Control M or Ctrl-M or ^M character whatever you call them creates the problem when present in Unix or Linux text files. The presence of these characters may lead to unexpected results or interrupt the working of shell script, configuration files, etc.
Unix /Linux is the most widely used operating system for server and database hosting. Developers create code, text scripts on Windows, and move those to Unix environment where script execution fails with Command not found andProgram exited with status 127 error.
What is ^M character?
Ctrl M or ^M character is the carriage return character. Those come in the file because of different line termination characters used by Unix and Windows/DOS operating systems. Unix uses line feed (LF) while windows use both carriage return (CR) and line feed (LF) as termination characters.
This carriage return (CR) character becomes ^M character in Unix/Linux.
See the below image for details.
Want to know more about line termination characters, just head on to Wikipedia.
Find ^M character in text files
You can use text editors like Notepad, Notepad++, Textpad, VI, or even Unix commands like cat/grep to find out the control M character in a text file.
Below we have covered all the methods,
1. Using Notepad++ text editor to find ^m character
Open the file in Notepad++. Click on View -> Show Symbol -> Show All Characters or click this “¶”menu on the toolbar to display all the characters including CR and LF. As mentioned above Unix only uses LF while Windows uses CRLF as a line termination character. While moving the Windows file to Unix, you need to remove this CR character.
2. cat -v command
On Unix/ Linux operating systems, you can use cat command with the -v option to display non-printing characters including ^M on the standard output as shown below.
$ cat -v texthost.progecho "hi how are you"^Mls^M^M
3. grep command
grep command in Linux/Unix allows you to search a string in a file. So run the grep command on the file as shown below to find out and display all the lines where the ^M character is present.
Note:-
To type “^M” – click Ctrl+V and Ctrl+M i.e you can hold the CTRL key and press V and M sequentially. V should be first.
$ grep ^M texthost.progecho "hi how are you"ls
4. vi text editor
vi is a screen-oriented text editor originally created for the Unix operating system. -b option available in the VI Editor displays this particular character.
Simply run vi -b <file name> to reveal ^M on the screen as shown below.
$ vi -b texthost.progecho "hi how are you"^Mls^M^M
Avoid ^M character in text files
You can use text editors like Notepad++ and TextPad to create files specific to the respective operating system. FTP software also helps to do explicit character conversion. Let’s discuss all these methods in detail and see how you can avoid Ctrl ^M character from a file.
1. Notepad++ or Textpad editor
Editors like Notepad++ or TextPad generally have a built-in option to save the file in Unix or Windows format. Depending on the target operating system you should save the file to avoid the character issue.
For Notepad++, navigate to Edit->EOL Conversion and choose the respective file format. Here you need to choose UNIX/OSX format.
2. FTP file in ASCII mode
sFTP or FTP software like WinSCP, FileZilla provides options to transfer files using Binary or ASCII mode. Always, transfer text files in ASCII mode from Windows to Unit or vice-versa to avoid this ^M character issue as these tools do explicit character conversion as per the destination operating system.
The ASCII Mode does explicit character conversion. Windows format file is automatically converted to Unix format file and vice versa. Binary Mode moves the file as is with all properties.
Remove ^M character in text files
The best is to avoid it using the method mentioned above, but if you still end up with it, then use the below methods to remove the ^M character.
1. Explicit character conversion using dos2unix and unix2dos
Unix operating system like Solaris provides built-in command/utility dos2unix and unix2dos to convert file from Unix to DOS and vice versa.
dos2unix converts the file from DOS to UNIX format and unix2dos converts the file from UNIX to DOS format. This command replaces CRLF to LF and LF to CRLF respectively.
This is the way you can use it.
dos2unix dos_format.txt unix_format.txtorunix2dos unix_format.txt dos_format.txt
2. vi editor
Use the VI editor in binary mode to search and replace the ^M character with null in a text file as shown in the below code snippet.
$ vi -b texthost.progecho "hi how are you"^Mls^M^MPress [ESC] Key and type below including : (do not copy type it):%s/^M//g:wq$ cat -v texthost.progecho "hi how are you"ls
3. sed command
You can also use the stream editor (sed) to find and replace ^M with space as shown in the below example.
$ sed -e "s/^M//g" texthost.prog > texthost.prog2$ cat -v texthost.prog2echo "hi how are you"lsmv texthost.prog2 texthost.prog
4. Perl
You can use the Perl search and replace to remove ^M and replace it with blank space. Check the below example.
$perl -pe 's/^M//g' texthost.prog > dummy$ cat -v dummyecho "hi how are you"lsmv dummy texthost.prog
If you do not want to use the mv command, run below
perl -i -pe 's/^M//g' texthost.prog
You can also create a backup of the original file using the -i.bak option.
$perl -i.bak -pe 's/^M//g' texthost.prog$ cat -v texthost.progecho "hi how are you"ls$ cat -v texthost.prog.bakecho "hi how are you"^Mls^M^M
5. tr command
Linux tr command with ‘-d’ option deletes the pattern instead of translating. Here, we delete the ‘\r’ (non-printable) character.
tr -d '\r' < texthost.prog >texthost.prog2
Summary
In this article, we have discussed the ^M character, why it appears in Unix/Linux text files, and how to find and remove it using various command-line tools. By understanding and applying these techniques, you can ensure the proper formatting of your text files and prevent potential issues when processing them on Unix/Linux systems.
Do let me know if I missed any.
You could also follow the discussion on StackExchange or StackOverflow for further details.
