This is a simple but annoying and persistent problem: text files edited on Windows and the uploaded to Unix acquire the “^M” character at the end of each line. In my experience, this little problem has been missed by armies of sysadmins and was the cause of massive production outages at some of the world’s largest companies.

In one case, some genius decided to edit NIS auto-mounter maps in Notepad and then put them back on the NIS master server via FTP. The new defective maps were quickly pushed to slave NIS servers across the enterprise, promptly bringing down production environments from Seattle to Sydney. It took many Unix admins a surprisingly long time to identify the problem.

The “^M” character is especially destructive when it finds its way into configuration files, scripts, password files, and various distributed access control lists. The first step is find affected files. The first example below will locate Shell script files in the current directory and let you know if any of them have “^M” problem.

find `pwd` -type f -regextype posix-extended -regex ".*\.(ba|k|c)?sh" | while read line
do
	if [ `cat -v "${line}" | grep -cE "\^M$"` -gt 0 ]
	then
		echo "${line}"
	fi
done

Similar to the example above, the following command will find and analyze configuration files.

find `pwd` -type f -regextype posix-extended -regex ".*\.c[o]?nf" | while read line
do
	if [ `cat -v "${line}" | grep -cE "\^M$"` -gt 0 ]
	then
		echo "${line}"
	fi
done

Removing “^M” is a bit more difficult than most people think. If you google the issue, you’ll find a lot bad advise along the lines of “sed -e “s/^M//” filename > newfilename” or “tr -d ‘^M’ < filename > newfilename”. These methods will only work if “^M” are printable, but in most cases these control characters will be hidden. One approach that is certain to work is to use “cat -v” like so:

cat -v filename | tr -d '^M$' > newfilename

We can combine this with the “while” loop above and add a couple of line to preserve ownership/permissions and make a backup copy of the original file:

find `pwd` -type f -regextype posix-extended -regex ".*\.(ba|k|c)?sh" | while read line
do
	if [ `cat -v "${line}" | grep -cE "\^M$"` -gt 0 ]
	then
		echo "Fixing ${line}"
		cat -v "${line}" | tr -d '^M$' > "${line}_ctrlmfixed"
		chown --reference="${line}" "${line}_ctrlmfixed"
		chmod --reference="${line}" "${line}_ctrlmfixed"
		/bin/mv -f "${line}" "${line}_`date +'%Y-%m-%d_%H%M%S'`"
		/bin/mv -f "${line}_ctrlmfixed" "${line}"
	fi
done

An alternative is to use “dos2unix” (yum -y install dos2unix) like so:

dos2unix < filename > newfilename

However, in my experience I ran into situations when for some reason dos2unix failed to remove DOS control characters.