Networking

Unix and Linux network configuration. Multiple network interfaces. Bridged NICs. High-availability network configurations.

Applications

Reviews of latest Unix and Linux software. Helpful tips for application support admins. Automating application support.

Data

Disk partitioning, filesystems, directories, and files. Volume management, logical volumes, HA filesystems. Backups and disaster recovery.

Monitoring

Distributed server monitoring. Server performance and capacity planning. Monitoring applications, network status and user activity.

Commands & Shells

Cool Unix shell commands and options. Command-line tools and application. Things every Unix sysadmin needs to know.

Home » Files

Deleting Lots of Files Quickly

Submitted by on November 17, 2015 – 9:03 am

I am not talking about hundreds or thousands of files. I am talking about hundreds of thousands. The usual “/bin/rm -rf *” may work, but will take a while. Or it may fail with the “argument list too long” error. So here are a few examples showing you how to delete many files in a hurry.

First, we need something to work with, like maybe a million empty files dumped into a single folder:

dir=/home/tmp ; mkdir -p ${dir} ; cd ${dir}
for i in `seq 1 1000000` ; do printf "file_${i}\n" ; done | xargs touch

Method 1: find + xargs (40 seconds)
time find ${dir} -type f | xargs -L 100 -P 100 /bin/rm -f

Method 2: find + delete (51 seconds)
time find ${dir} -type f -delete

Method 3: rsync (22 seconds)
mkdir /tmp/empty ; time rsync -a --delete-before /tmp/empty/ ${dir}/

Method 4: perl (23 seconds)
cd ${dir} ; time perl -e 'for(<*>){((stat)[9]<(unlink))}'

Method 5: rm (19 seconds)
time /bin/rm -f ${dir}

Moral of the story: if the number of files doesn’t exceed the argument size limit – don’t try to be clever and just use it.

Print Friendly, PDF & Email

Leave a Reply