Recently I ran into a small problem: I needed to find recently-modified files in a very large NFS filesystem. One of the high-level folders contained dozens of sub-folders with thousands of files in each. There is a significant performance penalty associated with placing such directory structures on network-mounted filesystems. Running the “find” command at the top of the filesystem would have taken over an hour and the problem here was not available bandwidth, but the time it takes for the “find” to request and receive attribute information for each folder and file.

The solution was to kick off multiple instances of “find” in parallel – one for each sub-folder. Each “find” thread would append its search results to the same text file, making it easy to monitor the progress. As you will see from the “top” output shown below, this approach puts the server under considerable load and should not be attempted on a production system.

Define to the top-level directory that contains multiple folders:

[root@server ~]# fs="/vol/big_nas_filesystem"

[root@server ~]# df -hP "${fs}"
Filesystem            Size  Used Avail Use% Mounted on
vnx:"${fs}"/  7.5T  6.6T  919G  88% "${fs}"

[root@server]# ls "${fs}"
00  10  20  30  40  50  60  70  80  90  a0  b0  c0  d0  e0  f0
01  11  21  31  41  51  61  71  81  91  a1  b1  c1  d1  e1  f1
02  12  22  32  42  52  62  72  82  92  a2  b2  c2  d2  e2  f2
03  13  23  33  43  53  63  73  83  93  a3  b3  c3  d3  e3  f3
04  14  24  34  44  54  64  74  84  94  a4  b4  c4  d4  e4  f4
05  15  25  35  45  55  65  75  85  95  a5  b5  c5  d5  e5  f5
06  16  26  36  46  56  66  76  86  96  a6  b6  c6  d6  e6  f6
07  17  27  37  47  57  67  77  87  97  a7  b7  c7  d7  e7  f7
08  18  28  38  48  58  68  78  88  98  a8  b8  c8  d8  e8  f8
09  19  29  39  49  59  69  79  89  99  a9  b9  c9  d9  e9  f9
0a  1a  2a  3a  4a  5a  6a  7a  8a  9a  aa  ba  ca  da  ea  fa
0b  1b  2b  3b  4b  5b  6b  7b  8b  9b  ab  bb  cb  db  eb  fb
0c  1c  2c  3c  4c  5c  6c  7c  8c  9c  ac  bc  cc  dc  ec  fc
0d  1d  2d  3d  4d  5d  6d  7d  8d  9d  ad  bd  cd  dd  ed  fd
0e  1e  2e  3e  4e  5e  6e  7e  8e  9e  ae  be  ce  de  ee  fe
0f  1f  2f  3f  4f  5f  6f  7f  8f  9f  af  bf  cf  df  ef  ff

Run a bunch of “find” threads in background, all outputting results into the same file:

[root@server]# find "${fs}" -maxdepth 1 -type d -print | grep -v "`echo ${fs}$ | awk -F'/' '{print $NF}'`" | while read line
do
    nohup find "${line}" -mtime -1 -print | tee -a /tmp/find_results.txt &
done

Run “top” to see just how hosed your server is:

top - 14:52:18 up 10 days, 23:02,  2 users,  load average: 206.50, 81.36, 30.95
Tasks: 884 total,   1 running, 883 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.8%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  12184028k total,  3987284k used,  8196744k free,    77076k buffers
Swap:  8388600k total,        0k used,  8388600k free,   151216k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5350 root      20   0 13772 1800  876 R  1.0  0.0   0:00.24 top
 4838 root      20   0  109m 1080  812 D  0.3  0.0   0:00.08 find
 4848 root      20   0  109m 1072  812 D  0.3  0.0   0:00.06 find
 4860 root      20   0  109m 1084  812 D  0.3  0.0   0:00.07 find
 ...
 5078 root      20   0  109m 1088  812 D  0.3  0.0   0:00.07 find
 5099 root      20   0  109m 1100  812 D  0.3  0.0   0:00.08 find
 5107 root      20   0  109m 1088  812 D  0.3  0.0   0:00.05 find
 5127 root      20   0  109m 1092  812 D  0.3  0.0   0:00.07 find
 ...

And keep an eye on the output file:

[root@server]# tail -f /tmp/find_results.txt