Bash scripting: lists and random things
Imagine you have an HPC cluster with a hundred compute nodes named node001-node100. The two commands below will help you generate a list of node names – either all name on one line or one name per line. Such a list can be useful as input in a “for” or “while” loops.
Node names on one line:
eval "echo node{`seq -w -s ',' 1 100`}"
Example:
[root@wil-dev-l04]# eval "echo node{`seq -w -s ',' 1 100`}" node001 node002 node003 ... node100
Node name one per line:
for i in `seq -w 1 100` ; do echo node$i ; done
Example:
[root@wil-dev-l04]# for i in `seq -w 1 100` ; do echo node$i ; done node001 node002 node003 ... node100
While we are on the subject of generating tediouly long lists of sequential things, here’s a quick example of generating every usable IP address for 192.168.0.1/24 subnet
for a in `seq 192 192` do for b in `seq 168 168` do for c in `seq 0 255` do for d in `seq 1 254` do echo "${a}.${b}.${c}.${d}" done done done done
Example:
[root@wil-dev-l04]# for a in `seq 192 192`; do for b in `seq 168 168`; do for c in `seq 0 255`; do for d in `seq 1 254`; do echo "${a}.${b}.${c}.${d}"; done; done; done; done 192.168.0.1 192.168.0.2 192.168.0.3 ... 192.168.255.254
You can also use shell brace expansion like so:
echo -e 192.168.{0..255}.{1..254}'\n'
This is simpler than the loop, however, this method should be avoided for very large ranges.
Here’s pointless but fun scripting exercise. First, we generate a directory structure going three levels deep and looking something like this:
level1_{1..10} ---level2_{1..10} ---level3_{1..10} -file{1..10}
So in the end we have 1111 directories and 10000 files.
target_dir="/tmp/source" ; mkdir -p "${target_dir}" for i in `seq 1 10` do mkdir -p "${target_dir}/level1_${i}" for j in `seq 1 10` do mkdir -p "${target_dir}/level1_${i}/level2_${j}" for k in `seq 1 10` do mkdir -p "${target_dir}/level1_${i}/level2_${j}/level3_${k}" for l in `seq 1 10` do touch "${target_dir}/level1_${i}/level2_${j}/level3_${k}/file_level3_${k}_${l}.file" done done done done
Next step is to configure an array of some common commands
set -a commands commands[0]='tar cvf archive_`expr $RANDOM % 1000`.tar' commands[1]="gzip -fr" commands[2]="/bin/rm -r" commands[3]="bzip2"
Now we run a “find” and for every file or directory execute a random command from the array
find "${target_dir}" | while read line ; do eval "${commands[$(echo `expr $RANDOM % 4`)]} `echo ${line}`" ; done
After this whole random mess is done, what we end up with is very different from the original 1111 directories and 10000 files:
[root@wil-dev-l04]# find . -type d | wc -l 530 [root@wil-dev-l04]# find . -type f | wc -l 4552 [root@wil-dev-l04]# find . -type f -name "*.tar" | wc -l 934 [root@wil-dev-l04]# find . -type f -name "*.gz" | wc -l 2562 [root@wil-dev-l04]# find . -type f -name "*.bz2" | wc -l 1056
What is the point of this? No point, just an exercise in randomness to sharpen our scripting skills.
In the IP sequencing script, {c} should be ${c}
Very useful scripts on this site… thank you!