Google Images is an extremely useful tool for webmasters, designers, editors, and just about anybody else who’s in a hurry to find just the right photo or clipart. However, this Google tool has a couple of annoying issues. First, the collection of images is updated once in a while. Not very often. This means that whenever you get your search results, there will be a bunch of dead links. Second, many Web sites prevent direct linking of images, so you cannot open just the image – you have to open the entire Web page where the image is located.

The idea behind the following BASH shell is to address these two small problems, as well as to save you some time downloading images. The script reads from a list of keywords formatted exactly as you would format your Google search query. For example, to find a bunch of photos of the Red Square in Moscow you would type something like: +”red square” +moscow. This will search for “red square” as a phrase plus the word “moscow”.

The search query is passed to Google as a URL along with the desired image size (icon, small, medium, large, xlarge, xxlarge). If you are looking only for the largest photos, you would specify the image size as “xxlarge”. However, if you are looking for image size from “large” and up, you would use the “pipe” character in place of a logical “OR”: large|xlarge|xxlarge. See the script below for an example.

The following step for the script is to count the number of search hits and download all Google pages listing the search results. A search result may say something like: “found 250 matches, displaying images 1-20”. Once again, the script formats an appropriate URL and sends it to Google to get search results 21-40, 41-60, etc. until it reaches 250.

All the search results are parsed to extract only the image URLs linking to the actual photos on various sites. These URLs are dumped into a file and fed to Wget. The direct linking issue is no longer a concern, because you open each image URL directly and your “HTTP Referrer” is blank. The downloaded images are written to a separate directory for each search query. A minimum file size filter is applied to the downloaded photos: all images smaller than, say, 30Kb are deleted. ImageMagick is used to enlarge images if they are smaller than a certain size.

You will need to create a file called “keywords.txt” in the same directory as the script. The file should contain one search query per line. Here’s an example:

+"red square" +moscow
+"times square" +"new york"
+bush +iraq +wmd +"big fat liar"

And here’s the script:

#!/bin/bash

# Collect and resize images from Google Images based on a keyword.

#------------------------------
# VARIABLES
#------------------------------

DATETIME=$(date +'%Y-%m-%d')
HOMEDIR="/WD120GB_01/misc/google"
SIZE="large|xlarge|xxlarge"
#SIZE="icon|small|medium|large|xlarge|xxlarge"

#------------------------------
# CONFIGURATION
#------------------------------

if [ ! -d "$HOMEDIR" ]
then
        echo "Home directory $HOMEDIR not found. Exiting..."
        exit 1
fi

if [ ! -f "${HOMEDIR}/keywords.txt" ]
then
        echo "Keyword list not found. Exiting..."
        exit 1
fi

#------------------------------
# IMAGE SEARCH & DOWNLOAD
#------------------------------

STATUS=1

i=0
cat "${HOMEDIR}/keywords.txt" | while read KEYWORD
do
        OUTDIR=$(echo "$KEYWORD" | sed 's/ /_/g' | sed 's/+/_/g' | sed "s/'//g" | sed 's/"//g' | sed 's/*//g' | sed 's/^_//g' | sed 's/__/_/g')
        OUTDIR=$(echo "${HOMEDIR}/${OUTDIR}_${DATETIME}")

        if [ ! -d "${OUTDIR}" ]
        then
                mkdir "${OUTDIR}"
        else
                echo "Directory ${OUTDIR} already exists. Skipping..."
                STATUS=0
        fi

        if [ $STATUS -eq 1 ]
        then
                URL="http://images.google.com/images?q=${KEYWORD}&svnum=100&hl=en&lr=&safe=off&sa=G&imgsz=${SIZE}"
                wget -U Mozilla -O "${OUTDIR}/results_${i}.txt" "$URL" -e robots=off

                cat "${OUTDIR}/results_${i}.txt" | sed 's/href/n/g' | grep imgurl | grep imgrefurl | sed 's/imgurl=/@/g' | sed 's/&imgrefurl/@/g' | awk -F'@' '{print $2}' > "${OUTDIR}/image_urls.txt"
                results=$(cat "${OUTDIR}/results_${i}.txt" | sed 's/border/n/g' | fgrep '&start=' | fgrep -v '&start=0' | sort | uniq | fgrep '
' | wc -l | awk '{print $1}')

                i=1
                while [ $i -lt $results ]
                do
                        (( START = i * 20 ))
                        URL="http://images.google.com/images?q=${KEYWORD}&svnum=100&hl=en&lr=&safe=off&sa=G&imgsz=${SIZE}&start=${START}"
                        wget -U Mozilla -O "${OUTDIR}/results_${i}.txt" "$URL" -e robots=off
                        cat "${OUTDIR}/results_${i}.txt" | sed 's/href/n/g' | grep imgurl | grep imgrefurl | sed 's/imgurl=/@/g' | sed 's/&imgrefurl/@/g' | awk -F'@' '{print $2}' >> "${OUTDIR}/image_urls.txt"
                        (( i = i + 1 ))
                done

                find "$OUTDIR" -type f -name "results_*.txt" -exec rm {} ;
                cat "${OUTDIR}/image_urls.txt" | fgrep '.jpg' | sort | uniq > /tmp/google_image_collector.tmp
                mv /tmp/google_image_collector.tmp "${OUTDIR}/image_urls.txt"

                if [ -f "${OUTDIR}/image_urls.txt" ]
                then
                        clear
                        COUNT=$(cat "${OUTDIR}/image_urls.txt" | wc -l | awk '{print $1}')
                        echo "Found $COUNT images matching $KEYWORD"

                        j=1
                        cat "${OUTDIR}/image_urls.txt" | while read LINE
                        do
                                wget -U Mozilla -nd -t 1 -T 5 -O "${OUTDIR}/photo_${j}.jpg" "$LINE" -e robots=off

                                if [ `ls -als "${OUTDIR}/photo_${j}.jpg" | awk '{print $6}'` -lt 10000 ]
                                then
                                        rm "${OUTDIR}/photo_${j}.jpg"
                                else
                                        convert -filter Cubic -resize '640x640< ' "${OUTDIR}/photo_${j}.jpg" "${OUTDIR}/photo_${j}.jpg"
                                        (( j = j + 1 ))
                                fi
                        done
                fi
        fi
done