Automatically Validate HTTP Proxies
Let’s say you downloaded a long list of Web proxy servers. Now you are stuck with the task of weeding out the proxies that are dead, slow, fake, or otherwise unusable. There are some applications out there that claim to validate and unblock proxy servers. The most common problem with these applications: they are excruciatingly slow. These apps also tend to get stuck once in a while. And, if your list of proxies is too long, these applications may crash altogether because of numerous memory leaks and other such examples of fine programming.
I would like to bring your attention to the following, hopefully, useful script that will go through some very long proxy lists in just a minute or two and will get rid of the trash. A few words about how it works are in order. I created a simple HTML page on my Web server (see $pvcurl variable below). This page contains a unique text string ($pvcstring variable).
The first step is to ping the proxy and see if it responds in a reasonable period of time. The ping commands are launched in background to speed up the process. If the proxy does respond, the next step is to use wget to see if you can download the $pvcurl and match the $pvcstring. If everything checks out, the proxy is added to the final list of good proxies. Just as the ping command, the wget threads are started in background mode with a 30-second timeout.
Download: proxy_validate.ksh
#!/bin/ksh
configure() {
pvcurl="https://www.krazyworks.com/pvc.html"
pvcstring="191628769290432845414226"
wget_timeout=30
proxyin="/tmp/proxylist.in"
if [ ! -f "$proxyin" ]
then
echo "Proxy list $proxyin not found. Exiting..."
exit 1
fi
proxyout="/root/proxylist.out"
if [ -f "$proxyout" ]
then
rm "$proxyout"
fi
}
cleanup() {
killall wget
for i in 1 2 3 4 5
do
if [ -f "/tmp/proxy_verify.tmp$i" ]
then
rm "/tmp/proxy_verify.tmp$i"
fi
done
}
wgetrun() {
if [ `wget -q --timeout=$wget_timeout --tries=1 -O - "$pvcurl" | grep -c "$pvcstring"` -eq 1 ]
then
echo "${proxy}:${port}" >> "$proxyout"
fi
}
pingrun() {
ping -q -c 1 -W 5 $proxy >/dev/null 2>&1
if [ $? -eq 0 ]
then
wgetrun &
fi
}
verify() {
sort "$proxyin" | uniq > "/tmp/proxy_verify.tmp1"
mv "/tmp/proxy_verify.tmp1" "$proxyin"
proxy_total=$(wc -l "$proxyin" | awk '{print $1}')
i=1
j=1
cat "$proxyin" | while read line
do
echo "Processing proxy $i of $proxy_total"
proxy=$(echo $line | awk -F':' '{print $1}')
port=$(echo $line | awk -F':' '{print $2}')
export http_proxy="${proxy}:${port}"
(( i = i + 1 ))
pingrun &
if [ $j -eq 100 ]
then
if [ `ps -ef | grep -c [w]get` -gt 100 ]
then
sleep $wget_timeout
killall wget
j=1
fi
else
(( j = j + 1 ))
fi
done
echo "Waiting for threads to finish ($wget_timeout seconds)..."
while [ `ps -ef | egrep -c "[w]get|[p]ing"` -gt 0 ]
do
sleep 5
done
}
# RUNTIME
configure
cleanup
verify
cleanup
6 Comments »
1 Pingbacks »
-
[…] a list of Web proxies. There are thousands of them and they are free to use. You can even grab a copy of a simple script I wrote that will quickly validate a huge list of proxies. A number of sites on […]
![wyryr1sn52gntzn65v4fpz5lc6v[1]](https://i2.wp.com/www.krazyworks.com/wp-content/uploads/2021/08/wyryr1sn52gntzn65v4fpz5lc6v1-80x80.jpg "Plundering Facebook Photo Albums")
![clfud46mxf2t5zph2l04zlpgl1d[1]](https://i2.wp.com/www.krazyworks.com/wp-content/uploads/2021/08/clfud46mxf2t5zph2l04zlpgl1d1-80x80.jpg "Scraping a Web Page in Bash")
![DSCF5361[1]](https://i1.wp.com/www.krazyworks.com/wp-content/uploads/2020/10/DSCF53611-80x80.jpg "Processing Videos with ffmpeg and Lightroom")
![Cat-Devouring-A-Bird-Pablo-Picasso[1]](https://i0.wp.com/www.krazyworks.com/wp-content/uploads/2020/10/Cat-Devouring-A-Bird-Pablo-Picasso1-80x80.jpg "Searching Twitter")
![91ebozy1e3b138waujy28agrngy[1]](https://i2.wp.com/www.krazyworks.com/wp-content/uploads/2021/08/91ebozy1e3b138waujy28agrngy1-80x80.jpg "Generating Random Text Files for Testing")
![iwoau71okwxqh4jv3k4ook32qps[1]](https://i2.wp.com/www.krazyworks.com/wp-content/uploads/2021/08/iwoau71okwxqh4jv3k4ook32qps1-80x80.jpg "Finding Cron Jobs")
![5gtysva4wkc8mzc1k3e66vaycxx[1]](https://i1.wp.com/www.krazyworks.com/wp-content/uploads/2021/08/5gtysva4wkc8mzc1k3e66vaycxx1-80x80.jpg "Understanding Memory Utilization in Linux")
![sy2zeb1vp2h8itlxfbbylw0jrm0[1]](https://i0.wp.com/www.krazyworks.com/wp-content/uploads/2019/12/sy2zeb1vp2h8itlxfbbylw0jrm01-80x80.jpg "Validating HTTPS Cache Peers for Squid")
![Photo-2019-10-07-14-39-03_0266[1]](https://i1.wp.com/www.krazyworks.com/wp-content/uploads/2019/11/Photo-2019-10-07-14-39-03_02661-80x80.jpg "Verifying SNMP Connectivity on Multiple Hosts")
Hi I have one year
I know that this letter is true and it is the menu that I’m ready to help
ATTENTION SIR,
Jordan International Bank Plc, London have acknowledged your contact information as the appointed representative and next beneficiary to Three Million, Two Hundred Thousand United States Dollar (USD $ 3.2 Million) deposited in our bank by late Dr.Adel Abolhassan,with Miss,Zainab Vanda Abolhassan,as the next-of-kin to it.
Miss. Zainab Vanda Abolhassan automatically inherits the said deposited fund with us as the next beneficiary because late Mr.Adel Abolhassan registered her name as the next-of-kin when the account was opened. The bank instructed her to appoint a trustee to act as proxy to her if she intends to reclaim the money now since her current status being a ward of the United Nations with refugee status restricts her from performing such transaction personally,
You are now requested to provide the bank with two legal authorizations for representative fund transfer called the (1A)Grant of Representation issued by Probate Registry and(2b) Change of Ownership Order from a legal attorney which must be legalized and validated by a Higher Court within the country where the next-of-kin presently resides for her to append signatory.
The Grant of Representation is mandatory to protect the interest of the bank, that of yours and Miss Zainab Vanda Abolhassan so that she can not testify in future that she did not authorize the transfer of her inheritance to your account. You are mandated to forward the requested documents without prolonged delay and provide the legal authorization documents must bear the signature of next-of-kin being Miss Zainab Vanda Abolhassan,
NOTE: On receiving the said demanded for legal authorizing documents for representative fund transfer, it will be verified after which we will credit the money to your provided Bank account through telex wire transfer which will reflect in your account after 48 Hours,
My internet service providor provide me pppoe(no user name no password & IP’s are automatically optained) internet service, can you tell me that how to share internet connection in another 2PC.
**I tried it by ICS,proxy, router, but no result
i am not sure what the hell is wrong with this site.
yesterday i bought a PREMIUM membership on megaupload and download megaupload manager,
then downloaded some files.
this morning i woke up and megaupload does not work, and says server can’t be found.
when i use the proxy (hide my asx) it works, but cannot download any file due to proxy.
looks like they just take the money then block you for not downloading any file, so they can save
some bandwidth for getting more PREMIUM users?
i live in Japan.
If there is any good people who knows a lot about computers or they are like a computer programmer please leave me their email. I want to know more about computers
ok well my friend got a new phone and didnt need her slvr l7 so she gave it to me. (we both have at&T) and so i put my chip in and stuff and i enter the unlock code and it says your number and my number comes up.. ok everything is fine.. BUT i have picture messaging on my plan and this phone wont send them or let me go on aim?? and sometimes a little in-call icon comes up on screen and im not in a call. also, i cannot acces the internet when it is in my plan and i can on other phones. can anyone help me?
Sometimes I get banned from a web site and most of the time it’s for silly reasons. Unfortunately, it’s very difficult to reset an ip address with windows 7. I tried entering the site through a proxy server, but the site still recognizes me. I’ve disabled cookies and the web site is in java. How does it still know my computer?
The web site is http://www.chat-avenue.com and the proxy server is http://www.freeproxyserver.net