Bash script for checking link status

Question

So I'm doing some SEO work I've been tasked with checking a couple hundred thousand back links for quality.  I found myself spending a lot of time navigating to sites that no longer existed.  I figured I would make a bash script that checks if the links are active first.  The problem is my script is slower than evolution.  I'm no bash guru or anything so I figured maybe I would see if there are some optimizations you folks can think of.  Here is what I am working with:
#!/bin/bash
while read line
do
#pull page source and grep for domain
curl -s "$line" | grep "example.com"
if [[ $? -eq 0 ]]
then
echo \"$line\",\"link active\" >> csv.csv else
echo \"$line\",\"REMOVED\" >> csv.csv
fi
done < <(cat links.txt)
Can you guys think of another way of doing this that might be quicker?  I realize the bottleneck is curl (as well as the speed of the remote server/dns servers) and that there isn't really a way around that.  Is there another tool or technique I could use within my script to speed up this process?
I will still have to go through the active links one by one and analyze by hand but I don't think there is a good way of doing this programmatically that wouldn't consume more time than doing it by hand (if it's even possible).
Thanks

Answer 1

I know it's been awhile but I've found myself in need of this yet again.  I've modified Xyne's script a little to work a little more consistently with my data.  The problem I'm running into now is that urllib doesn't accept IRIs.  No surprise there I suppose as it's urllib and not irilib.  Does anyone know of any libraries that will convert an IRI to a URI?  Here is the code I am working with:
#!/usr/bin/env python3
from threading import Thread
from queue import Queue
from urllib.request import Request, urlopen
from urllib.error import URLError
import csv
import sys
import argparse
parser = argparse.ArgumentParser(description='Check list of URLs for existence of link in html')
parser.add_argument('-d','--domain', help='The domain you would like to search for a link to', required=True)
parser.add_argument('-i','--input', help='Text file with list of URLs to check', required=True)
parser.add_argument('-o','--output', help='Named of csv to output results to', required=True)
parser.add_argument('-v','--verbose', help='Display URLs and statuses in the terminal', required=False, action='store_true')
ARGS = vars(parser.parse_args())
INFILE = ARGS['input']
OUTFILE = ARGS['output']
DOMAIN = ARGS['domain']
REMOVED = 'REMOVED'
EXISTS = 'EXISTS'
NUMBER_OF_WORKERS = 50
#Workers
def worker(input_queue, output_queue):
while True:
url = input_queue.get()
if url is None:
input_queue.task_done()
input_queue.put(None)
break
request = Request(url)
try:
response = urlopen(request)
html = str(response.read())
if DOMAIN in html:
status = EXISTS
else:
status = REMOVED
except URLError:
status = REMOVED
output_queue.put((url, status))
input_queue.task_done()
#Queues
input_queue = Queue()
output_queue = Queue()
#Create threads
for i in range(NUMBER_OF_WORKERS):
t = Thread(target=worker, args=(input_queue, output_queue))
t.daemon = True
t.start()
#Populate input queue
number_of_urls = 0
with open(INFILE, 'r') as f:
for line in f:
input_queue.put(line.strip())
number_of_urls += 1
input_queue.put(None)
#Write URL and Status to csv file
with open(OUTFILE, 'a') as f:
c = csv.writer(f, delimiter=',', quotechar='"')
for i in range(number_of_urls):
url, status = output_queue.get()
if ARGS['verbose']:
print('{}\n {}'.format(url, status))
c.writerow((url, status))
output_queue.task_done()
input_queue.get()
input_queue.task_done()
input_queue.join()
output_queue.join()
The problem seems to be when I use urlopen
response = urlopen(request)
with a URL like http://www.rafo.co.il/בר-פאלי-p1
urlopen fails with an error like this:
Exception in thread Thread-19:
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/threading.py", line 639, in _bootstrap_inner
self.run()
File "/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/threading.py", line 596, in run
self._target(*self._args, **self._kwargs)
File "./linkcheck.py", line 35, in worker
response = urlopen(request)
File "/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 160, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 473, in open
response = self._open(req, data)
File "/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 491, in _open
'_open', req)
File "/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 451, in _call_chain
result = func(*args)
File "/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 1272, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 1252, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/http/client.py", line 1049, in request
self._send_request(method, url, body, headers)
File "/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/http/client.py", line 1077, in _send_request
self.putrequest(method, url, **skips)
File "/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/http/client.py", line 941, in putrequest
self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 5-8: ordinal not in range(128)
I'm not too familiar with how character encoding works so I'm not sure where to start.  What would be a quick and dirty way (if one exists) to get URLs like this to play nicely with python's urlopen?