Large Java Cache Solutions

Question

I am implementing a web crawler which is expected to create an index of approximately 100,000 webpages. My current implementation has worked fine until I approached an index of approximately 30,000 webpages. The problem is being able to supply several web crawler "clients" with urls to crawl at a rate faster than the "clients" can actually crawl the webpage. Currently my database of urls to crawl is getting emptied by crawler clients at a faster rate than I can repopulate it.
The problem is for each new url I find, I need to check it against the entire index of "crawled urls" to ensure its a "new" url (I dont want to crawl a url more than once). My current implementation uses db4o and each look up, with a database of 30,000 urls, takes approx 0.8 seconds.
My question is, would it be better to keep a large scale cache of "crawled urls", rather than querying db4o each time I need to check if a url has been previously crawled?
I have found two possible cache systems:
http://jakarta.apache.org/jcs/index.html
and
http://ehcache.sourceforge.net/
Before I go about trying to work out how these systems work I just need to get some opinions on whether this is a correct solution to my problem.
Also if anyone has had any previous experience with these, or similar, systems could you point me in the direction of a good "learners" tutorial! They both seem rather complexat first glance.
Thanks

Answer 1

maybe try hashing the url down to a 16ish bit integer, then when you're looking for a url, first hash it, select everything in your table with that integer hash, and then search that subset for your url.
Like,
SELECT * FROM (SELECT * FROM urls WHERE urls.hash=myhashfunction('www.someurl.com')) WHERE urls.name='www.someurl.com';Instead of the embedded select statement you could perhaps have a stored view, and you may want to have the hash function be a client-side thing instead of a stored function.
I'm not really into database optimization though.
EDIT: to clarify, your insert would be:
INSERT INTO urls (hash,name) VALUES (myhashfunction('www.someurl.com'), 'www.someurl.com');Edited by: endasil on Oct 17, 2007 6:23 AM