Understanding Crawl Statistics

Question

Hi,
I set up SES web crawler to crawl some internal sites. At the end of the crawl I received following statistics:
Total number of documents fetched = 982
Total number of documents updated = 1,051
Document fetch failures = 13
Document conversion failures = 0
Total number of unique documents indexed = 56
Total data collected = 4,787,947 bytes
Total number of non-indexable documents = 926
Average size of fetched document = 85,499 bytes
Documents discovered = 1313
Documents rejected = 318
I could not understand:
1. Why certain number of documents were non indexable (as crawler has already rejected certain documents)
2. What is the difference between 'Document fetch failure' and 'documents rejected' (Documents rejected depends on certain types of url's not accepted by crawler)
3. What is the criterion for document rejection (I know a few images, audio, video are not accepted)? Is there any other criterion? I mean can a crawler plug in implementation has this capability and is it expected from crwler plugin to do so? I know while creating a source we can define what all url's to reject.
4. In crawler logs I get error like: "EQG-30173: 'Document url' Error reading document content" ? What could be the possible reason for this?
Thanks in advance.
Shakti

Answer 1

Non-indexable documents are generally formats which we don't index (eg jpg, gif files).
Rejected documents are usually those which are outside the boundary conditions for the crawl.
For example if you start a crawl at http://www.oracle.com/ then any links to (for example http://www.peoplesoft.com/ or even http://files.oracle.com/ will be outside the boundary. Check the second tab on the edit source pages.
The EQG-30173 error normally means that the web server has failed to respond, or has sent invalid information. This should correspond with the "Document fetch failures" - 13 documents in this case.