Incomplete Crawl of SQL Server Table Source

Question

We are currently experiencing a problem with a Database-based crawler in SES 10.1.8.2 where scheduled crawls produce different results than crawls that are executed manually. Crawler configuration is as follows:
- Database Source Type pointing at a rollup table in SQL Server.
- Rollup table is deleted and re-created each evening prior to SES indexing. Rollup table includes a "KEY" column that is the primary key.
- Schedule properties include a recrawl policy of "Process All Documents" and a crawling mode of "Automatically Accept All URLs for Indexing"
The symptom we are experiencing basically involves an incomplete crawl of the source. For example, a source table that contains 1,734 rows only translates into 567 indexed documents in SES. This coincides with the Browse counts when using the out-of-the-box version of SES search. If I then manually launch the schedule, the proper number of documents are processed and the Browse feature reflects the proper number.
One other piece of information is after examining the log for a scheduled crawl, a document that was not indexed does appear in the log as being queued, but we never see a log info line indicating it was processed.
The bottom line is a scheduled crawl fails to index all database table rows. A manual crawl consistently indexes everything properly.
Finally, we have made sure that we have minimal overlap of scheduled crawls in case this is a resource issue.
Any assistance would be greatly appreciated.

Answer 1

1. What is the use of the table compression?
Save disk space and sometimes also gain performance.
2. when do we need to compress the table ?
Need is maybe not the best word, but when we want to reduce disk space and/or make the performance gains, we would consider compression.
And, not to forget, if we are on Enterprise Edition. Compression is not available in other editions.
3. If i compress the table what will be the performance impact
There are two levels of compression: ROW and PAGE. ROW is basically a different storage format, which gives a more compact format for most data profiles. Not the least if you have plenty of fixed-length columns that are often NULL. ROW compression has a
fairly low CPU overhead. Since compression means that the data takes up less space, this means a scan of the full table will be faster. This is why you may gain performance.
Page compression is more aggressive and uses a dictionary. You can make a bigger gain in disk space, but the CPU overhead is fairly considerable, so it is less likely that you will make a net gain.
To find out how your system is affected, there is a stored procedure, of which I don't recall the name right now, which can give you estimated space savings. But if you also want to see the performance effects, you will need to run a test with your workload.
There is also columnstore, which also is a form a compression, and which for data warehouses can give enormous performance gains.
Erland Sommarskog, SQL Server MVP, [email protected]