Need Help on using CAS Incremental Crawl with JDBC data source

Question

Hi,
As part of one of the e-commerce implementations, we are implementing a delta pipeline which reads the data from the database views. We plan to crawl the data with the help of CAS JDBC data source. In order to optimize the data reads, we want to use the CAS capabilities of incremental crawl. We rely on CAS to be able to identify the updates available in the subsequent incremental crawl and only read the updated data for any given crawl unless we force a full crawl.
     We have tried implementing the above setup using JDBC data source. CAS reads from the database and stores it in the record store. The Full crawl works fine however; we observed some unexpected behavior during the incremental crawl run. Even when there is no change in the database, crawl metrics show that certain number of records have been updated in the record store and the number of updates differ in every subsequent run.
Any pointers what can be the issue ? Does CAS has the incremental crawl capability using JDBC data source ?
Regards,
Nitin Malhotra

Answer 1

Hi Ravi,
Generic Extraction is used to extract data from COPA tables. And delta method used to extract delta records (records created after initilization) is timestamp.
What is this timestamp?
Assume that, you have posted 10 records one after the other to the COPA tables. And we do initilization to move all these 10 records to BW system. Later another 5 records are added to COPA tables. How do you think the system identifies these new 5 records (delta records).?
It identifies based on a timestamp field (eg : Document created on ,a 16 digit decimal field).
Assume that, in our previous initilzation, say  "document created on" field for the last (or the latest) record is 14/11/2006, 18.00 hrs. and  timestamp is set to the 14/11/2006, 18.00hrs.  then when you do delta update, the system treats all the records with "document created on" field is greater than 14/11/2006, 18.00 hrs as delta records. This is how new 5 records are extracted to bw system and timestamp is again set to the new value besed in field in the records.(say 14/11/2006, 20.00hrs)
Assume that, you have two records with "document created on" field as 14/11/2006, 17.55hrs and 14/11/2006, 17.57hrs. and they were updated to the COPA table after 14/11/2006,20.00hrs (at this time last delta load is done)  due to some reason. How can you bring these two records? For this, purpose we can reset the timestamp at KEB5. In this eg, we can reset as 14/11/2006, 17.50 hrs) and do the delta again. So, the system picks all the records which are posted after 14/11/2006,17.50hrs again.  But remember that, doing this, you sending some of the records again (duplicate records). So, make sure that you sending ODS object. otherwise you end up with inconsistant data due to duplicate records.
Hope this helps!!
Vj