Match Performance

Question

Hi,
I have just started working on the DQ component to find matching records between two sources. Source 1 has about 10K records while Source 2 has 1M rows. Matching needs to be done based on two varchar columns (names). I need to update a column on Source 1 rows based on the matches with Source 2.
When I run my job, it takes forever to complete. If Source 2 has less number of records, job finishes fast. There are no keys for me to split the matching process into break groups or select candidates for matching. All 1M rows could potentially contain the matching rows.
What is the best approach?
Thanks,
Kranthi

Answer 1

Hello Kranthi,
This is an interesting use case. Typically we don't come across cases in batch deduplication where there are no possible break keys. If possible, can you describe what these two sources contain and why all the rows in second source be potential match? Are there no address elements, birthdates, partial IDs etc?
Having said that there are a few things you can do.
1. Use compare table functionality: For this you need to know couple of things. Are there any duplicates within Source1 itself or Source2 itself? If these two sources by themselves are "duplicate free" or you really don't care about duplicates within those sources then you should setup the compare table such that only records from Source1 are compared to records from Source2. This will eliminate unnecessary comparison and for sure will improve performance.
2. Make sure compare buffer is large: This is a Match performace setting not directly exposed in Match editor. But can be found in DSConfig.txt in DSInstall/bin folder. Typically we don't recommend changing this because default is sufficient. But your case seems extreme. Open the dsconfig.txt and search for "MemoryInKBForComparisons". The default is 4096KB. Change this value to something large (based on available RAM), say 65536KB. This will ensure Match can process all the data in RAM instead of going to disk again and again for all the data. By the way, to verify if your compare buffer is large enough, look at the Match contribution report for this job (from Data Services Management Console, Data Quality Reports). If the size of the largest break group is more than the number of records that can fit into compare buffer, you need to increase that. But in your case you are not doing any breaking at all.
3. Use phonetic versions of names as break keys: One possible approach is to use phonetic keys as break keys. So you will have to load the source2 records into a staging table with original data as well as phonetic version of the two columns. There is a function double_metaphone to create phonetic keys. Then you use can use this staged data in Candidate Selection with substring of phonetic keys in the selection criteria. (Don't use full phonetic keys, you may loose some duplicates and it will defeat the purpose). So you first candidate select the possible duplicates based on say first two characters of the phonetic key. This will reduce the number of rows getting compared. But you use the actual name data in the match criteria (not the phonetic keys). I am trying to find if there is a blueprint available that could show you a sample, but that's the basic idea.
You should consider #1 and #2 irrespective of the break strategy. There may be other ways too, but first lets try the above.
Hope this helps,
Abhiram