Reconciliation between two sources

Question

I have two tables A and B. There are around 5 crore records in table A. In table B i have a column called id. In all there are 25 ids and there are 6000 records for each id. So in total there are 150000. I want to join all the 5 crore records of table A with all the records for each id in table B one by one. i.e. 5 crore join with 6000 of id 1; then 5 crore join with 6000 records of id 2 and so on... can u please suggest a solution with optimized performance as it is a performance critical case?

Answer 1

Unless your join is on columns that represent a unique key in atleast one of the tables, you will find that your result contains duplicates*.
This is because SQL is multiset based and not set based.
SQL is supposed to be declarative. In that if you want to join tables A, B on columns x,y,z then you just do so, and leave the optimiser to worry about the "optimal implementation".
We all know in practise that this isn't the case, and this is why there are always clowns writing bad code and well paid jobs to fix the circus SQL.
However in a join between two tables, especially one not involving unique keys, there is no logical optimisation, although you will additionally to DISTINCT your result which will degrade performance further.
I suggest you revisit your table design and try to get some unique constraints on there, so that the optimiser may be enabled to do it's job for you and you may leverage the innate power of the set.
*unless of course either of the tables is empty, or contains only one row. Or the join columns happen to represent an unenforced unique key, in which case you'd better get an actual constraint on there before someone breaks it.
Message was edited by:
The Flying Spontinalli