SCD2 dimensions wrong behavior: same business identifiers on same load

Question

Hello,
We have the requisite (that I would consider normal on a DW project) of keeping history of all the changes that might occur to a specific field on the same day. This field loads into a scd type 2 attribute on a dimension.
Since I have daily loads on my DW, it could happen that, in the same load, different rows with the same business keys exist.
Consider the following scenario:
Table A: id (business identifier), event, dt_event
Table A loads into DIM_A: sk (surrogate key), id, event (trigger history), dt_begin(effective date), dt_end(expiration date)
First Load:
Table A:
1, 'OPEN_PROCESS', '01-06-2008 09:00:00'
1, 'CLOSE_PROCESS', '01-06-2008 15:00:00'
Dimension DIM_A:
1, 1, 'OPEN_PROCESS', '01-06-2008 09:00:00', '01-06-2008 08:59:59' <- Shouldn't be '01-06-2008 14:59:59'?
1, 1, 'CLOSE_PROCESS', '01-06-2008 15:00:00', NULL
Second Load:
Table A
1, 'REOPEN_PROCESS', '02-06-2008 14:00:00'
1, 'CLOSE_PROCESS', '02-06-2008 15:00:00'
Dimension DIM_A:
1, 1, 'OPEN_PROCESS', '01-06-2008 09:00:00', '01-06-2008 08:59:59'
2, 1, 'CLOSE_PROCESS', '01-06-2008 15:00:00', NULL
5, 1, 'REOPEN_PROCESS', '02-06-2008 14:00:00', '02-06-2008 13:59:59' <- Same issue with dates?
where's the new record? I should have 4 registers by now...
The row with sk=2 was updated twice on the last load: first it was updated with a new dt_end (as it should be) and next it was updated once again, keeping dt_end to a null value and sk=2 as my up to date record.
What I would like to have at the end of the second load is something like this:
1, 1, 'OPEN_PROCESS', '01-06-2008 09:00:00', '01-06-2008 14:59:59'
2, 1, 'CLOSE_PROCESS', '01-06-2008 15:00:00', '02-06-2008 13:59:59'
5, 1, 'REOPEN_PROCESS', '02-06-2008 14:00:00', '02-06-2008 14:59:59'
6, 1, 'CLOSE_PROCESS', '02-06-2008 15:00:00', NULL
What can I do to achieve this? Is this a problem with the dimension operator? Has anyone have any issues like this before?
Thank you for all your comments on this.
Regards,
Bruno

Answer 1

Thinking off the top of my head for a possible workaround, perhaps you could use an analytic function to partition the input data set
e.g.
1) Use something like ROW_NUMBER() OVER(PARTITION BY id ORDER BY dt_event ASC) to create a COLUMN data_partition
2) Add a filter to the DIMENSION load that will accept parameter P_DATA_PARTITION and have condition WHERE data_partition = P_DATA_PARTITION. This way you can run dimension load and load first set of changes, then second set etc.
3) Set up a process flow that will run dimension load X number of times with parameter P_DATA_PARTITION = 1 to X. Perhaps you can use the FOR LOOP operator in the Process Flow (I haven't used it so not sure)?
It's not ideal as you're running the same code several times (overhead) and also you need to fix the number of times it runs, unless you can use a function to determine this and pass into FOR LOOP.
Hope this is of some use.
Cheers
Si