How to prevent duplicate keys in archive database?

I am struggling with this problem.
Background: I'm working on a project where I have to make an archive database. The archive database should get
all data of the operational database. It should even save every update ever made, so it literally contains the entire history of the operational database (this is a must and the whole project revolves around this idea). So this is solved by using Change Data
Capture. After that the data should go through a staging area and eventually in the data warehouse database. I came out with a solution and worked it out in the prototype and it seemed to be working all fine. I stupidly forgot to include the foreign keys,
so the archive database didn't have the original structure but it should ofcourse (no wonder it went okay without too much hassle).
Problem: Because we want to store everything in archive, there will be duplicate primary keys (just for instance,
many same contact_id's because telephone number changes a couple of times). I thought to solve this by adding a new primary key which says is auto-increment and purely exist to make a record unique. But when it comes to foreign keys, it's impossible. You want
contact_id to be able to be duplicate and in that case it cannot be a primary key. But foreign key is only able to reference to a primary key or another unique key but not other normal columns.
Any advice on this? It's an absolute must to store all changes.

All of you, thanks for replying, I'm happy you're trying to help me out with this problem. 
Visakh and Louis, thanks that seems like the solution for this case indeed. Yes, the dimensional design appeals more to me as well.
I read the articles and watched some tutorials. But I can't work it around the solution that I had.
More background info: I use CDC to track all the changes done in the operational database and SSIS (following one of Matt Mason's tutorials and with a lot of alterations to make it fit for my project). I have this control flow (don't mind that
error haha):
(Oh apparently I cannot add images yet, so here's the link for the screenshot:) http://nl.tinypic.com/r/w0p1u0/8
Basically I create staging tables in my archive database next to my normal archive tables. Then start CDC control task to get the processing range and then it copies everything from the operational database (joined with a few CDC columns) to the staging
tables. After that the processing range ends so it will only get the rows it hasn't processed before. And then I do some updates on the staging tables and then finally insert everything into the archive tables. The staging tables then can be truncated. After
this the data will go to the staging area for transformations and then finally at last to the DWH. The reason for having a staging area between the archive and DWH is that the archive will not only be used as source for the DWH but also on it's own. The DWH
will not contain 100% the same stuff as the archive (like maybe some transformations, extra columns with calculated fields, plus some columns don't need to be in the DWH at all). When all the ETL stuff is done in SSIS, I have to use SSAS to define all the
facts, dimensions, cubes. 
Example: So I try to work with the SCD type 2. If I understood it correctly (and maybe I didn't): for example, the contact table in archive should have the surrogate key ID (the auto-increment one). The business key is the contact_id
and can be used uniquely with the time range columns. 
Following Visakh's post, the ID becomes the key that the foreign key will reference to. For example: 
Contact table:
ID: 1 contact_id: 100
Name: Glenn start_time: 2014-01-01
End_time: 2014-08-20
ID: 2 Contact_id: 100
Name: Danzig Start_time: 2014-08-20
end_time: NULL
Sorry, I couldn't style it as table somehow. So the employee changed his name. It makes sense that the time period tells when the first name was valid. 
Organisation table: 
ID: 1
org_id: 20 
Contact_id: 1
Start_time: 2014-01-01
End_time:NULL
(it references to ID instead of contact_id as suggested)
The employee belongs to an organisation. It references 1 which is still old data. But this is the last version of the organisation record. 
So then I need a table to link the 2: 
organisation_contact table
contact_id:100
org_id: 20
and then I need another one to join with the surrogate key?
ID: 1
org_id: 20
ID: 2
org_id: 20
(Guess it would make more sense to have org_id in the contact table but for now it's an example)
Problems: I don't quite understand how this works. From the example I saw you have to have another table (the fact table) to link it to the surrogate key. Would this mean I have to have facts and dimension tables in my archive database?
My intention was actually to have all records of the operational databases (all the updates too) in my archive. And after that create the facts and dimensions in the DWH with SSAS. The example looks like I should do it earlier. 
I don't know how to combine this with the cdc solution. I want to get all the data by using CDC. Like how every update gets registered in the accompanying CDC table. Then the archive will get the CDC data. But then how to combine this in use with SCD. I
have the surrogate key in archive (ID) and then I make the start and end time columns. I need to point all references to the ID and then make the other table to keep track of the contact_id (original PK) and another key. At last make another table to track
all the current data in the fact. 
Another question: Would you recommend the SCD task in SSIS. I read it was not that great if you have many rows to work with. What would you think is the best method to implement it. 
Thanks so much again.
EDIT: What about slowly changing dimensions type 4? It looks like you don't have to change the references of the foreign key then. Why do you prefer 2 over 4?

Similar Messages

  • How to display duplicate key in web report?

    Hi Experts,
    Can anybody tell me how to display duplicate key in web report?
    I know in the Bex analyzer, we can allow the duplicate key to be dispalyed via the 'query property' by right click, and in the 'display option' tab, there is an option which named 'forbid duplicate key', if we don't select this option, the duplicate key will be dispalyed in the report result in BEx analyzer.
    But how can I do this in Web report? Thanks in advance.
    Eileen

    Hi,
    <b>I know in the Bex analyzer, we can allow the duplicate key to be dispalyed via the 'query property' by right click, and in the 'display option' tab, there is an option which named 'forbid duplicate key', if we don't select this option, the duplicate key will be dispalyed in the report result in BEx analyzer.</b>
    Do the same and execute report in web.You can able to see the same.
    Cheers
    Karthik

  • How to prevent duplicate web pages from loading

    <blockquote>Locking duplicate thread.<br>
    Please continue here: [[/questions/930219]]</blockquote>
    how to prevent duplicate web pages from loading

    <s>Hi berternie, can you describe this in more detail?
    Are you saying the identical page loads in two different tabs? When does that happen -- when you click a link? or when you use a bookmark?
    Or do you have multiple tabs open every time you see your home page (i.e., when you start up, open a new window, or click the home icon)?</s>
    I see, you have more info in this thread: https://support.mozilla.org/en-US/questions/930219

  • How to Prevent duplicates on Combination of Lookup columns in sharepoint 2010 using infopath 2010 form.

    Hi All,
    I have list with some Lookup columns like  City, Pin, and Text Column Name. All these are required columns.
    Now I want to prevent duplicates while submitting InfoPath form if a Combination of  City,Pin & Name. (like a Composite primary in Database is used.)
    Can some one help me on how to achieve this using InfoPath  2010 Rules, writing  rule in Xpath.
    Thanks in Advance.

    1. Add a secondary data connection to the list where the form will be submitted.
    2. Prior to submit via rules, set the query fields in the above connection: City, Pin & Name with values entered in the form. Query the data source and check if the result has values.
    3. Show error messages accordingly if exists else continue with Submit.
    This post is my own opinion and does not necessarily reflect the opinion or view of Slalom.

  • How to prevent duplicate submission?

    Hello!
    I have a page for gathering some data:
    private function saveHander():void {
        // Gathering data into database.
    <mx:TextInput id="name" />
    <mx:TextInput id="age" />
    <s:Button id="Submit" label="Submit" click="saveHandler();" />
    When focus on submit button,I click at the button and press the space bar at the same time,saveHander function execute two times.
    How can I prevent duplicate submission? thanks!

    Assuming this is a form you are attempting to save, good UX practice is to disable the form once you begin local form validation (submit button clicked) prior to forming your data packet for save.
    Doing so will prevent the anomaly you are experiencing.
    In other cases where the above practice is not practical, you can setFocus to another component (off the submit button) because as long as the focus remains on the button, depressing the space bar will trigger the mouse event for the button. Research ADA compliance if you want more details.
    HTH. 

  • How to prevent duplicate records

    Hello all,
    This is just a general query. How do we prevent duplicate records at data target level(ODS/InfoCube).
    Thanks
    S N

    Hi,
    for ODS you can either specify in settings option "unique records" (only first combination of keys is saved, second one throws error message) or you can define "overwrite" for update rules to ODS (last occurrence of record is saved).
    For InfoCubes I think there isn't such setting and there is no possibility of overwriting existing records. So you have to load first to an ODS object to ensure unique records and update from ODS to InfoCube.
    Best regards,
    Björn

  • How to store duplicate keys in HashMap

    Hi,
    sun guys,
    please any one guide me how to store duplicate values in the haspmap.
    i think we need override the equals and hash code methods. am i right??
    if so guide me how to do it??
    thanks in advance,
    nagaraju.

    uppala wrote:
    Hi,Hi,
    sun guys,Most of the people who answer questions here, aren't from Sun.
    please any one guide me how to store duplicate values in the haspmap.
    i think we need override the equals and hash code methods. am i right??
    if so guide me how to do it??Associate the key with a list or a set, and place the values in that list/set.
    A put will then mean that you first call get to see if a list already is associated with the key. Place the value in the list in that case. Otherwise create a new list. Place the value in the list, and then call put with the list as value.
    Kaj

  • How to prevent duplicates in your contacts

    When syncing two iphones via icloud with your computer how do I prevent duplicate contacts between both phones, icloud and yahoo?

    I have 11 people designated under faces.
    Do you see the 11 People when using the "Faces" view from the source list in the Library Inspector or from the "Faces" button in the toolbar?
    If you are using the Faces button from the toolbar, you may have been excluding faces stacks you already created. Then, when you enter search strings, Aperture will suggest names from your contacts and you may create the same stack again.
    Also, if the search field is not empty, you may already have more "Faces" stacks than you are actually seeing.
    As you certainly know, you can always unite "Faces" stacks by dragging a stack onto the other to merge them. The merged stack will use the name of the stack you dragged to.
    Regards
    Léonie

  • How to prevent duplicate web pages loading

    when clicking on a link, two duplicate web pages load, how to prevent this

    This started when you installed Firefox 13? Which version did you have before -- Firefox 12 or Firefox 3.6?
    Does this occur on all sites or only on particular sites?
    Do you have a very sensitive mouse or touchpad that might be double-clicking the link?

  • How to prevent duplicate entry in Details block

    Dear All
    I am using Forms 10g.
    I have a detail block.
    There is a column called Ip_Address.
    There i want to prevent duplicate entry .
    How can i do this ?

    hey i have a requirement that to restrict duplicate entry in both block(both multi record).the blocks are DEPT(MATER)
    EMP(DETAIL)
    the associated fieds in master block DEPT.DEPT_NO,in detail EMP.EMP_ID .
    I have done' Kevin D Clarke’s calculated item ' both in master and detail block
    its working fine for master block but in case of detail block its only respond to the current session(i.e. if the new value is inserted or save it will restrict another record to be inserted of that last record's value),it does not restrict duplicate value enter ,checking with other existing records(which are saved in earlier session,though after query it is shown on the form)
    can anyone guide me why its not working

  • How to prevent duplicate vendor reference no on SAP B1 v8.8?

    Helo experts,
    I have SAP B1 v.8.8 and they system is allowing me to enter the same Vendor Reference for different POs but same BP.
    Is there a way to prevent duplicaton?
    I read in other post that there was an option in Admin'-System Initialisation-Document settings......to block duplication, but that does not show on my screen.
    Do you have any ideas or suggestions?
    Thank you

    Hello Gordon,
    Thanks for the link but I had already read those posts.
    I understand there is a way to prevent duplicate to enter vendor reference numbers.
    However, my question would be if the version that I am using would have the option to block release when When duplicated Vendor Reference No occurs as I cannot find it.
    Please advise,
    Thanks

  • How to prevent duplicate submit

    I am looking for a solution to prevent duplicate submit on the click of submit button or browser refresh. In Struts its quite easy to implement using toekn feature.
    Is there some feature in JSF which I can use to prevent duplicate submit?
    Please share your thought.

    Redirect after POST.

  • How to prevent Duplicate AP Invoices ??

    hi Experts
    End user Raised Po for 1000 quantities but good receipt Po comes one time 100,2nd time 200 ,3time 300 and so on...in between invoice raised for 100 ,2time 200 ... against GRPO..in between End user miss understand or communication some Duplicate bills are raised...
    how i restrict duplicate postings??
    note:(im using sap b1 2007 pl 17)
      Thanks & Regards
         AKR

    Hi
    Try this Stored Procedure in Test Environment.
    if (@object_type = '20' or @object_type = '18')
         and (@transaction_type= 'A' or @transaction_type= 'U')
    begin
         declare @venno as varchar(100)   
         if @object_type = '20'
         begin
              select @venno = NumAtCard
              from opdn
              where docentry =@list_of_cols_val_tab_del
              if 1 != (select count(docentry) from opdn with(nolock) where NumAtCard = @venno)
              begin
                   select @error = 1
                   select @error_message = 'Duplicate vendor ref no in PO Receipts. ! '
              end
         end
         else
         if @object_type = '18'
         begin
              select @venno = NumAtCard
              from opch
              where docentry =@list_of_cols_val_tab_del
              if 1 != (select count(docentry) from opch with(nolock) where NumAtCard = @venno)
              begin
                   select @error = 2
                   select @error_message = 'Duplicate vendor ref no in AP Invoices. ! '
              end
         end
    end
    With Regards
    Balaji Sampath

  • PT 8.51 Upgrade - How to find duplicate keys for Portal Registry Structure?

    I am upgrading peopletools for two applications (CRM 9.00 and FDM 9.00) from the 8.48.10 to 8.51.11 versions.
    For CRM, Task 4-15-9 Completing the Peopletools Conversion ran app engine PTUPGCONVERT ran smoothly.
    For FDM, I have the following error messages:
    The Navigation Pagelet does not exist. Portal = EMPLOYEE. Object = PT_SC_PGT_MAIN_MENU. Collection = PT_PTPP_PORTAL_ROOT (219,2083)
    Message Set Number: 219
    Message Number: 2083
    Message Reason: The Navigation Pagelet does not exist. Portal = EMPLOYEE. Object = PT_SC_PGT_MAIN_MENU. Collection = PT_PTPP_PORTAL_ROOT (219,2083)
    Duplicate key. Portal: EMPLOYEE, Obj Name: PT_SC_PGT_MAIN_MENU, Nodename: LOCAL_NODE, URL: s/WEBLIB_PTPP_SC.HOMEPAGE.FieldFormula.IScript_SCP (133,4)
    PT_SC_PGT_MAIN_MENU already exists. (96,3)
    Copy of the Main Menu Pagelet From PS_SITETEMPLATE to Portal: EMPLOYEE failed. (219,2111)
    Message Set Number: 219
    Message Number: 2111
    Message Reason: Copy of the Main Menu Pagelet From PS_SITETEMPLATE to Portal: EMPLOYEE failed. (219,2111)
    I checked table PSPRSMDEFN which does not have an entry for PT_SC_PGT_MAIN_MENU, under the Employee Portal. I tried to migrate the missing Portal Registry object using App Designer but again receive the "duplicate key" error.
    So it seems that I have to find the duplicate key and resolve it before I can migrate the missing object.
    Anyone know a quick way to figure out what the duplicate keys are?
    Thanks

    I tried several things to find the duplicates with no success.
    A couple of workarounds were attempted that resulted in the same "duplicate key" error, including:
    a) Re-copying file project PPLTLS84CUR
    b) Copy object 'PT_SC_PGT_MAIN_MENU" from Demo
    After opening an SR, the successful workaround was to use Data Mover to export from Demo the "EMPLOYEE" portal entries for "PT_SC_PGT_MAIN_MENU" from tables PSPRSMDEFN, PSPRSMSYSATTR and PSPRSMSYSATTRVL. The import to the target upgrade environment was successful. A re-run of PTUPGCONVERT finished successfully.
    The upgrade is progressing but where the duplicate keys are is still a mystery.
    Cheers

  • How to get access key while creating database view in R3

    Hi all experts,
    I am new to SAP so please forgive my ignorance.
    From R3, I am trying to create a new database view named ZVENDOR_ATTR  which will be based on LFM1.
    When i go to SE11 -> chose view and click create then it shows a pop up window that says "you are not registered as a developer. Please register in the Online Service System (OSS). In the OSS you will receive an access key."
    I don't know how can i get the access key and proceed now. Please note that i have installed this BW3.5 and R3 on my laptop. From the default user ddic, i have copied and created a new user 'sapuser1'. I am now using the sapuser1 to logon  and attempting to create the above mentioned Database view. It seems like this sapuser1 as it is copied from ddic is not a developer.
    So anybody can please help  how to create/get this access key to proceed  further.
    I will give full points. Please help.
    Thank you so much in advance
    ak

    Thanks All,
    Does it mean that i can not create any database object without the access key from service.sap.com? As i told that i installed this BW3.5 / R3 on my laptop to train myself but i don't have an account on service.sap.com.
    However i can login to R3 system as admin (i have access to the system using ddic or spcpic). Is there a way that a new user with deveoper role can be created by the admin without going to service.sap.com.
    Please advise.
    Thanks.
    ak

Maybe you are looking for

  • What if I want the Report Header on every page?

    Is there a way to tell Report Builder to have the main report header show up on every page in CF Report Builder 9? There doesn't seem to be any property in the property viewer when you select the main header that tells it to do that. The secondary he

  • Yoga 2 External Display Resolution

    I have Yoga 2 and when I conncet it to any external monitor the resolution of the monitor becomes very bad.  I'm using the micro to hdmi adaptor.  I do not know if there is a better adobtor or if there is some option I need to adjust.  Any idea?

  • Smartform page numbering

    Hi experts, I am displaying notification numbers in the ALV format. when the user selects few of the notification numbers and click on display button it should display the notification details using smartform. I have to trigger a new page for every n

  • MacOS 10.9 and InDesign CS4

    After installing MacOS 10.9 I got the error message 150:30 when trying to open InDesign. The suggested solutions like deleting the Folder FlexNet Publisher (which doesn't exist on my iMac) or reparing the permissions didn't work. Is there any other h

  • I just installed Maverick 10.9 and I'm not happy.  Too many things not working.  How can I get back to my OS 10.6.9?

    I just installed Maverick and I'm not happy.  Too many other things not working.  How can Iget back to 10.6.9?