Flattening CDC tables to use in a data warehouse

I have enabled CDC on 3 tables: [Employee, EmployeeAddress, and Address]. I want to flatten the data out then store it data in a warehouse so I can report what changed in a specific time frame. 
If a Address is updated the linking table (EmployeeAddress) is not updated since the link to the employee already exists so when i flatten out my data my CDC tables cant figure out the who the address belongs to.
It feels wrong to in some cases join my CDC table to my production tables. What is a best practice or recommendation for a scenario like this?
My tables look like:
Employee:
EmployeeId | Name
-------------+-------------
e1               | Bob Jones
e2               | jane Doe
EmployeeAddress:
EmployeeAddressId |EmplyeeId | AddressId
--------------------+----------+---------------
ea1                          | e1              | a1
ea2                          | e2              | a2
ea3                          | e2              | a3
Address:
AddressId | Address
----------+------------------------
a1            | 111 Some Street
a2            | 222 Some Street
a3             | 333 Some Street
SELECT *
FROM [cdc].[fn_cdc_get_all_changes_dbo_Employee](sys.fn_cdc_get_min_lsn('dbo_Employee'), sys.fn_cdc_get_max_lsn(), N'all update old') E
JOIN [cdc].[fn_cdc_get_all_changes_dbo_EmployeeAddress](sys.fn_cdc_get_min_lsn('dbo_EmployeeAddress'), sys.fn_cdc_get_max_lsn(), N'all update old') EA ON EA.EmployeeId = E.EmployeeId
JOIN [cdc].[fn_cdc_get_all_changes_dbo_Address](sys.fn_cdc_get_min_lsn('dbo_Address'), sys.fn_cdc_get_max_lsn(), N'all update old') A ON A.AddressId = EA.AddressId
Thanks!

Create a
dimensional model instead of flattening. Under normal circumstances this leads to model with a small fact table and some slowly changing dimension tables. The key problem is to find some meaningful measures (HR related, e.g. work time) and to determine
the appropriate grain and fact table type (transaction or snapshot (periodic, accumulating or temporal)).
See also
Kimball Dimensional Modeling Techniques.

Similar Messages

  • Table and Index compression in data warehouse - thoughts?

    Hi,
    We have a data warehouse with large fact tables and materialized views of this data.
    Approx 3 million inserts per day week-ends about 12 million.
    The fact tables we have expected to have 200 million, and couple with 1-3 billion.
    Tables partitioned and have bitmap indexes.
    Just wondered what thoughts were about compressing large fact tables and mviews both from point of view of ETL into them and reporting from them afterwards.
    I take it, can compress/uncompress accordingly without any problem?
    Many Thanks

    After compression, most SELECT statements would not get slower. Actually, many can get faster due to reduced IO and buffer needs.
    The situation with DMLs is more complex. It depends on the exact compression options (basic or advanced) and the DML (INSERT,UPDATE, direct load,..),but generally DML are negatively affected by compression.
    In a Data Warehouses (DWs), it is usually quite beneficial to compress partitions or tables that contain data that is not supposed to be modified (read only or read mostly). Please note that in many cases you do not have to compress while you are loading the data – you can do that later.
    You can also consider compressing some of your B-tree indexes (if you use them in your DW system).
    Iordan Iotzov
    http://iiotzov.wordpress.com/

  • Tables between OBAW and Oracle Data Warehouse in OBIA rpd are different

    Hi,
    The tables in the Data Warehouse DB is different as compared to the tables in the physical layer of the Oracle data Warehouse in OracleBIAnalytiscApps.rpd.
    When I click update Row Count, it gives me below error message:
    There was an error while updating row count for "Oracle Data Warehouse"."Catalog"."DBO"."W_CTRY_REGN_D":
    [nQSError :17001] Oracle Error Code: 942, message: ORA-00942:Table or view does not exist at OCICallSTmTEXECUTE
    Whereas when I click on some other tables for Row count there is not error.
    Also, When I am trying to look for some specific table which exist in Data Warehouse (that has the data), it does not exist in the rpd.
    Kindly assit me.

    hi,
    Have a look
    Not able toView Data in Answers
    Let me know
    thanks,
    saichand.v

  • Modelling tables in Transactional database and data warehouse

    Hello,
    Can any one please tell me what are the differences between modelling tables in a transactional databse compared with a data warehouse? And I want to know the process of multi dimensional modelling...

    Check here:
    https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/6ce7b0a4-0b01-0010-52ac-a6e813c35a84
    BI Data Modeling and Frontend Design [original link is broken]
    Regards,
    Luis

  • Using Sequence with Data Warehouse Table

    I am working on a Uni exercise on data warehousing and am trying to create and populate a table with a sequence on one database with data located on another database.
    For this particular table, I have to reset the primary key (primary keys on operational table start from 4000). In my notes, I have been told to create a sequence first and use yoursequence.nextval in the sub query. I am not very experienced with using sequences and am not sure what a subquery is in relation to them.
    I tried the following but it didn't work. Hopefully it will give you an idea as to what I am trying to achieve:
    create table Region as select region_id_seq, Region_Name, Management_area, Country from dw_op.Sales_Region@q_link;
    Any help with this would be great.

    Thanks again, Justin.
    My instructions tell me to use the schema name along with the db link and it created the table correctly so I'll just have to keep my eye on it.
    Do oyu know how I could use the function initcap to ensure all data is of the same format? I did a bit of research and found that I should be using column aliases which I duly tried but I received an error saying the following line needs a column alias:
    initcap(s.region_name),
    Just thought you may know how this can be done.
    Thanks.

  • Why do we gor for SAP BW rather than using the other Data Warehouse S/W's

    Hi All,
    Can any one tell me Why do we especially go for SAP BW, rather than other Warehouse S/W's available in the IT Market.
    Plzzzz..... Reply back me to [email protected]
    Regards,
    Kiran

    Hello Kiran Telkar,
    How r u ?
    1. Multi Dimentional Modelling Concept is available in BW
    2. Most of the 3rd party DataBases could be connected to BW
    3. Very effective Reporting Solution is given by BW
    4. Integration of Transactional Data & Master Data
    5. Tight integration with R/3 and the Business Content functionality, which handles core areas & processes.
    and more !
    Best Regards....
    Sankar Kumar
    +91 98403 47141

  • Why do we need SSIS and star schema of Data Warehouse?

    If SSAS in MOLAP mode stores data, what is the application of SSIS and why do we need a Data Warehouse and the ETL process of SSIS?
    I have a SQL Server OLTP database. I am using SSIS to transfer my SQL Server data from OLTP database to a Data Warehouse database that contains fact and dimension tables.
    After that I want to create cubes using SSAS form Data Warehouse data.
    I know that MOLAP stores data. Do I need any Data warehouse with Fact and Dimension tables?
    Is not it better to avoid creating Data warehouse and create cubes directly from OLTP database?

    Another thing to note is data stored in transactional system may not always be in end user consumable format for ex. we may use bit fields/flags to represent some details in OLTP as storage required ius minimum but presenting them as is would not make any
    sense to user as they would not know what each bit value represents. In such cases we apply some transformations and convert data into useful information for users to understand. This is also in the warehouse so that information in warehouse can directly be
    used for reporting. Also in many cases the report will merge data from multiple source systems so merging it on the fly in report would be tedious and would have hit on report server. In comparison bringing them onto common layer (warehouse) and prebuilding
    aggregates would be benefitial for the report performance.
    I think (not sure) we join tables in SSAS queries and calculate aggregations in it.
    I think SSAS stores these values and joined tables and we do not need to evaluates those values again and this behavior is like a Data Warehouse.
    Is not it?
    So if I do not need historical data, Can I avoid creating Data Warehouse?
    On the backend SSAS uses queries only to extract the data
    B/w I was not explaining on SSAS. I was explaining on what happens inside datawarehouse  which is a relational database by itself. SSAS is used to built cube (OLAP structures) on top of datawarehouse. star schema is easier for defining relationships
    and buidling aggregations inside SSAS as its simple and requires minimal lookups to be performed. Also data would be held at lowest granularity level which can easily be aggregated to required levels inside OLAP cubes. Cube processing is very resource
    intensive and using OLTP system would really have a huge impact on processing performance as its nnot denormalized and also doing tranformation etc on the fly adds up to complexity. Precreating a layer (data warehouse) having data in required format would
    make cube processing easier and simpler as it has to just cross join tables and aggregate data based on relationships defined and level needed inside the cube.
    Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

  • OBIEE reverse engineering to go from SQL Server to a data warehouse

    Hi,
    I'm new to data modeling for warehouses. We currently have an OBIEE environment set up where the data source was SQL Server transactional tables. The SQL Server data is to be moved to a non-Oracle data warehouse and I need to produce a logical data model for the warehouse folks at my company. Unfortunately, the SQL Server data was never modeled, so, I'm basing the model from the Logical and Physical diagram/relationships of OBIEE.
    My question is in regards to the validity of the following relationship to be used in a data warehouse based on what's currently in OBIEE. When I model this via Erwin, I'm wondering if I'm way off base in the relationships (modeling, not personal):
    Dimension 1 has a 0:M with Dimension 2
    Dimension 1 has a 0:M with Dimesion 3
    Dimension 2 has a 0:M with Dimension 3
    Both Dimension 2 and Dimension 3 have a 0:M with Fact 1
    Through the use of aliases and such, this does work in OBIEE. Will this work as a data model for a data warehouse environment?
    Thanks!

    I think you started with the wrong foot. I suggest you search in Google for "kimball methodology" and have a read at a few articles. Your DWH model should not be based on your transactional tables. You should ask your business users what "questions" they want to answer in the DWH. Then model your DWH base on that. You can not model a DWH without knowing what questions you need to answer. For instance if your business users want to know the sales per day and per branch you will a sales fact with a sales amount measure joining to two dimensions branch and time dimension. The number of facts will depend on the questions you need to answer, the type of data and the granularity of them.

  • Data Access Object for Data Warehouse?

    Hi,
    Does anyone know how the DAO pattern looks like when it is used for a data warehouse rather than a normal transactional database?
    Normally we have something like CustomerDAO or ProductDAO in the DAO pattern, but for data warehouse applications, JOINs are used and multiple tables are queried, for example, a query may contains data from the Customer, Product and Time table, what should the DAO class be named? CustomerProductTimeDAO?? Any difference in other parts of the pattern?
    Thanks in advance.
    SK

    In my opinion, there are no differences in the Data Access Object design pattern which have any thing to do with any characteristic of its implementation or the storage format of the data the pattern is designed to function with.
    The core pupose of the DAO design pattern is to encapsulate data access code and separate it from the business logic code of the application. A DAO implementation might vary from application to application. The design pattern does not specify any implementation details. A DAO implementation can be applied to group of XML data files, an Excel-based CSV file, a relational database, or an OS file system. The design is the same for all these, it is the implementation that varies.
    The core difference between an operational database and a strategic data warehouse is the purpose of why and how the data is used. It is not so much a technical difference. The relational design may vary however, there may be more tables amd ternary relationships in a data warehouse to support more fine-tuned queries; there may be less tables in a operational database to support insert/add efficiencies.
    The DAO implementation for a data warehouse would be based on the model of the databases. However the tables are set up, that is how the DAO is coded.

  • Designer Vs. Oracle Data warehouse builder

    Dear all,
    Currently I'm responsible of building a Data warehousing project using Oracle database. I'm trying to decide on a tool for modelling my datawarehouse. I have two options:
    1) Designer: we have some experience with this tool and we are using it for our main OLTP application.
    2) Oracle Data Warehouse builder: we are using this to design our ETL processes.
    I want to get some advice on whether the OWB is capable of modelling my datawarehouse and of doing a retrofit action. also, I try to standardize on the tools that are using in the Data Warehouse department (currently we are using only OWB).
    I will appreciate for any other advice to help in my selection process.
    Best Regards,
    Bilal

    Hi,
    In my experience this choice depends on the implementation of the datawarehouse. If you are building a "pure" Kimball style dimensional data warehouse you should be able to do this using OWB. I have architected such a DW in the past using only OWB, so I am speaking from experience.
    If on the other hand you are planning to implement an Inmon style CIF, if your requirements includes an operational data store (ODS), or if you for any other reason anticipate that you are going to be doing a lot of ER modeling, then I would not recommend using the current release of OWB for modelling. (Note however that there are significant improvements to the modelling capabilities in the Paris release of OWB, so this may change in the future)
    The advantage of improved maintainability when using a single tools needs to be weighted against the improved functionality if you choose a combination of the two. In the "two tool" scenario strict development and deployment routines need to be enforced to avoid that the model in Designer comes out of sync with the metadata in OWB. (Consider the effect of a developer making a change to a table definition in OWB and deploying it directly to the database without updating the model in Designer.)
    Hope this helps.
    Regards,
    Roald

  • Need Data warehouse Training in India...

    Hi,
    can somebody help me findout where I can learn Data warehousing in detail, may be along with some BI tools ?
    I m using a Teradata data warehouse but my source database is in Oracle and frontends are BI tools like Business Objects,
    Cognos, Actuate etc.
    Thanks in anticipation,
    DJ Panchal

    Dear Dharmendra,
    We are a cutting edge software training company, based in Ahmedabad, India, which brings real industry software training to you before anyone else. We were the pioneers in bringing the e-commerce/web technologies training in India and we are doing it again by bringing Business Intelligence (BI), Data Warehousing & related technologies, the hottest combination of software tools, which is growing in demand even in recession.
    The Tools we will be giving training in are COGNOS, Business Objects, Actuate, Teradata and Oracle 9i.
    Candidates from Overseas will get a cheaper Option in terms of Fees.
    Candidates within India will get the opportunity to learn real industry Tools.
    If you are Interested please contact us at [email protected] we will then provide you with further details.
    Thank You

  • Data Warehouse in Virtual Machine

    Hi all!
    I'm looking info about pros and cons of using an Oracle Data Warehouse in a Virtual Machine.
    I've been looking at google but I can't find something worthy.
    Do you know where can find a document or tips?
    Thanks!

    I am curious to know why you would want to run a Data Warehouse within a VM image. It would seem to me the benefits would be limited for most data warehouses unless you are using this as a way to quickly spin off QA, training, unit/integration testing environments using limited resources.
    There is nothing specific you need to do for data warehousing when running a database inside a VM but obviously you will need to setup your VM environment carefully. You can find general information on virtualization here: http://www.oracle.com/technology/tech/virtualization/index.html
    I would be interested in getting more information you about this requirement.
    Regards
    Keith Laker
    Senior Principal Product Manager, Data Warehousing

  • Data are not transfered to a table view using signal out

    Hi all,
    we have recently upgraded to SP16 and I'm now facing a problem on my model that worked fine when we were on SP15.
    I have a data service returning data to my model. The data are transfered to a signal out element and then displayed into a table view using a signal in element. Now the table view remains empty.
    I have checked that the dataservice returns data because if I directly connect my table view with the dataservice the data are displayed. But my requirement is to use signal in/out elements.
    This error was not appearing in SP15.
    Any ideas?
    Regards
    Panos

    After doing some tests I found out that if you have a model that has more than 2 signals in/out then the 3rd one will not work. This is definitely a bug.
    Hope that soon there will be a SP without any new bugs!

  • Use of end-date as part of Primary key in SAP Tables

    All,
    Any ideas on the rationale behind SAPs use of end-date as part of Primary key in general, specifically on condition Tables(A9xx for example) instead of Start-date? appreciate any help!!
    Best Regards
    TRP

    Hmm, another example is CSKS, this logic was probably designed already in the 1980s...maybe it was seen beneficial to be able to select the currently valid record by using the primary key only (the first record where DATBI is larger than system date, not possible when DATAB is the key instead).
    Maybe there are better explanations. Why do you need to know this, just curious?
    Thomas

  • Acrobat x pro crashes when I try to flatten a pdf after using typewriter to insert a date

    acrobat x pro crashes when I try to flatten a pdf after using typewriter to insert a date.
    Has anybody else seen this problem? Is there a fix?
    I wanted to download the demo of acrobat XI but it says the previous version will be removed and deactivate the license. That sucks!

    Yes it is up to date.
    I can use the stamp tool and flatten it after with no problems, but the typewriter tool crashes it upon attempting to flatten.

Maybe you are looking for

  • What Format Is Used for eprint Emails?

    I'm interested in generating eprint jobs from a Linux machine, so I'm wondering what format is used, and how you cope with email gateways which can accomodate messages of limited size. Can somebody assist please? Regards ..

  • Adobe Reader for Palm OS Tungston 5

    Does anyone know why my keyboard comes up when using Adobe Reader and won't go down when I tap the down arrow?  The keyboard goes up and down in all other functions, as it should, but the down arrow seems to be greyed out when ever I open Adobe.  I'm

  • IE 10 stopped retreiving verizon mailbox

    After logging into my verizon account.....and trying to retreive my mail using IE 10.....no part of my email will show up.  I get the top of the verizon screen....and the very bottom of the page......but nothing else. Tech support has told me it's MY

  • Tuning : log buffer space in 11gr2

    Hi, version : 11202 on hpux awr Top 5 events shows : Top 5 Timed Foreground Events : Event Waits Time(s) Avg wait (ms) % DB time Wait Class log buffer space 12,401 29,885 2410 55.83 Configuration My log_buffer size is : SQL> show parameter log_buffer

  • Publish iCal to internet

    Hi I have 6 Cals that I want to share with my coworkers on the web. I do not want them to be able to change them. I do not want each of them to subscribe to my 6 cals, too much hastle for them!! Currently my Cals are all synced with my MobileMe so th