Building a data warehouse

Good Morning Buddys,
Actually I need build a data warehouse and I've some doubts:
1. OLTP I have these tables (Adventureworks):
Production.Location
LocationID int primary key 
Name nvarchar(50)
CostRate smallmoney
Availability decimal(8,2)
ModifiedDate datetime
LocationFK int -- example
Production.ProductCategory
ProductCategoryID int primary key
Name nvarchar(50)
rowguid uniqueidentifier
ModifiedDate datetime
ProductCategoryFK int -- example
Production.ProductModel
ProductModelID int primary key
Name nvarchar(50)
CatalogDescription xml
Instructions xml
rowguid uniqueidentifier
ModifiedDate datetime
ProductModelFK int -- example
When I transfer to Staging is necessary I have foreign key columns? I mean with varchar, not int (foreign key) for example:
STAGING_Production.Location
LocationID varchar(20)
Name nvarchar(50)
CostRate varchar(20)
Availability varchar(20)
ModifiedDate varchar(20)
LocationFK varchar(20) - it's correct? 
STAGING_Production.ProductCategory
ProductCategoryID varchar(20)
Name nvarchar(50)
rowguid varchar(20)
ModifiedDate varchar(20)
ProductCategoryFK varchar(20) - it's correct?
I've no idea if is necessary foreign key columns transfer to staging but with another data type...
some one can help me please?

Ok ok
following:
1. OLTP I have these tables (Adventureworks):
Production.Location
LocationID int primary key 
Name nvarchar(50)
CostRate smallmoney
Availability decimal(8,2)
ModifiedDate datetime
Production.ProductCategory
ProductCategoryID int primary key
Name nvarchar(50)
rowguid uniqueidentifier
ModifiedDate datetime
LocationFK int - example
Production.ProductModel
ProductModelID int primary key
Name nvarchar(50)
CatalogDescription xml
Instructions xml
rowguid uniqueidentifier
ModifiedDate datetime
ProductCategoryIDFK int -- example
When I transfer to Staging is necessary I have foreign key columns? I mean with varchar, not int (foreign key) for example:
STAGING_Production.Location
LocationID varchar(20)
Name nvarchar(50)
CostRate varchar(20)
Availability varchar(20)
ModifiedDate varchar(20)
STAGING_Production.ProductCategory
ProductCategoryID varchar(20)
Name nvarchar(50)
rowguid varchar(20)
ModifiedDate varchar(20)
LocationFK varchar(20) data type? - example
Production.ProductModel
ProductModelID int primary key
Name nvarchar(50)
CatalogDescription varchar(20)
Instructions varchar(20)
rowguid varchar(20)
ModifiedDate varchar(20)
ProductCategoryIDFK varchar(20) -- example
What my doubt?
all columns in oltp (primary key and foreign key) exists on my olap table? i mean using data type varchar(20) - in columns : 
ProductCategoryIDFK varchar(20) 
LocationFK varchar(20) 
because i have no idea how can i load fact table without relationship...
you understood?

Similar Messages

  • Can one build a data warehouse using SQL rather than Warehouse Builder?

    I would like to build a data warehouse purely using SQL statements. Where can I find the data warehouse extension of SQL statements?

    I am exploring the internal workings of Warehouse Builder.
    I have written a SQL script to generate sample data to be inserted into tables, then write SQL script to do Extraction, Transformation and Loading using MERGE,, GROUP BY CUBE, DECODE, etc.
    If anyone has any experience of just using SQL to perform ETL, would you share your expeience here? Thanks.

  • Build a data warehouse for crystal report use

    OS: AIX 5.3 with Oracle 10g R2/Baan 5c
    I proposed my colleage to create a data warehouse for his heavily use of crystal report activity, so that his activites will not drag the normal database performance. The data will collect from the current Oracle 10 g database. Because I have never physically worked on the data warehouse, where should I start to look for information to proceed? I have an extra Sun Solris 8 box with a spare HD. I think it will be better to build the warehouse on that server, am I right? or the 10g R2 provides the capability to host another warehouse activity -- create a materialized view within? Thanks,

    Thanks for the help Ted
    Works like a charm now.
    Or atleast my very very simple XML report works.
    Now I just need to play around and see what I can do and what I cant.
    I just want to make sure I understand how this works.
    1) I start of with a XML file (or URL or other data source as explained in your BLOG)
    2) I then have to create blank report which will have its data source set to my XML file.
        This report (rpt file) will always have its data source set to the original XML location
    Can this be dynamically changed, or does a new rpt file need to be created each time
        (or atleast I will have to re-run "CreateXMLInlineSchemaReport" from your BLOG)
    3) I can now use this rpt file as I would any other rpt file but its data source will be the original XML file
       - If I update or change the XML file, the report will dynamically reflect these changes.
    4) To change the data source, I need to recreate a new rpt file as per point 2. Which will now be blank.
    5) Filtering.... if I want to filter the XML data do I use normal parameters. If I want to filter the data before
        I pull it do I need to change anything. In otherwords if I have a XML file with 1000 records, but I only want say
    5 of them, how do I only pull those 5 without first pulling the whole 1000 records. Hope that makes sense.
    As I said I am very new to this so some things still fly over my head, but I do play around abit so hopefully
    will find what I am looking for.
    Cheers
    Darren
    PS any other links to info on creating and running reports of XML data sources would be greatly appriciated

  • Suggestions of building this data warehouse

    Hi,
    We have 5 source systems and all the data are coming in to 5 different ODSes in our data warehouse.
    For reporting purposes,
    1). do you suggest that we feed each ODS into a one cube each and push them into a multiprovider
    or
    2) feed all 5 ODSes into a single cube
    3) What is the advantage of 1) over 2); and 2) over 1)
    4) Any suggestions on how best to configure the cubes and/or multiprovider?
    5) How should the keys fields of the ODSes be handled in the Cube?
    6) How best do your suggest that we handle this with respect to BW authorization?
    Thanks

    Hi,
    thanks for the response.
    I don't clearly get some of the points you made well.
    On 1) & 2), you indicated that
    "I would say 1 would be right way to do it. In that case, you can split ur KF values by dataprovider if ever you need them to comparision scenario.".
    What did you mean by "...ou can split ur KF values by dataprovider if ever you need them to comparision scenario"?
    I was also lost on the "split ur KF laues.." and " comparison scenario"
    I will appreciate and example to clarify this
    On 3) you wrote that 
    "Unless you dont have any field in each ODS to differentiate the data, putting all the data into a single cube would prevent you from spilting the data by dataprovider."
    Can you clarify (preferably with an exmaple) what you meant by ""Unless you dont have any field in each ODS to differentiate the data"?
    For performance purposes, won't multiple cubes be better than placing all the data in a single cube?
    I thought I read that the ODSes to multiple cubes and them pushing to a multiprovider has some advantages of being able to add or remove a cube whithout the need to modify the queries.
    On 4) what "data structure" were you refering to? ( this is logistic data).
    Thanks

  • SQL Server Data Warehouse Creation External or Build Your own

    Hi, 
    In our firm we have a data warehouse which has been created by an external supplier.
    The warehouse is being used for some reporting and dashboards but only for a single client, we have yet to fully utilize it.
    My question is should we build a new data warehouse or should we push forward with the data warehouse created by the external supplier. I know it would take some time to build a data warehouse however it means we will have full control over it and could
    overcome any issues we have with it in house.
    It would really help if someone could offer some advice.
    Thanks

    Hi Nasa1999,
    I am trying to involve someone more familiar with this topic for a further look at this issue. Sometime delay might be expected from the job transferring. Your patience is greatly appreciated.
    Thank you for your understanding and support.
    Regards,
    Katherine Xiong
    Katherine Xiong
    TechNet Community Support

  • How do I design this in data warehouse?

    I am working on building a data warehouse for insurance quote data.
    Each quote will have an applicant and can have an optional co-applicant. Each applicant and co-applicant will have prior auto insurance history, prior home insurance history, current auto insurance information and current home insurance information.
    So do I create Applicant and Insurance dimensions here?

    Hi Ashan,
    Just so you know.
    I completely reworked our methodology of building data warehouses back in 2012. The new way of building data warehouses is quite different to the old way.  The way you listed.
    The methodology presentation is on this link.
    https://www.youtube.com/watch?v=Df4CgOtrFq8
    Video channels are here. http://www.instantbi.com/videos/
    Downloads are here: http://www.instantbi.com/company/downloads/
    I have been doing BI since 91 and what we have done now is industry leading. 
    I am an MSDN so we do our development on MSFT first and then deploy where ever our clients want us to deploy.
    Best Regards 
    Peter Nolan

  • Data Warehouse Infrastructure

    I have a requirement to build a Data Warehouse and Analytics / Reporting capability with the following requirements...
    Maximum of 1TB for Production Data + DR + Test/Dev Env.
    SSIS (up to 25 sources), SSAS (cubes, 5 concurrent users) and SSRS (2 concurrent users, max 500 reports).
    I needs a Production, DR and Test/Dev Environment 
    I have been told that I will require 12 servers each having 4 cores and 12GB of storage (4 for Prod, 4 DR and 4 Test/Dev).
    To give you an idea of load we plan to have 1 full time ETL developer, 5 Data Analysts, 2 Reporting Analysts. We are quite a small business and don't have a particularly large
    amount of data. 
    The model has SQL Server, SSIS, SSAS, SSRS on different servers across each Environment. 
    Any idea if this is overkill? I also have an estimate of 110 days for Setting up the Servers, Installing the SQL Server software and general Infrastructure design activity.

    Agree. Overkill. Big overkill.
    I would recommend production/DR/Dev each have 2 servers. I'd put SSAS, SSRS and SSIS one one and the DB on the other.
    In production, SSAS/SSRS will be active during the daytime; SSIS will likely be active off hours. So putting all that on one box should be fine for sharing the load. The DB on a second box would be good since it will likely be busy during the daytime
    and night time. Four processors may be heavy depending on the types of queries and usage patterns. I suspect you can get by with 2 processor servers, but would recommend buying the 4 processor boxes for dev and production, get them configured and run
    some performance baselines before putting in the DR environment. Then, if you find the CPUs idling, you can always cut the DR environment to 2 processor boxes. Not sure it's worth the minor cost savings to save 2 processors on 2 boxes with that effort, but
    if you're looking to cut corners, you may find that a 2 processor per server DR environment is within your performance comfort zone.
    For the dev environment, one box may well handle it all, but I'd go for 2. On average, a Dev environment isn't all that busy, but when you need the horsepower, you need it. And since it's Development AND Test, you help yourself by having realistic production
    level performance on what you're testing. Four processors is fine, but max it out on memory.
    As for hard drives, be careful about configuration. You need the space on your DW server and maybe for the SSAS server depending on how the cubes are built (ROLAP/MOLAP). When you speak about amounts of data, be careful since you'll want a lot of indexes,
    and that can double the DB size for a DW. Your DW will also run faster if you have different filegroups for data/indexes/temp DB, but only if those different filegroups are on different physical media that work well in parallel. You can always get fancier
    with more filegroups to have different ones for staging tables, for segregating fact & dimension tables etc. But for this size DB, that's overkill as well.
    Mainly, I'd look at spending hardware $s on memory for the servers, but get less of them.
    Now... two questions...
    1) Can you clarify the disk space needs? How much total data space in one environment, without indexes? Based on that, add the same for indexes, add half as much (?) for TempDB and you have the core disk needs. Depending on how much it is,
    you can decide on RAID, filegroup configuration, etc. And if the disk space with indexes is small enough that it all fits in memory, then disk and filegroup configuration becomes inconsequential except for ETL loads.
    2) The 25 sources... can you clarify that? 25 source systems? Total of 25 source applications? Total of 25 tables? Curious, because I'm wondering about how long you'd keep 1 full time ETL developer busy.

  • Default dates for Data Warehouse

    We are currently building a data warehouse using Oracle 9i. We need to have values for default low dates and high dates eg to store in a closed date field for rows not closed. Any recommendations? The defaults must work correctly in tools such as Access and Business Objects.

    From my point of view such limits depend totaly of the business case.
    What I have seen several times is that the 1.1.3000 is used as upper limit. I never saw, for dates, anything like a lower limit.
    Which is your need of having a lower date?

  • Missing Zip - OBE 9i2, Data Warehouse

    All the downloads work except 8.7 - Where is the file?
    8 Build a Data Warehouse for Business Intelligence
    8.1 Setting Up the Data Warehousing Environment
    8.2 High Speed Data Loading and Rolling Window Operations
    8.3 Using the ETL Infrastructure of Oracle9i
    8.4 Using Basic Database Functionality for Data Warehousing
    8.5 Using Oracle9i Materialized Views
    8.6 Using Oracle Data Mining to Predict Behavior
    8.7 Using the OLAP Analytic Workspace
    8.8 Creating a Data Warehouse using Oracle Warehouse Builder
    8.9 Implementing Relational Business Analysis with SQL
    8.10 Analyzing Business Information

    Please try the following link:
    http://otn.oracle.com/products/oracle9i/htdocs/9iober2/obe9ir2/download/obe-dwh_olap.zip

  • Design the data warehouse around the reporting system?

    Hi All,
    A Jr. data warehouse developer resisted my suggestion to flatten out activity tables of differing grains into a single fact table.  (Think sales order header, sales order detail, and even a 3rd level of details to each sales order detail.)  Although
    he agreed that flattening out the fact tables into a single fact would be proper for a data warehouse, he's concerned that report developers will have an easier time querying the data warehouse with the 3 separate fact tables.  I'm not sure if it's because
    the report developers don't like learning new schemas or if their reporting tool is just severely limited, mainly because I've never used Cognos.  I assured him that a properly-designed data warehouse will save on query execution time, but he's concerned
    about the reporting tool and how it may not work so well with the data warehouse.  
    Did I give him the proper advice?  It seems like a data warehouse should be built properly regardless of reporting tool shortcomings.  Assuming this tool is lousy, maybe they need a new reporting system for their new data warehouse.
    Thanks,
    Eric

    Hi Eric,
    one of the hard and fast rules of building a data warehouse is that from a logical point of view the fact table presents data at a certain level of granularity and that you do not mix facts in fact tables. This is data warehousing 101.
    From your comment you seem to be suggesting mixing data of different granularity in the one table.
    Now, we have ways and means of co-habiting data that will appear as different fact tables in the one physical table. We control the physical placement of data in fact tables. But on SQL Server we would never mix facts at different granularities or representing
    different data in the one fact table. SQL Server supports that quite poorly.
    It is sad that in 2015 people are still messing up data warehouse project from pure ignorance of what is available. We have data warehouse data models that are extremely extensive but people just have to start from scratch and reinvent the wheel and fail over
    and over again. Sad but true.
    Best Regards 
    Peter Nolan

  • Designer Vs. Oracle Data warehouse builder

    Dear all,
    Currently I'm responsible of building a Data warehousing project using Oracle database. I'm trying to decide on a tool for modelling my datawarehouse. I have two options:
    1) Designer: we have some experience with this tool and we are using it for our main OLTP application.
    2) Oracle Data Warehouse builder: we are using this to design our ETL processes.
    I want to get some advice on whether the OWB is capable of modelling my datawarehouse and of doing a retrofit action. also, I try to standardize on the tools that are using in the Data Warehouse department (currently we are using only OWB).
    I will appreciate for any other advice to help in my selection process.
    Best Regards,
    Bilal

    Hi,
    In my experience this choice depends on the implementation of the datawarehouse. If you are building a "pure" Kimball style dimensional data warehouse you should be able to do this using OWB. I have architected such a DW in the past using only OWB, so I am speaking from experience.
    If on the other hand you are planning to implement an Inmon style CIF, if your requirements includes an operational data store (ODS), or if you for any other reason anticipate that you are going to be doing a lot of ER modeling, then I would not recommend using the current release of OWB for modelling. (Note however that there are significant improvements to the modelling capabilities in the Paris release of OWB, so this may change in the future)
    The advantage of improved maintainability when using a single tools needs to be weighted against the improved functionality if you choose a combination of the two. In the "two tool" scenario strict development and deployment routines need to be enforced to avoid that the model in Designer comes out of sync with the metadata in OWB. (Consider the effect of a developer making a change to a table definition in OWB and deploying it directly to the database without updating the model in Designer.)
    Hope this helps.
    Regards,
    Roald

  • Creation data warehouse par l'outil oracle warehouse builder  (OWB)

    Bonjour,
    Dans le cadre de mon PFE je veux construire un data warehouse par l'outil Oracle warehouse builder.
    je cherche des exemples et un tutorial en français pour faire l'ETL d'une base de données de l'ERP JD Edwards.
    SVP, aidez moi et merci.

    Bonjour,
    For examples try:
    http://www.oracle.com/technetwork/developer-tools/warehouse/otn-collateral-086331.html
    If you need them in French try changing the country / language on Oracle site, not sure what's available though as my French is not good enough to let me read documentation in that language.
    Bonne chance

  • Archiving audit trail using data warehouse builder.

    How do you archive audit trail using data warehouse builder ?
    Can someone please provide any steps in archiving the audit trail table aud$ ?

    Hi.
    Sorry 'bout my ignorance, what is a POC?
    About OWB Installation cenario, it's all up to you. If your target database is a 9i db and you don't want to have a new database instance, then you can place all OWB repositories in this instance. Remember that DB release must be at least 9iR2. For further information, check OWB Installation Guide.
    You'll find all OWB 10g R2 documentation in here:
    http://download-east.oracle.com/docs/cd/B31080_01/doc/nav/portal_booklist.htm
    OWB client may reside in any client machine or even in the host machine (if it's a windows server).
    If you prefer to have a brand new instance to host OWB Repositories, you must create the runtime rep. in the target database (9i), in order to avoid loading performance issues.
    Hope this helps
    Marcosq

  • Strategy in Data Warehouse Table Structure

    I'm building a relational data warehouse, and there are two approaches that seem almost interchangeable to me, despite being quite different from each other. 
    The first approach is rather simple.  I have a "User" table with a bunch of foreign keys, and then I have a bunch of other tables containing user attributes.  One table for "department," another for "payroll type,"
    another for "primary location," and so on for 20 different user attributes.
    The second approach, instead of using 20+ tables, combines this down into far fewer.  I would have an "Attribute Type" table and "Attribute" table.  These two, in conjunction with a bridge table, could accommodate as many
    attributes as necessary within three tables.  If the business wants to track a new "user-related" attribute, I don't need any new tables.  I would simply add the new attribute into the "Attribute Type" table as, say, "attribute
    21," and begin tracking it.  All the work could be done without ever adding new tables or columns.
    Both approaches seem to maintain (at least) 3NF.  Is one approach better in certain circumstances, and the other approach more appropriate at other times?  Any insight is appreciated!
    BrainE

    Hi Brian,
    The second approach with three tables is not really good here. Query Optimizer in SQL Server has a few enhancements for Star/Snowflake schemas in DW environment and 3-table schema would not be able to benefit from them. It would be also harder to maintain,
    load data and query. Finally, your attributes could have different data types, which you need to store. 
    I would suggest to go with first solution (multiple dimensions table) and follow a few extra rules:
    Avoid nullable attributes
    Choose attribute data types as narrow as possible
    Avoid string attributes. If needed create separate dimension tables for them
    Use columnstore indexes and
    Upgrade to SQL Server 2014 if it is all possible - there are multiple enhancementsin batch-mode processing there
    Thank you!
    Dmitri V. Korotkevitch (MVP, MCM, MCPD)
    My blog: http://aboutsqlserver.com

  • Using OBIEE for a custom Data Warehouse

    Hi Everyone,
    I am very new to OBIEE and I have a few questions about this product family.
    1. I have an existing custom build data warehouse, and I would like to know, is it possible to have build reports on this data warehouse?
    2. I understand that OBIEE comes with pre-built ETL jobs in Informatica, what kind of license is it? Is it possible to modify them, or even build new jobs that load into a non-OBIEE data warehouse?
    your answer will be greatly appreciated.
    Jeffrey
    Edited by: user3265404 on Oct 13, 2009 12:50 PM

    Its the same Informatica which can do all functions as a stand alone Infa. additionally it also has prebuilt adapters for source systems like Siebel, APPL, PSFT, JDE, SAP and some universal adapters, so the license included these also which is going to cost more than getting a informatica Licence from Informatica Corp. Moreover, OBI Apps 7.9.6 comes with Informatica 8.6 which is a little older version of the tool. Informatica is going to release version 9 in a couple of weeks.
    I see that you already have a datawarehouse, so why do you need a ETl tool again?
    OBI EE can directly report out of a datewarehouse, and also transactional systems as long as the metadata layer is built.
    PS: Am I clear?

Maybe you are looking for

  • New to Webdynpro for ABAP

    Hi Experts, I am working in Webdynpro Java and Portal Administration for the past three years. I am very much interested in learning Webdynpro ABAP. Please let me know what are the skills (in ABAP language especially) i need to know for working in We

  • Problems with Elements 7

    Hello, I have Windows Vista Home Premium on my laptop and I'm using Windows 8.  I've been using my Elements 7 since 2009.  I've been having problems with it - today I was trying to put some textures (layers) on one of my images and wasn't able to. Th

  • Integrating Pentaho(BI tool) with Liferay Portal (implementing MyFaces)

    Hey friends, i'm a newbie to the Science of Liferay (Portal Servers). and i kind of think this may not be an appropriate forum to post this query and i know that Many people might be bored by this discurssion and i'm sorry as i was not able to get an

  • Command line for replication setup for DS 4.16

    I am using Directory Server 4.16. I see example using command line for replication configuration in DS 5.0. Since 5.0 is quite different with 4.16. So my question is how to use command line to set up a Supplier in DS 4.16? I know we can use ldapmodif

  • Trying to convert from .mts to .flv

    I'm trying out Flash Video Encoder for creating .flv files to add to my web site. My Canon HD HG10 creates .mts video files, which aren't being uploaded when I try to "Add" them. This format is not in the drop-down list. Any suggestions? Scott