How validation & transformation handled in ETL

Hi,
I am new to OWB (We are using Oracle 8i). I have to extract data from source, do the required validation and transformation and then finally load the data into target table.I need help on validation and transformation handled in ETL.
Illustration:
source table emp_src & target table is emp_target. I have to validate deptcd (should present in dept master). While loading i have to replace these invalid dept code with some dummy value maintained in parameter table. Insert all the correct records and errorneous records with transformations into target table.
Also, i want to store all these invalid records in error table or error file.
My question:
1. How these can be handled in ETL
2. Should i have to modify the pl-sql procedure/jobs
generated by ETL.
3. Can i have my own procedures to accomplish the above
task
4. Can i execute my procedure within the ETL generated
procedure.
Thanks,
sanjay rastogi

I will try to answer your questions:
1) You could do this using a "key lookup" on deptcd against the dept master. The key lookup returns NULL for each invalid value. You could then use an expression to replace the null values with a default.
In order to log the errors you could use a "split", which enables you to insert the erroneous data into a log table.
2) No. I have yet to come across a situation where I have had to modify the generated PL/SQL. (Doing so will cause serious issues regarding maintenance, upgrades etc.)
3) You can use your own procedure. Simply write your PL/SQL package/procedure/function and import it into your OWB transformation library.
4) Yes. Same as before, import your own procedures, and use them in mappings as Transformations or in Expressions
Hope this helps.
Roald

Similar Messages

  • [ETL]How to transform a Source model into an arbitrary number of target models ?

    Hi all,
    I need to know how to transform a source model into an arbitrary number of target models which conforms to different target metamodels using ETL ? Is there any examples that can help me ?
    Thank you
    Mana

    Hi Antonio,
    Actully I mean the number of metamodels are known.For example, assume that there are two known metamodels in the target side, the first one is the target metamodel, and the second one is the trace metamodel .
    I would like to see an example of ETL transformation that run on a source model and generate the target model and the trace model as well. This statement is part of the chapter 5 of the ebook which says "it is often essential to be able to access/update additional models during a transformation (such as trace or configuration models). Building on the facilities provided by EMC and EOL, ETL enables specification of transformations that can transform an arbitrary number of source models into an arbitrary number of target models."
    could you please explain this more for me or provide me with an example.
    Thank you
    Mana

  • How can i handle error in my apex form

    Hi experts,
    I have designed a form where i record values.I have a create button and when i press it without entering any values i have the following error. How can i handle tat error so that if i dont enter any values i have an error message being displaying prompting the user to enter values
    ORA-01400: cannot insert NULL into ("LOTTO_BI"."HOURLY_SALES"."ACT_AMT")
         Error      Unable to process row of table HOURLY_SALES.

    As far as I understand your problem you just have to add a validation to your page processing.
    In the third step when creating a validation you can choose "Item not null".

  • Ways to retrieve Validation transform information for documentation

    Hi All,
    We have a requirement which needs to document validation rule in all validation transform of data flow in a project.
    Does anyone know if there is a way which we can retrieve validation rules from DI repository? E.g. a SQL statement.
    I searched the DI repo tables but no luck.
    Hope someone can give me some light.
    Thanks,
    Bobby

    In the Auto Documentation in the Management Console, you can browse all the objects in your repository and for the validation transform it will show the rules used in that transform. This is for browsing only though, the print feature does not print the details for any of the transforms, so also for the Validation transform the details (i.e. the validation rules) are not printed.
    Unfortunately, today there is not an easy way to extract the validation rules via a simple query. It is a common request though that we will address in a future release (import/export of validation rules).
    That being said, technically there are some options to get access to the rules. Not straightforward though...
    The validation rules are part of the transform definition, which on its turn is part of the whole dataflow definition. So they are not stored as separate objects in a table somewhere, but embedded into the ATL language we use to describe the DI objects. What you could do is export the repository to an ATL file (or select from AL_LANG_TEXT table in the repo) and scan this file for the validation rules. Below is an example of how such a rule would look like, in this case the rule name is "MyRuleName" and the rule is  : Query.JOB_KEY = 1.
    As a side note, the part that would be easy to get via a SQL query are the run-time statistics of the validation transform. These statistics (number of records passed/failed for each validation rule) are stored in the repo tables al_qd_* and are also used by the Validation Dashboards in the management console. Keep in mind that you need to check the option to 'collect  data validation statistics' in order to collect these details.
    CALL TRANSFORM Validation ()
    INPUT(Query)
    OUTPUT(Validation_Pass ( JOB_NAME varchar(192),
    JOB_KEY int,
    JOB_RUNID varchar(384),
    RUN_SEQ int,
    PATH varchar(765),
    OBJECT_NAME varchar(765),
    OBJECT_TYPE varchar(765),
    ROW_COUNT varchar(765),
    START_TIME varchar(765),
    END_TIME varchar(765),
    EXECUTION_TIME varchar(765),
    DATAFLOW_NAME varchar(765),
    JOB_ID int ) ,
    Validation_Fail ( JOB_NAME varchar(192),
    JOB_KEY int,
    JOB_RUNID varchar(384),
    RUN_SEQ int,
    PATH varchar(765),
    OBJECT_NAME varchar(765),
    OBJECT_TYPE varchar(765),
    ROW_COUNT varchar(765),
    START_TIME varchar(765),
    END_TIME varchar(765),
    EXECUTION_TIME varchar(765),
    DATAFLOW_NAME varchar(765),
    JOB_ID int,
    DI_ERRORACTION varchar(1),
    DI_ERRORCOLUMNS varchar(500) )  )
    SET("validation_rules" = '<?xml version="1.0"; encoding="UTF-8"?>
    <Rules collectStats="true" collectData="false" >
    <Column name= "Query.JOB_KEY"; enableValidation="true" noValidationWhenNull="false" >
    <RuleName> MyRuleName </RuleName>
    <Description></Description>
    <Expression uiSelection="1">
    <UIValue1>=</UIValue1>
    <UIValue2>1</UIValue2>
    <Custom> Query.JOB_KEY = 1 </Custom>
    </Expression>
    <Action sendTo="0" substOnFail="false" substValue="" />
    </Column>
    </Rules>

  • Transform handles are moving the object and not scaling.

    I've been running into trouble with the transform handles on objects moving the object instead of scaling/stretching the object. It seems to happen on the bottom and right sides vs the top and left sides. When I have the mouse over the handle it shows the scaling icon, but when I press the mouse it turns into the black selection arrow. Is anyone else experiencing this and know how to fix it?

    kjredin,
    I am afraid you have come across the Live Rectangle bug which is limited to the MAC versions starting with 10.7 and 10.8, but not 10.9 (Mavericks) or 10.10 (Yosemite). Hopefully, the bug will be fixed soon.
    So a switch to Mavericks or Yosemite with a reinstallation might be the way to solve it here and now.
    To get round it in each case, it is possible to Expand the Live Rectangles to get normal old fashioned rectangles, or to Pathfinder>Unite, or to use the Scale Tool or the Free Transform Tool.
    A more permanent way round that is to create normal old fashioned rectangles, after running the free script created by Pawel, see this thread with download link:
    https://forums.adobe.com/thread/1587587

  • Large field even after validation transform

    Hi, I am working on BODS 4.0 with Oracle 11.2 as Repo. I am using validation transform to calculate the length of the string . Validation will fail if record length is greater than 60. I am sending pass records to table temp1 and send failed records table temp2. I have defined the field length to 60 in temp1.
    After this when I run Job then I get error on table temp1 that record length is 65 and field is of 60. I am working with Russian data. Russian is multi byte character CYRILLIC data.
    Now if SAP has field defined as  varchar(60) then how can I make sure that only records with length 60 or less are send.
    Even though I am checking length 60 in validation but inactual it is taking more space and can not be inserted into field of length 60.
    Thanks,

    Hi,
    I've done the same test on BODS 3.1 ver and didn't see any error messages and Job run successfully with correct results.
    I've given one field as 90 chars.
    Please let me know if you need any further help . My emails is mohan.salla at telegraph.co.uk
    cheers

  • Validation/Transformation Question ....

    We need to use the  abbreviations in the Name field in the  Vendor master. The incoming data may be with full name but the business requirement is to use the abbreviations for example :-
    INCORPORATED  must be INC
    ORGANIZATION  must be ORG
    MANUFACTURING must be  MFG
    HOSPITAL must be  HOSP
    The above are some of the examples. The requirement is to replace the full name with abbreviations where ever that is present.
    How do we handle this in MDM. I noticed in the data manager there is a tab called transformation where we can specify the Table field name then the Old value and new value but donu2019t know how to execute it. Is Data Manager is the place to handle this?.
    Anybody help me with this?.

    Hi JP,
    Yes,you are right.Datamanager is responsible to do transfomations.As you said we have to
    specify the field name on which you want to create the transformation and also specify
    the value to be replaced in From field and new value in the To field.
    Suppose for example
    Specify INCORPORATED in from field and INC in To field, then INCORPORATED is replaced
    with INC.if you don't specify anything in To field, it just simply deletes the specified value
    available in from field.
    Token option is also availble to replace the values tokenwise.
    To get more details go through below link:
    Re: Transformation in matching mode in data manager
    Cheers
    Narendra

  • How to Exception Handler

    Hi Experts,
    How to handle exception in WebDynpro  GP callable objects.
    I created exception parameter in getdescription() method like
    IGPExceptionInfo processExc1 = technicalDescription.addProcessException("USER_NOT_FOUND");
      processExc1.setNameKey("USER_KEY");
      processExc1.setDescriptionKey("USER_DESCRIPTION_KEY");
      processExc1.setFatal(true);
    In my Process 2 levels, in block level i am able to see above created exception in exception tab. I create one extra action with same callable object mapped, this action is mapped to exception handler & selected repeat from dropdown.
    If any exception occurs execute method(like UMException) how to call handler from coding. I tryed this code to call handler
    String localizedMessage =textAccessor.getText("USER_NOT_FOUND");
      wdThis.wdFireEventTechnicalException(new GPTechnicalCallableObjectException(logger,localizedMessage,e));
    this is not working...   
    Please any one tell how to call handler from WebDynpro  callable object
    Regards
    Satya

    Inside a UI side Exception handling task-flow How can I distinguish validation exception and other exceptions..?
    Note: Validate entity method in EOImpl might be throwing validation exceptions. But it is not aware of any UI side exception handling.

  • How do I handle my Final SELECT based on a @ReportType Run Parameter

    So I have multiple CTEs in my SQL...
    WITH CTE_1 AS
    (SELECT...)
    CTE_2 AS
    (SELECT...)
    CTE_3 AS
    (SELECT...)
    My Final SELECT then will be based on my CTE drill-downs and the filtering will be based on the @ReportType that will be chosen within SSRS and Report Manager. How can I handle the varying portions of my FINAL SELECT based on @ReportType? Do I have to do
    this via dynamic SQL? I know that dynamic SQL is frowned upon. So this is what I'm talking about...
    SELECT ...
    CASE WHEN @ReportType = 'Renewal People'
    THEN INNER JOIN...
    WHEN @ReportType = 'New People'
    THEN NOT IN (SELECT...)
    END
    CASE WHEN @ReportType = 'Renewal People'
    THEN WHERE...
    WHEN @ReportType = 'New People'
    THEN WHERE...
    END

    > I know that dynamic SQL is frowned upon.
    That depends. I love (*) dynamic SQL for one thing:
    http://www.sqlusa.com/bestpractices/dynamicsql/
    Second, there are many things you cannot in static SQL:
    http://www.sqlusa.com/bestpractices2008/rebuild-all-indexes/
    * First choice is static SQL always.
    Kalman Toth Database & OLAP Architect
    SQL Server 2014 Database Design
    New Book / Kindle: Beginner Database Design & SQL Programming Using Microsoft SQL Server 2014

  • In ldap how do you handle to users logged in at the same time

    in ldap if i am logged in as one user and make a change to a page and another user makes another change how do i handle refresh the other persons logged in pages...

    Hi,
    to which page do you add a change and what has LDAP to do with this ?
    Frank

  • How do you handle multiple at the same address?

    My wife just got an iMac. She is using the Birthday feature in Address Book and iCal to track our friends and family birthdays and anniversaries.
    Question:
    How do you track it when multiple people are at the same address? For instance, our friends are a family of four, but they all live at the same address. Do you need to make a separate Address Book card for each of them, even the 1 year old baby? Or can you track multiple people in a single card with custom fields?
    If you do need to make multiple cards, then how do you handle mailing lists? For instance, if I make separate cards for all four of them, how do I make sure I only send on Christmas card when I use my Mac to print labels?
    Thanks for any advice.

    This is really an issue that you're going to have to resolve yourself since Apple's address book isn't really built with this in mind - darn it. For me there are three scenarios. First, a singleton. No problem. Second a family for which all members share the same information - acquantences to whom I don't send birthday greetings. These get one card and I'll put in both (or all) their names in the first name field (as in Bob and Jean). If there are children, in the last name field I'll add 'and Ken' or 'and the kids'. Third, a family for which some fields are different - perhaps cell phone number, birthday, etc.
    For mailing purposes, I create a holiday group and put the people I'll be sending cards to into the group.

  • How does IITTrackCollection handle ItemByName if there are multiple tracks by the same name?

    I am writing a program in C# that is trying to check if a track already exists in the iTunes Library before adding it (thus preventing duplicates).  The best method I have come up to do this with is to use 'MyPlaylist.ItemByName(SongName)' to retrieve a possible duplicate and then compare the rest of that track's attributes (i.e. the artist and album).  However, I have several examples of tracks in my library which have the exact same name, but are in fact completely different and unrelated.  How is this handled by the iTunes COM?  Or is there a better way to do it?

    Hi DSW,  
    Thanks for your post.
    In your query result, please check if these two users have the same Account Name. if they are two different Account Name in result, it indicate there’s two users have the same display name in your AD, please check that two users’ information in
    your AD. We suggest change one user’s display name in AD.  
    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click
    HERE to participate the survey.

  • How can we handle browser settings while dealing with the security ?

    Hi ,
    how can we handle browser settings while dealing with the security ?When we configured security in web.xml , during the first request the container is asking for the authentication credentials once they are provided it go's on. but when the user gives a fresh request from the second window within the same browser that time it is not asking for authentication. How can we overcome this.Is there anything to do with server configurations?
    How can we make the container no to keep the things or act like session?

    Ya... I am taking a small example need not happen always but a kind of possibility i am thinking off.
    once the user sign out and just left without closing the browser and a friend (suppose not a good friend ... just kidding...) of that user may open the same jsp or file .This time the security is breached. If that feature or property exists....
    I know what you might say ... the user will log-out before leaving where a programer might invalidate the session at the time of log out.
    Consider the case of a bad Programing or just a programer might forget to invalidate,At that time as a application administrator how can he solve that issue.
    Thanks.......
    Edited by: user8483670 on Jun 6, 2011 1:08 AM
    Edited by: user8483670 on Jun 6, 2011 1:09 AM

  • How best to handle multiple web sites using iWeb 08

    I currently am using iWeb08. I am maintaining two websites; The church site resides on a Rogers server. I use Transmit as a FTP loader. The other site is an engineering society's website that is posted on Apple's server. I have an ME account just for the latter group. iWebsites is used to alternate between the two sites. The church group has a .ca web address.
    Is it worth upgrading to iWeb 09? How does iWeb09 handle 2 websites? I wish I could publish both to the Mac account but have not figured out how. I must retain the rogers account for viewing the church site
    Has anyone deleted iWebsites from OX10.5.6? How was it done? I know that backups are mandatory before attempting these changes.
    Thanks,
    Bill

    In its current form, iWeb '09 doesn't handle publishing well at all!
    I keep all my sites on separate domain files, each in their own folder. Any site is launched in iWeb by double clicking the domain file. I start each new site from a new, blank domain file.
    I keep all my website folders in a folder in a second dock so that I can launch any site with two mouse clicks - faster than you can launch iWebsites!
    You can publish as many sites as you want to one MobileMe account but you can only have domain name forwarded as Cname. Any more have to use masking with all its inherent problems.
    I would be more inclined to dump MobileMe and publish both sites to a decent hosting company.
    The new iWeb '09 FTP works for some. I have tested it with my server - Host Excellence - which doesn't force you to upload to a Public_html folder and this goes as planned.
    Having said that, I still publish to a local folder as I optimize and upload my files using Web Site Maestro. Once you have published your site for the first time, this application will then use its Smart Handling feature to process the changed files only.
    Unless all your viewers use Macs you have to optimize to get your pages to download in that browser that all the Fred Flintstones of the world use.

  • How do you handle an incoming call while playing a game?

    If you're playing a game on your iPhone, how do you handle an incoming call? A call will stop my app from running, but how do I save the state and restore it when the user starts the app again?
    I have a bunch of objects on-screen and I need to record where they are so that the user doesn't lose the progress of their game.
    I've looked through the SDK, but I can't see anything obvious. Any ideas?

    I don't have a device to test with, but I suspect you'd want to save state in one of the UIApplicationDelegate callbacks:
    applicationWillResignActive:
    applicationWillTerminate:

Maybe you are looking for