Checking data quality

Hi all,
I have two databases: first is oracle and second not oracle, I copied tables from second server to the first(oracle) (there is a lot of data). Now I should have exacly the same data in both databases, but how to check it? Is there any tool or metodology? Of course I can check row by row but it can take a long time maybe checksum on columns...
I can write some script to check but first I have to find some method.
Any idea ?

- I'm not a fan of this sort of requirement because you're going to end up writing a pile of untested code in order to test the code and/or process you've already written. This is both inefficient and subject to the same set of bugs you'd apparently be trying to test for. If your unit tests of the migration process are correct, this testing is pointless, if your unit tests are incomplete, it's much more efficient to focus your effort there.
You could generate a hash for each row in each table in both databases, copy the hashes from one database to another, and compare the hashes, but if you are concerned that somehow the migration process is dropping rows, there hash checking code could just as easily drop or fail to properly compare rows.
Justin

Similar Messages

Data quality check or automation

Apart from passing the report to the user for testing are there ways the process can be automated for a data quality check and how?
Thanks.

Hi Dre01,
According to your description, you want to check the report data quality. Right?
In Reporting Services, the only way to check the report data is viewing the report. So for your requirement, if you want to make this data processing automatically. We suggest to create subscription, it will process the data automatically based
on the schedule and you will get the subscription of report to check if it shows data properly.
Reference:
Create, Modify, and Delete Standard Subscriptions (Reporting Services in Native Mode)
If you have any question, please feel free to ask.
Best Regards,
Simon Hou

How to do data quality check on XI?

Hi XI guru,
I am working on a project, our architect want XI to perform some data quality check to make sure the message contains correct data in future processing. Is there any existing solution or walk arround for XI to perform the data quality check?
For example: if field A and field B is not exist, then XI need to send a email to remind someone who support this interface.
For this kind of scenario, is that possible for XI to handle? What's the best option for XI?

Hi,
check all the condition in UDF and based on the condition result raise an alert.
follow below steps:
Michal Krawczyk
The specified item was not found.
Configuration steps are: go to transaction ALRTCATDEF
1) Define Alert Category
2) Create container elements which are used for holding an error messages.
3) Recipient Determination.
https://www.sdn.sap.com/irj/sdn/wiki?path=/display/xi/alert%2bconfiguration%2bin%2bxi
Alert can be triggered in different ways.
1) Triggering by Calling a Function Module Directly.
Triggering XI Alerts from a User Defined Function
chirag

CDS Data Quality Health Check Error -(Code: 207,103)

Using the CDS Individual Data Quality Health Check.
Created a new data source, snap shot and mapping. Cloned the job and modified the snapshot an mapping. Ran the Job and go the following :An error occurred whilst creating task '[I2A] Profile Individual Misc Data' : Failed to load process "[I2A] Profile Individual Misc Data" (Code: 207,103) - Any Suggestions

Determined the Error to be a missing processor. I must have deleted it by mistake. You can identify which one is missing by the processor Name [I2A] Profile Individual Misc Data

DATA QUALITY connection on INGRES

Post Author: ryefrancis
CA Forum: Data Quality and Data Insight
Hi,
Would like to ask if any you guys have any idea on the error prompt "NO COLUMNS SHOULD BE RETRIEVED FOR THIS TABLE. A POSSIBLE REASON IS THAT YOU DO NOT HAVE SELECT PERMISSION FOR THIS TABLE OR DATABASE"?
The database being used is INGRES. We already tried to update its
security settings and Grant Access still it won't establish a
connection.
But when we try using SQL statements instead of SQL HELPER we could get some reports.
Hope you could have any suggestions and solutions to this problem.
Good Day!!!

Ingres is not supported for Data Insight however if you will send some additional information we may be able to check if anything obvious is missing.
Two items that may help are the connection information from the Data Insight repository (iqi_connection table) for this source and screen shots of the populated connection dialogs.

Data Quality - Failed to create connection to MDR.

Hi All,
I have installed Data Quality 11.7.2 on windows 2003 server using SQL 2005 (Both SQL 2005 server and Data Quality are in the same server) , The install was successful. But when I try to Open 'Data Quality Project Architect' it throws an error
Failed to create connection to MDR.
Driver error messgage = qodbc_mssql:
Unable to connect, Database error message =] [microsoft][odbc sql server driver]
[ tcp/ip sockets] sql server does not exist or access denied
I checked the ODBC connection and tested it successfully too.
I am not able to pass through the 'Meta Data Repository Connection' as its failing in 'Setup SQL connection option'
Any help is appreciated !!!
Thanks
Ranjit Krishnan

Hi Paul,
Thanks for you reply, OBDC connection was created under System DSN, also tired the users DSN option too, but did not work.
Recreated the ODBC connection many times still no luck.
The 'Project Architect' client and the MDR (SQL 2005 Server) are on the same machine.
Thanks
Ranjit Krishnan

Question on CKM and Data Quality

As I understand, CKM only supports the check based on db constraints. If I want to have more complicated business logic built to the checking, does Data Quality sound like a good choice. Or other suggestions?
In my case, I will need to check the data in the source table based on the table data from different sources (both target and source tables). This should be doable through data quality, correct? I am new to ODI. When I first installed the ODI, I didn't choose to install data quality module. I suppose I can install DQ separately and link it back to ODI? Do they share the same master repository?
Sorry for the naive questions, you help is greatly appreciated.
-Wei

Hi,
My idea is just like:
for instance a FK validation:
create function F$_validate_FK (value1 number) return number
as
v_return number;
begin
select my_fk into v_return from any_table where column_fk = value1 ;
return v_return ;
exception
When NO_DATA_FOUND then
return -1;
end;
And at constraint you will have:
-1 = (select F$_validate(table1.column_to_be_validated) from dual)
Any record that have -1 as return will be not valide for the flow.
The F$ function can be created in a ODI procedure before the interface and dropped at end if you think to be necessary.
Make any sense?
(Maybe there are several syntax error in this example, I just write it and did not compilate, just to show the idea)
Edited by: Cezar Santos on 28/04/2009 10:20

Data Services and Data Quality Recommnded Install process

Hi Experts,
I have a few questions. We have some groups that have requested Data Quality be implemented along with another request for Data Services to be implemented. I've seen the requested for Data Services to be installed on the desktop, but from what I've read, it appears to be best to install this on the server side to allow for more of a central benefit to all.
My questions are:
1. Can Data Services (Server) install X.1 3.2 be installed on the same server as X.I 3.1 SP3 Enterprise?
2. Is the Data Services (CLIENT) Version dependent on if the Data Services (Server) install is completed? Basically can the u201CData Services Designeru201D be used without the Server install?
3. Do we require a new License key for this or can I use the Enterprise Server license key?
4. At this time we are not using this to move data in and out of SAP, just using this to read data that is coming from SAP.
From what I read, DATA Services comes with the SAP BusinessObjects Data Integrator or SAP BusinessObjects Data Quality Management solutions. Right now it's seems we dont have a need for the SAP Connection supplement, but definetly something we would implement in the near future. What would be the recommended architecture? A new Server with tomcat and cmc (seperate from our current BOBJ Enterprise servers)? or can DataServices be installed on the same?
Thank you,
Teresa

Hi Teresa.
Hope you are referring to BOE 3.1 (Business Objects Enterprise) and BODS (Business Objects Data Services) installation on the same server machine.
Am not an expert on BODS installation.
But this is my observation :
We had recently tested on a test machine BOE BOXI 3.1 SP3 (full build) installation before upgrade of our BOE system.
We also have BODS in our environment.
Which we also wanted to check whether we could keep on the same server.
So on this test machine, which already has BOXI 3.1 SP3 build, when i installed BODS server installation,
what we observed was that,
all the menus of BOE went away
and only menus of BODS were seen.
May be BODS installation overwrites/ or uninstalls BOE, if it already exists ?
I dont know. Though i could not fine any documentation, saying that we cannot have BODS and BOE on the same server machine. But this is what we observed.
So we have kept BODS and BOE on 2 different machines running independently and we do not see any problem.
Cheers
indu

Terms of payment in IDoc E1EDK18 have been transferred: Check data

hi Firends,
In my scenario , i have to convert incoming purchase order to sales order in my system . I am able to get message from partner and idoc is created to the message bt it contains error,
tht is Terms of payment in IDoc E1EDK18 have been transferred: Check data
related to this I had gone through note 388120 - Transfer of conditions and terms of payment.
which says that i have to maintain tht entry in SMME tcode.
My confusion here is, while I tested same scenario in development , and quality server , it was not giving any such error bt in prod system i m gettin this error.
Also if some one knows more details abt this note ,let me know.
Thanks ,
Brijesh Soni

Hi Brijesh,
you are more likely to get a response for this query if you create a thread in the SD forum.
Regards
Kenny Scott
XI/PI SDN Forum Moderator

Data quality in ODI 11g

Hi all,
I want to use DQ tool for validating the source(comlex file). My all validations are mathmetical and complicated.
So it possible by doing with Oracle data quality tool which in ODI11g?
Regards,
Suresh

I once used a small ETL tool esProc to combined with Data Quality( to analyse the stock data).
It is famous for handling the complicated mathematical computation and statistical analysis, and its performance is also acceptable.
Check here for details about esProc

The Data quality knowledge base property is empty

Hello
I build Knowledge base and published it successfully (using the DQS client wizard normally )
But when i open new ssis project and try to use the DQS component i can't use the knowloedge base i build , i get the message The Data quality knowledge base property is empty
How can i be sure that i published successfully (any query on repository ?)
What do i miss ?

Hello,
Use the Data Quality Client to check in the Open Knowledge Base
screen if the knowledge base was published. To do so, click Open Knowledge Base
in the home screen of Data Quality Client, and then check the Date Published column.
Alternately, you can also check the PUBLISH_DATE column in the DQS_MAIN.dbo.A_KNOWLEDGEBASE
table.
Thanks,
Vivek

SAP Data Quality Management: how to replace input fields

Hi all,
the input fields in the SAP DQM 4.0 Real time job Job_Realtime_DQ_SAP_Name_And_Address_Match are:
REC_NUM
NAME_FIRST
NAME_LAST
NAME1
NAME2
HOUSE_NUM1
STR_SUPPL1
STREET
CITY1
COUNTRY
POST_CODE1
REGION
ADDR_TYPE
MTCSTD1
MTCSTD2
PO_BOX
POST_CODE2
THRESHOLD
I want to replace one of the fields (i.e ADDR_TYPE) with "Account_group" column.
The idea is replace, not create of additional fields ( as described in SAP BusinessObjects Data Quality Management (DQM): Enhancements for more fields other than address data for Duplicate Check SAP Business Objects Data Quality Management ( DQM ) Enhancements for more fields other than address data for Duplicate Check.).
Could you please let me know if the standard XML file sent by the RFC server contains the Account_group column.
Any other suggestions and ideas are welcome
Thanks in advance
Goran

Hi,
the additional data tab is controlled by means of classes created in DMS.
if your requirement is something like,
project name
responsible person name etc..
for this - in CL01, you can create class, say Project XYZ
in CT04 - you can create characteristics as project name and person name..etc for this class.
in each characteristics you can maintain the values which will serve the requirement.
hope this helps U..!
Thank You,
Manoj
award points if useful..!

Verification of data quality in migration process

Hi All,
I am in a project that migration data from SQLserver to Oracle database. But my quesion is not performance but the check of data quality.
My procedures to move data is: a) Extract data to a flat file from SQLserver via a GUI tool b) ftp it to UNIX c) sqlldr to Oracle temp tables d) copy the data from temp tables to fact tables.
My point is to only check the sqlserver log file and sqlldr log file, if no error in them and the row counts match in SQLserver and Oracle, then we can say a,b,c are successful.
And since d is a third party stored procedure, we can trust its correctness. I don't see any point where the error could happen.
But the QA team think we have to do at least two more verification: 1. compare some rows column by column 2. sum some numeric columns to compare the results.
Can someone give me some suggestion on how you check the data quality in your migration projects, please?
Best regards,
Leon

Without wishing to repeat anything that's already been said by Kim and Frank this is exactly the type of thing you need checks around.
1. Exporting from SQL Server into a CSV
Potential to loose precision in data types such as numbers, dates, timestamps, or in character sets (unicode, utf etc)
2. Moving from windows to unix
Immediately there are differences in EOL characters
Potential for differences in character sets
Potential issues with incomplete ftp of files
3. CSV into temp tables with SQL Loader
Potential to loose precision in data types such as numbers, dates, timestamps, or in character sets (unicode, utf etc)
Potentail to have control files not catering for special characters
4. Copy from temp tables to fact tables
Potential to have column mappings wrong
Potential to loose precision in data types such as numbers, dates, timestamps, or in character sets (unicode, utf etc)
And I'm sure there are loads more things that could go wrong at any stage. You have to cater not only for things going wrong in the disaster sense i.e. disk fails, network fails, data precision lost, but also consider there could be obscure bug in any of the technologies you're working with. They're not things you can directly predict but you should have verification in place to make sure you know if something has gone wrong - however subtle.
HTH
David

Trillium Data quality connector for SAP

hi
this is my first time in this forum, so if this is not the correct area to post this, mods please move the appropriate area.
as the subject mentions, i just want to know if anyone have any business documents regarding this data quality connector for SAP. any kind of documentation will be helpfull. thanks

Please check the newly created MDM Data Enrichment page on SDN. It provides useful information: https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/webcontent/uuid/60368290-4d9e-2910-1480-bb55167ee3c3. [original link is broken]
Regards,
Markus

Address verification - Data Quality

Hi guys,
I am trying to do some research to understand if you (ORPOS) customers see a need for Address, Phone & EMail Verification to improve data quality?
If you do, please let me know where is your biggest pain with the data quality? which forms or module if you had an Address, Phone or EMail verification solution integrated would make your life and improve ROI for your company
Thanks!

Hello Ida,
Address Verification in OEDQ is comprised of the Address Verification API, and a Global Knowledge Repository (also known as Postal Address File).
A subscription to a Postal Address File must be purchased directly from a provider, and Oracle's prefered partner for this is Loqate, (http://www.loqate.com/).
See explanation here for details: https://blogs.oracle.com/mdm/entry/enterprise_data_quality_integration_ready
The Address Verification and Standardization service uses EDQ Address Verification (an OEM of Loqate software) to verify and clean addresses in either real-time or batch. The Address Verification processor is wrapped in an EDQ process – this adds significant capabilities over calling the underlying Address Verification API directly, specifically:
Country-specific thresholds to determine when to accept the verification result (and therefore to change the input address) based on the confidence level of the API
Optimization of address verification by pre-standardizing data where required
Formatting of output addresses into the input address fields normally used by applications
Adding descriptions of the address verification and geocoding return codes
The process can then be used to provide real-time and batch address cleansing in any application; such as a simple web page calling address cleaning and geocoding as part of a check on individual data.
The Installation and Configuration of Addess Verification with OEDQ and Loqate is documented here: Installing and Configuring Address Verification
Best regards,
Oliver.

Checking data quality

Similar Messages

Maybe you are looking for