Compare table data with data in csv file

Hi, I have a database table with ~32 milj rows. I want to compare the data from a csv file(~ 1500000 rows) with the data in a specific database column. And as a result I want to see what rows exists in both the csv file and db and what records in csv file are unique. The text I'm searching for is VARCHAR2. I am not allowed to export the database table so that's not an alternative. This is also a production environment so one need to be careful not disturbing the production.
Rgds J

2c0eaa24-46c9-4ff7-9f01-564f37567899 wrote:
Hi, I have a database table with ~32 milj rows. I want to compare the data from a csv file(~ 1500000 rows) with the data in a specific database column. And as a result I want to see what rows exists in both the csv file and db and what records in csv file are unique. The text I'm searching for is VARCHAR2. I am not allowed to export the database table so that's not an alternative. This is also a production environment so one need to be careful not disturbing the production.
Rgds J
How do I ask a question on the forums?
https://forums.oracle.com/message/9362002#9362002
Does CVS fille reside on DB Server system now?

Similar Messages

How to import data from excel or csv files to Oracle table

hello everybody,
I am new here and new in Oracle. I would like to know the steps how to import data from excel or csv files to Oracle table.
Let say I already have table inside the Oracle. Then my user give me the sets of data inside the Excel Worksheet.
So, how can I import the excel data into Oracle table.
Thank you in advance.
cheers,
shima

Even easier. Download JDeveloper 11G from this site.
Set up the database connection, right click on the table, select Import->Excel and specify your file to load it. On the import pop-up, you must view and update each tab indicating Columns, Data Types, and DML.
Columns -- move the selected columns that you want to load to the box on the right
Data Types -- select column name from second column to which the data for each column of the import file should load
DML -- click this tab to generate the INSERT SQL
Once done click 'Insert'

Download data from db to CSV file and need to handle chinese character

Hi , all , I need to write to alow to export data from db to csv file, now the problem is some column is chinese character, in both db and web page it can dispaly the chinese, but in Csv I can't display chinese character, what should I do?
Thnak you

HJava wrote:
I already have res.setCharacterEncoding("UTF-8"); in my servlet, and db
req.setCharacterEncoding("UTF-8");
Are you sure that you have set it on the response? You do not need to set it on the request. Have you read the article? Do you understand the world of characters? It look like you still don't understand it.
by the way, since csv is text file , how can I bold my University number??? in sb = new StringBuffer();
sb.append("University Number");Not possible with CSV since it is text/plain. Use real XLS if you want to add formatting to cells.

Performance question when compare date with date or char with char

Hello from Germany (Frankfurt) !
I am working on Oracle 9.2.0.8.0
and have to solve following problem:
Comparison of a date and char fields.
Now two ways to do it.
Either I compare char with char and convert date to char,
or I compare date with date and convert char to date.
Now the performace question. Which operation takes more effort for the database?
So why not to try and to see the results?
First create table:
CREATE TABLE TEST (
char_date VARCHAR2(8),
real_date DATE
NOLOGGING;
Then insert 1.000.000 rows
BEGIN
for x in 1..1000000
loop
insert into test (char_date, real_date) VALUES('19990101', TO_DATE('2006.01.01', 'YYYY.MM.DD'));
end loop;
COMMIT;
END;
Collect statistics
EXEC dbms_stats.gather_table_stats('TESTER', 'TEST');
Now run some selects for date to char conversion:
Elapsed: 00:00:00.00
SQL> select * from test t where TO_DATE(char_date, 'YYYYMMDD') > real_date;
no rows selected
Elapsed: 00:00:03.02
SQL> select * from test t where TO_DATE(char_date, 'YYYYMMDD') > real_date;
no rows selected
And some selects for char to date conversion:
Elapsed: 00:00:03.02
SQL> select * from test t where char_date > TO_CHAR(real_date, 'YYYYMMDD');
no rows selected
Elapsed: 00:00:02.05
SQL> select * from test t where char_date > TO_CHAR(real_date, 'YYYYMMDD');
no rows selected
Elapsed: 00:00:02.05
SQL>
As you see when I compare char with char and convert date to char it seems to be faster (almost 1 second)
Is the test correct?
I still not sure, what gets better performance...
Any idea?
Thanks!

Depends on whether you want the right results or not.
Why don't you run the following two queries and see if the difference in results tells you anything?:
with t as (select to_date('01/02/2007', 'dd/mm/yyyy') date_col from dual
           union all
           select to_date('02/02/2007', 'dd/mm/yyyy') from dual
           union all
           select to_date('03/02/2007', 'dd/mm/yyyy') from dual
           union all
           select to_date('03/03/2006', 'dd/mm/yyyy') from dual)
select *
from   t
where date_col < to_date('04/03/2006', 'dd/mm/yyyy');
with t as (select to_date('01/02/2007', 'dd/mm/yyyy') date_col from dual
           union all
           select to_date('02/02/2007', 'dd/mm/yyyy') from dual
           union all
           select to_date('03/02/2007', 'dd/mm/yyyy') from dual
           union all
           select to_date('03/03/2006', 'dd/mm/yyyy') from dual)
select *
from   t
where to_char(date_col) < '04/03/2006';

Unique file name to be decoded and to be updated in a table along with data

Hi
I'm working on a File to Proxy scenario, where the file names(10 chars length) are unique. These files will be available for XI in a source directory. My requirement is -- file name need to be decoded into 3 values and to be updated into a r/3 database table along with the file data.
Hope my requirement is clear.
Thanks.

You cannot see the field which stores the file name. The file name comes from payload at runtime.
Secondly no need to create any input parametes in your UDF, just edit your udf and delete the input parameter (default input is 'a') so that you don't have to map any constant to this UDF just map this UDF to target field.
e.g. UDF ---> target Field.
I did the same but am not able to activate the mapping.
Mapping activation Error:
Activation of the change list canceled Check result for Message Mapping hello_mapping | http://briks.com:
Mapping not sufficiently defined
Chennai.

Best data Structor for dealing with very large CSV files

hi im writeing an object that stores data from a very large CSV file. The idea been that you initlize the object with the CSV file, then it has lots of methods to make manipulating and working with the CSV file simpler. Operations like copy colum, eliminate rows, perform some equations on all values in a certain colum, etc. Also a method for prining back to a file.
however the CSV files will probly be in the 10mb range maby larger so simply loading into an array isn't posable. as it produces a outofmemory error.
does anyone have a data structor they could recomend that can store the large amounts of data require and are easly writeable. i've currently been useing a randomaccessfile but it is aquard to write to as well as needing an external file which would need to been cleaned up after the object is removed (something very hard to guarentee occurs).
any suggestions would be greatly apprechiated.
Message was edited by:
ninjarob

How much internal storage ("RAM") is in the computer where your program should run? I think I have 640 Mb in mine, and I can't believe loading 10 Mb of data would be prohibitive, not even if the size doubles when the data comes into Java variables.
If the data size turns out to be prohibitive of loading into memory, how about a relational database?
Another thing you may want to consider is more object-oriented (in the sense of domain-oriented) analysis and design. If the data is concerned with real-life things (persons, projects, monsters, whatever), row and column operations may be fine for now, but future requirements could easily make you prefer something else (for example, a requirement to sort projects by budget or monsters by proximity to the hero).

Wrong date format when import CSV files

When you import a CSV file that contains fields with German date formats, these fields are displayed incorrectly.
Example: Contents of the CSV file "01.01.14". After importing the corresponding cell in Numbers has the content "40178".
A reformat the cell to a date format is not possible.
How do I get the date in the new version of Numbers displayed correctly?

It seems that there are more than a few problems related to import/export with non-US localizations.
I, personally, don't have a solution to your problem. I started to adjust my Language & Region settings to test your problem but it was several settings, I didn't get it right, and I didn't want to mess up my computer so I set everything back to US/English.
The only workarounds I can suggest are
Insert a new column into your table and in it put a formula that adds the number to the date 01.01.1904. Or,
Edit the CSV in TextEdit to Replace All "." with "/". This will work if "." is used for nothing else but these dates.
I recommend the second one if it will work for you. Hopefully Apple is addressing problems such as the one you are seeing.

Export data with Dreamweaver (TXT, CSV)

Hi to all of the Adobe community.
My question to me is a little hard for my level of knowledge.
I'm using Dreamweaver using technology PHP / MySQL, in my database have a table in MySQL called mailing list, with the fields name and email.
I have a php page that I developed using the Dynamic Table where it returns me a list of names and e-mail registrations. What I would like to make is that at the end of this table I have a link or a button called export data. I wish that when you export this data be txt or csv format.
How?
Export data and further define the format to be exported.
I'm using Dreamweaver CS4.
Rodrigo Vieira da Silva Eufrasio
E-mail: rodrigo.mct @ gmail.com
Mobile: +55 11 8183-9484
Brazil - Osasco - SP

It is doable.
Assume for the moment that you are not paging the query results (that is, you are display ALL results at one time).
You would need to do the following when the button is clicked (the action calling a separate file for the processing):
Open a file on the server for writing ($ofile = fopen("data.txt","w");)
Repeat your query but instead of echo $row_Record set.... you would use fwrite($ofile, $row_Recordset...
In between fields you would need fwrite($ofile, "\t") to put in a tab or fwrite($ofile, ",") for a CSV,
At the end of every line you would need a <cr><lf>: fwrite ($ofile, "\r\n");
Close the file - fclose ($ofile);
Then use the header function to force the download of this file like this.
header('Content-disposition: attachment; filename=data.txt');
header('Content-type: text');
readfile('data.txt);
The header function has to be the first thing sent to the user. Any white space would cause it to fail. I haven't tried this part after a DB query, but it should work. If it doesn't you will have to invoke the header routine in a separate file.
Hope this helps.
Barry

Reading a flattened LabVIEW data string from a CSV file in C#?

We're recording a large amount of data into a CSV file to make it easily accessible from the C# end. I've a header with all the variable names then I write all the data (doubles) as flattened LabVIEW binary strings using flatten to string (little endian) to keep the size down. We are having issues converting these strings back to doubles on the C# end.
At the C# end we can read the LabVIEW repesentation of the DBL but have tried the BitConverter.ToDouble library function to convert it without luck. We use this method to convert the same flattened data coming over a TCP connection without any problems. I'm not writing the C# part of the program and have little experience with it, but if anyone could shed some light, it would be greatly appreciated.
Regards,
Andrew

It is all written as single values. It looks like we have found a method if we remove the header which contains the variable name as a string so the data is in a csv containing just the binary data strings. It looks like we'll go with this as the data will always be written in the same order so can be indexed by the C# program without the need for headers.
Thanks.

Import table - problems with date

Hi,
I want to import an table from text file with date column. The format in file is 12.01.2000 13:59:12.
When importing the preview of column value is 2000-12-01 13:59:12. I've then added the format YYYY-MM-DD HH24:MI:SS. After import all rows with date of year 1999 or below are failed.
Does anybody know why and how can I solve this problem?
Thanks
chrissy

Hi Dimitri,
yes all rows with date of year 2000 or higher are correct. It shows me the YYYY-DD-MM HH24:MI format if it recognizes the column as date. Somehow I get it worked that it is recognized as varchar, changed to date and set format to DD.MM.YYYY HH24:MI. So it was imported fine.
Thanks for your help,
chrissy

How to get sql query data and write into csv file?

I am writing to seek help, in how can I create bat. script which can execute the following logic:
connection to the database
run sql query
create CSV file
output query data, into CSV file
save the CSV file
osql
-S 84.18.111.111
-U adw
-P rem
-i "c:\query.sql"
send ""
sed -e 's/,\s\+/,/g' MCI_04Dec2014.csv > localNoSpaces.csv
-o "c:\MCI_04Dec2014.csv"
This what i have so far, and I am little struggling with the logic after creating CSV file. I am unable to get the above script to work, please advice further, where I may be going wrong.
Can you create if statement logic within window's script, to check for null parameters or data feeds?
Any hints would be most appreciated.

Thank you for your reply.
Apology for posting the code irrelevant to the forum, as I am still novice scripting user.
My goal is to create window's script which can compute the above logic and send the final output file (csv), to FTP folder.
Can this logic be implemented via bat. script, if so, is there a example or tutorial i could follow, in order to achieve my task.
Any help would be much appreciated.

How export one table along with data from one location to other location

Hi All,
I'm new in export/import practice.
Can anyone plz tell the steps along with commands to do the following:
1. I want to export a table with data from one location(computer) to other(computer) that are in same network.
2.Also from one user to another user.
I'm using oracle 10g.
regards
Sonia
Edited by: 983040 on Feb 19, 2013 11:35 PM

First of all read documentation
Oracle Export/Import : http://docs.oracle.com/cd/B19306_01/server.102/b14215/exp_imp.htm
Datapump Export/Import : http://docs.oracle.com/cd/B19306_01/server.102/b14215/dp_overview.htm
If you are using Datapump or Traditional Export/import you need to follow following steps
*1) Take User dump via EXPDP on Computer A .*
For EXP
exp username/password owner=Test file=D:\test.dmp log=D:\test.log
For EXPDP
expdp username/password schemas=TEST directory=TEST_DIR dumpfile=TEST.dmp logfile=TEST.log*2) Copy that to Computer B*
*3) Import dumpfile.*
For IMPDP Use remap_schema optionhttp://www.acehints.com/2012/05/data-pump-impdp-remapschema-parameter.html
For IMP use fromuser and touser option
one user to another user imp

Random Records are being skipped while uploading data in PSA from CSV File

Hi Experts,
I am facing issue in data uploading in PSA through CSV file, Random Records are being skipped while uploading data in PSA.
First Load
We have flat file (.txt in CSV format), which contains 380240 Records.
We are uploading the flat file data into PSA from Application Server. But it uploads Only 380235 records, 5 Records are being skipped.
Second Load
We have re-generated same file next day, which contains same No of Records (380240), but this time it uploads Only 380233 records, 7 Records are being skipped.
We found 1 skipped record (based on key columns combination, by cross verifying from source and PSA table) from First load. But same records (combination of key column) is available in second load. It means same records are not being skipped every time.
Earlier (5 months ago) , We have loaded 641190 Records from flat file in same PSA and all records (641190) were uploaded successfully.
There is no change is Source, PSA and flat file structure.
Thanks & Regards
Bijendra

Hi Bijendra,
Please check in the file if at the begining if it has got any excape sign then that record may be skipped so the records may be mssing
Please check the excape sign like ; if they are present at the beginign that recor entirely will be skipped.
Regards
vamsi

HELP - Export Files from Data Set into Excel (CSV File)

Hello,
I was hoping someone could help me with adobe forms since this is the first time we have used this function at my work. We created an enrollment form, posted it online and 700 employees completed the form (with the submit button it came back to my email). I received all the forms and added them to the dataset in adobe. Now that my forms are complete, I have tried several options to export them into a csv file and it is not working properly. When I use the export button it only brings over 125 of the 700. If I export by clicking on forms>manage form data>merge files into a spreadsheet it brings them all over but is moving my information on some of the rows. For example, some names will be in the name column, some are under social security, etc. I need the exact answer under the appropriate column becuase I can't tell which plan employees chose. I would really appreciate if someone could help. I have searched and searched for the reason this is doing this but I can't seem to locate an answer. Is there another step that I'm missing? Does anyone know why my rows are shifting? We tried to test this before we used it and it seemed to work fine but we only used 5 files. Please Please Please Help Me! This enrollment file is due by tomorrow!
Thank You,
Mary

I've moved your post to the Acrobat Windows user to user forums. You might also want to consider the Adobe moderated Acrobat forums here:
http://www.acrobatusers.com/forums/aucbb/index.php
Kind Regards,
Michelle

How to fill the table UD_SAPHR with data from SAP HR Connector

Hi, I`ve searched all the forrum but didn`t find how to map fields from connector form to the any field in OIM. I tryied with entity adapters but failed to reach any result. The problem is that I even cant put incoming data with connector to the table of the connector form (UD_SAPHR).
I will be grateful for any help. Below is the cut of the log. In few words - how to map, for example, field "Department=40000260" to the field in the USR table or UD_SAPHR table (or any other - to have this information in OIM).
OIM-9101Upd5, OS-Oracle Enterprise Linux4x64,SAP COnnector-SAP-ER90460, Database-Oracle 10gx64, Appication server-SOA(Oracle Application server)
INFO,01 Jul 2009 14:08:11,504,[XL_INTG.SAPHRMS],getDetailsForRecon(): *****************************
INFO,01 Jul 2009 14:08:11,505,[XL_INTG.SAPHRMS],getDetailsForRecon(): 00000017
INFO,01 Jul 2009 14:08:11,505,[XL_INTG.SAPHRMS],reconcileEmp(): SAPUserRecon for SAP : SAP HRMS IT Resource:EmployeeId=00000017
DEBUG,01 Jul 2009 14:08:11,505,[XL_INTG.SAPHRMS],SAPUserRecon:returnDate(): Employee Status***3
DEBUG,01 Jul 2009 14:08:11,505,[XL_INTG.SAPHRMS],EmpStatus***3
INFO,01 Jul 2009 14:08:11,505,[XL_INTG.SAPHRMS],getUserId(): Recon key is 00000017
INFO,01 Jul 2009 14:08:11,505,[XL_INTG.SAPHRMS],getUserId(): Hash Key is USR_UDF_PERSONNEL_NUMBER
INFO,01 Jul 2009 14:08:11,505,[XL_INTG.SAPHRMS],getUserId(): Get UserId of user with reconKey: 00000017
INFO,01 Jul 2009 14:08:11,506,[XL_INTG.SAPHRMS],getUserId():hmFilter: {USR_UDF_PERSONNEL_NUMBER=00000017}
INFO,01 Jul 2009 14:08:11,540,[XL_INTG.SAPHRMS],Result set rs contains : Thor.API.tcMetaDataSet@790a37
INFO,01 Jul 2009 14:08:11,540,[XL_INTG.SAPHRMS],getUserId(): Xellerate RS Count*** 0
DEBUG,01 Jul 2009 14:08:11,541,[XL_INTG.SAPHRMS],getUserId(): hmReturn ***{Create=0, UserLinked=0, userId=, UserKey=, XelUser=, Exist=0, User=}
INFO,01 Jul 2009 14:08:11,541,[XL_INTG.SAPHRMS],getUserId(): Recon key is 00000017
INFO,01 Jul 2009 14:08:11,541,[XL_INTG.SAPHRMS],getUserId(): Hash Key is USR_UDF_PERSONNEL_NUMBER
INFO,01 Jul 2009 14:08:11,541,[XL_INTG.SAPHRMS],getUserId(): Get UserId of user with reconKey: 00000017
INFO,01 Jul 2009 14:08:11,541,[XL_INTG.SAPHRMS],getUserId():hmFilter: {USR_UDF_PERSONNEL_NUMBER=00000017}
INFO,01 Jul 2009 14:08:11,570,[XL_INTG.SAPHRMS],Result set rs contains : Thor.API.tcMetaDataSet@282f10
INFO,01 Jul 2009 14:08:11,570,[XL_INTG.SAPHRMS],getUserId(): Xellerate RS Count*** 0
DEBUG,01 Jul 2009 14:08:11,570,[XL_INTG.SAPHRMS],getUserId(): hmReturn ***{Create=0, UserLinked=0, userId=, UserKey=, XelUser=, Exist=0, User=}
DEBUG,01 Jul 2009 14:08:11,570,[XL_INTG.SAPHRMS],sEmployeeUserID***
DEBUG,01 Jul 2009 14:08:11,570,[XL_INTG.SAPHRMS],UserKey***
DEBUG,01 Jul 2009 14:08:11,570,[XL_INTG.SAPHRMS],EmpStatus fetch from target System is 3
DEBUG,01 Jul 2009 14:08:11,570,[XL_INTG.SAPHRMS],EmpStatus Mentioned in Task scheduler is 3
INFO,01 Jul 2009 14:08:11,570,[XL_INTG.SAPHRMS],reconcileEmp():sUser get from Resource Object Table ***
INFO,01 Jul 2009 14:08:11,570,[XL_INTG.SAPHRMS],reconcileEmp():sXelUser get from Xellerate Table***
el Number=00000017, LinkedUserID=, Xellerate Type=End-User Administrator, First Name=п⌠п╣пҐпҐп╟пЄп╦п╧, Last Name=п■п╟п╡я▀пЄп╬п╡, Role=Consultant, Middle Name=FromHRMS=1}╣п╡п╦я┤, Userr
INFO,01 Jul 2009 14:08:11,571,[XL_INTG.SAPHRMS],Reconciliation of the user entered
DEBUG,01 Jul 2009 14:08:11,571,[XL_INTG.SAPHRMS],before create HR user reconciliation event***{PostalCode=, EndDate=9999/12/31 12:00:00 MSK, EmailAddress=, SSN=, State=, District=, TelephoneNumber=, Manager=, StartDate=1888/12/01 12:00:00 MSK, City=, EmplUserId=, UserLinked=0, UserLocked=0, FirstName=п⌠п╣пҐпҐп╟пЄп╦onnel Number=00000017, EmployeeId=00000017, Country=RU, Title=Mr, Department=40000260, MiddleName=п░пҐпЄя─п╣п╣п╡п╦я┤, LastName=п■п╟п╡я▀пЄп╬п╡}
INFO,01 Jul 2009 14:08:11,711,[XL_INTG.SAPHRMS],disableEnableXelUser(): Get UserKey of user with reconKey: 00000017 for SAP SAP HRMS IT Resource
DEBUG,01 Jul 2009 14:08:11,742,[XL_INTG.SAPHRMS],disableEnableXelUser(): Xellerate RS Count*** 0
INFO,01 Jul 2009 14:08:11,742,[XL_INTG.SAPHRMS],reconcileEmp(): User Reconciliation for Create & Modify User is completed.
INFO,01 Jul 2009 14:08:11,742,[XL_INTG.SAPHRMS],reconcileEmp() **** DONE RECONCILIATION

That is my problem. I get only those fields, that are in the xellerate user standard form. And I need also fields "Department" and "City".
This information connector gets from SAP (as it is seen in log) but doesn`t save it anywhere inside the OIM, though such fields exists in the UD_SAPHR table (and correspondend form) but this table remains empty after reconciliation.
DEBUG,26 Jun 2009 18:01:41,746,[XL_INTG.SAPHRMS],before create XL user reconciliation event***{Email=, Organization=Xellerate Users, User ID=00000072, Personnel Number=00000072, LinkedUserID=, Xellerate Type=End-User Administrator, First Name=п°пҐп╣, Last Name=п п╬п╪п©п╣пҐя│п╦я─я┐п╧п╨п╟, Role=Consultant, Middle NamomHRMS=1}
INFO,26 Jun 2009 18:01:41,746,[XL_INTG.SAPHRMS],Reconciliation of the user entered
DEBUG,26 Jun 2009 18:01:41,746,[XL_INTG.SAPHRMS],before create HR user reconciliation event***{PostalCode=, EndDate=9999/12/31 12:00:00 MSK, EmailAddress=, SSN=, State=, District=, TelephoneNumber=, Manager=, StartDate=1970/01/01 12:00:00 MSK, City=, EmplUserId=, UserLinked=0, UserLocked=0, FirstName=п°пҐп╣, Person Number=00000072, EmployeeId=00000072, Country=RU, Title=Mr, Department=40000240, MiddleName=п·я┌п©я┐я│п╨, LastName=п п╬п╪п©п╣пҐя│п╦я─я┐п╧п╨п╟}
Edited by: user6645106 on 03.07.2009 10:12

Compare table data with data in csv file

Similar Messages

Maybe you are looking for