Bulk insert and error logging

Hello,
I am trying to upload data from an excel file to database, using ADF Business components (11g) view objects. I have 2 questions related to this.
1- What is the best practice for inserting large amount of bulk data? Should i commit after every insert or after some batch size defined by me reached?
2- I want to log every unsuccessful upload attempt of a row, to database.. As i am rolling back the transaction after an error occurs, i need to make this logging process in another transaction. How can i open another transaction in AM Impl?
Thanks a lot for helps and ideas.

Deniz, you may want to consider ADF Desktop Integration (ADFdi) for integrating your ADF-based application with MS Excel.
More info: http://download.oracle.com/docs/cd/E12839_01/web.1111/e10139/toc.htm
ADFdi handles bulks uploads, validation failure reporting, and more.
Alex

Similar Messages

Issue with trigger, multi-table insert and error logging

I find that if I try to perform a multi-table insert with error logging on a table that has a trigger, then some constraint violations result in an exception being raised as well as logged:
<pre>
SQL> select * from v$version;
BANNER
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
PL/SQL Release 11.2.0.1.0 - Production
CORE 11.2.0.1.0 Production
TNS for 32-bit Windows: Version 11.2.0.1.0 - Production
NLSRTL Version 11.2.0.1.0 - Production
SQL> create table t1 (id integer primary key);
Table created.
SQL> create table t2 (id integer primary key, t1_id integer,
2 constraint t2_t1_fk foreign key (t1_id) references t1);
Table created.
SQL> exec dbms_errlog.create_error_log ('T2');
PL/SQL procedure successfully completed.
SQL> insert all
2 into t2 (id, t1_id)
3 values (x, y)
4 log errors into err$_t2 reject limit unlimited
5 select 1 x, 2 y from dual;
0 rows created.
SQL> create or replace trigger t2_trg
2 before insert or update on t2
3 for each row
4 begin
5 null;
6 end;
7 /
Trigger created.
SQL> insert all
2 into t2 (id, t1_id)
3 values (x, y)
4 log errors into err$_t2 reject limit unlimited
5 select 1 x, 2 y from dual;
insert all
ERROR at line 1:
ORA-02291: integrity constraint (EOR.T2_T1_FK) violated - parent key not found
</code>
This doesn't appear to be a documented restriction. Does anyone know if it is a bug?

Tony Andrews wrote:
This doesn't appear to be a documented restriction. Does anyone know if it is a bug?Check The Execution Model for Triggers and Integrity Constraint Checking.
SY.

Where is HTTP access and error log when using Embedded PL/SQL Gateway?

Hello,
I am using Embedded PL/SQL Gateway to run stored procedures.
I can't find access log and error log (something like Oracle HTTP Server has).
In listener.log are only simple data and there are no usernames and procedure names...
Thanks.
Marian

You might be better off asking this on the Apex forum: Oracle Application Express (APEX)

ODBC, bulk inserts and dynamic SQL

I am writing an application running on Windows NT 4 and using the oracle ODBC driver (8.01.05.00, that inserts many rows at a time (10000+) into an oracle 8i database.
At present, I am using a stored procedure to insert each row into the database. The stored procedure uses dynamic SQL because I can only determine the table and field names at run time.
Due to the large number of records, it tends to take a while to perform all the inserts. I have tried a number of solutions such as using batches of SQL statements (e.g. "INSERT...;INSERT...;INSERT..."), but the oracle ODBC driver only seems act on the first statement in the batch.
I have also considered using the FOR ALL statement and SQL*Loader utility.
My problem with FOR ALL is that I'm not sure it works on dynamic SQL statements and even if it did, how do I pass an array of statements to the stored procedure.
I ruled out SQL* Loader because I could not find a way to invoke it it from an ODBC statement. Secondly, it requires the spawining of a new process.
What I am really after is something similar the the SQL Server (forgive me!) BULK INSERT statement where you can simply create an input file with all the records you want to insert, and pass it along in an ODBC statement such as "BULK INSERT <filename>".
Any ideas??
null

Hi,
I faced this same situation years ago (Oracle 7.2!) and had the following alternatives.
1) Use a 3rd party tool such as Sagent or CA Info pump (very pricey $$$)
2) Use VisualC++ and OCI to hook into the array insert routines (there are examples of these in the Oracle Home).
3) Use SQL*Loader (the best performance, but no real control of what's happening).
I ended up using (2) and used the Rouge Wave dbtools.h++ library to speed up the development.
These days, I would also suggest you take a look at Perl on NT (www.activestate.com) and the DBlib modules at www.perl.org. I believe they will also do bulk loading.
Your problem is that your program is using Oracle ODBC, when you should be using Oracle OCI for best performance.
null

Bulk inserts and dynamic SQL

I am writing an application running on Windows NT 4 and using the oracle ODBC driver (8.01.05.00, that inserts many rows at a time (10000+) into an oracle 8i database.
At present, I am using a stored procedure to insert each row into the database. The stored procedure uses dynamic SQL because I can only determine the table and field names at run time.
Due to the large number of records, it tends to take a while to perform all the inserts. I have tried a number of solutions such as using batches of SQL statements (e.g. "INSERT...;INSERT...;INSERT..."), but the oracle ODBC driver only seems act on the first statement in the batch.
I have also considered using the FOR ALL statement and SQL*Loader utility.
My problem with FOR ALL is that I'm not sure it works on dynamic SQL statements and even if it did, how do I pass an array of statements to the stored procedure.
I ruled out SQL* Loader because I could not find a way to invoke it it from an ODBC statement. Secondly, it requires the spawining of a new process.
What I am really after is something similar the the SQL Server (forgive me!) BULK INSERT statement where you can simply create an input file with all the records you want to insert, and pass it along in an ODBC statement such as "BULK INSERT <filename>".
Any ideas??
null

Hi,
I faced this same situation years ago (Oracle 7.2!) and had the following alternatives.
1) Use a 3rd party tool such as Sagent or CA Info pump (very pricey $$$)
2) Use VisualC++ and OCI to hook into the array insert routines (there are examples of these in the Oracle Home).
3) Use SQL*Loader (the best performance, but no real control of what's happening).
I ended up using (2) and used the Rouge Wave dbtools.h++ library to speed up the development.
These days, I would also suggest you take a look at Perl on NT (www.activestate.com) and the DBlib modules at www.perl.org. I believe they will also do bulk loading.
Your problem is that your program is using Oracle ODBC, when you should be using Oracle OCI for best performance.
null

How to get count of records inserted and errored out in an email

Hi
I have following question
I want to send report statics of scenario i.e Number of rows inserted during the scenario and
Number of rows in error in the scenario in an email.Here in my scenario I am trying to insert data into Essbase database and when i try to use getNbInserts() and getNbErrors functions of ODI in an email body,its giving zero value even though there 140 records inserted and 10 errored out.Can any one let me know how we get this number of records inserted and number of records errored in email.
Thanks in advance
Regards
Baji

Hi
I have following question
I want to send report statics of scenario i.e Number of rows inserted during the scenario and
Number of rows in error in the scenario in an email.Here in my scenario I am trying to insert data into Essbase database and when i try to use getNbInserts() and getNbErrors functions of ODI in an email body,its giving zero value even though there 140 records inserted and 10 errored out.Can any one let me know how we get this number of records inserted and number of records errored in email.
Thanks in advance
Regards
Baji

Multi table insert with error logging

Hello,
Can anyone please post an example of a multitable insert with an error logging clause?
Thank you,

Please assume that I check the documentation before asking a question in the forums. Well, apparently you had not.
From docs in mention:
multi_table_insert:
{ ALL insert_into_clause
      [ values_clause ] [error_logging_clause]
      [ insert_into_clause
        [ values_clause ] [error_logging_clause]
| conditional_insert_clause
subqueryRegards
Peter

Access and Error Log Rotation

When turning on Archive for the logs. cron style. The log gets archived with the date etc, but the new files access and error do not get created. I have ns-cron on and the rotation occurs, but the files name only changes and it never create the new files and only keeps changing the file names with the new name.

You may be running into Problem 4684892 mentioned in the release notes:
http://docs.sun.com/source/817-5170-10/rn60sp7.html
The Administration Server and the cron daemon must be run as root for cron-based log rotation to function properly. You may have to modify your cron.conf user to run as root.
Thanks,
Manish

Message and Error Logging

Version : TimesTen Release 11.2.1.4.0 (32 bit NT)
In C:\TimesTen\tt1121_32\srv\info we have 2 TimesTen log files : tterrors.log and ttmesg.log
Based on C:\TimesTen\tt1121_32\srv\info\ttendaemon.options I expected these files to grow to a maximum of 1Mb each, then retain 10 copies of each. However I find they grow to unlimited size, e.g. ttmesg.log was 1.2 Gb last week before I manually renamed it. Am I missing a setting somewhere? This functionality was working correctly last month, and as far as I am aware I have not intentionally changed anything.
Contents of
C:\TimesTen\tt1121_32\srv\info\ttendaemon.options
# By default, turn verbose logging on
-verbose
# Commented values are default values
#-supportlog C:\TimesTen\TT1121~1\srv\info\ttmesg.log
#-maxsupportlogfiles 10
#-maxsupportlogsize 0x100000
#-userlog C:\TimesTen\TT1121~1\srv\info\tterrors.log
#-maxuserlogfiles 10
#-maxuserlogsize 0x100000
-tns_admin C:\oracle\product\111~1.0\db_1\NETWORK\ADMIN
-server 53385
Also included is directory listing to show it used to work.
Directory of C:\TimesTen\tt1121_32\srv\info
12/11/2007 18:32 470 snmp.ini
19/11/2009 22:50 3,244 cluster.oracle.ini
06/01/2010 15:17 378 ttendaemon.options
06/01/2010 15:18 <DIR> crs_scripts
08/01/2010 16:11 3,996 DBI4b475916.1~
12/01/2010 15:47 1,048,430 ttmesg.log.9
15/01/2010 01:55 1,048,605 ttmesg.log.8
17/01/2010 09:28 1,048,882 ttmesg.log.7
26/01/2010 01:03 1,048,610 ttmesg.log.6
26/01/2010 01:03 1,048,670 ttmesg.log.5
29/01/2010 21:37 1,048,500 ttmesg.log.3
29/01/2010 21:57 1,048,490 ttmesg.log.2
29/01/2010 22:17 1,048,483 ttmesg.log.1
29/01/2010 22:37 1,048,146 ttmesg.log.0
29/01/2010 23:47 1,048,977 tterrors.log.9
30/01/2010 03:08 1,052,803 tterrors.log.8
30/01/2010 06:39 1,048,725 tterrors.log.7
30/01/2010 09:50 1,048,851 tterrors.log.6
30/01/2010 13:09 1,048,633 ttmesg.log.4
01/02/2010 19:26 1,052,682 tterrors.log.5
22/02/2010 08:22 1,051,629 tterrors.log.3
22/02/2010 11:43 1,048,943 tterrors.log.2
22/02/2010 15:04 1,049,926 tterrors.log.1
22/02/2010 18:25 1,048,968 tterrors.log.0
23/02/2010 01:04 1,048,934 tterrors.log.4
01/03/2010 18:24 135,615,162 ttmesg.log.20100301
01/03/2010 18:24 2,135,473 tterrors.log.20100301
01/03/2010 18:26 6 timestend.pid
07/03/2010 19:42 <DIR> ..
07/03/2010 19:42 <DIR> .
08/03/2010 12:32 138,266,337 tterrors.log.20100307
08/03/2010 12:32 1,306,793,217 ttmesg.log.20100307
16/03/2010 06:05 31,213 tterrors.log
16/03/2010 06:45 3,996 DBI4b4b6484.0
16/03/2010 08:23 1,933,865 ttmesg.log
32 File(s) 1,605,773,244 bytes
3 Dir(s) 34,267,926,528 bytes free
C:\TimesTen\tt1121_32\srv\info>

This sounds like this bug:
Bug 9442841: AFTER MAXSUPPORTLOGFILES IS REACHED TTMESG.LOG CONTINUES TO GROW
This is published so you should be able to see the details of a potential workaround for it.
I believe this is unique to TT on Windows, and from the internal bug also logged for it, it states it has been fixed in 11.2.1.5 which has just been released (available on Windows 32-bit and 64-bit) from here:
http://www.oracle.com/technology/software/products/timesten/index.html

ESB DB adapter and error logging

Hi !
I need a DB adapter in ESB to read records from a table and do a logical delete (update a status column) .
I can get it to work om my laptop, but not in our dev machines.
I noticed that if I change the the 'mcf-properties' in the wsdl-file for the READ function like this
mcf.DriverClassName="oracle.jdbc.OracleDriver" mcf.PlatformClassName="oracle.toplink.platform.database.oracle.Oracle10Platform"
mcf.ConnectionString="jdbc:oracle:thin:@localhost:1521:ORCLAAAAAA"
mcf.UserName="soatest" mcf.Password="A932C53E63FFDE3D4A8267B2FCE4A0044C1B70BFD42DD194"
so that is wrong I never get any error in any log-file I found.
This makes is very tiresome to debug.
Is there a log where I can find an error message ?
If not , what should I do ?

Hello Everyone,
I am stuck with the ESB's "logging" behavior. Due to some problems we had increased the Logging level to FINEST for some and as a result there were too many messages - mainly "Traces" -generated in /../oc4j/log.xml Now the problem is that i am seeing that ESB maintains only 10 to 12 log files.
My query is does it archives the older log files somewhere- if yes then which location?
-Or does it simply overwrites the files (that doesnt seems likely)?
For example: While searching the OC4J Diagnostic Logs from Enterprise Manager Console it is showing that there are 10 selected log files.
But where are the older log files. We need those badly as there are some issues we are trying to fix hinged on the Error Messages.
Thanks
--debashis

SSIS BULK INSERT unsing UNC inside of ForEach Loop Container Failed could not be opened. Operating system error code 5(Access is denied.)

Hi,
I am trying to figure out how to fix my problem
Error: Could not be opened. Operating system error code 5(Access is denied.)
Process Description:
Target Database Server Reside on different Server in the Network
SSIS Package runs from a Remote Server
SSIS Package use a ForEachLoop Container to loop into a directory to do Bulk Insert
SSIS Package use variables to specified the share location of the files using UNC like this
\\server\files
Database Service accounts under the Database is runing it has full permission on the share drive were the files reside.
In the Execution Results tab shows the prepare SQL statement for the BULK insert and I can run the same exact the bulk insert in SSMS without errors, from the Database Server and from the server were SSIS package is executed.
I am on a dead end and I don’t want to re-write SSIS to use Data Flow Task because is not flexible to update when metadata of the table changed.
Below post it has almost the same situation:
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/8de13e74-709a-43a5-8be2-034b764ca44f/problem-with-bulk-insert-task-in-foreach-loop?forum=sqlintegrationservices

Insteresting how I fixed the issue, Adding the Application Name into the SQL OLAP Connection String Fixed the issue. I am not sure why SQL Server wasn't able to open the file remotely without this.

Auditing and Custom error logging

Guys,
Can one of you tell me how can we do auditing in ODI i.e like say if load files how many records the file have and how many we have loaded for each file and how many are
bad records etc and how many records we have inserted/updated in each table etc.
Basically some sort of report we need to send in the end as audit report after odi batch run every day
Can we do this in ODI?
Is it possible to do sort of custome error logging like what we do in pl/sql, like inserting into error log table when ever oracle error comes or any runtime error which we need to insert into table etc.
Can we do this kind of error handling ODI?
Cheers
Sri
Edited by: aranisrinivas on 26-Nov-2011 10:13

Just use below details for your required information
'<%=odiRef.getPrevStepLog("STEP_NAME")%>'
'<%=odiRef.getPrevStepLog("SESS_NO")%>'
'<%=odiRef.getPrevStepLog("MESSAGE")%>'
'<%=odiRef.getPrevStepLog("ERROR_COUNT")%>'
You can get more details from below tables.
1) snp_sess_txt_log -( It holds the scripts used for the task and session details)
2) snp_sess_task_log-(It holds the time details error msg and all)
3) snp_sess_task -( it holds the name of the task and technology , context details
Thanks

Bulk Insert Task Cannot bulk load because the file could not be opened.operating system error error code 3(The system cannot find the path specified.)

Following error i am getting after i chnaged the Path in Config File from
\\vs01\d$\\Deployment\Files\temp.txt
to
C:\Deployment\Files\temp.txt
[Bulk Insert Task] Error: An error occurred with the following error message: "Cannot bulk load because the file "C:\Deployment\Files\temp.txt" could not be opened. Operating system error code 3(The system cannot find the path specified.).".

I think i know whats going on. The Bulk Insert task runs by executing sql command (bulk insert) internally from the target sql server to load the file. This means that the SQL Server Agent of the target sql server should have permissions on the file you trying to load. This also means that you need to use UNC path instead to specify the file path (if the target server in on different machine)
Also from BOL (see section Usage Considerations - last bullet point)
http://msdn.microsoft.com/en-us/library/ms141239.aspx
* Only members of the sysadmin fixed server role can run a package that contains a Bulk Insert task.
Make sure you take care of this as well.
HTH
~Mukti
Mukti

Cannot fetch a row from OLE DB provider "BULK" with bulk insert task

Hi, folks:
I created a simple SSIS package. On the Control Flow, I created a Bulk INsert Task with Destination connection to a the local SQL server, a csv file from a local folder, specify comma delimiter. Then I excute the task and I got this long error message.
[Bulk Insert Task] Error: An error occurred with the following error message: "Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.The bulk load failed. The column is too long in the data file for row 1, column 1. Verify that the field terminator and row terminator are specified correctly.".

I got the same error with some additional error details (below). All I had to do to fix the problem was set the Timeout property for the SQL Server Destination = 0
I was using the following components:
SQL Server 2008
SQL Server Integration Services 10.0
Data Flow Task
OLE DB Source – connecting to Oracle 11i
SQL Server Destination – connecting to the local SQL Server 2008 instance
Full Error Message:
Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80040E14.
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80040E14 Description: "Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".".
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80040E14 Description: "The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.".
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80040E14 Description: "The Bulk Insert operation of SQL Server Destination has timed out. Please consider increasing the value of Timeout property on the SQL Server Destination in the dataflow.".
For SQL Server 2005 there is a hot fix available from Microsoft at http://support.microsoft.com/default.aspx/kb/937545

Back ground batch input processing error log handling

Dear Friends,
Currently im am scheduling job for batch processing of goods receipt through bapi, now after the succesfull run of this program im appending all the success and error logs in one internal table and sending this table to spool for future reference.
Is there any other way to handle the error logs??
bcoz in the method im using every time user has to run transaction sp01 to access that spool request to check the error logs.
Kindly suggest ....
Regards,
Sunny V

Best way will be creating Application Logs of the reports log.This will be forever in the system unless you delete it.
Few one time settings are required. After that you can use tcodes
SLG1 (Analyze application log) and
SLG2 (Application Log: Delete expired logs).
For more info take the help from below link:
[Create Application Log|http://help.sap.com/saphelp_nw04/Helpdata/EN/2a/fa0216493111d182b70000e829fbfe/frameset.htm]
Let me know if you find any difficulty in doing this.

Bulk insert and error logging

Similar Messages

Maybe you are looking for