OutOfMemoryError in batch insert
Hi All,
I load 180M records in one transaction using batchUpdate.
When I load first 4M I get the exception:
18:50:47,127 INFO ...:172 - Insert bulk records. Lines processed: 449334. Staging size: 4232105
[Full GC 1631120K->1631120K(1864192K), 3.8070750 secs]
[Full GC 1631120K->1625943K(1864192K), 5.9659140 secs]
[Full GC 1630588K->1630588K(1864192K), 3.9573270 secs]
[Full GC *1630588K*->1629580K(1864192K), 3.9293920 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.lang.reflect.Array.newArray(Native Method)
at java.lang.reflect.Array.newInstance(Array.java:52)
at oracle.jdbc.driver.BufferCache.get(BufferCache.java:226)
at oracle.jdbc.driver.PhysicalConnection.getByteBuffer(PhysicalConnection.java:7587)
at oracle.jdbc.driver.OraclePreparedStatement.setupBindBuffers(OraclePreparedStatement.java:2915)
at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10199)
at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:216)
at org.springframework.jdbc.core.JdbcTemplate$4.doInPreparedStatement(JdbcTemplate.java:898)
at org.springframework.jdbc.core.JdbcTemplate$4.doInPreparedStatement(JdbcTemplate.java:1)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:586)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:614)
at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:883)
"*Staging size: 4232105*" here is the amount of already executed inserts.
As you can see above, GC does not help much.
Is that possible, that statements remain bound in driver after batch is executed? Transaction is not autocommitted.
Thanks!
AShananin wrote:
I load 180M records in one transaction using batchUpdate.That won't work. Doesn't have anything to do with java.
When I load first 4M I get the exception: You are trying to batch 4 million records at one time? By calling executeBatch() after 4m records?
That isn't going to work.
My first choice. Don't use java. Your database will have an import tool. Use it. As a variation you can load the data into work tables and process it further there.
Second choice. Use a batch size of about 1000, or maybe 10,000 if you feel lucky.
Similar Messages
-
How can i get the failed row numbers in batch insertion?
Afeter execution of batch insert operation into a table, i use OCIAttrGet() to get the error number of insertion, as expected, it returns the correct number. however, when i use OCIParamGet() to get an error handle and then to get the exact failed row numbers , OCIParamGet() returns -2 , means "INVALID_HANDLE". can anyboy tell me why ? how can i get the exact failed rows?
the snapshot of my code :
int errNum = 0;
int nRet = 0;
nRet = OCIAttrGet((dvoid *)m_pIIOdbc->m_pStmtAll,OCI_HTYPE_STMT,&errNum,
0,OCI_ATTR_NUM_DML_ERRORS, m_pIIOdbc->m_pErrHandle);
if (errNum)
OCIError* pErrHndl;
int* pRow_off = new int[errNum];
for (int i = 0; i < errNum; i++)
nRet = OCIParamGet(m_pIIOdbc->m_pErrHandle,
OCI_HTYPE_ERROR, m_pIIOdbc->m_pErrHandle, &pErrHndl, i + 1);
nRet = OCIAttrGet (pErrHndl, OCI_HTYPE_ERROR, &pRow_off, 0,
OCI_ATTR_DML_ROW_OFFSET, m_pIIOdbc->m_pErrHandle);
delete []pRow_off;now the code is :
OCIEnv *m_pEnvHandle;
OCIError *m_pErrHandle;
OCIError *m_pErrHandle2;
OCIServer *m_pSrvHandle;
OCISvcCtx *m_pSvcHandle;
OCIStmt *m_pStmtExec;
OCIStmt *m_pStmtAll;
OCIError** m_pErrorHndl;
int* m_pRowOff; //the array that will contain the failed row numbers
int m_nArrayLength; //length of array in once batch operation
int m_nErrNum; //error number
int m_nSucCount; //successful record number
//initialization
//m_nArrayLength is a parameter passed in, so i can not tell the exact length of the array, dynamic space allocation is needed
m_pErrorHndl = new OCIError*[m_nArrayLength];
m_pRowOff = new int[m_nArrayLength];
//execution
OCIStmtExecute(
m_pSvcHandle,
m_pStmtAll,
m_pErrHandle,
(ub4)m_nArrayLength,
(ub4)0, (OCISnapshot *)NULL, (OCISnapshot *)NULL,
OCI_BATCH_ERRORS
//get successful records number
m_nSucCount = 0;
OCIAttrGet((dvoid *)m_pStmtAll,OCI_HTYPE_STMT,&m_nSucCount,
0,OCI_ATTR_ROW_COUNT, m_pErrHandle);
//get error numbers
m_nErrNum = 0;
OCIAttrGet((dvoid *)m_pStmtAll,OCI_HTYPE_STMT,&m_nErrNum,
0,OCI_ATTR_NUM_DML_ERRORS, m_pErrHandle2);
//get failed row numbers into the array
if (m_nErrNum)
//get failed row numbers into the array
for (int i = 0; i < m_nErrNum; i++)
//initiallize error handles OCIHandleAlloc((dvoid*)m_pEnvHandle, (dvoid**)&m_pErrorHndl,
OCI_HTYPE_ERROR, (size_t)0, (dvoid**)0);
OCIParamGet(m_pErrHandle, OCI_HTYPE_ERROR, m_pErrHandle2, (void**)&m_pErrorHndl[i], i);
OCIAttrGet (m_pErrorHndl[i], OCI_HTYPE_ERROR, &m_pRowOff[i], 0,
OCI_ATTR_DML_ROW_OFFSET, m_pErrHandle2);
now, if batch insert is executed, eg, there are 10 records to be inserted, thus m_nArrayLength is set to be 10. There are 5 records of the 10 have exist in database, so some of insert operations will fail. After execution, m_nErrNum is 5, and i can get the failed row numbers.Everything seems go well.
However, if batch updata is excuted, eg, 10 records to be updated, m_nArrayLength is set to be 10, and also there are 5 records of the 10 have exist in database. After execution, the OCIStmtExecute() returns 1, m_nErrNum got by
OCIAttrGet((dvoid *)m_pStmtAll,OCI_HTYPE_STMT,&m_nErrNum,
0,OCI_ATTR_NUM_DML_ERRORS, m_pErrHandle2)
is 0, and m_nSucNum is 5. Why?? how can I get the rows that fail? i have to know which rows do not exist in database?? -
Batch insert running numbered PDFs as background into corresponding numbered PDFs
My objective: What I want to do is actually simple. I want to replace the pages in a PDF ("the original pdf") with pages from another PDF ("the new pdf") but retain the crop margins of the original pages. Both the new and original PDF has the same number of pages (about 600 pages). The crop margins differ from page to page.
I can't do so directly as replacing a page replaces the crop margins of the original page with the crop margins of the new page. Neither is there an option to export and import crop margins.
The workaround I am using is do insert the new page into the original page as a background (with the "scale relative to target page" option unchecked), and deleting the page (it has not been OCRed, so it is just an image) object, revealing the underlying inserted page. This works, but only for 1 page, because I can only select, in the insert background dialog box, one page out of the new PDF to insert as background into the original PDF.
What I want to do instead is to insert as background corresponding pages of the new PDF into the pages of the original PDF, i.e. p 1 to p1, p2 to p2.
To try to get around this, I extracted all the pages in the new and original PDF as separate files. My idea is to create an action that will batch insert, the single paged PDFs in one folder as background into the corresponding single paged PDFs in another folder, i.e. insert 1.pdf in Folder A as background for 1.pdf in Folder B, 2.pdf in Folder A as background for 2.pdf in Folder B and on.
This is where I am stuck. How can I define an action to do that? Or can this be done by script? I have no experience with scripts.
Help!
Many thanks,
JayChange the Save command to "Save to local folder" (by clicking the arrow icon next to it), and then select where you want to save the edited files.
-
Batch insert from object variable
Any way to do this excluding the foreach loop container? I have millions of rows and if it goes through them 1 by 1 it will take hours maybe days. How can you do a batch insert from an object variable without putting the sql task inside
a foreach loop container?
thanks,
Mike
MikeI know how to use the dataflow task but it will not work with what I am trying to do as far as I know. The script I am running is against a netezza box that creates temp tables and then returns a data set that I store in an object variable.
I then want to take the data from that object variable and batch insert it into a table. The only way I can see this being done is by looping through the record set in a foreach loop while mapping the fields to variables and inserting records one
at a time like I am doing in the screenshot above. This takes way to long to do. I am trying to figure out a way to do a batch insert using the data in this object variable. Is this possible? I am using SSIS Version
10.50.1600.1
thanks,
Mike
Hi Mike,
Provide more details why you cannot use the data flow task for the processing.
Let me repeat . Loading millions of records in object variable will not work.
SSIS Tasks Components Scripts Services | http://www.cozyroc.com/ -
Hello,
While migrating from Kodo 3.4 to Kodo 4.1, I noticed significant drop in
insert performance. I traced it down to some strange batching behavior.
While most of the PCs were committed with batched inserts one particular
class refused to batch and would insert row by row resulting in 10 times
performance drop.
There is nothing special about the class. Its hierarchy is mapped on base
its table except for one of the lowest members which is mapped vertically.
Thank you very much
Alex Roytman
Peace Technology, Inc.
301-206-9696x103See my post "My experience migrating Kodo 3.4 to 4.1"
"Stefan Hansel" <[email protected]> wrote in message
news:[email protected]..
Abe White wrote:
This case is being handled elsewhere, but for the benefit of others:
Kodo 4 works around a date/timestamp batching bug in some Oracle drivers
by not batching rows with date/timestamp columns.-------------
Abe could you get a bit deeper into it ?
It sounds like kodo 3 was batching that rows and had no problem with it ?
We only ship with the suggested oracle driver 10.2.0.1.0.
50% of our customers use oracle, so it would be a pity if performance
drops.
Alex ... it sounds like you successfully migrated to kodo4.
Could you give a short summery on how 'difficult' it was in the end ?
From the docs it sounds very easy (except the sentence 'Many
confniguration
properties and defaults have changed.') but from the newsgroups it sounds
as if
you'd struggle a bit.
I'm about to schedule our migration for the first quarter of 2007 and any
hint
helping use to estimate the time will help us.
If you compare with migration from kodo 2 to 3 ... is it easier or worse ? -
Transaction with batch inserts
I have a process running where I'm reading a large flat file and doing batch inserts everytime I have 500 rows. I need to implement some safety mechanism where if any part of the process fails, the data isn't corrupted. Typically I would use transactions, but since I'm committing every 500 rows, it seems like it would be tough to apply a transaction to the entire process, i.e. inserting over a million rows in 500 batch sets.
Basically, everytime I run this process I'm replacing all of the data in the entire table. I know this isn't ideal, but that's the requirement.
Any recommendations?Well you can put 500 items in one transaction. If you get an error while entering a perticuler block you can roll back it and devide the block in to smaller blocks and insert them. Doing this repeatedly will isolate the records that are giving trouble.
Also if there is only a small number of posible errors you can validate the data before inserting to avoid any errors. -
I am having a performance issue with network traffic. Can anyone tell me how to if possible to make batch inserts to a Oracle Database.
ThanksAbsolutely, It's called batching.
There is a code sample in
ORACLE_HOME/jdbc/demo/samples/oci8/basic-samples/SetExecuteBatch.java
The feature is not specific to the OCI driver, it works with the KPRB driver within JServer.
Matthieu -
Are batch inserts supported in cx_oracle?
Oracle's JDBC package provides for batch inserting to increase performance. I've used it in writing an ETL tool in Java. I would like to do the same in Python, but I can't find anything that mentions the same batch insert capability in cx_oracle. Any ideas?
Have you tried http://www.oracle.com/technology/pub/articles/prez-python-queries.html?
Look for "Many at Once" section.
Przemek -
Batch insert filename in textpath script crashes Illustrator
Hi everyone!
First off: I am not a programmer. Just playing around with code and trying to get it to work for a specific task:
Here is a script made for the purpose of inserting a text with the file name in over 600 pdf files. This is suppose to work on all files in a selected folder.
The problem: Illustrator crashes.
A first test code worked but after a few edited files Illustrator crashed, so I tried to introduce a delay after each save in order to slow down the batch process.
$.setTimeout(function () {sourceDoc.close(SaveOptions.SAVECHANGES)}, 1000);
Unfortunately I did not save the working (almost working ) first test code.
No idea what to do next. The code works if I delete this line: sourceDoc.close(SaveOptions.SAVECHANGES);
Here is the complete script:
var destFolder, sourceFolder, files, fileType, sourceDoc, layers, writeText, finLabel;
// Select the source folder.
sourceFolder = Folder.selectDialog( 'Select the folder with Illustrator files you want to convert to PNG', '~' );
// If a valid folder is selected
if ( sourceFolder != null )
files = new Array();
fileType = prompt( 'Select type of Illustrator files to you want to process. Eg: *.ai', '*.pdf' );
// Get all files matching the pattern
files = sourceFolder.getFiles( fileType );
if ( files.length > 0 )
for ( i = 0; i < files.length; i++ )
sourceDoc = app.open(files[i]); // returns the document object
layers = unlock();
writeText = getFilename();
finLabel = remText();
sourceDoc.close(SaveOptions.SAVECHANGES); //if save command line is deleted the code works WTF???
$.setTimeout(function () {sourceDoc.close(SaveOptions.SAVECHANGES)}, 1000); // still crashes using delay ...
//alert( 'Files are saved as PNG in ' + destFolder );
else
alert( 'No matching files found' );
function unlock()
//get the total number of layers in the active document
doc = app.activeDocument;
var totalLayers = doc.layers.length;
//looping on layers to create one artboard per layer
for ( var i = 0 ; i < totalLayers ; i++){
var currentLayer = doc.layers[i];
//We don't want to deal with hidden layers
if(currentLayer.visible == false) continue;
//Unlock the layer if needed
currentLayer.locked = false;
function getFilename()
// Write text
var pointTextRef = app.activeDocument.textFrames.add();
pointTextRef.contents = app.activeDocument.name + "\n" + "YBS";
pointTextRef.top = 0;
pointTextRef.left = 0;
app.activeDocument.textFrames[0].textRange.characterAttributes.textFont=app.textFonts[31];
function remText()
// This works for search and replace :))))))
var active_doc = app.activeDocument;
var search_string = /_Template.pdf/gi; // g for global search, remove i to make a case sensitive search
var replace_string = '';
var text_frames = active_doc.textFrames;
if (text_frames.length > 0)
for (var i = 0 ; i < text_frames.length; i++)
var this_text_frame = text_frames[i];
var new_string = this_text_frame.contents.replace(search_string, replace_string);
if (new_string != this_text_frame.contents)
this_text_frame.contents = new_string;
Any ideas about what makes Illustrator crash?
Note: The application crashes after opening the first file.
Thanks for helping out!Thanks a lot Carlos!
After deleting line 23:
$.setTimeout(function () {sourceDoc.close(SaveOptions.SAVECHANGES)}, 1000);
the code worked on 499 files in one batch
Also Turns out Illustrator kept crashing because of a corrupted pdf file. -
Error Handling for batch inserts
Hi,
Firstly, apologies if this has been asked before but I haven't been able to find the answer.
I'm inserting a batch of rows, each of which could fail for different reasons (e.g. constraint, space, etc.). However, when I receive multiple failures and retrieve the error message it is always:
ORA-24381: error(s) in array DML
If I don't use the OCI_BATCH_ERRORS mode then it obviously only fails once and the correct error message is received for the first failure.
Any suggestions as to how to get the real error would be appreciated.
The code I'm using to lookup the errors is:
OCIAttrGet (m_stmtHandle, OCI_HTYPE_STMT,
&numErrors, 0, OCI_ATTR_NUM_DML_ERRORS,
m_errorHandle);
if (numErrors)
OCIError *errhndl = NULL;
ub4 row_off[100];
for (int i = 0; i < numErrors; i++)
OCIParamGet(m_errorHandle, OCI_HTYPE_ERROR,
m_errorHandle,
reinterpret_cast <void **> (&errhndl), i);
OCIAttrGet (errhndl, OCI_HTYPE_ERROR, &row_off, 0,
OCI_ATTR_DML_ROW_OFFSET, m_errorHandle);
OCIErrorGet(errhndl, 1, NULL, &errorCode, errorText,
maxErrorLen, OCI_HTYPE_ERROR);
Thanks,
StevenHi,
Please use the OCI forum for this question. This is for the C++ Call Interface. -
Hi Everyone
First of all, Thank you all for such a nice contribution on the forum. I have few questions I have with SAP and JDBC adaptor.
I was wondering if we could do batch update/insert in XI.
let us say I received 10 different records from IDOC. I would like to insert all the ten records into a destination oracle table. I know the way to do would be to do a insert one at a time. This would involve sending INSERT message every single time.
I was wondering if there is a way to do a batch update. Is it possible to send all the information at one single shot and have it updated.
any suggestions would be highly appreciated.
Regards
PuskarThe way I did this was to use the ability to have 1..n statements in a JDBC Adapter message. So you define the Insert/Update structure with the required access elements, then you can connect the recurring node on the source to the statment node on the target. You may need an intermediate mapping to pull the data cleanly out of the IDOC, depending on what you are retrieving. Make sure you set your statement line in the JDBC message to 1..unbounded occurances. It works great for me... You can even mix and match, having INSERT, UPDATE, EXECUTE, whatever different statements all in one JDBC Adapter call. If doing with stored procedures you need to be careful about how you do your transactions in the stored proc, because of the transaction handling used by JDBC adapter.
I did the multi insert model using a 1-N mapping then looping using a block, it was much messier and results in a lot of messages showing up in the RWB and SXMB_MONI. The all in one approach is much cleaner and executes much faster. -
CSOM and Batch Insert Performance
Can we Batch requests while inserting records into SharePoint lists?
Did anyone notice any performance issues while bulk inserting 75 rows of 15 columns per row into SharePoint 2013 using CSOM?
VHi,
You can use batch processing with csom. Ex.
function CreateListItems(objMyArray) {
var itemArray = [];
var clientContext = SP.ClientContext.get_current();
var oList = clientContext.get_web().get_lists().getByTitle('MyList');
for(index in objMyArray){
var curObject = itemArray[index];
var itemCreateInfo = new SP.ListItemCreationInformation();
var oListItem = oList.addItem(itemCreateInfo);
oListItem.set_item('Title', curObject.title);
oListItem.update();
itemArray[i] = oListItem;
clientContext.load(itemArray[i]);
clientContext.executeQueryAsync(onQuerySucceeded, onQueryFailed);
And it goes well.(No Performance issue) -
Batch inserts and insertion order
I'm inserting about 14,000 rows in one transaction, each by calling a stored procedure that has only the insert statement with a sequence.next_val to generate the ID. I don't commit or rollback in the procedure.
I call the stored procedure using a PreparedStatement and batching the calls. After 14,000 calls to addBatch() I call executeBatch().
The order in which I call is determined by the application and is critical because we want the ids to match that order.
I see the order of the rows following the id's values does not match the order expected.
Luckily I can export the same data that is inserted to a file as a "preview" and after I saw the order problem in the inserted data I exported that preview file and I was puzzled to see the order is fine there. This is the same application and I did not even restart it.
Moreover, looking at the code, it is impossible to insert the data in the order seen in the table, mainly because the rows that are sent together are appearing interleaved with rows that were sent afterwards. There are no duplicate rows, just out of order.
My questions are meant to rule out causes:
1) Is there any chance the Oracle Thin Driver is changing the order of the batched procedure calls?
2) Is there any chance the Oracle 9.2 is receiving the inserts in order but executing them in parallel so the insertion order (i.e. the order in which they get the sequence's value) is different to the order of the calls?
3) Is it possible that large batches are executed in parallel by the engine?
The database has 32 processors (runs solaris) and 48GB of RAM. I'm using JDK 1.4 and Oracle's thin driver for Java 1.4 (ojdbc14.jar).
Thanks in advance for your help.The order in which I call is determined by the
application and is critical because we want the ids
to match that order.A table does not have "an order". Unless you specify an ORDER BY clause with your SELECT the order of the returned data is not specified and never guaranteed to remain the same.
Luckily I can export the same data that is inserted
to a file as a "preview" and after I saw the order
problem in the inserted data How die you export (i.e. retrieve) the data?
Did you supply an ORDER BY? -
hello folks...
before procceding furthur I would like to thank the people who have helped me till date on this forumn..without whose help I would not have held on to my job.Thanks
I am wking on Oracle batch statements inorder to improove the efiicienecy of Inserts that run into millions of records.I have been able to improove the efficiency a lot by doing so (around 20-25%)
I have a feeling that I could improove the efficiency furthur by playing around with the number of Inserts done per batch.Presently I batch them arbitarily whenever certain criretia is met ( sometimes it is more than a 1000 inserts sometimes it is less than 10).
I have read a lot of articles on batching and searched this forumn too ...but I am not too certain where I should draw the line .on whether to use a batch or a normal insertion.
Since the data I am wking on(Biological data ) is too complicated to do create a test data and since the whole process takes in excess of 16 hrs it is difficult for me to do many arbitary testing (try different combinations of batch sizes) .
What I am looking for is "in a genearl in case" what MIGHT be a good batch size....
I know this is a bit too vague and there may no specific number ...I would appreciate your thoughts and views on it
Thanks for your timeHow many ms/insert do you get in your productionprogram?
I have not tested them individually by say a batch but
that is a good idea.to calculate the rate .Ballpark? "Program run with 1,000,000 inserts takes about half an hour"?
Inspired by the O'Reilly book chapter you showed, I tried PreparedStatement (program below). PreparedStatement seems way faster than a plain Statement, and scales better:
batch size 1 best time 25 = 120 insert/s
batch size 10 best time 21 = 1428 insert/s
batch size 50 best time 31 = 4838 insert/s
batch size 100 best time 37 = 8108 insert/s
batch size 500 best time 341 = 4398 insert/s
batch size 1000 best time 401 = 7481 insert/s
batch size 2000 best time 203 = 29556 insert/s
batch size 3000 best time 339 = 26548 insert/s
batch size 4000 best time 409 = 29339 insert/s
batch size 5000 best time 498 = 30120 insert/s
batch size 6000 best time 725 = 24827 insert/sMaxes out with a batch size of around 2000 at 30,000 inserts a second -- that's way more than I'd have thought possible! I've never done batching, only round-trip queries, so this is all new to me.
...Can anyone see a bug in the code below? I'm getting some weird results with batched PS's on Oracle: if a PS doesn't have any "?" parameters, just a constant insert statement, seems like addBatch() doesn't execute all the statements?? Replace the "?" with "1" and remove the setInt() call. I'm not the first to shout Java Is Broken, but here I have half a mind to say there's a bug in the Oracle driver here...
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
public class demo4
static Connection connection;
public static void main(String args[])
try {
DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver());
connection = DriverManager.getConnection(args[0], args[1], args[2]);
// Let the JIT compiler warm up a bit first
doit(100);
doit(100);
doit(100);
doit(1);
doit(10);
doit(50);
doit(100);
doit(500);
doit(1000);
doit(2000);
doit(3000);
doit(4000);
doit(5000);
doit(6000);
} catch (Exception e) {
System.err.println("exception in db test: " + e);
e.printStackTrace(System.err);
private static void doit(int batch_size)
throws SQLException
long best = -1;
clearTable();
int rounds = batch_size * 3;
for (int n = 0; n < 5; n++) {
long time = batch(batch_size, rounds);
if (best == -1 || time < best)
best = time;
System.out.println("batch size " + batch_size +
" best time " + best +
" = " + ((rounds * 1000) / best) +
" insert/s");
private static void clearTable()
throws SQLException
Statement stmt = connection.createStatement();
try {
stmt.executeUpdate("drop table demo4");
} catch (SQLException e) {
// Ignore, it's just saying "table does not exist"
stmt.executeUpdate("create table demo4 (" +
" a number(10)," +
" b number(10)," +
" c varchar(20)" +
connection.commit();
Util.close(stmt);
private static long batch(int batch_size, int rounds)
throws SQLException
long start_time = System.currentTimeMillis();
connection.setAutoCommit(false);
String sql = "insert into demo4 values (?, 2, 'hello')";
PreparedStatement stmt = connection.prepareStatement(sql);
for (int n = 0; ; n++) {
if (n == rounds || (n % batch_size == 0 && n != 0)) {
int counts[] = stmt.executeBatch();
for (int x = 0; x < counts.length; x++)
System.out.println("count " + x + " = " + counts[x]);
connection.commit();
stmt.clearBatch();
if (n == rounds)
break;
stmt.setInt(1, n);
stmt.addBatch();
Util.close(stmt);
long end_time = System.currentTimeMillis();
long time = end_time - start_time;
System.out.println("batch size " + batch_size +
" time: " + time + " ms");
return time;
public static class Util
public static void close(Statement s)
if (s == null)
return;
try {
s.close();
} catch (Exception e) {
System.err.println("Util.close: error closing Statement: " + e); -
How to control number of records in batch insert by eclipselink
Hi,
We are using eclipselink(2.2) to persist objects and we use following configuration -
<property name="eclipselink.jdbc.batch-writing" value="Oracle-JDBC" />
<property name="eclipselink.jdbc.batch-writing.size" value="5" />
however the number of records inserted is much more than 5( I have seen 5000 records being inserted ). How can we control the number of records inserted once?
Thanks.Binding can be configured using the "eclipselink.jdbc.bind-parameters" property, and is on by default - it should be on for jdbc batch writing.
Batch writing defaults to 100 statements, so I am not sure why it would include all statements in one batch unless it is not batching at all. If you set the logs to finest or all it should print of the values it is using for each property, and also show the SQL and statments it is executing. Can you turn on logging and post portions of the logs, particularly the part showing the transaction in question (though maybe only 6 lines of consecutive inserts).
Logging is controlled through the "eclipselink.logging.level" properties.
Best Regards,
Chris
Maybe you are looking for
-
I have purchased a new iPhone on the online store, but how come Apple can send me a repacking old device (with scar on both the border and the cable port)!!!! And after I phone to their hotline for the issue, they asking to give me hk$400 cash coupon
-
Setting a master page at the global level
Hi, We are purchasing a commercial branding solution to our company SharePoint Portals. My question is: how can we enforce so that any time a Farm Administrator creates a site collection, the site collection gets the default master page derived from
-
Hello everybody! My DTD is external to my XML file. In my XML file I refer to the DTD. I'm parsing my XML file with this piece of code: DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance(); DocumentBuilder builder = factory.newDocume
-
The only way to be able to open my browser again is to restart my computer.
-
Hello, In my communication manager i configured a trunk against a Gatekeeper. When i dial from an IP communicator to the Gatekeeper i have a 15 sec of delay but when i try from any hardphone the call is established quickly. In the device configurati