Is this the fastest way to insert 100k rows into a database?

I'm looking to insert ~100k rows into a database as fast as possible. I'm connected to the database through a small LAN with static IP adresses. The target database is running my SQL. Here's the fastest I've gotten the code so far. Task manager shows that it's only sending like 2.5mbps. If I take the size of the 2D array (64MB) and divide by the time it takes to transfer it currently, I calculate ~160kbps. Is that the best I can hope for with the NI DB Toolkit?
CLD (2014)

Hi,
Use DB Tool Insert Data VI to insert the data in the database.
You can insert complete row data at a time. No need to insert single-single element in database this will reduce your code performance.
Thanks and Regards
Himanshu Goyal | LabVIEW Engineer- Power System Automation
Values that steer us ahead: Passion | Innovation | Ambition | Diligence | Teamwork
It Only gets BETTER!!!

Similar Messages

What is the best way to insert massive data into an existing excel file?

dear gurus,
i am wondering that what is the best way to insert massive data into an existing excel file, more performance perspective.
the file is read from BDS , and we want to insert data into it .
the way i can think of is
1. OLE AUTOMATION
   i think performance will be a big problem
2. Office integration
    i am not sure it's facing the same performance issue ?
3 . XXL_SIMPLE_API/FULL_API
    I am not sure whether they can insert data into an existing excel file?
could you please give me some advices?
br.
jun

Hi,
If you want to APPEND data( add data to an existing excel file) from SAP, then use GUI_DOWNLOAD fm with APPEND = 'X' paramter.
Best regards,
Prashant

Is this the fastest way to copy files?

I'm looking for a way to take 1 file and make many copies of it. Is this fastest way you know of?
import java.io.*;
import java.nio.channels.*;
public static void applyFiles( File origFile, File[] files )
FileInputStream f1 = new FileInputStream( origFile.getAbsolutePath() );
FileChannel source = f1.getChannel();
FileOutputStream f2[] = new FileOutputStream[ files.length ];
FileChannel target;
for( int x=0; x < files.length; x++ )
    if( origFile.getAbsolutePath() != files[x].getAbsolutePath() )
      f2[x] = new FileOutputStream( files[x].getAbsolutePath() );
      target = f2[x].getChannel();
      source.transferTo(0, source.size(), target);
}

2 questions from your code...
1) I assume the last line should read
out.write(buffer,0,numRead);
2) Doesnt this just read in a piece at a time and write that piece to each file? Isn't that degrading performance to have so many files open and waiting at once? Would it make more sense to read into a StringBuffer, then write all the data to each file at a time?
Thanks
I'd have to say that your question is loaded. :)
Without knowing anything about your target system, I'd
have to say no, this is not the fastest way to copy a
file to many files. This will end up reading the file
once for every time you want to copy it. Which may
work fine if the file is small, but not when the file
that is being copied is larger then free ram. Or if
the file channel implementation sucks.
For the general case, where you don't know how big the
file will be beforehand I'd say that this is a better
algorithim.
public static void oneToManyCopy( File source, File[]
dest ) throws Exception(s) {
FileInputStream in = new FileInputStream( source );
FileOutputStream out[] = new
w FileOutputStream[dest.length];
for ( int i = 0 ; i < dest.length;++i)
out[i] = new FileOutputStream(dest);
byte buffer[] = new byte[1024]; // or whatever size
e you like
int numRead;
while ( ( numRead = in.read(buffer,0,buffer.length)
) ) > -1 ) {
for ( int i = 0 ; i < out.length; ++i )
out.write(buffer,0,numRead);

What is the proper way to insert an envelope into a hp9680?

What is the proper way to put an envelope into a HP 6980?

Hi,
Normally th software (driver) will shows you how to feed tha envelop for example like this.
Regards
BH
**Click the KUDOS thumb up on the left to say 'Thanks'**
Make it easier for other people to find solutions by marking a Reply 'Accept as Solution' if it solves your problem.

What is the efficient way of insert some bytes into a file?

Hello, everyone:
If I want to insert some bytes into a file (for example, insert the bytes before all the original content of the file, or append the bytes to a file), and the size of the original file is very big. I am wondering what is the efficient way? Where can I get some sample codes?
regards,
George

Thanks, DrClap.
I have tried your method and you are correct. I have written a simple program which can be used to insert "Hello World " to the start of a file ("c:\\temp\\input.txt"), and I have verified that it can work. Please help to see whether it is correct and whether it has a more efficient way.
public class TestDriver {
     public static void main(String[] args) {
          byte[] back_buffer = new byte [1024];
          byte[] write_buffer = new byte [1024];
          System.arraycopy("Hello World".getBytes(), 0, write_buffer, 0, "Hello World".getBytes().length);
          int write_buffer_length = "Hello World ".getBytes().length;
          int count = 0;
          FileInputStream fis = null;
          FileOutputStream fos = null;
          try {
               fis = new FileInputStream (new File("c:\\temp\\input.txt"));
               fos = new FileOutputStream (new File("c:\\temp\\output.txt"));
               while ((count = fis.read (back_buffer)) >= 0)
                    fos.write(write_buffer, 0, write_buffer_length);
                    System.arraycopy (back_buffer, 0, write_buffer, 0, count);
                    write_buffer_length = count;
               //write the last block
               fos.write(write_buffer, 0, write_buffer_length);
               fis.close();
               fos.close();
               //copy content back into original file
               fis = new FileInputStream (new File("c:\\temp\\output.txt"));
               fos = new FileOutputStream (new File("c:\\temp\\input.txt"));
               while ((count = fis.read (back_buffer)) >= 0)
                    fos.write(back_buffer, 0, count);
               fis.close();
               fos.close();
               //remove temporary file
               File f = new File ("c:\\temp\\output.txt");
               f.delete();
          } catch (FileNotFoundException e) {
               // TODO Auto-generated catch block
               e.printStackTrace();
               try {
                    fis.close();
               } catch (IOException e1) {
                    // TODO Auto-generated catch block
                    e1.printStackTrace();
               try {
                    fos.close();
               } catch (IOException e2) {
                    // TODO Auto-generated catch block
                    e2.printStackTrace();
          } catch (IOException e) {
               // TODO Auto-generated catch block
               e.printStackTrace();
               try {
                    fis.close();
               } catch (IOException e1) {
                    // TODO Auto-generated catch block
                    e1.printStackTrace();
               try {
                    fos.close();
               } catch (IOException e2) {
                    // TODO Auto-generated catch block
                    e2.printStackTrace();
}regards,
George

What is the best practice for inserting (unique) rows into a table containing key columns constraint where source may contain duplicate (already existing) rows?

My final data table contains a two key columns unique key constraint. I insert data into this table from a daily capture table (which also contains the two columns that make up the key in the final data table but are not constrained
(not unique) in the daily capture table). I don't want to insert rows from daily capture which already exists in final data table (based on the two key columns). Currently, what I do is to select * into a #temp table from the join
of daily capture and final data tables on these two key columns. Then I delete the rows in the daily capture table which match the #temp table. Then I insert the remaining rows from daily capture into the final data table.
Would it be possible to simplify this process by using an Instead Of trigger in the final table and just insert directly from the daily capture table? How would this look?
What is the best practice for inserting unique (new) rows and ignoring duplicate rows (rows that already exist in both the daily capture and final data tables) in my particular operation?
Rich P

Please follow basic Netiquette and post the DDL we need to answer this. Follow industry and ANSI/ISO standards in your data. You should follow ISO-11179 rules for naming data elements. You should follow ISO-8601 rules for displaying temporal data. We need
to know the data types, keys and constraints on the table. Avoid dialect in favor of ANSI/ISO Standard SQL. And you need to read and download the PDF for:
https://www.simple-talk.com/books/sql-books/119-sql-code-smells/
>> My final data table contains a two key columns unique key constraint. [unh? one two-column key or two one column keys? Sure wish you posted DDL] I insert data into this table from a daily capture table (which also contains the two columns that make
up the key in the final data table but are not constrained (not unique) in the daily capture table). <<
Then the "capture table" is not a table at all! Remember the fist day of your RDBMS class? A table has to have a key. You need to fix this error. What ETL tool do you use?
>> I don't want to insert rows from daily capture which already exists in final data table (based on the two key columns). <<
MERGE statement; Google it. And do not use temp tables.
--CELKO-- Books in Celko Series for Morgan-Kaufmann Publishing: Analytics and OLAP in SQL / Data and Databases: Concepts in Practice Data / Measurements and Standards in SQL SQL for Smarties / SQL Programming Style / SQL Puzzles and Answers / Thinking
in Sets / Trees and Hierarchies in SQL

Is this the fastest way to create a single color bufferedImage?

Here is my code but it's slow sometimes. Is there a better way?
BufferedImage bi = new BufferedImage(1600,1200, BufferedImage.TYPE_INT_RGB);
int[] rgbs = new int[1600*1200];
for (int j =0; j<1600*1200; j++)
   rgbs[j] = Color.RED;
bi.setRGB(0,0,1600,1200,rgbs,0,1600);

This will be even faster.
int w = 1600;
int h = 1200;
int masks[] = new int[4];
rv[0] = 0xff0000; // red
rv[1] = 0x00ff00; // green
rv[2] = 0x0000ff; // blue
rv[3] = 0xff000000; // alpha
SampleModel sm = new SinglePixelPackedSampleModel(DataBuffer.TYPE_INT,
                                                  w, h, masks);
final int rawData[] = new int[w * h];
DataBufferInt db = new DataBufferInt(rawData, rawData.length);
WritableRaster ras = Raster.createWritableRaster(sm, db, null);
ColorSpace cs = ColorSpace.getInstance(ColorSpace.CS_sRGB);
ColorModel cm = new DirectColorModel(cs, 32, masks[0], masks[1],
                                     masks[2], masks[3],
                                     false,
                                     DataBuffer.TYPE_INT);
BufferedImage img = new BufferedImage(cm, ras, false, null);
int value = 0xffff0000; // opaque red.
int l = rawData.length;
// Set the image to solid color of the value
for(int i = 0; i < rawData.length; i++)
    rawData[i] = value;

Insert Multiple Rows into a Database

Hi,
I've been working on a project that takes an xml document from JMS queue and inserts the data into db. But the problem here is that there can be any number of data sets. I've been using OEPE for my task, but no matter what I do, the data simply isn't inserted. Could any body please help me? Here is a sample message that I'm trying to insert, as well as the XSDs for the JCA and the input documents.
XSD for JMS Message:
<?xml version="1.0" encoding="windows-1252" ?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns="http://www.example.org"
targetNamespace="http://www.example.org"
elementFormDefault="qualified">
<xsd:element name="cannonical">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="orders">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="data" maxOccurs="unbounded">
     <xsd:complexType>
          <xsd:sequence>
               <xsd:element name="rowCode" type="xsd:string"/>
               <xsd:element name="productNumber" type="xsd:string"/>
               <xsd:element name="attributeType" type="xsd:string"/>
               <xsd:element name="attributeCode" type="xsd:string"/>
               <xsd:element name="transactionType" type="xsd:string"/>
               <xsd:element name="timestamp" type="xsd:dateTime"/>
          </xsd:sequence>
     </xsd:complexType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:schema>
JCA XSD:
<?xml version = '1.0' encoding = 'UTF-8'?>
<xs:schema targetNamespace="http://xmlns.oracle.com/pcbpel/adapter/db/top/InsertData" xmlns="http://xmlns.oracle.com/pcbpel/adapter/db/top/InsertData" elementFormDefault="qualified" attributeFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="OrderDataCollection" type="OrderDataCollection"/>
<xs:complexType name="OrderDataCollection">
<xs:sequence>
<xs:element name="OrderData" type="OrderData" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
<xs:complexType name="OrderData">
<xs:sequence>
<xs:element name="prodnum" type="xs:decimal"/>
<xs:element name="attrtype" minOccurs="0" nillable="true">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:maxLength value="20"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element name="attrcode" minOccurs="0" nillable="true">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:maxLength value="10"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element name="transtype" minOccurs="0" nillable="true">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:maxLength value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element name="ordertime" type="xs:dateTime" minOccurs="0" nillable="true"/>
</xs:sequence>
</xs:complexType>
</xs:schema>
Sample Input:
<?xml version="1.0" encoding="windows-1252"?>
<ProductData>
<header>
<rowCode>H</rowCode>
<interfaceCode>H6D9J7</interfaceCode>
<createdOn>
<date>
<year>2011</year>
<month>05</month>
<day>23</day>
</date>
<time>
<hour>02</hour>
<minute>05</minute>
<seconds>11</seconds>
</time>
</createdOn>
<description>Front Shop Item Attributes from FMS to JDA PMM</description>
</header>
<body>
<data>
<rowCode>D</rowCode>
<productNumber>0004567</productNumber>
     <attributeType>TTTTTT</attributeType>
<attributeCode>CCCCCCCjjj</attributeCode>
     <transactionType>A</transactionType>
<timestamp>
<date>
          <year>2011</year>
<month>02</month>
          <day>19</day>
     </date>
     <time>
     <hour>02</hour>
<minute>05</minute>
<seconds>11</seconds>
</time>
</timestamp>
</data>
     <data>
<rowCode>D</rowCode>
<productNumber>0004567</productNumber>
<attributeType>TTTTTT2</attributeType>
<attributeCode>CCCCCCjjj2</attributeCode>
<transactionType>A</transactionType>
<timestamp>
<date>
          <year>2011</year>
<month>02</month>
          <day>19</day>
     </date>
     <time>
     <hour>02</hour>
<minute>05</minute>
<seconds>11</seconds>
</time>
</timestamp>
</data>
</body>
<footer>
<rowCode>Z</rowCode>
<interfaceCode>H6D9J7</interfaceCode>
<numberOfRows>000000000002</numberOfRows>
</footer>
</ProductData>
I'll be really grateful if anybody can help me out. Thank you.

Check below link it may help you out
how to Insert multiple records using single DB Adapter
Thanks
AJ

Inserting Multiple Rows into Database Table using JDBC Adapter - Efficiency

I need to insert multiple rows into a database table using the JDBC adapter (receiver).
I understand the traditional way of repeating the statement multiple times, each having its <access> element. However, I am just wondering whether this might be performance-inefficient, as it might insert records one by one.
Is there a way to ensure that the records are inserted into the table as a block, rather than record-by-record?

Hi Bhavesh/Kanwaljit,
If we have multiple ACCESS tags then what happens is that the connection to the database is made only once. But the data is inserted row by row.
Why i am saying this?
If we add the following in JDBC Adapter..logSQLStatement = true. Then incase of multiple inserts we can see that there are multiple
<i>Insert into tablename(EMP_NAME,EMP_ID) VALUES('J','1000')
Insert into tablename(EMP_NAME,EMP_ID) VALUES('J','2000')</i>
Doesnt this mean that rows are inserted one by one?
Correct me if i am wrong.
This does not mean that the transaction is not guaranted. Either all the rows will be inserted or rolled back.
Regards,
Sumit

How to make a table copy - the fastest way????

Hi,
Well, I have the following task to do in the minimal possible time:
Task: duplicate a table in my instance database. The source table name is SPC.
The destination table name is SPC_REP. Both are partitioned.
The trouble is my source table has 40 million rows, but I need to copy only 27 million related to a column which I will use a condition in where clause.
Ok, so I have some questions:
Question 1)
What's the fastest way to provide this copy?
Export/import, Insert into, sqlloader (this one I've never used)
Question 2)
I am planning to do this by export/import utility. But I don't want to waste time doing first the export process and only after all doing import process. I would like to do export and import simultaneously to save time as table has so many rows.
Question 3)
If I use export/import I will not be able to create the copy table on the same schema as it is my source table, correct? Correct me, if I'm wrong...
Thank you all,
And any hint will be appreciated.

Export & import probably isn't the best solution here. It's not particularly fast, reuires that you create an additional copy of the data in your dump file, involves the cost of pulling all the data out of Oracle and puts it back in, etc.
Transportable tablespaces would be an option, assuming your partitioned table isn't in the same tablespace as a bunch of other opjects.
Personally, I would probably do a
CREATE TABLE <<copy name>>
AS
SELECT *
FROM <<original name>>
WHERE 1=2
INSERT /*+ APPEND */
INTO <<copy name>>
NOLOGGING
SELECT *
FROM <<original name>>
WHERE <<something>>Justin
Distributed Database Consulting, Inc.
http://www.ddbcinc.com/askDDBC
Message was edited by:
Justin Cave
APPEND & NOLOGGING may be faster so I added that.
Message was edited by:
Justin Cave

What is the fastest way of getting data?

With a scanning electron microscope, I need to scan a 512*512 pixel area with a pixel repetition of 15000 (two channels), meaning averaging over 15000 measurements. Simultaneously I have to adjust the voltage output for every pixel.
I am using a 6111E Multifunction I/O board in a 800MHz P3. The whole task has do be done as fast as possible (not more than 20 minutes altogether).
What is the fastest way to get this huge amount of data with averaging and output in between? (E.g. do I use buffered read with hardware triggering or is there a faster way?)

Using the NI-DAQ API (not LabView) will give you a significant amount of more control over what happens and when to the data stream; which translates to a more efficient program. But you need to program in C/C++ or Delphi then. The Measurement Studio provides ActiveX controls that are like the LabView ones for C&C++ (they�re slow like the LabView ones though � not a lot you can do about the Windows GDI).
What are you trying to sample 15000 times? The 512*512 pixel field?
That�s almost 15Gigs of data! And it means you need to process data at 12.8MB/s to finish it in 20 minutes. I hope you know C, x86 assembly and MMX.
I would setup a huge circular buffer (NI-DAQ calls them �double buffers�), about 30 seconds worth or so, to use with SCAN_Start. Then I would proces
s the actual buffer the card is DMA�ing the data into with a high priority thread. Progressively sum the scan values from the 16bit buffer (the samples are only 12 bit, but the buffer should still be 16bits wide) into a secondary buffer of DWORDs the size of the screen (512*512), and you�ll need two of those, one for each channel. Once the 15000 scans are complete, convert each entry into a float divide by 15000.0f, and store it in a third buffer of floats.
If you wish to contract this out, send me an email at [email protected]

In Acrobat Professional 8, what is the best way to insert/combine multiple pdf's together in a large

In Acrobat Professional 8, what is the best way to insert/combine multiple pdf's together in a large volume?
We have 300 pdf reports and need to insert a 2 page cover page infront of each report. Not sure if Batch processing is best???
Thanks for any tips.

Probably each cover page is different too. I would probably just bite the bullet and do each individually. I would create the 2 cover pages in WORD or other word processor and print to cover.pdf. Then open a PDF and Pages>Insert Pages or the cover.pdf to the front of the open PDF and save as to the current PDF. Then repeat 299 times. Each time you would make the appropriate change to the DOC file and print a new cover.pdf file (you might want to turn off open in Acrobat for this processing in the printer properties to save time). Probably a good idea to keep a list of the files to check off what has been done (you can generate a list in DOS by Start>cmd, change the directory to your location [cd path], and do "dir >>list.txt". That will give you a list to use.). There may be an easier way, but by the time you get it figured out you might be done this way.

What's the fastest way to share files live between 2 Macs in the same room?

Please can I have some advice on this scenario?
I'll need to share HD video footage between 2 new Mac Pros in the same room. 1 Mac will be used to upload the footage and the other to edit it using FCP. The footage can be stored either on hard drives in the edit machine or on desktop hard drives connected to the edit machine (or if there is a better option I'm open to advice).
What is the fastest way of sharing the files and what is the simplest way?
Any suggestions would be greatly appreciated.
Thanks in advance

What is the fastest way of sharing the files and what is the simplest way?
Fastest way? a fiber channel SAN connecting the two machines to a common fiber-channel based storage array. Can't beat it for performance, but it comes at a cost (think $20K as a starting point, depending on the amount of storage you need).
Simplest way? Some external media (thumb drive, external hard drive, etc.) that you shuffle between the two machines
Intermediate: a NAS-based storage box on your network, although be aware that real-time editing of HD video can overwhelm many low-end NAS boxes.

What's the fastest way to look up a User's Work Items?

Hi, my SCSM environment has a few different service desks and plenty of customer drive-by's. Sometimes when a customer comes up and asks for a status, the SD personnel can't find their ticket quickly. The fastest way right now is to 1.) click on CONFIGUATION
ITEMS; 2.) Search for the Affected User; 3.) Click on the Affected User; 4.) Click on the Affected User's Related Items; 5.) Search the related items for the proper Work Item; 6.) Load the Work Item.
It's not bad, but it's a lot of steps and I'm wondering if anybody has a faster method in place.

Just to poke holes in the salesmanship, Consider that you might have an easier time in doing any of the following:
Create a view that lists all open incidents, and make the first column Affected User. Analysts can filter or sort by this column to get quick info
The search box in the top right corner can be set to users, right click the down arrow to see all options.
the search box in the top right can be used to search work item IDs or titles.

What is the fastest way to transfer files and applications from an older iMac to a new MacBook Pro?

What is the fastest way to transfer files and applications from an older iMac to a new MacBook Pro?
I have a Firewire cable and Thunderbolt adapter, but no icons showing either Mac appear on either desktop.

The fastest way is to use Carbon Copy Cloner and a external drive formatted GUID OS X Ext. J in Disk Utlilty, then connecting to the new Mac and using Migration Assistant in the Utilites folder.
Even faster, if you can determine your going to replace all the apps from fresh sources anyway (like if the older Mac's OS X verison is old thus the apps) then just use a external drive and copy just your files to it, then connect to the new Mac and transfer over.
Some apps you can just grab the registration code and install it fresh on the new machine with the old code, talk to the developer about transfering the program, as long as it's deleted on the older Mac in the process.
It used to be Firewire Target disk mode was fastest, but since Thunderbolt came out...
Notice this support doc hasn't been updated since june 2012, no Thunderbolt info
https://support.apple.com/kb/HT1661

Is this the fastest way to insert 100k rows into a database?

Similar Messages

Maybe you are looking for