Best way to Insert Records into a DB

hi
i have around 400000(4 lakh ) records to insert into a db , whats the optimal way to do it ,
i tried it doin g using threads i could gain only 2 seconds for 4 lakh records ,suggest me a better way

Very hard thread, poor informations u give us can not help me too much to understand really the problem.
Where do u have to 40.000K (input) records?
Those records must be only added to output table?
How many rows, and how many indexes output table has?
The cost oh each insert depends also on how many indexes dbms has
to manage on oupy table.
In general to take about 2 seconds to add to a table a large amount of rows depends on many variables (hardware performances, dbms performances and so on)
If u ahve only to insert and your input secord statys on another table,
I think that a performance way to do it is insert select, so
insert into output select * from input
If your input records statys on a text file, the best way is to use
native dbms importer
Let me know something more....
Regards

Similar Messages

Efficient Way of Inserting records into multiple tables

Hello everyone,
Im creating an employee application using struts framework. One of the functions of the application is to create new employees. This will involve using one web form. Upon submitting this form, a record will be inserted into two separate tables. Im using a JavaBean (Not given here) between the JSP page and the Java file (Which is partly given below). Now this Java file does work (i.e. it does insert a record into two seperate tables).
My question is, is there a more efficient way of doing the insert into multiple tables (in terms of performance) rather than the way I've done it as shown below? Please note, I am using database pooling and MySQL db. I thought about Batch processing but was having problems writing the code for it below.
Any help would be appreciated.
Assad
package com.erp.ems.db;
import com.erp.ems.entity.Employee;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.Collection;
import java.util.ArrayList;
public class EmployeeDAO {
     private Connection con;
     public EmployeeDAO(Connection con) {
          this.con = con;
     // METHOD FOR CREATING (INSERTING) A NEW EMPLOYEE
     public void create(Employee employee) throws CreateException {
          PreparedStatement psemployee = null;
          PreparedStatement psscheduleresource = null;
          String sqlemployee = "INSERT INTO employee (FIRSTNAME,SURNAME,GENDER) VALUES (?,?,?)";
          String sqlscheduleresource = "INSERT INTO scheduleresource (ITBCRATE,SKILLS) VALUES (?,?)";
          try {
               if (con.isClosed()) {
                    throw new IllegalStateException("error.unexpected");
                        // Insert into employee table
               psemployee = con.prepareStatement(sqlemployee);
               psemployee.setString(1,employee.getFirstName());
               psemployee.setString(2,employee.getSurname());
               psemployee.setString(3,employee.getGender());
                        // Insert into scheduleresource table
               psscheduleresource = con.prepareStatement(sqlscheduleresource);
               psscheduleresource.setDouble(1,employee.getItbcRate());
               psscheduleresource.setString(2,employee.getSkills());
               if (psemployee.executeUpdate() != 1 && psscheduleresource.executeUpdate() != 1) {
                    throw new CreateException("error.create.employee");
          } catch (SQLException e) {
               e.printStackTrace();
               throw new RuntimeException("error.unexpected");
          } finally {
               try {
                    if (psemployee != null && psscheduleresource != null)
                         psemployee.close();
                         psscheduleresource.close();
               } catch (SQLException e) {
                    e.printStackTrace();
                    throw new RuntimeException("error.unexpected");
     }

Hi ,
U can use
set Auto Commit function here ..
let it be false first
and when u do with u r all queries ..
make it true
this function take boolean values
i e helful when u want record to be inserted in all or not at all..
Hope it helps

Best way to insert data into a non indexed partitioned table?

Hi,
I've a rather basic question that I haven't been able to find an answer to so hope that someone may be able to help.
We need to insert sales data into a partitioned table that is range partitioned on sales date. The incoming sales data can span a number of days, e.g. 31-JUL-2007 through to 04-AUG-2007 inclusive. I've come up with two approaches and would like to know which you would recommend and why?
Approach 1: Load data into each individual partition using a statement like:
INSERT /*+ APPEND */ INTO sales PARTITION (sales_20070731) SELECT * FROM sales_new WHERE sales_date = TO_DATE('31-AUG-2007','DD-MON-YYYY')
Approach 2: Load data into the entire table using a statement like:
INSERT /*+ APPEND */ INTO sales SELECT * FROM sales_new
Thanks

You need to compare both approaches by creating simple test case.
But there is no advantages in approach 1until you will have index on sales_date column.
With index these approaches are comparable and you can choose whatever best fits to your requirements.
Best Regards,
Alex

Best way to insert millions of records into the table

Hi,
Performance point of view, I am looking for the suggestion to choose best way to insert millions of records into the table.
Also guide me How to implement in easier way to make better performance.
Thanks,
Orahar.

Orahar wrote:
Its Distributed data. No. of clients and N no. of Transaction data fetching from the database based on the different conditions and insert into another transaction table which is like batch process.Sounds contradictory.
If the source data is already in the database, it is centralised.
In that case you ideally do not want the overhead of shipping that data to a client, the client processing it, and the client shipping the results back to the database to be stored (inserted).
It is must faster and more scalable for the client to instruct the database (via a stored proc or package) what to do, and that code (running on the database) to process the data.
For a stored proc, the same principle applies. It is faster for it to instruct the SQL engine what to do (via an INSERT..SELECT statement), then pulling the data from the SQL engine using a cursor fetch loop, and then pushing that data again to the SQL engine using an insert statement.
An INSERT..SELECT can also be done as a direct path insert. This introduces some limitations, but is faster than a normal insert.
If the data processing is too complex for an INSERT..SELECT, then pulling the data into PL/SQL, processing it there, and pushing it back into the database is the next best option. This should be done using bulk processing though in order to optimise the data transfer process between the PL/SQL and SQL engines.
Other performance considerations are the constraints on the insert table, the triggers, the indexes and so on. Make sure that data integrity is guaranteed (e.g. via PKs and FKs), and optimal (e.g. FKs should be indexes on the referenced table). Using triggers - well, that may not be the best approach (like for exampling using a trigger to assign a sequence value when it can be faster done in the insert SQL itself). Personally, I avoid using triggers - I rather have that code residing in a PL/SQL API for manipulating data in that table.
The type of table also plays a role. Make sure that the decision about the table structure, hashed, indexed, partitioned, etc, is the optimal one for the data structure that is to reside in that table.

What is the best way to insert massive data into an existing excel file?

dear gurus,
i am wondering that what is the best way to insert massive data into an existing excel file, more performance perspective.
the file is read from BDS , and we want to insert data into it .
the way i can think of is
1. OLE AUTOMATION
   i think performance will be a big problem
2. Office integration
    i am not sure it's facing the same performance issue ?
3 . XXL_SIMPLE_API/FULL_API
    I am not sure whether they can insert data into an existing excel file?
could you please give me some advices?
br.
jun

Hi,
If you want to APPEND data( add data to an existing excel file) from SAP, then use GUI_DOWNLOAD fm with APPEND = 'X' paramter.
Best regards,
Prashant

Best way to insert 300dpi photo's into Ilustrator file for print

I am creating a 5.5 X 8 postcard style card for print. I need to place 15 photographs that are aprox. 1.5 X 2.5 each. What is the best way to insert these files so that my file doesn't become too large? I will be making a PDF of the final artwork to send to the printer. Your guidance is much appreciated!!

Link don't embed the images. Use a layered format such as .psd, .tif, or .pdf. Psd is not lossy, .tif can be lossy if you modify the default settings, .pdf you choose how much compression you want, and a small amount of compression can make a much smaller file size.
I would recomend using .psd, in cmyk color space. When you make your pdf for printing is when you will decide how much compression and which pdf preset you will use(press ready or PDF X should be a good choice). That will be your important choice.

File To RFC Scenario- simply insert records into ztab

Hi
i have a requirment, where XI gets a flatfile and needs to map to RFC function module and then inserts records into ztable in SAP system. it doesn't require any response back.
proble here is:
i don't want to get RFC 'Response mesage' when i import RFC into XI IR. i require only 'Request message'.
please help me, how to create a RFC function module with out Response message, and it should contain only Request message.
i have created RFC in this way:
RFC doesn't vcontains neither import nor export parameters. and i have defined on table in TABLE tab which refers to a ztable created already.
Thanks in advance..
Regards,
Rajesh

Thankx Michal,
I have written ZRFC this way:
insert ZMM_AUTO_GR from INPUT_TABLE .
if sy-subrc = 0.
commit work.
else.
rollback work.
im not using neither import nor export parameters..but using only tables parameters. but when imported in XI im getting same table structure in both request an response.
please suggest what changes required in above code to invoke my ZRFC in async way.
you will not get any response in XI- what chages required in my code to dnt get .reponse in XI.
appreciate your help.
Regards,
Rajesh.

Inserting records into Table with check table logic in place

I want to insert records into a table, and have the check table to not allow invalid entries. Is there a function out there that will allow this? I am currently using the insert statement and it is working except that it is not giving an error when the value does not exist in the check table for the particular fields.
INSERT INTO ZSD_XREF VALUES WA_XREF.
That is the basic statement I'm using ZSD_XREF has several fields one being the material number field tied to the check table which happens to be MAKT file. The insert statement is not validating the material numbers using the check table, or at least it is not giving an error. Any ideas?

Paul,
Unfortunately, open SQL statements such as INSERT, UPDATE do not go through the check table logic as they directly hit the database layer. Check table checks are performed only if you go through application layer that is when you enter the same data through a screen.
You have to do the checks yourself.
Happy checking!!!
Srinivas

In Acrobat Professional 8, what is the best way to insert/combine multiple pdf's together in a large

In Acrobat Professional 8, what is the best way to insert/combine multiple pdf's together in a large volume?
We have 300 pdf reports and need to insert a 2 page cover page infront of each report. Not sure if Batch processing is best???
Thanks for any tips.

Probably each cover page is different too. I would probably just bite the bullet and do each individually. I would create the 2 cover pages in WORD or other word processor and print to cover.pdf. Then open a PDF and Pages>Insert Pages or the cover.pdf to the front of the open PDF and save as to the current PDF. Then repeat 299 times. Each time you would make the appropriate change to the DOC file and print a new cover.pdf file (you might want to turn off open in Acrobat for this processing in the printer properties to save time). Probably a good idea to keep a list of the files to check off what has been done (you can generate a list in DOS by Start>cmd, change the directory to your location [cd path], and do "dir >>list.txt". That will give you a list to use.). There may be an easier way, but by the time you get it figured out you might be done this way.

What is the best way of insertion using structured or binary type in oracle

what is the best way of insertion using structured or binary type in oracle xml db 11g database

SQL*Loader.

Is there a way to insert comments into numbers on an ipad, or an alternative?

Is there a way to insert comments into numbers on an ipad, or an alternative?

If you mean in the iBooks app - tap on Collections in the upper left corner. Then tap New. Name the collection. Tap Done. Then tap the Edit button in the upper right. Tap on the books that you want to move to select them. Tap Move in the upper let corner. Select the newly created collection.

I would like to know the best way to scan music into a pdf format which I can then sync with ipad

I would like to know the best way to scan music into a pdf format which I can then sycn with ipad

smgchandler,
Any camcorder that can connect via USB or that can save to an SD card can interface with an iPad if you have the iPad Camera Connection Kit so that you can watch your videos on the new iPad's beautiful retina display. You can find this on the Apple website or in most big box retail stores.
http://store.apple.com/us/product/MC531ZM/A?fnode=MTc0MjU4NjE

Inserting records into a table with all caps

Hello
I have a procedure that inserts records into a table. How do I ensure that the text values inserted are recorded all capital letters into the table?
Thanks.

You can use UPPER(..) function in your insert statement, so that values are converted to UPPER, before insert.
If you want to check at table level, you can achieve that by writting a before insert trigger and in that trigger check
IF UPPER(:new.<col>) != :new.<col> THEN
RAISE_APPLICATION_ERROR(-20101,'Error: Not all values are in upper case')
END IF;

Best way to insert a handmade signature

Best way to insert a handmade signature
I would like to know the best way to insert a handmade signature in a pdf acrobat document.
Best Regards
p.d. A tutorial perhaps ?

Yep you link in now fixed. what most likely happed is when you click on Link tool in the forum, when window open the http:// is already in the window. Most likely you clicked at the beginning of the http and pasted your link.
I always when this window open triple click on the http:// so everything is selected, then hit Delete (I use Mac - PC it would be backspace) so window is completely empty then I paste the link.
But I occasionally I let it happen myself.

Servlet inserting record into postgresql through hibernate

Hi
I have developed a servlet. This servlet is inserting record into postgres sql through hibernate using eclispe ide. When I run my code then I find the following exception
exception
java.lang.NullPointerException
     hibernate.example.FirstExample.doGet(FirstExample.java:54)
     javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
     javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
     org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
I have developed the following code
package hibernate.example;
public class Contact {
     private String firstName;
     private String lastName;
     private String email;
     private long id;
     public String getEmail() {
     return email;
     public String getFirstName() {
     return firstName;
     public String getLastName() {
     return lastName;
     public void setEmail(String string) {
     email = string;
     public void setFirstName(String string) {
     firstName = string;
     public void setLastName(String string) {
     lastName = string;
     public long getId() {
     return id;
     public void setId(long l) {
     id = l;
**Now the servlet is**
package hibernate.example;
import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.ServletConfig;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.hibernate.cfg.Configuration;
public class FirstExample extends HttpServlet     {
     Session session = null;
     public String getServletInfo() {
          return "Servlet connects to PostgreSQL database and displays result of a SELECT";
     public void doGet(HttpServletRequest request, HttpServletResponse response)
     throws IOException, ServletException {
          try{
               response.setContentType("text/html");
               PrintWriter out = response.getWriter();
               Configuration cfg = new Configuration().configure();
               SessionFactory factory = cfg.buildSessionFactory();
               session = factory.openSession();
               out.println("Inserting Record");
               Contact contact = new Contact();
               contact.setId(6);
               contact.setFirstName("Deepak");
               contact.setLastName("Kumar");
               contact.setEmail("[email protected]");
               session.save(contact);
               out.println("Done");
          catch(Exception e){
               System.err.println(e.getMessage());
          finally{
               session.flush();
               session.close();
**web.xml is**
<web-app>
          <database>
               <driver>
               <type>org.postgresql.Driver</type>
          <url>jdbc:postgresql://127.0.0.1:5432/ali</url>
          <user>ali</user>
          <password>ali</password>
               </driver>
          </database>
          <servlet>
          <servlet-name>FirstExample</servlet-name>
          <servlet-class>hibernate.example.FirstExample</servlet-class>
          </servlet>
          <servlet-mapping>
          <servlet-name>FirstExample</servlet-name>
          <url-pattern>/hb</url-pattern>
          </servlet-mapping>
</web-app>
**And contact.hbm.xml is**
<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC
"-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<hibernate-mapping>
<class name="hibernate.example.Contact" table="CONTACT">
<id name="id" type="long" column="ID" >
<generator class="assigned"/>
</id>
<property name="firstName">
<column name="FIRSTNAME" />
</property>
<property name="lastName">
<column name="LASTNAME"/>
</property>
<property name="email">
<column name="EMAIL"/>
</property>
</class>
</hibernate-mapping>
**And hibernate.cfg.xml is**
<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD//EN"
"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="hibernate.connection.driver_class">com.postgressql.jdbc.Driver</property>
<property name="hibernate.connection.url">jdbc:postgresql://127.0.0.1:5432/ali</property>
<property name="hibernate.connection.username">ali</property>
<property name="hibernate.connection.password">ali</property>
<property name="hibernate.connection.pool_size">10</property>
<property name="show_sql">true</property>
<property name="dialect">org.hibernate.dialect.PostgreSQLDialect</property>
<property name="hibernate.hbm2ddl.auto">update</property>

<mapping resource="contact.hbm.xml"/>
</session-factory>
</hibernate-configuration>
Pleas tell me that where I have done the mistake and how can I resolove the error.

ahmadgee wrote:
Thank u very much.
First I tried with out the servlet. I took a simple class. But it was displaying the null pointer exception.
then you still have the problem. do you understand the root cause?
After then I made a servlet. how was that going to help your null pointer exception?
Now it is displaying error hibernate.cfg.xml not found.that's part of the problem. where is the servlet reading that file and initializing the session factory? how is it finding that file?
i don't know where you have that .cfg.xml file, or where the recommended place is where hibernate expects to see it, but i'd bet it needs to be in the CLASSPATH, which means WEB-INF/classes.
figure out why you get that NPE first, though. the servlet is only making things worse.
%

Best way to Insert Records into a DB

Similar Messages

Maybe you are looking for