Reading and Writing to blob column is very slow

Hi
I want to write a serialized java object called 'engine' to a database table called javaObjectsDB. The serialized object could be of sizes anywhere between 20MB to 4GB. I am using the following code to write this 'engine' object
               * Write java object to BLOB in DB - Start
               Connection conn=null;
               conn = Util.getConnectionFromDS();
          PreparedStatement ps=null;
          String sql=null;
               Engine.startWatch();
               ByteArrayOutputStream baos = new ByteArrayOutputStream();
               Engine.endWatch("Time for ByteArrayOutputStream");
               Engine.startWatch();
               ObjectOutputStream oos = new ObjectOutputStream(baos);
               Engine.endWatch("Time for ObjectOutputStream");
               Engine.startWatch();
               oos.writeObject(this.engine_);
               Engine.endWatch("Time for oos.writeObject");
               oos.flush();
               Engine.startWatch();
               byte[] data = baos.toByteArray(); // in case of using ByteArrayOutputStream
               Engine.endWatch("Time for toByteArray");
               Engine.startWatch();
     sql="insert into javaObjectsDB values (?)";
     ps=conn.prepareStatement(sql);
     ps.setObject(1, data); // in case of using ByteArrayOutputStream
     ps.executeUpdate();
     Engine.endWatch("Time for prepare statement and insert");
               * Write java object to BLOB in DB - End
To read from database BLOB column back to the engine object I use the following code:
try {
               String readBlob = "SELECT javaObject FROM javaObjectsDB ";
               Statement readBlobStmt = conn.createStatement();
               ResultSet resultSet = readBlobStmt.executeQuery(readBlob);
               while (resultSet.next()) {
                    * ByteArrayStream *** Start
                    Engine.startWatch();
                    Blob blob = resultSet.getBlob(1);
                    Engine.endWatch("loadPlan::Time for getBlob");
                    Engine.startWatch();
                    byte [] data = blob.getBytes(1,(int)blob.length());
                    Engine.endWatch("loadPlan::Time for getBytes");
                    Engine.startWatch();
               ByteArrayInputStream bais = new ByteArrayInputStream(data);
               Engine.endWatch("loadPlan::Time for ByteArrayInputStream");
               Engine.startWatch();
               ObjectInputStream oin = new ObjectInputStream(bais);
               Engine.endWatch("loadPlan::Time for ObjectInputStream");
               Engine.startWatch();
               engine_ = (Engine)oin.readObject();
               Engine.endWatch("loadPlan::Time for oin.readObject");
               System.out.println("engine object prepared");
                    * ByteArrayStream *** End
               } //while of result Set
The time taken to write and read for a engine object of size 124MB are:
write - 41secs
read - 28secs
For a engine object of size 340MB
write - 3minutes
read - 1minute
This is a lot of time for such small object size, and since we are expecting sizes of 4GB, I need to have better performance. Please suggest what I can do to improve performance for read and write.
One thing I tried was using cached lobs but not of much use.

Welcome to the forum!
Unfortunately this is an Oracle forum and your question should be posted in the Java JDBC forum.
https://forums.oracle.com/forums/category.jspa?categoryID=288
Please create a question on the JDBC forum, post a link to that new question here, in this forum, and mark this question answered.
That way anyone seeing this question can followup with you in the other forum.
Thanks.

Similar Messages

Very slow painting while reading and writing doubles into file

for 15MB length file i = 7662080
for 50MB length file i = 12414368
Part of Code for writing into file follows like this:
try{
fos = new FileOutputStream("Angel.txt");
File f = new File("Angel.txt");
     if(f.length() >=4)
     f.delete();
fos = new FileOutputStream("Angel.txt");
dos = new DataOutputStream(new BufferedOutputStream(fos,1000000));
int x=0;
double y_last, y_new;
for(int j=0 ;j<i ;j++)
if(some condition)
y_new = ....;
try{
//previously in vectors
y_last = y_new;
vect.add(new Line2D.Double(x, y_last, x, y_new)_;
dos.writeDouble(y_new);
     }catch(Exception e){System.out.println(e);}
dos.close();
fos.close();
x++;
}catch(Exception excp){System.out.println(excp);}
part of code for reading from file follows like this:
public void paint(Graphics g)
try{
     double y1, y2 =0;
     Line2D.Double doub;
     raf = new RandomAccessFile("Angel.txt","r");
     dis = new DataInputStream(new BufferedInputStream(new FileInputStream(raf.getFD(),1000000)));
     raf.seek((rect.x*8));
     for (int i = 0/any value; (i < value as per choice); i++)
          g2.setStroke(new BasicStroke(0)); //2
          y1 = y2;
     y2 =dis.readDouble();
          doub=new Line2D.Double(i,y1,i,y2);
          g2.draw(doub);
dis.close();
     raf.close();
}catch(Exception excp){System.out.println(excp);}
I tried using Object Streams but NotSerializable Exception is thrown as Line2D.Double objects
are not serialized.
Any idea to make reading and writing into file specially from MB files faster is appreciated.

Why are you reading in the file in the paint method ?
Create your data once before painting.
I think you should explain what is your goal and what behavior you want.
Denis

Reading and Writing large Excel file using JExcel API

hi,
I am using JExcelAPI for reading and writing excel file. My problem is when I read file with 10000 records and 95 columns (file size about 14MB), I got out of memory error and application is crashed. Can anyone tell me is there any way that I can read large file using JExcelAPI throug streams or in any other way. Jakarta POI is also showing this behaviour.
Thanks and advance

Sorry when out of memory error is occurred no stack trace is printed as application is crashed. But I will quote some lines taken from JProfiler where this problem is occurred:
          reader = new FileInputStream(new File(filePath));
          workbook = Workbook.getWorkbook(reader);
          *sheeet = workbook.getSheet(0);* // here out of memory error is occured
           JProfiler tree:
jxl.Workbook.getWorkBook
      jxl.read.biff.File
             jxl.read.biff.CompoundFile.getStream
                   jxl.read.biff.CompoundFile.getBigBlockStream Thanks

Quickest method for reading and writing files

Hi
I need help regarding file operations.(Reading and Writing). Currently I am using BufferedReader and BufferedWriter to read and write files. But the files (XML) are very huge files(from 30 -50 mb). This is slowing the application to a great extent. Is there any other approach to perform the above mentioned operations on XML files in a fast manner.
Thank You
Mansoor.

Hi
Can u let me know how to use the java.nio pavkage for primitve data types(int,float..., boolean). I have tried it but found no success.
Thank You
Mansoor

Reading and writing a ByteArray

This question was posted in response to the following article: http://help.adobe.com/en_US/as3/dev/WS5b3ccc516d4fbf351e63e3d118666ade46-7d54.html

I believe that the example of Reading and Writing Objects needs a rewrite
First, the attempt to use comments to interleave a Flex-ified example with a Flash example of the uses of the ByteArray Class is a shortcut that ultimately fails. Take the time to create two, separately valid examples for the two different frameworks.
Second, the interlinea comments show a rather poor example of how to write clean code, and are entirely confusing to the person you you trying to help learn how to use your library. Look at the declaration of a variable called outFile:File, a comment about some function that is to be declared somewhere with the same name, outFile(), and indeed, if this code were to compile -- which I am sure it could not -- the compiler would see that you try to invoke an outFile() function. No one gets through an interview who writes code that way.
Third, I am not sure you understand how the readBytes and writeBytes methods work. In the upper example, where an order is being written to the file system, the writeBytes method uses data.length correctly. At the point in time that it is used, the XML object will have been written into the bytes:ByteArray, so it will have a proper length.
On the other hand, in the second half of the example, where the readBytes method similarly passes data.length, the reader has to be very lucky to note that at that point in time data.length will have a value of zero. The inBytes:ByteArray was only instantiated, not populated with anything, as, indeed it should not be, at that point in the logic flow. Most importantly, when the program does need to readBytes, there is no way for the program to know in advance what the length of the byte stream coming in from the file system will be. Your example would be more correct, and could make a useful point in showing that readBytes (data) is the correct way to pass the arguments because the default behavior will then be used and the default behavior will be to make data whatever length it needs to be in order to hold the content from the file system. The program can know the value of data.length after the read has taken place, but not before.

Problem on reading and writing from from a *.txt file

I get Problem on reading and writing from from a *.txt file. The following is the read() method...
The software said the DataInputStream is depreciated. Can anyone help me please?
public void read()
    File file = new File("C://Documents and Settings//Charles//My Documents//Brunel//EE2065//Assignment and Lab//Assignment 4 and Lab 4//data.txt");
    FileInputStream in = null;
    String str = "";
    try
      in = new BufferedReader(file);
      //in = new FileInputStream(file);
      for(;;)
        str = new BufferedReader(in).readLine();
        //str = new DataInputStream(in).readLine();
        if(str == null)
          break;
        System.out.print(str);
    in.close();
    catch(IOException e)
        System.err.println("execution error: " +e);
}

Thank you for your reply. I have made some change. However, there is an incompetable type found error.
in = new BufferedReader(new InputStreamReader(in));The following are all of the code.
public void read()
    File file = new File("C://Documents and Settings//Charles//My Documents//Brunel//EE2065//Assignment and Lab//Assignment 4 and Lab 4//data.txt");
    FileInputStream in = null;
    //BufferedReader in = null;
    String str = "";
    try
      in = new BufferedReader(new InputStreamReader(in));
      //in = new FileInputStream(file);
      for(;;)
        BufferedReader Bstr = new BufferedReader(new InputStreamReader(in));
        //str = new BufferedReader(in).readLine();
        //str = new DataInputStream(in).readLine();
        if(str == null)
          break;
        System.out.print(str);
    in.close();
    catch(IOException e)
        System.err.println("execution error: " +e);

File importer detected an inconsistency in the file stucture of (file name). Reading and writing this file's metadata (XMP) has been disabled.

I have duplicated a project to work on another computer. The project opens fine but when I import new footage/audio files I get this message. "File importer detected an inconsistency in the file stucture of (file name). Reading and writing this file's metadata (XMP) has been disabled." Then I can't play my timeline and Premier Pro crashes. I have to force quit and restart my computer to continue working. What does this error really mean? How do you rectify?
Our workflow requires that we duplicate projects to make updates because we are frequently revising but need to keep original project unchanged and intact.

I have a similar issue and message , but occurs when I import AVI clips from OnLocation CS4 to Premier Pro CS4.

I updated Itunes today to the latest version. Windows 7 64bit. None of my drivers work and get an error when itunes starts, about registry setting for reading and writing dvds and cds missing. Anyone else have the same issue. I downloaded itunes again, re

I updated Itunes today to the latest version. Windows 7 64bit. None of my drivers work and get an error when itunes starts, about registry setting for reading and writing dvds and cds missing. Anyone else have the same issue. I downloaded itunes again, reinstalled still have same issue.

I'd start with the following document, with one modification. At step 12 after typing GEARAspiWDM press the Enter/Return key once prior to clicking OK. (Pressing Return adds a carriage return in the field and is important.)
iTunes for Windows: "Registry settings" warning when opening iTunes

IPhone4 how through the USB mobile phone file reading and writing?

iPhone4 how through the USB mobile phone file reading and writing?

No idea what you are asking.
Please explain

Switcher - Reading and Writing External IEEE HDD's

Greetings,
i'm trying to read and write with my external ieee drive on windows and the mac pro. it's format is ntfs, and is read only on the mac pro.
i am admin and can't seem to change the permissions to read / write.
the over all idea is i need to go back and forth between operating systems reading and writing from both sides of the house.
here's a snapshot.
http://briankross.com/images/500GBIEEEHDD.jpg
Mac Pro Mac OS X (10.4.8)

Unfortunately, no. In order to convert the drive to FAT32 it must be re-partitioned which is a destructive process.
Why reward points?(Quoted from Discussions Terms of Use.)
The reward system helps to increase community participation. When a community member gives you (or another member) a reward for providing helpful advice or a solution to their question, your accumulated points will increase your status level within the community.
Members may reward you with 5 points if they deem that your reply is helpful and 10 points if you post a solution to their issue. Likewise, when you mark a reply as Helpful or Solved in your own created topic, you will be awarding the respondent with the same point values.

ImageIo: reading and writing a jpeg corrupts the image

Sometimes, ImageIo manages to read an image wrong. If I simply read() and write() the
image without any transformations, the image writtten will have its colors all wrong. The
output image format here doesn't matter.
An example image is: http://bram.name/~brama/pics/test.jpg
And output after reading and writing it: http://bram.name/~brama/pics/test_output.jpg
Is the image io lib to blame, or is it a broken image? The image displays fine in other programs.
Message was edited by:
Bram_A

Use this code to load the image.
From my experience it is much more stable and also faster.
My guess is that the problem is related to the ICC profile embeded in the JPEG.
JPEGImageDecoder decoder = JPEGCodec.createJPEGDecoder(new URL("http://www.stack.nl/~brama/pics/imageio.jpg").openStream());
BufferedImage image = decoder.decodeAsBufferedImage();

Lightweight library for reading and writing mp3-tags

Hi,
I'm currently evaluating libraries for reading and writing mp3-tags. All the projects I found so far where not maintained for a couple of years.
My question:
Is there currently a library, which one could call standard for this purpose
Cheers
Jonny

jonnybecker wrote:
I'm currently evaluating libraries for reading and writing mp3-tags. Don't you rather mean ID3 tags?
All the projects I found so far where not maintained for a couple of years.It may either be dead, or it may be so finished and free of bugs that no maintenance is needed anymore.
If you're occurring problems with it and if it is open source you can always consider taking a fork for own development.
Is there currently a library, which one could call standard for this purposeNo one comes to mind.

BLOB column inserts are slow

Hi All,
Have a table with 7 columns where 4 columns are of Varchar2 type, 2 columns are of NUMBER type and 1 column is of type BLOB.
Myself inserting the values to the table from JAVA program. Insertion to VARCHAR2 and NUMBER type columns are very much fast. But insertion to BLOB column is dead slow(data to BLOB column values are about 10KB).
Please help me in this regard to insert BLOB values fastly.
Regards/Sreekeshava S

Sreekeshava S wrote:
Running JAVA program in the same server as that of DB. Connecting how? IPC? TCP? Dedicated server? Shared server?
Calling Oracle how? Doing a SQL statement prepare per insert? Reusing the SQL cursor handle? Binding variables?
And inserting 250 records/ sec(during peak load and 50 records/sec during normal load). where each record is having size of 10K(Blob column size).And what is slow? You have NOT yet provided ANY evidence that points to the actual INSERT being slow.
As I have already explained, there are a number of layers from client to server - and any, or all of these, could be contributing to the problem.
Use your web browser and look up what instrumentation is. Apply it. Instrument your code. On the client. On the server. So you have evidence (call stats and metrics) to use to determine what and where the performance problem is. And not have to guess - and like most developers point your finger at the database in the false belief that your client code, client design, and client usage of the database, are all perfect.

Moving table columns is very slow

In my application moving table columns is very slow, with the rendering of table cells dragging behind the rendering of the table cell (using Java 1.5). I use a custom table cell renderer that subclasses a JPanel and contains a JTextField.
I checked I was overriding all the methods that DefaultTableCellRenderer does for performance reasons and found I wasnt overiding validate() and invalidate(), over-riding these the table column performance is now ok, but the cells are now not rendered correctly.
I also put a System.out.println into getTableRendererComponent() and found that even if I only move a column slightly the getTableRendererComponent() is called for every cell displayed rather than just the cells within the column moved/columns either side of it which is what i expect.
Any ideas what i should do. My own vague idea is to put some code into validate() and invalidate() to only call super implemntaions when really need to, but dont know how to work this out.

Hi, thanks for your help - ive created a full test case, code below. I have reworked the code so I dont need to do a removeAll() but the issues remaining are:
1. If I override validate() with an empty method nothing is displayed in the grid
2. The renderer is alot slower than the using default renderer even though ive removed stuff like borders.
The code below has validate without the empty method.
(My real renderer is a bit more complicated, and I do require to use a JPanel)
import com.jthink.jaikoz.settings.LAF;
import com.jthink.jaikoz.table.ID3Cell;
import javax.swing.*;
import javax.swing.table.*;
import java.util.Vector;
import java.awt.*;
public class SlowColumnMoveTest
    public static void main(String args[])
        new SlowColumnMoveTest();
    public SlowColumnMoveTest()
        Vector colNames = new Vector();
        colNames.add("col0");
        colNames.add("col1");
        colNames.add("col2");
        colNames.add("col3");
        colNames.add("col4");
        colNames.add("col5");
        colNames.add("col6");
        colNames.add("col7");
        colNames.add("col8");
        colNames.add("col9");
        Vector data = new Vector();
        for (int i = 0; i < 500; i++)
            Vector v = new Vector();
            v.add(i);
            for (int j = 1; j < 10; j++)
                v.add(String.valueOf((i + 1) * (j + 1)));
            data.add(v);
        JTable table = new JTable(new DefaultTableModel(data, colNames));
        table.getColumnModel().setColumnSelectionAllowed(true);
        table.setDefaultRenderer(Object.class, new SlowRenderer());
        JFrame frame = new JFrame("SlowColumnMoveTest");
        frame.setDefaultCloseOperation(JFrame.DISPOSE_ON_CLOSE);
        JScrollPane scrollPane = new JScrollPane(table);
        frame.add(scrollPane);
        frame.pack();
        frame.setVisible(true);
    static class SlowRenderer extends JPanel implements TableCellRenderer
        private static String PROPERTY_TABLE_BGCOLOUR = "Table.background";
        private static String PROPERTY_TABLE_FGCOLOUR = "Table.foreground";
        private static String PROPERTY_TABLE_SELECTION_BGCOLOUR = "Table.selectionBackground";
        private static String PROPERTY_TABLE_SELECTION_FGCOLOUR = "Table.selectionForeground";
        protected static Color tableBGColour = null;
        protected static Color tableFGColour = null;
        protected static Color selectionBGColour = null;
        protected static Color selectionFGColour = null;
        static
            tableBGColour = UIManager.getColor(PROPERTY_TABLE_BGCOLOUR);
            tableFGColour = UIManager.getColor(PROPERTY_TABLE_FGCOLOUR);
            selectionBGColour = UIManager.getColor(PROPERTY_TABLE_SELECTION_BGCOLOUR);
            selectionFGColour = UIManager.getColor(PROPERTY_TABLE_SELECTION_FGCOLOUR);
        protected JTextField text;
        public SlowRenderer()
            text = new JTextField();
            text.setBorder(null);
            text.setOpaque(false);
            text.setForeground(tableFGColour);
            setOpaque(true);
            this.setLayout(new BoxLayout(this, BoxLayout.X_AXIS));
            this.add(text);
        public Component getTableCellRendererComponent(final JTable table,
                                                       final Object value,
                                                       final boolean isSelected,
                                                       final boolean hasFocus,
                                                       final int row,
                                                       final int column)
            //System.out.println("rendering:col:"+column+":row:"+row);
            text.setVisible(true);
            text.setText(value.toString());
            if (isSelected)
                text.setForeground(selectionFGColour);
                this.setBackground(selectionBGColour);
            else
                text.setForeground(tableFGColour);
                this.setBackground(tableBGColour);
            return this;
        @Override
        public final boolean isOpaque()
            return true;
        @Override
        public void invalidate()
            //System.out.println("invvalidate");
            //super.invalidate();
        @Override
        public void validate()
            //System.out.println("validate");
            super.validate();
        @Override
        public void revalidate()
        @Override
        public void repaint(long tm, int x, int y, int width, int height)
        @Override
        public void repaint(Rectangle r)
        @Override
        public void repaint()
        @Override
        protected final void firePropertyChange(final String propertyName, final Object oldValue, final Object newValue)
        @Override
        public final void firePropertyChange(final String propertyName, final boolean oldValue, final boolean newValue)
        private static final class UIResource
            extends DefaultTableCellRenderer
            implements javax.swing.plaf.UIResource
}

Just got iPhone4 and installing apps on 3G is very slow. Is there something I can do to speed this up?

Just got iPhone4 and installing apps on 3G is very slow. Is there something I can do to speed this up?

Use a WiFi connection instead... or download them using iTunes on your computer, then sync them to the phone.

Reading and Writing to blob column is very slow

Similar Messages

Maybe you are looking for