OutOfMemory error while executing sql query

Question

Hello!
My program gets multiple datas from database in every ten minutes, and stores them in memory for hundreds of users, requesting datas via web-interface simoultaneously.
I dont have access to change database structures, write stored procedures, etc, just read from db.
There is a table in database with lot of million rows, and sometimes when I try to execute a SELECT on this table it takes minutes to get back the result.
To avoid waiting for database server for a long time, I set querytimeout to 30 seconds.
If the server throws back the execution with Query Timed out Exception, I want to 'forget' this data, and 0 value is acceptable because of fast run is more important. So I put the boolean broken variable to check if there is any problem with db server.
The size of the used memory is about 150Mb if things going well, but I set the max heap to 512 MB, just in case anything happens.
I'm logging all threads stacktrace, and free/used/allocated memory size in every 5 seconds.(threadwatching.log) 2.appendix
Sometimes, not in every case (I dont know what is this depends on), when I get the next phase of refreshing cached datas (you can see it below), the process reaches the fiorst checkpoint (signed in code below), starts to execute the sql query, and never reaches the second checkpoint , but used memory growing 50-60 Mb-os in every 5 seconds, as I can see in threadwatching.log until it reaches the max memory and throws OutOfMemory error: java heap space.
I'm using DbConnectionBroker for connection pooling, SQLCommandBean for handling Statements, PreparedStatements, etc, and jTDS jdbc connector.
SQLCommandBean closes statements, resultsets, so these objects doesnt stays open.
I cant figured out what causes the memory leak, if someone have an idea, please help me.
1. Part of the cached data refreshing (DataFactory.createPCVPPMforSiemens()):
        PCVElement element = new PCVElement(m, ProcessControlView.PPM);
        String s = DateTime.getDate(interval.getStartDate());
        boolean broken=false;
        int value = 0;
        for (int j = 0; j < 48; j++) {
            try {
                if (!broken) {
                    d1 = DateTime.getDate(new Date(start + ((j + 1) * 600000)));
                    sqlBean = new SQLCommandBean();
                    conn = broker.getConnection();
                    sqlBean.setConnection(conn);
                    sqlBean.setQueryTimeOut(30);
                    System.out.println(DateTime.getDate(new Date())+" "+m.getName()+"   "+j);// first checkpoint
                    value = SiemensWorks.getPCVPPM(sqlBean, statId, s, d1);
                    System.out.println(DateTime.getDate(new Date())+" "+m.getName()+"   "+j);// second checkpoint
                } else value=0;
            } catch (Exception ex) {
                System.out.println("ERROR: DataFactory.createPCVPPMforSiemens 1 :" + ex.getMessage());
                ex.printStackTrace();
                value = 0;
                broken=true;
            } finally {
                try {
                    broker.freeConnection(conn);
                } catch (Exception ex) {}
            element.getAvgValues()[j] = value;
        }2. SiemensWorks.getPCVPPM()
    public static int getPCVPPM(SQLCommandBean sqlBean,int statID,String start,String end)
            throws SQLException, UnsupportedTypeException, NoSuchColumnException {
        sqlBean.setSqlValue(SiemensSQL.PCV_PPM);
        Vector values=new Vector();
        values.add(new StringValue(statID+""));
        values.add(new StringValue(start));
        values.add(new StringValue(end));
        sqlBean.setValues(values);
        Vector rows=sqlBean.executeQuery();
        if (rows==null || rows.size()==0) return 0;
        Row row=(Row)rows.firstElement();
        try {
            float ret=Float.parseFloat(row.getString(1));
            if (ret<=0) ret=0;
            return Math.round(ret);
        } catch (Exception ex) {
            return 0;
    }3. Part of Threadwatching.log
2006-10-13 16:46:56 Name: SMT Refreshing Threads
2006-10-13 16:46:56 Thread count: 4
2006-10-13 16:46:56 Active count: 4
2006-10-13 16:46:56 Active group count: 0
2006-10-13 16:46:56 Daemon: false
2006-10-13 16:46:56 Priority: 5
2006-10-13 16:46:57 Free memory: 192,228,944 bytes
2006-10-13 16:46:57 Max memory: 332,988,416 bytes
2006-10-13 16:46:57 Memory in use: 140,759,472 bytes
2006-10-13 16:46:57 ---------------------------------
2006-10-13 16:46:57 0. Name: CachedLayerTimer
2006-10-13 16:46:57 0. Id: 19
2006-10-13 16:46:57 0. Priority: 5
2006-10-13 16:46:57 0. Parent: SMT Refreshing Threads
2006-10-13 16:46:57 0. State: RUNNABLE
2006-10-13 16:46:57 0. Alive: true
2006-10-13 16:46:57 java.io.FileOutputStream.close0(Native Method)
2006-10-13 16:46:57 java.io.FileOutputStream.close(Unknown Source)
2006-10-13 16:46:57 sun.nio.cs.StreamEncoder$CharsetSE.implClose(Unknown Source)
2006-10-13 16:46:57 sun.nio.cs.StreamEncoder.close(Unknown Source)
2006-10-13 16:46:57 java.io.OutputStreamWriter.close(Unknown Source)
2006-10-13 16:46:57 xcompany.smtmonitor.chart.ChartCreator.createChart(ChartCreator.java:663)
2006-10-13 16:46:57 xcompany.smtmonitor.chart.ChartCreator.create(ChartCreator.java:441)
2006-10-13 16:46:57 xcompany.smtmonitor.CachedLayerRefreshenerTask.run(CachedLayerRefreshenerTask.java:463)
2006-10-13 16:46:57 java.util.TimerThread.mainLoop(Unknown Source)
2006-10-13 16:46:57 java.util.TimerThread.run(Unknown Source)
Software runs well until I get the DataFactory.createPCVPPMforSiemens function in my code ->
2006-10-13 16:47:01 Name: SMT Refreshing Threads
2006-10-13 16:47:01 Thread count: 4
2006-10-13 16:47:01 Active count: 4
2006-10-13 16:47:01 Active group count: 0
2006-10-13 16:47:01 Daemon: false
2006-10-13 16:47:01 Priority: 5
2006-10-13 16:47:02 Free memory: 189,253,304 bytes
2006-10-13 16:47:02 Max memory: 332,988,416 bytes
2006-10-13 16:47:02 Memory in use: 143,735,112 bytes
2006-10-13 16:47:02 ---------------------------------
2006-10-13 16:47:02 0. Name: CachedLayerTimer
2006-10-13 16:47:02 0. Id: 19
2006-10-13 16:47:02 0. Priority: 5
2006-10-13 16:47:02 0. Parent: SMT Refreshing Threads
2006-10-13 16:47:02 0. State: RUNNABLE
2006-10-13 16:47:02 0. Alive: true
2006-10-13 16:47:02 java.util.LinkedList$ListItr.previous(Unknown Source)
2006-10-13 16:47:02 net.sourceforge.jtds.util.TimerThread.setTimer(TimerThread.java:174)
2006-10-13 16:47:02 net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:3734)
2006-10-13 16:47:02 net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:997)
2006-10-13 16:47:02 net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:320)
2006-10-13 16:47:02 net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:667)
2006-10-13 16:47:02 xcompany.database.sql.SQLCommandBean.executeQuery(SQLCommandBean.java:91)
2006-10-13 16:47:02 xcompany.smtmonitor.data.SiemensWorks.getPCVPPM(SiemensWorks.java:409)
2006-10-13 16:47:02 xcompany.smtmonitor.data.DataFactory.createPCVPPMforSiemens(DataFactory.java:6103)
2006-10-13 16:47:02 xcompany.smtmonitor.data.DataFactory.refreshProcessControlView(DataFactory.java:5791)
2006-10-13 16:47:02 xcompany.smtmonitor.CachedLayerRefreshenerTask.run(CachedLayerRefreshenerTask.java:514)
2006-10-13 16:47:02 java.util.TimerThread.mainLoop(Unknown Source)
2006-10-13 16:47:02 java.util.TimerThread.run(Unknown Source)
2006-10-13 16:47:06 Name: SMT Refreshing Threads
2006-10-13 16:47:06 Thread count: 4
2006-10-13 16:47:06 Active count: 4
2006-10-13 16:47:06 Active group count: 0
2006-10-13 16:47:06 Daemon: false
2006-10-13 16:47:06 Priority: 5
2006-10-13 16:47:08 Free memory: 127,428,192 bytes
2006-10-13 16:47:08 Max memory: 332,988,416 bytes
2006-10-13 16:47:08 Memory in use: 205,560,224 bytes
2006-10-13 16:47:08 ---------------------------------
2006-10-13 16:47:08 0. Name: CachedLayerTimer
2006-10-13 16:47:08 0. Id: 19
2006-10-13 16:47:08 0. Priority: 5
2006-10-13 16:47:08 0. Parent: SMT Refreshing Threads
2006-10-13 16:47:08 0. State: RUNNABLE
2006-10-13 16:47:08 0. Alive: true
2006-10-13 16:47:08 java.util.LinkedList$ListItr.previous(Unknown Source)
2006-10-13 16:47:08 net.sourceforge.jtds.util.TimerThread.setTimer(TimerThread.java:174)
2006-10-13 16:47:08 net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:3734)
2006-10-13 16:47:08 net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:997)
2006-10-13 16:47:08 net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:320)
2006-10-13 16:47:08 net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:667)
2006-10-13 16:47:08 xcompany.database.sql.SQLCommandBean.executeQuery(SQLCommandBean.java:91)
2006-10-13 16:47:08 xcompany.smtmonitor.data.SiemensWorks.getPCVPPM(SiemensWorks.java:409)
2006-10-13 16:47:08 xcompany.smtmonitor.data.DataFactory.createPCVPPMforSiemens(DataFactory.java:6103)
2006-10-13 16:47:08 xcompany.smtmonitor.data.DataFactory.refreshProcessControlView(DataFactory.java:5791)
2006-10-13 16:47:08 xcompany.smtmonitor.CachedLayerRefreshenerTask.run(CachedLayerRefreshenerTask.java:514)
2006-10-13 16:47:08 java.util.TimerThread.mainLoop(Unknown Source)
2006-10-13 16:47:08 java.util.TimerThread.run(Unknown Source)
2006-10-13 16:47:12 Name: SMT Refreshing Threads
2006-10-13 16:47:12 Thread count: 4
2006-10-13 16:47:12 Active count: 4
2006-10-13 16:47:12 Active group count: 0
2006-10-13 16:47:12 Daemon: false
2006-10-13 16:47:12 Priority: 5
2006-10-13 16:47:15 Free memory: 66,760,208 bytes
2006-10-13 16:47:15 Max memory: 332,988,416 bytes
2006-10-13 16:47:15 Memory in use: 266,228,208 bytes
2006-10-13 16:47:15 ---------------------------------
2006-10-13 16:47:15 0. Name: CachedLayerTimer
2006-10-13 16:47:15 0. Id: 19
2006-10-13 16:47:15 0. Priority: 5
2006-10-13 16:47:15 0. Parent: SMT Refreshing Threads
2006-10-13 16:47:15 0. State: RUNNABLE
2006-10-13 16:47:15 0. Alive: true
2006-10-13 16:47:15 java.util.LinkedList.addBefore(Unknown Source)
2006-10-13 16:47:15 java.util.LinkedList.access$300(Unknown Source)
2006-10-13 16:47:15 java.util.LinkedList$ListItr.add(Unknown Source)
2006-10-13 16:47:15 net.sourceforge.jtds.util.TimerThread.setTimer(TimerThread.java:175)
2006-10-13 16:47:15 net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:3734)
2006-10-13 16:47:15 net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:997)
2006-10-13 16:47:15 net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:320)
2006-10-13 16:47:15 net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:667)
2006-10-13 16:47:15 xcompany.database.sql.SQLCommandBean.executeQuery(SQLCommandBean.java:91)
2006-10-13 16:47:15 xcompany.smtmonitor.data.SiemensWorks.getPCVPPM(SiemensWorks.java:409)
2006-10-13 16:47:15 xcompany.smtmonitor.data.DataFactory.createPCVPPMforSiemens(DataFactory.java:6103)
2006-10-13 16:47:15 xcompany.smtmonitor.data.DataFactory.refreshProcessControlView(DataFactory.java:5791)
2006-10-13 16:47:15 xcompany.smtmonitor.CachedLayerRefreshenerTask.run(CachedLayerRefreshenerTask.java:514)
2006-10-13 16:47:15 java.util.TimerThread.mainLoop(Unknown Source)
2006-10-13 16:47:15 java.util.TimerThread.run(Unknown Source)
2006-10-13 16:47:17 Name: SMT Refreshing Threads
2006-10-13 16:47:17 Thread count: 4
2006-10-13 16:47:17 Active count: 4
2006-10-13 16:47:17 Active group count: 0
2006-10-13 16:47:17 Daemon: false
2006-10-13 16:47:17 Priority: 5
2006-10-13 16:47:20 Free memory: 23,232,496 bytes
2006-10-13 16:47:20 Max memory: 332,988,416 bytes
2006-10-13 16:47:20 Memory in use: 309,755,920 bytes
2006-10-13 16:47:20 ---------------------------------
2006-10-13 16:47:20 0. Name: CachedLayerTimer
2006-10-13 16:47:20 0. Id: 19
2006-10-13 16:47:20 0. Priority: 5
2006-10-13 16:47:20 0. Parent: SMT Refreshing Threads
2006-10-13 16:47:20 0. State: RUNNABLE
2006-10-13 16:47:20 0. Alive: true
2006-10-13 16:47:20 net.sourceforge.jtds.util.TimerThread.setTimer(TimerThread.java:171)
2006-10-13 16:47:20 net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:3734)
2006-10-13 16:47:20 net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:997)
2006-10-13 16:47:20 net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:320)
2006-10-13 16:47:20 net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:667)
2006-10-13 16:47:20 xcompany.database.sql.SQLCommandBean.executeQuery(SQLCommandBean.java:91)
2006-10-13 16:47:20 xcompany.smtmonitor.data.SiemensWorks.getPCVPPM(SiemensWorks.java:409)
2006-10-13 16:47:20 xcompany.smtmonitor.data.DataFactory.createPCVPPMforSiemens(DataFactory.java:6103)
2006-10-13 16:47:20 xcompany.smtmonitor.data.DataFactory.refreshProcessControlView(DataFactory.java:5791)
2006-10-13 16:47:20 xcompany.smtmonitor.CachedLayerRefreshenerTask.run(CachedLayerRefreshenerTask.java:514)
2006-10-13 16:47:20 java.util.TimerThread.mainLoop(Unknown Source)
2006-10-13 16:47:20 java.util.TimerThread.run(Unknown Source)
2006-10-13 16:47:23 Name: SMT Refreshing Threads
2006-10-13 16:47:23 Thread count: 4
2006-10-13 16:47:23 Active count: 4
2006-10-13 16:47:23 Active group count: 0
2006-10-13 16:47:23 Daemon: false
2006-10-13 16:47:23 Priority: 5
2006-10-13 16:47:26 Free memory: 4,907,336 bytes
2006-10-13 16:47:26 Max memory: 332,988,416 bytes
2006-10-13 16:47:26 Memory in use: 328,083,768 bytes
2006-10-13 16:47:26 ---------------------------------
2006-10-13 16:47:26 0. Name: CachedLayerTimer
2006-10-13 16:47:26 0. Id: 19
2006-10-13 16:47:26 0. Priority: 5
2006-10-13 16:47:26 0. Parent: SMT Refreshing Threads
2006-10-13 16:47:26 0. State: RUNNABLE
2006-10-13 16:47:26 0. Alive: true
2006-10-13 16:47:26 java.util.LinkedList.addBefore(Unknown Source)
2006-10-13 16:47:26 java.util.LinkedList.access$300(Unknown Source)
2006-10-13 16:47:26 java.util.LinkedList$ListItr.add(Unknown Source)
2006-10-13 16:47:26 net.sourceforge.jtds.util.TimerThread.setTimer(TimerThread.java:175)
2006-10-13 16:47:26 net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:3734)
2006-10-13 16:47:26 net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:997)
2006-10-13 16:47:26 net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:320)
2006-10-13 16:47:26 net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:667)
2006-10-13 16:47:26 xcompany.database.sql.SQLCommandBean.executeQuery(SQLCommandBean.java:91)
2006-10-13 16:47:26 xcompany.smtmonitor.data.SiemensWorks.getPCVPPM(SiemensWorks.java:409)
2006-10-13 16:47:26 xcompany.smtmonitor.data.DataFactory.createPCVPPMforSiemens(DataFactory.java:6103)
2006-10-13 16:47:26 xcompany.smtmonitor.data.DataFactory.refreshProcessControlView(DataFactory.java:5791)
2006-10-13 16:47:26 xcompany.smtmonitor.CachedLayerRefreshenerTask.run(CachedLayerRefreshenerTask.java:514)
2006-10-13 16:47:26 java.util.TimerThread.mainLoop(Unknown Source)
2006-10-13 16:47:26 java.util.TimerThread.run(Unknown Source)
2006-10-13 16:47:35 Name: SMT Refreshing Threads
2006-10-13 16:47:37 Thread count: 4
2006-10-13 16:47:38 Active count: 4
2006-10-13 16:47:38 Active group count: 0
2006-10-13 16:47:38 Daemon: false
2006-10-13 16:47:38 Priority: 5
2006-10-13 16:47:42 Free memory: 35,316,120 bytes
2006-10-13 16:47:42 Max memory: 332,988,416 bytes
2006-10-13 16:47:42 Memory in use: 297,672,296 bytes
2006-10-13 16:47:42 ---------------------------------
2006-10-13 16:47:42 0. Name: CachedLayerTimer
2006-10-13 16:47:42 0. Id: 19
2006-10-13 16:47:42 0. Priority: 5
2006-10-13 16:47:42 0. Parent: SMT Refreshing Threads
2006-10-13 16:47:42 0. State: TIMED_WAITING
2006-10-13 16:47:42 0. Alive: true
2006-10-13 16:47:42 java.lang.Object.wait(Native Method)
2006-10-13 16:47:42 java.util.TimerThread.mainLoop(Unknown Source)
2006-10-13 16:47:42 java.util.TimerThread.run(Unknown Source)
4. Tomcat default logging file:
2006-10-13 16:47:36 ERROR CachedLayerRefreshenerTask: external error: Java heap space
5. DbConnectionBroker (connection pooling) logging file:
Handing out connection 1 --> 10/13/2006 04:47:01 PM
Handing out connection 0 --> 10/13/2006 04:47:01 PM
Handing out connection 1 --> 10/13/2006 04:47:01 PM
Handing out connection 0 --> 10/13/2006 04:47:02 PM
Warning. Connection 0 in use for 3141 ms
Warning. Connection 0 in use for 24891 ms
----> Error: Could not free connection!!!
I would appreciate for any help.

Answer 1

What does your query bring back from this table?This is the query:
SELECT case sum(c.picked) when 0 then 0 else
((sum(c.picked)-(sum(c.picked)-(sum(c.vacuum)+sum(c.id
ent))))*cast((1000000/cast(sum(c.picked) as float))
as bigint)) end as PPM
FROM sip_comp c
LEFT JOIN sip_pcb pc ON pc.id=c.pcbid
LEFT JOIN sip_period p on p.id=pc.periodid
WHERE p.stationid=? AND pc.time BETWEEN ? AND ?Has anybody who knows SQL tried EXPLAIN PLAN to optimize this table? You're joining on a table with a million rows and you're wondering why the performance is poor?
What is the index situation with these tables?
.> When I execute it from query manager, it takes from 1
to 60 secs depend on servers availability. So how will that be any different for JDBC and Java?
..> You're right. Thats why I am here.
What I mean by that is we can't read minds, either. You need to get some hard data to tell you where the bottleneck is. Asking at a forum won't help.
But tell me, if the java process enters to this query
execution, and doesnt quit until OOM thrown, how can
be the problem in caching?I was guessing about caching, because I didn't know what the query was.
You expect a lot.
.> No.
Then how do you ever expect to solve this?
I tried YourKit Profiler at home, where I'm
developing software, but this OOM never thrown here,
even if I have the same database size.Then you aren't replicating the problem. You have to run it on the system that has the problem if you're going to solve it.
YourKit isn't an industry leader. How well do you know how to use it?
It just happened at the company where the system
runs, and I cannot run this profiler there because
the PC where my tomcat runs dramatically slowed.You have to run something to figure out what the problem is. What about Log4J, some trace logging statements and a batch job to harvest the log?
Bottom line: you've got to be a scientist and get some real data. We can theorize all we want here, but that won't get you to a solution.
%