JDBC, SQL*Net wait interface, performance degradation on 10g vs. 9i

Question

Hi All,
I came across performance issue that I think results from mis-configuration of something between Oracle and JDBC. The logic of my system executes 12 threads in java. Each thread performs simple 'select a,b,c...f from table_xyz' on different tables. (so I have 12 different tables with cardinality from 3 to 48 millions and one working thread per table).
In each thread I'm creating result set that is explicitly marked as forward_only, transaction is set read only, fetch size is set to 100000 records. Java logic processes records in standard while(rs.next()) {...} loop.
I'm experiencing performance degradation between execution on Oracle 9i and Oracle 10g of the same java code, on the same machine, on the same data. The difference is enormous, 9i execution takes 26 hours while 10g execution takes 39 hours.
I have collected statspack for 9i and awr report for 10g. Below I've enclosed top wait events for 9i and 10g
===== 9i ===================
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
db file sequential read 22,939,988 0 6,240 0 0.7
control file parallel write 6,152 0 296 48 0.0
SQL*Net more data to client 2,877,154 0 280 0 0.1
db file scattered read 26,842 0 91 3 0.0
log file parallel write 3,528 0 83 23 0.0
latch free 94,845 0 50 1 0.0
process startup 93 0 5 50 0.0
log file sync 34 0 2 46 0.0
log file switch completion 2 0 0 215 0.0
db file single write 9 0 0 33 0.0
control file sequential read 4,912 0 0 0 0.0
wait list latch free 15 0 0 12 0.0
LGWR wait for redo copy 84 0 0 1 0.0
log file single write 2 0 0 18 0.0
async disk IO 263 0 0 0 0.0
direct path read 2,058 0 0 0 0.0
slave TJ process wait 1 1 0 12 0.0
===== 10g ==================
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
db file scattered read 268,314 .0 2,776 10 0.0
SQL*Net message to client 278,082,276 .0 813 0 7.1
io done 20,715 .0 457 22 0.0
control file parallel write 10,971 .0 336 31 0.0
db file parallel write 15,904 .0 294 18 0.0
db file sequential read 66,266 .0 257 4 0.0
log file parallel write 3,510 .0 145 41 0.0
SQL*Net more data to client 2,221,521 .0 102 0 0.1
SGA: allocation forcing comp 2,489 99.9 27 11 0.0
log file sync 564 .0 23 41 0.0
os thread startup 176 4.0 19 106 0.0
latch: shared pool 372 .0 11 29 0.0
latch: library cache 537 .0 5 10 0.0
rdbms ipc reply 57 .0 3 49 0.0
log file switch completion 5 40.0 3 552 0.0
latch free 4,141 .0 2 0 0.0
I put full blame for the slowdown on SQL*Net message to client wait event. All I could find about this event is that it is a network related problem. I assume it would be true if database and client were on different machines.. However in my case they are on them very same machine.
I'd be very very grateful if someone could point me in the right direction, i.e. give a hint what statistics should I analyze further? what might cause this event to appear? why probable cause (that is said be outside db) affects only 10g instance?
Thanks in advance,
Rafi.

Answer 1

Hi Steven,
Thanks for the input. It's a fact that I did not gather statistics on my tables. My understanding is that statistics are useful for queries more complex than simple select * from table_xxx. In my case tables don't have indexes. There's no filtering condition as well. Full table scan is what I actually want as all software logic is inside the java code.
Explain plans are as follows:
======= 10g ================================
PLAN_TABLE_OUTPUT
Plan hash value: 1141003974
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | | 1 | 259 | 2 (0)| 00:00:01 |
| 1 | TABLE ACCESS FULL| xxx | 1 | 259 | 2 (0)| 00:00:01 |
In sqlplus I get:
SQL> set autotrace traceonly explain statistics;
SQL> select * from xxx;
36184384 rows selected.
Elapsed: 00:38:44.35
Execution Plan
Plan hash value: 1141003974
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | | 1 | 259 | 2 (0)| 00:00:01 |
| 1 | TABLE ACCESS FULL| xxx | 1 | 259 | 2 (0)| 00:00:01 |
Statistics
1 recursive calls
0 db block gets
3339240 consistent gets
981517 physical reads
116 redo size
26535700 bytes received via SQL*Net from client
2412294 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
36184384 rows processed
======= 9i =================================
PLAN_TABLE_OUTPUT
| Id | Operation | Name | Rows | Bytes | Cost |
| 0 | SELECT STATEMENT | | | | |
| 1 | TABLE ACCESS FULL | xxx | | | |
Note: rule based optimization
In sqlplus I get:
SQL> set autotrace traceonly explain statistics;
SQL> select * from xxx;
36184384 rows selected.
Elapsed: 00:17:43.06
Execution Plan
0 SELECT STATEMENT Optimizer=CHOOSE
1 0 TABLE ACCESS (FULL) OF 'xxx'
Statistics
0 recursive calls
1 db block gets
3306118 consistent gets
957515 physical reads
100 redo size
23659424 bytes sent via SQL*Net to client
26535867 bytes received via SQL*Net from client
2412294 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
36184384 rows processed
Thanks for pointing me difference in table scans. I infer that 9i is doing single-block full table scan (db file sequential read) while 10g is using multi-block full table scan (db file scattered read).
I now have theory that 9i is faster because sequential reads use continuous buffer space while scattered reads use discontinuous buffer space. Since I'm accessing data 'row by row' in jdbc 10g might have an overhead in providing data from discontinuous buffer space. This overhead shows itself as SQL*Net message to client wait. Is that making any sense?
Is there any way I could force 10g (i.e. with hint) to use sequential reads instead of scattered reads for full table scan?
I'll experiment with FTS tuning in 10g by enabling automatic multi-block reads tuning (i.e. db_file_multiblock_read_count=0 instead of 32 as it is now). I'll also check if response time improves after statistics are gathered.
Please advice if you have any other ideas.
Thanks & regards,
Rafi.