Same query on different RAC instance huge execution time difference

Question

we have 3 nodes oracle 10gR2 RAC based on ASM. The 3rd node is the latest one we added into the cluster. All instances on 3 nodes using the same system parameters and same OS and same hardware
i found a weird problem, a same query i executed on node1 is way more faster than on node3 (1.5mins VS. 18mins). Then i did trace, surprising found the fetch time for node3 is longer than node1. Both nodes are at very low load. here is part of tracing for both nodes:
node1:
call count cpu elapsed disk query current rows
Parse 1 3.40 3.47 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 2 19.13 97.04 69938 269374 0 7
total 4 22.54 100.51 69938 269374 0 7
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 5
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
row cache lock 132 0.00 0.03
library cache lock 13 0.00 0.00
library cache pin 9 0.00 0.00
rdbms ipc reply 6 0.00 0.00
latch: shared pool 1 0.00 0.00
SQL*Net message to client 2 0.00 0.00
gc current block 2-way 2061 0.00 0.67
gc cr multi block request 76610 0.18 7.71
gc current block 3-way 801 0.00 0.33
gc cr block 3-way 3 0.00 0.00
gc cr block 2-way 30 0.00 0.01
db file scattered read 1167 0.02 3.66
gc cr grant 2-way 24739 0.00 4.81
db file sequential read 37363 0.12 57.35
latch: KCL gc element parent latch 7 0.00 0.00
db file parallel read 1465 0.05 5.86
latch: cache buffers lru chain 1 0.00 0.00
gc current grant 2-way 1 0.00 0.00
gc cr block busy 10 0.17 0.37
gc cr disk read 1 0.00 0.00
SQL*Net more data to client 5 0.00 0.00
SQL*Net message from client 2 436.40 436.40
node3
call count cpu elapsed disk query current rows
Parse 1 2.58 2.64 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 2 11.35 299.22 37084 269364 0 7
total 4 13.94 301.86 37084 269364 0 7
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 5
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
library cache lock 3 0.00 0.00
row cache lock 10 0.00 0.00
SQL*Net message to client 2 0.00 0.00
gc cr grant 2-way 23126 0.01 5.01
db file sequential read 37080 1.30 277.16
gc cr multi block request 1 0.00 0.00
db file scattered read 1 0.31 0.31
read by other session 721 0.03 2.12
gc buffer busy 1248 0.03 4.78
latch: cache buffers chains 5 0.00 0.00
gc current block 3-way 332 0.00 0.17
gc current block 2-way 834 0.00 0.30
latch: KCL gc element parent latch 1 0.00 0.00
gc current grant 2-way 1 0.00 0.00
SQL*Net more data to client 5 0.00 0.00
SQL*Net message from client 2 76.12 76.12
And all the execution plan are the same.
Could anybody please tell me why this happens?
Thanks!

Answer 1

To me, it looks like node3 is not accessing storage as fast as node 1 is. Look at this:
db file sequential read 37363 0.12 57.35
db file sequential read 37080 1.30 277.16
both cases have roughly the same amount of reads/waits but the time waited is 5 times more in the second case. So this could indicate a problem, maybe something with the HBA or SAN path or something like that.
Bjoern