Vastly different performance of identically configured databases...

Question

I’m using Oracle Text to retrieve data from a large table (5 million rows or so). The index is created simply:
CREATE INDEX ALT_DEV_IDX ON devtable(searchfield) INDEXTYPE IS CTXSYS.CONTEXT;
…and then I do my best to make sure the thing is loaded into memory and stays there:
alter table devtable cache;
alter table devtable storage (buffer_pool keep);
alter table dr$alt_dev_idx$I storage (buffer_pool keep);
alter table dr$alt_dev_idx$R storage (buffer_pool keep);
alter index dr$alt_dev_idx$X storage (buffer_pool keep);
alter table DR$ALT_dev_IDX$R storage (buffer_pool keep) modify lob (data) (storage (buffer_pool keep));
SELECT /*+ FULL(ITAB) */ SUM(TOKEN_COUNT), SUM(LENGTH(TOKEN_INFO)) FROM DR$ALT_dev_IDX$I ITAB;
SELECT /*+ INDEX(ITAB) */ SUM(LENGTH(TOKEN_TEXT)) FROM DR$ALT_dev_IDX$I ITAB;
SELECT /*+ FULL(BTAB) */ SUM(ROWNUM) FROM devtable WHERE ROWNUM >= 900000;
No other tables are in use when testing. We have over 90GB of free memory on our servers, so memory is not constrained. However, I/O is not as good as it should be, because I’m limited to just two sets of mirrored paired hard disks.
On a spare Linux box running 64-bit Centos 5, with much worse I/O configuration (basically, just one mirrored pair), a lot less free RAM and a CPU that’s about three years older (and slower!) than the ones in the production servers, I set the following init.ora parameters (and nothing else):
*.audit_file_dest='/u01/app/oracle/admin/devdb/adump'
*.background_dump_dest='/u01/app/oracle/admin/devdb/bdump'
*.compatible='10.2.0.1.0'
*.control_files='/data/oradata/devdb/control01.ctl','/data/oradata/devdb/control02.ctl','/data/oradata/devdb/control03.ctl'
*.core_dump_dest='/u01/app/oracle/admin/devdb/cdump'
*.db_block_size=16384
*.db_domain=''
*.db_file_multiblock_read_count=16
*.db_name='dbsd'
*.job_queue_processes=10
*.open_cursors=300
*.pga_aggregate_target=2280652800
*.processes=700
*.remote_login_passwordfile='EXCLUSIVE'
*.sessions=775
*.sga_target=9000M
*.undo_management='AUTO'
*.undo_tablespace='UNDOTBS1'
*.user_dump_dest='/u01/app/oracle/admin/devdb/udump'
Retrieval of a very common search term (680,000+ hits) takes about 3 seconds and involves a very short, sharp I/O spike –but nothing too much to worry about, I think.
I set EXACTLY the same parameters on a Windows Server 2003 R2 Standard Edition box (64-bit) –apart from the obviously path-related ones. The same query runs in over 54 seconds and there is a massive and sustained I/O hit on the datafile drive (as measured by perfmon). In fact, the web front end used to do these searches simply times out and no results are ever actually displayed.
The Linux box is using just two AMD Opteron single core chips; the Windows box is using two quad-core Intel Core 2 Duos –which run at a higher clock speed than the Opteron, apart from anything else. So the Windows box has (effectively) at least 6 CPUs more than the AMD machines. It also has double the number of hard drives and double the amount of free RAM after the Oracle instance has started.
And yet with all those advantages, the Windows box runs at least 25 times slower than the Linux box for the same search term, using the same table and index structures.
What is going on?! Why does the Windows box keep hitting the hard disk (it’s definitely cause and effect: you can watch the Windows perfmon idle, press the Enter key on the search box and see the I/O bar immediately jump and stay there until either the page gives up or the results are returned. The next search term causes another I/O spike. And on and on: nothing ever seems to need only memory access, despite there being stacks of memory in which to cache the objects being accessed in their entirety)?
I’ve compared execution plans between the two servers, and they are identical in the number and type of steps they take (table access by index rowed domain index, sort order by, view and so on).
Does anyone know why Windows Oracle Text access should appear to demand physical I/O that refuses to be cached whereas Linux Oracle Text doesn’t?
Is there some subtlety of Oracle-on-Windows that I’ve missed configuring? Is there some known bug that I’m hitting?
The software is actually 10.2.0.3 64-bit, but we’re using 10.2.0.1 optimiser compatibility.
I would be grateful for any pointers since I am at a complete loss to explain the seemingly very different behaviour of Oracle on each platform.
This isn’t a Linux v. Windows, thing, either –just to head that one off before it happens! We have to run on Windows, so I’m not knocking it, but seeking to get Oracle running on it properly. I would be grateful for any insight you can offer.

Answer 1

How did you establish that? Did you query v$bh, or is this logical reasoning? (I/O is high, so data is not in cache)
As I said, I watched the Windows CPU and Disk I/O performance metrics for some time and established they were at a flat line. I typed in a search term and pressed Enter. Immediately, the I/O performance measure soared. Just before the search results were displayed, the I/O metric went back to flatline and stayed there. Do another search, the metric goes up; as soon as the results are displayed, the metric goes back to around about zero. Utterly repeatable. I didn't bother viewing V$BH, because this happens to be the only table (along with its associated indexes) that exist in this database, so there aren't exactly a lot of candidates for buffer cache occupants!
Windows servers are by default configured with a huge file system cache. This should be turned off.
Good point, but it is.
Also the background services should be given priority.
And, again, they are. I've also done things like switch on large pages so that the SGA is locked in real memory, not pages out to disk. But none of that makes a difference to the observed disk I/O behaviour.
Apart from that: the architecture is fundamentally different so you really shouldn't compare Windows and Linux.
As I said originally, I'm not. But I cannot ignore the fact that 3 year-old hardware that happens to be running Linux is out-performing brand new, state-of-the-art servers that are running Windows, especially when the init.ora is identical in each case. I'm not saying one is better than another, just that there's clearly an issue and I would like to know whether it is indeed architectural or whether it's something else (like a Windows-specific bug, for example).
I respect the findings of all Oracle experts, but this isn't simply a matter of 'Windows is three times slower'. For a start, it's more like 'ten times slower'... and anyway, it's a behaviour difference that is the real problem here: Oracle-on-Linux doesn't do I/O (or, very little, I should say) when a search is performed. Oracle-on-Windows does a massive I/O hit. That's not a performance issue, I would say, but a fundamentally different way of working. Something must be causing it. I would like to know what.
'Architecture is different' doesn't cut it. Sure, I know that the architectures are profoundly different... but 10 times faster with less than half the CPUs different? And 'I have to do IO when the other guy doesn't' different? I think just saying 'architecture' is a mantra response and doesn't really mean much.
If it really is true that 'it's just the architecture', then what you're really saying is 'Oracle doesn't work on Windows' -or at least 'Oracle Text doesn't work on Windows'. That's not actually the official line though, is it? If we take the more rational line of assuming that Oracle Text does work on Windows, I'd like to know how I can make it work well on Windows.