Impact of Controller Policy on ASM

Question

This is to share with everyone my discoveries in regard to Controller Policy for ASM (Automatic Storage Management) in a JBOD (just a bunch of disks) configuration.
As it turns out, it makes a big difference in performance how you set your controller policy. I tested various settings for the READ POLICY: ADAPtive read-ahead, (always) Read-Ahead, or NO-Read-Ahead; and CACHE POLICY: Cache I/O, or Direct I/O. WRITE POLICY was kept at Write-Back, with Write-Through not tested (Warning).
This test showed the best performance is attained if you set Read Policy to NO_READAHEAD, Cache Policy to DIRECTIO, and Write Policy to WRITEBACK. The SQL used are more suitable for OLAP than OLTP. (Time is in min:sec)
ADAP-------ADAP--------NO-RA-------RA------------Read Policy
CACHEIO--DIRECTIO--DIRECTIO--DIRECTIO--Cache Policy
9:13----------6:45----------5:19------------6:50-------------Complex CTAS 3tbl join
6:49----------5:58----------3:53------------7:02-------------Full Table Scan 120mil recs
4:41----------3:42----------3:39------------3:44-------------Full Scan of IOT Table 100mil rec
These tests were conducted on two Dell PowerVaults with 28 x 136 GB 15K RPM drives and one PERC RAID controller set to eliminate any RAID so ASM would provide all mirroring by scattering 1 MB Allocation Unit copies over various disks. Other specs: Windows Server 2003 SP2, Dell 1850 3.6 GHz Xeon PCU. Oracle 10g Standard Edition One. Parameter dbfile_multiblock_read_count is set to read 1 MB.
Your configuration will not match mine, but this test does show that it is worth the time to invest in empirical measurements of controller policy performance. DO NOT TAKE CONTROLLER POLICY FORGRANTED AND GO WITH THE DEFAULT POLICY OR ANY OUT OF THE BOX SOLUTION. This test showed that READ-AHEAD IS HARMFUL FOR ASM with 1MB AU and 1 MB dbfile_multiblock_read_count. I speculate that either Oracle ASM may be issuing its own intelligent read-ahead, or the ASM scatter of 1MB chunks renders single-disk read-ahead useless & excessive, or both.
I am very interested in reading your discoveries in tuning controller policy for ASM.

Answer 1

1: Why not try with 2 PERC adapters ? The single adapter would be a SPOF and a possible bottleneckGood question; and are correct. Normally we would use 2 controllers with 2 PowerVaults and cross-hatch the 4 cables. Howerver, in this case we are planning to grow to 4 PowerVaults, 4 cables, and 2 PERCs (the max), so I need to plan on 2 PV per PERC.
2. Can we see your init.ora Sure. We do data mining, so our settings favor high throughput with minimal availability needs. Due to the amount of data moved via single-threaded execution (this is SE1) we are I/O constrained.
processes = 275
sessions = 308
__shared_pool_size = 176160768
__large_pool_size = 8388608
__java_pool_size = 159383552
java_pool_size = 159383552
sga_target = 1375731712
db_block_size = 16384
__db_cache_size = 838860800
db_keep_cache_size = 16777216
db_recycle_cache_size = 159383552
compatible = 10.2.0.3.0
log_buffer = 15244288
db_file_multiblock_read_count= 64
db_create_file_dest = +DATA
undo_management = AUTO
undo_tablespace = UNDOTBS1
undo_retention = 36000
job_queue_processes = 10
cursor_sharing = FORCE
open_cursors = 300
optimizer_mode = FIRST_ROWS_1000
optimizer_index_cost_adj = 1
optimizer_index_caching = 75
query_rewrite_integrity = TRUSTED
pga_aggregate_target = 1560281088
optimizer_dynamic_sampling= 3
3. Can we see the DDL for the ASM Disks/Diskgroups and the tablespace definitions?Using diskpart.exe the disks were all set as:
create partition primary size=139264 align=64
One large +DATA disk-group was made over all of these disks, placing each disk in its own failure-group, with normal redundancy.
CREATE SMALLFILE TABLESPACE "SYSTEM"
DATAFILE '+DATA(DATAFILE)' SIZE 5120M
AUTOEXTEND ON NEXT 10240K
MAXSIZE 65535M
LOGGING
EXTENT MANAGEMENT LOCAL
SEGMENT SPACE MANAGEMENT MANUAL;
CREATE SMALLFILE TABLESPACE "SYSAUX"
DATAFILE '+DATA(DATAFILE)' SIZE 5120M
AUTOEXTEND ON NEXT 10240K
MAXSIZE 65535M LOGGING
EXTENT MANAGEMENT LOCAL
SEGMENT SPACE MANAGEMENT AUTO;
ALTER DATABASE ADD LOGFILE GROUP 1 ('+DATA(ARCHIVELOG)') SIZE 10240M;
ALTER DATABASE ADD LOGFILE GROUP 2 ('+DATA(ARCHIVELOG)') SIZE 10240M;
ALTER DATABASE ADD LOGFILE GROUP 3 ('+DATA(ARCHIVELOG)') SIZE 10240M;
CREATE BIGFILE TABLESPACE "REF1_DATA"
DATAFILE'+DATA(DATAFILE)' SIZE 150G
AUTOEXTEND ON NEXT 10G
MAXSIZE UNLIMITED
NOLOGGING
EXTENT MANAGEMENT LOCAL
UNIFORM SIZE 64M
SEGMENT SPACE MANAGEMENT AUTO;
CREATE BIGFILE TABLESPACE "USER_DATA"
DATAFILE'+DATA(DATAFILE)' SIZE 350G
AUTOEXTEND ON NEXT 5G
MAXSIZE UNLIMITED
LOGGING
EXTENT MANAGEMENT LOCAL
SEGMENT SPACE MANAGEMENT AUTO;
CREATE BIGFILE TABLESPACE "USER_INDEX"
DATAFILE'+DATA(DATAFILE)' SIZE 100G
AUTOEXTEND ON NEXT 5G
MAXSIZE UNLIMITED
NOLOGGING
EXTENT MANAGEMENT LOCAL
SEGMENT SPACE MANAGEMENT AUTO;
CREATE BIGFILE TEMPORARY TABLESPACE "TEMP"
TEMPFILE '+DATA(DATAFILE)' SIZE 184320M
AUTOEXTEND ON NEXT 640K
MAXSIZE 67092480M
EXTENT MANAGEMENT LOCAL UNIFORM SIZE 1024K;
ALTER DATABASE DEFAULT TEMPORARY TABLESPACE "TEMP";
CREATE BIGFILE UNDO TABLESPACE "UNDOTBS1"
DATAFILE '+DATA(DATAFILE)' SIZE 143360M REUSE;