Determine IOPS/Throughput for Data Warehouse

Question

We are migrating our DW to new servers but before we install Oracle we want
to check IOPS of new servers. As Oracle and other technical expert suggests we
are planning to create the DB with 32K block size with DBMRC=32 so total IO
size of 1 M but the test I performed does not support that theory. Please advise
where I am wrong OR what is the best way to check the throughput:
Here are my results:
1. Created a 2 GB file deleteme
2. For 32*32 results are: (block size=32, db_block_multi_read-count=32)
time dd if=deleteme of=deleteme2 bs=1048576 (32*32)
2048+0 records in
2048+0 records out
real 0m42.94s
user 0m0.00s
sys 0m1.22s
Result: 47.69 IOPS 1MB each.
2. For 32*8 results are: (block size=32, db_block_multi_read-count=8)
time dd if=deleteme of=deleteme2 bs=262144 (32*8)
8192+0 records in
8192+0 records out
real 0m28.12s
user 0m0.02s
sys 0m3.69s
Result: 291.32 IOPS 256K each.
Based upon above is it better to have IO size for the DB as 256K? Please advice.
Thanks in advance.

Answer 1

I/O throughout is a function of the transfer size, access type (read/write random/sequential), access range (e.g. 1 MB vs. 1 TB), arrival rate, and most importantly, the number of concurrent threads (load level). The dd command for instance is single threaded, while a file system or database can generate hundreds of concurrent threads.
Under the control of ORACLE, data must be written through the page cache (O_DSYNC flag in open()) and the database manages it own cache, so for the most part, a file system page cache is redundant and adds a lot of overhead to I/O when used with a database. Direct I/O (the forcedirectio option of UFS for example) is often the best choice with ORACLE.
In all cases the combination of application demand, like a 32 KB block size in ORACLE with multi-block read count set for 1 MB reads, with some number of shadow process or SQLNET threads, is going to generate a characteristic I/O workload as described by its distribution of level and composition (IO size, access type). This is the key to understanding the performance capability of any given storage resource, likewise it is the key to determining how much of a given resource is needed to target and achieve a specific performance level for a specific workload arrival rate, load level distribution and composition.
Also, when configuring a system for 1 MB I/O you have to adjust some kernel and filesystem parameters. The Solaris maxphys, SVM md_maxphys, VxVM vol_maxio and UFS maxcontig are examples. I have heard the maxphys is adaptive in Solaris 10, but I have yet to see it adapt, and the md_maxphys (if using SVM on top of hardware RAID LUNS for example) does not adapt, nor does maxcontig (newfs �C or tunefs �a) in UFS. Each filesystem has its own configuration parameters you need to look at to see how to effect allocation and in particular contiguous allocation sizes. VxFS for example will use a 64 KB file allocation size by default and needs to be adjusted to get > 64 KB I/O. There may be other parameters governing I/O size through your filesystem as well, but these are mostly an issue if you not using Direct I/O.
Once the workload is known the expected throughput for a complex combination of I/O sizes, and access types for a given load level can be estimated very accurately by weighted averages of the pure I/O size and access types as measured over the range of load level, from single thread to saturation for each I/O size and access type. The final result is according to Littles law: (X = N/R) * U, where X is IOPS, N is load level, R is response time (which is a function of load level and composition at the level of a single resource, which is determined by how many resources the load is being spread, as discussed above) and U is busy utilization�a measure of how sustained is the arrival rate.
The idea is to spread the load evenly over some number of physical resources such that per resource load level is in a comfortable and/or targeted range of response time for your SLA, understanding that response time is a function of load level and composition at the unit of resource level, and throughput is a function of load level and response time. This defines the capability of the storage system, for the given workload, how much of that capability is used defines head room and is a function of arrival rate.
There are tools available that you can get for free evaluation to find out what your workload is, and also to profile your storage resources to determine its capability.