Blocks have much more free space specified by pctfree moved off freelist.

Question

We have a table with pctfree=0 and pctused=90 and experienced high session against it. During the high session, we saw lots of "read by other session", cbc chain, and "buffer busy wait" caused by the same sql (INESRT). The row size of of an average 4K with long raw. After got the block info for the "read by other session" and "buffer busy wait", I dumped the block. The strange thing is that all the of blocks dumped are not in the freelist and the available free space is over 1K.
My idea is: When the session named waiting_a experienced "read by other session" to find the block to insert data, that block must be in the freelist, otherwise the waiting_a wouldn't try to read it in buffer and wouldn't be waiting for it to be ready. Very shortly other session reading_b which was reading the block put the data in buffer and did the insert operation, and the reading_b completed its work of insert. My dump should be after reading_b completed. So the dumped block should contain the data inserted by reading_b. After that the block was got off from freelist.
But why blocks with 0x520 (1312)and 0x104c (4172) freespace got off from freelist? As pctuse is 90, a block will be got off from freelist when free space is less than 10% of the block size, and these two blocks have much more freespace.
TEST-prod-test$> grep tosp TEST_ora_23095.trc
tosp=0x2a2
tosp=0x104c
tosp=0x520
TEST-prod-TEST$> grep avsp TEST_ora_23095.trc
avsp=0x2a2
avsp=0x104c
avsp=0x520
TEST-prod-test$> grep flg TEST_ora_23095.trc
scn: 0x0328.47520b02 seq: 0x02 flg: 0x04 tail: 0x0b020602
seg/obj: 0x14e05  csc: 0x328.47520b02  itc: 1  flg: -  typ: 1 - DATA
scn: 0x0328.47520f41 seq: 0x02 flg: 0x04 tail: 0x0f410602
seg/obj: 0x14e11  csc: 0x328.47520f41  itc: 1  flg: -  typ: 1 - DATA
scn: 0x0328.4752153a seq: 0x02 flg: 0x04 tail: 0x153a0602
seg/obj: 0x14e0d  csc: 0x328.4752153a  itc: 1  flg: -  typ: 1 - DATA

Answer 1

knowing Oracle specifically warns against using multiple blocksizes
Nope.  Oracle does not warn against using multiple blocksizes, anywhere in the documentation.
The only warnings I've seen (By Daniel Morgan, and unverified) claims that Oracle was negligent and only tested the SQL optimizer with an 8k blocksize.
I'm not sure that this is true, since it would constitute misfeasance.
I know that Oracle does extensive tesing on their new features, and the docs offer the multiple blocksize feature without reservation.
Uisng multiple blocksizes is a well established, proven concept.
DBA's have been using them for decades with IMS, IDMS and DB2 before Oracle came along.
I first saw multiple blocksized used with great success in the 1980's, and they are still in use today on many large databases.
provide a measurable idea what 'lots' constitute, and how much I/O is reduced in the shops you saw?There is load of verifiable evidence out there:
http://www.google.com/search?&q=site%3Awww.tpc.org+db_16k_cache_size
This UNISYS Oracle benchmark used multiple blocksizes to achieve optimal performance
db_cache_size = 4000M
db_recycle_cache_size = 500M
db_8k_cache_size = 200M
db_16k_cache_size = 4056M
db_2k_cache_size = 35430M
http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100883
The IBM Oracle Technical Brief titled "Oracle Architecture and Tuning on AIX" (November 2006) notes that careful evaluation is required before implementing multiple blocksizes:
While most customers only use the default database block size, it is possible to use up to 5 different database block sizes for different objects within the same database.
Having multiple database block sizes adds administrative complexity and (if poorly designed and implemented) can have adverse performance consequences. Therefore, using multiple block sizes should only be done after careful planning and performance evaluation.
https://metalink.oracle.com/metalink/plsql/f?p=130:14:4123909781200375285::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,46757.1,1,1,1,helvetica
Metalink Note:46757.1 titled "Notes on Choosing an Optimal DB BLOCK SIZE" says that there are some benefits from having larger blocksizes