Can insert performance be improved playing with env parameters?

Question

Below is the environment confioguration and results of my bulk load insert experiments. The results are from two scenarios that is also described below. The values for the two scenarios is separated by a space.
Environment Configuration:
setTxn     N
DeferredWrite Y     
Sec Bulk Load     Y
Post Build SecIndex Y
Sync Y
Column1 value reflects for the scenario:
Two databases
a. Database with 2,500,000 records
b. Database with 2,500,000 records
Column2 value reflects for the scenario:
Two databases
a. Database with 25,000,000 records
b. Database with 25,000,000 records
1. Is there a good documentation which describes what the environment statistics mean.
2. Looking at the statistics below, can you make any suggestions for performance improvement.
Looking at the below statistics is the:
Eviction Stats                    
nEvictPasses               3929          146066
nNodesSelected               309219          17351997
nNodesScanned          3150809     176816544
nNodesExplicitlyEvicted     152897     8723271
nBINsStripped          156322     8628726
requiredEvictBytes     524323     530566
CheckPoint Stats     
nCheckpoints     55     1448
lastCheckpointID     55     1448
nFullINFlush     54     1024
nFullBINFlush     26     494
nDeltaINFlush     116     2661
lastCheckpointStart     0x6f/0x2334f8     0xb6a/0x82fd83
lastCheckpointEnd     0x6f/0x33c2d6     0xb6a/0x8c4a6b
endOfLog     0xb/0x6f22e     0x6f/0x75a843     0xb6a/0x23d8f
Cache Stats     
nNotResident     4591918     57477898
nCacheMiss     4583077     57469807
nLogBuffers     3     3
bufferBytes     3145728     3145728
(MB)     3.00     3.00
cacheDataBytes     563450470     370211966
(MB)     537.35     353.06
adminBytes     29880     16346272
lockBytes     1113     1113
cacheTotalBytes     566596198     373357694
(MB)     540.35     356.06
Logging Stats          
nFSyncs 59     1452
nFSyncRequest     59     1452
nFSyncTimeouts     0     0
nRepeatFaultReads     31513     6525958
nTempBufferForWrite     0     0
nRepeatIteratorReads     0     0
totalLogSize     1117658932     29226945317
(MB)     1065.88     27872.99
lockBytes     1113     1113

Answer 1

Hello Linda,
I am inserting 25,000,000 records of the type:
Database 1
Key --> Data
[long,String,long] --> [{long,long}, {String}}
The secondary keys are on {long,long} and {String}
Database 2
Key --> Data
[long,Integer,long] --> [{long,long}, {Integer}}
The secondary keys are on {long,long} and {Integer}
i set the env parameters to non-transactional and setDeferredWrite(True)
using setSecondaryBulkLoad(true) and then build two Secondary indexes on {long,long} and {String} of the data portion.
private void buildSecondaryIndex(DataAccessLayer dataAccessLayer ) {
    try {
          SecondaryIndex<TDetailSecondaryKey, TDetailStringKey,
                                   TDetailStringRecord> secondaryIndex      = 
                                   store.getSecondaryIndex(
                                         dataAccessLayer.getPrimaryIndex() ,
                                         TDetailSecondaryKey.class,         
                                         SECONDARY_KEY_NAME
        } catch (DatabaseException e) {
              throw new RuntimeException(e);
We are inserting to 2 databases  as mentioned above.
NumRecs        250,000x2   2,500,000x2     25,000,000x2
TotalTime(ms)  16877             673623     30225781
PutTime(ms)    7684             76636   1065030
BuildSec(ms)   4952             590207     29125773
Sync(ms)       4241             6780     34978Why does building secondaryIndex ( 2 secondary databases in this case) take so much longer than inserting to the primary database - 27 times longer !!!
Its hard to believe that building of the tree for secondary database takes so much longer.
Why doesnt building the tree for primary database take so long. The data in the primary database is same as its key to be able to search on these values.
Hence its surprising it takes so long
The cache stats mentioned above relate to these .
Can you try explaining this. We are trying to figure out is it worth trying to build the secondary index later for bulk loading.