Insert Speed

Question

Hello.
I have a db which has 5M records and has 2 secondary associated dbs.
ENV uses 800M cache.
I have checked speed of insertion, and how long took to populate every 100,000 records.
before 3M records, it took about 10 sec. to insert every 100,000 records.
I think that's very nice.
But after 3M inserted, it took about 100 sec. to insert every 100,000 records.
overall, it took 20 minutes to insert 5M records.
Is there any solution improving performance without having more cache?
Thanks.
ps.
* Primary DB's file size is 4.9GB
* 2 Secondary DB's file size is 700MB
* I'm not using transactions.
Result of Insert Test
inserted rows seconds to insert 100,000 records
100,000      5.5
200,000      4.5
300,000      5.6
400,000      5.8
500,000      5.4
600,000      6.7
700,000      5.2
800,000      9.5
900,000      8.1
1,000,000      10
1,100,000      10.8
1,200,000      9.9
1,300,000      10.7
1,400,000      11
1,500,000      11.8
1,600,000      9.6
1,700,000      10.6
1,800,000      10.9
1,900,000      12.2
2,000,000      11
2,100,000      10.8
2,200,000      10.9
2,300,000      11.7
2,400,000      11.1
2,500,000      13.9
2,600,000      10.7
2,700,000      10.6
2,800,000      11.4
2,900,000      11.3
3,000,000      24.2
3,100,000      45.4
3,200,000      53
3,300,000      38
3,400,000      59.4
3,500,000      81.8
3,600,000      83.8
3,700,000      95.6
3,800,000      79
3,900,000      75.8
4,000,000      80.9
4,100,000      98
4,200,000      117.8
4,300,000      110
4,400,000      96
4,500,000      82
4,600,000      101
4,700,000      104
4,800,000      109
4,900,000      110
4,931,099      20
Message was edited by:
wertyu
Message was edited by:
wertyu
Message was edited by:
wertyu

Answer 1

Hello.
I have a db which has 5M records and has 2 secondary
associated dbs.
ENV uses 800M cache.
I have checked speed of insertion, and how long took
to populate every 100,000 records.
before 3M records, it took about 10 sec. to insert
every 100,000 records.
I think that's very nice.
But after 3M inserted, it took about 100 sec. to
insert every 100,000 records.You did not mention what hardware you run on. I can give you some numbers that I'm seeing on my MacBook Pro (2.4 Ghz, slow laptop disk)
My records are 40 bytes and my keys are 20 bytes. With two secondary indexes I'm inserting those with about 20.000 records per second. The insert speed is pretty constant, whether I'm testing with 200.000 or 10.000.000 million records.
The one thing that is completely killing performance is doing an import from some dump file on the same filesystem that ie being read record by record. If I do that then I see your speed pattern, first few go fast, then it drops.
What worked very well for me was to read all or large chunks of import files into memory. Then there is mostly write access to the database files and nothing else. Speeds up many times.
S.