Still seeing scalability problems

Question

While BDB 4.6 is definitely faster than earlier versions for databases that fit inside the BDB cache, I'm still seeing unreasonable slowdowns when the db is much larger than the cache. E.g. in OpenLDAP I'm loading a 50 million entry database (which results in about 69GB of data on disk) on a machine with 16GB of RAM, 12GB configured for the BDB cache. On a Linux system with ext2fs this load takes 7 hours. If I change the loader to open the environment with DB_PRIVATE, so that malloc() is used instead of the BDB shared environment, the load takes only 3 hours. Most of the excess time using the BDB shared environment is due to __env_alloc_free(); there are long periods of time where the loading process is using 100% CPU, in this function, with no actual progress being made on the load.
Another issue that I'm seeing in 4.6 that didn't occur in 4.2 - I also use a trickle_sync task which is run after every 8192 entries are written. In 4.2 this task pretty much always writes some number of pages on each invocation, and that number is relatively consistent (around 700 pages with a full cache) each time. In 4.6 this task starts out consistent, but somewhere around when the cache first fills completely the number of pages written drops, to about 16 pages per invocation, and then drops to 0 pages. From then on the trickle_sync task does nothing useful at all.

Answer 1

DWILCOX01 wrote:
Ansury wrote:
I think this only seems to happen when I post and then quickly edit the post within 1-2 seconds, so it may be a timing thing.
Do you always have to post and then quickly edit your message within 1-2 seconds?  How about writing a draft first and then post it only if you are sure it is exactly what you want to say.
I am fairly careful usually, but I'm used to having a PREVIEW function (like every real forum on the face of the planet has).  Not to mention, what you see in the text box (non-preview view) isn't always exactly what you see in the actual message. 
Ex. When you're posting source code and you need it to be formatted in a certain way so it's readable...  Or how about when you type a word that our 1950's-era language filter happens to dislike?  You can't tell whether it was "star'ed" until AFTER it's posted.  So now you have people thinking you're putting foul ******* language all over the forums when you're really typing f*r*i*g*g*i*n or "poopy" or something harmless.  A preview would at least save us from that nonsense, but this primitive jiveware doesn't even give us that.