PLease help: High Volume database hitting limits

Question

Hi,
I am using Berkeley DB to store real time ticks for market data. To give you an idea of the amount of data...
- I get roughly 15000 ticks/sec on an average
- by the end of the day the database file grows upto 50GB.
My system configuration is...
$ uname -a
Linux gritsbox.rsi.com 2.6.18-92.1.17.el5 #1 SMP Wed Oct 22 04:19:38 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
- Quad core 1.6 Ghz
- 4GB RAM
- I have set the Berkeley db cache to 2GB and I am using the BTREE access method.
I started out using one database to store everything and it works well for 3/4th of the trading day. However, Db:puts increasingly start to take longer and towards the end of the day the delay is so high that I start loosing ticks. I figured that this could be because of BTREE trying to rebalance a heavy tree on every PUT (is this correct?)
I then decided to open 10000 (yeah!!!) databases in the environment, 1 for each symbol(ticker) that i am subscribed to. my idea behind this was that balancing a much smaller tree would be faster for Berkeley DB. However when i run the code, i get a "Db::open: Cannot allocate memory" error after opening just 300 something databases.
Any suggestions would be much appreciated!
thanks
Nishant

Answer 1

Hi Bogdan, thanks for your reply. below is the response to your questions...
1) I havent written code to sort the records. I assume BTREE is storing them sorted.
2) This is a multi-threaded application
3) I have to access the database on demand. Other processes would set the curser to the last value in each database and return its value.
4) the 50GB file is created in my old approach where I was using one database to store everything. today I was trying to open 1 database for each symbol/ticker (so around 10000 databases) that resulted in the "Could not allocate memory" error. In the old approach a new 50GB file was created everyday.
5) This is the only code that writes to the database...
     //Now write the values to DB
     stringstream ss_key_;
     ss_key_ << bsid_ << time_;
     string s_val = "{" + ss_val_.str() + "\"d\":\"d\"}";     //adding a dummy key to take care of the last comma
     Dbt key((char *)ss_key_.str().c_str(), 25);     //BSID + Time
     Dbt value((char *)s_val.c_str(), s_val.length()+1);
     try {
          dbs_[1]->put(NULL, &key, &value, 0);
} catch (DbException &e) {
cout << "Error: " << e.what() << std::endl;
6) u_int32_t envFlags      = DB_CREATE | DB_INIT_MPOOL | DB_INIT_TXN | DB_INIT_LOCK | DB_THREAD;
u_int32_t dbFlags      = DB_CREATE | DB_THREAD;
7) No, I am not managing the cache.
I appreciate you taking the time to help me out.
thanks
Nishant