Insert times grow exponentially

Question

Hello everyone
We want to store (lots of) performance data into a database and we are evaluating also berkeley JE. Basically there are devices and performance data types and an m:n relationship, so n types can be measured per device. Values of the performance data (kpi) are stored together with the timestamp when the measurement happened. What I want to get back from the database, is the performances data of one type of a device over a timerange (which are the values for objId/kpiId pair between two timestamps).
The classes I use are the following:
@Persistent
public class KpiKey implements Serializable {
     @KeyField(1) Long objId;
     @KeyField(2) Long kpiId;
     @KeyField(3) Date ts;
@Entity
public class KpiEntry implements Serializable {
     @PrimaryKey
     private KpiKey kpiKey;
     private Double value;
}Now I found out that inserting new values is linear in time, as long as the KpiKey stays ascending, when inserting new values, i.e. when objId and kpiId are constant and only ts grows (~3.5s per 100'000 inserts).
When I start to randomize objId and kpiId and keep ts ascending, the insert times grow exponentially, starting with 3.5s per 100'000 entires for the first 2'000'000 and grow up to 165s per 100'000 entries between 2 and 4 million entries.
Do you have any recommendations how to optimize this?
Thank you very much in advance.
James

Answer 1

>
So I think this is more a conceptual issue then a configuration issue.
Could another approach with a different primary key configuration maybe solve this issue? Since I don't want to store entites, but values into the DB, a composite key will probably be the only solution - or am I wrong?
>
Each record needs a unique key. Since you need to perform lookups by all the fields in your key, I believe your schema is correct.
One thing that may be working against you is that you are storing very small records, and there is a per-record overhead. If you were to group multiple values into a single record (same kpiId, same objId, but a range of timestamps) you may see a performance benefit. However, this only makes sense if you always operate on a range of timestamps, and don't need to delete or modify individual values. Just a thought, in case you hadn't tried it.
--mark