Abrupt, loss in performance.

Question

Hi all.
I just found that doing certain database work like data normalization is far faster in java than Sybase.
Stats I have recorded:
Records                    | Sybase  | Java
less than 30,000 records:  | 2000R/S | 3000R/S
less than 800,000 records: | 2.5R/S ; | +2500R/S
more than 800,000 records: | ?       | 25R/S
public void process(){
UniqueList<Person> persons = new UniqueList<Person>();
//Create other unique lists for other tables.
while(log.next()){
cleaner.clean(log.getLine());
Record record = new Record(cleaner.getCleanedData());
record.setPersonsIDKey(persons.add(record.getPerson()));
//set other IDKeys for other tables
WhareHouse.storeRecord(record);
public class UniqueList<Unique>{
  int id=0;
  private HashMap<Unique,Integer> list = null;
  public UniqueList(){
    list = new HashMap<Unique,Integer>();
  public int add(Unique value){
    if(!list.containsValue(value)){
      listID.put(value,id);
      return id++;
    }else{
      return -1 * list.get(value).intValue();
} I have 2 questions to ask but let me ask the simpler of the 2 first.
*<Question 1>*
This is regarding Sets.
Why is it that Sets use both equals and hashCode to determine uniqueness, even when the documentation says
(o==null ? e==null : o.equals(e)). *</Question 1>*

Answer 1

kajbj wrote:
Execute it with gc printing so that you can see how often the gc is executing and for how long it's executing. It still sounds gc related.It looks that way.
Here are 3 sections (Begining, Middle, point of performance loss) of the log file from
java -XX:+PrintGC -Xloggc:log.log Normalizer------------------
0.075: [GC 896K->183K(5056K), 0.0017151 secs]
0.108: [GC 1079K->186K(5056K), 0.0005844 secs]
0.109: [GC 1082K->187K(5056K), 0.0003445 secs]
0.134: [GC 1083K->190K(5056K), 0.0002221 secs]
0.130: [GC 1086K->191K(5056K), 0.0001363 secs]
0.152: [GC 1087K->195K(5056K), 0.0002240 secs]
0.145: [GC 1091K->197K(5056K), 0.0001318 secs]
0.165: [GC 1093K->200K(5056K), 0.0002152 secs]
0.159: [GC 1096K->201K(5056K), 0.0001563 secs]
0.179: [GC 1097K->203K(5056K), 0.0002125 secs]
171.522: [GC 55672K->52095K(56024K), 0.0005632 secs]
171.533: [GC 55679K->52103K(56024K), 0.0004544 secs]
171.544: [GC 55687K->52111K(56024K), 0.0004587 secs]
171.554: [GC 55695K->52114K(56024K), 0.0004445 secs]
171.609: [Full GC 52150K->52150K(56024K), 0.1150563 secs]
171.736: [GC 56151K->52158K(65088K), 0.0046462 secs]
171.752: [GC 56254K->52165K(65088K), 0.0004931 secs]
171.778: [GC 56261K->52172K(65088K), 0.0004828 secs]
171.790: [GC 56268K->52181K(65088K), 0.0004062 secs]
171.802: [GC 56277K->52182K(65088K), 0.0003858 secs]
220.279: [GC 64706K->60616K(65088K), 0.0006192 secs]
220.292: [GC 64712K->60623K(65088K), 0.0005213 secs]
220.304: [GC 64719K->60632K(65088K), 0.0005268 secs]
220.317: [GC 64728K->60638K(65088K), -0.0130646 secs]
220.329: [GC 64734K->60649K(65088K), 0.0006524 secs]
220.329: [Full GC 64745K->60655K(65088K), 0.1366094 secs]
220.481: [Full GC 65087K->60661K(65088K), 0.1351008 secs]
220.645: [Full GC 65087K->60668K(65088K), 0.1205051 secs]
220.794: [Full GC 65087K->60663K(65088K), 0.1209526 secs]
220.930: [Full GC 65087K->60664K(65088K), 0.1348696 secs]
And from here it only does Full GC which takes about a tenth of a second