Inserting large amounts of data

Question

I am not sure if this is the best way of doing this but, I am trying to store icing data that we gather for general aviation purposes, in a spatial table in Oracle 8.1.7
The data set has about 10 altitude levels with about 10,000 points at each level. Each point corresponds to a a small rectangular region on the map.
We receive each data set every hour. The problem is that inserting these points in the database takes a long time. I am using JDBC with Oracle Update Batching with commit at every 100 inserts. The actual insert takes place in a stored procedure.
According to my benchmarking it takes about 10 seconds for each 100 inserts to take place. Here is the insert statement:
insert into ncar_detail
(ncar_detail_id,ncar_id,altitude,grid_value,grid_point)
values
(ncar_detail_seq.nextval,p_ncar_id,p_altitude,p_grid_value,
mdsys.sdo_geometry(2003, 8307,null,
mdsys.sdo_elem_info_array(1,1003,3),
mdsys.sdo_ordinate_array(p_ll_lon,
p_ll_lat,
p_ur_lon,
p_ur_lat)
If you have any comments on how to improve performance I appreciate it. The reason I am storing each point in the database is to be able to run proximity queries for hazard detection. Is this impractical or not?
Thanks
-Selim
null

Answer 1

Hi,
I don't know enough about SQL*Loader to help with the sequence number questions, but I'll see if I can get someone with more knowledge in that area to respond.
Regarding the speed of inserting:
Updating/adding data into spatial columns that have been indexed using r-trees is relatively slow. It may not be a jdbc issue at all.
There are a few things I would suggest:
try rebuilding the rtree index with a very large value for sdo_rtr_pctfree. This will leave room in the index structures for inserted data. After that if things are fast, occasionally rebuild the index.
also, how long does it take to build the index in general? If it is fast, try dropping the index and recreating it after the new data is inserted.
You may want to check the performance of queries and inserts using quadtree indexes.
You might want to have separate tables based on each level, or in 9i, partition the data based on the level.
Also in 9i, the rtree index on data with srid of 8307 (or any geodetic SRID) becomes a geodetic index so it is more accurate. But there are restrictions, one of which is the optimized rectangle will not be supported. I'm not sure you need full geodetic support, in which case you may want to remove the srid fields. Also, indexing geodetic layers will be a bit slower, and some operations will take a bit longer (for instance, length, distance, and area calculations will be excellent on the surface of the earth without having to project the data, but there are additional cpu cycles required because the geometry is non-euclidean). If you do need to keep the geodetic srid, and you want the geodetic features in 9i, then you should load the rectangles with all 4 corners defined (so have 5 coordinates, as the first and last have to be the same).
Hope this is useful.
dan