Converting large amounts of points - 76 million lat/lon's to spatial object...

Question

Hello, I need help.
Platform - Oracle 11g 64bit on Windows Enterprise server 2008 64bit.  64 GB of ram with 2 CPUs totalling 24 cores
Does any one know of a fast way to convert large amounts of points to a spatial object?  I need to convert 76 million lat/lon's to ESRI st_geometry or Oracle sdo_geometry.
Currently, I have setup code using pipelined parallel functions and multiple jobs that run concurrently.  It still takes over 2.5 hours to process all of the points.
Any pointers would be GREATLY appreciated!
Thanks
John

Answer 1

Hi,
Where is the lat/lon data at the moment?  In an external text file or in an existing database table as number attributes?
If they're in an external text file, then I'd probably use an external table to load them in as quickly as possible.
If they're in an existing database table, then you can just update the sdo_geometry column using:
update <table> set <geometry column> = sdo_geometry(2001, <your srid>, sdo_point_type(<lon column>, <lat column>, null), null, null)
where <lon column> is not null
and <lat column> is not null;
That should run very quick for you.  If you want to avoid the overhead of creating redo, you could use "create table .... as select...".  This example of creating 1,000,000 points runs in 9 seconds for me.
create table sample_points (geometry) nologging as
  (select sdo_geometry(2001, null,
  sdo_point_type(
  trunc(100000 * dbms_random.value()),
  trunc(100000 * dbms_random.value()),
  null), null, null)
  from dual connect by level <= 1000000);
I have setup code using pipelined parallel functions and multiple jobs that run concurrently
You shouldn't need to use pl/sql for this task.  If you find you do, then provide some sample code and we'll take a look.
Regards,
John O'Toole