Upgrading to uber SDO_GEOMETRY, looking for comments

Question

Good morning folks,
I see the new thread from Dalibor discussing an issue with what I am calling "uber" SDO_GEOMETRY - anyone got a better name?
Uber SDO_GEOMETRY is described here
http://download.oracle.com/docs/cd/E11882_01/appdev.112/e11830/sdo_migrat.htm#CHDBCHBG
Dalibor wins some kind of prize for the first time anyone has brought up uber SDO on the forum since the old thread ran its course years back
Re: SDO_GEOMETRY size limits here to stay?
Coincidentally this past week was my first shot at trying out the upgrade script and I thought I might post my experiences and see if anyone would have comments. This was just a test machine and I agree that each major error could have been followed up with a SR. But my overall goal is to determine if installing and running uber SDO_GEOMETRY is feasible in my production environment, so for better or worse these experiences are building those recommendations.
So I am not much interested in going much in length why anyone would want to upgrade. I think the old thread covers it. For my work at least, the issues have not gone away. No question a polygon with millions of vertices is nutty, but they are out there all over the place and growing in number each year. Here are two additional datasets I am working with that are "over the top" for normal SDO:
* National Wetlands Inventory (http://www.fws.gov/wetlands/data/DataDownload.html) - Largest polygon has 1,210,907 vertices.
* FEMA National Flood Hazard Layer (http://www.msc.fema.gov/webapp/wcs/stores/servlet/info?storeId=10001&catalogId=10001&langId=-1&content=nfhlResources&title=NFHL%20Resources) - Largest polygon has 1,537,698 vertices.
My current procedure is to load the datasets using ESRI's geometry format, identify the biggies, export the remainder and then reimport them as SDO_GEOMETRY.
E.g. toss the big polygons away and explain to my clients that Oracle Spatial cannot handle this type of thing.
My production instances remain on 10g so uber SDO is not yet a possibility but I am trying to think ahead.
So back to the present. I started with a fresh 11gR2 (no upgrade history) running PSU 11.2.0.1.2 on OpenSUSE on a rather modest machine with two processors and four gig of RAM. I have about 100 gig of Oracle Spatial data on this machine. Basically I ended up running the upgrade script four times of the course of about 5 days, each run required about 30 hours to process. Here are some notes:
* Make sure you spool the results! The script is quite "verbose" and does not have any error checking so the ALTER TYPE CASCADE part can fail but the script continues recompiling packages and rebuilding all your indexes. This is good in that the system is put back together whether it bombs or not. Bad in that you can waste a lot of time waiting to rerun the script.
* You need lots of undo! The first time it ran through it appeared to finish and it was not until I looked at the logs did I see that I ran out of undo. I just had a single smallfile undo and it topped out at 32 gig. In the end I created 4 undo datafiles and it appears I needed 100 gig of undo! So I believe you need as much undo as you have SDO_GEOMETRY data. The first thing you want to do afterwards is DESCRIBE MDSYS.SDO_ORDINATE_ARRAY to see if the type was altered. If you see the usual 1048576 then you bombed.
* Second time through I received an error that I had some spatial indexes in a failed state. Turns out as this is a development box there was an index stuck in "INPROGRS" state due to some flub on my part. This was enough to hose the upgrade. I would suggest folks check for this ahead of time using SELECT * FROM dba_indexes WHERE index_type = 'DOMAIN' AND domidx_status != 'VALID' and drop or rebuild those indexes.
* Third time through I received the cryptic error
ERROR at line 1:
ORA-22324: altered type has compilation errors
ORA-22332: a dependent object in schema "NAVTEQ_SF" has errors.
ORA-00600: internal error code, arguments: [qctostiix1], [], [], [], [], [],
[], [], [], [], [], []I had some sample Navteq data for use with the networking demo I never got functioning. At this point after four days of trying I was getting a bit impatient and just decided to drop that schema and try again. I have no idea why this navteg data caused an issue but the error never happened again with any other datasets.
* Fourth time through the alter type worked! But the script still crashed with
ERROR at line 1:
ORA-00600: internal error code, arguments: [qximgcn1], [], [], [], [], [], [],
ORA-06512: at line 14After eyeballing the script and the spool file it looks like it failed in the final step where it rebuilds all the spatial indexes. This seems right as I found sixty seven bad spatial indexes afterwards. However, it was easy enough to rebuild those by hand myself (I mean have ArcSDE do so).
So I call this a success! :)
So I am looking for anyone else's experiences they'd like to share.
With the amount of undo required, I think it might be a better route to just use the old school exp to export out all spatial data, run the upgrade and then imp it all back in? Or perhaps just dump out the large tables? Weird 00600 errors cause the DBAs (not part of my organization) that manage my production environment to bleed internally. Ideally I would like to avoid all that pain and suffering. Perhaps just dropping ALL the spatial indexes before the upgrade? I guess I would need to open an SR to see what caused the 00600s. I suspect the indexes.
So I would like to keep this thread tied to the upgrade process and/or planning/strategy for the upgrade process. There are a lot of other issues that we should start new threads over. For example, exchange of data between standard and uber instances as Dalibor is doing. It sure appears that datapump is never going to work and that only the old-school unsupported imp/exp utility can play this role? I have done some initial testing and ESRI's sdeimport/sdeexport tools seems to work fine between these instances. And none of this even scratches the performance issues.
So hey Dalibor, I know you went through this. Any comments?
Cheers!
Paul

Answer 1

Hi Mike,
Nice to hear from you. I would indeed be interested in your thoughts on the upgrade and the actual business case surrounding the upgrade or the decision not to upgrade. I am feeling rather skeptical at this point as the side-effects of the medicine so far are pretty serious. But yet the need to treat the disease remains. You can always drop me an email offline at my first name at my last name dot com or via LinkedIn. The nice things about a super-rare unpronounceable Polish last name is I am easy to find - and that's the bad thing too. Here is an update on my testing.
Per my earlier SR, the folks are Oracle have provided a backport for bug 10085580 for Linux 32-bit. It looks to work just fine and does the trick - problem solved. If one needs something other than Linux you will need to request the backport for your platform. The bug affects more than SDO_GEORASTER so I would recommend this patch as mandatory.
Moving along my next issue is that Oracle MapViewer does not seem to work with uber SDO_GEOMETRY.
MapViewer and uber (Very Large) SDO_GEOMETRY upgrade not compatible?
This is again a bit of a nonstarter as my client has a couple of applications using MapViewer against production instances. I dutifully submitted an SR on the matter yesterday. Almost to be expected the tech immediately asked why I would want to have such a large geometry and why don't I thin it! :)
It's no fun being in the middle on these things. I suppose I could ring up FEMA or Fish and Wildlife and tell them Oracle Corporation thinks their polygons are crazy. But as they are ESRI-shop folks, I think they would just be puzzled by the statement. If anyone has an opinion on how to broach the topic nicely with public agencies, I would love to hear it. Do we need to create a "citizens against huge polygons" lobby in DC? Again the ESRI shop folks are smirking.
The other issue in the thread started by Dalibor where the old imp/exp is erratically failing on the table creation statements is probably going to need an SR too. The work around is to just append the data and create your tables by hand ahead of time. On a more positive note ArcSDE seems utterly oblivious to the type change and I have not found any problems using ESRI tools to move stuff around.
Cheers,
Paul