Large exports fail

Question

Hi,
We've recently installed SQL Developer and are trying to use it to export tables/results of queries to .csv files. Some of these exports are very large (> 3 million records) and it seems that SQL Developer cannot handle them.
After it reaches about 1.7 million records, it stops without any error message and the csv file is empty.
I've read about this problem on other posts, and it seems to be because Java runs out of heap space. I've edited ide.conf in the /ide/bin folder and upped the min and max heap size, which did allow it to export more records, but still not as many as we need. There is a limit on this heap size too, something to do with not having enough contiguous space in memory.
It seems as though it's trying to push everything into memory and then writes it out to disk rather than writing to disk periodically.
Does anyone know of any workarounds or other clients that might handle exports better?
Thanks,
Aditya

Answer 1

In the ide.conf file there are some Java VM settings, VMOption -Mms and -Mmx or something like this. Bumping those up increases the Java heap size and allows more records to be exported...still wasn't enough for us though.
Have also tried Tora now. It followed a similar path, memory kept going up and up. Then it reached 800MB and just disappeared without a trace...
The alternative we've used is to create a PL/SQL procedure that will dump the results of a query to a file, and it uses the UTL_FILE package, which writes out to disk periodically rather than storing in memory, and it can handle exports of any size. There's procedures like this available out there, so try them if you're stuck.