Howto when i have very very large double array

Question

I have code my program about document retrieval.
I create double array in size [10000]x[8000] for keep value for calculte someting in only memory (about metrix operation).
I have set parameter to heap memory size at 1024m but it not enough, it has exception out of memory.
I use Notebook that have RAM 1G. Is this cause for exception?
Please help me to lead solution for my case.
Thank you.

Answer 1

anurag.kapur wrote:
A couple of points:
1. Your notebook has 1 GB of RAM and you are allocating 1024 MB (=1 GB) to the JVM heap, which is not possible, as you need some space for running various other Operating System processes as well.This is a good point. There are also things in the JVM itself other than the Java object heap that need space. Thread stacks, loaded classes, native libraries, etc.
2. Why have you allocated 1024 MB to the JVM heap? Did you choose any random number (obviously with some degree of educated guess)?
Your array size is 10000 * 8000 = 8 * 10^7
Thus total heap memory required = 8 * 10^7 * 4 Bytes [The int data type is a 32-bit signed two's complement integer]
= 320 * 10^6 Bytes
= 320 MBExcept that peterdog1234 said that the array was an array of doubles, at 8 bytes each, so about 640MB are needed for the array. If the algorithms allow the loss of range and precision, maybe the array could be an array of floats, not doubles. That would keep the array in about 320MB. That would help with the space requirements, but might be slower as all the operations will have to expand the floats to be operated on as doubles, then converted back to floats for storage in the array. But on some architectures that's not too bad.
I would use -XX:+PrintGCDetails to see how the heap has been partitioned into young generation (fast allocation and collection) and old generation (relatively long-lived data). You might have to reshape the heap to have at least 640MB in the old generation, and then whatever you can fit as the young generation, leaving some of your 1GB for the rest of the JVM, other programs running on the box, the operating system, etc. Maybe "-Xms768m -Xmx768m -XX:NewSize=64m -XX:MaxNewSize=64m" would be a good compromise. That gives you a 768MB Java object heap, with 64MB in the young generation and 704MB in the old generation, leaving 256MB (1GB-768MB) for "other things". Whether that works depends on now many other long-lived Java objects you have. But -XX:+PrintGCDetails will tell you that, too.