Image scale caching causing problems

Question

I have an app with a image that the user must be able to zoom and scroll.
When the image is first loaded it is at full view. When the user zooms the memory usage sky rockets, and during this zooming the app is very laggy.
Once the memory has added about 70Mb, the zooming runs without lag.
If the user waits however, the memory usage goes back down, the scaling lags again, and the memory fills again.
I have been able to get around this with making the height an odd number of pixels.
This causes the memory to stay at the +70Mb and prevent lagging, but this may not be a permanent solution.
Is there any way to prevent the GC from wiping the scale information? Or any way to manage how the cached scale data is stored?

Answer 1

The image is large. The app is for iOS and Android devices.
The images are 5464x3073 (should be 3072, but made it odd to prevent problem), they are simply scaled to fit the screen (usually about 0.2 scale).
Till this point is fine, obvious pause when loading the image, but when the user zooms in, the memory gets an extra 70Mb(if you zoom all the way in) and stays for about 10-20 seconds. During this time zoom is smooth. When the memory goes back down the process of zooming causes lag and memory usage again.