CFMX 6.1's Virtual Memory Use problem!!

I appologise for the long post in advance...
Ok... so I have this script that, using cfdirectory, will
check a directory for any files that may have been uploaded, if
there are files, it loops through the results and reads the files
one at a time, line by line, using the FileReader.cfc (Uses the
Java FileInputStream, InputStreamReader, and BufferedReader to
provide a way to incrementally read large files). The files are
just pipe "|" delimited data, each line represents a record for a
db table.
Now as it's reading each line, it will perform some basic
string parsing to clean up the file line to make sure the data is
valid, blah blah blah and then it will write that "cleaned" line to
another file using FileWriter.cfc (Java component once again). Once
it's completely done reading the original file, it will close it
and it will open the new "cleaned" version of the file, read it
(FileReader.cfc), create an INSERT statement and then update the
database table.
Now... this all works GREAT... until it has to loop through
more than a few files... 3 - 4 files are NO problem! works like a
charm, but throw 6 - 8 files at it and it dies, not a timeout mind
you but an actual "java.lang.OutOfMemoryError" (now, I've tried
making all the files exactly the same (just changed the name) and
the weird thing is, it takes longer and longer to process each as
it goes through the loop... I have the script write some stats as
it's looping:
FILE 1 STATS
Name: COA0607_Intranet1.DAT
Status: Import Successfull
Line Count: 32,971
Processing Time: 74,237ms
FILE 2 STATS
Name: COA0607_Intranet2.DAT
Status: Import Successfull
Line Count: 32,971
Processing Time: 82,018ms
FILE 3 STATS
Name: COA0607_Intranet3.DAT
Status: Import Successfull
Line Count: 32,971
Processing Time: 94,476ms
FILE 4 STATS
Name: COA0607_Intranet4.DAT
Status: Import Successfull
Line Count: 32,971
Processing Time: 145,108ms
I know what you guy are probably thinking; "Woah man... CF
isn't really meant to do that kind of processing...", I know, trust
me I know... however, I really neeeeeed it too lol.
Ok, so as the script is running, I watch the Virtual Memory
use of jrun.exe, processing say 3 - 4 of these files brings up the
usage to approx 300,000k which yes, is a LOT but that's fine...
this process is meant to run at night via a Scheduled Task...
When I run more than 4 files, things start to get ugly, keep
in mind that these are EXACTLY the same files just re-named
differently. The script will start lagging BIG time and on the last
file (usually the last file) I'll see the memory usage spike from
350,000K all the way up to 600,000K and that's when it throws the
"java.lang.OutOfMemoryError" and dies... I've tried commenting out
the part of the script that updates the db, but still get the same
problem...
So... what gives? How come CF Server does this??? I mean, it
runs fine for the first few files... and then WAM, it dies... sorry
for the long post... any insight here is VERY much appreciated...
it would be AWESOME if the wonderful folks at Adobe could shed some
light on this for me : )
CFMX 6.1 version: 6,1,0,83762
Windows XP Pro SP2
Intel P4 2.8Ghz
1Gb of Ram

quote:
Originally posted by:
Mr Black
300M memory usage while using "incremental" file reader??
Looks like it is "incremental" only in the sense that it increments
memory usage. Did you try non-Java C/C++ file reader tags?
Well I did try cffile originally... and it didn't even run...
lol

Similar Messages

Huge virtual memory use - 200GB

I'm using a MacBook Pro (10.8.3), 2.4 GHz Intel Core 2 Duo, 4 GB.
When I look at activity monitor I see VM size at >240 GB!
I only have firefox running. When I reboot I see a similar VM size.
I only have 20GB of free disk space.
While the system appears to be running ok, I'm troubled by the size of the VM memory - perhaps I have a virus?

Winds Up!!!! wrote:
When I look at activity monitor I see VM size at >240 GB!
Not a problem. From the Help:
VM size:
VM size is the total amount of virtual memory space reserved by the OS X and your apps. The actual amount of virtual memory being used is likely to be much less, because OS X and apps frequently reserve virtual memory space that they don’t use.
You may be thinking of swap space. Also from the Help:
Swap used:
The amount of hard disk space currently being used as virtual memory
Activity Monitor actually has pretty good Help. See the Monitor your computer's resources section.
I only have 20GB of free disk space.
That's probably enough.

Activity Monitor - Java Virtual Memory Use is 16,333.00 TB. How is this possible?

Activity Monitor: Virtual Memory column -
Java Real Memory @ 330 MB
Java Virtual Memory was 16,333 TB.
How could this be?
Running:
Safari
Mail
Pages
Activity Monitor open
Safari was incredibly slow today, as it has been a lot lately, but inconsistently. The other computers in the house had normal speed. I'm not a tech person. I attached a copy of the activity monitor - how it looks right now. I appreciate your thoughts and suggestions. Thank you.
bottom half

As Kappy says, the virtual memory information is mostly meaningless except to developers, and most of them do not really care.
In this case the Java VM usage is most likely a math error on the part of either the operating system, or Activity Monitor (when a 64-bit number goes negative, but is then displayed as an unsigned value, it can look like what you are seeing; ignore it).
What you ARE interested in is "Real Memory" usage, and who is using it. Look at those numbers in Activity Monitor.
If you want to see if pageout activity is affecting your performance, then start Applicaitons -> Utilities -> Terminal and run the command "sar -g 60 100" which will report pageout numbers once a minute for 100 minutes (adjust the numbers to suit your tastes). Mostly zero means no pages outs. Occassional spikes generally occur when starting an app or switching to an app which has been idle for awhile. Sustained pageouts starts to indicate a problem. High sustained pageouts means you could benefit from either having more RAM or running fewer concurrent applications.

Slow Lion - really high virtual memory use

Lion is running really slow on both my Mac Mini and my Mac Book Pro. Both systems are less than two years old, both with 4 GB of RAM. I checked Activity Monitor (right after restart, running Chrome, Mail, and iTunes only) and I get a VM size of 266.42 GB! Is this normal? And could this be the reason why it seems so dreadfully slow?

budusa,
Take a really deep breath and let it out. Relax. The universe is ok.
You have to step back at look at what your data is showing you.
Physical Memory: 4 GB
Virtual Memory: 266.42 GB < notice this larger than your physical.
Note the amount of memory used by your system, Wired, 675.4 MB. From what I understand so far of how OS X lion uses memory is that your system is using less a 1 gigabyte of memory.
Now note the amount of Inactive memory, 535.7 MB. I bet if you looked for the processes you used the amount they where accessing would equal that amount.
Also note your Active memory used, 1.52 GB. That was currently being used at the moment of your snapshot. Note also it could reflect was used recently also.
Ok, now look at your Free memory, 1.30 GB.
And the last part, the Used, 2.70 GB.
Do you notice something yet? Do you see it?
Ok, you have 4 GB of Physical RAM installed.
According to your information you used only, roughly, 2.70 GB of Physical RAM.
That means you used 2.70 of 4 GB of physical RAM.
Now if you notice your free memory, 1.30 GB.
If you take 2.70 - 4 = 1.30 < notice the answer, Free memory, 1.30 GB.
Basically your system used 2.70 GB of 4 GB to run the applications listed.
Look at your Virtual Memory, 266.42 GB. Notice it larger than 2.70 GB?
My understanding of how a 64 bit OS works, and the key item why it works better than 32 bit, is the Virtual Address Space. This is how the OS allows programs to run with out taking up all the physical memory. The programs go there to play as I like to say. Another way to look at it is do want only 4 GB's or 8+ Terabyte's of Virtual Address Space?
So, the Virtual Memory you see is ok and is normal.
Do not follow some advice I have seen here about switching programs to run in 32 bit mode. You defeat the purpose of 64 bit.
If you system is running slow then look at your installed programs. Note also that if you have a small hard drive it can cause sluggishness too.
I hope this helped. Maybe it confused. Sorry for making it long as I did.

Virtual Memory Allocation problem

When I run compile some source code in F90, I receive the following error message:
/usr/bin/ld: can't vm_allocate() buffer for output file of size 1699768758 ((os/kern) no space available)
What seems to be happening is that the system wants to allocate a large amount of memory. I have over 14 Gb of virtual memory available and over 2.5 GB of RAM available. Another forum ( http://www.archivum.info/gnu.gcc.help/2007-08/msg00083.html ) said the memory needs to be contiguous. I need to compile this, and I have sufficient memory to do so. What must I do to make this work?
I tried the following:
cd /usr/bin
sudo mv ld ld32
sudo ln -s ld64 ld
This did not solve the problem as now the error says:
ld64-62.1 failed: 32-bit pic-base out of range in atexitcommon from /usr/local/gcc/4.2.2-32/lib/gcc/powerpc-apple-darwin8.9.0/4.2.2/crt3.o

I'm assumming that your operating system is Windows NT 4, Windows 2000, or Windows XP. For others reading, this is a link to the above referenced KnowledgeBase article.
NI-DAQ does not require you to invoke the VirtualLock function. NI-DAQ will page lock the buffer passed to the SCAN_Start function. It is possible that, since the buffer has already been paged locked by your code, that there are insufficient resources (i.e., kernel-mode Page Table Entries (PTEs)) available for NI-DAQ to page lock the buffer.
Remove the invocation of the VirtualLock function and see if that helps.
Geoffrey Schmit
Fermi National Accelerator Laborary

How to check virtual Memory Info on Solaris 10

Hello,
Can anybody please tell me command used to calculate or check virtual memory configured on my solaris 10 T2000 servers.
From Solaris Management Console I did get this information but I was wondering which command it used to extract this information.
Thanks in Advance

Here is vmstat 2 2 s output on my machine
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy cs us sy id
0 0 0 18874528 3257488 19 47 32 3 3 0 1 2 3 0 0 964 16476 1115 1 1 98
0 0 0 21063104 5781064 36 135 28 0 0 0 0 0 0 0 0 1509 5049 1946 1 1 99
It says system has 21063104KB of Swap but my solaris management console shows
Physical Memory: 8184 MB
Virtual Memory : 21992 MB
Virtual Memory used : 7%(1537/21992)20455 MB Available
I want to know how Solaris Management Console calculated my Virtual Memory.
Thanks

Memory used by one instance on Windows 2003 server

Hi
I'm looking for a single query able to return the global allocated memory for my instance.
Actually i must search in Windows process and compute the global memory on oracle.exe process with VM Size + Used memory.
I'd like to find this information only from Oracle query.
Thanks in advance
Den

It doesn't work !
Actually system say:
- 1200 Mo for Memory Used
- 1320 Mo for virtual memory used
Show SGA or the other request doesn't wortk ! Show sga or v$sgastat don't include informations on PGA (first) !!!
1) Show SGA:
Fixed Size 456892
Variable Size 1107296256
Database Buffers 100663296
Redo Buffers 405504
2)
SELECT ROUND(SUM(bytes)/1024/1024,2)||' MB' total_sga,
ROUND(SUM(DECODE(name,'free memory',bytes,0))/1024/1024,2)||' MB' free
FROM v$sgastat;
TOTAL_SGA FREE
608,81 MB 250,16 MB
1 row selected.
Regards
Den

SIMBL Agent--what is it and why is its virtual memory footprint 30gb+?

Loud fan noises. Discovered something called SIMBL Agent was using well over 30 gigabytes of virtual memory. 2nd place was Safari at only 1.5!
So, the question of the day is
What is SIMBL Agent?
What is it doing?
And should I be concerned at 30gb+ of space being consumed?
(OK, so thats 3 questions of the day! )

Thanks for the reply! My understanding is that virtual memory uses hard drive space... so 30 GB+ is excessive, right? Or, has "virtual memory" taken on a new meaning in Leopard, or OS X.
Virtual Memory information is mostly meaningless, except to software developers and very few of them.
Virtual Memory is often times filled with holes of unused address space. The application code is mapped into this space, and the disk used for this is the actual executable code.
All the shared libraries and application frameworks that the application uses are mapped into the virtual address space, often times with empty gaps in the address space. Again the disk used are the shared library and framework files.
If the application has loadable modules, such as browser or photoshop add-ons, then these get mapped into virtual address space. And the disk for this comes from the add-on files. Also these add-ons are most likely mapped into the virtual address space with gaps between them and other code.
Then comes the data used by the application. This comes in different flavors. Some might be memory mapped files, where an explicit file accessed as if it is part of memory (vs doing read/write operations). If the file is huge, then the virtual address space needed to map it will be large as well. Or an application can memory map a bunch of smaller files, but if you have enough of them then that adds up to a lot of virtual address space.
NOTE: when memory mapping a file, the application has the opiton of specifying where in the virtual address space it would like the file to be mapped. So it is very possible for an application to specify they would like to map a file at the 29GB address offset, and everything from the lower address up to the now memory mapped file at 29GB is empty address space. No RAM and no disk associated with this gap.
So far everything I've mentioned does not need swapfile space, as real files are used for all of the above.
The application stack is mapped into the virtual address space. Since the stack grows as the application calls subroutines, it is necessary to leave lots of unmapped addresses for the stack to grow into. Since most stacks to not grow that big, a lot of that virtual address space is never mapped to RAM nor disk.
And there is the actual data being worked on by the application. This could be local variables used as part of the program flow control, or it could be buffers used to hold data the program is working on. This data would get allocated potential space in swapfiles (/var/vm/swapfile*), however, if the operating system does not run out of RAM, then it will not need to actually create any additional space in the swapfiles.
Now I have no idea what your 30GB application is doing with all of its virtual address space. And it is possible that it is using all of it, and forcing RAM to /var/vm/swapfile* files. If you really want to know, you can always use the vmmap command from a terminal (man vmmap).
However, in my experience, no one really cares. All they really want to know is whether their Mac is spending a lot of time paging. And for that I like to use Applications -> Utilities -> Terminal
sar -g 50 100
The command will display pageout information once a minute for 100 minutes (adjust values to suite your tastes).
Now you do your normal work. When you come back an look at the sar output, if it is mostly zero or some small values, or a momentary burst, then you really do not need to worry. If you have sustained high values, or if you notice a the Mac is slow at the same time you getting high pageout numbers, then you could benefit from more memory, or running fewer apps at the same time.

Low Virtual Memory-Satellite M115 S3154

I constantly have Low Virtual Memory problem and have to clear and reset my paging file regularly. System memory 504 Mb RAM on Satellite M115 S3154. Have paging file min & max set at 1000. Upon clearing paging file and reset everything runs fine for a while then slowly grinds to a halt until I can't do anything until I reset again. HAve also tried running System Mechanic to manage virtual memory, but problem continues. Any help would be greatly appreciated. I seem to have already done what previous posts have suggested. Thanks!
Solved!
Go to Solution.

stropical wrote:
When you say increase the physical memory what I have to do? Thanks
Increasing the physical memory means increasing the RAM, in this instance. Your computer spec says 2GB but upgradeable to 4GB. Here is a one person’s recommendation on how much virtual memory should be allocated, i.e., 2-3X your memory. (Some will recommend min if you have enough memory.)
http://www.gilsmethod.com/how_to_optimize_windows_vista_swap_file

High Virtual memory usage when using Pages 2.0.2

Hey there,
I was just wondering whether there had been any other reports of unusually high memory usage when using Pages 2.0.2, specifically Virtual memory. I am running iWork 06 on the Mac listed below and Pages has been running really slowly recently. I checked the Activity Monitor and Pages is using hardly any Physical memory but loads of Virtual memory (so much so that the Page outs are almost as high as the Page ins (roughly 51500 page ins / 51000 page outs).
Any known problems, solutions or comments for this problem? Thanks in advance

I don't know if this is specifically what you're seeing, but all Cocoa applications, such as Pages, have an effectively infinite Undo. If you have any document that you've been working on for a long time without closing, that could be responsible for a large amount of memory usage.
While it's good practice to save on a regular basis, if you're making large amounts of changes it's also a good idea to close and reopen your document every once in awhile, simply to clear the undo. I've heard of some people seeing sluggish behavior after working on a document for several days, which cleared up when the document was closed and reopened.
Titanium PowerBook Mac OS X (10.4.8)

How can one use the physical memory of our system rather than virtual memory while running Labview?

We have a Windows NT system with 2 Gb of physical memory and would like to utilize the RAM fully using Labview. What usually occurs is that Labview uses a tremendous amount of page file space while a majority of the physical memory is unused. Is there a way to configure Labview (or our system) to overcome this problem? It seems that our processes would be much faster if they were mainly using the physical memory. Is it possible to trick the system, by creating a RAM disk and allocating this as virtual memory space?

> We have a Windows NT system with 2 Gb of physical memory and would
> like to utilize the RAM fully using Labview. What usually occurs is
> that Labview uses a tremendous amount of page file space while a
> majority of the physical memory is unused. Is there a way to
> configure Labview (or our system) to overcome this problem? It seems
> that our processes would be much faster if they were mainly using the
> physical memory. Is it possible to trick the system, by creating a
> RAM disk and allocating this as virtual memory space?
LabVIEW the application doesn't know anything about physical versus
virtual memory. LV asks the OS for general purpose memory and goes
from there.
Drivers like DAQ and IMAQ will have a combination of general user
memory and page lo
cked physical buffers.
I'm not up on the details, but this is something that the OS is in
control of, so that is where you need to look for the solution. One
of the things to look at is the disk cache. By default, I think that
NT takes a percentage of the RAM to use for disk cache. With that
much RAM, this is probably unnecessary and is using too much.
Similarly, the OS may be attempting to keep the working set size
to a fraction of the total RAM to reserve space for other things.
It doesn't make sense to me, but then I don't work for MS.
Greg McKaskle

Why does my mac use virtual memory when I still have free physical memory?

I have a 2011 i7 quad core mac, I was hoping it would scream. Most of the time it does. However when trying to edit within FCPX I get a very disappointing experience with many pauses and pin wheels if I don't close every single other program.
I have 8GB of physical memory and when i'm experiencing these problems I see that i still have 1-2 gb of physical memory free or inactive. At the same time FCPX is only using 2gb of memory. I just happened to keep an eye on the VM page in/outs and noticed them going up.
Right now i'm doing some browsing and emailing, that's about it.. its sat with over 4gb of memory free or inactive and yet still the page in/outs is still going up occasionally. It's currently at over 2 million page ins, and over 1 million page outs.
So with so much physical memory free why is this happening!? At the moment the mac feels nice and responsive, but if i start trying to use FCPX i'll start to experience these slowdowns, stalls... whenever i see these i see my main hdd is being accessed whilst the pinwheel is displayed.. I mean i get it, its VM, the hdd is too full, a bit fragmented perhaps, its stalling... but i've got gigs of memory sitting free or inactive... why wont the OS use it!!!
Would my experience improve if i took the plunge and got 16gb of memory instead of 8gb!?
Thanks for your help!

Because without virtual memory, managing computer RAM is a royal pain in the ...
Virtual memory cost you nothing, and gains you huge benefits, even if you do not notice it
What cost you is when you need more real RAM than is available, and things are thown out of RAM, either back to the original file it came from (Read Only information), or pushed out to the swapfiles (/var/vm/*). Then the system has to wait for slower disk access. But even this is better than not being able to run the apps until you quit something else.
(speaking as someone that starting his professional life working with 1" punch paper tape, 80 columns cards, 7-track and 9-track mag tapes, 1MB disks (you heard me right 1 Megabyte), etc..., and trust me when I tell you that virtual memory is a god send to software development).
There are a lot of problems running a modern operating system with out virtual memory. For example all the shared libraries and frameworks that provide services to an application would all need to be compiled into the application, which means every application gets bigger and instead of having a single copy of the shared library or framework, you would have dozens of copies wasting your RAM.
Without virtual memory, you would be required to find a contiguous chunk of RAM to run your application. Think of this like going out to dinner by yourself, you can find any available table, but if you go to dinner with your extended family, you need a table for 10 to 15, and if you are going to dinner with your high school graduation class, you will need hundreds of seats all next to each other and a very large table. In the later situations you have to wait until the resturante has enough contiguous space, which means you have to wait until other diners finish. There may be lots of empty tables, but they are not together, and your group wants/needs to sit together. Virtual memory allows gathering any 4K chunk of RAM, building a virtual memory map for all those random 4K chunks, and make it look like one big contiguous chunk of RAM, so you can run your application right away, no waiting.
Going back to shared libraries and frameworks. This code will need to have addresses resolved so they branch to the correct locations during execution, and it will need to have addresses resolved on where its program variables are located in RAM. Using virtual memory, you can local a shared object into RAM, then place it in everyone's virtual memory map at the exact same RAM address. This means everyone can use the exact same code, and since everyone is using it at the same RAM address, it makes life so much easier for the operating system (translation, less work, less wasted CPU time, faster execution).
When a program wants to grow, for example a web browser loading a web page (and its images) into RAM, it needs to allocate additional RAM. In the contiguous RAM model, you need to get control of the RAM that imediately following your program, but if that RAM is being used by someone else, you have to wait until that program goes away.
Virtual memory provides protection from another program looking at and modifying your program's RAM. Malware would just love for virtual memory to go away.
You want virtual memory. What you do not want is excessive paging activity.
If you are concerned, then you can launch Applicaitons -> Utilities -> Terminal. Once you have a terminal command prompt, enter the following command:
sar -g 60 100
which will tell you the number of 4k pages written to /var/vm/pagefile ever minute for 100 minutes (modify the numbers to suit your tastes). You can then go about your normal usage, and come back later to see how much you have been using the pagefiles. If you have mostly zeros, and an occasional small burst, this is noise, and not worth worrying about. If you have sustained pageout activity, with higher numbers, then you should either consider running less things all at the same time, or looking for an application that is being greedy with its memory use (or has a memory leak), OR get more RAM for your Mac if you need to do all those things at once.
But do not complain about virtual memory. Life would be much worse without it. Then again if you have a better idea, write a research paper, and get operating system vendors (as well as hardware vendors) to implement your ideas. I am serious, as I've seen many accepted computing ideas be overturned by good new ideas.

After upgrading to Lion, the Finder will increase the virtual memory while using external HD

after upgrading to Lion, the Finder will increase the "virtual memory size" while using external HD. I check it using Activity Monitor and found that "Finder" wll increase the "Virtual Memory Size" and "Private Memory Size". and then after a while. the system told me my HD is full and hanging there.
My External HD is 1TB. and I have installed new version Paragon NTFS for Mac V.9.01. I am not sure it is related or not.
but I attached my external HD. the "Finder" will start to increase the "Virtual Memory Size".
please help out this case and solve this issue.
thank you !!!

The one has nothing to do with the other. Virtual memory size and Private memory size are used by developers to analyze their software. it has nothing to do with anything related to the user.

Using all memory, including virtual memory

For some reason Final Cut Pro 10.0.3 doesn't get past the loading screen, it's worked fine until yesterday - havent done any system updates, or program upgrades. When I start the program, I can see the square that says FCP, and I can see the interface loading in the background, but it never comes out of that loading screen. It uses all 18 GB of my memory, even after a restart, then it goes to my hard drive for more virtual memory. I have 37 GB free, it uses all that and then 'pauses' according to my Force Quit screen. I'm in a big rush to get this fixed - I work at a church, and this is the Friday before Easter, which is a really big weekend, and of course, people want to see these videos before the end of the day, or earlier.... Any help would be much appreciated!!

Thanks innocentius. I found the culprit, after much troubleshooting and help from Apple. When I loaded Final Cut, the loading screen would always get stuck 'loading the 'easter' event'. There was a PSD file in the easter event that I was working on yesterday that suddenly didn't like Final Cut. In the PSD, I had created a very complex vector shape, and I think Final Cut was trying to render the vector without success. But, it could have just been a corrupted PSD. Either way, I went to the Final Cut Events folder, found the PSD, moved it somewhere else, and voila, everything is back to normal!

Why all of a sudden I'm getting pop up's stating High CPU performance for plug in, anmd Performance allert, high memory use by firefox, never had these problems till I upgraded to the latesty Firefox offered???????

since I upgraded to the latest version of fire fox, I'm getting pop ups stating, And I quote! Performamce Alert,high memory use by Fire Fox, and High CPU Performance for plug in by Fire fox, never had this problem with the last version of Fire Fox, this has apparently is slowing down mu computer, and I was much better off with the old one

since I upgraded to the latest version of fire fox, I'm getting pop ups stating, And I quote! Performamce Alert,high memory use by Fire Fox, and High CPU Performance for plug in by Fire fox, never had this problem with the last version of Fire Fox, this has apparently is slowing down mu computer, and I was much better off with the old one

CFMX 6.1's Virtual Memory Use problem!!

Similar Messages

Maybe you are looking for