What can I do when it says "your system has run out of application memory?

Question

What can I do when it says "your system has run out of application memory?"

Answer 1

The message has nothing to do with low disk space.
There is excessive swapping of data between physical memory (that is, the memory chips on the logic board) and virtual memory (one or more files on the startup volume.) That activity is relatively slow and causes the whole system to be less responsive. It can happen for two reasons:
          A long-running process with a memory leak (a kind of bug)
          Not enough memory for your usage pattern
Tracking down a memory leak can be difficult, and it may come down to a process of elimination.
These instructions are for OS X 10.9 ("Mavericks.") The procedure may be slightly different for earlier versions of OS X.
When you notice the swap activity, open the Activity Monitor application and select All Processes from the View menu, if it's not already selected. Select the Memory tab. Click the heading of the Real Mem column in the process table twice to sort the table with the highest value at the top. If you don't see that column, select
          View ▹ Columns ▹ Real Memory
from the menu bar.
If one process (excluding "kernel_task") is using much more memory than all the others, that could be an indication of a leak. A better indication would be a process that continually grabs more and more real memory over time without ever releasing it. Here is an example of how it's done.
The processes named "Safari Web Content" render web pages for Safari. They use a lot of memory and may leak if certain Safari extensions or third-party web plugins are installed. Consider them prime suspects.
Another process often implicated in memory leaks is "inkjet4" or "inkjet8," which is a component of the HP printing software. If it's present, force-quit the process in Activity Monitor to solve the problem temporarily. Empty the print queues in the Printers & Scanners preference pane (which has a slightly different name in each recent version of OS X.) If you don't use an HP printer, remove the software. Otherwise, if the problem is recurrent, update the software (which may not help) or contact HP support.
"Wired" memory should be a small part of the total. That memory is not swapped, but it makes less physical memory available which may then result in swapping. If you have a lot of wired memory, that's usually an indication of a memory leak in a third-party program that modifies the operating system at a low level. Ask for guidance in that case.
If you don't have an obvious memory leak, your options are to install more memory (if possible) or to run fewer programs simultaneously.
The next suggestion is only for users familiar with the shell. For a more precise, but potentially misleading, test, run the following command:
sudo leaks -nocontext -nostacks process | grep total
where process is the name of a process you suspect of leaking memory. Almost every process will leak some memory; the question is how much, and especially how much the leak increases with time. I can’t be more specific. See the  leaks(1) man page and the Apple developer documentation for details.