Questions around size of the ARC cache - Why does it sometimes drop so low?

Question

I have an old SUN X4500 running solaris 11.1. The box is used as a streaming content server and i`ve noticed some odd behavior with the ARC size recently.
If i run the arc_summary script it frequently reports the size as down around 64MB - 128MB.
ARC Size:
Current Size: 141 MB (arcsize)
Target Size (Adaptive): 64 MB (c)
Min Size (Hard Limit): 64 MB (zfs_arc_min)
Max Size (Hard Limit): 15343 MB (zfs_arc_max)
Kstat somewhat agrees
kstat zfs:0:arcstats:size | grep size | awk '{printf "%2dMB\n",  $2/1024/1024+0.5}'
82MB
What is weird is i`ve also seen the ARC up in the multi-gb range and i`m curious what causes it to drop to such a low watermark.
According to TOP there is a lot of free memory sitting idle, and would probably be better used as ARC:
Memory: 16G phys mem, 11G free mem, 2048M total swap, 2048M free swap
Any thoughtsÉ

Answer 1

These are thoughts and postulations (and guesses) ... some gathered from recent postings.
The zfs_arc_cache will be sized between zfs_arc_min and zfs_arc_max as Solaris sees best fit especially with regards to the needs of other processes.
However if some pages in the cache have not been accessed recently, and he cache is above its target size, then they will be discarded until the cache reaches its target size.
Now as pointed out by http://www.cuddletech.com/blog/pivot/entry.php?id=979 .... the target is adaptive.
Now I suspect the target size will go up if you have just copied a number of large files, or done a full backup ( as will the amount in the zfs cache), however if will probably do down if theses files are not accessed in a while.
In Solaris 10 I dont think the target was implemented and the cache would grow up to 85% of physical ram if left unchecked. And this could cause problems for applications checking the amount of free real/virtual memory before startup; or even if they attempted to grab too much memory too quickly which might cause a crunch shortage and application startup fail.
I think strategy changed with Solaris 11 to attempt to address some of these issues (for all I know it may be backported to a later solaris 10 update).
And if you know things about your system better than the cache optimizer ... and you may because you may be able to extrapolate whereas it is essentially only able to interpolate ... then you may wish to tune zfs_arc_min
Please note this post is not authorative .... consider it more as thoughts and postulations and it may contain errors and misguidance.