Solaris 10 Zones (global/local) /proc fs is 97% full

Question

Hi Folks,
sorry in case of this thread has been discussed already but I couldn't find any info regardless.
I've a solaris 10 server on which have installed 3 zones and 2 of these zone giving me continuously alert regarding disk space.
I know when /proc fs is full is normal but in my case when I am looking into disk space of solaris zone:
-bash-3.00# df -ak
Filesystem 1024-blocks Used Available Capacity Mounted on
/ 70220321 67141358 2376760 97% /
/dev 70220321 67141358 2376760 97% /dev
proc 0 0 0 0% /proc
ctfs 0 0 0 0% /system/contract
mnttab 0 0 0 0% /etc/mnttab
objfs 0 0 0 0% /system/object
swap 18973688 224 18973464 1% /etc/svc/volatile
/platform/SUNW,Sun-Fire-T200/lib/libc_psr/libc_psr_hwcap1.so.1
70220321 67141358 2376760 97% /platform/sun4v/lib/libc_psr.so.1
/platform/SUNW,Sun-Fire-T200/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1
70220321 67141358 2376760 97% /platform/sun4v/lib/sparcv9/libc_psr.so.1
fd 0 0 0 0% /dev/fd
swap 19074320 100856 18973464 1% /tmp
swap 18973488 24 18973464 1% /var/run
-hosts 0 0 0 0% /net
As my understanding this issue filling up / fs as well, which totally strange to me.
Could someone advise pls why this mess?
Thanks

Answer 1

ostendali wrote:
Thanks for reply.
I've already ran du command, sorry for haven't posted here:
-bash-3.00# du -ash /* |sort -nr |grep GA more useful command is:
du -kd / | sort -n | tail -30
Why is it better?
- The "/" argument tells it to look at /. Your command says "look at all the stuff that is in /*". Note that /* does not include files and/or directories that start with ".". It does include file systems that are mounted on subdirectories of /.
- The -k option makes it so all of the units are the same. Thus, sort works better and you have an easy visual of the difference in space by the number of digits in the size.
- The -d option makes it so that it doesn't look in other file systems. You are trying to figure out why / is full. /proc is a different file system that is a representation of a bunch of kernel data structures. /proc takes no space on disk.
As you can see the fs is filled up from processes /proc and vhost/proc is not filling up any disk-based file system. When you read a file that is in /proc, you are either reading a representation of kernel data structures that are stored in memory or you are being referred off to a file in a different file system.
Notice that between your posts, you show that the root file system is about 70 GB in total size and /vhosts seems to be using 51 GB. That means that you are looking for less than 20 GB. If the du command I give above shows that / is using a lot less than df says it is using, try this command:
# find /proc/*/fd -type f -links 0 | xargs du -k | sort -n
258     /proc/12328/fd/24
Each line represents a file that has been removed but something still has it open. Because it is removed (no longer "linked" into a directory) it is invisible to du when traversing the normal file system. /proc provides a special view into it. In my example pid 12328 has file descriptor 24 open to a removed file. It is 258 KB in size. When pid 12328 dies, the space used by that file will be freed.
You can typically see what is in such a file using a command like
# less /proc/12328/fd/24
Many times you will see that it is a log file that is being written to and will never be read (because it is removed, after all). In such cases, you can free the space immediately with:
# cp /dev/null /proc/12328/fd/24