ANM 5.2 Virtual Appliance Internal filesystems problem

Question

Good day,
We have an ANM Virtual Appliance, version 5.2, were we login and can go no further. This was working for fine for approximately two and half weeks. We created a backup and re-loaded the system via CLI with the same result. We logged in again via SSH and we have noted the following:
cscoanmsa/admin# sh disk
temp. space 4% used (141244 of 4951688)
disk: 7% used (353916 of 5935604)
Internal filesystems:
  warning - /var is 100% used (89219000 of 89258112)
cscoanmsa/admin# sh application status ANM
% Error finding status information for the application: ANM
cscoanmsa/admin#
cscoanmsa/admin#
cscoanmsa/admin#
cscoanmsa/admin#
cscoanmsa/admin# sh application status ANM
The monit daemon 4.9 uptime: 23m
Process 'dcm'                       initializing
Process 'dal'                       initializing
Process 'ip-disc'                   Execution failed
Process 'licman'                    Execution failed
Process 'anm-fw-mon'                running
Process 'mysql'                     Execution failed
System 'cscoanmsa'                  running
Java Processes:
licman    : Stopped (None)  [N/A]
dcm       : Stopped (None)  [N/A]
dal       : Stopped (None)  [N/A]
ip-disc   : Stopped (None)  [N/A]
Other Processes:
anm-fw-mon: Running (3641)  [2012-06-15 15:48:55]
mysql     : Stopped (None)  [N/A]
cscoanmsa/admin# sh mem
total memory:    4148032 kB
free memory:     3616320 kB
cached:           342332 kB
swap-cached:           0 kB
Is there any way to access and clean out the /var directory from the CLI. is this achieved simply via the "delete" command with the full path ?
I believe this may be the issue, and we need to resolve this before we can move forward.
Any assistance or guidance would be greatly appreciated.
Thank you in advance.
Paul.

Answer 1

We've seen this on ANM 5.2.1 as well. In our case it was that /var/log/messages was taking up nearly 85GB on the var volume. ANM, at least in 5.2.1, doesn't seem to rotate this log file.
The way we fixed it was to mount the var volume using a Linux live CD like Knoppix and clear our /var/log/messages. Very unsupported, but gets you out of a hole:
Boot the ANM machine with Knoppix (KNOPPIX_V7.2.0DVD-2013-06-16-EN.iso)
As root run:
pvs (to work out LVM physical volumes and volume groups)
PV        VG
/dev/sda2 smosvg
vgchange -ay smosvg (to activate the volume group)
lvscan (to work out what logical volumes are available)
ACTIVE '/dev/smosvg/varvol'
mount /dev/mapper/smosvg-varvol /mnt
echo > /mnt/log/messages
umount /mnt
reboot