Space on boot volume keeps disappearing - crisis

Question

One of my customers has a Mac Pro running OSX Server 10.4.9. It was installed last October and was running fine until a few days ago. It's running afp, firewall, nat, dhcp, dns, mail, print, web (for webmail) and windows services.
The only non-Apple things running on the machine are Now Contact/Up To Date server, Timbuktu and Retrospect. I have the same combination running on the same model hardware elsewhere without problems.
The boot volume had about 85gb free and mail has its own dedicated volume. The boot volume is not shared or used by anyone.
Then I had a call from the customer saying mail wouldn't send out. When I looked at the machine, the problem was that there was exactly 0 bytes available on the boot volume and postfix couldn't find queuing space.
I deleted all the rubbish - dmg files from various installs, other files 'parked' on the dekstop etc. and got back about 8Gb. I made a note to look at the machine on my next visit and thought that was it.
The 8Gb disappeared within a couple of hours. This time I looked a bit harder (partly because there wasn't any rubbish left to delete). Naturally, there were no files visible to finder that would account for the space used. My frantic but fairly superficial spin through ls -l in a terminal didn't show me any 'new' folders I didn't recognise. Looking at Activity Monitor, I saw that there was almost constant disk write activity going on, even though the users were apparently 'hands off' their machines. The only process using any cpu time was the Now Contact event server - so I pounced on it, thinking it was bound to be the culprit, being a buggy piece of software at the best of times. When I killed the process, the disk writing stopped.
I got on the phone to Now Software and they told me every place they THOUGHT the program wrote to - nothing. I restarted the machine and made sure the Now Software stuff wasn't running. Everything seemed stable, except I still hadn't figured out where the lost space was. However, it started again the next night.
Over the past few days I've been fightng a running battle to keep space above zero. I've now deleted everything in applications, utilities, library etc that I can live without and I've even gone as far as clearing every log file, but the space keeps disappearing. I'm now down to 150Mb and there's nothing left to delete so I have to find out where it's all really going to.
I have done a file search on all the usual combinations - visible/inivisible, created/modified/bigger than/smaller than etc but nothing stands out.
Is the server being attacked? I've shut down windows and web, and SSH was always switched off. I've been watching system.log without seeing anything obvious happening. The server has been restarted many times so it's not a virtual memory thing (I think).
I'm depsarate. Where should I be looking?
Justin

Answer 1

Good grief. First of all, thank you very mich for introducing me to du.
Here's what I I learned (and it's obviously an accident since no-one on site would know how to do this deliberately):
There is a volume (a real hard drive) called Archive. It's got about 90-something Gb of stuff on it. There are also three other volumes but here's the thing about Archive. I failed to notice in my earlier quick scan that /volumes contains an item called 'Archive' AND an item called 'Archive 1' - each being about the same size. When, in the finder, I deleted a folder from Archive that they could live without, it was Archive 1 that got smaller as seen by du.
going cd /volumes/archive shows me just one folder (which doesn't exist on the real volume but did once).
going cd '/volumes/archive 1' shows me what appears to be all the folders that are supposed to be there (on Archive). I haven't trawled right through and done an entry-by-entry check.
On impulse, I've just loaded Workgroup Manager and looked at Sharing. Under 'All', the left hand pane shows the volumes I'd expect to see (and nothing else). Buuuuuuttttt....... under 'Share Points', the volume we know as Archive is called 'Archive 1' - with a custom afp name of 'Archive'.
What have they done? I don't know what to delete at this point; and the trouble is that I suspended backups last week (because the catalogue files were amongst the things I removed to get some space back). I'll have to do a manual backup of Archive, verify it, and then try deleting something and see what happens.
What I'd really like to know is what happened and how. For example, would anyone like to try and build a hypothesis whereby the server was brought up with Archive off-line, then, after some other event, Archive subsequently came on line? Is this how you might end up with two apparent volumes with the same name?
Justin