Xsan panic and recurring errors

Question

I had the volume go down in a panic last week. We were able to restart and get everything running but now have a persistent error which appears to be related. Both reference gethostbyname. I have dns setup for all of the systems on both the regular network and the metadata. One of the issues is that there is no secondary mdc, a problem the client refuses to rectify.
Process:         fsm [189]
Path:            /Library/Filesystems/Xsan/bin/fsm
Identifier:      fsm
Version:         ??? (???)
Code Type:       X86-64 (Native)
Parent Process:  fsmpm [179]
Date/Time:       2012-03-26 13:58:00.591 -0700
OS Version:      Mac OS X Server 10.6.8 (10K549)
Report Version:  6
Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Crashed Thread:  0  Dispatch queue: com.apple.main-thread
Application Specific Information:
PANIC: /Library/Filesystems/Xsan/bin/fsm "Server_comm_init GetHostByName failed" file server_comm.c, line 3455
Thread 0 Crashed:  Dispatch queue: com.apple.main-thread
0   libSystem.B.dylib                       0x00007fff848529da __pthread_kill + 10
1   libSystem.B.dylib                       0x00007fff848522fe pthread_kill + 83
2   fsm                                     0x00000001000ac22c 0x100000000 + 705068
3   fsm                                     0x00000001000ac787 0x100000000 + 706439
4   fsm                                     0x00000001000ea56a 0x100000000 + 959850
5   fsm                                     0x00000001000767f9 0x100000000 + 485369
6   fsm                                     0x0000000100030aa4 0x100000000 + 199332
7   fsm                                     0x0000000100000d94 0x100000000 + 3476
This error keeps cropping up in the current logs:
prod servermgrd[16235]: xsan: [16235/2112B0] ERROR: get_fsmvol_at_index: Could not connect to FSM because Cannot get host by name - No such file or directory

Answer 1

It might be worth while to clear your dns cache on the mdc if its also the dns.
~dscacheutil -flushcache
Is there anything else in the logs that seem relevent?
Have you run a permissions fix on the mdc anytime recently?
In your dns setting you have one host name for the public and one for the private?
internet - 192.168.1.100 mdc01."yourdomain"."internal"
private net - 192.168.2.100 mdc01-san."yourdomain"."internal"
Do you have any crostalk? (if only the private network is pluged in does it get a dhcp address? If you add the dns address to your static private port it can you get online?)