Windows 2000 Server Cluster Restars

Question

Hi,
Last friday one of server's cluster restarted without any logical reason. We found a minidump and a dr Watson log. From the user dump we have this information:
Microsoft (R) Windows Debugger Version 6.3.9600.17298 X86
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [E:\6 marzo error backup\user.dmp]
User Dump File: Only application data is available
************* Symbol Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 2000 Version 2195 UP Free x86 compatible
Product: LanManNt, suite: Enterprise TerminalServer SingleUserTS
Machine Name:
Debug session time: Sat Mar  7 01:14:08.144 2015 (UTC + 1:00)
System Uptime: 127 days 11:06:13.061
Process Uptime: not available
The call to LoadLibrary(ntsdexts) failed, Win32 error 0n2
    "El sistema no puede encontrar el archivo especificado."
Please check your debugger configuration and/or network access.
(688.bbc): Access violation - code c0000005 (!!! second chance !!!)
eax=4da59480 ebx=4db2c148 ecx=00000000 edx=784afca0 esi=4e065640 edi=4db2c0f8
eip=7846f113 esp=0348fdb8 ebp=0348fdec iopl=0         nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=0038  gs=0000             efl=00000202
ntdll!RtlDeleteCriticalSection+0x61:
7846f113 8901            mov     dword ptr [ecx],eax  ds:0023:00000000=????????
0:026> !analyze -v
*                        Exception Analysis                                  
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                         
***    Type referenced: ntdll!_PEB                                   
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                         
***    Type referenced: nt!IMAGE_NT_HEADERS32                         ***
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for Rtvscan.exe -
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for GEDataStore.dll -
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for SRTSP32.DLL -
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for vpmsece.dll -
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for SymProtectStorage.dll -
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                         
***    Type referenced: nt!_CONTEXT                                   ***
The call to LoadLibrary(ntsdexts) failed, Win32 error 0n2
    "El sistema no puede encontrar el archivo especificado."
Please check your debugger configuration and/or network access.
The call to LoadLibrary(ntsdexts) failed, Win32 error 0n2
    "El sistema no puede encontrar el archivo especificado."
Please check your debugger configuration and/or network access.
The call to LoadLibrary(ntsdexts) failed, Win32 error 0n2
    "El sistema no puede encontrar el archivo especificado."
Please check your debugger configuration and/or network access.
GetUrlPageData2 (WinHttp) failed: 12007.
FAULTING_IP:
ntdll!RtlDeleteCriticalSection+61
7846f113 8901            mov     dword ptr [ecx],eax
EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 7846f113 (ntdll!RtlDeleteCriticalSection+0x00000061)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000001
   Parameter[1]: 00000000
Attempt to write to address 00000000
CONTEXT:  00000000 -- (.cxr 0x0;r)
eax=4da59480 ebx=4db2c148 ecx=00000000 edx=784afca0 esi=4e065640 edi=4db2c0f8
eip=7846f113 esp=0348fdb8 ebp=0348fdec iopl=0         nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=0038  gs=0000             efl=00000202
ntdll!RtlDeleteCriticalSection+0x61:
7846f113 8901            mov     dword ptr [ecx],eax  ds:0023:00000000=????????
FAULTING_THREAD:  00000bbc
DEFAULT_BUCKET_ID:  NULL_POINTER_WRITE
PROCESS_NAME:  Rtvscan.exe
ERROR_CODE: (NTSTATUS) 0xc0000005 - La instrucci n en 0x%08lx hace referencia a la memoria en 0x%08lx. La memoria no se pudo %s.
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - La instrucci n en 0x%08lx hace referencia a la memoria en 0x%08lx. La memoria no se pudo %s.
EXCEPTION_PARAMETER1:  00000001
EXCEPTION_PARAMETER2:  00000000
WRITE_ADDRESS:  00000000
FOLLOWUP_IP:
RPCRT4!MUTEX::Free+c
77142496 c3              ret
APP:  rtvscan.exe
ANALYSIS_VERSION: 6.3.9600.17298 (debuggers(dbg).141024-1500) x86fre
LAST_CONTROL_TRANSFER:  from 77142496 to 7846f113
PRIMARY_PROBLEM_CLASS:  NULL_POINTER_WRITE
BUGCHECK_STR:  APPLICATION_FAULT_NULL_POINTER_WRITE
STACK_TEXT: 
0348fdb8 7846f113 ntdll!RtlDeleteCriticalSection+0x61
0348fdf4 77142496 rpcrt4!MUTEX::Free+0xc
0348fdfc 77135a63 rpcrt4!LRPC_CASSOCIATION::~LRPC_CASSOCIATION+0xe8
0348fe14 77135973 rpcrt4!LRPC_CASSOCIATION::`vector deleting destructor'+0x8
0348fe1c 77142978 rpcrt4!REFERENCED_OBJECT::FreeObject+0xb
0348fe24 771422b0 rpcrt4!REFERENCED_OBJECT::RemoveReference+0x18
0348fe2c 771358be rpcrt4!LRPC_CASSOCIATION::RemoveBindingHandleReference+0x37
0348fe34 77135815 rpcrt4!LRPC_BINDING_HANDLE::~LRPC_BINDING_HANDLE+0x42
0348fe4c 771357b8 rpcrt4!LRPC_BINDING_HANDLE::`vector deleting destructor'+0x8
0348fe54 771357a4 rpcrt4!LRPC_BINDING_HANDLE::BindingFree+0x32
0348fe64 77142caa rpcrt4!RpcBindingFree+0x4e
0348fe74 7cf460e6 ole32!CChannelHandle::~CChannelHandle+0xf
0348fe80 7cf460c7 ole32!CChannelHandle::Release+0x1c
0348fe88 7ce7cc6c ole32!OXIDEntry::ExpireEntry+0x1d
0348fe94 7ce8fca2 ole32!COXIDTable::FreeCleanupEntries+0x6e
0348ff10 7ceb5f93 ole32!COXIDTable::GetOxidsToRemove+0xce
0348ff2c 7ceb5e6e ole32!CROIDTable::ClientBulkUpdateOIDWithPingServer+0x1a5
0348ff90 7ceb5d74 ole32!CROIDTable::WorkerThreadLoop+0x37
0348ff98 7ce97f2c ole32!CRpcThread::WorkerLoop+0x22
0348ffb0 7ce97f8b ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x1a
0348ffbc 7945b388 kernel32!BaseThreadStart+0x52
STACK_COMMAND:  .ecxr ; kb ; dps 348fdb8 ; kb
SYMBOL_STACK_INDEX:  1
SYMBOL_NAME:  rpcrt4!MUTEX::Free+c
FOLLOWUP_NAME:  MachineOwner
MODULE_NAME: RPCRT4
IMAGE_NAME:  RPCRT4.dll
DEBUG_FLR_IMAGE_TIMESTAMP:  42a06799
FAILURE_BUCKET_ID:  NULL_POINTER_WRITE_c0000005_RPCRT4.dll!MUTEX::Free
BUCKET_ID:  APPLICATION_FAULT_NULL_POINTER_WRITE_rpcrt4!MUTEX::Free+c
ANALYSIS_SOURCE:  UM
FAILURE_ID_HASH_STRING:  um:null_pointer_write_c0000005_rpcrt4.dll!mutex::free
FAILURE_ID_HASH:  {73dc9e1e-67d1-8f64-5cc1-10543104d0b2}
Followup: MachineOwner
Could anybody help us to understand what happened? We have Symantec 11 Client in this server.
Best Regards and thanks in advance.

Answer 1

I see that the process
Rtvscan.exe is highlighted in the dump. This belongs to Symantec: http://www.neuber.com/taskmanager/process/rtvscan.exe.html
Also, I see multiple access violation errors.
Please note that Windows Server 2000 is no longer supported by Microsoft. However, this OS version seems to be supported for the Symantec Version you use. If the problem gets reproduced again then I would recommend removing Symantec AV and trying again.
If this fixes the issue then you might want to contact Symantec Support for assistance.
This posting is provided AS IS with no warranties or guarantees , and confers no rights.
Ahmed MALEK
My Website Link
My Linkedin Profile
My MVP Profile