Solaris Logical Disks (Filesystems) not being discovered. WinRM CIM_ERR_INVALID_CLASS Error.

Question

Hello,
currently I am testing the Solaris monitoring with different kind of Solaris servers. We have one single system, a RAC database cluster of two servers, and one "Global-Zone" server hosting a "Non-Global-Zone". We're using SCOM
2012 SP1 with Agent Version 1.4.0-906 (sparc).
My problem is, that only on one of the 5 servers the Logical Disks (Filesystems) are discovered correctly. All Solaris systems are using ZFS.
The one that works just fine is the single system. In Health Explorer under Hardware Availability Rollup I can find every filesystem that is also configured in the /etc/vfstab. A check with WinRM from the Management Server works fine (winrm enumerate hxxp://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/SCX_FileSystem?__cimnamespace=root/scx
-auth:basic -remote:https://<servername>:1270 -username:scomuser -password:<pass> -skipCAcheck -skipCNcheck -encoding:utf-8)
The same WinRM command does not work for the other servers. The WinRM command failes with following error code:
Fault
    Code
        Value = SOAP-ENV:Sender
        Subcode
            Value = wsa:DestinationUnreachable
    Reason
        Text = CIM_ERR_INVALID_CLASS
    Detail
        FaultDetail = hxxp://schemas.dmtf.org/wbem/wsman/1/wsman/faultDetail/InvalidResourceURI
Error number:  -2144108485 0x8033803B
The WS-Management service cannot process the request. The resource URI is missing or it has an incorrect forma
t. Check the documentation or use the following command for information on how to construct a resource URI: "w
inrm help uris".
As far as I can see in the scx.log, there are some messages like:
2013-04-25T12:19:33,668Z Warning    [scx.core.common.pal.system.disk.diskdepend:6241:7] No link exists between the logical device "rpool12/export/home" at mount point "/export/home" with filesystem "zfs". Some
statistics will be unavailable.
2013-04-25T14:10:34,775Z Info       [scx.core.providers.diskprovider:6241:11] BaseProvider::EnumInstances() - Invalid class - //<servername>/root/scx:SCX_FileSystem
2013-04-25T14:13:27,810Z Warning    [scx.core.providers.osprovider:6241:11] BaseProvider::EnumInstances() - Calling opendir() returned an error with errno = 24 (Too many open files) - [/export/home/serviceb/ScxCore_URSP1_SunOS510_sparc/source/code/shared/scxsystemlib/process/processenumeration.cpp:58]
Do you have any idea what is going wrong and how to fix it?
Thanks & Regards,
Holger

Answer 1

Holger,
I believe it might be the (Too many files open) issue causing your problems. Try the following and let me know if it helps...
First confirm if this is the issue. Run the following command to find out the Solaris server's current file descriptor limit:
ulimit -n
Next determine the number of file descriptors in use by the SCX Agent processes. A list of file descriptors is located under /proc/<Process ID>/fd, where <Process ID> is the process ID of the agent process running on the Solaris server which
can be obtained using the ps command. Or use the following command to list the PID’s and number of file descriptors for any processes containing microsoft/scx which should cover all of the SCX Agent processes:
for i in `ps -ef | grep microsoft/scx | grep -v grep | awk '{print $2}'` ; do echo PID $i: `ls -l /proc/$i/fd | wc -l` ; done
Note: You must execute the above commands using sh, bash, or other Bourne-compatible shell. Be sure the commands are on a single line. If the result is a > sign, then press CTRL-C, check to make sure quotes and backticks are closed, and try again.
If the number of file descriptors returned is near the file descriptor limit obtained using ulimit, follow this procedure to increase the limit by putting a ulimit command in the SCX Agent startup script.
1.      
Find and edit the scx-cimd startup script, which should be located at /opt/microsoft/scx/bin/tools/scx-cimd in Solaris 10 and 11, or at /etc/init.d/scx-cimd in Solaris 8 and 9:
vi /opt/microsoft/scx/bin/tools/scx-cimd
2.   In the file, locate the section responsible for starting the agent daemon.
start)
        # Start daemon
        echo "Starting $DESC"
        $DAEMON $OPTIONS
        exit $?
3.      
Below the echo statement, but before the line with $DAEMON $OPTIONS, insert this command on its own line:
ulimit -n 1024
The end result should look like:
start)
        # Start daemon
        echo "Starting $DESC"
        ulimit -n 1024
        $DAEMON $OPTIONS
        exit $?
4.      
Save the file and restart the agent:
scxadmin -restart all
Regards,
-Steve