WAAS WAE Alarm 'mstore_key_retrieval'

Question

Hello,
I am supporting an environment that has 30+ remote WAEs deployed with a CM at the HQ.
All remote WAE's Versions = Cisco Wide Area Application Services (universal-k9) Software Release 4.2.3b (build b4 Oct  4 2010)
HQ's CM version = Cisco Wide Area Application Services (universal-k9) Software Release 4.4.3 (build b4 Aug 22 2011)
On 4 of these WAEs, I currently am receiving encryption key alarms:
WAE#show alarms detail support
Critical Alarms:
        Alarm ID                 Module/Submodule               Instance
   1 mstore_key_retrieval      cms                          ssl_mstore_key          
     Apr 11 18:36:16.026 CDT, Processing Error Alarm, #000002, 3000:700008
     Unable to generate and/or retrieve SSL managed store encryption key from the Key Manager
     /alm/crit/cms/mstore_key_retrieval_failure:
         CMS/Management agent failed to generate and/or retrieve SSL managed store encryption key from Key Manager.
     Explanation:
         This alarm indicates one of following issues: Central
         Manager device(s) is not reachable.  Secure store on
         Central Manager is initialized but not open.  Key Manager
         process on Central Manager device is not running or failing
         to respond.  Key Manager is unable to process key
         generation or retrieval request.   If this issue is
         present, the WAE device will not be able to process  a
         configuration update received from the Central Manager if
         it  contains SSL certificate/key information.
     Action:
         Check if Central Manager device is reachable (TCP
         connections from the WAE to the Central Manager on port
         443) Check following log files for additional information
         about the error: /local1/errorlog/kc.log on WAE
         /local1/errorlog/km/km.log on CM
   2 mstore_key_failure        sslao                        mstore_key_failure      
     Apr 11 18:39:07.518 CDT, Processing Error Alarm, #000006, 26000:26002
     Failed to open SSL store due to failure in getting key from Central Manager.
     /alm/crit/sslao/mstore_key_failure:
         SSL managed secure store key retrieval failure.
     Explanation:
         The SSL accelerator is unable to get the SSL secure store
         key from the Central Manager.
     Action:
         Check the connection with the Central Manager.
The explanations and actions match the alarm book , but in addition to that, in the Cisco WAAS Monitoring Guide, it also states:
Alarm 700008 (mstore_key_retrieval_failure) CMS/Management agent failed to generate and/or retrieve SSL managed store encryption key from Key Manager.
Severity: Critical
Category: Processing
Description: This alarm indicates one of following issues:
–The WAAS Central Manager device is not reachable
–Secure store on WAAS Central Manager is initialized but not open
–The Key Manager process on the WAAS Central Manager device is not running or failing to respond
–Key Manager cannot process key generation or retrieval request. If this issue is present, the WAAS device cannot process a configuration update received from WAAS Central Manager if it contains SSL certificate and key pair information.
Action: Check to see if the WAAS Central Manager device is reachable (TCP connections from the WAE to the WAAS Central Manager on port 443). Check the following log files for additional information about the error:
–On WAE: /local1/errorlog/kc.log on WAE
–On WAAS Central Manager: /local1/errorlog/km/km.log
Action: Fix the clock on the device or the primary WAAS Central Manager.
For a complete list of alarm conditions, see the Alarm Book located in the WAAS 4.2.1 Software Download area on Cisco.com.
Using this information, I've checked the following:
TCP 443 is reachable from the WAE to the CM (I can telnet from each WAE to the CM on TCP 443)
Time is correct on the WAEs and CM ('show ntp status' and 'show clock' are consistent)
Secure store on CM is open ('show cms secure-store' on the CM shows that the mode is in 'Open' state),
Verified that the key manager process is running (Looking at the CM's KM log shows plenty of action that it's working for other WAEs)
Here is some information I gathered from the WAEs' kc.log files and the CM's km.log (slightly scrubbed):
From the WAEs' kc.log files:
pool-1-thread-1] INFO  CommClientAbstractRPC - Send key retrieval request to CM 10.x.x.x for token d1b77e45-ce60-4332-a92d-3d3cb17d35cf
pool-1-thread-1] WARN  CommClientAbstractRPC - Received error response from KM(20,No key found for token d1b77e45-ce60-4332-a92d-3d3cb17d35cf from device 17111)
From the CM's km.log file:
[pool-1-thread-4] INFO - retrieveKey request, token=d1b77e45-ce60-4332-a92d-3d3cb17d35cf from device WAE1/17111
[pool-1-thread-4] INFO - Checking secure store open
[pool-1-thread-4] INFO - Loading KEK from data server
[pool-1-thread-4] INFO - ticket 17111 (1327767406332, 1327767392433, 13899, 10000)
[pool-1-thread-4] WARN - No key found for token d1b77e45-ce60-4332-a92d-3d3cb17d35cf from device 17111
*** Going through these logs, I've seen other devices have the same issue, and eventually a WAE records the following:
[main] ERROR DeviceInfo - /state/node.dat (No such file or directory)
java.io.FileNotFoundException: /state/node.dat (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(Unknown Source)
at java.io.FileInputStream.<init>(Unknown Source)
at com.cisco.waas.kc.DeviceInfo.retrieveNodeInfo(DeviceInfo.java:65)
at com.cisco.waas.kc.DeviceInfo.<init>(DeviceInfo.java:47)
at com.cisco.waas.kc.DeviceInfo.getInstance(DeviceInfo.java:37)
at com.cisco.waas.kc.comm.CommClientAbstractRPC.retrieveKey(CommClientAbstractRPC.java:149)
at com.cisco.waas.kc.RetrieveKeyCommand.execute(RetrieveKeyCommand.java:43)
at com.cisco.waas.cli.CLICommand.execute(CLICommand.java:114)
at com.cisco.waas.cli.AbstractCLI.process(AbstractCLI.java:28)
at com.cisco.waas.kc.KeyClient.main(KeyClient.java:40)
[main] ERROR DeviceInfo - /state/node.dat (No such file or directory)
java.io.FileNotFoundException: /state/node.dat (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(Unknown Source)
at java.io.FileInputStream.<init>(Unknown Source)
at com.cisco.waas.kc.DeviceInfo.retrieveNodeInfo(DeviceInfo.java:65)
at com.cisco.waas.kc.DeviceInfo.<init>(DeviceInfo.java:47)
at com.cisco.waas.kc.DeviceInfo.getInstance(DeviceInfo.java:37)
at com.cisco.waas.kc.comm.CommClientAbstractRPC.initKey(CommClientAbstractRPC.java:40)
at com.cisco.waas.kc.InitKeyCommand.execute(InitKeyCommand.java:40)
at com.cisco.waas.cli.CLICommand.execute(CLICommand.java:114)
at com.cisco.waas.cli.AbstractCLI.process(AbstractCLI.java:28)
at com.cisco.waas.kc.KeyClient.main(KeyClient.java:40)
*** Followed with what appears to be a new SSL key being generated ***:
[main] INFO  DeviceInfo - loaded device info, hash  H04Fer5il3b/9oanDZXx/7aBnIo=
[pool-1-thread-1] DEBUG CMProber$ProbeWorker - Sending CM probe request to CM 10.x.x.x
[pool-1-thread-1] DEBUG CMProber$ProbeWorker - CM 10.x.x.x returned :primary:4.4.3.0.4
[pool-1-thread-1] DEBUG CMProber$ProbeWorker - Primary CM address 10.x.x.x version 4.4.3.0.4
[main] DEBUG CommClientAbstractRPC - CM version 4.4.3
[main] INFO  CommClientAbstractRPC - Send key initialization request to CM 10.x.x.x key type SSL
[main] INFO  CommClientAbstractRPC - Received new token for generated key SSL/cbe3d6fc-875e-4b61-baeb-528c55cb3597
[main] INFO  DeviceInfo - loaded device info, hash  H04Fer5il3b/9oanDZXx/7aBnIo=
[pool-1-thread-1] INFO  CommClientAbstractRPC - Send key retrieval request to CM 10.0.65.234 for token cbe3d6fc-875e-4b61-baeb-528c55cb3597
[main] INFO  CommClientAbstractRPC$1 - Successfully retrieved key from CM for token cbe3d6fc-875e-4b61-baeb-528c55cb3597
*** And the CM records the following ***:
[pool-1-thread-4] INFO - initKey request from device WAE2/30129 key type SSL
[pool-1-thread-4] INFO - Checking secure store open
[pool-1-thread-4] INFO - Loading KEK from data server
[pool-1-thread-4] INFO - Return crypto of type : 0
[pool-1-thread-4] INFO - Checking secure store open
[pool-1-thread-4] INFO - Loading KEK from data server
[pool-1-thread-4] INFO - Loading KEK from data server
[pool-1-thread-4] INFO - Generated new key WAE2/SSL token cbe3d6fc-875e-4b61-baeb-528c55cb3597
I'm wanting to know why this occurs on some boxes and not others, and what triggers the process for a WAE to stop repeatedly sending key retrieval requests with a token that the CM has repeatedly replies with the key not being found and performing an initial key request.
Thanks!

Answer 1

Hi all, I got into the same issue and looking at a solution I found a way to clear those alarms whithout re-registering the WAE/WAVE. Here it goes...
WAE##sh accelerator
Accelerator     Licensed        Config State    Operational State
cifs            Yes             Enabled         Running
epm             Yes             Enabled         Running
http            Yes             Enabled         Running
mapi            Yes             Enabled         Running
nfs             Yes             Enabled         Running
ssl             Yes             Enabled         Disabled  ---> your SSL AO is probably down due the issue
video           No              Enabled         Shutdown
WAE#sh alarms
Critical Alarms:
        Alarm ID                 Module/Submodule               Instance
   1 mstore_key_retrieval      cms                          ssl_mstore_key
   2 mstore_key_failure        sslao                        mstore_key_failure
Major Alarms:
None
Minor Alarms:
None
WAE#crypto pki managed-store initialize
All certificate/private keys in SSL managed store will be deleted and optimized SSL traffic will be interrupted. Are you sure you want to continue(yes/no)? [no]:yes
Restarting SSL accelerator. Done.
After a couple of minutes alarms will be cleared and SSLAO will be back UP.
WAE#sh accelerator
Accelerator     Licensed        Config State    Operational State
cifs            Yes             Enabled         Running
epm             Yes             Enabled         Running
http            Yes             Enabled         Running
mapi            Yes             Enabled         Running
nfs             Yes             Enabled         Running
ssl             Yes             Enabled         Running
video           No              Enabled         Shutdown
WAE#sh alarms
Critical Alarms:
None
Major Alarms:
None
Minor Alarms:
None
In case you have the issue in the Core WAE (where the cms secure-store is opened), you might need to initialize it.
Regards,
Fernando