Corrective Action on Agent Unreachable?

Question

Let me first explain exactly what I am looking to do. My current setup is a repeating job that recreates connections (tunnels) to the agent boxes every 15 minutes to ensure that connections stay up. This can get hung up and drag the network down. Ideally, I would like to setup an automated system to correct the issues. If an Agent becomes unreachable, I want to initiate a corrective action that will try to recreate the connection. I was told that the corrective action would be run ON THE AGENT, which would inevitably fail if there was no connection. So a 2nd task would be added to the corrective action in response to the first failed task, that would run the connection scripts on the OEM box and re-establish connection. If that fails as well, then I would like a notification to be sent out (incident rules typically).
That is my ideal set up. Here are the issues I am running into. The first is that the Agent Unreachable does not seem to be an editable metric. It is listed as a metric called Response > Status, under All Metrics. However, under Metric settings, it can not be edited, and thus, I do not see a way to tie a corrective action to this metric. Once I do tie the corrective action to this metric, do I need to set up a job to run in the corrective action or is the corrective action basically a job that runs in response to an event? That is how I understood it...just looking for verification. It would be greatly appreciated if I could be pointed in the correct direction.

Answer 1

I'm not sure you can do this through a metric alert corrective action, but it sounds like a great enhancement request. You didn't mention the EM version that you're using, but if you're on EM12c you can implement something like this through an advanced notification method. There's more than one way to do it but here's what I just tested and confirmed will work:
1) Generate an ssh key for the oracle user running the OMS, and save the key with no password
2) Copy the generated public key to $HOME/.ssh/authorized_keys for the user running the agent on the host where you would like to restart the agent
3) Create a shell script owned by the OMS oracle user that contains the line "ssh [email protected] /pathtoagent/bin/emctl start agent"
4) Log in to EM12c as user SYSMAN and create an advanced notification method that calls the shell script created in step 3
5) Create an incident ruleset applying to the agent target
6) Create an rule in the ruleset that applies to incoming incidents where Category = Availability and Severity = Critical
7) Create an action for that rule that calls the advanced notification method from step 4
Then stop the agent to test it. Once the incident is generated, the notification method will run, ssh from the OMS host to the agent host, and start up the agent again.
It doesn't look like you can pass parameters to the notification methods, so you may need to create a separate notification method and separate shell script for every agent you monitor.
You could probably also do this through an SNMP trap advanced notification if you have other management software on your network that could catch the trap and respond to it appropriately.
But for me the real question would be: why are your agents going unreachable so often that you need to bother with this?