Backup ACS server not used by switch.

Question

I am experiencing a strange issue: During a primary ACS failure, our switches are not resorting to the backup ACS for login authentication, except for enable mode. This means we can only use the emergency local login, but once logged in we cannot enable due to the switch attempting to authenticate that to the backup ACS.
Once I created the local user in the backup ACS I was able to log in, and after I removed then re-addded the primary server as a TACACS host it worked as expected - using the backup only. I can't help but think there is some minor command I am missing so that the switches will recognize the failure of the primary ACS.
What am I missing that a failure of an ACS server does not cause the switches to use other configured servers?

Answer 1

Richard,
I have reviewed the information, however, the debugs are not clear enough as the only outputs displayed other than Accounting logs are the following lines:
012697: Jan  3 22:37:16.866 GMT: AAA/AUTHEN/LOGIN (0000094B): Pick method list 'default'
012698: Jan  3 22:37:24.743 GMT: AAA/AUTHEN/LOGIN (0000094B): Pick method list 'default'
There are known issues with IOS devices not triggering the fallback/failover to the secondary ACS/TACACS+ server when the primary returns an "ERROR" response. "ERROR" refers to a process failure on the server side dropping the request and would not be the same as User Invalid or Bad Password responses which are failures referring to the Authentication information and not the process itself.
Would it be possible for you to collect a capture on the Secondary ACS switchport while the primary is down in order to determine if the IOS device is reaching the secondary server at all?
Known issue:
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsd48175
Symptoms
AAA does not failover to the backup tacacs server defined when it receives ERROR
from the primary server .
Conditions
Occurs when tacacs is configured for authentication, and backup servers are
configured. When the primary server returns error due to csauth not running on
the primary server, in that case  authentication request does not fail over to
secondary server.
Frequency:
Not a common scenario.
Workaround:
None
NOTES
1) If you have an ACS for Windows (3.x or 4.x) then you can install Wireshark on the Windows Server and collect the capture.
2) If you have an ACS Appliance (3.x or 4.x) or an ACS 5.x you might need to configure a SPAN session on the switch.
After collecting the capture you can use Wireshark > Edit > Preferences > Protocols > TACACS+  > TACACS+ Encryption Key > type the shared secret value. This will  allow you to review the unencrypted packets.
You can filter the capture as well using ip.addr==x.x.x.x where x.x.x.x is the IOS device IP address.
Feel free to share the capture with me as well along with the shared secret key. I would gladly review the information.
NOTE: If the capture shows no traffic going to the secondary unit a useful test would be to configure the "Secondary" server as the primary on the IOS and verify if it works that way.
NOTE: If possible, a capture on the primary server switchport while it is down might be useful in order to verify how is the IOS determining that the primary server is down as I do not see it trying to contact the primary either... We should see atleast timeouts when contacting the primary ACS.
Regards.