Surprises on Replacing failed primary unit

Question

Dear friends,
I have done failover for firewalls umpteen number of times but yesterday it failed for some reason.
I had replaced the failed primary unit with a fresh one and i had expected that it will detect the secondary unit as active and try to begin config replication from it but rather it wiped off the secondary unit's config. I dont think that i faulted in the sequence but let me share with you what i did:
1. Put the four or five lines of failover configuration (except the failover command) and did a no shut on the failover interface (management0/0)
2. Ran the failover command
Instead of getting the config from the active unit, it started forcing the configs to the other unit. To restore, i had to reload the active unit to restore its config. After that i reloaded the fresh unit and now the failover happened as expected.
I think that i should forced a reload of the new unit before trying to establish failover.
Has anyone tried this in a fail-proof way during production hours? if yes, can you please share with me the steps?
I did not ask for downtime because i was confident but i resulted in bringing down the ASA for 5 minutes because of the unexpected failover action.
Thanks a lot
Gautam

Answer 1

Dear kureli,
Thanks a lot for the efforts you took. I really appreciate it.
Here's the exact sequence of steps that happened:
1.  When primary unit failed, secondary got active and i dont remember if sh fail showed "secondary- not detected" or "secondary - failed"
2.  I replaced the faulty primary unit with another primary unit and said no shut on the m0/0 failover interface and also put all the failover commands except "failover" command.
3. I made sure that the new primary unit runs the same code (i checked only the main code version, i did not check the asdm version similarity). The asdm versions were different on both boxes though.
4. After powering up the box and connecting cables, i said failover. It then prompted me saying that SSL license is not the same on both units and disabling failover.
5. I applied for an activiation key from [email protected] and then got the SSL license from them.
6. Next day i went back to the customer and installed the license key. After installing the license key, i said failover. It gave me the message "No response from mate"
7. I then said no failover to disable failover on the new primary unit.
8. I then went to secondary active unit and said failover as failover was disabled
9. I then went back to primary unit and said failover
10. This is where blank config replication started !!
11. Reloaded secondary unit to undo the blank running config
12. Went to Primary unit and disconnected the failover cable. Rebooted the primary unit and connected the failover cable.
13. Secondary came up as active, primary then came up, and this time primary honored the secondary as active and did config replication
14. All was well then!!
Not sure still why this happened and it was a bit shameful for me to see this happening after 3.5 years of firewalling experience.
Anyways, i am willing to learn and improve from now on.
Probably next time, i would try to make sure that i apply the failover configs, reload, and while reload connect the failover cable.
I think the learning lesson is that if the unit reloads, the reloaded unit always honors the currently active unit and does not try to override its role.
This is what worked for me.
Thanks a lot
Gautam