Cisco ASA 5585-X SSP-20 8.4(2) - TCP Syslog problem

Question

Hi,
We have a firewall service environment where logging is handled with UDP at the moment.
Recently we have noticed that some messages get lost on the way to the server (Since the server doesnt seem to be under huge stress from syslog traffic). We decided to try sending the syslog via TCP.
You can imagine my surprise when I enabled the "logging host <interface name> <server ip> tcp/1470" on an ASA Security context and find out that all the connections through that firewall are now being blocked. Granted, I could have checked the command reference for this specific command but I never even thought of the possibility of a logging command beeing able to stop all traffic on a firewall.
The TCP syslog connection failing was caused by a missmatched TCP port on the server which got corrected quickly. Even though I could now view log messages from the firewall in question in real time, the only message logged was the blocking of new connections with the following syslog message:
"%ASA-3-201008: Disallowing new connections."
Here start my questions:
- New connections are supposed to be blocked when the the TCP Syslog server aint reachable. How is it possible that I am seeing the TCP syslog sent to the server and the ASA Security Context is still blocking the traffic?
- I configured the "logging permit-hostdown" after I found the command and it supposedly should prevent the above problem/situation from happening. Yet after issuing this command on the Security Context in question, connections were still being blocked with the same syslog message. Why is this?
- Eventually I changed the logging back to UDP. This yet again caused no change to the situation. All the customer connections were still being blocked. Why is this?
- After all the above I removed all possible logging configurations from the Security Context. This had absolutely no effect on the situation either.
- As a last measure I changed to the system context of the ASA and totally removed the syslog interface from the Security Context. This also had absolutely no effect on the situation.
At the end I was forced to save the configuration on the ASAs Flash -memory, remove the Security Context, create the SC again, attach the interfaces again and load the configuration from the flash into the Security Context. This in the end corrected the problem.
Seems to me this is some sort of bug since the syslog server was receiving the syslog messages from the SC but the ASA was still blocking all new connections. Even the command "logging permit-hostdown" command didnt help or changing back to UDP.
It seems the Security Context in question just simply got stuck and continued blocking all connections even though in the end it didnt have ANY logging configurations on.
Seems to me that this is quite a risky configuration if you are possibly facing cutting all traffic for hundreds of customers when the syslog connection is lost or the above situation happens and isnt corrected by any of the above measures we took (like the command "logging permit-hostdown" which is supposed to avoid this situation alltogether).
- Jouni

Answer 1

Hi,
I FINALLY had the time to look at this issue as I was testing something else in our lab too.
In short, here is what I did:
I configured the TCP logging in the same way as in the original post
I configured the TCP logging giving the commands in different order
Did some other tests related to the proble
Device used: ASA 5585-X
Software: 8.4(2)
Original Device and software : ASA 5585-X running 8.4(1)9
Heres the above scenarions and what actually happened
Original situation
Before doing any changes the test firewall context in question is working normally and the log sent by UDP/514 is arriving to the Syslog server as usual.
I now change the syslog to TCP by giving a command "logging host tcp/1471" (actual port being TCP/1470)
The firewall immediatly starts blocking all connections going through it.
I change the configuration to the correct port TCP/1470 after which log starts appearing in my realtime view on the syslog server. The firewall context in question is still sending only the message "Disallowing new connections" even though the TCP -port on the Syslog server is clearly reachable and the connection is active.
After this I try to do the suggest "clear local-host all" command. This has no effect on the firewall context. No connections are getting through. No connections/xlates are formed on the firewall. I can only see the firewall doing DNS queries with its outside interface (related to another configuration).
After this I try to start correcting the situation the same way as before. I add "logging permit-hostdown" command which has no effect on the situation. I remove all logging configurations and it doesnt have any effect on the situation.
After this I activate UDP logging and can see the logs arriving on the syslog server but again I can only see "Disallowing new connections" message.
In the end I have no other option (to my knowledge) other than to delete the Security Context and create it again with same interfaces and with the configuration saved to the Flash -memory of the ASA.
After this the connections work like usual. (UDP logging in the saved configuration)
Giving the configurations in different order
After I've created the firewall again and all is working I have another try in configuring the TCP Syslog while giving the commands in different order.
First I add the command "logging permit-hostdown" command
Then I add the command "logging host tcp/1470"
After this logs start arriving on the syslog server and connections work as usual. Seems giving the "logging permit-hostdown" first before any other configurations is the right way to go.
Removing the "logging permit-hostdown" command
After I saw that everything was working I tried to remove the "logging permit-hostdown" command and see what happens. Everything worked fine.
Configuring wrong TCP port to "logging host" command
I decide to try and change the TCP port used to a wrong one and see if anything happens. (logging permit-hostdown is active). Firewall works as usual. Naturally no logs can be viewed at the syslog server.
Configuring the TCP Syslogging without "logging permit-hostdown" but with correct port
Finally I tried to configure the TCP Syslogging on ASA with the correct TCP port without issuing the "logging permit-hostdown" command. Everything seemed to work fine after this.
So in conclusion it seems that IF you don't have the "logging permit-hostdown" command issued before you start configuring "logging host tcp/xxxx" , you might run into problems IF you don't have matching settings on the ASA sending the log and the Syslog server receiving the log.
There doesnt seem to be any easy way to correct the situation (with the connections getting blocked) after you have once messed up the configurations. Seems your only option is to reconfigure the Security Context (which is easy) or if this problem exists in the same way in a single ASA you will have to reboot the device which means longer downtime than reconfiguring a context.
There would still be a couple of things to test but at the moment I have no more time for this. I will update if there is any new information.
- Jouni