TMS lost HTTPS communciation to devices, but not really!!!

Question

Hey all,
Another weird one.
TMS - 14.3.2
Monitoring multiple device via HTTPS.
I'm seeing some odd issues from TMS where it seems to think that it has lost HTTPS communication with multiple devices. We have many hundreds of devices that we monitor across multiple organisations in TMS (thankfully, we don't use it to schedule!) such as VCS-Es, VCS-Cs, MXP, C and SX series devices. Yesterday it appeared as though TMS wanted to stop monitor a LOT of these devices. Most of the VCS-Cs, MXP and other devices that seemed to all loose communication are in organisations outside of our main campus.
However, the odd thing is, that they haven't really! Its a "phantom" lost communication!
If you navigate to the connection tab in TMS and hit the "Save/Try" button (in fact you only need to navigate to this page, or the edit setting page, or force a refresh etc.), HTTPS communication is established immediately. In fact, we can also RDP into the TMS box and navigate to any and all devices from a browser with no issues using HTTPS. You can even have a browser session open to a device an have an auto refresh running (say on a camera/call page), with no problems. Further, if you navigate the the affected device (such as VCS Control) and look in the "System --> External Manger" menu, the device shows that our TMS management server is active on HTTPS. However, after a couple of minutes, TMS again thinks it has lost HTTPS connection!!
The really odd thing is that we haven't lost access to all devices. Those on campus are OK, as are the VCS-Es on the core, but a most of the devices in other organisations have got this "phantom" lost communication - weirdly not all of them though!!!
Recently, the backbone network provider installed some DDoS devices on the MPLS network that caused havoc with our neighbour zones, effectively killing our entire VC network. I'm wondering if these devices might also be responsible for these problems? So, I need to know what is different between the TMS HTTPS monitoring communication, and actually forcing a refresh of this connection or navigating to a HTTPS server on a device via a browser.
I have tried a packet capture on TMS, looking at device whose HTTPS response remain unaffected and a VCS Control that is affected, but it show no traffic on either, so I don't really know what I'm looking for (I was expecting a heartbeat or some-such).
Any suggestions, please let me know, before I go back to the core network provider....
Cheers
Chris

Answer 1

Hey Wayne/Adam,
Riddle me this....
Ok, in my situation all appears to be working again. How? Well, I don't really understand, but settle down and I'll tell you a story...
I ran some captures on TMS and on a couple of devices (one that was working and one that wasn't). I turned off things like provisioning just to eliminate as much traffic as possible between the devices and TMS. However, the captures showed nothing unusual between the either device and TMS. Of I could see traffic in both directions with no issue, but one devices still times out and ended with the "No HTTPS connection" error. Unusually I saw no heartbeat type traffic after a refresh - just nothing on both devices either to or from TMS.
We noted that the devices that were not timing out in TMS were also accessible from other machines in our campus. We ask our customers to tie down this management to the single IP address of TMS, but sometimes they allow our campus range. So we though, ah, firewall, yet TMS can still talk to these odd units, so again I didn't understand.
Anyhow, my colleague noted an error in the Windows Event log referring to DNS resolution (Event ID 1014 - which looked like a partial reverse lookup such as 100.100.in-addr.arpa for our class B campus network). However, the vast majority of our system we mange form TMS are through direct IP so I didn't think much of it. Anyhow, I though we would update the DNS settings on the TMS NIC to point to other DNS servers and an odd thing happened.
Firstly, TMS started to "see" the phantom devices. Secondly, the DNS settings didn't actually take. I definitely OK'ed the properties box on the NIC settings, but the 'ipconifg' details and 'nslookup' still showed the old DNS server addresses. Even more weird, I re-opened the properties box and the new services were in place. I then closed the properties box down again and re-opened and as if by magic, the old servers re-appeared! All of this was watched by my colleague who was as baffled as I.
I have no idea what just happened here and why the DNS setting may even have had an impact on this problem, but I'll take the win and walk away.
Our TMS is virtualised running on Windows 2008 R2 and I'm wondering if the adapter settings might have got messed up somehow, but to be honest, I don't really know.
I doubt this will help you Wayne, but its worth a shot. If it doesn't work and If TMS is a VM, it may be worth added an second adapter and recovering the old one.
Cheers
Chris