Flexconnect APs stopped joining

Question

Hello Community,
A customer of mine has a centralized 2504 WLC with 7.2 code running.  They have 1142N APs deployed locally as well as in remote sites (3) in FlexConnect mode.  For no apparent reason last Thursday all the remote APs disassociated with the controller and could not rejoin.  All the local APs remained up and unaffected.
No changes to the WLAN, LAN, Firewall or MPLS WAN occured to cause this.
The customer opened a TAC case and their determination was that ports 5246-5247 were not getting thru.  When the customer engaged me this morning I had him run a packet capture on the Sonicwall firewall to prove out if the CAPWAP signals were leaving and returning across the WAN.  Sure enough we can see this bi-directional traffic (pic attached).  Also, I had the MPLS provider run a trace at the far end and they see the same traffic leave the remote site. 
And then an odd thing happened; one of the APs at one of the remote sites all of a sudden Joined the controller.  So I tried rebooting the AP that is located in the same office, and it fails to Join.  When I look on the controller under AP Join statistics, the last activity shows the controller receiving a Discovery Request and response is sent, but no further Config Request and response or Join Request and response.
Frankly a little stumped, we are re-engaging TAC, but thought that maybe someone in the community has run across this scenario before. 
I also thought that it may be good to upgrade to 7.4, just on spec.
As always any feedback, advice and comments are appreciated and welcome.
Thanks.

Answer 1

I just checked, and that world-mode command isn't even available on the radio interface, so it must only be available for Autonomous APs.  I wondered if that was the case.  I'd love to try the remote AP swap, but the closest remote site is about 2000km away.  may come to that.  I'll see if they have a spare lying around we can try that with.
When I look at the AP Join stats for a failed AP, theres a Discovery Request Received, and a Discovery Response sent, but nothing past that.
When I do a #show ap join stats detailed the reason for failure shows as "Unavailable"
I did confirm that all the working APs have a country code of CA, which makes sense if they just received that from the controller upon joining, so that may just be a rat-hole I'm chasing down.
We do see that traffic leave and come back across the edge firewall, so it must be traversing, just not the full process.  If it was blocked it should be totally blocked, not mid-way thru?