I have quite often run into issues with clients where they certain web applications no longer work after a Firewall Migration or implementation of Web Proxies (such as BlueCoat).
The scope of the issue tended to be the webUI mgmt interfaces of our various network and server devices – i.e. Aruba, Nutanix, NetApp, HP iLo, etc. I should also note that these all had custom HTTPS ports.
The symptom is that their Browser throws one of many errors (see screenshots below):
Chrome: “This site can’t provide a secure connection”
IE and Edge: “Can’t connect securely to this page”
Firefox: “Secure Connection Failed”
I racked my brain trying to find the root cause, however due to the ambiguous error messages thrown by each browser I tested with, I found myself going nowhere. I checked all of the usual spots in the Windows Event Log (Application, System, NetworkProvider, etc) – all devoid of information specific to this issue.
Most of the advance online was to check SSL/TLS settings in the browser. Which I did and that did not help. I reverted those settings because I don’t want insecure protocols enabled. While editing settings I remembered that Firefox uses its own settings and is not (usually) derived from Group Policy or Registry settings – instead, Mozilla uses their own management approach for browser configuration (it does actually offer a way to manage them via Group Policy but it is archaic and hardly ever used in Enterprise scenarios in my experience). I actually prefer Firefox personally because of the way it works and how you can bring your settings with you, which is important for when you have many different machines that you rotate between.
Back on topic! The fact that I was receiving these errors in Firefox made me toss out the window any suspicion that it was policy settings, because my clients are not managing Firefox at all. I also checked the ECC curve settings (which this client did have set), but quickly ruled out that they were causing issues by using a fresh build before domain join which did not have those settings applied in the build. No luck though – but now I knew it was a global issue not unique to their AD environment and GP settings.
So I started looking elsewhere in my freshly built image sans hardening – in the Registry, in particular, the browser settings for SSL/TLS. No luck there – still default settings. On to the global settings for Schannel. Nope – nothing there either, still default settings.
At this point before wasting any more time, I tested out a Linux OS and my mobile iPhone (which turned out to be a bit of a pain because of their radius authentication on office wired and wireless networks). I received the same errors as before, so now I knew this was not a Windows issue at all.
My suspicions were arisen, so I wondered what would happen if I tried their guest Wifi. I did – and no issues at all! GitHub and other problematic sites (which for them was their cloud-based conferencing/collaboration apps). To my surprise they worked! OK, now I knew this had to be a network issue. I just had to convince their network team….
Have I ever mentioned that I love Wireshark? Well I do. A packet capture showed that upon connecting to the website, the Browser would send a Client Hello and we see a successful TCP handshake, but then nothing happens, so the client tries again, but then just gives up and fails. There is no helpful information in the tcp dump – no wonder our browser had no idea what the error was!
Much back and forth with a network support engineer ensued, as he insisted that the port was open on our firewall. Duh! I could telnet to the port and clearly send and receive traffic (i.e. the TCP handshake). However, I made my point that we see the Client Hello message but receive no Server Hello in return.
I asked if we were doing any type of advanced SSL/TLS protection on the firewall, and he responded that no we were not (as well as reminded me that we weren’t doing SSL Decrypt).
To make a long story short, it was the Application Identification functionality of the recently installed Palo Alto firewalls. Because the mgmt network was on a subnet firewalled off from the office networks and most of the datacenter ranges, the traffic to these web mgmt pages were being routed through their new Palo Alto’s (for which routing was being incrementally changed as the scope of their rollout continued).
The fix was just to add several App-ID items to the configuration along with their custom ports. Voila! Problem = solved.
I am baffled that Palo Alto’s firewalls apparently don’t have the definitions for many of these incredibly common webUI mgmt. pages — but they don’t (yet). Palo Alto if you’re reading this, step up your game! Maybe slip an error code in the traffic so that the browser is aware that the site is blocked and can inform the user (or in this case, the admin!).
Even if they don’t address it directly, it would be nice if PA could detect based on certificate when in doubt. After all, I didn’t spend a few hours going through the trouble of creating and deploying customized certificates from our internal PKI to all of these devices all for nothing!
I hope this guide can help some other Windows Admins and Security Professionals who come across this issue. I see Palo Alto’s becoming more and more popular over the last few years, especially due to their SSL Decrypt capabilities (which can be a nightmare to setup BTW…). Because who wants to maintain proxy servers these days anyway? LOL. I am somewhat serious though – no matter how well your proxy deployment is built on the server-end, you will still be inundated with user errors for when they forget to turn the proxy on or off on their laptop’s browser settings. And good luck with DHCP Option 252 vs. Group Policy; .PAC file definitions and WPAD queries…. So good on Palo Alto for designing a solution that can do web-filtering on the Firewall itself. Seriously, it’s about time.
Never A Null Moment 