I have rebuilt my FMC and ASA Firepower and I'm looking into SSL decryption. I need to do the following:
- make an internal CA to generate a cert used for presenting to the client
- make a self-signed certificate that will protect the proxied content
- distribute and install the cert-chain for the CA on the client systems (or be happy with 'this website is not secure, do you want to proceed' error messages)
Has anyone made a CA that can issue a suitable cert? It looks like I need to issue a cert for "*.com" etc... is this possible or am I missing something here?
Windows Active Directory is your best friend here for internal clients.
If this is to be a solution for external users' devices, though, you'll need a commercial certificate and then be sure that a connectivity path exists to the commercial CA.
This question specifically is for inside -> out requests. eg a user accessing youtube (example).
Do you know what the cert should look like? I'm not sure how it's supposed to look. Will one cert do the job?
I've done some more digging and found this: http://docs.fortinet.com/uploaded/files/2041/using-a-custom-certificate-for-SSL-inspection.pdf
So does the SSL inspection module (in my case, FirePOWER) generate a new cert on the fly for each connection? And uses the signing CA cert that I'll generate from Windows CA. I feel like I'm missing how this is going to work, hence confusion on my part.
The user thinks he's connecting to an HTTPS website, but he's actually connecting to the proxy via an internal SSL cert. The proxy then looks at all the traffic and gets it into an SSL session with the outside site. As traffic returns, the proxy decrypts it locally for inspection before encrypting it with the internal cert to ship back to the user.
Think of it as a big spy network in the post office, opening every letter, inspecting the contents, and then putting those contents into a new envelope to send on its way. There is no direct connection to the ultimate endpoint, so all SSL encryption is handled between an endpoint and the proxy.
So the normal way this all works for inside to outside traffic is as follows.
1. You generate a intermediate CA cert from your internal CA (or create a separate CA cert)
2. Install the public cert for the CA created in step one on all the systems you want to intercept. If you don't own the endpoint good luck.
3. Install the private CA key, and public cert on the device doing the SSL intercept. Configure correctly based on the device.
Now when your client tries to connect to say https://www.networking-forums.com the SSL intercept device will intercept the handshake, and identify the hostname. It will setup a new connection with https://www.networking-forums.com, and then on the fly generate a new key/cert signed by the installed CA for that site. This certificate should match any issues with the real certificate (i.e. expired, wrong hostname, untrusted CA, etc). It then presents this certificate to the end device. You have now successfully intercepted an SSL session wthout throwing any warnings.
Some things to keep in mind.
You have to install your CA cert on the end point to get rid of the cert errors. Typically in an enterprise environment you push it out with a GPO. If trying to do this with BYOD, or customer devices good luck. It is more of a pain than it is worth. Your CA will not be trusted, and everyone will get certificate warnings for every site they go to.
Another item is to verify that your device is matching issues with the real certificate. If you go to a website with an expired SSL cert then the certificate generated on the fly should also be expired. Some SSL intercept solutions always present valid certs to the end device, and ignores errors from the real server. This hides real issues with certs, and actually causes more risk.
That CA key on your intercept device is now the most important file on your network. With it I can spoof anyone I want to your systems.
Some applications/appliances will not work with intercept. Some systems have hard coded CA certs that can not be updated. You will need to configure bypasses for these. I suggest you limit where bypassed systems can go on the internet as you will not see that traffic. These systems typically are just doing call homes for licensing, and have very few destinations. If you allow it to the entire internet you now have a encrypted path for someone to exfil data.
Legal/Politics is important. Users don't like their data being intercepted. Make sure you are clear that you are doing it. Don't try to hide it. Someone will notice. Also there are privacy laws that you may not want to intercept health care sites, banking sites, etc. This can get messy quickly. Make sure you are covered. The last thing you want is all your users having their bank accounts owned, and tracing it back to your intercept box being the source of the leak.
-Otanx
Regarding phone homes... Google, Apple, and Microsoft traffic needs to be allowed undisturbed or their devices will freak out and disconnect from the network.
Also Google certificate pinning. It's killing the SSL decryption market
I understand how this works in terms of traffic flow. I'm just not understanding the certificates aspect.
For the sites that can't be inspected, I'm going to have the fall back action to simply allow the traffic not decrypted. With this, the SSL connection is established between the client and real server.
At the moment all I have is plain http inspection for malware. It's a bit crappy since SSL certs are free and so easy for anyone to set up sending malware via HTTPS. Plus most sites are HTTPS these days. I want to do this on a best effort.
Next problem is most of our user devices are apple Mac's. Good luck pushing certs via gpo 😊
Quote from: deanwebb on May 31, 2017, 02:58:54 PM
Regarding phone homes... Google, Apple, and Microsoft traffic needs to be allowed undisturbed or their devices will freak out and disconnect from the network.
Thanks for this how can I tell if a cert is pinned? I'm expecting the fallback "allow connection not decrypted" to take care of this but I want to confirm.
Quote from: Otanx on May 31, 2017, 02:44:00 PM
The last thing you want is all your users having their bank accounts owned, and tracing it back to your intercept box being the source of the leak.
Scary one.
Quote from: Dieselboy on May 31, 2017, 06:16:29 PM
Quote from: Otanx on May 31, 2017, 02:44:00 PM
The last thing you want is all your users having their bank accounts owned, and tracing it back to your intercept box being the source of the leak.
Scary one.
Yep. This is why I don't like SSL interception. Yes, I know that bad guys use encryption to do bad things. But good guys use it, as well, and there's an expectation of privacy that's worth respecting. Block access to Tor nodes, implement DNS security, don't have a default route that points to the Internet gateway, you should be OK. For DLP, don't allow encryption software on company PCs and shut down external hard drives, including printers that have hard drive functions. Shut 'em all off.
I have antimalware (amp) on desktops. I could say that HTTPS will be protected by the endpoints, negating the need for sale decryption for end users.
Unfortunately I need a default route for cloud services​, main one being Cisco spark.
I'm still thinking of setting up SSL inspection for non-desktops.
Plus I want to do outbound ssl inspection but that's a whole lot easier since I already make the server certs.
Reminds me of that saying but with a twist
pick 2 only:
Security
Convenience
User acceptance
I will see if I can dig up my how to generate a CA certificate stuff. I stole it from a web site somewhere. Basically you just use openssl to generate a key, and a csr. Then sign the csr with the same key. This is now a self signed CA. Import the key and cert into the intercept box, and the cert onto the clients.
What Mr. Webb means by no default route is that you configure all your clients to use the proxy, and then they don't need to get to the internet as all they will go to is the proxy. The proxy needs a default out, but it should be sitting in your DMZ, and can have a default. I am not a fan of the no default route stuff, and just block at the firewall. There are too many different stupid apps/devices that can't have proxies configured.
Certificate pinning it does not work as most people think. If you import your CA into the browser then pinning will accept that CA as OK. The pinning RFC says this is acceptable (RFC 7469 section 2.6). See the FAQ below from Chromium for how they handle user defined CAs. I don't have Chrome, but I can confirm that both Apple and Google allow imported CAs to do intercept just fine as of five minutes ago in Firefox with a green lock. Pinning is not a protection against corporate SSL intercept. It is a protection against a "real" CA issuing bogus certificates which has happened a few times.
http://www.chromium.org/Home/chromium-security/security-faq#TOC-How-does-key-pinning-interact-with-local-proxies-and-filters-
-Otanx
Quote from: Otanx on May 31, 2017, 08:03:44 PM
I will see if I can dig up my how to generate a CA certificate stuff. I stole it from a web site somewhere. Basically you just use openssl to generate a key, and a csr. Then sign the csr with the same key. This is now a self signed CA. Import the key and cert into the intercept box, and the cert onto the clients.
Thanks mate this would be a big help and would be exactly what I'm looking for!
Quote from: Otanx on May 31, 2017, 08:03:44 PM
What Mr. Webb means by no default route is that you configure all your clients to use the proxy, and then they don't need to get to the internet as all they will go to is the proxy. The proxy needs a default out, but it should be sitting in your DMZ, and can have a default. I am not a fan of the no default route stuff, and just block at the firewall. There are too many different stupid apps/devices that can't have proxies configured.
Yes I get that and thats the reason why I can't do this, too. Spark is just one of the apps and it's not proxy aware (yet).
Regarding the proxy, I tried to find info on this but as far as I can tell, I can't use the firepower module or FMC as a proxy itself. The FirePOWER module just has traffic passed through it ie a transparent proxy. I can set up a traditional web forward proxy, though. Its traffic will then be inspected by the FirePOWER module upstream. Is this what you mean? I've had this task on my list anyway and then this came up when I was trying to decide how to proceed from here.
Thanks for the info on cert pinning :)
I checked with HR and we have a policy which an snip of this says:
Quote
[company name removed] will regularly monitor and may copy, access or disclose any information or files stored, processed or transmitted on its communication systems and devices, including internal or external communications (including emails), documents stored on the network, internet usage, duration and sites visited, automated scanning of an employee's files to identify viruses or other malicious codes and logging individual keystrokes.
This includes monitoring the [company name removed] systems and devices outside of regular business hours and personal communications where they were communicated through [company name removed] communication systems and devices. Therefore staff members who use [company name removed] communication systems and devices for personal purposes must not consider this information to be private as they will not have the same personal privacy rights as they would if they were using private communication systems or devices.
This should cover SSL inspection.
PS.
I removed the only line in my SSL decryption rule today because I'm going to come back to this later. The line was a rule to this website. So this meant the policy was empty although it was still attached to the access-control-policy.
For some reason this caused the SFR module to drop a whole bunch of SSL traffic. Not good. Beat this in mind. :squint:
Quote from: Dieselboy on May 31, 2017, 10:35:15 PM
I checked with HR and we have a policy which an snip of this says:
Quote
[company name removed] will regularly monitor and may copy, access or disclose any information or files stored, processed or transmitted on its communication systems and devices, including internal or external communications (including emails), documents stored on the network, internet usage, duration and sites visited, automated scanning of an employee's files to identify viruses or other malicious codes and logging individual keystrokes.
This includes monitoring the [company name removed] systems and devices outside of regular business hours and personal communications where they were communicated through [company name removed] communication systems and devices. Therefore staff members who use [company name removed] communication systems and devices for personal purposes must not consider this information to be private as they will not have the same personal privacy rights as they would if they were using private communication systems or devices.
This should cover SSL inspection.
It may be illegal in your country though, you should check with your legal department instead :)
https://www.sans.org/reading-room/whitepapers/legal/generation-firewalls-employee-privacy-global-enterprise-35467
I can't find my customized write up at home. Here is the site I used when I did this. Besides changing the obvious stuff like company name I think the only change I made was the intermediate CA cert I created was 2048 not 4096. The system I was playing with only supported 2048 CA certs. I wold hope Cisco can do 4096. The write up is pretty good. Just follow the directions on creating the root pair, and intermediate pair. The root pair needs to go offline (USB stick), and the intermediate pair goes on the FirePOWER box. If you want to start issuing certificates for internal websites instead of doing self signed you can create a second intermediate pair, and use it for signing other stuff. The directions should work on a Mac. I don't have one here to test with, but I think they have openssl on them.
https://jamielinux.com/docs/openssl-certificate-authority/introduction.html
-Otanx
That SANS link is a really good write up on the privacy issues. I just skimmed it, but going to read the entire thing later when I have some time.
-Otanx
Quote from: Otanx on May 31, 2017, 08:03:44 PM
I will see if I can dig up my how to generate a CA certificate stuff. I stole it from a web site somewhere. Basically you just use openssl to generate a key, and a csr. Then sign the csr with the same key. This is now a self signed CA. Import the key and cert into the intercept box, and the cert onto the clients.
What Mr. Webb means by no default route is that you configure all your clients to use the proxy, and then they don't need to get to the internet as all they will go to is the proxy. The proxy needs a default out, but it should be sitting in your DMZ, and can have a default. I am not a fan of the no default route stuff, and just block at the firewall. There are too many different stupid apps/devices that can't have proxies configured.
Certificate pinning it does not work as most people think. If you import your CA into the browser then pinning will accept that CA as OK. The pinning RFC says this is acceptable (RFC 7469 section 2.6). See the FAQ below from Chromium for how they handle user defined CAs. I don't have Chrome, but I can confirm that both Apple and Google allow imported CAs to do intercept just fine as of five minutes ago in Firefox with a green lock. Pinning is not a protection against corporate SSL intercept. It is a protection against a "real" CA issuing bogus certificates which has happened a few times.
http://www.chromium.org/Home/chromium-security/security-faq#TOC-How-does-key-pinning-interact-with-local-proxies-and-filters-
-Otanx
I haven't read the RFC but my understanding is that certificate pinning will explicitly mandate the specific certs that is valid in the trust chain I.e. you replace with something signed by your CA it's invalid. Not sure exactly how but example is Discordx. Palo for example has a default bypass list where it don't even try decryption as it knows it will break, it caused a minor ruckus when made public as it was doing the bypass silently.
Looked it up and I don't agree with your take. Certificate pinning will mandate the expected public key, if you change it then no dice. See here https://www.owasp.org/index.php/Certificate_and_Public_Key_Pinning
It explicitly calls out interception and that you have to import the interception proxy public key into the pinset otherwise the interception will break. There is no traditional trust chain anymore that's the point
It's also what our Palo, source fire and checkpoint consultants tell me LOL
Btw it calls out that RFC as contentious specifically for that feature you're talking about
Technically you are correct that certificate pinning fails if the certificate chain presented does not match the pin. However, real life does not match theory.
When doing SSL intercept you have to be clear about three major use cases. The first one is web browsing from normal browsers. By default the major browsers will ignore pinning failures if the CA is a user imported CA. This setting can be changed, but this is the default. This means that as an enterprise I can intercept all my normal users web surfing from managed endpoints by importing a CA cert on the systems. I can now intercept and scan encrypted traffic for bad stuff in a central location. Trying to intercept non managed devices that I can't push a CA cert to is the problem, and usually not worth trying. Just dump BYOD/personal stuff on different network, and treat them as untrusted.
The second use case is non-browser applications that use SSL for transport. This is where the interception can break. It is up to the application developer how they want to handle certificate pinning. They can decide to reject all pin failures, or what I find more common is they don't use the system CA store, and don't supply any way to import your own CA cert into their store. This breaks the intercept. This is actually becoming less of an issue as everyone is doing everything "in the cloud" or "from your web browser" No more thick clients, and the intercept works because I am back to using the browser. If you look at the PA bypass list most of those entries fall into this category.
The third use case is appliances or server applications using SSL for call home, or license validation, etc. This is really a subset of the second use case, but I like to break it out separately because it is a different risk level. These are a crap shoot if you can import your own CA or not. I found most don't do certificate pinning at all (yet), but a lot do not have a way to import a different CA cert. The mitigating factor is most appliances or server applications only connect to a few different sites so I can bypass interception for that appliance to those few sites. This category is actually growing as developers are learning that designing their own protocols is a pain, and just use SSL.
In the end can you intercept traffic? Yes. Is interception going to be impossible any time soon? No. Is it worth doing? Maybe. Should you do it? Politics.
-Otanx
Be prepared to pull your hair out. When everything (and I mean everything) stops working. Windows updates, google drive, adobe, SMTP/ActiveSync/etc/etc/etc.
Thanks for the clarification on IRL vs theory, I'll go back and question our sec guys again on what they're seeing in the field
No problem. I had to dig into this for a customer in the past. It isn't as hard as it sounds once you understand all the terminology, but it still isn't easy. I am also ignoring all the privacy and legal concerns that go with it.
-Otanx
From what I hear, pretty much what lynk says. For certificate pinned sites, browsing is fine, but server updates etc. all break, and not all those mechanisms have a method for importing your CA, so its basically a giant headache and you end up configuring SSL inspect bypass for 'problem' destinations.
Our resident Palo oracle basically told me point blank that he sees SSL inspection dead in 12 months, having said that he is notoriously pessemistic LOL
Quote from: wintermute000 on June 06, 2017, 08:21:41 PM
From what I hear, pretty much what lynk says. For certificate pinned sites, browsing is fine, but server updates etc. all break, and not all those mechanisms have a method for importing your CA, so its basically a giant headache and you end up configuring SSL inspect bypass for 'problem' destinations.
Our resident Palo oracle basically told me point blank that he sees SSL inspection dead in 12 months, having said that he is notoriously pessemistic LOL
I do hope it goes away. It's a damn stupid way to do a damn stupid thing.
If you want to read everyone's web traffic and email, do it at the endpoint. If the endpoint can't be viewed, block it. If the endpoint can't be blocked, you should have spent money on that NAC solution instead of the SSL decryption solution...
Thanks everyone!
Quote from: deanwebb on June 07, 2017, 09:07:39 AM
If you want to read everyone's web traffic and email, do it at the endpoint. If the endpoint can't be viewed, block it. If the endpoint can't be blocked, you should have spent money on that NAC solution instead of the SSL decryption solution...
^ This :)
I see that you're saying SSL inspection will break certain things. So with that, why not just exclude those certain things from inspection? At least then you'll be inspecting most things as a best effort. You could create a pass-through rule based on URLs or destination networks. Then, what's is not specified as excluded / passed-through, will hit the inspection rule for SSL. A bit more management but...
Anything excluded becomes an attack vector.
"The hackers utilized a flaw in the Windows Update service..." Want that as a headline?
:problem?:
Nothing is perfect. To quote Shrek "Security is like an onion, it has layers". The problem for any of these solutions (endpoint or central inspection) are appliances/software where you can't install extra software, and that don't allow adding CAs to trust lists. Any system like that means exception to the inspection policy. Now you have an encrypted link you can't see. An attacker can use that either as an ingress if there is a vulnerability, or as a exfil channel to get data out bypassing DLP. It is a risk acceptance decision in the end. You can just exempt that endpoint from inspection all together, that is an easy low work solution. You can exempt it from inspection for specific destinations. This is more work, but more secure. You could make the decision not to deploy the device at all, and accept the loss of that capability.
I think inspection of encrypted traffic however you do it is important for enterprise networks. Too many attacks now are using encrypted channels to bypass inspection. So either watching it at the end point, or with an intercept you should be watching. While writing this I thought it would be interesting to see malware that exfils data by posting to facebook (I think there is one that uses twitter).
My opinion differs from wintermute's PA guy. I think interception will grow, and become easier in the coming years. We will see. However, I think it is going to differ heavily between different markets. Financial, Medical, and Government will see big increases where employee privacy is secondary to preventing exfil of data. Other markets may not be as interested in deploying it to not deal with privacy issues.
-Otanx
It's not just Fin-Med-Gov-Oil that will want to shut down this sort of thing. We're dealing with highly sophisticated attackers targeting *everyone*, with the targets still thinking that a firewall takes care of everything.
Make an exception, that's where the action of bad actors will go. Get systems that can be used - without exception - and you have something better to work with. The problem will be in all the people asking for exceptions, all of which will present a valid business reason for the exception. Any one of those people asking for the exception can be an inside guy who sees this exception as his or her open door.
We go back to the guys getting exceptions for Microsoft and Google and Apple... and then other vendors... and then we get to someone that says there's a VPN that needs to be open for an outside partner and the cert won't work with the SSL man in the middle you have running... do we check to see if it actually breaks, or do we just take the guy's word at face value and make exception number 145?
100% agree. Everyone is being attacked. I just think that outside of a few sectors the overhead of all the exceptions, the politics and legal concerns of spying on employees will make this all look too hard.
-Otanx
Like Micro segmentation took over from intra zone firewalls, and endpoint based approach will probably take over IMO. I'm also curious how ZScaler etc do it.
I'm looking at NAC :mrgreen: :XD:
Quote from: Dieselboy on June 21, 2017, 08:06:47 AM
I'm looking at NAC :mrgreen: :XD:
This is pertinent to my interests.
It's been on my list of things to implement since I started this job years ago.. Came up in a meeting this week. I've not been able to do much as I've been labbing bitlocker and filevault. I'm about done with that. Hope it never gives me problems!
As part of Cisco's big release yesterday they announced a new "Encrypted Traffic Analytics" See the link below. I think I have to admit defeat, and interception is dying.
https://newsroom.cisco.com/press-release-content?type=webcontent&articleId=1854555
-Otanx
Quote from: Otanx on June 21, 2017, 09:15:39 AM
As part of Cisco's big release yesterday they announced a new "Encrypted Traffic Analytics" See the link below. I think I have to admit defeat, and interception is dying.
https://newsroom.cisco.com/press-release-content?type=webcontent&articleId=1854555
-Otanx
I'm thinking... netflow...
Quote from: Otanx on June 21, 2017, 09:15:39 AM
As part of Cisco's big release yesterday they announced a new "Encrypted Traffic Analytics" See the link below. I think I have to admit defeat, and interception is dying.
https://newsroom.cisco.com/press-release-content?type=webcontent&articleId=1854555
-Otanx
Nice! That's good stats!
To add on to this thread, I've now been given the go ahead to buy something like FirePOWER for our remote office (didn't even need to do the POC funny route-map - yay!).
Should I be looking at something else other than firepower? I've seen it bagged often.
Palo Alto.
If his boss cringes at the cost of riverbed support wait till he sees a Palo BOM
Sent from my Pixel C using Tapatalk
Quote from: wintermute000 on June 23, 2017, 06:06:17 PM
If his boss cringes at the cost of riverbed support wait till he sees a Palo BOM
Haha, $_$
I've never even touched a Palo unit but I am keen to!
I'm still working out a BoM for our remote office but looks like a ASA 5506 would probably do the job. Cisco have mentioned their new 2100 firewalls which I know nothing about.
Would someone mind if they please share a Palo model number to get me started with research?
Quote from: wintermute000 on June 23, 2017, 06:06:17 PM
If his boss cringes at the cost of riverbed support wait till he sees a Palo BOM
Sent from my Pixel C using Tapatalk
It's not so much the cost, but their support has been crap last year. One guy said to me he didn't know how to help me as he didn't know about the feature. In contrast, for another vendor they're about 30% of the cost and smash them with their engineers knowledge and support ethic. Obviously not Cisco.
yeah palo support is rubbish.
The vendor you're talking about, wouldn't happen to start with F would it?
30% of the cost can't be checkpoint ROFL.
really?
I have had about 17 TAC calls in the past 3 months, all of which have been very knowledgeable, and very helpful. Between the weird issues, and this projects bizarre requirements we have needed they have been very good (8/10).
The issue with outbound decryption is their list is great for ~60-70% of the stuff, but it is the weird crap that gets you (looking at you adobe)
Quote from: Otanx on June 21, 2017, 09:15:39 AM
As part of Cisco's big release yesterday they announced a new "Encrypted Traffic Analytics" See the link below. I think I have to admit defeat, and interception is dying.
https://newsroom.cisco.com/press-release-content?type=webcontent&articleId=1854555
-Otanx
So how do we utilise ETA "Encrypted Traffic Analytics"? Will it be included in sourcefire / firepower?
It's metadata analysis via software defined access and tetration AFAIK. I'd expect it to come into source fire as well surely