Like many online services, we at Telerivet spent much of last week responding to the OpenSSL Heartbleed vulnerability (see our post-mortem here).
While the lessons of Heartbleed are still fresh in our minds, now is the time for the software community to consider how we can improve our collective security and prepare for the next zero-day vulnerability. Whether the next vulnerability is in an open-source library, operating system, firmware, one’s own software, or a third-party service, many of the same lessons apply.
Our top 4 recommendations for online services to limit the fallout from the next vulnerability like Heartbleed:
1. Improve your SSL configuration
Heartbleed is the perfect advertisement for Perfect Forward Secrecy. Thousands of web services all realized this week that their SSL private keys may have been compromised. Those without PFS enabled also realized that someone who had been recording encrypted sessions could have used their private keys to decrypt up to two years worth of previous traffic to their servers.
But don’t just stop at enabling PFS — why not take this opportunity to enable HSTS in order to prevent against SSL stripping too?
SSL configuration is hard to get right. Qualys SSL Labs makes it easy to test your server’s SSL setup and make sure it’s both secure and compatible with all the browsers out there. Try running their SSL server test on your own server, and follow their suggestions to improve your SSL config until you get an A or A+.
2. Subscribe to security announcement lists
Many of us were lucky to learn about Heartbleed very quickly on Monday when it quickly reached #1 on Hacker News. But system administrators shouldn’t rely on visiting the right website at the right time in order to react quickly to major vulnerabilities.
Fortunately, most major projects have an email list to alert users of security vulnerabilities.
Since it can be impractical to subscribe to announcement lists for every library and software package you depend on (like openssl-announce, where Heartbleed was first publicly announced), joining the security announcement list for your server operating system would be good enough for almost everyone. Here are a few:
When the next big zero-day vulnerability happens, don’t leave it up to chance whether you find out about it immediately or after it's already too late.
3. Limit your exposure when using third-party services
Many online services use various third-party services for functionality like customer support, CRM, analytics, payments, and ads that they rely on to safeguard private data about their customers. Many of these services are integrated via JavaScript snippets within a web app, while others are integrated on the server side.
Vulnerabilities in any of these third-party services could compromise customer data just as easily as vulnerabilities in the main service itself. Yet most services’ Heartbleed post-mortems haven't discussed the risk of vulnerabilities in the third party services they use.
Commonly used services like Google Analytics, Mixpanel, KISSmetrics, Optimizely, Intercom, Stripe, and others would likely be a highly attractive target during a vulnerability like Heartbleed — either for intercepting information of millions of end users, or (worse) MITMing these services and injecting JavaScript code into the websites that use them.
In the case of Heartbleed, all of the services above were affected for at least some time before being patched. It’s hard to say with certainty that nobody used Heartbleed to extract user data from any of these services. And now that we know for sure that attackers can use Heartbleed to steal SSL private keys, it’s hard to say that nobody MITMed them either, though that would be more difficult to do on a large scale.
When assessing the security of your own application, keep in mind that your security is essentially no better than the least secure service whose JavaScript you embed on your site. Each third-party JavaScript you add increases the risk that a vulnerability in a system you have no control over could destroy the foundation of your web app’s security.
I say this as a former JavaScript embed developer myself. In a previous role, it would have been technically possible for me, acting entirely on my own, to inject arbitrary JavaScript code into thousands of websites. Today, thousands of engineers and system administrators have similar or greater abilities. In addition, the ability to inject arbitrary code is now a feature of A/B testing services such as Optimizely. In this case, your web app’s overall security is no stronger than the least secure Optimizely login password of any of your employees.
To prepare for the next zero-day vulnerability, online services should evaluate and limit their exposure to third-party services, especially services embedded via JavaScript. Self-hosting JavaScript files, when possible, can often limit the potential of compromised third-party services to inject malicious JavaScript code into your own website.
In addition, online services can prepare themselves by making it easy to immediately disable third-party JavaScripts that are vulnerable or not known to be safe when a vulnerability is announced.
Likewise, services that provide JavaScript snippets should prepare a way to disable their own service if they know that they are exploitable but are unable to patch themselves (as happened this week when everyone was waiting on Amazon to update their ELBs).
4. Design your systems with defense in depth
The Heartbleed vulnerability was surprising in some ways — how was such a critical part of the world’s security infrastructure broken for so long with so little review? But it isn’t as surprising as one might think. There is an unmeasurably massive amount of code in the wild that almost everyone online relies on every day. And in any of those obscure lines of code there might be one simple programming mistake that was overlooked in code review, missed by a static analyzer, and deployed into production. It has happened before and it will surely happen again.
As a developer, this means that it’s necessary to design your systems with multiple levels of security to mitigate the damage if one part of your system’s security fails, whether because of a vulnerability in your own code or someone else’s code.
In the case of Heartbleed, helpful mitigation methods would have included:
-
For session ID exposure: requiring users to re-authenticate when changing IP addresses.
-
For password exposure: requiring two-factor authentication, or requiring users to approve logins from new IP addresses.
-
For secret key exposure: enabling Perfect Forward Secrecy so that a MITM attacker couldn't decrypt previous encrypted sessions.
In general, imagine that an attacker was able to get through one level of your security, and think about how you can still prevent them from doing something even worse.
For example, imagine that an attacker somehow gained read-access to your database through a SQL injection vulnerability, or somehow gained access to your database backups. Then it would help if sensitive fields in your database were encrypted with a key stored outside the database.
Or imagine one of your employees’ laptops was stolen. It’d help if any customer data or secret keys on it were encrypted in a TrueCrypt volume.
Or imagine that an attacker already had root access to one of your database servers. Then you're in big trouble, but it would still help if you used Tarsnap with a write-only key so that they couldn’t just delete all your data along with all your backups.
And although “security by obscurity” is a bad foundation for a cryptosystem, obscurity is a perfectly valid additional layer of “swiss cheese” in a security system. Simply making things harder to find — like choosing a non-default port for your SSH servers — can at least buy you valuable time when the next zero-day is being exploited.
Since it is inevitable that another “Heartbleed” will happen again, defense in depth is the best way to make sure your users stay as protected as possible.
The two most secure types of software are the software that never ships and the software that does nothing at all. Unfortunately, neither type is very useful to anyone. In practice, we all have to make tradeoffs between competing priorities like shipping, adding features, and improving security. While the lessons of Heartbleed are still fresh, let’s all at least take a moment to make sure we’re well prepared for the next big zero-day vulnerability.
What do you think the software community should do to prepare for the next Heartbleed?