4 Ways Online Services Can Prepare for the Next Heartbleed

Like many online services, we at Telerivet spent much of last week responding to the OpenSSL Heartbleed vulnerability (see our post-mortem here).

While the lessons of Heartbleed are still fresh in our minds, now is the time for the software community to consider how we can improve our collective security and prepare for the next zero-day vulnerability. Whether the next vulnerability is in an open-source library, operating system, firmware, one’s own software, or a third-party service, many of the same lessons apply.

Our top 4 recommendations for online services to limit the fallout from the next vulnerability like Heartbleed:

1. Improve your SSL configuration

Heartbleed is the perfect advertisement for Perfect Forward Secrecy. Thousands of web services all realized this week that their SSL private keys may have been compromised. Those without PFS enabled also realized that someone who had been recording encrypted sessions could have used their private keys to decrypt up to two years worth of previous traffic to their servers.

But don’t just stop at enabling PFS — why not take this opportunity to enable HSTS in order to prevent against SSL stripping too?

SSL configuration is hard to get right. Qualys SSL Labs makes it easy to test your server’s SSL setup and make sure it’s both secure and compatible with all the browsers out there. Try running their SSL server test on your own server, and follow their suggestions to improve your SSL config until you get an A or A+.

2. Subscribe to security announcement lists

Many of us were lucky to learn about Heartbleed very quickly on Monday when it quickly reached #1 on Hacker News. But system administrators shouldn’t rely on visiting the right website at the right time in order to react quickly to major vulnerabilities.

Fortunately, most major projects have an email list to alert users of security vulnerabilities.

Since it can be impractical to subscribe to announcement lists for every library and software package you depend on (like openssl-announce, where Heartbleed was first publicly announced), joining the security announcement list for your server operating system would be good enough for almost everyone. Here are a few:

When the next big zero-day vulnerability happens, don’t leave it up to chance whether you find out about it immediately or after it's already too late.

3. Limit your exposure when using third-party services

Many online services use various third-party services for functionality like customer support, CRM, analytics, payments, and ads that they rely on to safeguard private data about their customers. Many of these services are integrated via JavaScript snippets within a web app, while others are integrated on the server side.

Vulnerabilities in any of these third-party services could compromise customer data just as easily as vulnerabilities in the main service itself. Yet most services’ Heartbleed post-mortems haven't discussed the risk of vulnerabilities in the third party services they use.

Commonly used services like Google Analytics, Mixpanel, KISSmetrics, Optimizely, Intercom, Stripe, and others would likely be a highly attractive target during a vulnerability like Heartbleed — either for intercepting information of millions of end users, or (worse) MITMing these services and injecting JavaScript code into the websites that use them.

In the case of Heartbleed, all of the services above were affected for at least some time before being patched. It’s hard to say with certainty that nobody used Heartbleed to extract user data from any of these services. And now that we know for sure that attackers can use Heartbleed to steal SSL private keys, it’s hard to say that nobody MITMed them either, though that would be more difficult to do on a large scale.

When assessing the security of your own application, keep in mind that your security is essentially no better than the least secure service whose JavaScript you embed on your site. Each third-party JavaScript you add increases the risk that a vulnerability in a system you have no control over could destroy the foundation of your web app’s security.

I say this as a former JavaScript embed developer myself. In a previous role, it would have been technically possible for me, acting entirely on my own, to inject arbitrary JavaScript code into thousands of websites. Today, thousands of engineers and system administrators have similar or greater abilities. In addition, the ability to inject arbitrary code is now a feature of A/B testing services such as Optimizely. In this case, your web app’s overall security is no stronger than the least secure Optimizely login password of any of your employees.

To prepare for the next zero-day vulnerability, online services should evaluate and limit their exposure to third-party services, especially services embedded via JavaScript. Self-hosting JavaScript files, when possible, can often limit the potential of compromised third-party services to inject malicious JavaScript code into your own website.

In addition, online services can prepare themselves by making it easy to immediately disable third-party JavaScripts that are vulnerable or not known to be safe when a vulnerability is announced.

Likewise, services that provide JavaScript snippets should prepare a way to disable their own service if they know that they are exploitable but are unable to patch themselves (as happened this week when everyone was waiting on Amazon to update their ELBs).

4. Design your systems with defense in depth

The Heartbleed vulnerability was surprising in some ways — how was such a critical part of the world’s security infrastructure broken for so long with so little review? But it isn’t as surprising as one might think. There is an unmeasurably massive amount of code in the wild that almost everyone online relies on every day. And in any of those obscure lines of code there might be one simple programming mistake that was overlooked in code review, missed by a static analyzer, and deployed into production. It has happened before and it will surely happen again.

As a developer, this means that it’s necessary to design your systems with multiple levels of security to mitigate the damage if one part of your system’s security fails, whether because of a vulnerability in your own code or someone else’s code.

In the case of Heartbleed, helpful mitigation methods would have included:

  • For session ID exposure: requiring users to re-authenticate when changing IP addresses.

  • For password exposure: requiring two-factor authentication, or requiring users to approve logins from new IP addresses.

  • For secret key exposure: enabling Perfect Forward Secrecy so that a MITM attacker couldn't decrypt previous encrypted sessions.

In general, imagine that an attacker was able to get through one level of your security, and think about how you can still prevent them from doing something even worse.

For example, imagine that an attacker somehow gained read-access to your database through a SQL injection vulnerability, or somehow gained access to your database backups. Then it would help if sensitive fields in your database were encrypted with a key stored outside the database.

Or imagine one of your employees’ laptops was stolen. It’d help if any customer data or secret keys on it were encrypted in a TrueCrypt volume.

Or imagine that an attacker already had root access to one of your database servers. Then you're in big trouble, but it would still help if you used Tarsnap with a write-only key so that they couldn’t just delete all your data along with all your backups.

And although “security by obscurity” is a bad foundation for a cryptosystem, obscurity is a perfectly valid additional layer of “swiss cheese” in a security system. Simply making things harder to find — like choosing a non-default port for your SSH servers — can at least buy you valuable time when the next zero-day is being exploited.

Since it is inevitable that another “Heartbleed” will happen again, defense in depth is the best way to make sure your users stay as protected as possible.


The two most secure types of software are the software that never ships and the software that does nothing at all. Unfortunately, neither type is very useful to anyone. In practice, we all have to make tradeoffs between competing priorities like shipping, adding features, and improving security. While the lessons of Heartbleed are still fresh, let’s all at least take a moment to make sure we’re well prepared for the next big zero-day vulnerability.

What do you think the software community should do to prepare for the next Heartbleed?

Heartbleed Security Update

Last Monday, a major security vulnerability called “Heartbleed” was announced in OpenSSL, security software that is widely used by web services in order to protect the privacy of communication on the internet. The Heartbleed bug affected many internet services — including Google and Yahoo — making it possible for their web servers to leak users’ private information, including account passwords.  The privacy and security of our users is extremely important to us, so we’ve written this update to explain how Heartbleed affects Telerivet users, as well as the actions Telerivet has already taken to protect your privacy and security.

Most importantly, Telerivet’s production servers were not vulnerable to the “Heartbleed” bug, since we were fortunately using a version of OpenSSL that did not have the bug.  However, some of our internal testing servers were vulnerable until last Monday afternoon, and we also inadvertently ran one unpatched server for less than 5 minutes on Thursday. Also, some third-party services that Telerivet uses to handle messaging, customer support, analytics, and payments (Twilio, Nexmo, Intercom, KISSmetrics, Sift Science, and Stripe) were vulnerable to Heartbleed until last Monday or Tuesday, but have since been patched and secured.

After a thorough security audit of Telerivet’s systems, we have no reason to believe that any of Telerivet users’ private information was exposed due to the Heartbleed vulnerability.  We’re not requiring users to change their Telerivet password, although any concerned users are encouraged to do so. In particular, it may be a good idea to change your password if you use the same password on other services that were vulnerable.

It’s possible that the vulnerabilities in third-party services may theoretically have allowed certain Telerivet customer information such as names, email addresses, and payment information to be exposed, although all of these services currently report no evidence that any data was improperly accessed due to the Heartbleed bug.

We have already rotated the secret keys and SSL certificates that Telerivet’s servers use to encrypt communications, while revoking our old certificates, in order to ensure that Heartbleed poses no future risk to Telerivet users. (Although the date on our SSL certificates hasn’t changed, they actually are using entirely new keys.) In addition, we notified a small number of users who logged in to the unpatched server during a five-minute interval on Thursday, and subsequently regenerated our SSL keys and certificates. We have also required all Telerivet staff to reset all passwords for all services impacted by the Heartbleed vulnerability.

In addition to taking immediate steps to respond to this particular vulnerability, we’ve also taken this opportunity to review Telerivet’s security for other areas for improvements, and have already implemented several changes to make our security even stronger than before. In particular, Telerivet’s SSL implementation now supports Perfect Forward Secrecy and HTTP Strict Transport Security, and our SSL implementation now receives an “A+” grade from Qualys’s SSL tester.

Heartbleed was likely the most serious security vulnerability in several years, and affected nearly every web user and every web service. Fortunately, there is currently no evidence that any private information related to Telerivet has been compromised due to the Heartbleed vulnerability. And even better, companies everywhere are now taking this opportunity to strengthen their security even further to prevent other vulnerabilities in the future, which will be good for the average user in the long run.

(Updated 4/13 to add additional third-party services affected by Heartbleed)