Archive | security RSS feed for this section

If Your Business Uses Rails 2.3 You Need To Move To A Supported Option ASAP

Posted on June 17, 2013 by in Rails, Ruby, security

Executive summary for your CTO: If your company runs Rails 2.3 apps, switch to Rails LTS, a commercially supported fork of Rails.  If you do not do this, and do not take one of the more elaborate mitigation steps as described below, your Rails applications will be compromised, you will lose the servers you run on, and your business will suffer substantial losses.

Two of my businesses run on top of Rails 2.3.X, because they were started when that was the best option for writing production software on Rails.  This is true of many, many commercial Rails applications (loosely defined as “ones which make money for businesses”), both ones which are used only internally and ones which are public-facing, like SaaS sold on the typical pay-for-it-monthly plan.

Rails 2.3.X is, with the release of Rails 4, currently unsupported.   What does that mean?  It means that the Rails Core team has washed their hands of dealing with security problems on the platform.   This means that if, for example, Rails had another bug or series of bugs like the string of Severity: Apocalyptic vulnerabilities disclosed in January, individual users would be responsible for arranging for their own response rather than just waiting for a new release and bumping their gem version.

Why Is Anyone Still on Rails 2.3.X?

I know many folks in the Rails community wonder why anybody could still be using a years-old system.  Suffice it to say that Rails 2.3.X remains a productive, viable way to develop applications.  Some sites which are under active development daily still run on 2.3 — Github, as one prominent example, runs on an internal fork.  (More on them later.)  They’re one of a dozen prominent companies I could name with the main application still on 2.3.  Other companies have software which is more or less done running on 2.3 — for example, many companies run internal HR, accounts receivables, CMSes, etc which were written by boutique Rails consultancies back in 2010 and haven’t been upgraded because HR hasn’t really changed all that much.  Some companies even run across multiple versions of Rails, where the main app might be running on a fairly recent version but the supporting cast of web services, admin dashboards, and internal company tools are at whatever version they were at when they were created.

Much love to Rails Core for continuing to push the state of the art in web development forward, but not every app needs to be on that bus.  Bingo Card Creator, for example, still has several hundred thousand dollars of commercial viability left in it, but Bingo Card Creator is done.  It has achieved everything it ever wants to do in terms of bringing bingo cards into an AJAX-y web application, and doesn’t have any possible upside for moving to the new hotness.  That said, it still needs to avoid losing users’ passwords/credit cards to hackers, and it can’t host a botnet within my firewall.  That would be, ahem, problematic.

[An aside: Let me sketch out “problematic” a bit: Appointment Reminder — also on 2.3.X — has patient information from several large hospitals in it.  Leaking this information would be, obviously, severely negative for my customers, their patients, and my business.  In addition, I’d be subject to fines which scale to how many bytes of information I have control over…  yeah, ouch, I know.  I’m insured against that risk, but if I tell my insurance company “Health and Human Services wants $600,000 from me.  I guess you just lost your bet with me that I’d be competent enough to secure my server eh?” one of the first questions they’re going to ask is “Did you actually take those security measures you promised you take, like say using all commercially reasonable means to maintain patches on your applications?”, and if I answered in the negative they’d say “Wonderful, this loss isn’t covered under the policy you bought.  Hope you have $600,000 lying around.  Thanks for your business.”  Data security is serious business for many people.]

The Rails Core Team Is Not At Present Interested In Supporting 2.3.X Going Forward

A few people, including myself, have told Rails Core that their businesses use 2.3 apps and that continued support for 2.3 would be a major win for them.  Rails Core is uninterested in supporting 2.3 going forward.  To be totally clear on this point: that’s fine.  Rails is an OSS project which implicitly requires a $X million budget a year in employee salaries to continue running, and those salaries are overwhelmingly paid by for-profit companies.  Those companies graciously allow their employees to contribute their time and IP back to the community.  If 2.3 support isn’t a priority for Rails Core and/or their employers, then obviously they’re free to take the project in directions which are.

One reason why 2.3 support is not a priority is that, as every maintenance programmer knows, supporting old software involves a lot of boring scutwork.  Rails Core felt there wasn’t a manpower or monetary budget for underwriting this scutwork.  Some people expressed the opinion that folks asking for 2.3 support were freeloaders who were prepared to contribute neither patches nor money towards the effort.  That’s a prediction that, in many cases, I would have had a lot of sympathy for.  It’s very easy to criticize people who actually do things for a living on Github/Twitter/etc.  To head this off, when I made feelers about maintaining 2.3 support, I made it explicit that my company would cut a $10,000 check to make it happen.  (Kalzumeus Software is, to put it mildly, not nearly where most of the money is being made in Rails 2.3 apps, but $10,000 is trivially affordable at even our microscopic scales, and I thought this would demonstrate that The Enterprise (TM) would be able to fund this effort, in much the same way that The Enterprise (TM) spends billions of dollars a year on software maintenance.)

I was told, informally, that that was a nice gesture but it wouldn’t sway minds.  So I started looking at other options.

Your Options If You Are Currently On Rails 2.3.X

1)  Do nothing and, with probability of 100%, get your server owned.

Have I beaten this drum enough?  Thump thump thump.  If any server operated by your company is on an unpatched version of 2.3, that server will be compromised by the adversary.  Every Rails 2.3 application is, as of today, unpatched.  (Did you backport the fixes for e.g. the translate helper method XSS bug?  No?  Then you’re running unpatched.  That bug is actually from 2011.  The ones from Q3 2013 are going to be much more interesting.) You cannot afford to remain like this.  You will wake up one day wondering “Why is my web server trying to download and run a rootkit?” … well, if you’re lucky enough to detect it prior to being rooted.  Since covering this subject in depth in January I have heard harrowing tales of, e.g., CTOs at well-regarded companies who will not allow engineers to upgrade from vulnerable versions of Rails because they’re afraid the patched versions will cause regressions in their applications.  That will be a very interesting excuse to trot out to management or the board when asked why you lost 100,000 credit cards or why your entire network has joined a botnet.

2)  Rewrite your applications for Rails 3 or Rails 4, which are currently supported.

You can upgrade an application from Rails 2.3 to Rails 3.  Upgrading point releases is (aspirationally) a fairly painless experience on Rails, but the Rails 2 to Rails 3 transition is a serious engineering project for non-trivial applications.  A short list of issues:

  • Many of the gems/plugins which you might be using with your current application will not be compatible with Rails 3.  You will either have to upgrade them, find a (hopefully compatible) fork for Rails 3/4, or fork them yourself and +1 the number of old projects you’re maintaining.  Some gems/plugins which have variants which are compatible with Rails 3 have had breaking API or behavioral changes in the interim.
  • Rails 3 made a really good decision to enable HTML escaping by default, for preventing XSS attacks.  If you’re coming to this from an old application, you are likely going to have to dig in and either root out or mark harmless every time your views/helpers/model objects/etc return HTML when you expect them to.  That can be an involved project.
  • Most Rails projects of non-trivial size will play with the Rails internals, via calling private methods or monkeypatching core behavior.  For example, Bingo Card Creator takes over the standard way of locating layouts to support more extensive A/B tests.  Since the internal structure of Rails changed dramatically when it absorbed Merb, you’re going to have to rewrite that sort of code from scratch.  (If you’re watching from the sidelines, you might think this is inherently a dangerous practice, but be that as it may it is virtually universal in the community.)
  • You might decide to switch from Ruby 1.8 to Ruby 1.9, depending on how quickly you jump up the Rails version ladder.  That will make it a very, very painful transition for you, because the standard library API has breaking changes and the language syntax itself has breaking changes.  There are also non-breaking changes where the behavior of code will just change overnight — e.g. Date.parse(“5/6/2013″) can be May 6th or June 5th depending on what version of Ruby you are running.  And the encoding issues.  Oh God.

My ballpark estimate for upgrading Bingo Card Creator to Rails 3, assuming market rates for Rails contractors, is approximately $20,000.  It is substantially higher for Appointment Reminder.  Neither of those applications even approach the complexity of some of the commercial Rails 2 applications.  It’s easy to imagine many companies spending six figures or more on the upgrade.  (If this is difficult for you to imagine, consider that the fully-loaded monthly cost of a full-time intermediate Rails programmer is approximately $20,000 in some localities, so if you use more than 5 man-months, there you go.  There are many, many companies for which that would be not nearly enough to complete the upgrade.)

This assumes that your application is in a happy place to begin upgrading today — for example, it has been well-maintained, has very good test coverage, and you have the original programmers on staff.  Unfortunately, many applications in maintenance mode are not in this position.  They might have been ordered back in 2010 for $40,000 from a boutique consultancy that no longer exists, for example.  That doesn’t inhibit a company’s day-to-day use of the application in 2013, because CRUD apps don’t suddenly stop working even if the original programmer dies, but would substantially complicate a very in-depth maintenance project.

3)  Support Rails 2.3 yourself

You can, of course, fork the Rails 2.3 code and improve on it yourself, including by developing and incorporating security patches.  Github took it upon themselves to do this, for example.  It’s viable if you have a team of very capable programmers and the bandwidth to drop everything you are doing at 3 AM in the morning and address new Rails vulnerabilities.  (Again, if you don’t do that, your server gets owned.)

This option is not for the faint of heart.  Some parts of the Rails framework are very ambitious engineering — take ActiveRecord, for example.  As is true with most software projects, even the people who wrote it the first time don’t fully understand what it is doing.  You get to start working on rebuilding their undocumented organic knowledge now, and doing security patches later.  You might think that you have a helpful ally in the test suite but it is not actually deterministic, because many of the tests rely on things like e.g. the ordering of keys in a hash, and this is not guaranteed on Ruby 1.8.

If you’re Github or another large consumer of Rails, you might be able to guarantee sufficient availability of competent, security-focused programmers such that you can both proactively find vulnerabilities in Rails 2.3 and reactively patch announced vulnerabilities before your app servers join their friendly neighborhood botnet.  This is not viable for all companies which use Rails 2.3 in a commercial fashion — for example, if you’re an insurance company which has a line-of-business Rails application, it is highly unlikely that you have the engineering bandwidth to do this.  My company has the equivalent of 0.2 full-time intermediate programmers working for it, and if I were to keep on top of Rails security, I’d probably have no time for improving my applications  or for writing code to support marketing/sales goals.

4)  Pray that someone supports Rails 2.3 for you

Rails Core obviously aren’t the whole of the Rails community.  It is possible that somebody else will take up the challenge of supporting Rails 2.3.  For example, Github has their fork, as mentioned.  It is possible that you can just sit downstream of that fork and have things work for you.  I know a few people who are implicitly dependent on the Github fork, for example.

Why didn’t I go down that route?

  • Github’s fork currently supports Github’s needs, but it isn’t a formal commitment to the community or anything.  They could drop it at any time, without notice.  (I might be outside the loop, but I don’t even remember it being announced as existing.)  This makes it difficult to rely upon for consequential business decisions, like “Can I afford to commit all weeks in October 2013 to other projects or do I need to keep something in reserve for a panic-upgrade from github/rails to whatever my next best option is then?”
  • As an informal project, there is no release process for Github Rails.  You’ll have to build your own gems for it, on your own schedule, without guidance as to where known-good commits are on their work-in-progress branch.  This could be problematic if you slurped down a security patch and got Github-specific code which, e.g., broke Ruby 1.8 compatibility (check their commit history).
  • Github’s fork of Rails 2.3 is, to my cursory view, not designed to be exactly compatible with 2.3 as used by places other than Github.  It is an extraction from their working system.  I can’t be sure that moving my systems to Github Rails isn’t going to cause them to break in all sorts of fun and subtle ways.  (For example, see this commit, which rips out the default YAML processor.  That is, in isolation, a decision that I’m totally on-board with and am reasonably sure won’t break my app.  One commit down, 53 more to audit.)
  • I don’t think “I had no formal relationship with Github but decided to rely on them because they’re good people” is an answer that lawyers at Health and Human Services or my insurance company would tolerate as a commercially reasonable business practice, in the event that a compromise of my site eventually happens.

5)  Pay for a commercial fork of Rails 2.3

So after looking through the other options, and getting turned down by Rails Core on my attempt to essentially buy support for Rails 2.3 from them, I started looking for other people to take my money.  It turns out that there is a consultancy in Germany called Makandra which maintains an internal fork of Rails, because they have ongoing support obligations for 50ish applications currently in commercial use by their customers.  They also have a team of a dozen Rails experts.  This puts them in a pretty good position vis-a-vis new security issues: they’re already committed to fixing them quickly (since they have 50 apps depending on it), and they have sufficient engineers to be able to do that.

So I made them a proposition: If they cleaned up their fork such that it could be consumed outside their company, and contractually promised to maintain that fork against newly discovered vulnerabilities in Rails, I would be willing to pay them a substantial amount of money for that.  This frees me from having to do it myself, and I have high confidence that “We engaged a firm of engineering experts to handle that aspect” will pass muster with customers, regulators, and my insurance company.  This is similar to the Long Term Support option that you could buy from e.g. RedHat if you didn’t want to be on the latest version of their distribution.  The name of the fork is, appropriately, Rails LTS.  You can buy access to it and use it in production today.

It seemed like a bit of a waste for this relationship to only benefit our companies, given that Rails is OSS and we both continue to benefit from the community, so we hashed out a way to make it work well for everybody.  Makandra has substantial ongoing engineering expenses in doing the scutwork that Rails Core correctly observed the community is unwilling to do, and it cost them even more to get it into a state where it was releasable for consumption outside the company.  Accordingly, “I’ll just pay you to do that, and you OSS the code” wasn’t going to work for them.

So we split the difference: we agreed that I would be Customer Zero for Rails LTS, guaranteeing that it was commercially viable for them to do the engineering work necessary to release it.  They will sell it to anybody, for very, very reasonable rates relative to the amount of money Enterprise software companies usually charge for commercial support agreements.  If you buy it, you get support integrating it into your application and, more importantly, guarantees with regards to timelines for them to address new issues.  This means that you can sleep soundly know that it is someone’s contractual obligation to drop everything and address any Priority: The World Is Burning vulnerabilities which get discovered in August.

Normally, when two enterprises get into a support contract with each other, the community doesn’t explicitly benefit.  I wasn’t thrilled with that notion, so after some negotiation we agreed on a model which preserves Makandra’s interests but also benefits from the community: all code produced under this arrangement is OSSed under the MIT license (the Rails standard) 10 days after being produced.  This means that if you need it to keep a commercial service up and running, that is something you can buy, but if you have a hobby project and are OK with a 10-day window for possible exploitation, you can just slurp down patches from their Github account as they become available.  Nothing the community currently has gets taken away as a result of this arrangement: you can still choose to use the Rails Core version of 2.3 on your own recognizance (and get no security patches), or you can buy Rails LTS (and get new patches written in a guaranteed timeframe for new issues), or you can use the community version of Rails LTS (which did not exist but for me bankrolling it) and get your security patches taken care of for free.

The community patches are, naturally, available for Rails Core to backport if they would like to do so.  I approached them about doing that, and they say it is a possibility if Rails LTS turns out to be a sustainable thing.  That was prior to it being funded and released.

You Might Have An Objection To This.  I’d Like To Preemptively Answer It.

“But Patrick, switching to a fork is hard!”

One of the core design goals of Rails LTS was making sure that you can upgrade to it without requiring that you modify your app.  If your Rails app uses Bundler, it’s as simple as changing a line in your Gemfile and running “bundle update”, then redeploying.  If you don’t use Bundler, you can build the gems locally and then deploy them as per these instructions.  I tried doing that for Bingo Card Creator, which didn’t use Bundler, but it turned out to be faster to just migrate to Bundler.  (If you have a 2.3 app that hasn’t done that yet, instructions for switching are here.  This took approximately 45 minutes for me, mostly as a result of not having a clear all-in-one-place view of my app’s dependencies.)

Since Rails LTS is exactly the current version of Rails plus security patches and tests (diff them if you don’t believe me), it is highly unlikely to cause problems for your application.  They’ve promised that featurewise it is frozen in amber and, unless a security issue necessitates it, the API will not change as a result of security releases.  (This is a policy whose adoption would be widely beneficial in the Rails community.  cough)  Both of my applications worked with flying colors.  There’s an optional “hardened mode” you can activate (one line in your environment file) to disable some features of Rails which are often unused and have recently been demonstrated problematic, like the “Rails can take XML requests and create objects from them then insert them directly into the params hash, including perhaps deserializing embedded YAML in the XML, which can result in arbitrary code execution” (a feature which, to put it mildly, is not required by much of the community).

“Can we trust this to actually, you know, work?”

Great question.  They’re sufficiently credible to me that I cut them a check for $10,000 and have essentially delegated to them a portion of the application security burden for my business, which is a bet-the-business issue for me if I can’t get it right.  They have a strong record of bringing client projects in, including ones which are technically ambitious.  They have a demonstrated commitment to open source and to Rails 2.3’s ongoing viability as a platform.  I think that’s the best testimonial I can give them, but feel free to spend a few hours spelunking in their code if you want to feel more secure about making the decision.  (If you want to look at the Rails LTS code in particular, can I draw your attention to the now-actually-working test suite which newly includes tests for the security patches released this year?)

“Why would we pay money for OSS code?”

You’re not really paying money for OSS code which already exists, you’re buying the guarantee that new code gets written at non-deterministic times in the future before the lack of it causes an emergency for your business.  It’s closer to an insurance policy than it is to source code.

You’d pay money for this for the same reason you pay engineer salaries: because the code provides business value to you.  If you’re a CTO, you should understand that there is definite value in not having to bring your applications down because their servers have been rooted.  You should also understand that drop-everything-hair-on-fire vulnerability resolution is not something your engineers are particularly good at and not something which you want periodically disrupting them from shipping applications to your paying customers.  You can either pay your engineers to continue monitoring and patching Rails for you, or you can buy Rails LTS.  Your call.  Their entry-level plan costs ~$200 a month, which is approximately the cost of one or two engineer-hours.  (Are you a Rails dev?  If you think you’re capable of identifying and fixing Rails framework issues and you don’t charge that much, raise your rates.  Are you a CTO?  If you think your engineers are capable of this and they don’t charge that much, prohibit them from emailing me, so that I don’t introduce them to more rational firms.)

If you don’t have $200 in the budget, and you at a real business, you either need a) a reality check about your business or b) a heart-to-heart discussion about what will happen when your servers join a botnet.  If you’re running a hobby project or are still in the first few weeks of running your business, then you should switch to the community version of Rails LTS, which is a substantial improvement to your security at no monetary cost.  Or you can just wait until your server gets owned, because that reliably makes people discover budget for security.  (It also could very well result in firings, so I wouldn’t rush to this option.)

“The Rails LTS plans are too expensive!  I would pay for this but only if it cost, like, $5 a month!”

Horsepuckey.  The hypothetical person saying this is a textbook pathological customer: they’re both deeply irrational (if the app’s security was worth $5 a month then the right answer is probably to shut it off and save the server cost) and likely to be far, far, far too much headache for professional Rails engineers to have to deal with.  I’m glad their mail is not going to be in the same inbox as mine when I ask questions about new security issues.

“I could totally duplicate all that work for free!  And I’m going to do that!  Nyaa nyaa!”

In the immortal words of OSS passive-aggressivity, patches welcome.  It would be an awesome thing for everybody if you did a lot of free work.  After finishing your free dev work, you also need to be a dab hand at working the politics of a successful open source project, again for free.  They won’t necessarily be actively hostile to your pull requests, but keep in mind that Rails Core does not consider 2.3 a priority and signaled strong distaste for the notion of having to do any release management for 2.3 patches even if the code were given to them on a silver platter.

(But horsepuckey, you’re not going to write free patches for Rails 2.3, or you’d have commits you could point to demonstrating your willingness to do this over the last 6 months.)

“You’re using your evil marketer wiles on me in pursuit of your selfish interests!”

I am, indeed, attempting to get you onto Rails LTS rather than unpatched 2.3.  Why?  Because as a fan of Rails and long-time member of the community, whose business has benefited enormously from it being available, I do not want to see other community members get rooted.

I have no direct financial interest in you deciding to buy a support contract for Rails LTS.  I don’t get a commission and I’m not an investor, I’m just the anchor customer.  My $10,000 is where my mouth is.  Well, no, my $10,000 is presumably sitting in a German bank account, but you get the general idea.

How Do We Switch?

It’s all on their website.  Switching to the community edition (free) is a one-line change in your Gemfile, and you can and should do that today if you’re already on 2.3.18.  (If not, upgrade to 2.3.18 first.  Then, have a frank discussion about security priorities, since you should have been on 2.3.18.)  To get a support contract and your username/password to access their private git repository (which gets the patches immediately on release rather than 10 days later), get in touch with them via the website.  When I did it it required a paper contract and a wire transfer to Germany, so it won’t complete as quickly as git clone will, but you’ll be on the way to getting this resolved before the next round of patches drop.

P.S. Remember, you need to have all of your Rails apps patched continuously, not just “the main one.”  If you miss e.g. a staging server, a simple service which hooks into the main app, an analytics side-project knocked up by an intern, or an old installation of Redmine, then that box will be rooted, and if that box is inside your firewall you can basically assume you will lose ever box attached to it.

What The Rails Security Issue Means For Your Startup

Posted on January 31, 2013 by in security, startups

January has been a very bad month for Ruby on Rails developers, with two high-severity security bugs permitting remote code execution found in the framework and a separate-but-related compromise on rubygems.org, a community resource which virtually all Ruby on Rails developers sit downstream of.  Many startups use Ruby on Rails.  Other startups don’t but, like the Rails community, may one day find themselves asking What Do We Do When Apocalyptically Bad Things Happen On Our Framework of Choice?  I thought I’d explain that for the general community.

Nota bene: I’m not a professional security researcher.  Mostly, I sell software.  In the course of doing that, I (very occasionally) do original security research.  I did no significant amount for these bugs, aside from mitigating them for my own applications.  I am currently engaged in a Ruby on Rails security safari, and anticipate publishing the results of that in February, after responsible disclosure to the relevant security teams.  If you don’t know enough to know whether I’m trustworthy with regards to generic advice, pay someone you trust to get advice on this.

Don’t skip this post because you’re not a Rails developer.  If you’re reading this blog, this matters to you.

Background: What Has Been Happening in Rails-land?

Ruby on Rails recently released two sets of security patches (announcements here and here), in response to related vulnerabilities discovered in the frameworks.

How bad were those bugs? Severity: Apocalyptic.  They permitted attackers to execute arbitrary code on virtually ever Ruby on Rails application, without requiring that the application do anything to enable the attack other than “be hooked up to the Internet.”

What does “execute arbitrary code” mean?  Literally, it means that the attacker can choose to have your server execute any code they can dream up.  In practice, it means that you lose the server that the code is executing on.  Any further access to that server or applications on should be assumed to be compromised.

What went wrong?  This has been covered in more detail by security researchers, in posts such as here and here.  A brief description: Ruby on Rails makes extensive use of a serialization format called YAML, most commonly (you might think) for reading e.g. configuration files on the server.  The core insight behind the recent spat of Rails issues is that YAML deserialization is extraordinarily dangerous.  YAML has a documented and “obvious” feature to deserialize into arbitrary objects.   Security researchers became aware in late December that just initializing well-crafted objects from well-chosen classes can cause arbitrary code to be executed, without requiring any particular cooperation from the victim application.  Since then, the bug hunt has been on: security researchers have been actively finding lots of ways in the Ruby on Rails code base, and in related code bases, to cause the application to deserialize YAML which is in some way under the control of the attacker.

So far this has included:

  • Rails, for programmer convenience, used YAML to implement JSON deserialization.  JSON is designed to get into Rails quite easily indeed — just POST it at the server, wham, YAML.load(attacker_data) happened.  (The actual mechanics of achieving that were more complicated, but that’s the practical upshot.)
  • Rails allows XML documents to include YAML attributes.  That decision has caused a bit of head scratching, since it seems like a curious choice for most programmers in the community, but be that as it may this allowed posting XML at Rails apps to be trivially exploited.
  • Rubygems used YAML to hold metadata about each gem submitted to it.  An attacker was able to create a malicious gem, cause the Rubygems web application to evaluate the metadata contained in it, and thereby compromise the Rubygems server infrastructure.
  • February will see more compromises, with my certainty of this prediction approaching my certainty that the sun will rise tomorrow.  There exist many, many other code paths in Rails to get to YAML.load().  Some of them will be found to be amenable to attackers, either (worst case) for all or substantially all Rails applications or (still bad case) to Rails applications whose application logic involuntarily cooperates with the attack.  (i.e. In the worst case, attackers root every unpatched Rails app on the Internet.  In the best case, attackers only root some apps and they often have to have an expert do a modicum of marginal work to do so.)

Ruby on Rails security sucks lolz amirite? No.  Well, no to the nuance.  Software security does, in general, suck.  Virtually every production system has security bugs in it.  When you bring pen testers in to audit your app, to a first approximation, your app will lose.  While Ruby on Rails cherishes its Cool-Kid-Not-Lame-Enterprise-Consultingware image, software which is absolutely Big Freaking Enterprise consultingware, like say the J2EE framework or Spring, have seen similar vulnerabilities in the past.  The recent bugs were, contrary to some reporting, not particularly trivial to spot.  They’re being found at breakneck pace right now precisely because they required substantial new security technology to actually exploit, and that new technology has unlocked an exciting new frontier in vulnerability research.  It sucks for users of Rails that Rails is currently on the bleeding edge — believe me, after having lost 3 consecutive nights to patching my own applications, I know — but it would suck much, much worse if the Bad Guys had found these first and just proceeded to remote-own every Rails app on the Internet.  That is, by the way, an achievable scenario.

Was anyone actually compromised?  Yes.  The first reported compromise of a production system was in an industry which hit the trifecta of amateurs-at-the-helm, seedy-industry-by-nature, and under-constant-attack.  It is imperative that you understand that all Rails applications will eventually be targeted by this and similar attacks, and any vulnerable applications will be owned, regardless of absence of these risk factors.

Will anyone else be compromised?  Yes.  Thousands upon thousands of Ruby on Rails applications will be compromised using these vulnerabilities and their spiritual descendants, and this will happen for years.

  • Many Rails developers have not reacted to this news with the alacrity they should have.  (See next question.)  These applications may be compromised already.
  • There are many Rails applications which were created years ago, which are not under active development any more, for whom no-one is responsible for applying security patches.  Any of these applications which are publicly routable on the Internet will be compromised.
  • There are many Rails applications which are installed by end users, some of whom do not have security expertise.  For example, Redmine — an open source developer productivity tool — is commonly installed at individual companies.  Every publicly accessible Redmine instance which is not patched will be compromised.
  • Ruby on Rails lacks a CMS with the mindshare of, say, WordPress, which is good, because every unpatched Ruby on Rails CMS delivered to a non-technical company to serve as their website or backend to their mobile application will be compromised.
  • There are many developers who are not presently active on a Ruby on Rails project who nonetheless have a vulnerable Rails application running on localhost:3000.  If they do, eventually, their local machine will be compromised. (Any page on the Internet which serves Javascript can, currently, root your Macbook if it is running an out-of-date Rails on it. No, it does not matter that the Internet can’t connect to your localhost:3000, because your browser can, and your browser will follow the attacker’s instructions to do so. It will probably be possible to eventually do this with an IMG tag, which means any webpage that can contain a user-supplied cat photo could ALSO contain a user-supplied remote code execution.)
  • Many companies — including ones which do not even consider themselves Ruby on Rails shops — nonetheless have a skunkworks project running somewhere.  For example, they might have a developer who coded a quick one-off analytics app, which is accessible outside the firewall so that sysadmins could check server loads from home.  If the app is on the public Internet, it will be compromised.
  • Many Ruby on Rails shops have good development practices and no longer have the “monorail” anti-pattern, where everything their company does is in one gigantic Rails app.  They have already patched most of their main apps, but they missed one.  Maybe it is the customer support portal at admin.example.com.  Maybe it is a publicly accessible staging server at EC2 spun up by a developer who has since left the company and not shut down because, hey, $20 a month.  Maybe it is a 20% project by a junior engineer which he has on the back burner for the moment.  It doesn’t matter why this app was forgotten: if it is publicly accessible, it will be compromised.

What was the proper way to react to these patches?  Patch immediately, install a workaround immediately, or pull the plug on your application.  (“Pull the plug” means disconnect it from the Internet or shutdown the server while you get a mitigation plan into place.)  You should have distinct memories of you or someone under your employ having at least two separate incidents in the last four weeks in which they dropped everything they were doing and immediately took action to resolve these problems.  Immediately means exactly that: right now, not during the next schedule code spring, not tomorrow, not in an hour.

I was up at 3 AM Japan time applying these patches, twice.  If the next patch drops at 3 AM your local time, someone should be applying it immediately.  Computers can count to big numbers very quickly indeed.  A six hour window between a patch dropping and the start of business the next day is more than enough for an automated scanner running on a botnet to have tried compromising substantially every Rails app on the Internet.  (Do you disagree?  You are overestimating how hard it would be to find your application.)

Aren’t you exaggerating?  Our application isn’t particularly high risk!  We aren’t high-profile, it doesn’t have obvious monetary return for exploiting it, etc etc. Good thing you aren’t really saying that, but you might be at an Internet cafe next to an engineer who has poor reading comprehension, so help me explain this to him: nobody needs to care about your application to compromise it using these vulnerabilities. They can be exploited in a totally automated manner over the open Internet, requiring zero knowledge of e.g. what version of Ruby you are running, what version of Rails you are running, what your URL structure looks like, etc.  (Somebody suggested “How would you determine which servers were running Ruby on Rails?”  Answer: It’s absolutely trivial to detect Rails applications in a scalable fashion, but why bother?  Fire four HTTP requests at every server on the Internet: if the server is added to your botnet, it was running a vulnerable version of Ruby on Rails.)

Aren’t you exaggerating? Clearly this would take a lot of specialized expertise to exploit! Yep… the first time. Now that people know how the exploitation is done, however, you could do it by just copy/pasting one of the proof-of-concept scripts or b) running a browser bookmarklet. (I am not passing out that browser bookmarklet, because I think that would inevitably lead to mischief, but just know that you’re rootable in a click if you didn’t take action on this. And, by the way, have been for three weeks or so now.)

We’re A Startup.  What Happens If We Lose A Server?

If you lose one server, you will lose every server, with very high confidence.  If, for example, you are a Python-using shop which had a Redmine instance running around with no code on it, and you lose that Redmine server, you can expect a skilled attacker to then pivot from that privileged location within your network to start compromising other servers on your network.  At this point, you need to have done absolutely everything right to make it impossible for that skilled attacker to prevail, and you almost certainly have not.  (Compelling evidence that you’re not as good as you think you are: you already had one vulnerable application which could be compromised over the open Internet.  To a certain philosophy, that isn’t your fault, but the attacker gets root regardless of whose fault it is.)

The actual steps a pen-tester would take to root your other boxes are pretty academic after they have one.  (For example, you can start probing other machines on the network for vulnerable services, use credentials found on your compromised machine to suborn other machines, take over routing hardware using vulnerable administration panels and then start intercepting all network traffic, etc etc.)  Just take it as a given, you will lose.  Companies much larger and smarter than you lose everything when this happens, essentially every time it happens.

We’re A Startup.  What Happens If We Lose Every Server?

A short preview of coming attractions:

  • You will lose the next several weeks out of your schedule dealing with this issue.
  • You will have to take down all of your applications and rebuild all your servers from scratch.
  • You can assume the attacker now has a copy of your source code, all credentials you have, all your databases, and all information you had like e.g. log files.
  • Do you take credit cards?  Were you taking credit cards through an exploited application?  You now have a PCI-reportable data breach.
  • Your local jurisdiction may have legal requirements that you notify the people whose data just got exposed.
  • You now have a public relations nightmare on your hand.
  • In addition to compromising any customer data you possessed, you have made it possible for diligent attackers to compromise those customers elsewhere.  The most trivial example is, if you did not implement password storage correctly, you have just handed the attackers a list of email addresses and associated passwords which they can now re-use on higher value targets like e.g. bank accounts or Gmail, because many users re-use their passwords everywhere.  (You use bcrypt?  Wonderful.  Did  the attackers turn it off when they rooted all your applications?  Can you conveniently check that, knowing that you cannot trust the contents of any logs on those compromised servers?  No?  OK, so instead of losing all the passwords, we can upper bound exposure at only all users who logged in since the attack started.  That’s an improvement… sort of.)

Basically, it’s Very Bad, but not the end of the world.  You’ll probably need expert help to get through it, like you would need if e.g. you got sued.  Unfortunately, lawsuits generally give you weeks of notice and progress slowly, but security vulnerabilities often give you negative several hours notice and get worse for every minute left unchecked.

We’re A Startup.  We Don’t Use Ruby on Rails So We’re Totally Cool, Right?

Can you enumerate every account on the Internet where you have a password and also every service consumed by your business?  Go ahead, take as long as you need: it is very important that you don’t miss one.

OK, let’s start with the obvious: Look for analytics providers and other folks on that list who have instructed you to embed JS on your website.  If I do this exercise, I come up with at least three results here.  Do any of them use Ruby on Rails?  (Are you sure?  Remember, if they have at least one Rails app on their network…)  Great.  If they didn’t patch in a timely manner, you should assume that Javascript you’re embedding on your website is in the hands of the enemy.  It is now a cross-site scripting vulnerability against every page it is embedded on.  Do you embed it on e.g. log in pages or anywhere your admins expose their own all-powerful admin cookies?  Boo, now the enemy has your password / cookies / etc.

Alright, let’s move down the line: Look for anybody who implements OAuth/Facebook Connect/etc.  Do any of them use Ruby on Rails?  Are you sure?  If they haven’t patched, you’ve handed the union of all privileges over the linked accounts to the attackers.

Alright, let’s move down the line: Consider everybody who has a copy of a password which you re-use elsewhere.  (You shouldn’t be re-using passwords, or variants of passwords, but I ignored that advice for years so I’m betting a lot of you did, too.  Maybe not you, specifically, but you know that chap in marketing who is great with people but thinks MSWord is complicated?  Consider whether he has access to anything sensitive in your company.  He does?  Well, sucks to be you then, but good on your for password security.)  Do any of them use Ruby on Rails?  Are you sure?  Did they use bcrypt/scrypt/etc to properly secure passwords at rest, and did they patch these vulnerabilities fast enough to prevent attackers from pulling them off of the wire?  Are you sure?  If you’re not sure of all of these things, consider every password compromised and take action appropriately.

One of my friends who is an actual security researcher has deleted all of his accounts on Internet services which he knows to use Ruby on Rails.  That’s not an insane measure.  (It might even be inadequate, because all the folks who are compromised are probably going to lose their database backups as well.  Well, if they have database backups.)

These are just a sample of ways in which these vulnerabilities can ruin your day.  They are very much not an exhaustive list.  If you believe in karma or capricious supernatural agencies which have an active interest in balancing accounts, chortling about Ruby on Rails developers suffering at the moment would be about as well-advised as a classical Roman cursing the gods during a thunderstorm while tapdancing naked in the pool on top of a temple consecrated to Zeus while holding nothing but a bronze rod used for making obscene gestures towards the heavens.

Somebody Dropped A 0-Day On Rubygems. What If It Happens To Me?

Yes, that certainly sucks royally.  Rubygems wasn’t even exploited using the patched Rails vulnerabilities — an attacker just learned something which worked (again, we’re on the leading, bleeding edge of security research here), applied it in a novel fashion, and compromised the Rubygems application.  As of me writing this it looks like we avoided the Ruby-ecosystem-wide apocalypse that would have happened if they had started backdooring gems, but let’s just focus on the immediate fallout: their system got compromised.  What if one of yours did, like that?

The first step is a preventative inoculation: If you run an application on the Internet, you should today establish a security contact page.  It only needs to include two things: a working, monitored email address and a PGP key.  Bonus points for giving some sort of public recognition to people who report security vulnerabilities to you in a responsible matter.  This helps to co-opt some security researchers so that they e.g. get in touch with you about the problem prior to just going ahead an exploiting it.  Software security has a curious system of social norms, where scalp collecting both builds both karma and pseudo-currency.  It’s bizarre, but just take this on faith: having a security page with a working email gives you a certain amount of We Should Avoid #’#(ing Their #()#% Up Without Asking First street cred.  (Naturally, like any social norm, that is a preventative measure rather than a panacea.  However, given that it is a well-understood norm, it gives you a bit of an edge in the PR battle should someone decide to just drop a 0-day on you.)

Good security pages to pattern after: 37signals (I particularly like how this page works for responsible disclosure while, in a dual-audience fashion, also doubles as being great marketing copy), Twilio, Heroku (again, dual audience), etc.

Have a plan for responding to security incidents. I call mine the Big Red Button. Thomas, a security consultant friend of mine, accurately observed that these probably caused the first Big Red Button events that many folks in the Rails community have ever had to deal with. We should learn from our experiences here.

For example: I pushed the Big Red Button at 3 AM in the morning, twice this month, to apply critical security patches and work-arounds.

So did I do a great job of addressing this problem? No, I did a pretty effing atrocious job of addressing this problem. Specifically, I have two old-as-the-hills Rails apps running on 2.3.X at the moment. Waaaaay back in 2010, Mongrel and Rails had a bit of a compatibility issue, and I solved it via a monkeypatch. The monkeypatch relied on a hardcoded version number, which I have been hand-incrementing every time I update Rails. It’s literally on the redeploy checklist, next to a note “TODO: This is stupid and should be fixed when I get a moment.”

I did three Rails app upgrades locally, three test suite runs, and three sets of smoke tests when applying one of these patches. The one in the middle happened to be Appointment Reminder, which is an application that has to be up during the US workday. Unfortunately, because I was exhausted while following my deployment and smoke test checklists, I a) forgot to bump the version number in that monkeypatch and b) did not follow the part of the smoke test which would have clued me on to “This is going to cause log-ins to fail on some browsers.” That resulted in some breaking downtime for some customers during the US workday, and me having to send an apology to all customers. That sucked horribly.

I have now fixed my monkeypatch to not require hard-coding the Rails version, simplified some of my deploy procedures, and am working in the next several months on beefing up my testing suite. Also, lesson learned about resolving “TODO: This is stupid” when it would take 5 minutes to do rather than having it blow up in my face.

There, that’s an experience I went through. Now you know the punchline, so hopefully you don’t have to go through it as well.

Similarly, we can observe:

  • We need an updated list of all applications running on our servers, so that we know when a problem with a technology stack affects them, even though this sounds like a boring Big Freaking Enterprise IT Department requirement. (And gulp their dependencies.)
  • For each tech stack we support, we need at least one expert following the primary source for security news for that tech stack.
  • We need whomever is responsible for product development and/or ops to, effectively, carry a pager for drop-everything-and-do-it-now resolution of security issues, just like we’d do for e.g. the server has fallen over or “our building is, physically, on fire.”
  • These requirements suggest minimizing the number of tech stacks we work with, even if that means passing up the new hotness occasionally.
  • Just like we have e.g. insurance on the building physically burning down, we should have some upfront investment in security. Good forms might include security training, outside consulting, or (if we’ve got a lot of money) contributing work towards securing tech stacks we rely on.

You Should Be At Defcon 2 For Most Of February

Big security vulnerabilities tend to be discovered in bunches.

Why does this happen?

  • Blood in the water attracts sharks. Some of my security friends would hate this phrasing, because “researchers don’t cause vulnerabilities, they find vulnerabilities”, but as a businessman who depends on software for his livelihood, I had exactly zero days of the last six years spent sleepless because of the latent vulnerability in Rails, but two days this month due to the pressing need for immediate mitigation. There are many more eyes pouring over Rails — and related projects — more closely now than typically. Many of them are white hats (yay!). Some aren’t. In general, there is a virtually infinite need for software security expertise, just like there is an infinite need for software, and there is a crushing lack of expertise which can meet it. Some folks who are capable of finding vulnerabilities are, due to attention/topicality/renewed interest/commercial potential/etc, now looking at Rails as of today.
  • Technology marches on. After you have a new exploit vector to play with, you can start applying some of the technology used to discover / develop / exploit it against other code bases, code paths, etc etc. For example, the first Rails vulnerability was parlayed within a day into a similar vulnerability in the MultiXml gem. The same underlying “YAML is very dangerous” realization enabled the Rubygems compromise. If I were working on e.g. Django, I would strongly suspect that security researchers are going to see whether they can find similar patterns on Django — it wouldn’t be the first time, since e.g. HMAC tampering vulnerability disclosures in Rails were followed up by similar findings on Django the same week.

I previously had a version of this post queued up right after the first bug dropped, but didn’t hit Publish because I got busy that weekend and thought it wouldn’t be timely anymore. That post included the lines “I will bet $1,000 at 100-to-1 odds that Rails suffers another code execution vulnerability before the end of January.” If you had hypothetically taken that bet, you would have lost.

You should expect February to be a very trying month for the Rails community and startups in general. Your security team should be at Defcon 2: be ready to respond to patches with particular alacrity, and expect there to be failures in the ecosystem outside of your ability to control them. For example, I’d make sure that you can rebuild systems without requiring access to Github / Rubygems / etc, and that’s (unfortunately) the tip of the iceberg.

This Sounds Like A #$()#%ing Disaster

That is primarily because this is a #$()#%ing disaster.

For my part, in addition to taking steps to fortify my own businesses, I’m (as time permits) doing some pro-bono security work on Rails. I do not have results which can be published yet. I strongly suspect based on early research that I will, in February, and I strongly suspect that other researchers (both white hats and the Bad Guys) are much, much better at this than I am.

Get ready. It will get worse before it gets better.