Archive | Rails RSS feed for this section

If Your Business Uses Rails 2.3 You Need To Move To A Supported Option ASAP

Posted on June 17, 2013 by in Rails, Ruby, security

Executive summary for your CTO: If your company runs Rails 2.3 apps, switch to Rails LTS, a commercially supported fork of Rails.  If you do not do this, and do not take one of the more elaborate mitigation steps as described below, your Rails applications will be compromised, you will lose the servers you run on, and your business will suffer substantial losses.

Two of my businesses run on top of Rails 2.3.X, because they were started when that was the best option for writing production software on Rails.  This is true of many, many commercial Rails applications (loosely defined as “ones which make money for businesses”), both ones which are used only internally and ones which are public-facing, like SaaS sold on the typical pay-for-it-monthly plan.

Rails 2.3.X is, with the release of Rails 4, currently unsupported.   What does that mean?  It means that the Rails Core team has washed their hands of dealing with security problems on the platform.   This means that if, for example, Rails had another bug or series of bugs like the string of Severity: Apocalyptic vulnerabilities disclosed in January, individual users would be responsible for arranging for their own response rather than just waiting for a new release and bumping their gem version.

Why Is Anyone Still on Rails 2.3.X?

I know many folks in the Rails community wonder why anybody could still be using a years-old system.  Suffice it to say that Rails 2.3.X remains a productive, viable way to develop applications.  Some sites which are under active development daily still run on 2.3 — Github, as one prominent example, runs on an internal fork.  (More on them later.)  They’re one of a dozen prominent companies I could name with the main application still on 2.3.  Other companies have software which is more or less done running on 2.3 — for example, many companies run internal HR, accounts receivables, CMSes, etc which were written by boutique Rails consultancies back in 2010 and haven’t been upgraded because HR hasn’t really changed all that much.  Some companies even run across multiple versions of Rails, where the main app might be running on a fairly recent version but the supporting cast of web services, admin dashboards, and internal company tools are at whatever version they were at when they were created.

Much love to Rails Core for continuing to push the state of the art in web development forward, but not every app needs to be on that bus.  Bingo Card Creator, for example, still has several hundred thousand dollars of commercial viability left in it, but Bingo Card Creator is done.  It has achieved everything it ever wants to do in terms of bringing bingo cards into an AJAX-y web application, and doesn’t have any possible upside for moving to the new hotness.  That said, it still needs to avoid losing users’ passwords/credit cards to hackers, and it can’t host a botnet within my firewall.  That would be, ahem, problematic.

[An aside: Let me sketch out “problematic” a bit: Appointment Reminder — also on 2.3.X — has patient information from several large hospitals in it.  Leaking this information would be, obviously, severely negative for my customers, their patients, and my business.  In addition, I’d be subject to fines which scale to how many bytes of information I have control over…  yeah, ouch, I know.  I’m insured against that risk, but if I tell my insurance company “Health and Human Services wants $600,000 from me.  I guess you just lost your bet with me that I’d be competent enough to secure my server eh?” one of the first questions they’re going to ask is “Did you actually take those security measures you promised you take, like say using all commercially reasonable means to maintain patches on your applications?”, and if I answered in the negative they’d say “Wonderful, this loss isn’t covered under the policy you bought.  Hope you have $600,000 lying around.  Thanks for your business.”  Data security is serious business for many people.]

The Rails Core Team Is Not At Present Interested In Supporting 2.3.X Going Forward

A few people, including myself, have told Rails Core that their businesses use 2.3 apps and that continued support for 2.3 would be a major win for them.  Rails Core is uninterested in supporting 2.3 going forward.  To be totally clear on this point: that’s fine.  Rails is an OSS project which implicitly requires a $X million budget a year in employee salaries to continue running, and those salaries are overwhelmingly paid by for-profit companies.  Those companies graciously allow their employees to contribute their time and IP back to the community.  If 2.3 support isn’t a priority for Rails Core and/or their employers, then obviously they’re free to take the project in directions which are.

One reason why 2.3 support is not a priority is that, as every maintenance programmer knows, supporting old software involves a lot of boring scutwork.  Rails Core felt there wasn’t a manpower or monetary budget for underwriting this scutwork.  Some people expressed the opinion that folks asking for 2.3 support were freeloaders who were prepared to contribute neither patches nor money towards the effort.  That’s a prediction that, in many cases, I would have had a lot of sympathy for.  It’s very easy to criticize people who actually do things for a living on Github/Twitter/etc.  To head this off, when I made feelers about maintaining 2.3 support, I made it explicit that my company would cut a $10,000 check to make it happen.  (Kalzumeus Software is, to put it mildly, not nearly where most of the money is being made in Rails 2.3 apps, but $10,000 is trivially affordable at even our microscopic scales, and I thought this would demonstrate that The Enterprise (TM) would be able to fund this effort, in much the same way that The Enterprise (TM) spends billions of dollars a year on software maintenance.)

I was told, informally, that that was a nice gesture but it wouldn’t sway minds.  So I started looking at other options.

Your Options If You Are Currently On Rails 2.3.X

1)  Do nothing and, with probability of 100%, get your server owned.

Have I beaten this drum enough?  Thump thump thump.  If any server operated by your company is on an unpatched version of 2.3, that server will be compromised by the adversary.  Every Rails 2.3 application is, as of today, unpatched.  (Did you backport the fixes for e.g. the translate helper method XSS bug?  No?  Then you’re running unpatched.  That bug is actually from 2011.  The ones from Q3 2013 are going to be much more interesting.) You cannot afford to remain like this.  You will wake up one day wondering “Why is my web server trying to download and run a rootkit?” … well, if you’re lucky enough to detect it prior to being rooted.  Since covering this subject in depth in January I have heard harrowing tales of, e.g., CTOs at well-regarded companies who will not allow engineers to upgrade from vulnerable versions of Rails because they’re afraid the patched versions will cause regressions in their applications.  That will be a very interesting excuse to trot out to management or the board when asked why you lost 100,000 credit cards or why your entire network has joined a botnet.

2)  Rewrite your applications for Rails 3 or Rails 4, which are currently supported.

You can upgrade an application from Rails 2.3 to Rails 3.  Upgrading point releases is (aspirationally) a fairly painless experience on Rails, but the Rails 2 to Rails 3 transition is a serious engineering project for non-trivial applications.  A short list of issues:

  • Many of the gems/plugins which you might be using with your current application will not be compatible with Rails 3.  You will either have to upgrade them, find a (hopefully compatible) fork for Rails 3/4, or fork them yourself and +1 the number of old projects you’re maintaining.  Some gems/plugins which have variants which are compatible with Rails 3 have had breaking API or behavioral changes in the interim.
  • Rails 3 made a really good decision to enable HTML escaping by default, for preventing XSS attacks.  If you’re coming to this from an old application, you are likely going to have to dig in and either root out or mark harmless every time your views/helpers/model objects/etc return HTML when you expect them to.  That can be an involved project.
  • Most Rails projects of non-trivial size will play with the Rails internals, via calling private methods or monkeypatching core behavior.  For example, Bingo Card Creator takes over the standard way of locating layouts to support more extensive A/B tests.  Since the internal structure of Rails changed dramatically when it absorbed Merb, you’re going to have to rewrite that sort of code from scratch.  (If you’re watching from the sidelines, you might think this is inherently a dangerous practice, but be that as it may it is virtually universal in the community.)
  • You might decide to switch from Ruby 1.8 to Ruby 1.9, depending on how quickly you jump up the Rails version ladder.  That will make it a very, very painful transition for you, because the standard library API has breaking changes and the language syntax itself has breaking changes.  There are also non-breaking changes where the behavior of code will just change overnight — e.g. Date.parse(“5/6/2013″) can be May 6th or June 5th depending on what version of Ruby you are running.  And the encoding issues.  Oh God.

My ballpark estimate for upgrading Bingo Card Creator to Rails 3, assuming market rates for Rails contractors, is approximately $20,000.  It is substantially higher for Appointment Reminder.  Neither of those applications even approach the complexity of some of the commercial Rails 2 applications.  It’s easy to imagine many companies spending six figures or more on the upgrade.  (If this is difficult for you to imagine, consider that the fully-loaded monthly cost of a full-time intermediate Rails programmer is approximately $20,000 in some localities, so if you use more than 5 man-months, there you go.  There are many, many companies for which that would be not nearly enough to complete the upgrade.)

This assumes that your application is in a happy place to begin upgrading today — for example, it has been well-maintained, has very good test coverage, and you have the original programmers on staff.  Unfortunately, many applications in maintenance mode are not in this position.  They might have been ordered back in 2010 for $40,000 from a boutique consultancy that no longer exists, for example.  That doesn’t inhibit a company’s day-to-day use of the application in 2013, because CRUD apps don’t suddenly stop working even if the original programmer dies, but would substantially complicate a very in-depth maintenance project.

3)  Support Rails 2.3 yourself

You can, of course, fork the Rails 2.3 code and improve on it yourself, including by developing and incorporating security patches.  Github took it upon themselves to do this, for example.  It’s viable if you have a team of very capable programmers and the bandwidth to drop everything you are doing at 3 AM in the morning and address new Rails vulnerabilities.  (Again, if you don’t do that, your server gets owned.)

This option is not for the faint of heart.  Some parts of the Rails framework are very ambitious engineering — take ActiveRecord, for example.  As is true with most software projects, even the people who wrote it the first time don’t fully understand what it is doing.  You get to start working on rebuilding their undocumented organic knowledge now, and doing security patches later.  You might think that you have a helpful ally in the test suite but it is not actually deterministic, because many of the tests rely on things like e.g. the ordering of keys in a hash, and this is not guaranteed on Ruby 1.8.

If you’re Github or another large consumer of Rails, you might be able to guarantee sufficient availability of competent, security-focused programmers such that you can both proactively find vulnerabilities in Rails 2.3 and reactively patch announced vulnerabilities before your app servers join their friendly neighborhood botnet.  This is not viable for all companies which use Rails 2.3 in a commercial fashion — for example, if you’re an insurance company which has a line-of-business Rails application, it is highly unlikely that you have the engineering bandwidth to do this.  My company has the equivalent of 0.2 full-time intermediate programmers working for it, and if I were to keep on top of Rails security, I’d probably have no time for improving my applications  or for writing code to support marketing/sales goals.

4)  Pray that someone supports Rails 2.3 for you

Rails Core obviously aren’t the whole of the Rails community.  It is possible that somebody else will take up the challenge of supporting Rails 2.3.  For example, Github has their fork, as mentioned.  It is possible that you can just sit downstream of that fork and have things work for you.  I know a few people who are implicitly dependent on the Github fork, for example.

Why didn’t I go down that route?

  • Github’s fork currently supports Github’s needs, but it isn’t a formal commitment to the community or anything.  They could drop it at any time, without notice.  (I might be outside the loop, but I don’t even remember it being announced as existing.)  This makes it difficult to rely upon for consequential business decisions, like “Can I afford to commit all weeks in October 2013 to other projects or do I need to keep something in reserve for a panic-upgrade from github/rails to whatever my next best option is then?”
  • As an informal project, there is no release process for Github Rails.  You’ll have to build your own gems for it, on your own schedule, without guidance as to where known-good commits are on their work-in-progress branch.  This could be problematic if you slurped down a security patch and got Github-specific code which, e.g., broke Ruby 1.8 compatibility (check their commit history).
  • Github’s fork of Rails 2.3 is, to my cursory view, not designed to be exactly compatible with 2.3 as used by places other than Github.  It is an extraction from their working system.  I can’t be sure that moving my systems to Github Rails isn’t going to cause them to break in all sorts of fun and subtle ways.  (For example, see this commit, which rips out the default YAML processor.  That is, in isolation, a decision that I’m totally on-board with and am reasonably sure won’t break my app.  One commit down, 53 more to audit.)
  • I don’t think “I had no formal relationship with Github but decided to rely on them because they’re good people” is an answer that lawyers at Health and Human Services or my insurance company would tolerate as a commercially reasonable business practice, in the event that a compromise of my site eventually happens.

5)  Pay for a commercial fork of Rails 2.3

So after looking through the other options, and getting turned down by Rails Core on my attempt to essentially buy support for Rails 2.3 from them, I started looking for other people to take my money.  It turns out that there is a consultancy in Germany called Makandra which maintains an internal fork of Rails, because they have ongoing support obligations for 50ish applications currently in commercial use by their customers.  They also have a team of a dozen Rails experts.  This puts them in a pretty good position vis-a-vis new security issues: they’re already committed to fixing them quickly (since they have 50 apps depending on it), and they have sufficient engineers to be able to do that.

So I made them a proposition: If they cleaned up their fork such that it could be consumed outside their company, and contractually promised to maintain that fork against newly discovered vulnerabilities in Rails, I would be willing to pay them a substantial amount of money for that.  This frees me from having to do it myself, and I have high confidence that “We engaged a firm of engineering experts to handle that aspect” will pass muster with customers, regulators, and my insurance company.  This is similar to the Long Term Support option that you could buy from e.g. RedHat if you didn’t want to be on the latest version of their distribution.  The name of the fork is, appropriately, Rails LTS.  You can buy access to it and use it in production today.

It seemed like a bit of a waste for this relationship to only benefit our companies, given that Rails is OSS and we both continue to benefit from the community, so we hashed out a way to make it work well for everybody.  Makandra has substantial ongoing engineering expenses in doing the scutwork that Rails Core correctly observed the community is unwilling to do, and it cost them even more to get it into a state where it was releasable for consumption outside the company.  Accordingly, “I’ll just pay you to do that, and you OSS the code” wasn’t going to work for them.

So we split the difference: we agreed that I would be Customer Zero for Rails LTS, guaranteeing that it was commercially viable for them to do the engineering work necessary to release it.  They will sell it to anybody, for very, very reasonable rates relative to the amount of money Enterprise software companies usually charge for commercial support agreements.  If you buy it, you get support integrating it into your application and, more importantly, guarantees with regards to timelines for them to address new issues.  This means that you can sleep soundly know that it is someone’s contractual obligation to drop everything and address any Priority: The World Is Burning vulnerabilities which get discovered in August.

Normally, when two enterprises get into a support contract with each other, the community doesn’t explicitly benefit.  I wasn’t thrilled with that notion, so after some negotiation we agreed on a model which preserves Makandra’s interests but also benefits from the community: all code produced under this arrangement is OSSed under the MIT license (the Rails standard) 10 days after being produced.  This means that if you need it to keep a commercial service up and running, that is something you can buy, but if you have a hobby project and are OK with a 10-day window for possible exploitation, you can just slurp down patches from their Github account as they become available.  Nothing the community currently has gets taken away as a result of this arrangement: you can still choose to use the Rails Core version of 2.3 on your own recognizance (and get no security patches), or you can buy Rails LTS (and get new patches written in a guaranteed timeframe for new issues), or you can use the community version of Rails LTS (which did not exist but for me bankrolling it) and get your security patches taken care of for free.

The community patches are, naturally, available for Rails Core to backport if they would like to do so.  I approached them about doing that, and they say it is a possibility if Rails LTS turns out to be a sustainable thing.  That was prior to it being funded and released.

You Might Have An Objection To This.  I’d Like To Preemptively Answer It.

“But Patrick, switching to a fork is hard!”

One of the core design goals of Rails LTS was making sure that you can upgrade to it without requiring that you modify your app.  If your Rails app uses Bundler, it’s as simple as changing a line in your Gemfile and running “bundle update”, then redeploying.  If you don’t use Bundler, you can build the gems locally and then deploy them as per these instructions.  I tried doing that for Bingo Card Creator, which didn’t use Bundler, but it turned out to be faster to just migrate to Bundler.  (If you have a 2.3 app that hasn’t done that yet, instructions for switching are here.  This took approximately 45 minutes for me, mostly as a result of not having a clear all-in-one-place view of my app’s dependencies.)

Since Rails LTS is exactly the current version of Rails plus security patches and tests (diff them if you don’t believe me), it is highly unlikely to cause problems for your application.  They’ve promised that featurewise it is frozen in amber and, unless a security issue necessitates it, the API will not change as a result of security releases.  (This is a policy whose adoption would be widely beneficial in the Rails community.  cough)  Both of my applications worked with flying colors.  There’s an optional “hardened mode” you can activate (one line in your environment file) to disable some features of Rails which are often unused and have recently been demonstrated problematic, like the “Rails can take XML requests and create objects from them then insert them directly into the params hash, including perhaps deserializing embedded YAML in the XML, which can result in arbitrary code execution” (a feature which, to put it mildly, is not required by much of the community).

“Can we trust this to actually, you know, work?”

Great question.  They’re sufficiently credible to me that I cut them a check for $10,000 and have essentially delegated to them a portion of the application security burden for my business, which is a bet-the-business issue for me if I can’t get it right.  They have a strong record of bringing client projects in, including ones which are technically ambitious.  They have a demonstrated commitment to open source and to Rails 2.3’s ongoing viability as a platform.  I think that’s the best testimonial I can give them, but feel free to spend a few hours spelunking in their code if you want to feel more secure about making the decision.  (If you want to look at the Rails LTS code in particular, can I draw your attention to the now-actually-working test suite which newly includes tests for the security patches released this year?)

“Why would we pay money for OSS code?”

You’re not really paying money for OSS code which already exists, you’re buying the guarantee that new code gets written at non-deterministic times in the future before the lack of it causes an emergency for your business.  It’s closer to an insurance policy than it is to source code.

You’d pay money for this for the same reason you pay engineer salaries: because the code provides business value to you.  If you’re a CTO, you should understand that there is definite value in not having to bring your applications down because their servers have been rooted.  You should also understand that drop-everything-hair-on-fire vulnerability resolution is not something your engineers are particularly good at and not something which you want periodically disrupting them from shipping applications to your paying customers.  You can either pay your engineers to continue monitoring and patching Rails for you, or you can buy Rails LTS.  Your call.  Their entry-level plan costs ~$200 a month, which is approximately the cost of one or two engineer-hours.  (Are you a Rails dev?  If you think you’re capable of identifying and fixing Rails framework issues and you don’t charge that much, raise your rates.  Are you a CTO?  If you think your engineers are capable of this and they don’t charge that much, prohibit them from emailing me, so that I don’t introduce them to more rational firms.)

If you don’t have $200 in the budget, and you at a real business, you either need a) a reality check about your business or b) a heart-to-heart discussion about what will happen when your servers join a botnet.  If you’re running a hobby project or are still in the first few weeks of running your business, then you should switch to the community version of Rails LTS, which is a substantial improvement to your security at no monetary cost.  Or you can just wait until your server gets owned, because that reliably makes people discover budget for security.  (It also could very well result in firings, so I wouldn’t rush to this option.)

“The Rails LTS plans are too expensive!  I would pay for this but only if it cost, like, $5 a month!”

Horsepuckey.  The hypothetical person saying this is a textbook pathological customer: they’re both deeply irrational (if the app’s security was worth $5 a month then the right answer is probably to shut it off and save the server cost) and likely to be far, far, far too much headache for professional Rails engineers to have to deal with.  I’m glad their mail is not going to be in the same inbox as mine when I ask questions about new security issues.

“I could totally duplicate all that work for free!  And I’m going to do that!  Nyaa nyaa!”

In the immortal words of OSS passive-aggressivity, patches welcome.  It would be an awesome thing for everybody if you did a lot of free work.  After finishing your free dev work, you also need to be a dab hand at working the politics of a successful open source project, again for free.  They won’t necessarily be actively hostile to your pull requests, but keep in mind that Rails Core does not consider 2.3 a priority and signaled strong distaste for the notion of having to do any release management for 2.3 patches even if the code were given to them on a silver platter.

(But horsepuckey, you’re not going to write free patches for Rails 2.3, or you’d have commits you could point to demonstrating your willingness to do this over the last 6 months.)

“You’re using your evil marketer wiles on me in pursuit of your selfish interests!”

I am, indeed, attempting to get you onto Rails LTS rather than unpatched 2.3.  Why?  Because as a fan of Rails and long-time member of the community, whose business has benefited enormously from it being available, I do not want to see other community members get rooted.

I have no direct financial interest in you deciding to buy a support contract for Rails LTS.  I don’t get a commission and I’m not an investor, I’m just the anchor customer.  My $10,000 is where my mouth is.  Well, no, my $10,000 is presumably sitting in a German bank account, but you get the general idea.

How Do We Switch?

It’s all on their website.  Switching to the community edition (free) is a one-line change in your Gemfile, and you can and should do that today if you’re already on 2.3.18.  (If not, upgrade to 2.3.18 first.  Then, have a frank discussion about security priorities, since you should have been on 2.3.18.)  To get a support contract and your username/password to access their private git repository (which gets the patches immediately on release rather than 10 days later), get in touch with them via the website.  When I did it it required a paper contract and a wire transfer to Germany, so it won’t complete as quickly as git clone will, but you’ll be on the way to getting this resolved before the next round of patches drop.

P.S. Remember, you need to have all of your Rails apps patched continuously, not just “the main one.”  If you miss e.g. a staging server, a simple service which hooks into the main app, an analytics side-project knocked up by an intern, or an old installation of Redmine, then that box will be rooted, and if that box is inside your firewall you can basically assume you will lose ever box attached to it.

How To Use SSL To Secure Your Rails App Against FireSheep And Other Evils

Posted on October 25, 2010 by in Rails, Ruby

The post on Hacker News today about FireSheep, a Firefox addon which lets you trivially compromise the web application cookies/sessions of anyone on the same wireless network, gave me the much-needed impetus to upgrade my business to SSL security.  For the last several years, I haven’t encrypted traffic between the server and the user.  My theory was that my customer’s didn’t store anything security critical in their elementary school bingo card games, but the increasing amount of information available to the admin (me) plus a customer support story from this morning convinced me that compromise would be a Very Bad Thing.

Implementing SSL in Rails was very painful and resulted in my site being down or unusable for a large portion of my customers for much of the day.  (If I were doing it again, I would have paid the extra few bucks to set up a staging environment with its own certificate and verified everything worked on that prior to trying to fight my way through the process.) Luckily, since I am time-shifted from them by over 12 hours, few noticed.  In the interests of saving you and your customers some difficulty, I thought I would write up what I learned.

What You Need Before You Get Started

  1. A SSL certificate from a certificate authority which is trusted by the major browsers.  GoDaddy sells them for $12.99 and they are perfectly adequate.
  2. SslRequirement and AssetHostingWithMinimalSsl, both plugins by DHH.
  3. Rails to be fronted by Nginx.  The explanation for Apache is similar but the magic server configuration is different. (If you use Nginx+Passenger, can skip the Mongrel-specific bit below.)

Nginx configuration

Nginx manages configurations on a per-server basis, and cannot have a single server declaration listen to both HTTP and HTTPS requests. We’re going to get around having to copy/paste our entire configuration (and maintain it in two places) by DRYing it into a single external snippet, then including it twice.

Right now, your Nginx config looks something like this:

server {
    listen       80;

    //You have a lot of stuff here.
}

Cut everything out of the server declaration (yes, everything) and externalize it into a separate file. It should now look like:

server {
    listen       80;
    // This path is relative to the conf directory
    include apps/EverythingJustCut.conf;
}

You can now create a separate declaration for your SSL server without causing much overlap:

  server {
  listen 443;
  ssl on;

  #GoDaddy's instructions will walk you through setting these up
  ssl_certificate /usr/local/nginx/conf/ssl/your_certificate_combined.crt;
  ssl_certificate_key /usr/local/nginx/conf/ssl/your_key_pair.key;

    include apps/EverythingJustCut.conf;
}

I remember setting up the SSL certificate to be more of a nuisance than a difficulty, but if not, you can find very detailed instructions online.

Now, if you are using Mongrel, you need to do one more thing: withing EverythingJustCut.conf, you’ve got to find the place where Nginx is proxying to Mongrel and have Nginx set the X_FORWARDED_PROTO to https for HTTPS requests. This is the one and only signal that Rails, running on Mongrel, is going to get that a particular request is for SSL or not. This is trivial to do if you know you have to do it: just find all your existing proxy_set_header statements and add this after them:

  proxy_set_header X_FORWARDED_PROTO $scheme;  // http or https, evaluated per request by Nginx

Very Carefully Scrub Your Website For Absolute URLs

Absolute URLs containing the scheme (i.e. anything starting with http:// ) are dangerous to your application, because if your page transmitted over HTTPS references a “certain type of resource” on HTTP, your browser may display a Scary Error Message to your users. Unhappily, the browsers get to pick what constitutes a security-critical resource.

The general rules for maintaining SSL security on HTTPS pages are:

  • Javascript files: must always be loaded from HTTPS
  • Image files: must always be loaded from HTTPS, except for Firefox and Safari
  • CSS files: must always be loaded from HTTPS, except for Safari
  • other files: may or may not be loaded from HTTPS

Do you think you’re going to remember that? Yeah, me neither. Hence, you don’t want any hardcoded http:// anywhere. Let Rails take care of it for you with AssetHostingWithMinimalSsl: all you have to do is be consistent about always loading e.g. images through the image_tag or image_path helper, CSS and Javascript through their associated helpers, etc, and you will always get the proper behavior. The tough part is that you’re going to have to take IE and drive through every page on your site verifying that it does not accidentally include a resource transmitted in the clear.

The configuration for AssetHostingWithMinimalSsl goes in your config/environments/production.rb file and is trivial:

config.action_controller.asset_host = AssetHostingWithMinimumSsl.new(
  "http://images%d.example.com",  #In this example, I have images1..images4 .example.com configured in DNS
  "https://www.example.com"  #Your SSL-secured domain
)

Note that it is very easy to miss a http:// URL hidden in a CSS file, Javascript file, or analytics-package JS include somewhere. If you do that, even in an unused CSS selector, you will cause the browser to throw Big Scary Errors. Test that you have gotten everything very thoroughly prior to proceeding.

Require SSL for Any Security Sensitive Actions

Why are we requiring SSL? To prevent against an attack where the bad guy can sniff the cookie out of there air, thereby appropriating it for himself and logging in as the logged-in user (either by compromising a session ID or, in Rails, compromising the user_id you probably stored in the tamper-proof CookieStore). So what do we have to SSL? Everything Rails sends or accepts a session cookie with. What is that? Everything that an actual browser can access. (If you are, like me, in the unenviable situation where some URLs are going to be hit by user agents which cannot support either HTTPS or cookies, don’t forget that requiring SSL for all actions won’t help you.)

The SslRequirement plugin makes it easier to do this sitewide than it otherwise would be:

#in application_controller.rb
 unless RAILS_ENV == "production"
    def self.ssl_required(*splat)
      false
    end

    def self.ssl_allowed(*splat)
      false
    end
  else
    include SslRequirement
  end

This sets it so that SSL is required when we say it is in production only, and in other environments the statements which set up SSL requirements are merely silently ignored. Those functions are:

  • ssl_required: takes a list of :symbols representing action names to require SSL for. Should be nearly all of your actions. If someone tries to access one of these actions over HTTP, they will be redirected to HTTPS (+).
  • ssl_allowed: takes a list of :symbols representing action names to allow SSL for. If they aren’t required and aren’t allowed, then they’ll be redirected to HTTP if they try getting to this action over HTTPS.

You set them in each controller, on a per controller basis. There does not appear to be a handy mass assignment option like :all for this method, unlike most of the before_filter and similar things you find in Rails.

Note there is a subtlety here: if you let Rails share a session cookie over both HTTP and HTTPS, then if it is ever used over HTTP, byebye cookie security. This means you can either be very, very careful that you never let anyone access Rails actions over HTTP (and you pray a malicious attacker never tricks your users into clicking to a valid link to your website), or you ban your session cookie from being sent over HTTP at all:

#production.rb
config.action_controller.session = {
    :key     => 'name_of_session_goes_here',
    :secret          => 'you need to fill in a fairly long secret here and obviously do not copy paste this one',
    :expire_after    => 14 * 24 * 3600, #I keep folks logged in for two weeks
    :secure => true #The session will now not be sent or received on HTTP requests.
  }

This option will probably break your site, possibly subtly, the first time you switch it on. Test thoroughly. I haven’t got it 100% working for myself yet.

Host downloadable files on SSL? You just broke IE.

After several hours of frustration, failing my way forward, and finally getting things working on Chrome/Firefox, I received a bug report from an IE using customer who couldn’t download her bingo cards. Some deep Googling revealed that IE, for architectural reasons known only to the IE team, does not play well with downloadable files over SSL unless you set some very specific headers:

  response.headers["Pragma"] = " "
  response.headers["Cache-Control"] = " "
  response.headers["Content-Type"] = "application/pdf"  #Put your favorite MIME type here
  response.headers["Content-Disposition"] = "attachment; filename=#{filename}"  #
  response.headers["X-Accel-Redirect"] = "/path/to/file.pdf"
  render :nothing => true

Note I am using X-Accel-Redirect to have the file served directly through Nginx, rather than through Mongrel, as described here.

Conclusion

I hope that saved you and your customers some pain and insecurity. If you haven’t done this yet, and I think there are many Rails apps as open to exploitation as I was until this afternoon, I suggest you download FireSheep and see how quickly any idiot with wireless can compromise your admin session. Then, fix this as soon as possible.

Security Lessons Learned From The Diaspora Launch

Posted on September 22, 2010 by in Rails, Ruby

Last week, Diaspora — the OSS privacy-respecting social network — released a “pre-alpha developer preview” of their source code.  I took a look out it, mostly out of curiosity, and was struck by numerous severe security errors.  I then spent the next day digging through their code locally and trying to get in touch with the team to address them, privately.  In the course of this, I mentioned obliquely that the errors existed on Hacker News, and subsequently was interviewed by The Register and got quoted in a couple of hundred places.

The money quote most outlets went for was:

The bottom line is currently there is nothing that you cannot do to someone’s Diaspora account, absolutely nothing.

I’d like to back up that contention, now that it is safe(r) to do so.

Reporting security bugs is a funny business: any description of the error sufficient to resolve it is probably sufficient to create exploit code.  This is why I was fairly circumspect about the exact mechanism for the errors, and why Steve Klabnik also mostly declined to give specifics when describing the state of the codebase.  Since the specific errors I reported are now patched, I’m going to disclose what they were, so that budding Rails developers who care about security do not inadvertently give attackers the ability to do anything they want.

By the way, if you’re looking for Rails security advice, I recommend the official guide and the OWASP list of web application vulnerabilities, which would have helped catch all of these.  Web application security is a very deep topic, and often involves unforeseen circumstances caused by the interaction of complicated parts which are not totally under the developers’ control.  That said, nobody should be making errors like these.  It hurts us as developers, it hurts our ecosystem, and it endangers our users in spite of the trust they have put in us.

I found somewhere in the ballpark of a half-dozen critical errors — it depends on how you count pervasive mistakes that undermined virtually every class in the system.  There were three main genres.  All code samples are pulled from Diaspora’s source at launch, were reported to the Diaspora team immediately, and have been reported to me as fixed.

Authentication != Authorization: The User Cannot Be Trusted

Code:

#In photos_controller.rb
def destroy
    @album = Album.find_by_id params[:id]  # BUG
    @album.destroy
    flash[:notice] = "Album #{@album.name} deleted."
    respond_with :location => albums_url
end

This basic pattern was repeated several times in Diaspora’s code base: security-sensitive actions on the server used the params hash to identify pieces of data they were to operate on, without checking that the logged in user was actually authorized to view or operate on that data. For example, if you were logged in to a Diaspora seed and knew the ID of any photo on the server, changing the URL of any destroy action from the ID of a photo you own to an ID of any other photo would let you delete that second photo. Rails makes exploits like this child’s play, since URLs to actions are trivially easy to guess and object IDs “leak” all over the place. Do not assume than an object ID is private.

(There is a second error here, by the way: the code doesn’t check to see if the destroy action is called by an HTTP POST or not. This means that an overenthusiastic browser might follow all links from a page, including the GET link to a delete action, and nuke the photo without any user action telling it to do so.)

You might think “Surely Diaspora checks to see if you’re logged in, right?” You’re right: they use Devise, a library which handles authentication, to verify that you can only get to the destroy action if you’re logged in. However, Devise does not handle authorization — checking to see that you are, in fact, permitted to do the action you are trying to do.

Impact:

When Diaspora shipped, an attacker with a free account on any Diaspora node had, essentially, full access to any feature of the software vis-a-vis someone else’s account. That’s pretty bad, but it gets even better when you combine it with other errors.

How to avoid this:

Check authorization prior to sensitive actions. The easiest way to do this (aside from using a library to handle it for you) is to take your notion of a logged in user and only access members through that. For example, in my software, any action past a login screen has access to a @user variable. If an action needs to access one of their print_jobs, it calls @user.print_jobs.find(params[:id]). If they have subverted the params hash, that will find no print_job (because of how associations scope to the user_id) and they’ll instantly generate an ActiveRecord exception, stopping any potential nastiness before it starts.

Mass Assignment Will Ruin Your Day

Code:

#users_controller.rb
def update
  @user = User.find_by_id params[:id]  # <-- uh oh, no auth check
  prep_image_url(params[:user])

  @user.update_profile params[:user]  #  root_url)
end

#user.rb
def update_profile(params)
  if self.person.update_attributes(params)  #  <-- insert input directly to DB
  ...
  end
end

Alright, so we know that if we forget authorization then we can do arbitrary bad things to people. In this case, since the user update method is insecured, we can meddle with their profiles. But is that all we can do?

Unseasoned developers might assume that an update method can only update things on the web form prior to it. For example, this form is fairly benign, so maybe all someone can do with this bug is deface my profile name and email address:

Diaspora Profile Update Page

This is catastrophically wrong.

Rails by default uses something called “mass update”, where update_attributes and similar messages accept a hash as input and sequentially call all accessors for symbols in the hash. Objects will update both database columns (or their MongoDB analogues) and also call parameter_name= for any :parameter_name in the hash that has that method defined.

Let’s take a look at the Person object to see what mischief this lets us do. (Right, instead of updating the profile, update_profile updates the Person: Diaspora’s internal notion of the data associated with one human being, as opposed to the login associated with one email address (the User). Calling something update_profile when it is really update_person is a good way to hide the security implications of code like this from a reviewer. Names matter — make sure they’re accurate.) What methods and fields do you expose…

#Person.rb
class Person
  ...
  key :url,            String
  key :diaspora_handle, String, :unique => true
  key :serialized_key, String   #This is your public/private encryption key pair.  OOPS.

  key :owner_id,  ObjectId   #You don't want me changing this one either...

  one :profile, :class_name => 'Profile'
  many :albums, :class_name => 'Album', :foreign_key => :person_id
  belongs_to :owner, :class_name => 'User'   #... because it lets me own you!  Literally!

end

#User.rb
one :person, :class_name => 'Person', :foreign_key => :owner_id  #Oh dear.

This is painful: by changing a Person’s owner_id, I can reassign the Person from one account (User) to another, allowing me to both deny arbitrary victims from their use of the service and also take over their account, allowing me to impersonate them, access their data at will, etc etc. This works because one in MongoDB picks the first matching entry in the DB it can find, meaning that if two Person have the same owner_id, your account will non-deterministically control one of them. So I’ll assign your Person#owner_id to be my #owner_id, which gives me a fifty-fifty shot at owning your account. If that is annoying for me, I can always assign my Person#owner_id to have some nonsense string, de-linking them and making sure current_user.person finds your data when I’m logged in.

But wait, there is more!: Note the serialized_key column. Can you guess what that is for? Well, if you follow some spaghetti in the User class, that is their serialized public/private encryption key pair. You might have heard that Diaspora seeds use encryption when talking between each other so that the prying eyes of Mark Zuckerberg can’t read your status updates. Well, bad news bears: the attacker can silently overwrite your key pair, replacing it with one he generated. Since he now knows your private key, regardless of how well-implemented your cryptography is, he can read your messages at will.

This is what kills most encryption systems in real life. You don’t have to beat encryption to beat the system, you just have to beat the weakest link in the chain around it. That almost certainly isn’t the encryption algorithm — it is some inadequacy in the larger system added by a developer who barely understands crypto but who trusts that sprinkling it in magically makes it better. Crypto is not soy sauce for security.

Is this a hard attack? No. You can do it with no tool more complicated than Firefox with Firebug installed: add an extra parameter to the form, switch the submit URL, own any account you like. It took me two minutes to find this vulnerability (I looked at the users controller first, figuring it was a likely place for bad stuff to happen if there was bad stuff to be found), and started trying to get the word to the Diaspora team immediately. It literally took longer to get Diaspora running than it took to create a script weaponizing this.

Steps to avoid: First, fix the authentication. That won’t prevent this attack, though — I can still screw up my Person by changing it’s owner_id to be yours (and do this an arbitrary number of times), virtually guaranteeing that I can successfully disassociate your account from your person.

After you fix authentication, you need to start locking down write access to sensitive data. Start by disabling mass assignment, which should be off in an public-facing Rails app. The Rails team keeps it in because it saves lines of code and makes the 15 minute blog demo nicer, but it is an easy security hole virtually anywhere it exists. Consider it guilty until proven innocent.

Second, if your data store allows it, you should explicitly make as much as feasible unwritable. ActiveRecord lets you do this with attr_readonly — I’m not sure whether you can do it with MongoMapper or not. There is almost certainly no legitimate reason for owner_id to be reassignable.

NoSQL Doesn’t Mean No SQL Injection

Code:

def self.search(query)
  Person.all('$where' => "function() { return this.diaspora_handle.match(/^#{query}/i) ||
               this.profile.first_name.match(/^#{query}/i) ||
               this.profile.last_name.match(/^#{query}/i); }")
end

Diaspora uses MongoDB, one of the new sexy NoSQL database options. I use a few myself. They have a few decades less experience getting exploited than the old relational databases you know and love, so let’s start: I claim this above code snippet gives me full read access to the database, including to serialized encryption keys.

What the heck?!

Well, observe that due to the magic of string interpolation I can cause the string including the Javascript to evaluate to virtually anything I want. For example, I could inject a carefully constructed Javascript string to cause the first regular expression to terminate without any results, then execute arbitrary code, then comment out the rest of the Javascript.

We can get one bit of data about any particular person out of this function: whether they are in the result set or not. However, since we can construct the result set at will, we can make that a very significant bit indeed. One thing Javascript can do is take a string and convert it to a number. I’ll elide the code for this because it is boring, but it is fairly straightforward. After I have that Javascript, I can run a binary search to get someone’s serialized_key. “Return Patrick if his serialized key is more than 2^512. OK, he isn’t in the result set? Alright, return Patrick if is key is more than 2^256. He is in the result set? Return him if his key is more than 2^256 + 2^255. …”

If their key has 1,024 bits (wow, so secure), it will take me roughly 1,024 and change accesses to find it. That will take me, hmm, a minute? Two? I can now read your messages at will.

I think MongoDB will let me do all sorts of nastiness here aside from just reading parts of the person object: for example, I strongly suspect that I can execute state-changing Javascript (though I didn’t have any luck making a Lil Bobby Tables to drop the database, but I only spent about two minutes on it) or join the Person document with other documents to read out anything I want from the database, such as User password hashes. That might be a fun project for someone who is not a complete amateur.

Code injection: fun stuff for attackers, not quite so fun.

How to avoid this:

Don’t interpolate strings in queries sent to your database! Use the MongoDB equivalent of prepared statements. If MongoDB doesn’t have prepared statements, don’t use it for your security-critical projects until it does, because you will be exploited.

Take Care With Releasing Software To End Users

Since making my public comments, I have heard — over and over again — that none of the above matters because Diaspora is in secret squirrel double-plus alpha unrelease and early adopters know not to put any data in it. False. As a highly anticipated project, Diaspora was guaranteed to (and did) have publicly accessible nodes available within literally hours of the code being available.

People who set up nodes might be intelligent enough to evaluate the security consequences of running them. That is actually false, because there are public nodes available, but we’ll run with it. Even if the node operators understand what they are doing, their users and their users’ friends who are invited to join The New Secure Facebook are not capable of evaluating their security on Diaspora. They trust that, since it is on their browser and endorsed by a friend, it must be safe and secure. (This is essentially the same process by which they joined facebook — the zuckers.)

How would I have handled the Diaspora release? Well, candidly, I wouldn’t have released the code in the current state, and instead would have devoted non-trivial effort to securing it prior to release. If you put a gun to my head and said “Our donations came from 6,000 people who want to see progress, give me something to show them”, I would have released the code that they had with the registration pages elided, forcing people to only add new users via Rake tasks or the console. That preserves 100% of the ability of developers to work on the project, and for news outlets to take screenshots, without allowing technically unsophisticated people to successfully sign up to the Diaspora seed sites.

I don’t know if the Diaspora community understands how bad their current security posture is right now. Looking at the public list of public Diaspora seeds, while the team has put a bold disclaimer that the software is insecure (which no one will read because no one reads on the Internet — welcome to software, guys), many of the nodes are explicitly appealing as safer options which won’t reset their DB, so you won’t lose your work if you start on them today. That is irresponsible.

Is Diaspora Secure After The Patches?

No. The team is manifestly out of their depth with regards to web application security, and it is almost certainly impossible for them to gather the required expertise and still hit their timetable for public release in a month. You might believe in the powers of OSS to gather experts (or at least folks who have shipped a Rails app, like myself) to Diaspora’s banner and ferret out all the issues. You might also believe in magic code-fixing fairies. Personally, I’d be praying for the fairies because if Diaspora is dependent on the OSS community their users are screwed. There are, almost certainly, exploits as severe as the above ones left in the app, and there almost certainly will be zero-day attacks by hackers who would like to make the headline news. “Facebook Competitor Diaspora Launches; All Users Data Compromised Immediately” makes for a smashing headline in the New York Times, wouldn’t you say?

Include here the disclaimer that I like OSS, think the Diaspora team is really cool, and don’t mean to crush their spirits when I say that their code is unprofessional and not ready to be exposed to dedicated attackers any time soon.

Rails Fails To Update Serialized Columns On Save

Posted on June 28, 2008 by in Rails

One of the problems I hit earlier today, which literally cost 2.5 hours to resolve, was that I have a Rails model which uses a serialized column in it.  The column contains a hash of options for the object.  Fairly simple stuff.

As it turns out, after overwriting one of the parameters in the hash and saving it, the object in the database would be out of sync with the ActiveRecord object still in memory (even though save returned true), which lead to hilarity the next time someone grabbed the object from the DB and got partially out-of-date data.  (Inconsistent out-of-date data.  Failure.)

As it turns out, the issue is a new feature Rails 2.1 — partial updates of database records.  Rails keeps a list of which attributes are “dirty” and only updates those when you call ActiveRecord#save.  Unfortunately, touching the contents of a hash does not mark it as dirty.

I was absolutely clueless how this was happening (I stupidly upgraded to 2.1 on a whim, thinking that I hadn’t written any significant code yet so incompatibilities wouldn’t bother me — true enough, the only incompatibility was with my mental model of how ActiveRecord worked), but I successfully diagnosed the why — a dirty bit wasn’t getting set for the serialized object.  OK, no problem, the same thing happens at the day job with our Java system — which means I can use the same hacky solution to the problem.

def before_save

  options_hash_clone = options.clone  #shallow copy of options hash

  options = {} # sets dirty bit for options hash

  options = options_hash_clone # restores options hash to original content, ensuring save updates it in DB

end

which will, indeed, set the dirty bit. 

As it turns out, there is a cleaner way to do this if partial updates aren’t a requirement for your model:

#there are a number of places you could put this — the model class itself strikes me as decent

ModelNameGoesHere.partial_updates = false

A big thanks to this post for putting me on the scent of the problem.  Seems I missed quite a bit of discussion on Google about it since I was not hitting the right keywords apparently.  The fact that this is on by default appears to be one of those opinionated software practices where “opinionated” means “it is our opinion that you should be as familiar with the change log as the core team before you install a new production release”.  (Sorry if I sound bitter.  2.5 hours.)

Brilliant Bit of Javascript for Redirecting Downloaders

Posted on March 12, 2008 by in microISV, Rails, web design

One of my uISV buddies, Ethan (king of language learning software), took me to task earlier today for spending so much time optimizing my download page when I could just eliminate it entirely and link the download direct to the download button in most cases.  I had always had these issues with that solution:

  • My users don’t necessarily know what to do with a window that pops up
  • If I do an HTTP Refresh or Javascript redirect, many browsers pop a security warning
  • I have to discriminate between Mac and PC users somehow
  • It is impossible to track that conversion for AdWords purposes, currently

Examining Ethan’s code made it really easy to avoid the first two issues:

function SetUpRedirect()
{
var destination = “http://www.bingocardcreator.com/free-trial.htm“;
setTimeout(“window.location='”+destination+”‘”,3000);
return true;
}

If you stick that in the OnClick attribute of a link pointing at your favorite executable, three seconds after clicking the link and having the download initiate, the user’s browser goes to the download page in the background.  This causes no security warning, scores them as a download conversion with the appropriate code on the page, and presents graceful fallback behavior if they don’t know what to do with the window that just popped up, since you can give them instructions.

Ahh, but what to do about the difference between Windows and Mac computers, which need different installers?  First, we make a controller method to handle it in Rails:

def free_trial_download
    if request.user_agent.downcase =~ /mac/
      send_file “public/files/BingoCardCreator.zip”, :type => “application/zip”
    else
      send_file “public/files/BingoCardCreatorInstaller.exe”, :type => “application/exe”
    end
  end

 That essentially says “If I’m positive you’re using a Mac, initiate a download of the zip file.  Otherwise, initiate a download of the exe file.”  (Obviously since 92% of my downloads are PC users I want to err on the side of caution.) 

Then, with a simple route added to routes.rb:

map.downloadFreeTrial ‘free-trial/download’, :controller => ‘static’, :action => ‘free_trial_download’

we get a simple URL which is platform agnostic and which decides, on the server side, which version of the file to give them.  You can then decorate your links to the platform-agnostic URL with the code to redirect the page to the download page in the background, with Analytics click tracking, and what have you.  Easy peasy!  One less step in the conversion funnel, and instantaneous recovery of a large portion of the 20% of folks who bounce out of the funnel at the download page.

WARNING: send_file will cause your Rails process to block while that IO transfer takes place under certain older versions of Rails (not in 2.0 in my testing).  This will cause requests coming to the same Mongrel after the download to wait until the download completes to start, which if you have a 56k modem user could potentially cause your basic site access to be delayed for minutes.  Not good news!

My site has two Mongrels running, very few dynamic requests, and very small executables.  If your site doesn’t have this profile, instead of using send_file, 302 redirect the browser to the appropriate file and let your web server handle the request before Rails does.

WARNING NUMBER TWO: You don’t want bots hitting that action, so its time for a good-old robots.txt exclusion of it.  Note that deploying this sitewide will cause your free trial page to lose quite a bit of the juice you’re sending to it.  However, given that that page is typically linked far and wide on the Internet and doesn’t include much interesting content on it (which would distract from the conversion to the trial!), you can probably live with that tradeoff.

Quick request: if you run an obscure browser or a Mac, kindly use my OS-agnostic link and tell me if it works for you.  (You should get a prompt to download BingoCardCreator.zip )

Because You Can't Quite Get Enough Transparency…

Posted on November 26, 2007 by in Rails, stats

I really wanted to post how Daily Bingo Cards was doing statswise today, but probably will not have the time.  (The short version: the snowflake queries are loving me and owning a top 10 spot on every possible variation of “thanksgiving bingo cards” is worth 1.5X owning the 11th spot on [thanksgiving bingo cards] itself.  Don’t ask me how you can rank for a phrase that competitive in less than 2 months of work.)  While I know the analysis is the really interesting bit, for the stats geeks in the audience I decided to make my website stats public in real time.  Enter a Rails plugin named Sitealizer, about five minutes of work, and powie, stats for anyone.

Want to take a gander?  Daily Bingo Cards stats.  At the moment it should be showing search queries, referrers, and the like for about the last 24 hours.  You’ll note that it is hardly as tricked out as Google Analytics (one nice feature Analytics lacks: it tells you what crawlers are hitting your site and at what rates), but it is good enough to keep me more or less honest when discussing traffic numbers.

Rails SEO Tips 90% Completed

Posted on November 19, 2007 by in Rails, SEO

Too many projects, too little time.  I got most of my Rails SEO hints page completed tonight, after finally implementing more of the suggestions I was making in Daily Bingo Cards itself.

The Table of Contents

The page is still a bit of a work in progress, of course.  I intend to keep it updated and continue gradually expanding the content.  Plus it is 2 AM and I really have no effort to do make the code samples more pretty (what can you expect — I built them by hand in notepad — lots and lots of ampersands, let me tell you). 

If you have any comments about the article, feel free to leave them here.  If you know any Rails developers who might be interested in the resource, please feel free to pass it on to them.

SEO Tips for Ruby on Rails

Posted on October 29, 2007 by in Rails, SEO

I’m working on this article as another bit of linkbait, and its about 33% of the way finished at the moment, but I thought I would give you guys a sneak peek.  If you have any comments, please, feel free.  If you want to blog or otherwise link to it, go right ahead, although it is very much a work-in-progress at this point.

The excerpt:

There is much to love about the Ruby on Rails framework. Don’t Repeat Yourself. It Just Works. Massive productivity gains, happiness returning to the world of boring CRUD apps, and a certain sense of panache in programming. However, while Rails has sensible defaults it doesn’t get everything right out of the box. This article focuses on how you can improve the search engine optimization (SEO) of your Rails site the Ruby way and get a

  • more usable,
  • more popular,
  • and more profitable application — with less work!

You can read the rest of it at Rails SEO tips, located at Daily Bingo Cards.  Why did I put it over there?  Frankly, I expect this to make the rounds a few times in the Rails community, many of whom have their own blogs, and I expect it to get linked to heavily.  There isn’t a Definitive Rails SEO Resource yet, and that page has delusions of grandeur. 

My blog is PR5, has a few hundred inbound links, and has little direct impact on my monthly bottom line.  Daily Bingo Cards is PR0, has about two inbound links, and has the potential to double my take-home pay.  Choosing to get the links over there rather than over here was not a hard decision.  Granted, the inbound links will not be that targeted to start out, but they’ll greatly help get the trust-ball rolling while I wait a few weeks to start ranking for my targetted snowflake queries.

P.S. When I post this to the social networking sites, for the ones which value a little bit of controversy with their morning coffee, the title is going to be “Default Routes Considered Harmful, and Other Rails SEO Tips”.  If you’re in the less-geeky end of the pool the reference might not make sense to you, but trust me, Considered Harmful is a (heated!) conversation starter around the Slashdot set.  I’m not saying it just to be controversial, though — leaving the default routes in a publicly-accessible Rails application is a bad idea, for the reasons I go over in the article.

Marketing A Superhero Novel (Web Admin Hackery Galore)

Posted on August 23, 2007 by in Rails

Around the same time I started Bingo Card Creator, my little brother got bitten by the Do A Cool Project bug and decided to become a published fiction author.  He is writing a superhero novel and, aside from this post being a totally transparent attempt to give him some SEO juice to get him kicking, I’d like to recount what we’re doing to market a product that, strictly speaking, doesn’t exist yet.  This is mostly going to be a technical overview — if folks have interest, I’ll go over some of the softer side of marketing at a later date. 

 Keep in mind that I am a programmer, not a system administrator.  Don’t do these instructions on a production box without testing it first, OK?  They worked for me, but if they don’t for you my liability is limited to refunding the amount you paid for this blog post.

Priority #1 — Setup a website.  I’m sure you’re shocked.

Specifically, because my brother is more skilled in dialogue writing than in web server administration, I set him up with WordPress to make the barriers to content creation nothing.  He had already had a blog he used for writing on WordPress.com, so moving him to hosted WordPress wasn’t much work from his perspective (we could even import all the old posts).  From my perspective, it was a bit of work getting SuperheroNation.com to coexist peacefully with Kalzumeus’ Rails test site on a single 256 MB Slicehost VPS, but after a night it was up and running.  (You can do this to host Rails and WordPress on the same domain, but the instructions are different and I’ll give them to you at a later date). 

Why go with hosted WordPress rather than WordPress.com?  It gives the site room to grow, and gives my brother control — he can incorporate Google Analytics throughout via a plugin (ahem, DO IT), and when he is ready to start selling the book throwing up e-junkie’s Fat Free Cart, a nice custom skin, and other features is very easy.  If you’re just starting your own site, for SEO purposes if nothing else, put it on a domain you control rather than wordpress.com.  It is a pain in the butt to set up, but moving a blog OFF wordpress is infinitely harder than getting WordPress up and running somewhere else and then making the changes you want down the road.

Here’s what you need to do, assuming you’re serving Rails with Apache proxying to a Mongrel cluster (works like a beauty even at 256MB, incidentally, although I haven’t tested it under severe load):

 1)  First, you’ll want to get WordPress installed.  If you’re hosting two domains on the same box, I’d recommend distinct directory structures for them — the non-Rails domain here resides in /var/www/superheronation/, and the Rails stuff is in a completely different path set automatically by deprec (see below).  After creating the directory, go over to wordpress.org, download the zip file, unzip it, and stick it in a subdirectory named wordpress (so, in my example, /var/www/superheronation/wordpress) .  We’re putting it in wordpress rather than in something descriptively named because if we change blogging platforms later we can do most of the job just by changing one line in a config file.  (By the way, wondering what to call your blog’s main user-visible directory?  I like “blog”, but a high value keyword works better for SEO… at least for now.)

1b) Skip this step if not a Slicehost customer.  There is a problem with installing WordPress.org on Slicehost — by default, Slicehost setups don’t come with a mail server, and for reasons only God knows WordPress.org dies with a silent error if PHP can’t mail you your password during an install.  Don’t ask me what the security rationale for that one is.  Happily, WordPress.org is open source and you can quickly hack together a solution — go to the wp-admin/upgrade-functions.php file, comment out (put a # sign in front of ) this line:

wp_new_blog_notification($blog_title, $guessurl, $user_id, $random_password);

and this line:

$random_password = substr(md5(uniqid(microtime())), 0, 6);

replacing it instead with

$random_password = ‘for_love_of_little_apples_change_me';

Now you can follow wordpress.org’s 5 minute install directions without blowing stuff up.  Or, you could, if your web server was actually serving up your blog yet.  Its not, since you haven’t told it that your second domain exists.

2)  Locate your Apache config file for your Rails installation.  If you did the setup for your Rails site using deprec (*highly* recommended — it will save your sanity and many days of tweaking config files, and it works beautifully with Slicehost), this will be in /usr/local/apache2/conf/apps/nameOfYourApp.conf .  Copy it to another .conf file of your choice.

3)  Working on your second copy, edit the VirtualHost declaration to read

<VirtualHost www.nameofyourdomain.com:80>

That is Apache speak for “If the web browser asks for anything under this domain, use the following options”.  You’re going to point the DocumentRoot to /var/www/superheronation/ or wherever you put this domain, and set the ServerName and ServerAlias from whatever your Rails domain is to whatever your WordPress domain is.  Now replace the catchall VirtualHost (VirtualHost *:80) with the same sort of name-based virtual host, in the nameOfYourApp.conf file.

4)  Now for the magic — we want Apache’s awesomely powerful rewrite engine to send the appropriate things to WordPress, while leaving the rest to Rails.  Here’s exactly what you need after RewriteEngine On for the seperate domain case:

# Let apache handle the PHP files – all requests that get past this rule
# are routed to the mongrel cluster (aka Rails)
#  – wordpress installation assumeed to be in ‘public/wordpress’
#  – Options: NC – case insensitive
#  –          QSA – query string append
#  –          L – last rule, aka stop here if rewriterule condition is matched

# Prevent access to .svn directories
  RewriteRule ^(.*/)?\.svn/ – [F,L]
  ErrorDocument 403 “Access Forbidden”

  # Check for maintenance file and redirect all request
  RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
  RewriteCond %{SCRIPT_FILENAME} !maintenance.html
  RewriteRule ^.*$ /system/maintenance.html [L]

  # OMIT THIS LINE if you have don’t want to automatically redirect everything from the domain to the blog.
  RewriteRule ^/$ %{DOCUMENT_ROOT}/wordpress/ [NC,QSA,L]

RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteRule . %{DOCUMENT_ROOT}/wordpress/index.php [L]

 

  # Rewrite to check for Rails cached page
  #RewriteRule ^([^.]+)$ $1.html [QSA]

  # Redirect all non-static requests to cluster
  #RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  #RewriteRule ^/(.*)$ balancer://deprec_will_fill_this_in_cluster%{REQUEST_URI} [P,QSA,L]

5)  Cleanup tasks:

Make sure Apache knows that .php files are first class citizens by editing /usr/local/apache2/conf/httpd.conf and replacing the line with DirectoryIndex to read

DirectoryIndex index.php index.html

And, at the way bottom of that file, comment out the Include statement and replace the NameVirtualHost directives with

NameVirtualHost www.rails_domain.com:80
Include conf/apps/rails_application_name.conf
NameVirtualHost www.wordpress_domain_name.com:80
Include conf/apps/wordpress_config_file.conf

That should be it.  You can now restart Apache (“sudo /etc/init.d/httpd restart”), and you should be able to access your WordPress blog and complete installation of it.  I’d HIGHLY recommend changing your Admin password, creating another Admin user with a non-Admin name, and changing your Preferences -> Permalinks to a non-default option which includes your post title, for SEO purposes (the 3rd option works nicely). 

6)  Don’t forget to update whoever holds your DNS records to point your domains to your slice’s IP address.  For me, this involves telling GoDaddy to use ns1.slicehost.net, ns2, and ns3 as DNS servers for superheronation.com, and then going into the Slicehost config and telling Slicehost to point superheronation.com and www.superheronation.com to the IP address of the slice I bought.  (Handily listed on the bottom of that screen.)

And there you have it!  Two websites running off of two very different technology stacks on a $20 a month VPS.  They’ll both perform marvelously under load, too…  I hope.

Spending Money, Incurring Headaches

Posted on June 7, 2007 by in Kalzumeus, Rails

Kalzumeus is getting to the point where I want to start periodic deployments to the Internets to test it (yes, yes, that DOES mean I’ll be able to announce it publicly, sometime this month if development doesn’t hit snags), so I went shopping for a hosting service.  I eventually settled on TextDrive, which appears to have a decent reputation for Ruby on Rails hosting.  $124 later I was the proud owner of a new “startup” hosting package for a year.  I’ll be spending another $120 or so later to get a certificate for the domain but for now I don’t have any data that needs https so its a waste.

Sidenote: Its a wonderful age we live in where you can buy 95% of the tools you need to run a business for less than $10 a month.  The founder of Textdrive, a guy named Jason who clearly has some chops in the provisioning enterprise class server deploments department, reckons you should budget 10% of sales for infrastructure.  So, for example, if you plan on getting $10,000 a month in sales your monthly server/colo fees/electricity/etc bill should be about $1,000.  This may well be true for larger installations, but I’m pretty sure you can get by on about $30~$50 a month for $10,000 in sales as a uISV with some care given to app selection.  I’ll break down the math for you some other day, after I have an app and benchmark numbers I can point you to, to demonstrate that I’m not shooting smoke out of my hindquarters.

Anyhow, yesterday was a comedy of errors.  These are 100% my doing — I bit off a lot more than I could chew with technologies I do not have a full conceptual understanding of yet.  Example: I spent three hours trying to manhandle my local SVN repository onto the server.  The textdrive docs got me set up with a new repository in 3 minutes, but actually importing the information to it was a constant battle with Netbeans.  I eventually lost my most recent work to corrupted settings (“What do you MEAN I only have 3 class files?!”) after twiddling stuff with text editors, and had to cleansweep my local copy of everything, then restore from the local repository.  Then I got out Cygwin (bless you, Cygwin), killed all the .svn directories, deleted and remade the remote repository, and imported the newly-checked-out-locally repository into the remote repository.  One more open of Netbeans later and I was done.  Blaaaaah.  On the plus side, both Kalzumeus and Bingo Card Creator now exist safely outside of my hard disk.

Incidentally: its far, far simpler to set up hosting with GoDaddy than it is with TextDrive.  The difference in the level of control you get for TextDrive is night and day though.  I’m eventually going to have both Apache and Lighttpd running on the server, Apache proxying everything and making sure the /blog/ folder goes to WordPress, with Lighttpd putting things to the Rails application.  Try that on Godaddy… if you dare.

Then, despite the fact that it was running rather late, I wanted to get some actual coding done rather than losing a night just to wrangling with tools.  Again, 100% due to my own negligence, I lost a lot of time debugging why a transaction wouldn’t work.  Well, OK, 98% to my own negligence.  You see, Rails takes an awful lot of syntatic shortcuts which are deeply meaningful but documented in, I kid you not, exactly one throw away line in The Book.  And, unfortunately, the emphasis on “create code whose intent is crystal clear” gets lost in some of these shortcuts.

Example: build vs new vs create.  One throwaway line will teach you that create actually creates the object in your database while build/new just create it in memory, a sort of Key Distinction when you’re deciding “What do I need to guard with a transaction block?”  And, for more fun, build and new are interchangeable… except when they’re not.

@user = User.build #OK

@user = User.new #OK, same as above

@new_friend = @user.friends.build #OK

@new_friend = @user.friends.new # No new method on friends!  Whoops!

@new_friend = @users.friends.create # Works, but note different database behavior!

Anyhow, I will soon be establishing a blog at kalzumeus.com (you could visit there now, but its a fairly boring “coming soon” page that was autogenerated).  That will be getting most of my future programming/business oriented and professional development posts.  This way I can keep the blog for the actual product site focused on the customers rather than on myself or an interest the customers don’t share (“helping uISVS make money”).  I won’t be abandoning this blog in the near future, though.

Older posts