Archive by Author

Bingo Card Creator Year In Review 2009

My name is Patrick McKenzie and for the last three years and change I’ve run a small software business selling Bingo Card Creator, which creates… OK, so I’m not the world’s most creative namer.  I traditionally publish stats and other observations from the business that I think are interesting. You can see my automatically compiled statistics and reports for 2006, 2007, and 2008 elsewhere on my blog.  2009 technically isn’t over yet but business typically comes to a standstill after this point in the year, so the overall financial picture is likely accurate in broad strokes, with perhaps a few hundred in sales yet to happen and some expenses which may or may not happen in this calendar year depending on the vagaries of when vendors charge my credit card.

Business Stats for This Year:

Sales: 1049 (up from 815 — 29% increase)

Refunds: 24 (unchanged from last year, down from 2.9% to 2.3% of sales)

Sales Net of Refunds: $31,156.18 (up from $21,141.60 — 47% increase)

Expenses: $12,630.47 (up from $12,318.54 )

Profits: $18,525.71 (up from $8,823.06 — 110% increase)

Approximate wage per hour worked: $125 ~ $150 (I have never been good with timesheets.  Sorry.)

Web Stats For This Year

(All stats are for bingocardcreator.com unless otherwise noted)

Visits: 546,000

Unique Visitors: 470,000

Page Views: 1.6 million

Traffic sources of note: Google (48%), AdWords (20%), “My Sprawling Bingo Empire” (see below) 4%.

Trial downloads: 56,000 (flat from last year)

Trial signups to online version: 17,000 (new this year)

Guest signups to online version:  8,500 (new this year)

Approximate download-to-purchase conversion rate: 1.17%

Approximate online trial-to-purchase conversion rate: 2.33%

Narrative Version:

The defining event for my business this year was releasing the web version of my application this summer, which was extraordinarily successful for me.  In addition to the (clearly visible above) massive increase in conversion rates it afforded me, it has substantially decreased my support burdens, development costs per feature added, and headaches due to version incompatibilities.  (Well, with the noted exception of that time I cost myself $3,500 due to a CSS bug.)

I’m currently making more than enough money to live off of on Bingo Card Creator (I live rather simply in a rural prefecture in Japan — think Kansas with rather less white people) and, depending on the month, have made more than my day job a few times.  (My highest month in sales was October at $4.5k… and that is even with the $3.5k lost to that bug.)

Things That Went Right

  • The launch of the web application.
  • Raising prices, from $24.95 to $29.95.
  • My development of A/Bingo, a Rails A/B testing framework.  In addition to collecting numerous mentions from luminaries in the community, actually using A/Bingo has been key to my ongoing conversion optimization efforts.  (Last year, for example, I had approximately 1.4% conversion rates for the downloadable version of the software.)  A/Bingo is also deployed in production at over a dozen other businesses, helping to make other people lots of money, which makes me very happy.
  • Iterating on the Christmas Bingo Cards experiment from last year to develop a stable of mini-sites, currently mostly centered on holiday bingo from Valentines through Halloween.  While a full ROI breakdown is outside the scope of this article, systematizing the process, automating the deployments, and using freelancers to do a lot of the repetitive work has resulted in greater than a 10x ROI.  This isn’t quite as impressive as the returns on the freelancer-produced content -> SEO engine that made my year in 2008 and continues to produce dividends, but it has been both fun and profitable.  I hope to improve more on it next year.
  • Mailing list marketing through Mailchimp.  In the course of signing up for the online trial of my site, many teachers choose to accept a semi-monthly newsletter from me.  Since the typical user behavior on my site is to use it for an immediate need and forget about it, this gives me further bites at the conversion apple.  (It costs me about 24 cents to get a trial download or trial signup through AdWords at the margin, but only 2 cents to remind someone that their account still exists and that it would be perfect for the Halloween Thanksgiving Christmas festivities around the corner.)  I feel like I’ve only scratched the surface of email marketing, and it will pay for all my Christmas presents this year and then some.
  • Meat and potatoes SEO, marketing, customer support, and all that jazz.

What Didn’t Go Quite Right

  • Did I mention I lost $3.5k to a CSS bug?
  • AdWords has made me very unhappy at several points this year, from when they turned off my account to their inability to approve my new ad copy within a month of submission to the strange partial limitation for “gambling content” that my account is sporadically flagged for.  (Attempts to resolve this through AdWords customer support have been…  you know what, in the Spirit of Christmas (TM), I think I’m just not going to go there.)
  • I again failed at my goal to launch a 2nd product, largely due to lack of time and mental bandwidth.  (Although I suppose the online version practically counts as a second product.)
  • I spent a whole lot of time implementing online features in the desktop version of my app — probably something on the order of half the time I spent making the online version.  This was largely done on the theory “Hey, it adds value to the desktop offering for current customers, who wouldn’t want to switch to the web app.”  I’m too cool to actually ask users about their feelings beforehand, of course.  Fans of the Lean Startup can already predict how this story ends up, right?  To date, twenty users have touched those three features.  Let’s see: add marginal feature to application, make twenty users happy, or for just a few hours more develop and market a new application which doubles my revenue streams.  Decisions decisions!
  • I am not satisfied with the level of attention I gave the business at some points during the year.  In particular, communication with my freelancers was subpar, and on three occasions I took more than 24 hours to respond to a customer.
  • I was sort of hoping to hit the nice round $20k number for profit.  Didn’t quite get there.

Plans For 2010

  1. In 2007, I had vague dreams of someday going full time on this.  In 2008, I had a half-formed wish that maybe I would go full-time on this in 2008.  In 2009… I have a date circled in red on my calendar, forms from the tax office and immigration authority ready for submission as soon as the new year starts, and I’ve had the first conversation with my boss to inform him that I am considering my options.
  2. I’d like to make $45,000 in sales from BCC.  This is probably a little on the low side, considering that if I don’t have a repeat of the CSS bug it would only require sustaining my current performance rather than markedly improving.  But, hey, I don’t want to get too caught up in hitting this to the exclusion of my other business and personal goals for the year.
  3. I’d like to have about $30,000 in profit from BCC.  This is a bit on the aggressive side if you assume $45,000 in sales — it means I’ll have to get most of the growth out of doing the old things better rather than just spending up through AdWords.  (Which would be wonderful if it were possible, don’t get me wrong.)
  4. Personal goals: I’d like to attend (and, ideally, present something fun) at an industry event overseas and at one in Japan.  (I got a bit of an invitation to talk software design in Osaka this January.  Jeepers.)  I’d also like to spend a lot more time with my family, which will be easier after I no longer have to ask for permission four months in advance to fly out to see them.  (I’m still not entirely decided on where I’ll end up with my business, but the great thing about it is the whole thing fits in a laptop bag, so aside from some irksome tax and visa issues I have a lot of flexibility in where I live.)

Bringing A/B Testing To The Fortune 5 Million

After writing an A/B testing library and blogging about the subject for a couple of years, I somehow unwittingly sleepwalked into being that most loathsome of creatures, a technology evangelist.  This means that periodically I get emails from folks with the Next Big Thing who want my opinion on it.  I rather enjoy this, as I’m passionate about the subject, can usually find ideas to steal become inspired by, and always enjoy a good tech demo.

One of the folks who asked my opinion was Paras Chopra over at Wingify.  Wingify is an analytics startup which, from my brief impression of using it, was trying to be an awful lot of things to an awful lot of people.  That is sort of the nature of the beast with enterprise software.  I did the requisite “install the tracking code” thing, commented to Paras that I would have felt a little lost if I didn’t breathe analytics systems, and more or less forgot about it.

But Paras and his team kept iterating and they produced a real gem — probably the best single piece of software I’ve seen in this field.  As Paras explains:

One of the biggest [lessons] I have had is that there is a huge gap in what we (split testing community) profess and what small businesses actually adopt. I have [learned] that many website owners are curious about split and multivariate testing but don’t have a clue where to start and what to use. Though [the] Google Website Optimizer guys are doing a great job to the community by evangelising split testing, … the difficulty in using the tool and fiddling with code leaves most people wondering how to really [use] testing.

I think this is an extraordinarily good job of learning from your customers, like that “customer development” that seems to be going the rounds these days.  Wingify initially set out to be an enterprise-class analytics system, but when trying to sell it Paras et al found out that the customers don’t need an enterprise-class analytics system.  They need the Microsoft Word of Internet marketing, a simple pick-up-and-go tool that you can install, employ, and benefit from without needing cooperation from the operations, marketing, and engineering teams which you either a) don’t have or b) can’t get to do your bidding.

This is sort of like the process I went through with making A/Bingo, except from another direction.  I couldn’t use the market-leader A/B testing tool (Google Website Optimizer) because it was built for non-technical marketers and made too many compromises to be useful for me.  Paras’ customers can’t use GWO because it isn’t nearly non-technical enough — it still requires inserting multiple chunks of Javascript code, knowing HTML to make alternatives, being comfortable with regular expressions and URLs, etc.  These aren’t core concerns if your market is Rails developers, but if your market is e.g. real estate agents with 5 page brochureware sites who want to split test the call to action to join their mailing list without having to engage a freelancer to do it, they’re huge stumbling blocks.

Genius UI

Visual Website Optimizer, Wingify’s new product, has such a UI for creating A/B tests so simple it will crush the life out of all other solutions for non-technical users:

  1. VWO opens your site in a browser.
  2. You click the element on the page you want to A/B test.
  3. You click “Add variation.”
  4. You add a variation by typing into a WYSIWYG editor.  (TinyMCE, if I don’t miss my guess.  Score one for OSS.)
  5. Copy/paste the Javascript we give you into your page.  You don’t have to identify sections, massage your HTML, or create alternate URLs.  We do that %(#$ for you.

For example, here is me clicking on the headline for BCC:

And I think I’ll rewrite it to say “PC or Mac” instead of computer.

Dead easy.  The remainder of the process involves a bit of Javascript cut/pasting and some URL specifying.  (The interface for this could still be improved a little bit, to be sure.  I don’t think it is quite at the level where it needs to be for non-technical folks to intuitively grasp it.  But, hey, ship and iterate, right?)

Why I’m Particularly Impressed With This

Aside from the work going on in the background to make this process so pain-free (that is live, unaltered HTML they’re working with, and I didn’t do anything when coding it to make it particularly easy for them to rewrite the DOM model on the fly with their injected Javascript), this software impresses me as a business.  It solves a clear need for a huge number of small businesses, and brings a powerful technique to people who would never have been able to use it before.  Moreover, it does it so disruptively, embarrassingly better than Google does that it puts a smile on my face.  I like Google, don’t get me wrong, they’ve made me a lot of money.  But all the kings horses and all the kings men apparently can’t deliver a UI as good as a small team.

A quick note: Paras et al are from India.  After a few years of doing outsourcing management I’m quite happy to see a young team producing something very worthwhile rather than doing the traditional thing and being a wee little cog in a giant corporation working on grinding out back-office software.  I have to say one thing, though, and it is straight out of the Economic History of Japan playbook: people don’t take copiers seriously, and it is a hard impression to shake once you’ve gotten it attached to you, fairly or otherwise.  The design “inspired by” Basecamp is… ahem…  well, suffice it to say that it does not demonstrate nearly as much originality as the software does.  I’d hate for folks to write this startup off just for that, but first impressions matter.

Rather than taking (deserved) lumps for flying the Jolly Roger, I’d suggest folks to either use an open source web design, one of the attractive reasonably priced templates the Internet is overflowing with, or hire somebody with design skills to bang out something decent for v1.0.  StyleShout (OSS) and ThemeForest (paid, but sinfully cheap) both have very attractive Web 2.0-y designs.  After you have revenue you can always improve it to your heart’s content, particularly if you’ve kept the design mostly separate from your program logic.  (A taller order than it needs to be in PHP, I know…)

Want To Try It Out?

Paras provided me with an invite code (“patrick-bcc” without quotes), which will work for the first 30 folks who use it.  Visual Website Optimizer is free when it is in beta, and will be a paid tool after that.

I’d expect pricing to be “reasonable”, although my advice (to Paras and anyone else) is to charge more than he thinks it is worth.  Trust me, it would be cheap at ten times the price if it worked for dentist offices, real estate agents, car mechanics, and the other constituents of the Fortune 5 Million.  One conversion at the margin is potentially worth hundreds or thousands of dollars, and even a single lead would pay for a month of most software-as-a-service products.

It Seems I Have A Podcast Now

I was just checking referrals today and found some from HearABlog, a new startup which narrates blogs for consumption on the go.  Apparently they used my blog as an example to demo their service, making podcasts of the following three posts:

  • Practical Metaprogramming: Storing User Preferences (download)
  • Tracking Down a Subtle Analytics Bug (download)
  • The IE CSS Bug Which Cost Me A Month’s Salary (download)

The quality is quite good — I’m especially impressed at their oral descriptions of the screenshots and graphs that were crucial to understanding some of those posts.  The translator in me approves immensely.  My only constructive criticism: Cal-zoo-ME-us, not Cal-zoo-MAY-us.  Because even made up words have a canonical pronunciation, even if only in my mind. :)

I’m not sure if strictly speaking creating derivative works from this blog is copyright kosher without asking me first, but in the interests of supporting people doing cool things, HearABlog has my explicit permission to produce and distribute audio recordings of all my blog posts on this site, and that will continue until I say otherwise.  (I have no particular plans of ever doing it but, hey, you never know.)

A request to people doing cool things in the future: I have an email address a marked aversion to saying “No”, so please, feel free to tell me about it in advance.

Edited to add: I just went back and checked my email, and Pablo from HearABlog did try to get in touch with me on October 29th.  Unfortunately, he managed to email me when I was on a plane over the Pacific, and I didn’t see the email later in the crush of payment notifications.  (A high class problem, to be sure.  Halloween is my busy season.)

Practical Metaprogramming with Ruby: Storing Preferences

// <![CDATA[
if (typeof window.Delicious == "undefined") window.Delicious = {}; Delicious.BLOGBADGE_GRAPH_SHOW = false; Delicious.BLOGBADGE_TAGS_SHOW = false;// ]]>

All code in this article is copyright Patrick McKenzie 2009 and released under the MIT license. Basically, you can feel free to use it for whatever, but don’t sue me.

The other day on Hacker News, commenting on a recent Yehuda Katz explanation of the nuts and bolts of metaprogramming, I mentioned that I though discussions of programming theory are improved by practical examples of how the techniques solve problems for customers.  After all, toy problems are great, but foos and bars don’t get me home from the day job quicker or convince my customers to pay me money.

My claim: Metaprogramming allows you to cut down on boilerplate code, making your programs shorter, easier to write, easier to read, and easier to test. Also, it reduces the impact of changes.

I’m going to demonstrate this using actual code from my PrintJob class, which encapsulates a single request to print out a set of bingo cards in Bingo Card Creator.  PrintJobs can have any number of properties associated with them, and every time I implement a new feature the list tends to grow.  For example, when I added in the ability to print in color, that required adding five properties. This pattern is widely applicable among many times of one-to-many relationships where you never really look at the many outside of the context of their relationship to the one — user preferences would be an obvious example.

There are a few ways you can do this in Rails. The most obvious is put each property as a separate column in your table. This means that

  1. you’d do a database migration (downtime! breakage! unnecessary work!) every week you add a new property.

If you’re getting into the associations swing of thing, you might consider creating a has_many relationship between PrintJob and PrintJobProperties, with each PrintJobProperty having a property_name and a property_value.  Swell.  Now you need to:

  1. Do twenty joins every time you inspect a single PrintJob.
  2. Add a bunch of unique constraints (in your DB or via Rails validations — I hope you have earned the favor of the concurrent modification gods) to prevent someone from assigning two properties of the same name to the same print job.
  3. Have very intensely ugly syntax for accessing the actual properties.  (Lets see, print_job.options.find_or_create_by_name(“foo”).value = “bar” ought to do it.)

Instead of either of these methods, I save the properties to an options hash, and then serialize that to JSON to save to and load from the database column.  Rails takes care of most of the details for me.

Enter The Metaprogramming Magic

However, this means I would have to write code like print_job[:options][:background_color], which is excessively verbose, and every time I referred to it I would need to possibly provide a default value in the event it was nil.  Too much work!

Instead, we’ll use this Ruby code:

#goes in /lib/current_method.rb

#Returns the method name ruby is currently executing.
#I use this just to make my code more readable to me.
module CurrentMethodName
  def this_method
    caller[0][/`([^']*)'/, 1]
  end
end

#goes in /app/models/print_job.rb
class PrintJob  5, :columns => 5, :column_headers => "BINGO", :free_space => "Free Space!", :cards_per_page => 1, :card_count => 1, :page_size => "LETTER", :title => nil, :title_size => 36, :font => "Times-Roman", :font_size => 24, :good_randomize => true, :watermark => false, :footer_text => "Omitted for length", :call_list => true, :background_color => COLOR_WHITE, :second_background_color => COLOR_GREY, :border_color => COLOR_BLACK,
:text_color => COLOR_BLACK, :color_pattern => "plain"
}

#set up accessors for options
  DEFAULT_OPTIONS.keys.each do |key|
    define_method key do
      unless options.nil?
        options[this_method.to_sym]
      else
        nil
      end
    end

    define_method "#{key}=".to_sym do |value|
      unless options.nil?
        options[this_method.to_s.sub("=","").to_sym] = value
      else
        options = {}
        options[this_method.to_s.sub("=","").to_sym] = value
      end
    end
  end

#Other stuff omitted. Sorry, I'm not OSSing my whole business today.
end

What This Code Does

OK, what does this do? Well, first I define a bunch of default options, which are later used (code not shown) to initialize PrintJobs right before they’re fed into the actual printing code. Each default option is used to create a getter/setter pair for PrintJob, so that instead of typing print_job[:options][:background_color] I can just type print_job.background_color. You’ll notice that it also note that both setters and getters pre-initialize the options array if I haven’t done it already. This saves me from accidentally forgetting to initialize it and then winding up calling nil[:some_option].

Why This Code Is Useful

Clearly this saves keystrokes for using getters/setters, but how does it actually save work? Well, because each of the properties are now methods on the ActiveRecord object (the PrintJob), all of the Rails magic which you think works on columns actually works on these options, too. This includes:

  • validations
  • form helpers
  • various pretty printing things

Since card_count is just another property on the PrintJob ActiveRecord object, Rails can validate it trivially. Try doing that within a hash — it isn’t fun. I sanity check that card_count (the number of cards printed for this print job) is an integer between 1 and 1,000, and additionally check that, for users who aren’t registered, it is between 1 and 15. (I’ve omitted the message which tells folks trying to print more to upgrade.)

  validates_numericality_of :card_count, :greater_than => 0, :less_than => 1000
  validates_numericality_of :card_count, :greater_than => 0, :less_than => 16,
:unless => Proc.new {|print_job| print_job.user && print_job.user.is_registered?}

Here’s an example of a portion of the form helper from the screen where most of these options are set:

#Just a part of the form.

   4, :title => "Total number of cards to print."%>

Ordinarily in the above code you’d expect card_count to correspond to a column in the database, and then the column would cause there to be card_count and card_count= methods on PrintJob, and this would be used to initialize the above text field. Well, Rails doesn’t really care how those methods came to be — they could be placed there by ActiveRecord magic, or attr_accessor, or defining by hand, or creative use of metaprogramming, as above. It takes about 7 lines to define a getter/setter pair in most languages. I have twenty properties listed up there. Instant savings: 140 lines of code.

Similarly, I’m saved from having to write a bunch of repetitive non-sense in the controller, too.

def some_controller_method
  @print_job = PrintJob.new
  @print_job.sensible_defaults! #initializes defaults for options not already set
  #update all parameters relating to the print job
  params[:print_job].each do |key, value|
  if @print_job.options.include? key.to_sym
    @print_job[:options][key.to_sym] = value
    end
  end
end

This walks over the param map for things being set to the PrintJob and, if they’re an option, sets them automatically. This saves about twenty lines of manual assignment. (Nota bene: PrintJob.new(param) will not work because the virtual columns are not real columns. In general, I hate mass assignment in Rails anyhow — sooner or later it will bite my hindquarters for security if I use it, so I assign only those columns which I know to be safe. Note that nothing in the options hash is sensitive — after all, they’re just user options.)

This controller is extraordinarily robust against change. When I added five extra options to the print jobs to accommodate my new features (font, color, and pattern selection), I didn’t change one single line of the associated controllers.

But wait, there’s more! You see, 200 lines of negacode (code that you don’t have to write) means 200 lines of code that you don’t have to read, test, maintain, or debug. I didn’t have to change the controller at all. I didn’t have to check to see if the new properties were automatically initialized to their starting values, since the code which performed the initialization was already known to work. I didn’t have to debug typos made in the accessors. It all just worked.

This is the power of metaprogramming. The less boilerplate code you have to write, understand, read, test, debug, and maintain, the more time you can spend creating new features for your customers, doing marketing, or doing other things of value for your business. The last three features I added caused five new properties to be added to my PrintJob model. I just diffed the pre- and post-commit code in SVN. Three features required required:

  • No change to the schema.
  • No change to the controller.
  • 35 lines to implement three features in the model. (Counting the white space.)

(The view required about 25 lines of new code, mostly inline Javascript, because a good UI for picking colors is tricky. I ended up liberally borrowing from this chap, who has an awesome color picker in Prototype that was amenable to quick adaptation to the needs of technically unsophisticated users to who wouldn’t think #F00F00 is a color.)

This is a much, much more effective use of my time than writing several hundred lines worth of boilerplate model code, then repeating much of it in XML configurations, just so that I can actually have access to the data needed to begin implementing the features. (How did you guess I program in Java?)

A Note To DBAs In The Audience

Yeah, I hear you — stuffing preferences in JSON is all well and good, but doesn’t this ruin many of the benefits of using SQL? For instance, isn’t it true that I can no longer query for PrintJobs which are colored red in any convenient manner? That is absolutely true. However, it isn’t required by my use cases, at all.

PrintJobs are only ever accessed per user and per id, and since both of those have their own columns, having all the various options be stored in an opaque blob doesn’t really hurt me. I regret not being able to do “select sum(card_count) from print_jobs;” some times, but since I don’t have to calculate “total number of cards printed since I opened” all that frequently, it is perfectly adequate to just load the entire freaking table into memory and then calculate it in Ruby. (It takes about 5 seconds to count: 217,264 cards printed to date. Thanks users!)

Note Regarding Security

Programmatically instantiating methods is extraordinarily dangerous if you don’t know what you’re doing. Note that I create the methods based on keys used in a constant hash specified in my code. You could theoretically do this from a hash created anywhere — for example, Ruby will happily let you create methods at runtime so you might decide, eh, PrintJob needs a method for everything in the params hash. DO NOT DO THIS. That would let anyone with access to your params (which is, well, anyone — just append ?foo=bar to the end of a URL and see what happens) create arbitrarily named methods on your model objects. That is just asking to be abused — setting is_admin? to 1 or adding can_launch_nuclear_weapons to role, for example.

#Fixing a WordPress bug.  Don't mind me.

Tracking Down A Subtle Analytics Bug

I have a confession to make: I often trust code that I thinks works.  For example, after I’ve got analytics code up and running, and verify to my satisfaction that it in fact increments the count of registrations by one when I sign up, I generally assume “OK, that is the last time I have to worry about that.”

This is, perhaps, a weakness.  For the last several months, for example, I’ve periodically checked in to Mixpanel or my A/Bingo stats and thought “Hmm, that is a lot less conversions than I expected over that time interval”, but I knew my conversion code worked and it wasn’t displaying zeroes, so I mentally pushed it out of mind.

Today, I was finally motivated into checking what was up when I reviewed the statistics for my recent Halloween promotions.  Portions of these did exceptionally well: for example, in the week before Halloween, I had over 2,000 trial signups.  In the course of writing a blog post about this, I tried to figure out how many of those trial signups were caused by the Halloween promotion, and to do this I opened my statistics at Mixpanel.  (Mixpanel is an analytics service with a wonderful API.  I highly recommend them. They’re blameless for the bug I’m about to talk about.)

Mixpanel, however, reported that over the same week, I only had 375 people open the sign up form and some 180 of them actually sign up.  While it is quite common for multiple analytics sources to disagree about exact numbers, a discrepancy that large meant that there was probably a bug somewhere.

Having previously had some issues with my Mixpanel integration, the first place I looked was my log files, to see if I was actually calling the Mixpanel API.  Some light grepping suggested that I was, in about the frequencies I expected to.  Then I thought “Hmm, I wonder if the unique IDs I am passing with each visitor are, in fact, unique?”

A quick check on the command line suggested, no, I was in fact duplicating IDs.  By the scads.  This resulted in me pulling out the code which assigned Mixpanel IDs:

  def fetch_mixpanel_id
    user_id = #code which sets current user ID, or nil for "no logged in user"

    mixpanel_id = session[:mixpanel_id]
    if (user_id)
      mixpanel_id = Rails.cache.read("mixpanel_id_for_#{user_id}") || mixpanel_id
      mixpanel_id ||= user_id
      Rails.cache.write("mixpanel_id_for_#{user_id}", mixpanel_id)
    else
      mixpanel_id ||= rand(10 ** 10).to_i
    end
    session[:mixpanel_id] = mixpanel_id
    #omission for clarity
  end

I was flummoxed, because the code appeared correct, if a little convoluted on first glance. The idea is to persist the Mixpanel ID for a given user — first, by stuffing it in their session cookie, and second, by stuffing it in Memcached so that if they bounce between multiple different machines I can still track them after they log in. If they don’t have an ID set anywhere, it gets randomized — that is the rand (10 ** 10) call.

Given that my user login code was correct (or there would be much, much more serious problems than analytics failing — people would be seeing my admin screens, bingo cards would bleed across accounts, cats and dogs would be friends, etc) and I trust Memcached not to have critical data-corruption bugs, the only place that made sense for introducing the error was the random statement. Which, of course, makes no sense at all — 10 digit random numbers should, by definition, not routinely collide. (There is the birthday paradox to worry about, I know, but the odds of that happening with only 15,000 accounts on the system were fairly slim and the odds of it causing the current issue were too low to measure.)

So I dropped into my Rails console and used it to inspect the memcached IDs, to see if I could find any patterns.

  mixpanel_ids = (1..User.count).inject({}) {|hash, user_id| hash[user_id] = Rails.cache.read("mixpanel_id_for_#{user_id}"); hash}
  counts = mixpanel_ids.to_a.map {|tuple| tuple[1]}.inject({}) {|hash, id| hash[id] ||= 0; hash[id] +=1; hash}
  counts.to_a.sort {|a,b| a[1]  b[1]}.reverse[0..9]

For those of you not fluent in Ruby, these are just some quick, basic operations on data sets. Ruby excels at them and the ability to do them in real time on the console is fantastic. (I don’t suggest doing them with truly massive data sets but when you count things in thousands, oh well, Slicehost gave me 8 cores for a reason.)

Anyhow, this exploration showed that the most common IDs were repeated literally hundreds of times each. However, it wasn’t a uniform distribution — instead, it looked like exponential decay, from the most repeated ID having 800 copies to a long tail of truly unique unique IDs.

As soon as I saw that pattern, I knew what had to be causing it: an srand bug. srand sets the random seed for your process. The random seed is used to generate what random number the next call to rand gives you. Two pieces of code which execute rand with the same random seed will always get the same random number. Since this is not desirable 99% of the time, if you execute rand before setting a seed through srand, srand is initialized to a string composed of the current time, process ID, and a sequence number. This virtually guarantees that you’ll get a unique random seed, and thus you get mostly unique streams of random numbers.

This narrowed down the possible causes of my bug quite precipitously: either there was a bug in Ruby’s srand (unlikely) or I was setting srand determinalistically somewhere. So I did a quick search through my code base and, yep, there is was:

if (options[:good_randomize])
  srand
else
  srand(12345678)
end

This lovely bit of poorly thought out code is in the code which controls printing bingo cards. Each card has the words scrambled, but trial users (whose options[:good_randomize] gets set to false) don’t get truly scrambled cards. They get a sequence of cards which is always the same, capped at 15 cards, so that no many how many times they try to print they can never create more than 15 unique cards. This encourages them to purchase the software, which removes that limitation.

Since that code snippet is executed for every print job, every user either sees a properly random random seed or the fixed random seed. However, after the code finishes, the random seed lives on in the Mongrel process. This was the ultimate cause of my bug.

Imagine a sequence of events like so:

1) Trial user Bob prints out a single bingo card, which sets the random seed to the deterministic value and uses 25 calls to rand.
2) Jane comes to the site for the first time and has a random number assigned to her. It is guaranteed to be the 26th element (call it R26) in the sequence from a fresh call from the deterministic random seed.
3) Bob prints out another bingo card, which sets the random seed to the deterministic value and uses 25 calls to rand.
4) Frederick comes to the site for the first time and has a random number assigned to her. It is guaranteed to be the 26th element (call it R26) in the sequence from a fresh call from the deterministic random seed.

Thus, Jane and Frederick end up getting the same “random” ID assigned to them. Thus, when I report that both Jane and Frederick signed up for the free trial, Mixpanel decides “Patrick’s conversion code is reporting the same event twice, OK, no biggie: I’m going to discard that second funnel event.”

In addition, since both Jane and Frederick shared the same ID for their A/Bingo identity, both of them would always see the same A/B test alternatives. That wouldn’t be too bad, except they also were counted as the same person for conversion purposes, so if Jane converted Frederick’s conversion wouldn’t be counted.

In actual practice, many users still end up getting random IDs, since any particular Mongrel’s random seed got unborked when a paying user printed out a bingo card… at least until the next trial user printed out a bingo card on that Mongrel. (Random seeds are maintained at the process level in Ruby, and each Mongrel is a separate process.)

Anyhow, long story short:

1) No users except me were adversely affected by this bug — it only affected analytics and A/B test results.
2) Other folks using my public Mixpanel APIs and A/Bingo A/B testing code were unaffected — it was only the interaction of this code with the srand(12345678) code that caused the issue.
3) Two months of my analytics and A/B test data is corrupted.

I’m not sure it matters to the bottom line of the A/B test results, which are shockingly robust against programming errors like this: as long as the source of error doesn’t correlate with your A/B choices, the errors will be evenly distributed across both alternatives, so it just washes out. For example, it is likely upwards of 60% of data for the A/B tests was thrown out as “duplicate” erroneously, but since the “duplicates” were evenly distributed across A and B this borks the measured totals and percentages but not the measured significances. That’s some comfort to me, cold as it is.

The moral of the story: don’t use srand(constant) anywhere in your production code, even if it is the easiest way to get what you want, because sooner or later you will use rand() in some unrelated code called by the same process and now you have extraordinarily subtle bugs caused by reuse of random numbers. If you absolutely must use srand(constant), call srand() on a per-request basis to clear out anything which may be lurking around from previous requests. (For example, in a application-wide before filter.)

The IE CSS Bug That Cost Me A Month’s Salary

//
I run a small business selling software (downloadable and online) which lets parents and teachers make bingo cards.  It is October, which is the busiest season on the educational bingo calendar, largely because Halloween is coming up.  Kids in school + candy-fueled frenzy + secular(ish) holiday + desire for fun activity = bingo bonanza!

However, due to an IE CSS bug, my Halloween experience is best described as “bobbing for poisoned apples”.

The Design Improvement That Wasn’t

On September 21st my designer and I got together and we created a new candidate design for my sign up and login pages, which are both critical to conversions for my business.   The intent was to A/B test the new pages against the old design.  One critically important niggle that didn’t bother me at the time: while ideally an A/B test would test exactly the old HTML against exactly the new HTML, for obscure implementation reasons we ended up testing the old HTML plus one little change versus the new HTML.

That one little change was replacing the old sign up button (the stock HTML form submit one) with a graphical button.  I thought this was such an obviously beneficial change that it was not worth taking on extra coding complexity to keep it the old way.  Stupid, stupid, stupid.

Thanks to the magic of subversion I was able to recover exactly the HTML that was displayed on my site after the upgrade.  (This is one of the two A/B alternatives but it is exactly identical to the other one in the area that is relevant.)  Feel free to open that in your browser and see if you can spot the problem.

At this point I have to mention that I generally use IE8 or Chrome, and my designer uses Firefox.  This means I had never seen the new version of my site in IE6 or IE7 until trying to do tech support from an Internet cafe last night.  (Being a proper Japanese salaryman, I had missed the last train and stopped at the cafe to answer customer emails prior to staying at a hotel.)

Whereupon I discovered a slight problem.  In IE7, the site looked more like:

New Version Of Site in IE7

Hmm, the signup form has no visible sign up button.  The log in form also had no visible log in button.

I started panicking but was in no position to fix the bug at 2 AM in the morning from an Internet cafe, although I made a game attempt to do so.  (Word to the wise: trying to fix a Rails application by editing files in vi through a web console to your Slicehost VPS is not recommended.)

How Well Do Forms Without Buttons Convert?

Answer: not well at all.

You can still submit a form by hitting the Enter button even if you can’t see the submit button.  This is, however, not standard user behavior in my niche.  Let’s see what that did to conversion rates.  Using custom segments (a power-user feature from Google Analytics that I’ve literally never had any useful purpose for before), I’ve graphed the conversion rates among IE8 users (orange line), IE6/7 users (green line), and the site average (blue line).

As you can see, IE6/7 and IE8 are pretty much neck and neck in the month before the bug is introduced.  This makes sense, as we would not intuitively assume those two user populations would be very different from each other.  You can also see that they both track the site average reasonably well, which is practically true by definition as IE users make up the majority of visitors on my site (about 65%, give or take).

Then, after the bug gets introduced, the conversion rates for IE6/7 take a nosedive relative to IE8.  The site average in blue also declines, solely because it is being pulled lower by the underperformance of the old IEs.

In concrete terms, the conversion rate of IE6/IE7 users was 3.40% after the bug.  (Note that this is not the conversion rate among people actually viewing the login/registration screens.  Google Analytics doesn’t let me conveniently break the numbers down for that.)  The conversion of IE8 users, by comparison, was 9.53%.  In other words, if you trust my intuition that IE6/7 users are about as likely to convert as IE8 users all else being equal (which you should, just by eyeballing that graph), then I lost about 64% of my conversions since September 21st.

Putting The Damage In Economic Terms

I’m a glutton for punishment so I’m going to do the painful math.  Between September 21st and October 23rd, when I fixed the bug, I had roughly $4,500 worth of sales, of which IE6/7 users comprised about $2,080.  Since the IE6/7 bug is essentially meaningless after you get through registration (since the overwhelming majority of users pick Remember Me and never see either the sign in or sign up forms again), I’m going to make the simplifying assumption that the bug ripples straight to my bottom line.

We can then use simple math to figure out how much the 64% of sales I threw away was probably worth.  Ouch, I am wincing as I type this: $3,750 in lost sales.  That unfortunately flows almost straight to the bottom line, since all of my major costs (advertising, hosting, etc) happen whether I make the sale or not — I probably lost in excess of $3,400 in profits.  That is more than my monthly salary.

How The Heck Do You Overlook That?

I wish I could blame Microsoft for my stupidity on this one, but aside from not testing the page in IE6/IE7 (which would have shown the bug immediately), I had ample opportunities to discover that there was a problem somewhere.  However, other things in the chaotic, fractally complex system that is a small business concealed them from me.

Indication #1: There are two types of signups on my site, trial signups (where I capture an email address) and anonymous guest signups (where I don’t).  Historically, I push the trial signups much harder, and my users signed up for them at a multiple of the guest signups.  (Guest accounts scarcely convert for me at all, which is why I don’t push them much.  However, an A/B test proved that removing the option didn’t increase the number of trials or paying customers, so I left the option in.)

Thus, when the mix of trial and guest users suddenly went from 880 : 350 to 792: 506, I should have said “Wait a minute, the guest accounts are getting much more popular.  Why is that?”  The answer to that is: Unsophisticated users, like the ones who make up most of my audience, will fill in their information and then see ‘Sign in as guest’ and ‘Cancel’.  Given only these two choices, clearly they want to ‘Sign in as guest’ rather than figure, “hmm, typically guest accounts don’t have usernames or passwords associated with them, I should email Patrick and ask what happened to the real sign up button.”

And indeed, I did see the explosion of guests on my stats, but I had changed the graphing option from showing daily counts to showing weekly counts, and the disparity wasn’t big enough to catch my sustained attention.  (The inflection point on September 21st is immediately visible on the daily graph.)  After considering the matter for a few minutes I said “Eh, Halloween is coming up, which always broadens my market.  Maybe it is bringing in some less interested folks.  Oh well, nothing to worry about.”

Indication #2:

I check my conversion rates on AdWords roughly once a week, largely because if they start to suffer Google stops running my ads and I lose a lot of money.  I then noticed that the numbers were below where I expected them to be (“Hmm, that’s funny, a few weeks ago I was getting 24 – 27%, now it is saying 18%… what is up?”) but I never ran the following graph to see the full range.

Yeah, just from eyeballing that, you can see the problem right?  Visualizations are wonderful for seeing problems that you already know to look for.  However, they are poor for informing you of problems that aren’t visible in the default visualization, since you won’t spend the time to slice the data to see the problem if you don’t know it exists.

Honestly, I don’t know why I didn’t immediately punch the panic button when I saw my AdWords conversion rates start declining.  Again, October always throws my numbers into a total mess, and I was very busy in my day job and in my various seasonal initiatives for taking advantage of Halloween.  So busy, in fact, that I offset most of the decline in sales, and so wasn’t given any advance warning by, e.g., having sales plunge to a fraction of normal.

Indication #3: What finally clued me into looking for this problem was a pattern in customer support requests, of all things.  As I mentioned earlier, guest accounts are historically rarer than trial accounts and they convert very, very poorly.  As a matter of fact, I think I probably had one sale to a guest account in history prior to the bug being introduced.

The code that I have which upgrades accounts in response to people paying me via Paypal and Google Checkout doesn’t handle guest accounts very well, because they’re anonymous.  It upgrades the account fine, but doesn’t register the email address they used for their payment with the guest account, which means the account still can’t log in after logging out.  (Since the guest account is still anonymous.)  However, since the system knows that the Registration Key issued to the purchaser is in use (on their guest account), if for some reason they log out or switch computers, they’ll be both unable to register a new account (because their Registration Key is in use) and unable to log into their guest account (because it is anonymous).  That is a pretty nasty Catch 22.

I’ve known about that Catch 22 for a while, but fixing it was not high on my list of priorities.  All of the following has to happen for it to actually bite a customer:

  • They start using a guest account.
  • They don’t give an email address when prompted by the fairly frequent upsells within the guest account.
  • They nonetheless purchase the software within their guest session.
  • They choose to use the online version of the software rather than the downloadable version (a not-insignificant portion of customers think the online version is the trial and the downloadable version is the “real deal” — well, if they pay me money, they can think whatever they please).
  • They clear their cookies.
  • They try to log in again.

Prior to the bug being introduced, I thought “This sequence of events is about as likely as flipping a coin and having it land on edge.”  And, indeed, it only ever happened to one person.  I fixed her record manually, looked at the code which caused the problem, and figured that an immediate fix was impossible.

<tangent>Why not just set their email address to the one the payment processor just gave me?  Well, Google Checkout has an option which it labels “Opt out of marketing email from this merchant” which, if the customer checks it, gives me an email address like “John.234982385@checkout.google.com” for them.  Obviously, my customers don’t know those random addresses, so using them for a login name is out of the question.  I was also worried that many people’s Paypal addresses are not the ones they use on a daily basis, so using them for logins was suboptimal, so I deferred finding a solution to the bug until later.  After all, it scarcely affected anyone, the cures were temporarily worse than the disease, and manually correcting their records was suboptimal but worked.</tangent>

Then I got emails from three people in the same day who were hit by the guest-purchase bug.  That was finally the eight sigma event that convinced me that my process was really, really out of control.  Whereupon I started investigating, beginning (fortuitously) with trying to log into my admin page on the site while on the Internet cafe’s IE7, and noticing that there was no login button.

Technical Mumbo Jumbo

If you’re a CSS geek, you might be interested in knowing why the buttons vanished.  Fair enough: the HTML input element had a background image specified and a text-indent:-9000px property applied to it, which puts the standard text for the button way off the screen, leaving only the background image visible.  However, IE6/IE7 treat text-indent as moving the background image as well, so the entire control essentially vanished, leaving empty, unclickable space where it used to be.

The fix was fairly simple.  I am truly indebted to Laurence at My Drupal Blog for posting about it.  He probably saved me an ulcer.  Essentially, I identify which browsers are using IE and then apply some sort of hacky stylings to make the background image display visibly, but cause the Sign Up text to be invisible by blending it with the background.  It is very hacky — sort of like patching a wound to your artery with duct tape — but you can’t be too choosy when you’re leaking over $100 a day and have the CSS skills of a vole rat.

On The Bright Side

I don’t want to give you the wrong impression: despite this very, very serious issue, my business has been doing very well in October.  The usual seasonal fluctuation, my seasonal promotions, and all the various improvements I’ve been making in the last few months will still make it my best month ever, and historically the last week of October sees a strong spurt in sales, so it might even be the best month ever by quite a margin.

I also got a much-needed kick in the pants to remind me to test all changes, even the stupid little one-line changes, in all the browsers my customers routinely use.  One positive lesson learned is that IE8’s compatibility mode reproduced this bug exactly, so that will be an easy way to do this going forward.  Ideally I’d have some sort of automated test suite set up to catch this sort of thing (using Selenium or what have you), but that isn’t in the cards at present.  At the very least, I’m going to upgrade my automated stats tracking put big red warnings on my dashboard when the business starts to fail sanity checks.  An obnoxious red warning about conversion rates would have caught this a month ago, within less than 24 hours of the bug going into production.

I hope you learned something from my experience, and am quite willing to take additional lumps in the comments if you have any suggestions for good ways of preventing a similar bug in the future. Happy Halloween!

How To Do A Seasonal Promotion for Your App

//

Many software/web-app developers naively assume that demand for their product is roughly static year-round, despite the fact that this is true for very, very few industries. Retail (in-a-box) software lives and dies by its Christmas numbers. B2B software often sees spiky behavior around the accounting periods and business cycles of its target sector. If you take a look at the data, it is likely that your business also has hot and cold periods — so how do you exploit that?

For example, I sell software which makes bingo cards for elementary schoolteachers.  Since a huge number of my customers use bingo as a fun diversionary activity rather than a regular instructional tool (nothing wrong with either approach, incidentally), I get big spikes of interest around holidays.  Due to peculiarities of how American schools and religion relate to each other, the ideal holiday for my purposes is one where a) class is in session for b) a non-religious holiday which c) does not have an obvious history-focused lesson plan.

If you take a quick look at the school calendar for those three criteria, you might find yourself saying Halloween.  And if you do, congratulations: the spike in traffic I get in October every year as a result of  interest in Halloween routinely makes October my best month of the year.

So knowing that, this year I’ve done a bit of optimization for Halloween, which you might be able to adapt to your own businesses:

Mini-Sites

The keyword profile for new customers coming in as a result of seasonal interest is generally very, very different from your everyday customer.  SEOing for these transient customers requires doing a bit of work, and (especially for those of us who do not have the budgets of Amazon) significantly altering the front page of the site to fit a Halloween theme (with the appropriate keywords everywhere) might be a bit too radical.

Enter the mini-site: you can take a handful of pages or, if you’re feeling generous, an entire new domain name, and then use an on-target visual design with optimized keywords to make it very, very obvious that you’re attentive to the desires of your holiday searchers.  As competition for the holiday keywords is likely fairly low relative to the “head of the long tail” keywords in your niche, you can rank a mini-site fairly easily.

For example, I registered halloweenbingocards.net and put up a simple WordPress installation with 5 pages about, well, Halloween bingo.  It looks much, much more Halloween-y than my main site, and empirically has been converting rather well for the first two weeks of October.  (I know from experience last year, when I had no mini-site but did have pages with Halloween-related content on my main site, that the 300 or so visits a day it is getting this week are about to increase by more than an order of magnitude over the next three weeks.)

The total cost for this site (domain registration, content written by freelancers, directory listings for SEO purposes, etc) was less than $500 and probably another 10 hours ($1,000) of my time.  (I wrote the first draft of it by hand, myself, before getting smart and automating/outsourcing things.  You live and you learn.  My next dozen ideas in this general vein are getting done much more efficiently.)

Incidentally: That WordPress design is based on a free template from WP Design, who graciously gave me permission to use it commercially (it comes with a Creative Commons non-commercial license by default, and I only realized that when writing this blog post, and promptly offered to pay for their blessing, because if software developers can’t respect license terms then we are doooooomed).  It is beautiful and saved me a lot of time in designing the site, and I’ll probably be going back to them for design projects in the future.

PPC Campaigns/Landing Pages

Mini-sites require a fair bit of work and creativity to do right, but if you can’t do anything else, you can quickly create PPC campaigns for the seasonal keywords.  Many businesses would use a holiday as a wonderful excuse to do a periodic sale — that is wonderful for you if it works.  However, with just a little work you can respin your existing content into an on-target landing page and then toss up some AdWords at it.

AdWords optimization is generally a little outside of my comfort zone, but here’s what I did:

  1. I ran a report on what URLs were showing my ads on the Content Network, and eyeballed them for the ones which were specific to Halloween.
  2. I banned those URLs from showing ads in my default ad group.
  3. I created a new ad group for Halloween-themed ads, and put the ads into it as Managed Placements, plus added in some Halloween related keywordery.
  4. I created some Halloween-themed creatives, to catch the clicks on the Halloween themed pages that my ads would be showing on.  For example:
  5. I re-used a quick variation on my main AdWords landing page, replacing the usual content with a bit of Halloween content.

This whole process took less than 15 minutes, mostly as a result of futzing around in the AdWords interface, as the landing page was already pre-written.  When I spent some time reworking my default landing page a few months ago, I linked it to the I-can’t-believe-its-not-a-CMS that drives most of the content on my website.  This lets me re-use any of my pre-existing content (bingo card word lists which I pay my freelancer to write for me) as a landing page just by changing the URL for it — for example, if instead of http://www.bingocards.com/lpc/halloween I were to write http://www.bingocards.com/lpc/christmas it would suddenly be a Christmas-oriented landing page.

I strongly recommend that you have some way to quickly generate landing pages like this — if not a custom-built CMS lurking in the background, at least consider having a pre-made template which can be customized and uploaded in a few minutes.  This lets you throw up a quick seasonal campaign anytime you get the inkling to — I don’t know if Dropbox does better on Boxing Day but if you can launch campaigns this quickly there is no reason why they can’t find out in the time it takes to brew coffee.

Work Your Mailing Lists

Many software companies offer interested people the opportunity to opt-in to receiving mails about special promotions & etc.  (I actually didn’t do this for almost three years, because I was worried about being spammy.  Then I dipped my toe in the water and found, to my surprise, that not only can you do email marketing in a non-spammy fashion, some people actually enjoy it so much that they’ll email you to ask why the September newsletter hasn’t arrived yet.  Really!)

For example, when folks sign up for the free trial of Bingo Card Creator, they’re given the opportunity to opt-in to a monthly-ish newsletter.  These folks are solid gold to talk to: all of them have used my product and expressed an interest in hearing about how it can make their classes run smoother.  Halloween is the perfect opportunity to get in touch with them — to hum a few bars, “Hey guys, do you have a Halloween activity planned yet?  Oh, you were too swamped and are putting it off to the last minute?  How about bingo, using that software you already tried out this summer?  It is fun, will fit in your class time, lets you use the candy you were planning on distributing as marketers, etc etc sales pitch.”

For bonus points, you can re-use or re-adapt mini-sites or PPC landing pages for your email audience.  I won’t be doing 100% re-use, since folks who have signed up to the free trial before are at a different point in the sales cycle than folks clicking on those PPC campaigns: they know the software works for their needs, they just haven’t seen $30 worth of value for it yet, so they need to be told how much hassle it is going to save them in late October.  Folks coming in from the PPC campaign, by comparison, know they want to play bingo in late October, but they don’t know that my software can help them do that and that I am not some evil scammer on the Internet.  (Fear and distrust of the Internet and the evil virus-spreading Spammy McPhishertons on it runs rampant in my market, so I spend significant efforts trying to demonstrate that I’m on the up-and-up.  That would make a good topic for a blog post, actually…)

You can also offer incentives to responding to an email.  Discount coupons are traditional, but as web-savvy software engineers we can be much, much more devious.  If you have a game or StackOverflow-type site you can offer a time-limited themed achievement.  I’m planning on offering to give them a Thanksgiving activity for free if they log in before October 31st.  (If that sounds wacky: giving them a Halloween activity for free would destroy any need for them to pay me money right now, but giving them a Thanksgiving activity for free creates a powerful incentive to come to your site and listen to your best possible Halloween sales pitch.  If they weren’t going to log in in October anyhow, giving them a freebie for Thanksgiving costs me nothing, since they were vanishingly unlikely to decide “Hey, I think I’ll log into that dormant account and whip something up for Thanksgiving, fall in love, and decide to purchase it.”)

I want to expand on that last idea a little bit, because it is so powerful for software developers: any IP you create can be replicated essentially indefinitely for no marginal cost, much like your main product.  However, customers don’t see the world that way — they live in a world where scarcity is a reality, and accordingly perceive value in things they don’t have but might want.  Thus, you can trade your customers access to IP in return for going further towards a conversion with you, and scale this offer across all your customers.  The economics of this are staggeringly efficient compared to e.g. PPC advertising, and you’re giving the offer to folks who are already pre-disposed to liking you, so the conversion rates should be much higher than similar techniques aimed at “cold” prospects.

Both making the offer and fulfilling it can be automated, so it costs a static amount of labor no matter how many customers you have and how many take you up on the offer.  If you’re smart and make this process repeatable, it actually takes less time every time you do a similar promotion, since you’ll have the infrastructure already and all you have to do is create or buy the premium.

Trick or Treat

Well, there you have it — three simple techniques you can adapt to almost any holiday or seasonal promotion.  I hope they got the juices flowing a bit.  If you’ve got any fun ideas in the same vein feel free to leave a comment — I love brainstorming with folks.

Look for a post in early November on how these techniques actually ended up working for me.  Early results look rather positive.

Work Less, Get More Done: Analytics For Maximizing Productivity

For three years now I’ve been running a small software business in my spare time.  It has been a very educational experience, especially in showing me that many things we think we know about software, programming, business, and the like are wrong.  This is a bit of a shock, especially for well-worn chestnuts which have intuitive appeal, which we have come to invest with moral significance, and (most importantly, because we all think we’re smart) we’ve believe so self-evidently true as to make investigation a waste of time.

For example: I have come to the conclusion, over the last three years, that working hard is overrated.  This is an idea I have been kicking around for a while, but it was thrown into sharp relief by a blog post entitled The Only Alternative Is To Work Harder, by a gentleman named Paras Chapra over at Wingify.  Paras and I have corresponded over email a few times, so I say as one analytics junkie to another: the notion that working longer hours is correlated to better business results is a pernicious social pathology.

Working Longer Hours Is Not A Competitively Defensible Advantage

For the last five years, I’ve been a Japanese salaryman, and have often worked 70 hour weeks out of a sense of social obligation.  I understand, very well, the social pressures which could lead someone to write “If your nearest competitor or neighbour works X hours, you must work for X+1 hours.”  It is just a terrible strategy.  Your competitor can adopt it as easily as you can, and then you’re playing a game of multiplayer endurance chicken against everyone else in your market.  You can’t win but you can certainly all lose, by ending up with an entire community where soul-crushing hours are normative.  (There are certain tendencies to this among Silicon Valley startups.  Take it from a Japanese salaryman, guys: it is a disease so vicious that in addition to hobbling businesses it damages society itself.  We’re barely beginning to recover from it decades later.)

Why can’t you win?  Well, suppose that longer hours are indeed the key to success and that Paras is willing to work longer hours than 99.9% of the population of India.  I don’t know how many hours that is at the 99.9% level, but call it 82 a week.  He’ll work 82 hours and then find, oh shoot, over a million people are willing to work more.

Working harder is a particularly bad idea for startups because you are likely competing with people with resources which, relative to yours, are infinite.  I compete with several educational publishers who employ tens of thousands.  Paras competes with Google.    Our competitors have more man-hours in a week than we’ll have in the next decade.  Engaging them on those terms is madness.

Why Smart People Keep Falling For This

Working long hours is near the perfect storm of meme spreadability.

It flatters the sensibilities of many religions — Max Weber was putting the Industrial Revolution down to the Protestant Work Ethic a hundred years ago.  It sometimes confuses me that it is so popular with atheist Japanese salaryman and agnostic Silicon Valley founders.  Then again, programmers at most companies work on a schedule designed to maximize the productivity of illiterate 18th century water loom operators, so expecting rationality might be excessively optimistic.

There is also a sense that working less is somehow, you know, cheating or immoral.  Take the reactions to 4 Hour Workweek (which is, incidentally, a tonne of self-promotion and self-help book hucksterism concealing an ounce of solid productivity gold): a lot of people are emotionally invested in the way that they have always done things being the correct, proper, morally acceptable way of doing things.  A lot of my fellow software developers feel the need to work long days because otherwise their customers will see them as lazy, despite the fact that their customers can only perceive the external indicia of the labor rather than the labor itself.  My customers don’t know that I average four hours a week on my business any more than they know I average 70 hours a week on my day job: all they see is the web site.

It is easy to fall into this trap of productivity being defined in terms of observed effort exerted because in the typical face to face organization it is easy to see who is “working hard” and very difficult to measure actual productivity.  The manager (and peer employees) always knows who was putting in heroic efforts at 10:00 PM last night.  However, your organization almost certainly doesn’t track productivity in any rational, systematic fashion.

This is a pity, because you can’t improve what you don’t measure.  Web analytics has taught us that our cherished beliefs about web design don’t matter an iota in the face of actual customer behavior.  Customer development has shown us that most of the time we’re spending enormous amounts of resources (including time) on products which will fail for lack of demand.  What is the use in one-upping the competition and spending 83 hours to their 82 when you’re both producing something that no one even wants?

Poor Metrics For Productivity

Of course, if you pick a poor metric and try to optimize for it, bad things happen.  For example, many large Internet publishers put auto-refresh code into their pages to inflate page views.  That adds no lasting business value (although it might succeed in getting a few more scraps from CPM advertisers prior to them abandoning the campaign in favor of models that actually work).

Similarly, my day job used to measure engineering productivity in, essentially, hours.  If I worked 20% more hours in August than I did in July, I was 20% more productive in August.  I remember one of my “very productive” weeks: after sleeping in a hotel because I was not physically capable of commuting home, I got to work and started pulling tickets, then opened the appropriate file and started coding.  I then continued for 90 minutes chasing rabbits before I realized that a) I had started working on a file totally different than the feature being discussed and b) the problem I thought I was fixing was not written in the ticket, anywhere.  I had literally hallucinated the entire request.  I then proceeded to spend the rest of the day committing crash bugs into the trunk, confusing our contractors with instructions that might as well have come from someone abusing drugs, and generally spinning my wheels.

Under the productivity tracking system we had at the time, that was my most productive day ever.  You don’t want to emulate that example.  For more examples of things not to emulate, see the Mythical Man Month.

Good Metrics For Productivity

I sell software online in my business. It is very, very easy to identify the direct driver of business value: selling software.  One conversion is worth $30.  It is even pretty easy to identify contributors to that: working up the funnel, trial signups aren’t worth as much as sales, but given their observed conversion rate 70 cents is a decent estimate.  “Casting to currency” lets you compare the worth of two different things, easily.  An intervention which generates 10 sales is less worthwhile than one which generates 4,000 trial signups (all else equal).

In practice, this works well for tracking productivity, too.  Just assign an arbitrary value to tasks, based on your best guess of how much value they add for the business.  Then, track how long it takes to complete the tasks, and figure out where you’re adding disproportionate amounts of value and where you are spinning your wheels.  Do more of the former, less of the latter.

When in genuine doubt about the value, guess what it would cost to have somebody else do it for you.

The Pseudo-Wage

When I started my business I thought it would be amazing if I eventually earned $100 per hour working on it.  (This is a princely sum for 20-something programmers in Japan.)  Its funny, people generally get very good with practical experiences of the mathematical properties of averages in school and then totally forget about that experience in business.  If you want a 92 average, you’d better not routinely get 60s on your homework.  If you want to earn $100 an hour, you’d better not busy yourself with $5 an hour tasks.

Examples:

  • How much is mailing a CD worth?  Like many software developers, I offer CDs of my software.  The primary purpose is to get the sale from people who would not buy if it were purely virtual.  Thus, my labor in generating CDs is worth however many marginal sales the CD option nets me, plus the marginal revenue from actually paying more for the CD.  If I had A/B tested the option, I’d have a very rigorous notion of how much CDs are worth, but we’ll go with a guess: 25% of sales of the CDs would not have happened but for the CD option, thus a single CD is worth $5 (what I charge) + $7.50 (a marginal $30 sale every 4 CDs) = $12.50.  When you consider that I’d have to do all the burning, enveloping, addressing, licking, and mailing myself (in very small batch sizes), its reasonable to assume that doing CD fulfillment myself would be worth on the order of $50~60 an hour.
  • How much is A/B testing worth?  I generally try for about 5 new A/B tests a month.  If the successful ones combine to a 5% improvement (not infrequent) and you generously assume that takes 10 hours, then over the course of a year I’ll earn over $1,500 from them, for over $150 an hour.

You’re Measuring Productivity.  Now, Improve It

Clearly, this means that instead of licking more stamps I should be writing more A/B tests, right?  Sure, I can lick one stamp and write one A/B test, but my time is my most important and limited business asset, so I have to be sparing with it at the margin.

“But Patrick, that is all well and good, but you still have to mail those CDs!”, I hear you say.  No, I have to make sure the CDs get mailed, which is a distinction with a difference.  No middle American schoolmarm perceives additional value over having my saliva the envelope her software came in.  I have long-since outsourced the actual production and mailing of the CDs to SwiftCD.  They cost me about $6 each and, when I was still typing orders in manually, increased the amount of orders I could fill in an hour from about 4~5 to about  10 (note, again, small batch sizes mean this was dominated by setup-and-teardown time as I lost productivity to friction turning on my computer, opening email, copy/pasting over details, checking them, then turning off the computer and running to work).  That results in a significant increase in my psuedo-wage: after deducting the extra cost of outsourcing versus insourcing, I still make ~$100 an hour versus ~$50 doing everything myself.

Productivity Technique #1: Outsource.

This is the first of three major tactics for improving productivity: outsource anything that your personal presence does not add value to.  Equivalently, outsource anything where the replacement price is less than your desired pseudo-wage.

Remember that outsourcing imposes an overhead cost, and payment is due in your time rather than dollars.  Often, this overhead swamps the actual monetary cost of the project.  This kills many outsourcing development projects, because communicating (detailed, constantly changing) customer requirements consumes an enormous amount of time on the part of both sides.

In my own business, I very rarely outsource development, for this reason.  I also don’t outsource direct interaction with customers, because I feel that as a small business knowing what my customers want is an important differentiator.  Everything else, though, is on the table.

My most important use of outsourcing is getting content written for my website.  My software creates bingo cards, and offering bingo cards in the exact niche a searcher is looking for helps convince them to sign up for the free trial.  For example, if you’re looking for a Halloween bingo activity to play with class, the fact I have one ready to go increases your likelihood of sourcing your immediate and future bingo card needs with me rather than making them yourself or going to a competitor.

I used to write my bingo cards myself, and I’m fairly good at it, but eventually I figured that while it was a worthwhile activity it didn’t really get all that much more worthwhile as a result of me doing it.  Instead, I put out a call on my blog for freelancers, and eventually worked out a mutually rewarding relationship with a highly-educated American teacher.  She bangs out the cards on her own schedule, and once a month I click “Post” on my backend interface and then mail her a check.

The economics of this arrangement are so staggeringly efficient that people tell me I have to be lying about it.  The pages my freelancer writes for me were visited 65,000 times in September, producing roughly $1,300 worth of sales for me through getting people into my various conversion funnels.    I did less than five minutes of work to maintain the freelancing relationship in September.  Do you want to do the math?

There is no possible way for me to achieve results like that merely by lengthening the number of hours I work in a day.  Additionally, because I’m capable of producing like that (via, e.g., leveraging freelancers intelligently in a business process which uses software to make them efficient and then extracts business value out of their labor), there is no need to spend excessively long hours working for the sake of working.

Productivity Technique #2: Automate Your Processes.

Software developers should really spend more of their time creating tools for themselves, but we’re hardly the only vertical this applies to.  One of the best points of an awesome lecture on lean startups is that a startup’s most important product is the process the startup uses to create products.

This idea is so powerful that I say, without a hint of exaggeration, it has changed history.  A huge portion of Japan’s rise to global prominence was a result not of working harder, morning radio exercise routines, of the superiority of wet rice cultivation for creating productive societies (all of these, and more, were explanations advanced by authors on the syllabi for the courses I took in the process of completing my East Asian Studies degree — which will teach you to trust academia).  It was by having a few decades of head-start on process improvement as a science.

(I’d be remiss if I didn’t point out that folks have also identified process improvement as a cause of Japan’s economic boom.  However, “Toyota considers it a worthwhile use of time to shave 15 seconds off the construction of an automobile” is so counter-intuitive and unsexy that people reach for the much more obvious, morally correct, visible explanation that Toyota workers work so much harder than GM workers.  This is despite inconvenient facts such as the fact that Toyota can reproduce their productivity advantage in American factories using American workers, and has, repeatedly.  Journalists and management consultants never let facts get in the way of a good narrative.)

Process improvement works for small companies, too!  For example, my use of freelancers rather than myself to write bingo cards is one example of a process improvement, but when I started the process was horrific.  They’d create the word lists (as text files), then have to manually run them through Bingo Card Creator, take screenshots, do manually cropping, etc etc.  This is a lot more efficient than having me do the same steps (their time is cheaper than mine, and I can always find another freelancer but finding a 25th hour in a day is sort of tricky), but there was clearly room for improvement.

I actually outsourced creation of the first improvement: it was a script which, given specially named text files as input, would open up Bingo Card Creator, emulate a user on the keyboard clicking on the particular buttons, and then open up PDF viewer, emulate hitting Print Screen, open up MSPaint, etc etc.  It was a system held together entirely by duct tape, but it worked.  It took me an hour every month to supervise the script and correct unhandled exceptions but it worked.  The productivity of my freelancers immediately increase.  This let them get more work done in less time.  I then bumped up their wages and the resulting more-money, less-busywork combination has kept them sufficiently happy with the arrangement that it has continued without a serious hitch for more than a year now.

(Incidentally, I eventually spent some time and replaced the duct tape.  Remember, it was still taking me an hour a month and I still had access to a computer: clearly, there was room for improvement.  I eventually tightened up the automation, decreasing it from 60 minutes to 40 minutes, then refactored some of the automation and got it down to 30 minutes, then leveraged an unrelated technology change I made and got it down to the current five minutes.  Five minutes isn’t zero minutes, either, so I still have work to do.)

Process improvement takes time.  Consider it an investment in your business’ future, and charge yourself for it.  I outsourced the first draft of that automation and got valuable results for $100.  (Why didn’t I write it myself?  Automating interfaces is Not My Bag, Baby and it would have taken me far more than an hour to do.)  As it is, all the development I’ve ever done on every draft of that system, plus all the handling of freelancers, adds up to less than twenty hours.

Productivity Technique #3: Eliminate Unproductive Uses of Time.

The final technique for maximizing your productivity is eliminating uses of time which do not add business value. We’ve all got them, most of them probably unknown to us because we don’t track our uses of time well.  (I use RescueTime.  If you don’t, install it today and spend some time aggressively categorizing what websites are worthwhile for you to be on.  Consider this a very, very effective process improvement.)

For example, as I mentioned earlier, I’m a bit of an analytics junkie.  RescueTime reported to me that I was spending upwards of 1 hour a day on Google Analytics.  I had an honest conversation with myself on whether I was getting $100 worth of insights every single freaking day out of Google Analytics.  The answer was, no, not really — I was essentially engaged in a nervous habit, opening Analytics and drilling down through a million reports to feel productive when I wasn’t mentally ready to actually be productive.  In particular, I had a few favorite screens I’d check too often — trial downloads per day, for example.

I eventually killed that habit by surfacing that number on my admin dashboard for my website (where I have to go to resolve routine customer service inquiries).  That way, the number flashes by the corner of my eye once every morning as I do productive work.  It is also set so that it gets cached for 24 hours, so that the monkey-brain WoW playing scrolling-numbers-are-dopamine-cast-into-integers side of me can’t get fascinated into sitting on the page and hitting refresh all day.

That sure isn’t going cold turkey, though.  A better example: I used to spend much, much more time on blogging than I do currently.  This is my 447th post to this blog, and I consider the blog one of the main contributors to my business doing as well as it has.  However, over the last three years, I’ve learned something: 90% of the value from the blog is in far less than 10% of the posts.  I’ve always written a mix of long-form, packed essays (you have now suffered through at least one — congratulations!) and shorter articles about more minor topics.  The shorter articles typically take me about an hour to write, whereas this article took me about four.  However, my long essays produce more traffic, more discussion, more links, and better writing (this last one subjective, the others less so) than the shorter articles — and vastly out of proportion to the time invested.  On a links-per-hour scale its better than a hundred to one in favor of the articles, I kid you not.  This has lead me to gradually curtail most of the minor posting to my blog.

You probably have something which could stand elimination in your business.  If you’re using Lean Startup-style processes, you should find out which of your development efforts are being 100% wasted because your customers don’t want what you’re making.  The solution to this is simple: stop that development.  You may spend time on social media when you could be working.  Work more, play when you get home.  (Says the guy who has racked up in excess of 200 hours on Hacker News — I never said I was particularly good at this.)

Maybe you spend time developing social media sites when you could be developing something people would pay for. ( Just kidding, Twitter.)

Well, OK, mostly just kidding.

Worker Smarter, Not Harder.  Then Go Home.

“Working harder” is a poor strategy which your competitors can trivially replicate.  Instead, spend some time measuring what tasks add value to your business and at what imputed wages.  Outsource those tasks which are below your desired imputed wage, automate any task where appropriate, and simply don’t do things which don’t add value.

I generally resist the urge to put a call to action in my blog posts, but just this once: working for the sake of work is a waste of time, resources, and human potential.  Try some of the suggestions from this post and see if you can’t cut a sliver of time off your work week — 15 minutes, an hour, whatever you can do.  If it works out, spend that time doing something which matters to you: read your kid a bedtime story, volunteer for your church, play WoW, whatever.  We should work to live, not live to work, and maximizing what the economists call “leisure” and what I’d rather rebrand as “stuff of lasting importance” should be a major goal for our careers and businesses.

A/B Testing Signup Page

I spent a day with my designer today and came up with the following.  The old registration page:

It empirically works (fairly well, actually) but it is spartan.  (The design is actually a slightly hacked version of the CSS which I downloaded from my Wufoo account — with their permission, naturally.  Yay for Creative Commons.  I don’t actually use Wufoo for storing any data but I love the form designer and pay $10 a month just to access it.)

The new, redesigned hotness:

I like this because it is more visually engaging and contains the sidebar, which I can stuff full of reasons they should sign up without impacting the readability of the main form.  It is also much more “on brand” for Bingo Card Creator, with the presence of my trademark (well, not really) H1 tag.

We’ll see which one ends up converting better.

While I was at it, I decided to try making the experience better for users coming in from my mini-sites.  Remember how I have a Halloween bingo page?  You might want to open that for a second.  The transition from that page to my site was sort of jarring — you move over domains, which normal users could care less about, but you also move from a black and orange color scheme to my soothing pastel blues.  Even with the guide text on the new page about Halloween bingo cards, I was worried about losing the “scent” — having a user figure “Whoops, this isn’t getting me to what I wanted” and closing out prior to signing up.

Accordingly, I did a bit of work to my template to automatically name the H1 with the card’s class, and all I have to do is upload a header image and write a wee bit of CSS and, presto-changeo:

Halloween Signup

Now it is much more obvious that I’m offering Halloween bingo, as promised.

BCC 3.0 — Got a moment to beta test?

Hideho everybody.  It is a five day weekend in Japan and I’m making the most of it.  On the agenda:

  1. Release BCC 3.0, finally.
  2. Add two new features to the online version.
  3. Finish a major article I’m writing.
  4. Start a new A/B test with the redesigned signup screens my designer is writing across the table as we speak.

If you’ve got a few minutes, I’d appreciate if you could download the Windows or Mac betas and tell me if they work properly on your system.  (If you can successfully print out one page of any Wizard, it works!)

New features:

  • It can save and load files from the clooooooooooooud.
  • It can export PDFs, for registered users at any rate.  (By calling a PDF creation web service, because I swear anything that reduces Java coding makes me happy.)
  • I have improved the registration process, transparently to the user, such that the majority of them will never have to see a Registration Key.  They just need to click OK.
  • Various bug fixes and improvements.