Kalzumeus Software

//
I run a small business selling software (downloadable and online) which lets parents and teachers make bingo cards.  It is October, which is the busiest season on the educational bingo calendar, largely because Halloween is coming up.  Kids in school + candy-fueled frenzy + secular(ish) holiday + desire for fun activity = bingo bonanza!

However, due to an IE CSS bug, my Halloween experience is best described as “bobbing for poisoned apples”.

The Design Improvement That Wasn’t

On September 21st my designer and I got together and we created a new candidate design for my sign up and login pages, which are both critical to conversions for my business.   The intent was to A/B test the new pages against the old design.  One critically important niggle that didn’t bother me at the time: while ideally an A/B test would test exactly the old HTML against exactly the new HTML, for obscure implementation reasons we ended up testing the old HTML plus one little change versus the new HTML.

That one little change was replacing the old sign up button (the stock HTML form submit one) with a graphical button.  I thought this was such an obviously beneficial change that it was not worth taking on extra coding complexity to keep it the old way.  Stupid, stupid, stupid.

Thanks to the magic of subversion I was able to recover exactly the HTML that was displayed on my site after the upgrade.  (This is one of the two A/B alternatives but it is exactly identical to the other one in the area that is relevant.)  Feel free to open that in your browser and see if you can spot the problem.

At this point I have to mention that I generally use IE8 or Chrome, and my designer uses Firefox.  This means I had never seen the new version of my site in IE6 or IE7 until trying to do tech support from an Internet cafe last night.  (Being a proper Japanese salaryman, I had missed the last train and stopped at the cafe to answer customer emails prior to staying at a hotel.)

Whereupon I discovered a slight problem.  In IE7, the site looked more like:

Hmm, the signup form has no visible sign up button.  The log in form also had no visible log in button.

I started panicking but was in no position to fix the bug at 2 AM in the morning from an Internet cafe, although I made a game attempt to do so.  (Word to the wise: trying to fix a Rails application by editing files in vi through a web console to your Slicehost VPS is not recommended.)

How Well Do Forms Without Buttons Convert?

Answer: not well at all.

You can still submit a form by hitting the Enter button even if you can’t see the submit button.  This is, however, not standard user behavior in my niche.  Let’s see what that did to conversion rates.  Using custom segments (a power-user feature from Google Analytics that I’ve literally never had any useful purpose for before), I’ve graphed the conversion rates among IE8 users (orange line), IE6/7 users (green line), and the site average (blue line).

As you can see, IE6/7 and IE8 are pretty much neck and neck in the month before the bug is introduced.  This makes sense, as we would not intuitively assume those two user populations would be very different from each other.  You can also see that they both track the site average reasonably well, which is practically true by definition as IE users make up the majority of visitors on my site (about 65%, give or take).

Then, after the bug gets introduced, the conversion rates for IE6/7 take a nosedive relative to IE8.  The site average in blue also declines, solely because it is being pulled lower by the underperformance of the old IEs.

In concrete terms, the conversion rate of IE6/IE7 users was 3.40% after the bug.  (Note that this is not the conversion rate among people actually viewing the login/registration screens.  Google Analytics doesn’t let me conveniently break the numbers down for that.)  The conversion of IE8 users, by comparison, was 9.53%.  In other words, if you trust my intuition that IE6/7 users are about as likely to convert as IE8 users all else being equal (which you should, just by eyeballing that graph), then I lost about 64% of my conversions since September 21st.

Putting The Damage In Economic Terms

I’m a glutton for punishment so I’m going to do the painful math.  Between September 21st and October 23rd, when I fixed the bug, I had roughly $4,500 worth of sales, of which IE6/7 users comprised about $2,080.  Since the IE6/7 bug is essentially meaningless after you get through registration (since the overwhelming majority of users pick Remember Me and never see either the sign in or sign up forms again), I’m going to make the simplifying assumption that the bug ripples straight to my bottom line.

We can then use simple math to figure out how much the 64% of sales I threw away was probably worth.  Ouch, I am wincing as I type this: $3,750 in lost sales.  That unfortunately flows almost straight to the bottom line, since all of my major costs (advertising, hosting, etc) happen whether I make the sale or not — I probably lost in excess of $3,400 in profits.  That is more than my monthly salary.

How The Heck Do You Overlook That?

I wish I could blame Microsoft for my stupidity on this one, but aside from not testing the page in IE6/IE7 (which would have shown the bug immediately), I had ample opportunities to discover that there was a problem somewhere.  However, other things in the chaotic, fractally complex system that is a small business concealed them from me.

Indication #1: There are two types of signups on my site, trial signups (where I capture an email address) and anonymous guest signups (where I don’t).  Historically, I push the trial signups much harder, and my users signed up for them at a multiple of the guest signups.  (Guest accounts scarcely convert for me at all, which is why I don’t push them much.  However, an A/B test proved that removing the option didn’t increase the number of trials or paying customers, so I left the option in.)

Thus, when the mix of trial and guest users suddenly went from 880 : 350 to 792: 506, I should have said “Wait a minute, the guest accounts are getting much more popular.  Why is that?”  The answer to that is: Unsophisticated users, like the ones who make up most of my audience, will fill in their information and then see ‘Sign in as guest’ and ‘Cancel’.  Given only these two choices, clearly they want to ‘Sign in as guest’ rather than figure, “hmm, typically guest accounts don’t have usernames or passwords associated with them, I should email Patrick and ask what happened to the real sign up button.”

And indeed, I did see the explosion of guests on my stats, but I had changed the graphing option from showing daily counts to showing weekly counts, and the disparity wasn’t big enough to catch my sustained attention.  (The inflection point on September 21st is immediately visible on the daily graph.)  After considering the matter for a few minutes I said “Eh, Halloween is coming up, which always broadens my market.  Maybe it is bringing in some less interested folks.  Oh well, nothing to worry about.”

Indication #2:

I check my conversion rates on AdWords roughly once a week, largely because if they start to suffer Google stops running my ads and I lose a lot of money.  I then noticed that the numbers were below where I expected them to be (“Hmm, that’s funny, a few weeks ago I was getting 24 – 27%, now it is saying 18%… what is up?”) but I never ran the following graph to see the full range.

Yeah, just from eyeballing that, you can see the problem right?  Visualizations are wonderful for seeing problems that you already know to look for.  However, they are poor for informing you of problems that aren’t visible in the default visualization, since you won’t spend the time to slice the data to see the problem if you don’t know it exists.

Honestly, I don’t know why I didn’t immediately punch the panic button when I saw my AdWords conversion rates start declining.  Again, October always throws my numbers into a total mess, and I was very busy in my day job and in my various seasonal initiatives for taking advantage of Halloween.  So busy, in fact, that I offset most of the decline in sales, and so wasn’t given any advance warning by, e.g., having sales plunge to a fraction of normal.

Indication #3: What finally clued me into looking for this problem was a pattern in customer support requests, of all things.  As I mentioned earlier, guest accounts are historically rarer than trial accounts and they convert very, very poorly.  As a matter of fact, I think I probably had one sale to a guest account in history prior to the bug being introduced.

The code that I have which upgrades accounts in response to people paying me via Paypal and Google Checkout doesn’t handle guest accounts very well, because they’re anonymous.  It upgrades the account fine, but doesn’t register the email address they used for their payment with the guest account, which means the account still can’t log in after logging out.  (Since the guest account is still anonymous.)  However, since the system knows that the Registration Key issued to the purchaser is in use (on their guest account), if for some reason they log out or switch computers, they’ll be both unable to register a new account (because their Registration Key is in use) and unable to log into their guest account (because it is anonymous).  That is a pretty nasty Catch 22.

I’ve known about that Catch 22 for a while, but fixing it was not high on my list of priorities.  All of the following has to happen for it to actually bite a customer:

• They start using a guest account.
• They don’t give an email address when prompted by the fairly frequent upsells within the guest account.
• They nonetheless purchase the software within their guest session.
• They choose to use the online version of the software rather than the downloadable version (a not-insignificant portion of customers think the online version is the trial and the downloadable version is the “real deal” — well, if they pay me money, they can think whatever they please).
• They clear their cookies.
• They try to log in again.

Prior to the bug being introduced, I thought “This sequence of events is about as likely as flipping a coin and having it land on edge.”  And, indeed, it only ever happened to one person.  I fixed her record manually, looked at the code which caused the problem, and figured that an immediate fix was impossible.

<tangent>Why not just set their email address to the one the payment processor just gave me?  Well, Google Checkout has an option which it labels “Opt out of marketing email from this merchant” which, if the customer checks it, gives me an email address like “John.234982385@checkout.google.com” for them.  Obviously, my customers don’t know those random addresses, so using them for a login name is out of the question.  I was also worried that many people’s Paypal addresses are not the ones they use on a daily basis, so using them for logins was suboptimal, so I deferred finding a solution to the bug until later.  After all, it scarcely affected anyone, the cures were temporarily worse than the disease, and manually correcting their records was suboptimal but worked.</tangent>

Then I got emails from three people in the same day who were hit by the guest-purchase bug.  That was finally the eight sigma event that convinced me that my process was really, really out of control.  Whereupon I started investigating, beginning (fortuitously) with trying to log into my admin page on the site while on the Internet cafe’s IE7, and noticing that there was no login button.

Technical Mumbo Jumbo

If you’re a CSS geek, you might be interested in knowing why the buttons vanished.  Fair enough: the HTML input element had a background image specified and a text-indent:-9000px property applied to it, which puts the standard text for the button way off the screen, leaving only the background image visible.  However, IE6/IE7 treat text-indent as moving the background image as well, so the entire control essentially vanished, leaving empty, unclickable space where it used to be.

The fix was fairly simple.  I am truly indebted to Laurence at My Drupal Blog for posting about it.  He probably saved me an ulcer.  Essentially, I identify which browsers are using IE and then apply some sort of hacky stylings to make the background image display visibly, but cause the Sign Up text to be invisible by blending it with the background.  It is very hacky — sort of like patching a wound to your artery with duct tape — but you can’t be too choosy when you’re leaking over $100 a day and have the CSS skills of a vole rat.

On The Bright Side

I don’t want to give you the wrong impression: despite this very, very serious issue, my business has been doing very well in October.  The usual seasonal fluctuation, my seasonal promotions, and all the various improvements I’ve been making in the last few months will still make it my best month ever, and historically the last week of October sees a strong spurt in sales, so it might even be the best month ever by quite a margin.

I also got a much-needed kick in the pants to remind me to test all changes, even the stupid little one-line changes, in all the browsers my customers routinely use.  One positive lesson learned is that IE8’s compatibility mode reproduced this bug exactly, so that will be an easy way to do this going forward.  Ideally I’d have some sort of automated test suite set up to catch this sort of thing (using Selenium or what have you), but that isn’t in the cards at present.  At the very least, I’m going to upgrade my automated stats tracking put big red warnings on my dashboard when the business starts to fail sanity checks.  An obnoxious red warning about conversion rates would have caught this a month ago, within less than 24 hours of the bug going into production.

I hope you learned something from my experience, and am quite willing to take additional lumps in the comments if you have any suggestions for good ways of preventing a similar bug in the future. Happy Halloween!

For three years now I’ve been running a small software business in my spare time.  It has been a very educational experience, especially in showing me that many things we think we know about software, programming, business, and the like are wrong.  This is a bit of a shock, especially for well-worn chestnuts which have intuitive appeal, which we have come to invest with moral significance, and (most importantly, because we all think we’re smart) we’ve believe so self-evidently true as to make investigation a waste of time.

For example: I have come to the conclusion, over the last three years, that working hard is overrated.  This is an idea I have been kicking around for a while, but it was thrown into sharp relief by a blog post entitled The Only Alternative Is To Work Harder, by a gentleman named Paras Chapra over at Wingify.  Paras and I have corresponded over email a few times, so I say as one analytics junkie to another: the notion that working longer hours is correlated to better business results is a pernicious social pathology.

Working Longer Hours Is Not A Competitively Defensible Advantage

For the last five years, I’ve been a Japanese salaryman, and have often worked 70 hour weeks out of a sense of social obligation.  I understand, very well, the social pressures which could lead someone to write “If your nearest competitor or neighbour works X hours, you must work for X+1 hours.”  It is just a terrible strategy.  Your competitor can adopt it as easily as you can, and then you’re playing a game of multiplayer endurance chicken against everyone else in your market.  You can’t win but you can certainly all lose, by ending up with an entire community where soul-crushing hours are normative.  (There are certain tendencies to this among Silicon Valley startups.  Take it from a Japanese salaryman, guys: it is a disease so vicious that in addition to hobbling businesses it damages society itself.  We’re barely beginning to recover from it decades later.)

Why can’t you win?  Well, suppose that longer hours are indeed the key to success and that Paras is willing to work longer hours than 99.9% of the population of India.  I don’t know how many hours that is at the 99.9% level, but call it 82 a week.  He’ll work 82 hours and then find, oh shoot, over a million people are willing to work more.

Working harder is a particularly bad idea for startups because you are likely competing with people with resources which, relative to yours, are infinite.  I compete with several educational publishers who employ tens of thousands.  Paras competes with Google.    Our competitors have more man-hours in a week than we’ll have in the next decade.  Engaging them on those terms is madness.

Why Smart People Keep Falling For This

Working long hours is near the perfect storm of meme spreadability.

It flatters the sensibilities of many religions — Max Weber was putting the Industrial Revolution down to the Protestant Work Ethic a hundred years ago.  It sometimes confuses me that it is so popular with atheist Japanese salaryman and agnostic Silicon Valley founders.  Then again, programmers at most companies work on a schedule designed to maximize the productivity of illiterate 18th century water loom operators, so expecting rationality might be excessively optimistic.

There is also a sense that working less is somehow, you know, cheating or immoral.  Take the reactions to 4 Hour Workweek (which is, incidentally, a tonne of self-promotion and self-help book hucksterism concealing an ounce of solid productivity gold): a lot of people are emotionally invested in the way that they have always done things being the correct, proper, morally acceptable way of doing things.  A lot of my fellow software developers feel the need to work long days because otherwise their customers will see them as lazy, despite the fact that their customers can only perceive the external indicia of the labor rather than the labor itself.  My customers don’t know that I average four hours a week on my business any more than they know I average 70 hours a week on my day job: all they see is the web site.

It is easy to fall into this trap of productivity being defined in terms of observed effort exerted because in the typical face to face organization it is easy to see who is “working hard” and very difficult to measure actual productivity.  The manager (and peer employees) always knows who was putting in heroic efforts at 10:00 PM last night.  However, your organization almost certainly doesn’t track productivity in any rational, systematic fashion.

This is a pity, because you can’t improve what you don’t measure.  Web analytics has taught us that our cherished beliefs about web design don’t matter an iota in the face of actual customer behavior.  Customer development has shown us that most of the time we’re spending enormous amounts of resources (including time) on products which will fail for lack of demand.  What is the use in one-upping the competition and spending 83 hours to their 82 when you’re both producing something that no one even wants?

Poor Metrics For Productivity

Of course, if you pick a poor metric and try to optimize for it, bad things happen.  For example, many large Internet publishers put auto-refresh code into their pages to inflate page views.  That adds no lasting business value (although it might succeed in getting a few more scraps from CPM advertisers prior to them abandoning the campaign in favor of models that actually work).

Similarly, my day job used to measure engineering productivity in, essentially, hours.  If I worked 20% more hours in August than I did in July, I was 20% more productive in August.  I remember one of my “very productive” weeks: after sleeping in a hotel because I was not physically capable of commuting home, I got to work and started pulling tickets, then opened the appropriate file and started coding.  I then continued for 90 minutes chasing rabbits before I realized that a) I had started working on a file totally different than the feature being discussed and b) the problem I thought I was fixing was not written in the ticket, anywhere.  I had literally hallucinated the entire request.  I then proceeded to spend the rest of the day committing crash bugs into the trunk, confusing our contractors with instructions that might as well have come from someone abusing drugs, and generally spinning my wheels.

Under the productivity tracking system we had at the time, that was my most productive day ever.  You don’t want to emulate that example.  For more examples of things not to emulate, see the Mythical Man Month.

Good Metrics For Productivity

I sell software online in my business. It is very, very easy to identify the direct driver of business value: selling software.  One conversion is worth $30.  It is even pretty easy to identify contributors to that: working up the funnel, trial signups aren’t worth as much as sales, but given their observed conversion rate 70 cents is a decent estimate.  “Casting to currency” lets you compare the worth of two different things, easily.  An intervention which generates 10 sales is less worthwhile than one which generates 4,000 trial signups (all else equal).

In practice, this works well for tracking productivity, too.  Just assign an arbitrary value to tasks, based on your best guess of how much value they add for the business.  Then, track how long it takes to complete the tasks, and figure out where you’re adding disproportionate amounts of value and where you are spinning your wheels.  Do more of the former, less of the latter.

When in genuine doubt about the value, guess what it would cost to have somebody else do it for you.

The Pseudo-Wage

When I started my business I thought it would be amazing if I eventually earned $100 per hour working on it.  (This is a princely sum for 20-something programmers in Japan.)  Its funny, people generally get very good with practical experiences of the mathematical properties of averages in school and then totally forget about that experience in business.  If you want a 92 average, you’d better not routinely get 60s on your homework.  If you want to earn $100 an hour, you’d better not busy yourself with $5 an hour tasks.

Examples:

• How much is mailing a CD worth?  Like many software developers, I offer CDs of my software.  The primary purpose is to get the sale from people who would not buy if it were purely virtual.  Thus, my labor in generating CDs is worth however many marginal sales the CD option nets me, plus the marginal revenue from actually paying more for the CD.  If I had A/B tested the option, I’d have a very rigorous notion of how much CDs are worth, but we’ll go with a guess: 25% of sales of the CDs would not have happened but for the CD option, thus a single CD is worth $5 (what I charge) + $7.50 (a marginal $30 sale every 4 CDs) = $12.50.  When you consider that I’d have to do all the burning, enveloping, addressing, licking, and mailing myself (in very small batch sizes), its reasonable to assume that doing CD fulfillment myself would be worth on the order of $50~60 an hour.
• How much is A/B testing worth?  I generally try for about 5 new A/B tests a month.  If the successful ones combine to a 5% improvement (not infrequent) and you generously assume that takes 10 hours, then over the course of a year I’ll earn over $1,500 from them, for over $150 an hour.

You’re Measuring Productivity.  Now, Improve It

Clearly, this means that instead of licking more stamps I should be writing more A/B tests, right?  Sure, I can lick one stamp and write one A/B test, but my time is my most important and limited business asset, so I have to be sparing with it at the margin.

“But Patrick, that is all well and good, but you still have to mail those CDs!”, I hear you say.  No, I have to make sure the CDs get mailed, which is a distinction with a difference.  No middle American schoolmarm perceives additional value over having my saliva the envelope her software came in.  I have long-since outsourced the actual production and mailing of the CDs to SwiftCD.  They cost me about $6 each and, when I was still typing orders in manually, increased the amount of orders I could fill in an hour from about 4~5 to about  10 (note, again, small batch sizes mean this was dominated by setup-and-teardown time as I lost productivity to friction turning on my computer, opening email, copy/pasting over details, checking them, then turning off the computer and running to work).  That results in a significant increase in my psuedo-wage: after deducting the extra cost of outsourcing versus insourcing, I still make ~$100 an hour versus ~$50 doing everything myself.

Productivity Technique #1: Outsource.

This is the first of three major tactics for improving productivity: outsource anything that your personal presence does not add value to.  Equivalently, outsource anything where the replacement price is less than your desired pseudo-wage.

Remember that outsourcing imposes an overhead cost, and payment is due in your time rather than dollars.  Often, this overhead swamps the actual monetary cost of the project.  This kills many outsourcing development projects, because communicating (detailed, constantly changing) customer requirements consumes an enormous amount of time on the part of both sides.

In my own business, I very rarely outsource development, for this reason.  I also don’t outsource direct interaction with customers, because I feel that as a small business knowing what my customers want is an important differentiator.  Everything else, though, is on the table.

My most important use of outsourcing is getting content written for my website.  My software creates bingo cards, and offering bingo cards in the exact niche a searcher is looking for helps convince them to sign up for the free trial.  For example, if you’re looking for a Halloween bingo activity to play with class, the fact I have one ready to go increases your likelihood of sourcing your immediate and future bingo card needs with me rather than making them yourself or going to a competitor.

I used to write my bingo cards myself, and I’m fairly good at it, but eventually I figured that while it was a worthwhile activity it didn’t really get all that much more worthwhile as a result of me doing it.  Instead, I put out a call on my blog for freelancers, and eventually worked out a mutually rewarding relationship with a highly-educated American teacher.  She bangs out the cards on her own schedule, and once a month I click “Post” on my backend interface and then mail her a check.

The economics of this arrangement are so staggeringly efficient that people tell me I have to be lying about it.  The pages my freelancer writes for me were visited 65,000 times in September, producing roughly $1,300 worth of sales for me through getting people into my various conversion funnels.    I did less than five minutes of work to maintain the freelancing relationship in September.  Do you want to do the math?

There is no possible way for me to achieve results like that merely by lengthening the number of hours I work in a day.  Additionally, because I’m capable of producing like that (via, e.g., leveraging freelancers intelligently in a business process which uses software to make them efficient and then extracts business value out of their labor), there is no need to spend excessively long hours working for the sake of working.

Productivity Technique #2: Automate Your Processes.

Software developers should really spend more of their time creating tools for themselves, but we’re hardly the only vertical this applies to.  One of the best points of an awesome lecture on lean startups is that a startup’s most important product is the process the startup uses to create products.

This idea is so powerful that I say, without a hint of exaggeration, it has changed history.  A huge portion of Japan’s rise to global prominence was a result not of working harder, morning radio exercise routines, of the superiority of wet rice cultivation for creating productive societies (all of these, and more, were explanations advanced by authors on the syllabi for the courses I took in the process of completing my East Asian Studies degree — which will teach you to trust academia).  It was by having a few decades of head-start on process improvement as a science.

(I’d be remiss if I didn’t point out that folks have also identified process improvement as a cause of Japan’s economic boom.  However, “Toyota considers it a worthwhile use of time to shave 15 seconds off the construction of an automobile” is so counter-intuitive and unsexy that people reach for the much more obvious, morally correct, visible explanation that Toyota workers work so much harder than GM workers.  This is despite inconvenient facts such as the fact that Toyota can reproduce their productivity advantage in American factories using American workers, and has, repeatedly.  Journalists and management consultants never let facts get in the way of a good narrative.)

Process improvement works for small companies, too!  For example, my use of freelancers rather than myself to write bingo cards is one example of a process improvement, but when I started the process was horrific.  They’d create the word lists (as text files), then have to manually run them through Bingo Card Creator, take screenshots, do manually cropping, etc etc.  This is a lot more efficient than having me do the same steps (their time is cheaper than mine, and I can always find another freelancer but finding a 25th hour in a day is sort of tricky), but there was clearly room for improvement.

I actually outsourced creation of the first improvement: it was a script which, given specially named text files as input, would open up Bingo Card Creator, emulate a user on the keyboard clicking on the particular buttons, and then open up PDF viewer, emulate hitting Print Screen, open up MSPaint, etc etc.  It was a system held together entirely by duct tape, but it worked.  It took me an hour every month to supervise the script and correct unhandled exceptions but it worked.  The productivity of my freelancers immediately increase.  This let them get more work done in less time.  I then bumped up their wages and the resulting more-money, less-busywork combination has kept them sufficiently happy with the arrangement that it has continued without a serious hitch for more than a year now.

(Incidentally, I eventually spent some time and replaced the duct tape.  Remember, it was still taking me an hour a month and I still had access to a computer: clearly, there was room for improvement.  I eventually tightened up the automation, decreasing it from 60 minutes to 40 minutes, then refactored some of the automation and got it down to 30 minutes, then leveraged an unrelated technology change I made and got it down to the current five minutes.  Five minutes isn’t zero minutes, either, so I still have work to do.)

Process improvement takes time.  Consider it an investment in your business’ future, and charge yourself for it.  I outsourced the first draft of that automation and got valuable results for $100.  (Why didn’t I write it myself?  Automating interfaces is Not My Bag, Baby and it would have taken me far more than an hour to do.)  As it is, all the development I’ve ever done on every draft of that system, plus all the handling of freelancers, adds up to less than twenty hours.

Productivity Technique #3: Eliminate Unproductive Uses of Time.

The final technique for maximizing your productivity is eliminating uses of time which do not add business value. We’ve all got them, most of them probably unknown to us because we don’t track our uses of time well.  (I use RescueTime.  If you don’t, install it today and spend some time aggressively categorizing what websites are worthwhile for you to be on.  Consider this a very, very effective process improvement.)

For example, as I mentioned earlier, I’m a bit of an analytics junkie.  RescueTime reported to me that I was spending upwards of 1 hour a day on Google Analytics.  I had an honest conversation with myself on whether I was getting $100 worth of insights every single freaking day out of Google Analytics.  The answer was, no, not really — I was essentially engaged in a nervous habit, opening Analytics and drilling down through a million reports to feel productive when I wasn’t mentally ready to actually be productive.  In particular, I had a few favorite screens I’d check too often — trial downloads per day, for example.

I eventually killed that habit by surfacing that number on my admin dashboard for my website (where I have to go to resolve routine customer service inquiries).  That way, the number flashes by the corner of my eye once every morning as I do productive work.  It is also set so that it gets cached for 24 hours, so that the monkey-brain WoW playing scrolling-numbers-are-dopamine-cast-into-integers side of me can’t get fascinated into sitting on the page and hitting refresh all day.

That sure isn’t going cold turkey, though.  A better example: I used to spend much, much more time on blogging than I do currently.  This is my 447th post to this blog, and I consider the blog one of the main contributors to my business doing as well as it has.  However, over the last three years, I’ve learned something: 90% of the value from the blog is in far less than 10% of the posts.  I’ve always written a mix of long-form, packed essays (you have now suffered through at least one — congratulations!) and shorter articles about more minor topics.  The shorter articles typically take me about an hour to write, whereas this article took me about four.  However, my long essays produce more traffic, more discussion, more links, and better writing (this last one subjective, the others less so) than the shorter articles — and vastly out of proportion to the time invested.  On a links-per-hour scale its better than a hundred to one in favor of the articles, I kid you not.  This has lead me to gradually curtail most of the minor posting to my blog.

You probably have something which could stand elimination in your business.  If you’re using Lean Startup-style processes, you should find out which of your development efforts are being 100% wasted because your customers don’t want what you’re making.  The solution to this is simple: stop that development.  You may spend time on social media when you could be working.  Work more, play when you get home.  (Says the guy who has racked up in excess of 200 hours on Hacker News — I never said I was particularly good at this.)

Maybe you spend time developing social media sites when you could be developing something people would pay for. ( Just kidding, Twitter.)

Well, OK, mostly just kidding.

Worker Smarter, Not Harder.  Then Go Home.

“Working harder” is a poor strategy which your competitors can trivially replicate.  Instead, spend some time measuring what tasks add value to your business and at what imputed wages.  Outsource those tasks which are below your desired imputed wage, automate any task where appropriate, and simply don’t do things which don’t add value.

I generally resist the urge to put a call to action in my blog posts, but just this once: working for the sake of work is a waste of time, resources, and human potential.  Try some of the suggestions from this post and see if you can’t cut a sliver of time off your work week — 15 minutes, an hour, whatever you can do.  If it works out, spend that time doing something which matters to you: read your kid a bedtime story, volunteer for your church, play WoW, whatever.  We should work to live, not live to work, and maximizing what the economists call “leisure” and what I’d rather rebrand as “stuff of lasting importance” should be a major goal for our careers and businesses.

I spent a day with my designer today and came up with the following.  The old registration page:

It empirically works (fairly well, actually) but it is spartan.  (The design is actually a slightly hacked version of the CSS which I downloaded from my Wufoo account — with their permission, naturally.  Yay for Creative Commons.  I don’t actually use Wufoo for storing any data but I love the form designer and pay $10 a month just to access it.)

The new, redesigned hotness:

I like this because it is more visually engaging and contains the sidebar, which I can stuff full of reasons they should sign up without impacting the readability of the main form.  It is also much more “on brand” for Bingo Card Creator, with the presence of my trademark (well, not really) H1 tag.

We’ll see which one ends up converting better.

While I was at it, I decided to try making the experience better for users coming in from my mini-sites.  Remember how I have a Halloween bingo page?  You might want to open that for a second.  The transition from that page to my site was sort of jarring — you move over domains, which normal users could care less about, but you also move from a black and orange color scheme to my soothing pastel blues.  Even with the guide text on the new page about Halloween bingo cards, I was worried about losing the “scent” — having a user figure “Whoops, this isn’t getting me to what I wanted” and closing out prior to signing up.

Accordingly, I did a bit of work to my template to automatically name the H1 with the card’s class, and all I have to do is upload a header image and write a wee bit of CSS and, presto-changeo:

Now it is much more obvious that I’m offering Halloween bingo, as promised.

Hideho everybody.  It is a five day weekend in Japan and I’m making the most of it.  On the agenda:

1. Release BCC 3.0, finally.
2. Add two new features to the online version.
3. Finish a major article I’m writing.
4. Start a new A/B test with the redesigned signup screens my designer is writing across the table as we speak.

If you’ve got a few minutes, I’d appreciate if you could download the Windows or Mac betas and tell me if they work properly on your system.  (If you can successfully print out one page of any Wizard, it works!)

New features:

• It can save and load files from the clooooooooooooud.
• It can export PDFs, for registered users at any rate.  (By calling a PDF creation web service, because I swear anything that reduces Java coding makes me happy.)
• I have improved the registration process, transparently to the user, such that the majority of them will never have to see a Registration Key.  They just need to click OK.
• Various bug fixes and improvements.

[Editor’s note: now available in Belorussian translation.]

Breaking up has always been difficult for me.  I tend to fall in love with being in love, and continue a relationship well past the point of futility.  And so it is with my oldest love, writing desktop software.

I’m sorry, desktop apps.  We just don’t have a future together anymore.  Its not you, its me.

A bit of background: for the last three years I’ve sold Bingo Card Creator, a desktop app which pretty much does what it says on the tin.  It has gone from being a passing fancy to a rather lucrative hobby to, well, a bit more than that over the years.  As I gradually became more invested in the business of writing desktop software, I got more and more snippy about the periodic desktop versus webapp flamewars, and defended the ascendancy of desktop software.

What Changed My Mind

Over roughly the same period my day job has changed and transitioned me from writing thick clients in Swing to big freaking enterprise web apps.  I’ve learned SQL, Rails, etc and used them to fairly decent effect in selling Bingo Card Creator, which is a Swing app (if all you have is a hammer…).  This summer, I decided to try stepping my web programming skills up a notch, and released a web version of Bingo Card Creator.  It has exceeded all my expectations: in ease of writing, in features, in sales, in support burden, in marketability, etc.  In game theory terms, it strictly dominates the desktop version, when seen from the eyes of the developer at any rate.

If I were starting out today, I would, without a shadow of a doubt, write a web app instead of a desktop app, for these reasons:

The Shareware Funnel Is Lethal

I have never used the word “shareware” to describe Bingo Card Creator, because I think that it is an anacronism that my customers do not understand, but among fellow technically inclined people it describes the business model succinctly.  Someone visits your website, downloads your trial, and hopefully purchases your program.  That process is called a funnel, and if you break it down into concrete steps, the shareware funnel is long and arduous for the consumer:

1. Start your web session on Google, like everyone does these days.
2. Google your pain point.
3. Click on the search result to the shareware site.
4. Read a little, realize they have software that solves your problem.
5. Mentally evaluate whether the software works on your system.
6. Click on the download button.
7. Wait while it downloads.
8. Close your browser.
9. Try to find the file on your hard disk.
10. Execute the installer.
11. Click through six screens that no one in the history of man has ever read.
12. Execute the program.
13. Get dumped at the main screen.
14. Play around, fall in love.
15. Potentially weeks pass.
16. Find your way back to the shareware site.  Check out price.
17. Type in your credit card details.  Hit Checkout.

I could go into more detail if I wanted, but that is seventeen different opportunities for the shareware developer to fail.  If you don’t catch the download in the 30 seconds people give your website, no sale.  If your customer can’t find the file after they download it, no sale.  If it requires a JRE upgrade and after restarting their computer they’ve forgotten what they were working on, no sale.  If they play around with it, close it, and can’t remember how to open it again, no sale.  If they get to the sales page and can’t operate your shopping cart, no sale.

Is it any wonder why shareware has typical conversion ratios of 1% or less?

Web Applications Convert Better

A web application doesn’t have to be downloaded or installed, never requires a restart, and never requires a contextual change just to open up a purchasing page.  As a result, the conversion ratio is higher.  Much higher.  Here are the actual stats from Bingo Card Creator.  I’m looking at conversions from my best performing AdWords campaign only, because that minimizes sources of variation like, e.g., the different types of traffic I’ve gotten in the last 2 months (while the webapp was available) versus in the last three years.

Visitor to Free Trial:

• Downloaded: 18 ~ 22%
• Web App: 22% ~ 26%

Trial to Purchase:

• Downloaded: 1.35%
• Web App: 2.32%

This is essentially the same application.  If anything, the online version has less features, and it has 2 months of development whereas the downloadable application has had 3 years of improvements made to it.  Yet the online version outsells my desktop application almost two to one.

Your AdWords Strategy Is Very Sensitive To Conversion Rates

A portion the numerical disparity is because I have started to react to, e.g., the difference in conversion rates of advertising and promote accordingly.  A sale of either nets me the same amount of money, about $28.  However, if you break out the math on how much AdWords costs per sale (cost per click divided by conversion rate to trial divided by conversion rate to purchase):

• Downloadable version: $20 AdWords CPA
• Web App: $9 AdWords CPA

(You’re welcome, Google.)

This doesn’t just save me money, it helps me trounce my competitors.  For example, if my competitors are selling downloadable software, and they are equally as skilled as I am about writing AdWords ads and optimizing their websites, then it should also cost them about $20 a sale to advertise on AdWords.  (This explains why I never see ads for the competitors who try to gain volume by undercutting my price — if you’re going to price at $23.95, you’d better be a crackerjack SEO because you simply cannot afford to outbid me in AdWords.)

Decreasing my cost of customer acquisition by over half lets me bid more for my AdWords to gain additional volume.  For example, for the longest time my AdWords strategy was more or less monetizing traffic other people couldn’t be bothered with, while larger brands producing e.g. printed bingo supplies went after the head terms like [bingo cards].  With vastly improved conversion rates, I might be able to advertise profitably on those terms, increasing my volume and making me very, very happy.  As it is, I have walked up bids a bit and am getting 25% more inventory than I usually do.

Web Applications Are Easier To Support

Many desktop developers hate customer support with a burning passion in their soul.  I actually enjoy it, but I enjoy making it unnecessary even more, as there is no customer support experience so good as avoiding the problem in the first place.

Support requests from last 50 customers:

• Desktop Application: 15
• Web Application: 3

I’ve had three years to streamline the website, purchasing process, and application for my desktop app, and that has helped me greatly reduce the number of inquiries I get.  Even after all that work, the main culprits are pretty much the same as ever: installation issues, lost registration keys, and bugs present in old versions of the software that are still floating around download sites.

Web apps, by comparison:

• Have no installation issues, because there is no installation.
• Do not require registration keys.  (Technically, because I allow users to use both the desktop and web application, I issue them one — but it is immediately applied to their account via creative abuse of e-junkie and cookies.  Most customers get to use their software immediately without actually reading the bit in the email sent to them — or failing to read it, as happens quite often.)
• Never have an accessible version of the software older than the most recent one.  By comparison, if you were to Google [bingo card creator version 1.04] (which hasn’t been distributed in, hmm, two years or so), you’d find it on hundreds of download sites.

The Age Of The Pirates Is Coming An End, Jack

I’m famously lackadaisical about software piracy, preferring to concentrate on satisfying paying customers rather than harming their experience with anti-piracy methods.  However, the existence of pirates is a stitch in my craw, particularly when any schoolmarm typing the name of my software into Google is prompted to try stealing it instead:

You want to take a quick stab at how many pirates have circumvented the copy protection on the online version?  Bwa.  Hah.  Hah.

I once remarked to Paul Graham that the future of software was with pervasive integration with the server simply because that means that downloading the client doesn’t let you pirate the software any more than downloading Firefox lets you pirate Basecamp.  (Ironically, I made that point in a defense of desktop software as a business model.  Mea maxima culpa!  Theoretical utility of desktop software is one thing, but I can’t ignore what my numbers are telling me.)

Phone Home vs. Google Analytics

One of the curious traits among software developers is that, speaking as a group, we feel something like “I own what happens on my machine and nothing should happen without my say-so”.  This generally leads to a severe reluctance to “phone home” from the application to the developer’s server — even reports on very innocuous data like “Did I steal this software or not?” is often tarred with the label spyware.

On the Internet, privacy expectations have evolved a bit in the last few years.  The overwhelming majority of the public has been told that they’re being tracked via cookies and could not care less.  If you write a privacy policy, they won’t even bother reading it.  Which means that you can disclose in your privacy policy that you track non-personally identifying information, which is very valuable as a software developer.

• What features of your software are being used?
• What features of your software are being ignored?
• What features are used by people who go on to pay?
• What combination of settings is most common?
• What separates the power users from the one-try-and-quit users?

Tracking all of these is very possible with modern analytics software like, e.g., Mixpanel.  You can even wrestle the information out of Google Analytics if you’re prepared to do some extra work.  You can do it in a way which respects your users’ privacy while still maximizing your ability to give them what they want.

Some people may be under the impression that users will tell you what they want.  Nope — most of them will assume you are like every other business they have ever dealt with, where their opinion doesn’t matter, and the software is offered take-it-or-leave-it.  And they just left it!

Things I learned about Bingo Card Creator customers which I never knew before I had an online app:

• The most common word used in bingo cards is — ready for it — “baby”.  I completely underestimated the demand for Baby Shower bingo cards, and avoided making an official set for years.  As soon as I had the top ten word list (which was all baby shower words) I fixed that.
• The more features I add to the software, the worse it sells.  (This is, needless to say, highly unintuitive to most software developers.)
• Most customers purchase within two hours of signup, so it is absolutely imperative that their first use of the software exceed all their expectations.

Web Apps Can Be Customized Per User

Downloadable software pretty much has to treat every user identically by default.  There are very limited ways to segment users, and no way to report the results of experiments.  For web apps, however, if you have a halfway decent A/B testing library (like, say, the free one I wrote for Rails developers), you can experiment with having multiple versions of the application available concurrently, and see which one performs best.

The data collected by A/B testing has helped me:

• simplify my options screens to avoid confusing users
• improve the first-run experience
• write instructions such that they’re easier to follow

In addition to changing program behavior randomly, you can segment your users.  I have only scratched the surface of how powerful this is, and it is already producing solid results for me:

Don’t treat your newbies like you treat your power-users. You have a database.  It records all their actions since the dawn of time.  Use it.  I have a couple very simple heuristics for “is probably still getting used to the software” and, if you are, the software treats you with kid gloves.  For example, it hides complex, seldom used options by default.  It gives you instructions to a degree that a power-user might find insulting.  (I don’t have the artistic skills to draw a little animated paperclip but I would if I could!  It looks like you’re trying to make a bingo card.  Need help?)

Give your customers a “credit” score.  I have a particular heuristic which segments users into four buckets.  It isn’t exactly FICO, but it does successfully predict conversion rates: they range from 10% in bucket A to 0.01% in bucket D.  Bucket C is interesting, though — they convert some of the time, but don’t seem to be getting quite the value out of Bingo Card Creator that Bucket A does.

I wonder if Bucket C would feel differently if they got a $5 coupon in the email.

Meanwhile, it looks like Bucket D isn’t very interested in paying me money under any circumstances, but if I had a scratch-my-back-to-get-it-free option, I could place it prominently on their dashboards.

Long Cycles Mean Low Innovation.  Short Cycles Mean Fast Innovation.

This sort of thing is very difficult to do with desktop apps, because you can’t reliably collect data on what approaches work, and you have the build/test/deploy/distribute cycle to worry about.  It takes months for a new version of the desktop application to hit more than half of my users, and I give out upgrades for free.

By comparison, I can literally have an A/B test coded, tested, and deployed globally in under a minute, for ones which are fairly low impact.  Relocating a button, for example, requires two lines of code, a SVN commit, and a quick server restart.  I start getting data immediately.  By comparison, doing that on my desktop app would require 15 minutes of building, then waiting weeks while the new trials percolated from my website to the various download sites, and probably unforseen issues on Mac OS X 10.4 because apparently in a past life I must have stepped on Pharaoh Jobs’ cat.

Recently, a desktop developer’s mailing list that I’m on commented that a release weekly development cycle is unsustainable, bordering on suicide.  As a desktop developer, I agree, it would break me.  As a web application developer — I have released 67 updates to Bingo Card Creator in the past 7 weeks, and this isn’t even my day job.  A button here, some textual edits there, seven A/B tests, etc etc, and pretty soon you’re looking at the magic of compounding 1% improvements.

Speaking of Magic

I love desktop applications.  I prefer them to web apps almost any chance I get. You can keep your Google Docs, Excel is superior in almost every way.

As a developer, I love getting permanent presence right in front of the user (on their desktop, naturally).

My customers love desktop applications.  They love the “physicality”.  They love the perceived security (the number of people who purchased backup CDs and then proceeded to only use the webapp is downright distressing to me).  They love that the application has first-class OS support, feels native, copies and pastes right, works with double clicking files, etc etc.

But at the end of the day, I’m an engineer.  I follow the numbers where they lead me.  The numbers say that sales in this August were 60% over those of last August, despite a major blowup with Google that should have cost me dearly.  All of my attempts to distill wisdom from the statistics have lead to one conclusion: the cumulative advantages of the web application, in my advertising, in my on-site experience, within the application, within my development process, and within my purchase funnel are just stupendously superior to the desktop app.

I’m sorry, desktop apps.  We had good times together, but we’re through.

[Edit to add: I’m going to continue supporting all customers of Bingo Card Creator, regardless of how they choose to get it. The next major release will almost certainly be its last. The webapp, and my future webapps, seem to be much better investments.]

Thanks to the Ruby on Rails site, Rails Inside, and everyone else who has so generously written about A/Bingo, the Ruby on Rails split testing framework.  Memo to self: next time, incorporating a pun into the name is fine, but incorporating a pun more limiting than the actual feature set is perhaps not a good idea.

If you have any requests or questions about A/Bingo, you can either leave a comment or drop me an email.  patrick@ either of my domains should work fine.

[Edited to add: Thanks for your interest!  The position has been filled.  If you’re interested in hearing from me the next time I need somebody and/or know of an opportunity, my email is always open.]

I’m again in the market for some freelance talent, and before I go to one of the body shopping sites and get inundated with proposals by folks offering to clone eBay for $25, I thought I would put the word out here.

The Project

Bingo Card Creator, my small software business, is intensely seasonal.  Many of my customers want to play, for example, Halloween bingo or Christmas bingo with the software.  It is in my interest to make it obvious to customers that that is possible, so I am developing a total of 13 mini-sites which pitch various holiday bingo activities.  The sites generally include bingo cards for the particular lesson, instructions for how to play bingo, suggestions on how to incorporate them into the lesson plan, and what have you.

What I Have

• Two reference implementations for the type of writing I am going for.  (See above.)
• The pictures, PDFs, and other resources you’ll need to include on the sites.
• WordPress installed and an administration account configured on all sites.
• Designs for the sites — this is not a coding project.
• A brief one-page style guide.

What I Need The Freelancer To Do

• Five pages per site describing how to play <pick the holiday> bingo and how Bingo Card Creator fits into that.  This is on the order of 500 to 800 words per page, plus associated formatting.  I will provide the 5 topics per each site.
• Basic WordPress configuration: turn comments off, configure sidebar, copy/paste a line into Google Analytics plugin, set a particular page as the front page, etc.
• Most importantly: the writing needs to be warm, inviting, and grammatically flawless.  I sell to teachers.  They notice these things.

Business Considerations

• I am willing to pay $100 per site completed, via either Paypal or (if you live within the United States) check.  I have been known in the past to award bonuses for exceptional work.
• I have previously paid in excess of $2,000 to freelancers without incident.  If you wish, I can provide references that I am a very no-nonsense client who provides clear direction, does not micromanage, and pays scarily promptly.
• I am flexible on the timeframe for the project, but my sense of things is that each site is a few hours work, and that someone working very part-time on the project would complete it in about two months.

I would prefer working with a single person for this project but am willing to split it up if I have multiple qualified applicants.  The ideal applicant would have substantial experience with expectations in American elementary schools, as either a parent or teacher, but I’m willing to consider anyone.

If this sounds like something you’d be interested in, please mail me at (my first name) @bingocardcreator.com  , and include a short one-paragraph explanation why you think you’d be a good fit.  If you aren’t personally interested in this, but know somebody who it would be a good fit for, please feel free to tell them about it.

I look forward to working with you on a fairly fun and mutually rewarding project.  Thanks!

I finally got serious about putting a blog on the Bingo Card Creator site today.  Since Bingo Card Creator is powered by Rails, ideally I’d be using a Rails blog platform, but frankly, they’re all vastly inferior to WordPress.  I could have slapped up a WordPress blog on a subdomain and called it a day, but I always advise people to put blogs on subdirectories for SEO reasons, so I thought I’d do it properly.

It is easy to bork this setup if you don’t understand WordPress and Nginx configuration files.  Make backups of all server config before you start modifying them.

Extracting a WordPress Theme from your Rails App

If you’re not a PHP-speaking web designer, this step can take a lot of time.  Or, alternatively, you can just pay $10 and use ThemesPress.  They have a web-app where you copy/paste in some HTML and CSS and they spit out a widget-enabled WordPress theme for you for $10.  This is well worth your time.

1. Look at your main CSS file.  Identify any images which have relative URLs.  Replace them with absolute URLs.  This will save you from having to upload the images to ThemesPress to see the preview, and besides, you should be doing this anyhow for static assets so that you can serve them from images{1,2,3}.example.com and speed up page loading.
2. Load the main page for your site.  Click view source.  Strip out the place where Rails loads your CSS.  Replace your sidebar contents with {SIDEBAR}.  Replace your main body contents with {CONTENT}.  Replace any relative URLs to images with absolute URLs as in step #1.  Save this file for a second.
3. Go over to ThemePress, skip the image uploading step, and follow the onscreen directions.
4. Pay them their $10.  Bwahaha, you now have a WordPress theme for your Rails app.

Set WordPress Up At A Private URL

1. I find the easiest way to do this is to create a new subdomain (blog.example.com, for example) and have Apache listen to requests on a non-standard port.  If you’re not paranoid about other people seeing your work in progress, you can put it on port 80 like usual.  I suggest not using a private IP at this stage, as you’ll find it difficult to diagnose problems in your browser until you get to the Nginx config step.
2. Install WordPress like you normally would.
3. Verify that the theme you created works well, make a post, etc.
4. Go to WordPress Settings and change the Blog URL to be your final blog URL.  For example, http://www.example.com/blog rather than blog.example.com.  That URL won’t work right now.  Don’t worry, we’ll fix it.

Tweak Nginx Settings So That It Reverse Proxies Your Blog

1. I could discuss the rationale for these settings a bit more, but they are a bit quirky.  If you copy/paste them, they should mostly work.  Good enough, right?
2. Restart Nginx.
3. Verify that you can access your main blog page and your admin page.  Do not proceed without doing this.

Change Your WordPress Settings

• WordPress has two URLs listed in settings, one WordPress URL and one Blog URL.  Your WordPress URL still reads something like blog.example.com.  Since Nginx is now set up properly, go ahead and change that to http://www.example.com/blog/ , like your blog URL.
• You now have NO need to publicly access the WordPress server, since the only user who will ever see it is Nginx, while proxying for you.  You can go ahead and move it to a private IP.  This will mean Google doesn’t come along and access blog.example.com by accident, then think “Hey, he is stealing content from himself.

There you have it, one blog on your Rails app!  Now start writing.

P.S. Don’t forget to tweak your Rails templates to, you know, link to the blog like a first class citizen of your website.

… you have to phone Chase in the middle of the night to say the $600 charge to GoDaddy was authorized and it was a very good deal, too, thank you very much.

A few dozen domains times two year renewal adds up really, really quickly…  (I actually paid GoDaddy $160 to become a member of their buyer’s club for two years, which knocks some money off the cost of domains.  In particular, for whatever reason, .net domains get a 53% discount off the list price.  Considering I own over a dozen of the suckers and they’re normally fairly expensive, that was enough to justify the membership by itself.)

Which reminds me — I really need to get some freelancers to help me fill these things out.

I passed a bit of a milestone yesterday: I’ve sold $50,000 of Bingo Card Creator.

It is an arbitrary and meaningless milestone, but it put a smile on my face nonetheless.  I’m cautiously optimistic that the next $50,000 won’t take 3 years.  (Profits before taxes are somewhere in the $27,000 range if I’ve managed to tally all my receipts correctly — always unlikely until tax time rolls around.)

Wish me luck — the school year sales started this week and it appears that I will be moving out of the summer doldrums in style.  The online version is converting very well (1.7% of all trials, including a large portion of trials that wouldn’t reasonably have converted yet because it is too early).  My new ad campaigns are posting my highest conversion rates ever, courtesy of the online version being easy to sign up for.  My first newsletter, sent yesterday via Mailchimp, resulted in 2 sales ($60) for about $2 worth of email postage.

All in all, things are going pretty well business-wise.  More updates as they happen.