Kalzumeus Software

[日本の方へ：読者が日本語版を翻訳してくださいました。ご参照してください。]

I run a small software business in central Japan.  Over the years, I’ve worked both in the local Japanese government (as a translator) and in Japanese industry (as a systems engineer), and have some minor knowledge of how things are done here.  English-language reporting on the matter has been so bad that my mother is worried for my safety, so in the interests of clearing the air I thought I would write up a bit of what I know.

A Quick Primer On Japanese Geography

Japan is an archipelago made up of many islands, of which there are four main ones: Honshu, Shikoku, Hokkaido, and Kyushu.  The one that almost everybody outside of the country will think of when they think “Japan” is Honshu: in addition to housing Tokyo, Nagoya, Osaka, Kyoto, and virtually every other city that foreigners have heard of, it has most of Japan’s population and economic base.  Honshu is the big island that looks like a banana on your globe, and was directly affected by the earthquake and tsunami…

… to an extent, anyway.  See, the thing that people don’t realize is that Honshu is massive. It is larger than Great Britain.  (A country which does not typically refer to itself as a “tiny island nation.”)  At about 800 miles long, it stretches from roughly Chicago to New Orleans.  Quite a lot of the reporting on Japan, including that which is scaring the heck out of my friends and family, is the equivalent of someone ringing up Mayor Daley during Katrina and saying “My God man, that’s terrible — how are you coping?”

The public perception of Japan, at home and abroad, is disproportionately influenced by Tokyo’s outsized contribution to Japanese political, economic, and social life.  It also gets more news coverage than warranted because one could poll every journalist in North America and not find one single soul who could put Miyagi or Gifu on a map.  So let’s get this out of the way: Tokyo, like virtually the whole island of Honshu, got a bit shaken and no major damage was done.  They have reported 1 fatality caused by the earthquake.  By comparison, on any given Friday, Tokyo will typically have more deaths caused by traffic accidents.  (Tokyo is also massive.)

Miyagi is the prefecture hardest hit by the tsunami, and Japanese TV is reporting that they expect fatalities in the prefecture to exceed 10,000.  Miyagi is 200 miles from Tokyo.  (Remember — Honshu is massive.)  That’s about the distance between New York and Washington DC.

Japanese Disaster Preparedness

Japan is exceptionally well-prepared to deal with natural disasters: it has spent more on the problem than any other nation, largely as a result of frequently experiencing them.  (Have you ever wondered why you use Japanese for “tsunamis” and “typhoons”?)  All levels of the government, from the Self Defense Forces to technical translators working at prefectural technology incubators in places you’ve never heard of, spend quite a bit of time writing and drilling on what to do in the event of a disaster.

For your reference, as approximately the lowest person on the org chart for Ogaki City (it’s in Gifu, which is fairly close to Nagoya, which is 200 miles from Tokyo, which is 200 miles from Miyagi, which was severely affected by the earthquake), my duties in the event of a disaster were:

• Ascertain my personal safety.
• Report to the next person on the phone tree for my office, which we drilled once a year.
• Await mobalization in case response efforts required English or Spanish translation.

Ogaki has approximately 150,000 people.  The city’s disaster preparedness plan lists exactly how many come from English-speaking countries.  It is less than two dozen.  Why have a maintained list of English translators at the ready?  Because Japanese does not have a word for excessive preparation.

Another anecdote: I previously worked as a systems engineer for a large computer consultancy, primarily in making back office systems for Japanese universities.  One such system is called a portal: it lets students check on, e.g., their class schedule from their cell phones.

The first feature of the portal, printed in bold red ink and obsessively tested, was called Emergency Notification.  Basically, we were worried about you attempting to check your class schedule while there was a wall of water coming to inundate your campus, so we built in the capability to take over all pages and say, essentially, “Forget about class.  Get to shelter now.”

Many of our clients are in the general vicinity of Tokyo.  When Nagoya (again, same island but very far away) started shaking during the earthquake, here’s what happened:

1. T-0 seconds: Oh dear, we’re shaking.
2. T+5 seconds: Where was that earthquake?
3. T+15 seconds: The government reports that we just had a magnitude 8.8 earthquake off the coast of East Japan.  Which clients of ours are implicated?
4. T+30 seconds: Two or three engineers in the office start saying “I’m the senior engineer responsible for X, Y, and Z universities.”
5. T+45 seconds: “I am unable to reach X University’s emergency contact on the phone.  Retrying.”  (Phones were inundated virtually instantly.)
6. T+60 seconds: “I am unable to reach X University’s emergency contact on the phone.  I am declaring an emergency for X University.  I am now going to follow the X University Emergency Checklist.”
7. T+90 seconds: “I have activated emergency systems for X University remotely.  Confirm activation of emergency systems.”
8. T+95 seconds: (second most senior engineer) “I confirm activation of emergency systems for X University.”
9. T+120 seconds: (manager of group)  “Confirming emergency system activations, sound off: X University.”  “Systems activated.”  “Confirmed systems activated.”  “Y University.”  “Systems activated.”  “Confirmed systems activated.” …

While this is happening, it’s somebody else’s job to confirm the safety of the colleagues of these engineers, at least a few of whom are out of the office at client sites.  Their checklist helpfully notes that confirmation of the safety of engineers should be done by visual inspection first, because they’ll be really effing busy for the next few minutes.

So that’s the view of the disaster from the perspective of a wee little office several hundred miles away, responsible for a system which, in the scheme of things, was of very, very minor importance.

Scenes like this started playing out up and down Japan within, literally, seconds of the quake.

When the mall I was in started shaking, I at first thought it was because it was a windy day (Japanese buildings are designed to shake because the alternative is to be designed to fail catastrophically in the event of an earthquake), until I looked out the window and saw the train station.  A train pulling out of the station had hit the emergency breaks and was stopped within 20 feet — again, just someone doing what he was trained for.  A few seconds after the train stopped, after reporting his status, he would have gotten on the loudspeakers and apologized for inconvenience caused by the earthquake.  (Seriously, it’s in the manual.)

Everything Pretty Much Worked

Let’s talk about trains for a second.  Four One of them were washed away by the tsunami. All Japanese trains survived the tsunami without incident. [Edited to add: Initial reports were incorrect.  Contact was initially lost with 5 trains, but all passengers and crew were rescued.  See here, in Japanese.]  All of the rest — including ones travelling in excess of 150 miles per hour — made immediate emergency stops and no one died.  There were no derailments.  There were no collisions.  There was no loss of control.  The story of Japanese railways during the earthquake and tsunami is the story of an unceasing drumbeat of everything going right.

This was largely the story up and down Honshu.  Planes stayed in the sky.  Buildings stayed standing.  Civil order continued uninterrupted.

On the train line between Ogaki and Nagoya, one passes dozens of factories, including notably a beer distillery which holds beer in pressure tanks painted to look like gigantic beer bottles.  Many of these factories have large amounts of extraordinarily dangerous chemicals maintained, at all times, in conditions which would resemble fuel-air bombs if they had a trigger attached to them.  None of them blew up.  There was a handful of very photogenic failures out east, which is an occupational hazard of dealing with large quantities of things that have a strongly adversarial response to materials like oxygen, water, and chemists.  We’re not going to stop doing that because modern civilization and it’s luxuries like cars, medicine, and food are dependent on industry.

The overwhelming response of Japanese engineering to the challenge posed by an earthquake larger than any in the last century was to function exactly as designed.  Millions of people are alive right now because the system worked and the system worked and the system worked.

That this happened was, I say with no hint of exaggeration, one of the triumphs of human civilization.  Every engineer in this country should be walking a little taller this week.  We can’t say that too loudly, because it would be inappropriate with folks still missing and many families in mourning, but it doesn’t make it any less true.

Let’s Talk Nukes

There is currently a lot of panicked reporting about the problems with two of Tokyo Electric’s nuclear power generation plants in Fukushima.  Although few people would admit this out loud, I think it would be fair to include these in the count of systems which functioned exactly as designed.  For more detail on this from someone who knows nuclear power generation, which rules out him being a reporter, see here.

• The instant response — scramming the reactors — happened exactly as planned and, instantly, removed the Apocalyptic Nightmare Scenarios from the table.
• There were some failures of important systems, mostly related to cooling the reactor cores to prevent a meltdown.  To be clear, a meltdown is not an Apocalyptic Nightmare Scenario: the entire plant is designed such that when everything else fails, the worst thing that happens is somebody gets a cleanup bill with a whole lot of zeroes in it.
• Failure of the systems is contemplated in their design, which is why there are so many redundant ones.  You won’t even hear about most of the failures up and down the country because a) they weren’t nuclear related (a keyword which scares the heck out of some people) and b) redundant systems caught them.
• The tremendous public unease over nuclear power shouldn’t be allowed to overpower the conclusion: nuclear energy, in all the years leading to the crisis and continuing during it, is absurdly safe.  Remember the talk about the trains and how they did exactly what they were supposed to do within seconds?  Several hundred people still drowned on the trains.  That is a tragedy, but every person connected with the design and operation of the railways should be justifiably proud that that was the worst thing that happened.  At present, in terms of radiation risk, the tsunami appears to be a wash: on the one hand there’s a near nuclear meltdown, on the other hand the tsunami disrupted something really dangerous: international flights.  (One does not ordinarily associate flying commercial airlines with elevated radiation risks.  Then again, one doesn’t normally associate eating bananas with it, either.  When you hear news reports of people exposed to radiation, keep in mind, at the moment we’re talking a level of severity somewhere between “ate a banana” and “carries a Delta Skymiles platinum membership card”.)

What You Can Do

Far and away the worst  thing that happened in the earthquake was that a lot of people drowned.  Your thoughts and prayers for them and their families are appreciated.  This is terrible, and we’ll learn ways to better avoid it in the future, but considering the magnitude of the disaster we got off relatively lightly.  (An earlier draft of this post said “lucky.”  I have since reworded because, honestly, screw luck.  Luck had absolutely nothing to do with it.  Decades of good engineering, planning, and following the bloody checklist are why this was a serious disaster and not a nation-ending catastrophe like it would have been in many, many other places.)

Japan’s economy just got a serious monkey wrench thrown into it, but it will be back up to speed fairly quickly.  (By comparison, it was probably more hurt by either the Leiman Shock or the decision to invent a safety crisis to help out the US auto industry.  By the way, wondering what you can do for Japan?  Take whatever you’re saying currently about “We’re all Japanese”, hold onto it for a few years, and copy it into a strongly worded letter to your local Congresscritter the next time nativism runs rampant.)

A few friends of mine have suggested coming to Japan to pitch in with the recovery efforts.  I appreciate your willingness to brave the radiological dangers of international travel on our behalf, but that plan has little upside to it: when you get here, you’re going to be a) illiterate b) unable to understand instructions and c) a productivity drag on people who are quite capable of dealing with this but will instead have to play Babysit The Foreigner.  If you’re feeling compassionate and want to do something for the sake of doing something, find a charity in your neighborhood.  Give it money.  Tell them you were motivated to by Japan’s current predicament.  You’ll be happy, Japan will recover quickly, and your local charity will appreciate your kindness.

On behalf of myself and the other folks in our community, thank you for your kindness and support.

[本投稿を日本語にすると思っておりますが、より早くできる方がいましたら、ご自由にどうぞ。翻訳を含めて二次的著作物を許可いたします。詳細はこちらまで。

This post is released under a Creative Commons license.  I intend to translate it into Japanese over the next few days, but if you want to translate it or otherwise use it, please feel free.]

[Edit: Due to overwhelming volume and a poor signal-to-noise ratio, I am closing comments on this post, but I encourage you to blog about it if you feel strongly about something.]

Apologies for not posting this earlier — I put notices on my business websites but forgot that a lot of folks know me solely through the blog:

• I live in Gifu, which is quite far from the earthquake epicenter.  We got shaken up a bit, but no permanent damage was done.  We’re landlocked so, unless the mountains fall into the sea, tsunamis are not an issue for us.
• The people I’m close to in Japan are all OK.
• We really appreciate your expressions of concern and prayers.
• If you are wondering “What can I do?”, every day is a good day for charity.  I recommend the Red Cross or your local favorite charity.  In particular, disaster relief charities will use money collected today to help the folks affected by the next major incident, and it is highly probable that they are less well-situated than Japan is — we’re probably as well-prepared as anybody could be.

Thanks as always.  We’ll pull through this, don’t worry.

Regards,

Patrick

Last week, I had an opportunity to talk with Andy Brice, who sells software for wedding seating plans and the like.  He is an absolute genius with AdWords, and gave me some ideas on ways to improve my performance.  I immediately started to implement them, full with the excitement of a new project and wondering why I don’t spend more time optimizing AdWords.

Oh, right.

There were another 15 ads which I added last Friday-ish and are still Under Review.  Under Review is Google-speak for “We aren’t sure that this ad complies with our policies yet.”  While an ad is Under Review, it doesn’t show anywhere, and you aren’t learning anything by having it.

Dealing With Shades of Grey

Google has a variety of businesses which it does not want to or legally cannot do business with.  To prevent them from using AdWords, they exercise prior restraint on AdWords copy, not letting their ads run until a human at Google has approved them.

One of the businesses that Google doesn’t want advertising (in the US, at any rate) is gambling.  Bingo is a form of gambling.  Bingo Card Creator is not a form of gambling — it is a form of software which helps elementary schoolkids learn to read.  This makes it rather hard to write focused, relevant advertisements responsive to customer queries like [how do I make a US presidents bingo card] which sell Bingo Card Creator without using the word “bingo” anywhere.

Google is, to all appearances, just using a keyword-based blacklist.  I guess all the eats-Bayesian-classifiers-for-lunch PhDs work in search and Gmail spam filtering, where they’ve clearly got an aptitude for understanding that words can have multiple meanings.  OK, fine, but at least the remaining boffins can do a blacklist correctly?

Well, not so much.

• Using Google’s Copy Ad feature to copy an ad, word for word, between ad groups will cause the new copy to go back into review purgatory.  This is despite that theoretically being a content-neutral action and a core task for advertisers, because many flavors of AdWords optimization rely on keywords being partitioned correctly into focused ad groups.
• Changing so much as a character of the ad, including landing page URLs, will cause the ad to get flagged again.  This only affects good advertisers.  Bad advertisers can presumably figure out how to serve whatever content they want on http://example.com/approved .  Pulling a bait-and-switch is absolutely trivial, since you have full control over what your own server serves to users.  This rule only inconveniences compliant advertisers, who get thrown into review purgatory every time they e.g. try to add another tracking parameter to their landing pages, switch from http:// to https://, etc etc.  I get the feeling I’m supposed to create five copies of each ad, pointing to /lp1 … /lp5 with identical content, and then if I need to do any testing I should get crafty with redirects or what have you later.  That’s insane - it is extra work that is directly against the spirit of the rules and unlike actual compliance it works.

Scalable Communication Methods

According to Google:

We review all ads in the order they’re received, and we work to review all ads in our program as quickly as possible, usually within 1 to 2 business days or sooner.

If there were only 48 hours of lag time inserted every time I touched an AdWords ad, this would be annoying but tractable.  It would lengthen my time through the idea creation/validation loop (Lean Startup  fans know why that is a Very Bad Thing), but I could still get work done by batching all my edits together and then twiddling my thumbs for 48 hours.

Sadly, Google routinely falls short of their announced level of service.  And when I say “Falls short”, I mean “Ads can sit for weeks ‘Under Review’ and never be approved.”

This leads you to have to contact Google Customer Service to be able to get Google to give permission to give Google money.

Google Customer Service: Welcome to Kafka

The first rule of Google Customer Service is that Google does not have Customer Service.  They prefer what Chief Engineer Matt Cutts describes as “scalable communication methods”: there are like a bazillion of you, there are only a few tens of thousands of us, instead of actually speaking to a human being you should read a blog post or watch a video or talk to a machine.  It is a wonderful, scalable model… when things work.

Anything which introduces a mandatory customer service interaction with Google is a process designed for failure.  AdWords approvals requires a customer service interaction.  Catch-22, to mix literary metaphors.

The “scalable communication methods” like AdWords Help have this to say about contacting customer service with regards to ad approvals:

Our Support team won’t be able to help you expedite this process.

That is not actually a true statement (which, incidentally, describes much of AdWords Help).  Length of time from ad submission to approval is, in my experiences, unbounded (literally, weeks can go by without approval).  Length of time from complaining to Support to approval: a day or two.  The most helpful Google employee I’ve ever Internet-met (name withheld to protect him from whatever dire punishments await someone who attempts to help customers) told me that my workflow should literally be 1) Submit ad 2) Submit ticket to get ad looked at, if I persistently fell into Under Review.

Google apparently knows it, too, since they have special-cased out the CS interaction for dealing with Ad approvals:

After filling in everything, I hit Submit expecting to be taken to a page which had an “OK, now actually tell us what the problem” comment box was.  No need — it has been optimized away!  Google doesn’t even want that much interaction.  (The last time I went through this — sometime last year — I recall there being a freeform field, limited to 512 characters or so.  I always use it to explain that I am not a gambling operation and if they want confirmation they can read the AdWords case study about my business.)

Google’s computers then weighed my support request and found it wanting:

Dear Advertiser,

Thank you for your e-mail. We understand you’d like us to review your ad.
When you submit new ads or make changes to existing ads, they’re
automatically submitted for review.

We work to review all ads in our program as quickly as possible. You
should receive an email notification stating the approval status of your
ads pending review within the next 3 business days. You can view the
status of your ad any time in your account. The “Status” column in the
“Ads” tab displays information on the current state of an individual ad
variation.

For a list of Ad Approval Statuses, visit
http://adwords.google.com/support/aw/bin/answer.py?hl=en&answer=138186

We are working as quickly as possible to get everyone up and running and
should get to yours soon! If you have a different question, which doesn’t
refer to pending ad approval, please get back to us via the ‘Contact Us’
link in the Help Center at https://adwords.google.com/support/aw/?hl=en.
Be sure to choose the category that is most relevant to your question.

Sincerely,

The Google AdWords Team

Well, at least the templating engine correctly replaced $BRUSHOFF_LETTER, but in terms of customer communication:

• You asked me to put in my name… you might want to think about using it.
• As much as I appreciate your False! Enthusiasm! if the next line of your letter is going  to be Eff Off And Die then maybe you should take out the exclamation points and give them to a Ruby programmer.  (We can always use more.)
• If the original timeline was 1-2 business days and the timeline three days later is “within 3 business days”, can we update them so that they quote it consistently?  Or maybe put something like “We get to 98.2% of approvals within 3 business days.”  (Or 2.89% of approvals within 3 business days, as the case may be.)

Google’s Isolation From Market / Customer Pressure

Google theoretically values my business — I pay them $10,000 a year and would love to pay more.  Indeed, they can find my email address and have a human contact it when they want to do ad sales.  (I got an offer recently to set up a call with one of their AdWords Strategists to discuss optimization of my account… which is great, but previous experience leads me to believe he would use the same reports I have access to, make decisions with little understanding of my business, and then leave it to me to actually schedule the new ads/keywords and run headlong into Pending Review purgatory.)  But they are not doing very well lately at convincing me they actually care.  And they’ll still make a bazillion dollars without that, so no harm done.

In normal markets, I would be strongly tempted to take my business to vibrant competitive offerings.  Sadly, Google is pretty much the only game in town for viable CPC advertising: even if Microsoft/Yahoo exorcized the abominations haunting their UIs, they would not have enough inventory to matter for me in my niche (I’ve tried before).

Which leaves me with only one option: trying out my own scalable communication methods, and hoping someone in the Googleplex reads this and takes action to unbork this process (ideally, for a large class of advertisers).  It is the Internet equivalent of putting a message in a bottle and then throwing it into the ocean, but that is still an improvement on the normal channels.

YCombinator just released their semi-annual application for companies to incubate.  One of the new questions this time around is “How will you get users?”  I think that is a great question to think about for everybody in business — perhaps the great question to think about.  Customer acquisition is one of the easiest places to screw up as a startup, particularly for technical founders (who, in their previous lives, have probably never had to do it for anything).

I’m not applying to YC this time around, but I always fill out the application to force myself to talk through my business strategy.  I had one thought which sounded worthwhile enough to share: customer acquisition can be hacked.  You can take the current conventional wisdom in the market of how to get customers to use solutions, identify it’s weak points, and aggressively target them.  That can, potentially, be as important (or more important) than the same applied to the actual product.

Enemy #1: The Technology Adoption Cycle

Let’s assume that you’re capable of successfully identifying a problem customers have and solving it.  Those are both highly non-trivial, but put them out of scope for the moment: people’s hair is on fire, you’re selling fire extinguishers, life should be grand.  Life is often not quite so grand, because you can produce a wonderful product which creates value and fail to sell it to folks.

Most startups are not creating an entirely new solution out of whole cloth.  Somewhere out there people are currently experiencing the problem you are solving, and they’re dealing with it somehow.  They might be ignoring it or gritting their teeth.  They might be using some inferior solution which they got from your competitors, you have competitors (you should have competitors — if you don’t, you probably aren’t doing something people care about).

Your competitors had to see people through a product adoption cycle:

1. Identify people with the problem
2. Teach them that the solution exists
3. Successfully sell them on the solution
4. Prevent them from leaving the solution for a competing solution

In actual practice, this adoption cycle is frequently long and arduous.  (If it were short and easy, there wouldn’t be any money in it.)

Your competitors, if they are established businesses, are probably very good at maneuvering customers through the technology adoption cycle as it exists in the market today.  For example, if grading students is a problem, your competitor might very well be successfully selling school districts on their gigantic consultingware grading solution which costs six figures an installation.  Since they can still make the rent and keep the lights on, you can infer that their business probably works.   Their marketing team is generating sufficient leads, their sales team converts some of them.

But you probably don’t want to do what they’re doing, because they’re better at being them than you are.

Hacking The Product Adoption Cycle

Startups are not the world’s most obvious choice of employment for people who enjoy coloring between the lines.  If you execute the competition’s playbook for acquiring customers, you are probably going to get crushed by them, because

• they know more about the market than you do
• they have a commanding head start
• they have large amounts of resources to throw at the problem

On the other hand, it is entirely possible that:

• they have stopped learning about the market
• they have a commanding head start running in a suboptimal direction
• they have large amounts of resources which, for reasons of switching costs and politics, can’t be reallocated to more efficient approaches

These statements aren’t just true about the product — sure, they might have a crufty old VB6 app and you have the new Node.js hotness.  They are equally true about the customer acquisition process.  You’re competing with their business, not with their product, so you could possibly either focus your innovation on customer acquisition or, more likely, use innovation on both customer acquisition and product in a mutually supportive manner.

Examples Of Hacking This

Freemium isn’t a business model so much as it is a customer acquisition tactic.  In markets dominated by expensive solutions with huge switching costs and uncertainty about success with technology changes, freemium can be very compelling: the self-serve model allows you to do less consultative sales (with the multi-month purchase cycles, large sales teams, and politicking that entails) and instead focus your efforts on getting leads and converting them.  This plays to a very different skill set versus traditional enterprise B2B sales, and it is a much more forgiving of small teams, since you’re deputizing your free users as internal sales champions and praying that they can do the consultative sales that your non-existent sales force isn’t doing.  This also lets you crack into markets where any model which requires consultative sales automatically is priced out of the market — essentially, anything where customer lifetime value is less than $75,000, give or take.

Monthly billing is another hack.  Customers are irrational and their processes are broken.  One artifact of those practices is that there is a stepwise increase in difficulty if prices increase by $1, as long as the price was already at whatever the company’s magic number is for maximum to be put on a corporate credit card or signed for on a non-manager’s authority.    Monthly billing defeats this step function because even if the total lifetime cost of the solution goes up the largest amount ever billed at once might well cross under that critical threshhold again.  This means that there is no longer a total no-man’s land between $1,000 and $75,000 in lifetime value. (Is this a hack?  Yes.  If you bill a Fortune 500 company product manager $80 a month, you are essentially conspiring with him against his accounting controls.  Not that there is anything wrong with that.  You can even explicitly sell that as a benefit to him, just like you sell SaaS as allowing him to avoid having to talk to IT to get the stuff he needs to do his job.)

Online marketing expertise hacks through the ridiculous inefficiencies of offline marketing.  Many startups can run rings around their traditional competitors in online marketing, for example due to savvier SEO that leverages their strengths in execution speed, technological savvy, and community ties.  For example, my wee little business competes directly with Scholastic Publishing, who has 10,000 employees and access to public capital markets.  They also couldn’t spell SEO if you handed them a set of alphabet flash cards, which is good news for me.  You would think that “Well, a business which doesn’t have online marketing expertise could just hire for it”, but after you get past the level of “let’s make a website — it should probably have title tags and some of those keywords, too”, everyone who tries this finds that it is murderously difficult to hire competent SEOs right now.  (If you disagree, I have some clients who would love to meet you.)  At the same time, I couldn’t possibly compete with the relationships which get their competing products on shelves at tens of thousands of retail locations… but then again, I don’t have to pay 50 cents of every dollar of sales to the retailer, either.

Taking A Hack From Tactic To Strategy

I think this isn’t exactly a new insight.  There are lots of folks who, when asked for their marketing plan, will say “Oh, we’re going to get lots of search traffic” (indeed, that is probably second only to “it will grow virally” in terms of signaling “has probably not thought this through.”)  What separates hopes and dreams of future success from very valuable businesses is a strategy which, with execution and refinement over time, will actually achieve the goals.

We often hear products described using something like “It’s like Facebook, except for dogs.”  How about, instead, describing businesses like “It’s like Quicken, except Quicken sells primarily through boxed software channels and we’re going to sell primarily through banks which will deal with us for a cut of the sale price and the ability to deepen relations with small business customers, who consume lots of high margin services and stay locked in for decades at at time.”  (That may or may not actually be true.)

We often accept previous experience or minimal proof-of-concept prototypes/MVPs in lieu of a functioning product when evaluating whether someone is capable of executing on building something.  Why not do essentially the same for proving that one is likely to get customers?  A previous background in revenue maximization through negotiating cross selling deals for banks, or evidence that you have enthusiasm from a few bankers who like the concept and want to hear more when you have something to show, demonstrates a certain likelihood that marketing challenges will be overcome like technical challenges will be overcome.

Similarly, for a startup hoping to make inroads for SEO, I’d be thinking less along the lines of “we’ll sprinkle some SEO on our website” and more along the lines of specific plans for scalable content generation, securing backlinks at scale, and winning the support of influencers either in the niche or in other addressable niches which your competition may not be aware are relevant to that facet of their business.

Product Supports The Marketing And Vice Versa

I have a wee little heresy as an engineer: I think that you can make a perfectly viable business out of a product which is not better than competitors, solely by improving on the method of selling it.  Farmville (and whatever Zynga has reskinned it as this week), for example, is not superior to all other options for entertainment… it just beats the pants off of most of their viral spread patterns, because promoting your use of the game is the core gameplay mechanic.  (You can also do this in more socially beneficial ways than Farmville, don’t worry.  I have a competitor in the bingo business whose product is very close in quality to my product.  They sell to schools via a catalog.  I sell to teachers via a website.  Despite solving the same problem for the same end-users our businesses are like ships passing in the night.  Hilariously, at least a few of my customers actually own both pieces of software, because the people who buy from the catalogs never bothered telling the people who use the websites.)

However, this doesn’t mean you can’t innovate on both the marketing and the product.  Indeed, since they feed off each other, that is probably substantially more effective than innovating on one or the other.  Imagine what a juggernaut World of Warcraft would be if they nailed their game’s quality as much as they did and also had Zynga’s viral loop and monetization model.  That hypothetical WoW could probably deal with Chinese net regulators by buying China.

(It’s easy to say this in retrospect: empirically, millions of employed adults with lots of disposable income spend much of their free time playing WoW.  They spend huge amounts of money on buying status for themselves — cars, diamonds, big houses.  They clearly value their experience in the game.  Therefore, they should be willing to buy status in the game, too… and since buying status is more being seen as having paid lots of money than it is about any particular artifact received, this should go over very well.  I mean, crikey, in a world where encrusted mollusk discharges say “I love you” anything is possible… anyhow, it is easy to say that in hindsight.  The challenge for startups is identifying that sort of synergy between customer adoption and the product in advance, and communicating that it is likely enough to happen to risk betting on.)

Hacking A Non-Computer System Whose Source Is Closed And Updated Continuously

We all know the first iteration of the product is going to suck (hopefully in the sense of “not meet customers needs” more than “a broken, unreliable mess”).  The first iteration of the marketing strategy is also going to suck (hopefully in the sense of “fail to generate the expected level of success” rather than “like shouting to an audience of deaf ants during a hurricane”).  Just like you can use the Lean Startup principles to modify your product and marketing message such that it comes closer to achieving a match with what some people actually need, you can also use spiritually similar disciplines to iterate on customer acquisition strategies.  There is as large a solution space in them as there is in the product space.  Maybe you need to try SEO and see that it doesn’t do a great job in your market, for your customers, while an affiliate channel performs better.  If you’re experimenting, measuring, and moving with a purpose as opposed to the traditional method of “throw stuff at the wall and see what sticks”, you will hopefully have a bit of success.

I’d love to hear if you have comments.

[Memo to self: Prior to ever actually applying for YC, I should practice thinking big thoughts and then writing small thoughts.  Those form fields are tiny!]

“Your single best investment is your own education.” “The average new graduate is drowning in $22,000 in debt.” “A degree in English is just as valuable as a degree in Biology — it teaches you critical thinking!” “Follow your dreams and you’ll find financial success whatever your major is!”

You’ve probably heard a thousand pieces of educational/career advice like these, and if your family/friends/advisors are anything like mine they’ve virtually never been substantiated by data.  That is a shame, because choosing a course of study is one of the largest transactions you’ll ever make — the sticker price at my alma mater was close to $140,000 and prices nationwide have risen faster than inflation for virtually a generation now.  We have the Blue Book to tell you what a ten year old beater is worth down to the dollar, there are entire industries devoted to assessing every type of security to determine their valuation, and the closest thing most students have to insight on the degree selection process is getting advice from Aunt Sue.  This is insane.

Information Asymmetry In Employment Outcomes

Any college could rectify this situation virtually overnight: take that lovely list of alumni that they obsessively curate for squeezing donations, send out a two question survey (“What did you major in?” and “What was your salary this year?”), and give a sociology grad student a bowl of ramen to do some very simple number crunching.  No college will actually do this because transparency goes directly against their interests: if all degrees from a particular institution are valued at “An uncertain, but certainly large and roughly constant number”, then the standard practice of pricing them all identically makes sense.  If not, it is the academic equivalent of pricing stocks by length of ticker symbol.

(I understand many folks enjoy the non-economic component of their education.  I did and do, too, but since I’ve never spent $120,000 and signed myself up for a decade of debt for the sheer enrichment offered by attending a ballet or reading about the Japanese economy I can only conclude that I don’t value it anywhere near what I paid for my degrees.  Your objection to the narrowness of this study is duly noted, though.)

You might assume that the government would track this, somewhere, but you’d be wrong.  The Census Bureau produces a report every ten years tying level of education (associate’s degree, bachelor’s degree, master’s degree, etc) to salary, which invariably produces the result “More education is better”, but they don’t answer very interesting questions like “Is a bachelor’s degree in computer science better than a master’s in elementary education?” or “Are there fields with a sharply diminishing return to additional education?”

However, the Bureau of Labor Statistics does very comprehensive work in administering a National Compensation Survey, which gathers huge amounts of raw data about employment hours and wages broken down by region, job (over 800 classifications available, from CEOs to ship loaders), and a few other axes.  They use this and other studies to produce a variety of government reports, such as the Occupational Outlook Handbook, which does a good job of providing per-career advice but probably intentionally omits comparisons between careers.

The Bureau of Labor Statistics also makes the data from the NCS available for download on their website.  It is hefty — over a gigabyte of plain text — but it contains a virtual treasure trove of value… if you just know how to read the map.

Liberating Conclusions From Open Data

Recently, a big buzzword in the tech community has been Open Data: the notion that the huge, monstrous streams of raw facts collected by the government can be exploited for our benefit if they are merely shared.  I think this is mostly true: the best single example I’ve heard of is that since your local health authorities inspect every restaurant’s premises as a matter of course they must know where they all are located, and therefore one should be able to get those locations from them and jumpstart the creation of a guide to local restaurants without having to find every one by yourself (a monumental undertaking).

However, raw facts are uninteresting.  Here’s a line from the BLS data:

NWU009910010200000000000016260,2008,M07,69.71

Scintillating stuff, right?  What we are really interested in is what those facts can teach us — in particular, what can they teach us that allows us to make decisions such as what to major in.  This is where your friendly neighborhood computer programmer comes in: with a bit of elbow grease and a laptop, you can reduce the 856,000-odd facts in the government’s salary data to some useful conclusions about college majors.

A Bit Of Science And A Dash of Art

Sadly, there are limitations to what we can accomplish with the BLS data: it groups salary data by occupation rather than by major and degree level.  The BLS separately identifies for each occupation what the probable degree level is, but going from occupation to degree requires a bit of guesswork.  Rather than associating all occupations with sets of related degrees by hand, and injecting my own biases into the analysis, I decided to crowd source the problem: I paid Mechanical Turk workers for their two cents (quite literally) on what degree an e.g. elementary school teacher likely had.  This produced answers like Education, Early Childhood Education, Teaching, English, etc.

I then used an arbitrary level of consensus as a cutoff, and was able to pair ~60 very popular degrees (Computer Science, English, etc) and ~250 less common ones (Vocational Education, Media Technology, etc) with associated occupations.  Some additional number crunching let me construct rough estimate salary-versus-age curves for the occupations, which could then be reduced into a simple net present value calculation.  Long story short: a lot of data, a bit of science, and a dollop of absolute voodoo — it’s sort of like most social science, except I’m going to be honest about the voodoo upfront.

After doing the calculations I used Ruby on Rails and some open source graphing libraries to present the results in a comprehensible, searchable fashion — similar to the data visualizations done by the New York Times, which are some of the best work they produce.  (Check this one on the geography of the recession for the general feel.)

Why Go To All This Trouble?

Short story: Intellectual interest plus a nice paycheck.

Longer story: I do very occasional consulting work for a variety of clients.  In case you haven’t noticed from the six-figure sticker prices, offering degrees is a very big business.  Any flowing river of cash that large attracts, as if by magic, a variety of service providers around it.  In education, one major problem colleges have is finding prospective customers to sell degrees to.  This is hardly a unique problem for businesses.  (Colleges may prefer to phrase this as “students” to “award” degrees to, because they are intellectually committed to a view of themselves as custodians of the lamp of human knowledge rather than rapacious money-grubbing institutions.  I don’t know of any reason they can’t be both.)

One thing colleges — from the Ivies to state schools to online for-profit institutions — spend absolutely gobsmacking amounts of money on is “lead generation”.  Basically, since a percentage of applicants will eventually matriculate (and pay five or six figures for the privilege), when a qualified prospect fills out an application that is an economically beneficial event.  You can compare this to a conversion to the free trial of a web service.  Universities are willing to pay quite a bit of money if you can induce someone to apply: the payout varies by university and agreement, but payments in the $10+ region just for requesting an application packet are not uncommon.  (And if you had some magic way of generating sought after candidates — say, highly qualified African American students — you could almost certainly negotiate much, much higher payouts.  There might still be some Marxists on the faculty but it is all capitalists in the administrations.)

Anyhow, with universities offering to pay for lead generation, there is an entire value chain created from the ether: sites to capture the leads, affiliate programs to direct folks to the lead capture sites, advertising to attract visitors to the affiliates, publishers to create content which displays advertising, etc.  One of the publishers in the industry, Online Degrees, hired me with an open brief: make something compelling for our website.  I thought since universities, academics, and the government have failed to produce any actionable data on which degrees make sense to go after, I could do some independent research on the subject.  Online Degrees.org will host it on their website, and in the course of providing value to potential students researching the subject, they’ll have an opportunity to display ads for degree programs.

You might be concerned about the impartiality of this.  I don’t blame you.  I’ve got no particular dog in this fight: I get paid by the hour no matter what degree wins.  (Cards on the table: I have degrees in Computer Science — which is in the data set — and East Asian Studies, which is not.)

Online Degrees.org obviously has a vested interest in convincing you that a having a degree is better than not having one, but they’re agnostic about which one you apply for.  Indeed, they’d love to tell you which fields are better than others because somebody in the industry needs to have the credibility to say that e.g. culinary school is tantamount to grand theft (and most of the victims take out loans for the privilege of going through it).

Besides, do you really have a better alternative?  If I had a PhD in Sociology, would that make me a less biased source of information on the desirability of becoming a cheap source of exploitable labor a master’s candidate in Sociology?

Anyhow, I have been intellectually interested in this subject for several years now.

Quick Overview Of Results

For the results of most particular interest to you, take a gander at the degree value calculator.

Regular readers of this blog are mostly technologists of one flavor or another, and degrees in technical majors do very, very well.  Computer Science and Computer Engineering are near the top among all options for bachelor’s degrees.  It is narrowly bested by a handful of degrees tailored around resource extraction: for example, if you study Geology, Big Oil will apparently pay you Big Bucks.

Hard sciences such as Physics and Biology pay rather less well than I would have expected.  Degrees in the humanities perform about as poorly as people often joke.  The largest surprise to me was that degrees, even advanced degrees, in some caring professions (like Social Work) are apparently terrible options.  Looking at the underlying data suggests that this because many social workers do it as a part time job.  (That is a recurring theme among many jobs that I expect people would classify as more likely to be female than the typical occupation.  Food for thought the next time someone brings up the wage gap.)

You can see the results of this research on their website.  [Edit as of 2/19/2013: You’ll have to search for this directly, due to link rot.]

Questions?  Comments?  Criticisms?  I’d love to hear them.

I’m Patrick McKenzie.  For the last few years, I’ve run a small software company which, most prominently, makes Bingo Card Creator.  It creates… well, you probably get the idea.  I recently launched Appointment Reminder, which… yeah.  I also do occasional consulting, shockingly not as Pay Me Lots Of Money And I Solve Your Problems LLC.

In addition to publishing pretty much live stats about the business, I always do a year-end wrapup (see: 2006, 2007, 2008, 2009) covering my thoughts on the year.  I hope folks find it interesting or informative.

Disclaimers: Stats are accurate as of publication, but the year isn’t quite over yet.  Ordinarily the last two weeks of December are fairly slow, but I would expect there to be a few hundred dollars more sales and possibly a few hundred more in expenses, depending on the timing of people charging my credit card.

My business is a good deal more complicated now than it was previously, which changes how open I can be about some bits of it.  See below.

The Big Change

After several years working as a Japanese salaryman, I quit my day job and went full time on my business as of April 1st of this year.  This was the best decision I have ever made.  Words cannot describe how happier I am with everything about my life: I see my family more often, I see friends more often, I travel more, I work less, I make more money, I’m healthier, my apartment is cleaner (well, OK, modestly cleaner), and I enjoy work a whole lot more than I ever did when I was working for somebody else.  Self-employment is awesome.  There are occasionally challenges to it, but since I spent a few years dipping my toes into the water prior to doing it full-time, I have had very little of the “Uh oh, I might not be able to make rent” uncertainty that a lot of folks report.

Bingo Card Creator

Bingo Card Creator remains my bread and butter for the time being, but I think this is likely to be the last year for that.

Stats:

Sales: 1,422 (up 38% from last year’s 1,049)

Refunds: 20 (down from 24 last year, to 1.4% of sales from 2.3%)

Sales Net Of Refunds: $42,589.90 (up 33% from $31,855.08)

Expenses: $16,685.24 (up from $15,485.28)

Profits: $25,904.66 (up 58% from $16,369.80)

Wage per hour: $200~250ish, based on my best guess of time spent on BCC (yeah, I went “full time” and work less than ever.  This is mostly because BCC is mature software.)

Web Stats:

(All stats are from bingocardcreator.com unless otherwise specified.)

Visits: 777,000 (up from 546k)

Unique visitors: 655,000 (up from 470k)

Page views: 2.7 million (up from 1.6 million)

Traffic sources of note: Google (44%), AdWords (20%), Binghoo (11%)

Trial downloads: 21,000 (down from 56,000)

Trial signups for online version: 72,000 (up from 17,000)

Approximate online trial to purchase conversion rate: 1.7%

Narrative Version:

The major change in BCC this year was unofficially deprecating the downloadable version of the software, for a variety of reasons.  This led to a huge cut in my support investment — the downloadable version generates 10 times the work per customer that the web app version does — which helped enormously when I dropped BCC into maintenance mode, which it spent over half the year in.  (Maintenance mode means I answer questions and collect payments and that is about it.)  I did a bit of experimentation over the summer in terms of conversion funnels, and also did some major work in October with regards to seasonal promotions and using my mailing list.

My conversion rate for BCC is slipping steadily.  This is sort of ironic, because it is the result of an unalloyed good thing for the business: as I get better at getting organic search traffic, the population of people using my web site moves from the teacher-heavy target-rich-environment it has historically been and towards a broader audience who don’t have quite such a pressing need for bingo cards.  Sales go up, since converting 1% of a big number is still a good thing, but it dilutes my aggregate conversion rate.  Oh well.

I’m mildly disappointed that I missed my sales and profitability targets this year (by about $2k and $4k respectively).  Oh well.

What Went Right:

• Deprecating the downloadable version reduces my work and stress level attributable to BCC enormously.
• Getting serious about using MailChimp and email marketing in general. My Halloween, Thanksgiving, and Christmas mail blasts made well over a thousand dollars in sales for me, for about $30 worth of virtual postage stamps and perhaps twenty minutes of writing.  One thing to note for next year: there does not seem to be a substantial difference in conversion rates for when I put a $5 off coupon in the mail versus when I don’t, so I shouldn’t next time.  Also, given that a huge percentage of folks bounce on the password screen coming from the email, I need to think about either putting a token in the URL to let them in automatically, or making dedicated landing pages for these campaigns.  (I don’t have good numbers for how effective the autoresponder sequence is — i.e. automated emails I send to people on the first and sixth days after they sign up.)
• Meat and potatoes SEO continues to be my bread and butter (how is that for a mixed metaphor).  My conversion rate has been gradually sliding as I get more parents in for holiday bingo activities through my Sprawling Bingo Empire, but 1% of a very large number is still worthwhile.  I should probably get serious about optimizing those sites individually, but my desire is waning.

What Didn’t Work So Well:

• In the wake of the FireSheep release, I decided to implement SSL for Bingo Card Creator, right in the middle of Halloween busy season.  This broke my pages in several different ways, from causing security popups on the login screen in IE (whoops) to not showing key images on landing pages in some browsers (whoops).  I really should have put off that implementation another few weeks, or thought through my testing strategy for it better.
• I don’t have a staging server for Bingo Card Creator yet.  Having seen the enormous advantages from having a staging server through my Appointment Reminder, I am retroactively dinging myself for not making one in the last four years.
• My Halloween promotion could have been handled better: $6,000 in sales is nothing to sneeze at, but I’m still of the opinion I could have broken $10k with a little better execution.  Maybe next year.
• AdWords has been on auto-pilot for virtually the entire year, and when it goes onto auto-pilot it seems to optimize for Google’s bottom line over mine.  I should block off some time to get it under control, and aggressively cut weakly performing aspects of my campaigns.

Consulting

Last Christmas Thomas Ptacek at Matasano (whose office I am in as we speak) suggested to me that people would pay for my expertise in software marketing.  I was skeptical, but put the word out quietly that I was available for hire, and did three large projects this year for a few clients.  The only one I can publicly comment on at the moment is that I worked for Matasano, on stuff.  My general field of expertise is in engineering marketing outcomes: A/B testing, SEO, and the like.  Basically, I bring the fanatical iterative improvement mindset and apply it to things other than bingo cards.

I love talking about what I do.  Unfortunately, consulting clients pay a lot of money to get me to shut up.  This means, for example, not blabbing on the specifics of new projects which are as-yet unreleased, and not blabbing on the particulars of engagements.  Since I had only a handful of clients, giving exact numbers tends to give a wee bit too much detail, since if you happen to know one data point then you can guess the other two fairly easily.  So treat these as very ballpark numbers:

Consulting sales: $45,000

Consulting expenses: $10,000 (Plane tickets and hotel rooms get expensive quick, what can I say.)

Consulting Profits: $35,000

I know somebody is going to ask my rate, and I’m torn between a desire to quote it and the knowledge that there is absolutely nothing good that can come of quoting it.  The reality for consultants is that clients pay you exactly how much you can negotiate with them.  Everybody knows this and yet everybody would be shocked, shocked to know that someone else got a better deal.  In addition to causing problems with existing clients, it complicates raising rates for new clients in the future.  (Given that I could fill 100% of the hours I wish to, have been saying ‘Nope, sorry, not available’ to interesting work frequently, and now have a few CEOs singing my praises, my rates are only going up next year.)

What Went Right:

• Client selection.  All three clients knew me well from the Internets, all had confidence in my ability to do what they wanted me to do, and all were model clients: they knew what they want, they communicated perfectly, and they paid all invoices in a timely fashion.
• Pricing: Aside from frightening my bank a few times when I got large wire transfers from America, charging a lot of money is a great idea in every possible way.  It makes your clients respect your time more, keeps you motivated, and helps pay the rent during the lean summer months of the bingo calendar.

What Didn’t Work So Well:

• Newbie consultant blues: I did my first consulting project for a friend.  Unfortunately, due to a combination of it being my first gig ever and my first experience with using Heroku, I greatly underestimated the amount of time the project would actually end up taking.  What I thought would be 20 hours over a two week period stretched into many more hours over months and months.  Luckily, my client was sympathetic, but I ended up doing a lot of writeoffs for good will and diverting my attention for longer than I wanted to.
• Juggling consulting with project work: I worked 90 hours a week and still had cycles to spare for BCC.  How hard could it possibly be to do 10 hours of consulting in a week and still get productive work done?  Very freaking hard, because the 10 hours tended to be splayed across several days, require creativity and focus to execute, and not include the whole email/sales/support dance that comes with consulting.  This is in no way a criticism of the client, it is just to illustrate how consulting works: I got a request from a C-level exec at a company you’ve heard of in April, and despite us being mutually enthusiastic about the project it took forty emails and five months until billable work actually started on it.  I had to gracefully extricate myself from my clients and block off November and December to get Appointment Reminder launched.

Appointment Reminder

I finally released Appointment Reminder in early December.  (Don’t worry, I haven’t forgotten about writing the rest of the series about developing it.)  I have one customer already and a handful of prospects currently trying the software out, so revenue is negligible as of yet.  Now I just have to do the other 90% of marketing the software, which I am going to do a bit of in December and then start in earnest in January.  The plan is, unsurprisingly if you know me, heavily reliant on organic SEO.  I have also had a lot of interest about whitelabeling the software, and will do that as well.  That gives me a built-in on-the-ground sales force of people with intimate knowledge of potential clients and the desire to sell them my service — and judging from the numbers thrown around by one of the guys interested in white labeling, that could be “quite lucrative indeed.”

In terms of technical direction, I’ve engaged a freelancer to get it working on iDevices (I could have done it myself, but I’ve got plenty on my plate as it is).  I am about 60% of the way to getting a very impressively hacky solution working on every major US smartphone, because many prospects have asked to be able to access schedules when on the move.  The implementation is so devious that if I had a mustache, I would be stroking it while cackling maniacally.

Semi-exciting news: I received an acquisition offer from a foreign telecommunications firm.  The CEO believed AR fit a hole in their product line, and offered me their estimated development budget for it if I would sell them the whole business.  That was a generous number relative to the amount of work I have put in, but it would not have been lifechanging for me or my family.  I declined and told him I’d like to try to run the business myself and see where it takes me.

The stats:

Sales: Nothing yet.  (Well, one test transaction to make sure Paypal/Spreedly works.  Spreedly is impressively painless, by the way.)

Expenses: ~$1,600 (a few hundred bucks in design work, $800 of Twilio credit, and one or two other things.  Servers and online services which I also use for BCC got filed under BCC because I’m lazy and my bookkeeping software doesn’t support splitting bills: a more accurate accounting would be closer to $3,000.)

What Went Right:

• It exists and mostly functions! These are both handy properties in software one wants to sell.
• MVP available for several months.  I didn’t have the cycles to create AR back in April/May, but I did get the MVP of it — basically, an interactive demo of the core of the service — out early.  This helped me in both getting a bit of age and trust on Google prior to the official launch, and also in getting a base of interested prospects to try selling to (I’m currently talking to a few of them).

What Didn’t Work So Well:

• Delaying release: April, May, June, July, August, September: that is six months, and I got very, very little accomplished (not one line more than the MVP, as a matter of fact).  The distraction from consulting work, working on BCC, and reacclimating myself to a human existence after years of salarymanhood just totally destroyed any desire to do heavy lifting on a new project.  I’m very obliged to the HN-based folks who started a Launch Something in November mini-sprint, which helped get me the energy to actually sit down at the computer for six hours a day five days a week and bang it out.
• Insufficient pre-launch marketing: I really should have been constantly adding new content to the website for the last six months.

Goals for 2011

Bingo Card Creator

• Sales of $60k, profits of $40k
• Getting AdWords back under control
• Getting the holiday promotions ready for all those domains I own but have never successfully exploited yet

Appointment Reminder

• 200 paying customers by December of 2011.  This implies the revenue run rate will be somewhere north of $10k then.  Cross my fingers: it might be well north of that, if SEO or white labeling works well.
• I’m about 70% certain that I’m going to hire a front end developer for AR.

Consulting

• Perhaps a wee bit more of it.
• At higher rates.
• Knock things out of the park for clients #1 ~ #3, and tell stories when possible.

See you all in 2011!

I worked for almost three years as a cog in a Japanese megacorporation, and one of the best parts about that experience (perhaps even worth the 70 hour weeks) was that they taught me how to be a professional engineer.  Prior to doing so, my workflow generally involved a whole lot of bubble gum, duct tape, and praying.  I spent a lot of time firefighting broken software as a result, to the detriment of both my customers and myself.  Talking to other software developers has made me realize that I’m not the only person who was never taught that there are options superior to bubblegum.  If you aren’t lucky enough to work at a company that has good engineering in its very DNA, you’re likely to not know much about them.

This strikes me as the industry’s attitude to source control a few years ago, prior to a concerted evangelization movement by people like Joel Spolsky.  It is virtually impossible to overstate how much using source control improves software development.  Our industry has changed in major ways since 2000, but our best practices (and knowledge of those best practices) are lagging a few years behind.  We could really use a Joel Test 2010 edition, for a world where “you should have build scripts for desktop software which can complete the build in one step” is largely an anachronism and where the front page to the website is no longer hand-coded in Notepad but, rather, is a shipping piece of software which can break in two hundred ways.

You’re not going to get the Joel Test 2010 here, mostly because I’m not Joel and there is no particular reason any company should judge its development practices relative to mine.  What I would like to give is some practical pointers for implementing three practices which, if you’re not already doing, will greatly improve the experience of writing software for the web:

1. Staging servers
2. Version control workflows
3. Tested, repeatable deployments

Staging Servers

What is a staging server?  The basic idea is that it is staging = production – users. (If you’re Facebook, Google, or IMVU, you are lightyears ahead of this article and have some system where there are multiple levels of staging/production and where you can dynamically change them.  You already have geniuses working on your infrastructure.  Listen to them.  This article is for people who don’t have any option between “code runs on developer’s laptop” and “code runs in production.”)

Why do we have staging servers?  So that anything that is going to break on production breaks on the staging server first.  For this reason, you want your staging server to be as similar to the production environment as you can possibly make it.  If the production environment processes credit cards, the staging environment processes credit cards.  This means that if, e.g., your configuration for the payment gateway is borked, you’ll find out about that on the staging server prior to pushing it live to production and, whoopsie, not actually being able to get money from people.  If your production server uses Ruby 1.9, your staging server uses 1.9.  If the production server uses memcached on port 12345, the staging server uses memcached on port 12345.

(Many folks have systems which exist on more than one physical machine.  I don’t — I’m a small business where 2 GB of RAM is enough for anything I want to do.  If you have multiple machines, strike “staging server” and read as “staging system” below: all the benefits for having a separate staging server are still beneficial when your staging environment actually has fifteen physical servers running 47 VMs.)

Setting up a staging server should be easy.  If it is not easy, you already have a problem in your infrastructure, you just don’t know it yet: you’ve cobbled together your production server over time, usually by manually SSHing into it and tweaking things without keeping a log of what you have done.  (Been there, done that, got the “I Created A Monster” T-shirt.)  There isn’t a written procedure or automated script for creating it from the bare metal.  If you had that procedure written, you should be able to execute it and create a staging server that works inside of an hour.

Most people won’t be able to do this if they haven’t given thought to the matter before.  That is fixable, and should be fixed.  It has substantial benefits: if you have a repeatable procedure for provisioning a production system, then when disaster strikes you will be able to confidently execute that procedure and end up with a production system.  (Confidence is important since you’ll probably be terrified and rushed when you need to do this, and rushed terrified people make unnecessary mistakes.)

If you’re working on Rails, I highly recommend using Deprec/Capistrano with all new projects.  In addition to making it very easy to get a full Rails stack working on your deployment environment of choice , it helps automate routine deployment and server maintenance, and has mostly sensible defaults.  (I have only one quibble with deprec : it installs software from source rather than using your system’s package manager.  That means that upgrading e.g. Nginx two years down the road is needlessly hard and error prone, when instead you could just have used apt-get in the first place and then updating is a piece of cake.)

You can also use Fabric, Chef, Fog, or a similar system to script up building new environments.  Pick whichever strikes your fancy.  Try to recreate your production environment, down to a T, on another host at your VPS/cloud/etc provider, or on another physical machine if you actually still own machines.  Keep tweaking the script until it produces something which actually matches your production environment.  You now have a procedure for creating a staging server, and as an added bonus it also works for documenting your production environment in a reproducible fashion.

One nice thing about keeping your server configuration in scripts rather than just splayed across fifteen different places on the server (/etc/environment, /etc/crontab, /usr/local/nginx/conf/apps/AppName.conf, etc) is that it lives in source control.  Your cron jobs?  If they’re in source control, you’ll have a written record of what they are, what they’re supposed to do, and why they just blew up when you bork the underlying assumptions eight months down the line.  Your Nginx config?  If it is in source control, you’ll understand why you added that new location setting for static images only.  The voodoo in your postfix config?  A suitably descriptive commit note means you’ll never have to think about reproducing the voodoo again.

After you have the script which will produce your staging environment, you probably want to make a minimum number of alterations from production.  Many companies will want their staging environment to be non-public — that way, customers don’t see code before it is ready, and critical issues never affect the outside world.  There are many ways to do this: ideally, you’d just tweak a setting on your firewall and bam, nobody from the public Internet can get to your staging environment.  However, this is a wee bit difficult to pull off for some of us.  For one, I don’t actually have a hardware firewall (I use iptables on each VPS individually).

My staging environment simply includes a snippet in Nginx which denies access to everyone except a particular host (which I can proxy through).  This breaks integration with a few outside services (e.g. Twilio and Spreedly, which needs callbacks), so I make exceptions for the URLs those two need to access.  The more complicated your staging server configuration gets relative to production, the more likely you are to compromise its utility.  Try to avoid exceptions.

That said, there are a couple that are too valuable to not make.  For example, my staging server has a whitelist of email addresses and phone numbers owned by me.  Through the magic of monkeypatching, attempting to contact anyone else raises an exception.  That sounded a little paranoid until that day when I accidentally created an infinite loop and rang every number in the database a hundred times.  (My cell phone company loves me, but folks who accidentally collided with test data sure would not have.)

How do you get data to populate the staging server?  I use seed scripts and add more data by hand.  (I also have DB dumps available, but they tend to go stale against the current schema distressingly quickly: I recommend seed scripts.)  You can also dump the production DB and load it into the staging DB.  Think long and hard before you do this. For one, it is likely to be way, way the heck out of bounds for regulated industries.  For another, your staging server is probably going to periodically be insecure — insecurity is failure and failure is what the staging server is for.  Slurping all of the data out of a staging environment has caused many companies smarter than you to have to go into disaster management mode.  Please be careful.

So you’ve got a staging server?  Now what?

At the simplest, you access your staging server with a browser and try to break things.  When you break things, you fix things, then you redeploy the staging server and try to break them again.  This is what you are probably doing right now with production, except that your customers don’t have to see broken things when you break things.

Eventually, you can script up attempts to break things, using e.g. Selenium.  Then when you break things, you add them to the list of things that Selenium tries to break.  If you run that against the staging server after every code check in (a process known as continuous integration), you’ll quickly catch regressions before they disrupt paying customers.  This is a wee bit harder than just having a staging server — OK, a lot harder — but you’ll get clear, obvious advantage out of every increment of work you do on this path, so don’t let present inability to be Google prevent you from getting started.

Version Control & Deployment Workflows

Everyone should use version control, but people tend to use version control differently.  Git is very popular in the Rails community, but there are probably no two companies using git the same way.  The key thing is that you agree with your team on how you use version control — document your assumptions, document your processes, then apply them religiously.  This will reduce conflicts on the team, reduce mistakes, and help you get more out of your tools.

There are a million ways to use version control and most of them are perfectly OK.  I’m going to mention mine, but it isn’t the canonical Right Way, it is just one way which works for a (very) small company.  Yours will likely be different, but you can see some of the things which go into design of a version control workflow.

Assumptions I Make About Life, The Universe, And Everything

1. I use git.  Git has notion of branches, tags, and remotes (physically distinct repositories) — if you don’t know what these are, Google for [getting started with git].
2. I generally work alone or with a very small team. (This assumption underpins very important parts of my workflow.  It won’t expand very well to a 200 man distributed team, but it might well work for 2 ~ 5 people.)
3. There is exactly one canonical repository, origin.  Developers maintain other repositories on their workstation.  Automated processes like deployment happen only with reference to the origin.  Code existing outside of the origin does not officially exist yet, for any purpose.  The history preserved in the origin is, in principle, sacred.
4. There is a branch called deploy.  The HEAD of deploy (the most recent code on it) is presumptively ready to be put into production.
5. Tags are used to take snapshots of the code base and preserve them in amber with a human readable name.  Right before we deploy to either production or staging, the HEAD gets tagged, so that we can easily find it later, with a simple naming convention (I use production_release_X and staging_release_X, where X just increments upwards — some people might prefer timestamps).  Production release tags are never deleted.  Staging tags get periodically culled when convenient to do so.
6. Development of any feature expected to take longer than a few hours happens on a feature branch.  (I do occasional work right on deploy locally, for issues of the “Minor copy edit on dashboard.” variety.  This would be one of the first things to go if I were working on a larger team.)

So how does this actually work in practice?  Let’s say I’m implementing a new feature.  I create a new branch to work on.  I code a bit, creating local commits with wild abandon any time I have accomplished something which I don’t want to lose.  When I believe code to be functional, I fire a capistrano task which tags the current head of my branch, pushes that tag to origin, and deploys it to the staging server.  I then continue testing on the staging server, for example verifying that Twilio integration actually works with Twilio (which cannot conveniently access localhost:3000 on my laptop).  I continue writing code, committing, tagging, and pushing to the staging server until the feature is ready.

Then, I switch back to the deploy branch and merge in my feature branch (with –no-ff, which creates a commit message just for the merge — this handily groups the twenty or thirty commits I just made into one easily readable story for myself later).  I then tag a production release (manually — this is entirely to force me to think through whether I’m ready for a production release), verify that there is no diff between it and the most recent staging release, and then push the new tag to origin.  I then fire the Capistrano task which checks out the new deployment tag and restarts the server.

What does this get me versus my previous SVN workflow for Bingo Card Creator, which was “Work only on one branch, commit stuff when I think it is ready, and deploy the trunk manually on occasion”?

1. I cause much less downtime for the production server due to reasons like svn commit -m ‘Whoops, forgot a setting in production.rb’ and svn commit -m ‘r1234 introduced dependency on foobar gem without putting it in environment file, causing rake gems:install to not load it.  Mongrels then failed to restart.’
2. My deploy branch has a relatively clean history, so when things start to break next year in production, finding the change sets which eventually caused the breakage will be less of a needle in the haystack search than finding them in SVN is.  SVN’s history is 1800 unedited commits, recording my stream of consciousness as they happened.  My stream of consciousness is frequently stupid, particularly when I’m panicking because the server is down.
3. This decouples the staging server from production in a clean fashion (so that I can advance the staging server a feature or three at a time if I want to), but guarantees that when I’m actually ready to deploy, I’m deploying exactly what did not break on the staging server.
4. Tagging releases gives you an Oh Crikey button, as in Oh Crikey, that last release broke stuff.  You can quickly rollback the deploy to a known good tag, isolate the changes which broke production, and fix them.
5. Deploy scripts manage releases with multiple moving parts a lot better than I do, even when I’m working from a checklist.

By the way, git gives you many options for recovering from problems — even severe problems — without requiring either gymnastics or a full-blown CSI investigation to discover what happened later.  For example, let’s pretend I just deployed tag production_deploy_82, and have discovered some issue serious enough to require an immediate rollback to production_deploy_81, which is known to be good:

#Assuming we are on our local workstation on the deploy branch.

git branch something-in-here-is-broken

git reset --hard production_deploy_81  #All changes made between deploy 81 and 82 just vanished from the deploy branch locally.

#Clean up the deploy, using any option discussed below.

git checkout something-in-here-is-broken  #Those changes which you just disappeared are now living on this branch, ready for you to fix.  After you've fixed and verified it works on the staging server (and, ahem, that you have addressed the issue that allowed this to get OKed for release last time), you merge this branch back into deploy, and do a tag-and-release cycle.

How you clean up the mess on the server is up to you: good options include “deploy 81 again”, “tag a release 83 equivalent to 81, then deploy it”, and “rollback to the copy of 81 which still exists on the server.”  (Capistrano includes deploy:rollback, which will do exactly this.)  Any of these will work, just always do it the same way to avoid stepping on each others’ toes.  I prefer tagging a new release so that I can add a descriptive message explaining why 82 just created an emergency.

This is important because it leaves a paper trail — if you’re pulling a release from production, something just went seriously wrong with your processes.  Emergencies are not supposed to happen – anything that lets an issue get that far isn’t just a one-off failure of whatever broke, it is a series of failures of the systems/processes designed to prevent failures from getting that far.  After you’ve put out the fire, investigate what went wrong and tweak your processes such that a similar failure in the future gets caught prior to bringing down production.  The sleep you save may be your own.

Scaling this to more programmers: Do whatever works for you!  I would probably create a staging branch and have folks integrate stuff into the staging branch when it was ready to go to the official staging environment.  I also might make per-developer staging environments: since creating one from the bare metal is supposed to be essentially free, let them all have their own where they can be reckless without spoiling the experience of other developers.  We can worry about code interaction on the “real” staging server.  Then, have folks communicate when they consider everything they have on staging ready for release, and release when everybody says it is ready.

The important thing is that, whatever process you use, you document it, teach it, and enforce it.

Stuff Your Deployment Script Might Not Do Today But Probably Should

1. Depending on your scale and how you use e.g. memcached, it might be safe to purge the cache on every re-deploy, which will prevent some hard to diagnose bugs.  At a certain scale, this is virtually a recipe for taking your site down in a cache stampede, but I’m not Facebook and having capacity problems means that I am probably already vacationing at my hollowed-out volcano lair.
2. Tell everybody on the team that you just deployed.  I know some teams who have an IRC channel with a bot who announces redeploys.  A quick email CCed to five developers also probably suffices.
3. Restart worker processes.  This is easy to forget but, if you do it by hand, you’ll eventually forget and then have two versions of the application in production at once.  If you’re not prepared for that, it will bite you on the hindquarters when, e.g., the application servers ask the workers to execute methods that the workers do not know now exist in the code base.
4. Do sanity checks.  You can go arbitrarily deep with complexity here.  For a first cut, mine for Appointment Reminder restarts the application server, counts ten seconds, then tries to access an internal URL.  If the application server isn’t up, or if the action at that URL blows up for any reason, the deployment script fails the deploy, rolls back to a known-good version, and sends me a very crossly worded email.  (You can do this for the staging server, too.)
5. Integrate with other systems which manage the state of your code/business.  For example, I use Hoptoad.  Hoptoad keeps track of exceptions and mails you when they happen, in such a fashion that your inbox doesn’t get buried by e.g. Googlebot deciding to do an impromptu fuzz test on your website.  I mark all exceptions as resolved every time I deploy to the environment they happened in.  You could also e.g. update an internal wiki by adding a new page specific to the deployment, automatically update your bug tracker to change the status of the bugs that you (presumably) just squashed, or start a new cohort for your stats tracking.

Is There Anything You Would Like To Add?

The trend towards openness with regards to technical practices on the Internet is a major, major win for everybody in our industry.  Best practices do not have to be passed from master to apprentice as oral lore.  Like OSS, since they’re often glue rather than competitive advantages (for many companies — not all), sharing them mostly doesn’t hurt you and can improve the situation of everybody.  So, in that spirit, if you’ve got anything you’d like to add, particularly for how you do things or how you would adapt this advice outside the scope of a very small business, I’d love to hear about it either in the comments or on your own blog (feel free to leave a link if it is relevant.)