Kalzumeus Software

Last week, I had an opportunity to talk with Andy Brice, who sells software for wedding seating plans and the like.  He is an absolute genius with AdWords, and gave me some ideas on ways to improve my performance.  I immediately started to implement them, full with the excitement of a new project and wondering why I don’t spend more time optimizing AdWords.

Oh, right.

There were another 15 ads which I added last Friday-ish and are still Under Review.  Under Review is Google-speak for “We aren’t sure that this ad complies with our policies yet.”  While an ad is Under Review, it doesn’t show anywhere, and you aren’t learning anything by having it.

Dealing With Shades of Grey

Google has a variety of businesses which it does not want to or legally cannot do business with.  To prevent them from using AdWords, they exercise prior restraint on AdWords copy, not letting their ads run until a human at Google has approved them.

One of the businesses that Google doesn’t want advertising (in the US, at any rate) is gambling.  Bingo is a form of gambling.  Bingo Card Creator is not a form of gambling — it is a form of software which helps elementary schoolkids learn to read.  This makes it rather hard to write focused, relevant advertisements responsive to customer queries like [how do I make a US presidents bingo card] which sell Bingo Card Creator without using the word “bingo” anywhere.

Google is, to all appearances, just using a keyword-based blacklist.  I guess all the eats-Bayesian-classifiers-for-lunch PhDs work in search and Gmail spam filtering, where they’ve clearly got an aptitude for understanding that words can have multiple meanings.  OK, fine, but at least the remaining boffins can do a blacklist correctly?

Well, not so much.

• Using Google’s Copy Ad feature to copy an ad, word for word, between ad groups will cause the new copy to go back into review purgatory.  This is despite that theoretically being a content-neutral action and a core task for advertisers, because many flavors of AdWords optimization rely on keywords being partitioned correctly into focused ad groups.
• Changing so much as a character of the ad, including landing page URLs, will cause the ad to get flagged again.  This only affects good advertisers.  Bad advertisers can presumably figure out how to serve whatever content they want on http://example.com/approved .  Pulling a bait-and-switch is absolutely trivial, since you have full control over what your own server serves to users.  This rule only inconveniences compliant advertisers, who get thrown into review purgatory every time they e.g. try to add another tracking parameter to their landing pages, switch from http:// to https://, etc etc.  I get the feeling I’m supposed to create five copies of each ad, pointing to /lp1 … /lp5 with identical content, and then if I need to do any testing I should get crafty with redirects or what have you later.  That’s insane - it is extra work that is directly against the spirit of the rules and unlike actual compliance it works.

Scalable Communication Methods

According to Google:

We review all ads in the order they’re received, and we work to review all ads in our program as quickly as possible, usually within 1 to 2 business days or sooner.

If there were only 48 hours of lag time inserted every time I touched an AdWords ad, this would be annoying but tractable.  It would lengthen my time through the idea creation/validation loop (Lean Startup  fans know why that is a Very Bad Thing), but I could still get work done by batching all my edits together and then twiddling my thumbs for 48 hours.

Sadly, Google routinely falls short of their announced level of service.  And when I say “Falls short”, I mean “Ads can sit for weeks ‘Under Review’ and never be approved.”

This leads you to have to contact Google Customer Service to be able to get Google to give permission to give Google money.

Google Customer Service: Welcome to Kafka

The first rule of Google Customer Service is that Google does not have Customer Service.  They prefer what Chief Engineer Matt Cutts describes as “scalable communication methods”: there are like a bazillion of you, there are only a few tens of thousands of us, instead of actually speaking to a human being you should read a blog post or watch a video or talk to a machine.  It is a wonderful, scalable model… when things work.

Anything which introduces a mandatory customer service interaction with Google is a process designed for failure.  AdWords approvals requires a customer service interaction.  Catch-22, to mix literary metaphors.

The “scalable communication methods” like AdWords Help have this to say about contacting customer service with regards to ad approvals:

Our Support team won’t be able to help you expedite this process.

That is not actually a true statement (which, incidentally, describes much of AdWords Help).  Length of time from ad submission to approval is, in my experiences, unbounded (literally, weeks can go by without approval).  Length of time from complaining to Support to approval: a day or two.  The most helpful Google employee I’ve ever Internet-met (name withheld to protect him from whatever dire punishments await someone who attempts to help customers) told me that my workflow should literally be 1) Submit ad 2) Submit ticket to get ad looked at, if I persistently fell into Under Review.

Google apparently knows it, too, since they have special-cased out the CS interaction for dealing with Ad approvals:

After filling in everything, I hit Submit expecting to be taken to a page which had an “OK, now actually tell us what the problem” comment box was.  No need — it has been optimized away!  Google doesn’t even want that much interaction.  (The last time I went through this — sometime last year — I recall there being a freeform field, limited to 512 characters or so.  I always use it to explain that I am not a gambling operation and if they want confirmation they can read the AdWords case study about my business.)

Google’s computers then weighed my support request and found it wanting:

Dear Advertiser,

Thank you for your e-mail. We understand you’d like us to review your ad.
When you submit new ads or make changes to existing ads, they’re
automatically submitted for review.

We work to review all ads in our program as quickly as possible. You
should receive an email notification stating the approval status of your
ads pending review within the next 3 business days. You can view the
status of your ad any time in your account. The “Status” column in the
“Ads” tab displays information on the current state of an individual ad
variation.

For a list of Ad Approval Statuses, visit
http://adwords.google.com/support/aw/bin/answer.py?hl=en&answer=138186

We are working as quickly as possible to get everyone up and running and
should get to yours soon! If you have a different question, which doesn’t
refer to pending ad approval, please get back to us via the ‘Contact Us’
link in the Help Center at https://adwords.google.com/support/aw/?hl=en.
Be sure to choose the category that is most relevant to your question.

Sincerely,

The Google AdWords Team

Well, at least the templating engine correctly replaced $BRUSHOFF_LETTER, but in terms of customer communication:

• You asked me to put in my name… you might want to think about using it.
• As much as I appreciate your False! Enthusiasm! if the next line of your letter is going  to be Eff Off And Die then maybe you should take out the exclamation points and give them to a Ruby programmer.  (We can always use more.)
• If the original timeline was 1-2 business days and the timeline three days later is “within 3 business days”, can we update them so that they quote it consistently?  Or maybe put something like “We get to 98.2% of approvals within 3 business days.”  (Or 2.89% of approvals within 3 business days, as the case may be.)

Google’s Isolation From Market / Customer Pressure

Google theoretically values my business — I pay them $10,000 a year and would love to pay more.  Indeed, they can find my email address and have a human contact it when they want to do ad sales.  (I got an offer recently to set up a call with one of their AdWords Strategists to discuss optimization of my account… which is great, but previous experience leads me to believe he would use the same reports I have access to, make decisions with little understanding of my business, and then leave it to me to actually schedule the new ads/keywords and run headlong into Pending Review purgatory.)  But they are not doing very well lately at convincing me they actually care.  And they’ll still make a bazillion dollars without that, so no harm done.

In normal markets, I would be strongly tempted to take my business to vibrant competitive offerings.  Sadly, Google is pretty much the only game in town for viable CPC advertising: even if Microsoft/Yahoo exorcized the abominations haunting their UIs, they would not have enough inventory to matter for me in my niche (I’ve tried before).

Which leaves me with only one option: trying out my own scalable communication methods, and hoping someone in the Googleplex reads this and takes action to unbork this process (ideally, for a large class of advertisers).  It is the Internet equivalent of putting a message in a bottle and then throwing it into the ocean, but that is still an improvement on the normal channels.

YCombinator just released their semi-annual application for companies to incubate.  One of the new questions this time around is “How will you get users?”  I think that is a great question to think about for everybody in business — perhaps the great question to think about.  Customer acquisition is one of the easiest places to screw up as a startup, particularly for technical founders (who, in their previous lives, have probably never had to do it for anything).

I’m not applying to YC this time around, but I always fill out the application to force myself to talk through my business strategy.  I had one thought which sounded worthwhile enough to share: customer acquisition can be hacked.  You can take the current conventional wisdom in the market of how to get customers to use solutions, identify it’s weak points, and aggressively target them.  That can, potentially, be as important (or more important) than the same applied to the actual product.

Enemy #1: The Technology Adoption Cycle

Let’s assume that you’re capable of successfully identifying a problem customers have and solving it.  Those are both highly non-trivial, but put them out of scope for the moment: people’s hair is on fire, you’re selling fire extinguishers, life should be grand.  Life is often not quite so grand, because you can produce a wonderful product which creates value and fail to sell it to folks.

Most startups are not creating an entirely new solution out of whole cloth.  Somewhere out there people are currently experiencing the problem you are solving, and they’re dealing with it somehow.  They might be ignoring it or gritting their teeth.  They might be using some inferior solution which they got from your competitors, you have competitors (you should have competitors — if you don’t, you probably aren’t doing something people care about).

Your competitors had to see people through a product adoption cycle:

1. Identify people with the problem
2. Teach them that the solution exists
3. Successfully sell them on the solution
4. Prevent them from leaving the solution for a competing solution

In actual practice, this adoption cycle is frequently long and arduous.  (If it were short and easy, there wouldn’t be any money in it.)

Your competitors, if they are established businesses, are probably very good at maneuvering customers through the technology adoption cycle as it exists in the market today.  For example, if grading students is a problem, your competitor might very well be successfully selling school districts on their gigantic consultingware grading solution which costs six figures an installation.  Since they can still make the rent and keep the lights on, you can infer that their business probably works.   Their marketing team is generating sufficient leads, their sales team converts some of them.

But you probably don’t want to do what they’re doing, because they’re better at being them than you are.

Hacking The Product Adoption Cycle

Startups are not the world’s most obvious choice of employment for people who enjoy coloring between the lines.  If you execute the competition’s playbook for acquiring customers, you are probably going to get crushed by them, because

• they know more about the market than you do
• they have a commanding head start
• they have large amounts of resources to throw at the problem

On the other hand, it is entirely possible that:

• they have stopped learning about the market
• they have a commanding head start running in a suboptimal direction
• they have large amounts of resources which, for reasons of switching costs and politics, can’t be reallocated to more efficient approaches

These statements aren’t just true about the product — sure, they might have a crufty old VB6 app and you have the new Node.js hotness.  They are equally true about the customer acquisition process.  You’re competing with their business, not with their product, so you could possibly either focus your innovation on customer acquisition or, more likely, use innovation on both customer acquisition and product in a mutually supportive manner.

Examples Of Hacking This

Freemium isn’t a business model so much as it is a customer acquisition tactic.  In markets dominated by expensive solutions with huge switching costs and uncertainty about success with technology changes, freemium can be very compelling: the self-serve model allows you to do less consultative sales (with the multi-month purchase cycles, large sales teams, and politicking that entails) and instead focus your efforts on getting leads and converting them.  This plays to a very different skill set versus traditional enterprise B2B sales, and it is a much more forgiving of small teams, since you’re deputizing your free users as internal sales champions and praying that they can do the consultative sales that your non-existent sales force isn’t doing.  This also lets you crack into markets where any model which requires consultative sales automatically is priced out of the market — essentially, anything where customer lifetime value is less than $75,000, give or take.

Monthly billing is another hack.  Customers are irrational and their processes are broken.  One artifact of those practices is that there is a stepwise increase in difficulty if prices increase by $1, as long as the price was already at whatever the company’s magic number is for maximum to be put on a corporate credit card or signed for on a non-manager’s authority.    Monthly billing defeats this step function because even if the total lifetime cost of the solution goes up the largest amount ever billed at once might well cross under that critical threshhold again.  This means that there is no longer a total no-man’s land between $1,000 and $75,000 in lifetime value. (Is this a hack?  Yes.  If you bill a Fortune 500 company product manager $80 a month, you are essentially conspiring with him against his accounting controls.  Not that there is anything wrong with that.  You can even explicitly sell that as a benefit to him, just like you sell SaaS as allowing him to avoid having to talk to IT to get the stuff he needs to do his job.)

Online marketing expertise hacks through the ridiculous inefficiencies of offline marketing.  Many startups can run rings around their traditional competitors in online marketing, for example due to savvier SEO that leverages their strengths in execution speed, technological savvy, and community ties.  For example, my wee little business competes directly with Scholastic Publishing, who has 10,000 employees and access to public capital markets.  They also couldn’t spell SEO if you handed them a set of alphabet flash cards, which is good news for me.  You would think that “Well, a business which doesn’t have online marketing expertise could just hire for it”, but after you get past the level of “let’s make a website — it should probably have title tags and some of those keywords, too”, everyone who tries this finds that it is murderously difficult to hire competent SEOs right now.  (If you disagree, I have some clients who would love to meet you.)  At the same time, I couldn’t possibly compete with the relationships which get their competing products on shelves at tens of thousands of retail locations… but then again, I don’t have to pay 50 cents of every dollar of sales to the retailer, either.

Taking A Hack From Tactic To Strategy

I think this isn’t exactly a new insight.  There are lots of folks who, when asked for their marketing plan, will say “Oh, we’re going to get lots of search traffic” (indeed, that is probably second only to “it will grow virally” in terms of signaling “has probably not thought this through.”)  What separates hopes and dreams of future success from very valuable businesses is a strategy which, with execution and refinement over time, will actually achieve the goals.

We often hear products described using something like “It’s like Facebook, except for dogs.”  How about, instead, describing businesses like “It’s like Quicken, except Quicken sells primarily through boxed software channels and we’re going to sell primarily through banks which will deal with us for a cut of the sale price and the ability to deepen relations with small business customers, who consume lots of high margin services and stay locked in for decades at at time.”  (That may or may not actually be true.)

We often accept previous experience or minimal proof-of-concept prototypes/MVPs in lieu of a functioning product when evaluating whether someone is capable of executing on building something.  Why not do essentially the same for proving that one is likely to get customers?  A previous background in revenue maximization through negotiating cross selling deals for banks, or evidence that you have enthusiasm from a few bankers who like the concept and want to hear more when you have something to show, demonstrates a certain likelihood that marketing challenges will be overcome like technical challenges will be overcome.

Similarly, for a startup hoping to make inroads for SEO, I’d be thinking less along the lines of “we’ll sprinkle some SEO on our website” and more along the lines of specific plans for scalable content generation, securing backlinks at scale, and winning the support of influencers either in the niche or in other addressable niches which your competition may not be aware are relevant to that facet of their business.

Product Supports The Marketing And Vice Versa

I have a wee little heresy as an engineer: I think that you can make a perfectly viable business out of a product which is not better than competitors, solely by improving on the method of selling it.  Farmville (and whatever Zynga has reskinned it as this week), for example, is not superior to all other options for entertainment… it just beats the pants off of most of their viral spread patterns, because promoting your use of the game is the core gameplay mechanic.  (You can also do this in more socially beneficial ways than Farmville, don’t worry.  I have a competitor in the bingo business whose product is very close in quality to my product.  They sell to schools via a catalog.  I sell to teachers via a website.  Despite solving the same problem for the same end-users our businesses are like ships passing in the night.  Hilariously, at least a few of my customers actually own both pieces of software, because the people who buy from the catalogs never bothered telling the people who use the websites.)

However, this doesn’t mean you can’t innovate on both the marketing and the product.  Indeed, since they feed off each other, that is probably substantially more effective than innovating on one or the other.  Imagine what a juggernaut World of Warcraft would be if they nailed their game’s quality as much as they did and also had Zynga’s viral loop and monetization model.  That hypothetical WoW could probably deal with Chinese net regulators by buying China.

(It’s easy to say this in retrospect: empirically, millions of employed adults with lots of disposable income spend much of their free time playing WoW.  They spend huge amounts of money on buying status for themselves — cars, diamonds, big houses.  They clearly value their experience in the game.  Therefore, they should be willing to buy status in the game, too… and since buying status is more being seen as having paid lots of money than it is about any particular artifact received, this should go over very well.  I mean, crikey, in a world where encrusted mollusk discharges say “I love you” anything is possible… anyhow, it is easy to say that in hindsight.  The challenge for startups is identifying that sort of synergy between customer adoption and the product in advance, and communicating that it is likely enough to happen to risk betting on.)

Hacking A Non-Computer System Whose Source Is Closed And Updated Continuously

We all know the first iteration of the product is going to suck (hopefully in the sense of “not meet customers needs” more than “a broken, unreliable mess”).  The first iteration of the marketing strategy is also going to suck (hopefully in the sense of “fail to generate the expected level of success” rather than “like shouting to an audience of deaf ants during a hurricane”).  Just like you can use the Lean Startup principles to modify your product and marketing message such that it comes closer to achieving a match with what some people actually need, you can also use spiritually similar disciplines to iterate on customer acquisition strategies.  There is as large a solution space in them as there is in the product space.  Maybe you need to try SEO and see that it doesn’t do a great job in your market, for your customers, while an affiliate channel performs better.  If you’re experimenting, measuring, and moving with a purpose as opposed to the traditional method of “throw stuff at the wall and see what sticks”, you will hopefully have a bit of success.

I’d love to hear if you have comments.

[Memo to self: Prior to ever actually applying for YC, I should practice thinking big thoughts and then writing small thoughts.  Those form fields are tiny!]

“Your single best investment is your own education.” “The average new graduate is drowning in $22,000 in debt.” “A degree in English is just as valuable as a degree in Biology — it teaches you critical thinking!” “Follow your dreams and you’ll find financial success whatever your major is!”

You’ve probably heard a thousand pieces of educational/career advice like these, and if your family/friends/advisors are anything like mine they’ve virtually never been substantiated by data.  That is a shame, because choosing a course of study is one of the largest transactions you’ll ever make — the sticker price at my alma mater was close to $140,000 and prices nationwide have risen faster than inflation for virtually a generation now.  We have the Blue Book to tell you what a ten year old beater is worth down to the dollar, there are entire industries devoted to assessing every type of security to determine their valuation, and the closest thing most students have to insight on the degree selection process is getting advice from Aunt Sue.  This is insane.

Information Asymmetry In Employment Outcomes

Any college could rectify this situation virtually overnight: take that lovely list of alumni that they obsessively curate for squeezing donations, send out a two question survey (“What did you major in?” and “What was your salary this year?”), and give a sociology grad student a bowl of ramen to do some very simple number crunching.  No college will actually do this because transparency goes directly against their interests: if all degrees from a particular institution are valued at “An uncertain, but certainly large and roughly constant number”, then the standard practice of pricing them all identically makes sense.  If not, it is the academic equivalent of pricing stocks by length of ticker symbol.

(I understand many folks enjoy the non-economic component of their education.  I did and do, too, but since I’ve never spent $120,000 and signed myself up for a decade of debt for the sheer enrichment offered by attending a ballet or reading about the Japanese economy I can only conclude that I don’t value it anywhere near what I paid for my degrees.  Your objection to the narrowness of this study is duly noted, though.)

You might assume that the government would track this, somewhere, but you’d be wrong.  The Census Bureau produces a report every ten years tying level of education (associate’s degree, bachelor’s degree, master’s degree, etc) to salary, which invariably produces the result “More education is better”, but they don’t answer very interesting questions like “Is a bachelor’s degree in computer science better than a master’s in elementary education?” or “Are there fields with a sharply diminishing return to additional education?”

However, the Bureau of Labor Statistics does very comprehensive work in administering a National Compensation Survey, which gathers huge amounts of raw data about employment hours and wages broken down by region, job (over 800 classifications available, from CEOs to ship loaders), and a few other axes.  They use this and other studies to produce a variety of government reports, such as the Occupational Outlook Handbook, which does a good job of providing per-career advice but probably intentionally omits comparisons between careers.

The Bureau of Labor Statistics also makes the data from the NCS available for download on their website.  It is hefty — over a gigabyte of plain text — but it contains a virtual treasure trove of value… if you just know how to read the map.

Liberating Conclusions From Open Data

Recently, a big buzzword in the tech community has been Open Data: the notion that the huge, monstrous streams of raw facts collected by the government can be exploited for our benefit if they are merely shared.  I think this is mostly true: the best single example I’ve heard of is that since your local health authorities inspect every restaurant’s premises as a matter of course they must know where they all are located, and therefore one should be able to get those locations from them and jumpstart the creation of a guide to local restaurants without having to find every one by yourself (a monumental undertaking).

However, raw facts are uninteresting.  Here’s a line from the BLS data:

NWU009910010200000000000016260,2008,M07,69.71

Scintillating stuff, right?  What we are really interested in is what those facts can teach us — in particular, what can they teach us that allows us to make decisions such as what to major in.  This is where your friendly neighborhood computer programmer comes in: with a bit of elbow grease and a laptop, you can reduce the 856,000-odd facts in the government’s salary data to some useful conclusions about college majors.

A Bit Of Science And A Dash of Art

Sadly, there are limitations to what we can accomplish with the BLS data: it groups salary data by occupation rather than by major and degree level.  The BLS separately identifies for each occupation what the probable degree level is, but going from occupation to degree requires a bit of guesswork.  Rather than associating all occupations with sets of related degrees by hand, and injecting my own biases into the analysis, I decided to crowd source the problem: I paid Mechanical Turk workers for their two cents (quite literally) on what degree an e.g. elementary school teacher likely had.  This produced answers like Education, Early Childhood Education, Teaching, English, etc.

I then used an arbitrary level of consensus as a cutoff, and was able to pair ~60 very popular degrees (Computer Science, English, etc) and ~250 less common ones (Vocational Education, Media Technology, etc) with associated occupations.  Some additional number crunching let me construct rough estimate salary-versus-age curves for the occupations, which could then be reduced into a simple net present value calculation.  Long story short: a lot of data, a bit of science, and a dollop of absolute voodoo — it’s sort of like most social science, except I’m going to be honest about the voodoo upfront.

After doing the calculations I used Ruby on Rails and some open source graphing libraries to present the results in a comprehensible, searchable fashion — similar to the data visualizations done by the New York Times, which are some of the best work they produce.  (Check this one on the geography of the recession for the general feel.)

Why Go To All This Trouble?

Short story: Intellectual interest plus a nice paycheck.

Longer story: I do very occasional consulting work for a variety of clients.  In case you haven’t noticed from the six-figure sticker prices, offering degrees is a very big business.  Any flowing river of cash that large attracts, as if by magic, a variety of service providers around it.  In education, one major problem colleges have is finding prospective customers to sell degrees to.  This is hardly a unique problem for businesses.  (Colleges may prefer to phrase this as “students” to “award” degrees to, because they are intellectually committed to a view of themselves as custodians of the lamp of human knowledge rather than rapacious money-grubbing institutions.  I don’t know of any reason they can’t be both.)

One thing colleges — from the Ivies to state schools to online for-profit institutions — spend absolutely gobsmacking amounts of money on is “lead generation”.  Basically, since a percentage of applicants will eventually matriculate (and pay five or six figures for the privilege), when a qualified prospect fills out an application that is an economically beneficial event.  You can compare this to a conversion to the free trial of a web service.  Universities are willing to pay quite a bit of money if you can induce someone to apply: the payout varies by university and agreement, but payments in the $10+ region just for requesting an application packet are not uncommon.  (And if you had some magic way of generating sought after candidates — say, highly qualified African American students — you could almost certainly negotiate much, much higher payouts.  There might still be some Marxists on the faculty but it is all capitalists in the administrations.)

Anyhow, with universities offering to pay for lead generation, there is an entire value chain created from the ether: sites to capture the leads, affiliate programs to direct folks to the lead capture sites, advertising to attract visitors to the affiliates, publishers to create content which displays advertising, etc.  One of the publishers in the industry, Online Degrees, hired me with an open brief: make something compelling for our website.  I thought since universities, academics, and the government have failed to produce any actionable data on which degrees make sense to go after, I could do some independent research on the subject.  Online Degrees.org will host it on their website, and in the course of providing value to potential students researching the subject, they’ll have an opportunity to display ads for degree programs.

You might be concerned about the impartiality of this.  I don’t blame you.  I’ve got no particular dog in this fight: I get paid by the hour no matter what degree wins.  (Cards on the table: I have degrees in Computer Science — which is in the data set — and East Asian Studies, which is not.)

Online Degrees.org obviously has a vested interest in convincing you that a having a degree is better than not having one, but they’re agnostic about which one you apply for.  Indeed, they’d love to tell you which fields are better than others because somebody in the industry needs to have the credibility to say that e.g. culinary school is tantamount to grand theft (and most of the victims take out loans for the privilege of going through it).

Besides, do you really have a better alternative?  If I had a PhD in Sociology, would that make me a less biased source of information on the desirability of becoming a cheap source of exploitable labor a master’s candidate in Sociology?

Anyhow, I have been intellectually interested in this subject for several years now.

Quick Overview Of Results

For the results of most particular interest to you, take a gander at the degree value calculator.

Regular readers of this blog are mostly technologists of one flavor or another, and degrees in technical majors do very, very well.  Computer Science and Computer Engineering are near the top among all options for bachelor’s degrees.  It is narrowly bested by a handful of degrees tailored around resource extraction: for example, if you study Geology, Big Oil will apparently pay you Big Bucks.

Hard sciences such as Physics and Biology pay rather less well than I would have expected.  Degrees in the humanities perform about as poorly as people often joke.  The largest surprise to me was that degrees, even advanced degrees, in some caring professions (like Social Work) are apparently terrible options.  Looking at the underlying data suggests that this because many social workers do it as a part time job.  (That is a recurring theme among many jobs that I expect people would classify as more likely to be female than the typical occupation.  Food for thought the next time someone brings up the wage gap.)

You can see the results of this research on their website.  [Edit as of 2/19/2013: You’ll have to search for this directly, due to link rot.]

Questions?  Comments?  Criticisms?  I’d love to hear them.

I’m Patrick McKenzie.  For the last few years, I’ve run a small software company which, most prominently, makes Bingo Card Creator.  It creates… well, you probably get the idea.  I recently launched Appointment Reminder, which… yeah.  I also do occasional consulting, shockingly not as Pay Me Lots Of Money And I Solve Your Problems LLC.

In addition to publishing pretty much live stats about the business, I always do a year-end wrapup (see: 2006, 2007, 2008, 2009) covering my thoughts on the year.  I hope folks find it interesting or informative.

Disclaimers: Stats are accurate as of publication, but the year isn’t quite over yet.  Ordinarily the last two weeks of December are fairly slow, but I would expect there to be a few hundred dollars more sales and possibly a few hundred more in expenses, depending on the timing of people charging my credit card.

My business is a good deal more complicated now than it was previously, which changes how open I can be about some bits of it.  See below.

The Big Change

After several years working as a Japanese salaryman, I quit my day job and went full time on my business as of April 1st of this year.  This was the best decision I have ever made.  Words cannot describe how happier I am with everything about my life: I see my family more often, I see friends more often, I travel more, I work less, I make more money, I’m healthier, my apartment is cleaner (well, OK, modestly cleaner), and I enjoy work a whole lot more than I ever did when I was working for somebody else.  Self-employment is awesome.  There are occasionally challenges to it, but since I spent a few years dipping my toes into the water prior to doing it full-time, I have had very little of the “Uh oh, I might not be able to make rent” uncertainty that a lot of folks report.

Bingo Card Creator

Bingo Card Creator remains my bread and butter for the time being, but I think this is likely to be the last year for that.

Stats:

Sales: 1,422 (up 38% from last year’s 1,049)

Refunds: 20 (down from 24 last year, to 1.4% of sales from 2.3%)

Sales Net Of Refunds: $42,589.90 (up 33% from $31,855.08)

Expenses: $16,685.24 (up from $15,485.28)

Profits: $25,904.66 (up 58% from $16,369.80)

Wage per hour: $200~250ish, based on my best guess of time spent on BCC (yeah, I went “full time” and work less than ever.  This is mostly because BCC is mature software.)

Web Stats:

(All stats are from bingocardcreator.com unless otherwise specified.)

Visits: 777,000 (up from 546k)

Unique visitors: 655,000 (up from 470k)

Page views: 2.7 million (up from 1.6 million)

Traffic sources of note: Google (44%), AdWords (20%), Binghoo (11%)

Trial downloads: 21,000 (down from 56,000)

Trial signups for online version: 72,000 (up from 17,000)

Approximate online trial to purchase conversion rate: 1.7%

Narrative Version:

The major change in BCC this year was unofficially deprecating the downloadable version of the software, for a variety of reasons.  This led to a huge cut in my support investment — the downloadable version generates 10 times the work per customer that the web app version does — which helped enormously when I dropped BCC into maintenance mode, which it spent over half the year in.  (Maintenance mode means I answer questions and collect payments and that is about it.)  I did a bit of experimentation over the summer in terms of conversion funnels, and also did some major work in October with regards to seasonal promotions and using my mailing list.

My conversion rate for BCC is slipping steadily.  This is sort of ironic, because it is the result of an unalloyed good thing for the business: as I get better at getting organic search traffic, the population of people using my web site moves from the teacher-heavy target-rich-environment it has historically been and towards a broader audience who don’t have quite such a pressing need for bingo cards.  Sales go up, since converting 1% of a big number is still a good thing, but it dilutes my aggregate conversion rate.  Oh well.

I’m mildly disappointed that I missed my sales and profitability targets this year (by about $2k and $4k respectively).  Oh well.

What Went Right:

• Deprecating the downloadable version reduces my work and stress level attributable to BCC enormously.
• Getting serious about using MailChimp and email marketing in general. My Halloween, Thanksgiving, and Christmas mail blasts made well over a thousand dollars in sales for me, for about $30 worth of virtual postage stamps and perhaps twenty minutes of writing.  One thing to note for next year: there does not seem to be a substantial difference in conversion rates for when I put a $5 off coupon in the mail versus when I don’t, so I shouldn’t next time.  Also, given that a huge percentage of folks bounce on the password screen coming from the email, I need to think about either putting a token in the URL to let them in automatically, or making dedicated landing pages for these campaigns.  (I don’t have good numbers for how effective the autoresponder sequence is — i.e. automated emails I send to people on the first and sixth days after they sign up.)
• Meat and potatoes SEO continues to be my bread and butter (how is that for a mixed metaphor).  My conversion rate has been gradually sliding as I get more parents in for holiday bingo activities through my Sprawling Bingo Empire, but 1% of a very large number is still worthwhile.  I should probably get serious about optimizing those sites individually, but my desire is waning.

What Didn’t Work So Well:

• In the wake of the FireSheep release, I decided to implement SSL for Bingo Card Creator, right in the middle of Halloween busy season.  This broke my pages in several different ways, from causing security popups on the login screen in IE (whoops) to not showing key images on landing pages in some browsers (whoops).  I really should have put off that implementation another few weeks, or thought through my testing strategy for it better.
• I don’t have a staging server for Bingo Card Creator yet.  Having seen the enormous advantages from having a staging server through my Appointment Reminder, I am retroactively dinging myself for not making one in the last four years.
• My Halloween promotion could have been handled better: $6,000 in sales is nothing to sneeze at, but I’m still of the opinion I could have broken $10k with a little better execution.  Maybe next year.
• AdWords has been on auto-pilot for virtually the entire year, and when it goes onto auto-pilot it seems to optimize for Google’s bottom line over mine.  I should block off some time to get it under control, and aggressively cut weakly performing aspects of my campaigns.

Consulting

Last Christmas Thomas Ptacek at Matasano (whose office I am in as we speak) suggested to me that people would pay for my expertise in software marketing.  I was skeptical, but put the word out quietly that I was available for hire, and did three large projects this year for a few clients.  The only one I can publicly comment on at the moment is that I worked for Matasano, on stuff.  My general field of expertise is in engineering marketing outcomes: A/B testing, SEO, and the like.  Basically, I bring the fanatical iterative improvement mindset and apply it to things other than bingo cards.

I love talking about what I do.  Unfortunately, consulting clients pay a lot of money to get me to shut up.  This means, for example, not blabbing on the specifics of new projects which are as-yet unreleased, and not blabbing on the particulars of engagements.  Since I had only a handful of clients, giving exact numbers tends to give a wee bit too much detail, since if you happen to know one data point then you can guess the other two fairly easily.  So treat these as very ballpark numbers:

Consulting sales: $45,000

Consulting expenses: $10,000 (Plane tickets and hotel rooms get expensive quick, what can I say.)

Consulting Profits: $35,000

I know somebody is going to ask my rate, and I’m torn between a desire to quote it and the knowledge that there is absolutely nothing good that can come of quoting it.  The reality for consultants is that clients pay you exactly how much you can negotiate with them.  Everybody knows this and yet everybody would be shocked, shocked to know that someone else got a better deal.  In addition to causing problems with existing clients, it complicates raising rates for new clients in the future.  (Given that I could fill 100% of the hours I wish to, have been saying ‘Nope, sorry, not available’ to interesting work frequently, and now have a few CEOs singing my praises, my rates are only going up next year.)

What Went Right:

• Client selection.  All three clients knew me well from the Internets, all had confidence in my ability to do what they wanted me to do, and all were model clients: they knew what they want, they communicated perfectly, and they paid all invoices in a timely fashion.
• Pricing: Aside from frightening my bank a few times when I got large wire transfers from America, charging a lot of money is a great idea in every possible way.  It makes your clients respect your time more, keeps you motivated, and helps pay the rent during the lean summer months of the bingo calendar.

What Didn’t Work So Well:

• Newbie consultant blues: I did my first consulting project for a friend.  Unfortunately, due to a combination of it being my first gig ever and my first experience with using Heroku, I greatly underestimated the amount of time the project would actually end up taking.  What I thought would be 20 hours over a two week period stretched into many more hours over months and months.  Luckily, my client was sympathetic, but I ended up doing a lot of writeoffs for good will and diverting my attention for longer than I wanted to.
• Juggling consulting with project work: I worked 90 hours a week and still had cycles to spare for BCC.  How hard could it possibly be to do 10 hours of consulting in a week and still get productive work done?  Very freaking hard, because the 10 hours tended to be splayed across several days, require creativity and focus to execute, and not include the whole email/sales/support dance that comes with consulting.  This is in no way a criticism of the client, it is just to illustrate how consulting works: I got a request from a C-level exec at a company you’ve heard of in April, and despite us being mutually enthusiastic about the project it took forty emails and five months until billable work actually started on it.  I had to gracefully extricate myself from my clients and block off November and December to get Appointment Reminder launched.

Appointment Reminder

I finally released Appointment Reminder in early December.  (Don’t worry, I haven’t forgotten about writing the rest of the series about developing it.)  I have one customer already and a handful of prospects currently trying the software out, so revenue is negligible as of yet.  Now I just have to do the other 90% of marketing the software, which I am going to do a bit of in December and then start in earnest in January.  The plan is, unsurprisingly if you know me, heavily reliant on organic SEO.  I have also had a lot of interest about whitelabeling the software, and will do that as well.  That gives me a built-in on-the-ground sales force of people with intimate knowledge of potential clients and the desire to sell them my service — and judging from the numbers thrown around by one of the guys interested in white labeling, that could be “quite lucrative indeed.”

In terms of technical direction, I’ve engaged a freelancer to get it working on iDevices (I could have done it myself, but I’ve got plenty on my plate as it is).  I am about 60% of the way to getting a very impressively hacky solution working on every major US smartphone, because many prospects have asked to be able to access schedules when on the move.  The implementation is so devious that if I had a mustache, I would be stroking it while cackling maniacally.

Semi-exciting news: I received an acquisition offer from a foreign telecommunications firm.  The CEO believed AR fit a hole in their product line, and offered me their estimated development budget for it if I would sell them the whole business.  That was a generous number relative to the amount of work I have put in, but it would not have been lifechanging for me or my family.  I declined and told him I’d like to try to run the business myself and see where it takes me.

The stats:

Sales: Nothing yet.  (Well, one test transaction to make sure Paypal/Spreedly works.  Spreedly is impressively painless, by the way.)

Expenses: ~$1,600 (a few hundred bucks in design work, $800 of Twilio credit, and one or two other things.  Servers and online services which I also use for BCC got filed under BCC because I’m lazy and my bookkeeping software doesn’t support splitting bills: a more accurate accounting would be closer to $3,000.)

What Went Right:

• It exists and mostly functions! These are both handy properties in software one wants to sell.
• MVP available for several months.  I didn’t have the cycles to create AR back in April/May, but I did get the MVP of it — basically, an interactive demo of the core of the service — out early.  This helped me in both getting a bit of age and trust on Google prior to the official launch, and also in getting a base of interested prospects to try selling to (I’m currently talking to a few of them).

What Didn’t Work So Well:

• Delaying release: April, May, June, July, August, September: that is six months, and I got very, very little accomplished (not one line more than the MVP, as a matter of fact).  The distraction from consulting work, working on BCC, and reacclimating myself to a human existence after years of salarymanhood just totally destroyed any desire to do heavy lifting on a new project.  I’m very obliged to the HN-based folks who started a Launch Something in November mini-sprint, which helped get me the energy to actually sit down at the computer for six hours a day five days a week and bang it out.
• Insufficient pre-launch marketing: I really should have been constantly adding new content to the website for the last six months.

Goals for 2011

Bingo Card Creator

• Sales of $60k, profits of $40k
• Getting AdWords back under control
• Getting the holiday promotions ready for all those domains I own but have never successfully exploited yet

Appointment Reminder

• 200 paying customers by December of 2011.  This implies the revenue run rate will be somewhere north of $10k then.  Cross my fingers: it might be well north of that, if SEO or white labeling works well.
• I’m about 70% certain that I’m going to hire a front end developer for AR.

Consulting

• Perhaps a wee bit more of it.
• At higher rates.
• Knock things out of the park for clients #1 ~ #3, and tell stories when possible.

See you all in 2011!

I worked for almost three years as a cog in a Japanese megacorporation, and one of the best parts about that experience (perhaps even worth the 70 hour weeks) was that they taught me how to be a professional engineer.  Prior to doing so, my workflow generally involved a whole lot of bubble gum, duct tape, and praying.  I spent a lot of time firefighting broken software as a result, to the detriment of both my customers and myself.  Talking to other software developers has made me realize that I’m not the only person who was never taught that there are options superior to bubblegum.  If you aren’t lucky enough to work at a company that has good engineering in its very DNA, you’re likely to not know much about them.

This strikes me as the industry’s attitude to source control a few years ago, prior to a concerted evangelization movement by people like Joel Spolsky.  It is virtually impossible to overstate how much using source control improves software development.  Our industry has changed in major ways since 2000, but our best practices (and knowledge of those best practices) are lagging a few years behind.  We could really use a Joel Test 2010 edition, for a world where “you should have build scripts for desktop software which can complete the build in one step” is largely an anachronism and where the front page to the website is no longer hand-coded in Notepad but, rather, is a shipping piece of software which can break in two hundred ways.

You’re not going to get the Joel Test 2010 here, mostly because I’m not Joel and there is no particular reason any company should judge its development practices relative to mine.  What I would like to give is some practical pointers for implementing three practices which, if you’re not already doing, will greatly improve the experience of writing software for the web:

1. Staging servers
2. Version control workflows
3. Tested, repeatable deployments

Staging Servers

What is a staging server?  The basic idea is that it is staging = production – users. (If you’re Facebook, Google, or IMVU, you are lightyears ahead of this article and have some system where there are multiple levels of staging/production and where you can dynamically change them.  You already have geniuses working on your infrastructure.  Listen to them.  This article is for people who don’t have any option between “code runs on developer’s laptop” and “code runs in production.”)

Why do we have staging servers?  So that anything that is going to break on production breaks on the staging server first.  For this reason, you want your staging server to be as similar to the production environment as you can possibly make it.  If the production environment processes credit cards, the staging environment processes credit cards.  This means that if, e.g., your configuration for the payment gateway is borked, you’ll find out about that on the staging server prior to pushing it live to production and, whoopsie, not actually being able to get money from people.  If your production server uses Ruby 1.9, your staging server uses 1.9.  If the production server uses memcached on port 12345, the staging server uses memcached on port 12345.

(Many folks have systems which exist on more than one physical machine.  I don’t — I’m a small business where 2 GB of RAM is enough for anything I want to do.  If you have multiple machines, strike “staging server” and read as “staging system” below: all the benefits for having a separate staging server are still beneficial when your staging environment actually has fifteen physical servers running 47 VMs.)

Setting up a staging server should be easy.  If it is not easy, you already have a problem in your infrastructure, you just don’t know it yet: you’ve cobbled together your production server over time, usually by manually SSHing into it and tweaking things without keeping a log of what you have done.  (Been there, done that, got the “I Created A Monster” T-shirt.)  There isn’t a written procedure or automated script for creating it from the bare metal.  If you had that procedure written, you should be able to execute it and create a staging server that works inside of an hour.

Most people won’t be able to do this if they haven’t given thought to the matter before.  That is fixable, and should be fixed.  It has substantial benefits: if you have a repeatable procedure for provisioning a production system, then when disaster strikes you will be able to confidently execute that procedure and end up with a production system.  (Confidence is important since you’ll probably be terrified and rushed when you need to do this, and rushed terrified people make unnecessary mistakes.)

If you’re working on Rails, I highly recommend using Deprec/Capistrano with all new projects.  In addition to making it very easy to get a full Rails stack working on your deployment environment of choice , it helps automate routine deployment and server maintenance, and has mostly sensible defaults.  (I have only one quibble with deprec : it installs software from source rather than using your system’s package manager.  That means that upgrading e.g. Nginx two years down the road is needlessly hard and error prone, when instead you could just have used apt-get in the first place and then updating is a piece of cake.)

You can also use Fabric, Chef, Fog, or a similar system to script up building new environments.  Pick whichever strikes your fancy.  Try to recreate your production environment, down to a T, on another host at your VPS/cloud/etc provider, or on another physical machine if you actually still own machines.  Keep tweaking the script until it produces something which actually matches your production environment.  You now have a procedure for creating a staging server, and as an added bonus it also works for documenting your production environment in a reproducible fashion.

One nice thing about keeping your server configuration in scripts rather than just splayed across fifteen different places on the server (/etc/environment, /etc/crontab, /usr/local/nginx/conf/apps/AppName.conf, etc) is that it lives in source control.  Your cron jobs?  If they’re in source control, you’ll have a written record of what they are, what they’re supposed to do, and why they just blew up when you bork the underlying assumptions eight months down the line.  Your Nginx config?  If it is in source control, you’ll understand why you added that new location setting for static images only.  The voodoo in your postfix config?  A suitably descriptive commit note means you’ll never have to think about reproducing the voodoo again.

After you have the script which will produce your staging environment, you probably want to make a minimum number of alterations from production.  Many companies will want their staging environment to be non-public — that way, customers don’t see code before it is ready, and critical issues never affect the outside world.  There are many ways to do this: ideally, you’d just tweak a setting on your firewall and bam, nobody from the public Internet can get to your staging environment.  However, this is a wee bit difficult to pull off for some of us.  For one, I don’t actually have a hardware firewall (I use iptables on each VPS individually).

My staging environment simply includes a snippet in Nginx which denies access to everyone except a particular host (which I can proxy through).  This breaks integration with a few outside services (e.g. Twilio and Spreedly, which needs callbacks), so I make exceptions for the URLs those two need to access.  The more complicated your staging server configuration gets relative to production, the more likely you are to compromise its utility.  Try to avoid exceptions.

That said, there are a couple that are too valuable to not make.  For example, my staging server has a whitelist of email addresses and phone numbers owned by me.  Through the magic of monkeypatching, attempting to contact anyone else raises an exception.  That sounded a little paranoid until that day when I accidentally created an infinite loop and rang every number in the database a hundred times.  (My cell phone company loves me, but folks who accidentally collided with test data sure would not have.)

How do you get data to populate the staging server?  I use seed scripts and add more data by hand.  (I also have DB dumps available, but they tend to go stale against the current schema distressingly quickly: I recommend seed scripts.)  You can also dump the production DB and load it into the staging DB.  Think long and hard before you do this. For one, it is likely to be way, way the heck out of bounds for regulated industries.  For another, your staging server is probably going to periodically be insecure — insecurity is failure and failure is what the staging server is for.  Slurping all of the data out of a staging environment has caused many companies smarter than you to have to go into disaster management mode.  Please be careful.

So you’ve got a staging server?  Now what?

At the simplest, you access your staging server with a browser and try to break things.  When you break things, you fix things, then you redeploy the staging server and try to break them again.  This is what you are probably doing right now with production, except that your customers don’t have to see broken things when you break things.

Eventually, you can script up attempts to break things, using e.g. Selenium.  Then when you break things, you add them to the list of things that Selenium tries to break.  If you run that against the staging server after every code check in (a process known as continuous integration), you’ll quickly catch regressions before they disrupt paying customers.  This is a wee bit harder than just having a staging server — OK, a lot harder — but you’ll get clear, obvious advantage out of every increment of work you do on this path, so don’t let present inability to be Google prevent you from getting started.

Version Control & Deployment Workflows

Everyone should use version control, but people tend to use version control differently.  Git is very popular in the Rails community, but there are probably no two companies using git the same way.  The key thing is that you agree with your team on how you use version control — document your assumptions, document your processes, then apply them religiously.  This will reduce conflicts on the team, reduce mistakes, and help you get more out of your tools.

There are a million ways to use version control and most of them are perfectly OK.  I’m going to mention mine, but it isn’t the canonical Right Way, it is just one way which works for a (very) small company.  Yours will likely be different, but you can see some of the things which go into design of a version control workflow.

Assumptions I Make About Life, The Universe, And Everything

1. I use git.  Git has notion of branches, tags, and remotes (physically distinct repositories) — if you don’t know what these are, Google for [getting started with git].
2. I generally work alone or with a very small team. (This assumption underpins very important parts of my workflow.  It won’t expand very well to a 200 man distributed team, but it might well work for 2 ~ 5 people.)
3. There is exactly one canonical repository, origin.  Developers maintain other repositories on their workstation.  Automated processes like deployment happen only with reference to the origin.  Code existing outside of the origin does not officially exist yet, for any purpose.  The history preserved in the origin is, in principle, sacred.
4. There is a branch called deploy.  The HEAD of deploy (the most recent code on it) is presumptively ready to be put into production.
5. Tags are used to take snapshots of the code base and preserve them in amber with a human readable name.  Right before we deploy to either production or staging, the HEAD gets tagged, so that we can easily find it later, with a simple naming convention (I use production_release_X and staging_release_X, where X just increments upwards — some people might prefer timestamps).  Production release tags are never deleted.  Staging tags get periodically culled when convenient to do so.
6. Development of any feature expected to take longer than a few hours happens on a feature branch.  (I do occasional work right on deploy locally, for issues of the “Minor copy edit on dashboard.” variety.  This would be one of the first things to go if I were working on a larger team.)

So how does this actually work in practice?  Let’s say I’m implementing a new feature.  I create a new branch to work on.  I code a bit, creating local commits with wild abandon any time I have accomplished something which I don’t want to lose.  When I believe code to be functional, I fire a capistrano task which tags the current head of my branch, pushes that tag to origin, and deploys it to the staging server.  I then continue testing on the staging server, for example verifying that Twilio integration actually works with Twilio (which cannot conveniently access localhost:3000 on my laptop).  I continue writing code, committing, tagging, and pushing to the staging server until the feature is ready.

Then, I switch back to the deploy branch and merge in my feature branch (with –no-ff, which creates a commit message just for the merge — this handily groups the twenty or thirty commits I just made into one easily readable story for myself later).  I then tag a production release (manually — this is entirely to force me to think through whether I’m ready for a production release), verify that there is no diff between it and the most recent staging release, and then push the new tag to origin.  I then fire the Capistrano task which checks out the new deployment tag and restarts the server.

What does this get me versus my previous SVN workflow for Bingo Card Creator, which was “Work only on one branch, commit stuff when I think it is ready, and deploy the trunk manually on occasion”?

1. I cause much less downtime for the production server due to reasons like svn commit -m ‘Whoops, forgot a setting in production.rb’ and svn commit -m ‘r1234 introduced dependency on foobar gem without putting it in environment file, causing rake gems:install to not load it.  Mongrels then failed to restart.’
2. My deploy branch has a relatively clean history, so when things start to break next year in production, finding the change sets which eventually caused the breakage will be less of a needle in the haystack search than finding them in SVN is.  SVN’s history is 1800 unedited commits, recording my stream of consciousness as they happened.  My stream of consciousness is frequently stupid, particularly when I’m panicking because the server is down.
3. This decouples the staging server from production in a clean fashion (so that I can advance the staging server a feature or three at a time if I want to), but guarantees that when I’m actually ready to deploy, I’m deploying exactly what did not break on the staging server.
4. Tagging releases gives you an Oh Crikey button, as in Oh Crikey, that last release broke stuff.  You can quickly rollback the deploy to a known good tag, isolate the changes which broke production, and fix them.
5. Deploy scripts manage releases with multiple moving parts a lot better than I do, even when I’m working from a checklist.

By the way, git gives you many options for recovering from problems — even severe problems — without requiring either gymnastics or a full-blown CSI investigation to discover what happened later.  For example, let’s pretend I just deployed tag production_deploy_82, and have discovered some issue serious enough to require an immediate rollback to production_deploy_81, which is known to be good:

#Assuming we are on our local workstation on the deploy branch.

git branch something-in-here-is-broken

git reset --hard production_deploy_81  #All changes made between deploy 81 and 82 just vanished from the deploy branch locally.

#Clean up the deploy, using any option discussed below.

git checkout something-in-here-is-broken  #Those changes which you just disappeared are now living on this branch, ready for you to fix.  After you've fixed and verified it works on the staging server (and, ahem, that you have addressed the issue that allowed this to get OKed for release last time), you merge this branch back into deploy, and do a tag-and-release cycle.

How you clean up the mess on the server is up to you: good options include “deploy 81 again”, “tag a release 83 equivalent to 81, then deploy it”, and “rollback to the copy of 81 which still exists on the server.”  (Capistrano includes deploy:rollback, which will do exactly this.)  Any of these will work, just always do it the same way to avoid stepping on each others’ toes.  I prefer tagging a new release so that I can add a descriptive message explaining why 82 just created an emergency.

This is important because it leaves a paper trail — if you’re pulling a release from production, something just went seriously wrong with your processes.  Emergencies are not supposed to happen – anything that lets an issue get that far isn’t just a one-off failure of whatever broke, it is a series of failures of the systems/processes designed to prevent failures from getting that far.  After you’ve put out the fire, investigate what went wrong and tweak your processes such that a similar failure in the future gets caught prior to bringing down production.  The sleep you save may be your own.

Scaling this to more programmers: Do whatever works for you!  I would probably create a staging branch and have folks integrate stuff into the staging branch when it was ready to go to the official staging environment.  I also might make per-developer staging environments: since creating one from the bare metal is supposed to be essentially free, let them all have their own where they can be reckless without spoiling the experience of other developers.  We can worry about code interaction on the “real” staging server.  Then, have folks communicate when they consider everything they have on staging ready for release, and release when everybody says it is ready.

The important thing is that, whatever process you use, you document it, teach it, and enforce it.

Stuff Your Deployment Script Might Not Do Today But Probably Should

1. Depending on your scale and how you use e.g. memcached, it might be safe to purge the cache on every re-deploy, which will prevent some hard to diagnose bugs.  At a certain scale, this is virtually a recipe for taking your site down in a cache stampede, but I’m not Facebook and having capacity problems means that I am probably already vacationing at my hollowed-out volcano lair.
2. Tell everybody on the team that you just deployed.  I know some teams who have an IRC channel with a bot who announces redeploys.  A quick email CCed to five developers also probably suffices.
3. Restart worker processes.  This is easy to forget but, if you do it by hand, you’ll eventually forget and then have two versions of the application in production at once.  If you’re not prepared for that, it will bite you on the hindquarters when, e.g., the application servers ask the workers to execute methods that the workers do not know now exist in the code base.
4. Do sanity checks.  You can go arbitrarily deep with complexity here.  For a first cut, mine for Appointment Reminder restarts the application server, counts ten seconds, then tries to access an internal URL.  If the application server isn’t up, or if the action at that URL blows up for any reason, the deployment script fails the deploy, rolls back to a known-good version, and sends me a very crossly worded email.  (You can do this for the staging server, too.)
5. Integrate with other systems which manage the state of your code/business.  For example, I use Hoptoad.  Hoptoad keeps track of exceptions and mails you when they happen, in such a fashion that your inbox doesn’t get buried by e.g. Googlebot deciding to do an impromptu fuzz test on your website.  I mark all exceptions as resolved every time I deploy to the environment they happened in.  You could also e.g. update an internal wiki by adding a new page specific to the deployment, automatically update your bug tracker to change the status of the bugs that you (presumably) just squashed, or start a new cohort for your stats tracking.

Is There Anything You Would Like To Add?

The trend towards openness with regards to technical practices on the Internet is a major, major win for everybody in our industry.  Best practices do not have to be passed from master to apprentice as oral lore.  Like OSS, since they’re often glue rather than competitive advantages (for many companies — not all), sharing them mostly doesn’t hurt you and can improve the situation of everybody.  So, in that spirit, if you’ve got anything you’d like to add, particularly for how you do things or how you would adapt this advice outside the scope of a very small business, I’d love to hear about it either in the comments or on your own blog (feel free to leave a link if it is relevant.)

Author’s note: This post is gigantic — over 10,000 words.  I don’t recommend reading this in one sitting — bookmark it and come back later.
//
Early this October, I was privileged to attend the Business of Software conference as both attendee and a speaker (see disclaimer waaaaay at the bottom of this post).  It was far and away the best conference I’ve ever been to.  You should go to it next year — I know I will, if I have to swim from Japan to Boston to do so.  (If you would like to but don’t think it is possible, see waaaaay at the bottom of this post.)

The featured speakers were, virtually without exception, remarkable.  The 7.5 minute Lightning Talks (one of which I presented — more about that after they post the video, you will be amused or your money back) were often entertaining and at least a few of them were packed with better detail than 15 Powerpoint slides have any right to be.

The phenomenal content of the conference, however, is just an excuse to get a few hundred ferociously smart people in the same hotel for three days.  I got to finally meet in person a few of the folks who either personally assisted me in my early days as a businessman or have been tremendously influential in my thinking — a short list of shoutouts includes Ian Landsman, Dave Collins, Joel Spolsky, Eric Sink, Eric Ries, Peldi from Balsamiq, and I could put another dozen names here without breaking a sweat.  One speaker at one point asked all attendees who had founded a software business to stand at one point, and by my eye probably 60% of the room stood up.  During breakout sessions, I heard from everyone from folks doing small software businesses like myself to a company which sells monitoring software for nuclear power plants.

The spirit of generosity at the conference was off-the-charts amazing.  However, many conversations were either by explicit agreement or common courtesy off the record — not everybody publishes sales stats, and that is their right — so I’m going to err on the side of caution and only repeat things I learned from the public sessions.  That said: beg, borrow, or steal tickets to it next year.  I can’t repeat that enough.  Although the conference swung to business topics far more often than technical topics, almost everyone I talked to both inspired and holding a notebook full of actionable advice to try.

My notes are incomplete and my memory is fallible, but here were some of the takeaways I got from the public presentations.  These are more a combination of my notes and thoughts than transcriptions of what was said.  If you want those, wait for the videos to be released.

Seth Godin

Seth posted a riff on his remarks on his blog.  The general tone and content was not surprising if you’ve followed him for a while.  (If you don’t, do so — aside from being a great communicator, he’s one of the better big-picture thinkers on marketing that I know of.)

Specifically, he says that the software business is undergoing radical changes because technical competence is no longer a scarce commodity: with open source tools, an increasing supply of trained programmers, and the explosion of cost-lowering innovations for creating and marketing software on the Internet, things which were previously the purview of a technical elite are now within what talented teenagers can cook up from their kitchen table.  The marketing side of the equation is also changing, since increased crowding means that the traditional model of paying salesmen to push software has devolved into using humans as inefficient spam bots.  (See his whole spiel about Permission Marketing here.)

A brief guide to evaluating opportunities:

• Who can I reach with my marketing message?
• Will they talk to people on my behalf?
• Can I earn and maintain permission to continue the conversation with them?
• Will they pay for my solution to their problems?

Perceptive readers will note that technical difficulty appears nowhere above.

Best four words of talk: “Ideas that spread, win.”  Software is now about creating a tribe or otherwise achieving leadership of one.  Take AutoCAD, for example: it isn’t a product, it is a movement: we architects who use AutoCAD are here to change the industry away from using pencil&paper drafting.  You are either in the movement or opposed to it, because it is an existential threat to your business, but you certainly aren’t neutral on the question.  Boring software creates no movements and has to be sold the old fashioned (expensive, ineffective) way.

Another example of movements is the trend towards measuring things, which was a recurring theme during the conference.  Toyota succeeded in changing the face of manufacturing because they got religion about statistics, and a similar sea change is finally happening in software.  This meta-trend is creating a lot of extraordinarily valuable companies directly and also making the rest of us leaner and meaner.  (Bingo Card Creator has a more sophisticated approach to metrics than many Fortune 500s.  That’s a shame, but also a tremendous opportunity for smart, savvy software firms.)

I was quite amused by a throw-away example of the difference between capitalists and socialists inspired by pin-making machines: prior to the industrial revolution, it took a skilled artisan a day to hone 5 pins (small bits of metal which one uses to, e.g., attach fabric to each other) by hand.  A pin-making machine lets any 5 people who can be trusted to learn a simple, repetitive set of instructions make 10,000 pins a day.  Socialists hearing this thought “Oh no, that is going to totally destroy the livelihood of pinmakers.”  Capitalists hearing this thought “Memo to self: be the guy who owns the pin-making machine.”  As the cost of software approaches zero, you want to be the guy who owns the machine.

Relatedly, with regards to your future employment prospects: draw a Venn diagram of job skills you have and skills which can be automated (or, by extension, anything which can be adequately explained by any amount of writing on paper, because anything that can be written down can get shipped to India).  Anything in the intersection represents career suicide.

Seth claims that no “competent” people work at his current software company, Squidoo, because “if we only needed competency, we can get that from [outsourcing]”.  You hire exceptional people and let them execute on things which require exceptional talent, and write down things which only require competency.  Low-wage grunts can do the competent stuff.  “Your job is a platform, not a list of tasks.”

Some personal thoughts: Seth was bearing on charging for software because of competition from OSS competitors, preferring instead to lead movements where joining was free (to maximize the size of the movement).  Happily for those of us who sell software, almost all OSS projects are incompetent at marketing, which matters more than technical execution.  Seth gave an example of trying to sell water: what are you going to do, make water wetter?  That is a wonderful example, since bottled water didn’t exist as a category 50 years ago and sales are going up every year, over a century after America solved the problem of giving everybody what amounts to free, infinite, drinkable water.  It amused me greatly to hear that bottled water was going away while attending a conference at a luxury hotel which sold $4 bottles of bottled water placed in the restroom literally next to the tap of ever-flowing unmetered municipal goodness.

David Russo

David gave a presentation about hiring at software companies, in particular about hiring to create and sustain a corporate culture as the company approaches scalability.  I regret I was insufficiently attentive during it, as this topic is several years ahead of need for me.

Briefly, there are five models for corporate culture in the literature:

• Bureaucracy  (the traditional BigCorp model)
• Autocracy   (Apple-style, where the company is an extension of the Will of Jobs)
• Engineering (Google, where the engineers run the asylum)
• Star culture (a large consulting firm or, perhaps, 37Signals, where individual contributors are expected to be stellar and you can put up with rough edges because, hey, rockstars)
• Commitment — a consensus culture with loyalty receiving strong priority (and, as you can probably guess by position on the list, the clear favored choice of the presentation)

There was an example of Westinghouse having a non-engineering CEO who proved to be disastrous for the company.  Afterwards, the board resolved never to hire a non-engineer again, because the culture of Westinghouse was ferociously engineering-dominated and when engineers talk to each other they break out the secret engineer decoder ring.  The CEO, who couldn’t speak the language or achieve respect among the engineers, was sidelined and failed at achieving his objectives.

Culture generally grows organically out of a business rather than being imposed from the top-down.  Changing a firm’s culture midstream is very painful, and Harvard studies suggest it has quantifiable negative impacts: 50% reduction in odds of an IPO, tripled likelihood of business failure, and 3% less growth monthly.  (The engineer in me winces at the compounding there.)

David’s advice was to invest in employees: hire brilliant people, and let them be brilliant.  The outcomes they achieve are more important than the work itself.

Dharmesh Shah

Dharmesh is CEO (most recently) of Hubspot, a marketing tool for small businesses.  They have raised quite a bit of money and, more importantly, have a lot of paying customers, consistent growth, and a revenue curve which virtually defines “up and to the right”.  You should be reading his blog already.

He presented a few techniques from Hubspot which were “interesting and not obvious”.  For example, successful SaaS companies which IPOed had a median capital raise of $42 million dollars.  It doesn’t match with our expectation that you can make a SaaS startup in your garage, partially because SaaS companies essentially offer their customers upfront financing (by spending their cost of customer acquisition in a front-loaded fashion and then recouping it over the 4-5 year lifespan of that relationship).

A key metric to keep track of: customer churn rate, because a 50% decrease in it doubles customer average lifetime value (LTV).  One of Dharmesh’s periodic hobbyhorses is that if you know LTV and know the cost to aquire a customer (CAC), then if CAC is less than LTV by a fair bit, you take as much investment as humanly possible, spin straw into gold, and walk away with money hats.  (This is far and away the best pitch for a software company taking VC money that I’ve ever heard.  See also the video of him speaking last year, if I remember right he covers it there.)

There are a few ways to measure churn: as percent of customers, as percent of revenue (better accounts churning faster than entry-level accounts: uh oh!), and “discretionary churn”, which is churn among accounts who are actually able to make the decision to cancel and are not, e.g., locked into a one-year contract with you at the moment.  However, overoptimizing on churn rates is myopic, because absence of churn does not indicate the presence of delight, and customer delight is a key to long-term success at marketing the product.

One particular way Hubspot maximizes for customer delight is by creation and use of a Customer Happiness Index (CHI), which is a backtested predictive measure of someone’s likeliness of churning next month.  CHI is dominated by three factors and adjusted as they get more data:

1. Frequency of use (more use = less churn)
2. Breadth of use (customers who use more features churn less)
3. Use of “sticky features” (there exist some features which are so screamingly better than alternatives that a customer who uses and gets value from them is very reluctant to stop using the service)

If I can call out #3 for a moment: many companies I have worked with have “sticky features”, although not all of them know it.  (Some day when I don’t have another few thousand words to write I’ll tell you about what they are for Bingo Card Creator and how I know.)

After you have CHI available in a transparent fashion throughout your organization, you can exploit it in a variety of ways:

• Tell your sales representatives to call customers with low CHI proactively and ask them what you can do to make them happy.  This simple technique saves 1/3 of canceled accounts, and is virtually free.
• Judge sales reps not just on sales but on CHI of new accounts.  (This counters the incentive to armtwist to sell to marginal customers.)
• Judge new features / policies / etc based on their impact to CHI.  Concerned about whether charging for consulting for setup is a good play for the business or not?  A/B test doing it versus not doing it, and optimize for CHI.  (I get the feeling that they might not explicitly be doing A/B testing… but they should be.  Sorry, hobbyhorse of mine.)

All meetings at Hubspot include a teddy bear named Molly, who is a stand-in for the customer (they have two customer archetypes: Margaret the marketing director and Olly the owner, collectively, Molly).  When they have contentious debates internally, they are frequently decided by someone saying “Hey, what would Molly want us to do here?”  Brilliant hack, practically free.

Hubspot sells services, primarily consulting for setting up new clients.  This represents a fraction of their revenue — only 7% or so — and decreases their gross margins, because consulting services are intrinsically lower margin than selling software products.  But “services are low margin… except when they’re not”: since getting people set up for success increases their CHI and wildly decreases churn rate, selling someone a few hundred bucks worth of engineer time at low margins can lead to thousands of dollars of marginal increase in LTV delivered by high margin renewals of the core product (which has 90+% margins).  Star this advice, gentlemen, because it has obvious implications for B2B SAAS startups.

So if you know customers and the company benefit hugely by increasing consumption of consulting services, why charge for them?  Answer: perceived value increases consumption, and perceived value is driven by cost.  When you give people 5 hours of free telephone support with an engineer, they — being successful businesspeople — find difficulty clearing 5 hours free in their schedule, fail to show up for the call, get distracted, etc.  When you charge them $500 for the call, they damn well show up.  (Patrick notes: why hello, recent discussion about Chargify free customers and support costs.)

In terms of explaining the value proposition of your software to customers, cribbing from Kathy Sierra: don’t make _____ software, make _____ superstars.  Make your customers awesome at their jobs / tasks, and they will pay any price and/or follow you to the gates of Hell.

Hubspot is big about being transparent to a fault within the company, particularly by using a company-wide Wiki which includes financial details, plans, and pretty much everything except salary information.  (Apparently, that being public is viewed as possibly toxic to morale without having obvious upsides.  Having financial details open to employee audit has obvious upsides: employees can tell you when you’ve let the entrepreneur reality distortion field go wild in the board meeting.)

Best sentence about branding I’ve heard recently: “Brand is what people say about you after you have left the room.”

Eric Ries

I will refrain from giving an in-depth summary of this talk because if you have read his blog you pretty much know the gist of it before.  If you haven’t read his blog, start doing so.  With the obligatory disclaimer about letting any methodology take over your higher brain processes, the Lean Startup movement (and some of the specific, implementable tactics it suggests) are so effective I sometimes worry about whether established competitors will try to illegalize them.  The best effect of this presentation was taking the idea away from the Silicon Valley echo chamber (where “pivot” has become something of a buzzword) and bringing it to an audience of people who largely sell things to people for money, but who still could benefit immensely from some of the Lean Startup lessons.

If you want an hour-long video introduction by Eric to the Lean Startup, you can see parts 1 and 2 of a previous explanation on the Internet.

Some specific lessons I haven’t heard before in a couple years of lurking on his blog or I thought were worth highlighting:

• The key takeaway: Lean Startup matters because the traditional model of development wastes people’s lives.  They pour out years of hard-charging startup workweeks making software which doesn’t actually improve the lives of any customer because it solves a problem no one has.  This waste of potential is criminal and can be fixed.
• Machine shops and software firms are self-regulating ecosystems if you do them right.  For example, if you use continuous deployment and five whys, when you start doing development too fast, you’ll get the brakes applied by having to make investments in quality control repeatedly because you’re making too many mistakes.  This lets you find the optimal pace of learning without having to fiat an inaccurate number (“You will write 10,000 lines of code a week!” — which as we all know means nothing, and definitely nothing good) from on high.
• The Agile methodology is a wonderful solution to the peculiar problems of internal IT departments at big companies, where the problem is extraordinarily well-known (“We need our existing payroll system, except on a new architecture where extending it doesn’t require sacrificing goats”) and solutions are unknown (“What is the Rails code to do this?”).  It has less to offer startups, where both the problem and solution are unknown.  (“Do people even need the thing we’re fumbling forward on building right now?”)
• Since the only people you can plausibly get to use a new product are early adopters, any additional quantum of work done to get the product ready for mainstream users is a form of waste.  (This is a bold statement with interesting implications.  I’m not sure I agree with it, but it sounds potentially very powerful.)

Scott Farquhar

The president of Atlassian had some thoughts gained from bootstrapping the company from 2 people to a $60 million capital raise and beyond.  Regrettably for my notes, this talk was structured as a List of N Things, which ensured that I wrote down N things and grabbed few supporting details.

1. Start with 2 founders.  For example, when their password database was compromised while Scott was off on his honeymoon in the wilds of Africa, knowing that his cofounder had the situation under control alleviated a portion of the anxiety, and after an emergency flight back to Australia Scott was able to let the cofounder get some sleep.
2. You need a business model.
3. Dogfood your products.
4. Measure everything.
5. Always be marketing.  Atlassian excels at this — for example, they had a charity give-away where they gave licenses of JIRA (a bug tracker) away for a song to small companies ($5 for 5 licenses or something to that effect), donated the profits to charity, and got massive good press from the deal while also increasing sales in the market segments which drive most of their actual revenue.  They also had very appealing marketing of another few initiatives: calling something the Stimulus Package, for example, was an excellent PR hook in the financial crisis environment.  I wish I had better notes here.  Memo to speakers: please do not autocommoditize your best points by numbering them.
6. Your first idea will fail.  Atlassian was originally going to market plugins for an obscure product from a company in Europe.  It turns out the bug tracker they wrote to make that product was more valuable than the product itself — shades of Fog Creek and CityDesk here.  (We heard this theme several times during the conference: Peldi later said that Mockups was originally intended strictly as a plugin for another software, and the standalone version which it grew into now dominates Balsamiq’s revenues.)
7. Long term thinking is good.
8. Know when to switch gears.
9. Build somewhere you want to work.  Many precious anecdotes here about, e.g., organizing a city-wide scavenger hunt for employees.
10. Experiences are more effective at motivating employees than and customers than money.  This comports with both social science research I’ve read and my own experience.  (I have recent experience of this at a company I consulted for, but since it is identifying, I’ll wait until they announce what we did together to go into it.  Still: it is amazing how effective turning money into an experience is at motivating people than awarding them the money directly, even though this is crazily irrational.  Japanese companies understand this better than anyone: “why pay people so that they can buy objects suggesting social status when you can just award social status directly”, which is the Rosetta Stone for so much of corporate life here it is scary.)

Jason Cohen

I was flagging by this point in the conference and a little terrified about my upcoming speech, so my notes aren’t that great.  Jason had a lot of amusing anecdotes from taking Smart Bear code review software from a one-man operation in a small room in Texas through to a sale to another company.  (Despite being told, repeatedly, that “Smart Bear” was unprofessional and the name would scare off big Fortune 500s, the company which bought him out later reorganized their portfolio of products under that brand name.)

One amusing story was about Sales Guru Frank.  He was a silver-haired slick talker who, for a mere 50% of the equity of the company, promised to sell Smart Bear software to the sorts of huge corporate clients which would bring it from a six figure business to a smashing success.  Jason had an absolutely priceless anecdote about how us engineers perceive the sales process: “You know, it’s when you’re in the boardroom drinking some expensive brand of alcohol and you start the conversation with ‘Hey, did you see the game?’ and everybody understands what game you meant and then, suddenly, sales happen.”  It turns out that sales is a little more complicated than this (for details, see the Paul Kenny presentation) and, thankfully, Jason avoided getting suckered by Frank.

Similar fun stories abounded but I was too terrified of my upcoming presentation to write them down (d’oh).

The single best takeaway was how to evaluate advice you receive from noted software bloggers.

There are two competing motivations for people who start software companies: wanting to maximize one’s financial outcome as quickly as possible (you want to be Rich) and wanting to sculpt the perfect personal niche in which you will be respected and enjoy your day-to-day work (you want to be King).  There are two fairly well-known markets for software: B2B (business to business) and B2C (business to customer).

This gives us the traditional four quadrant graph, which Jason decorated with some examples:

The take-away: only listen to advice given from people in the same quadrant as yourself, and you’ll save yourself a lot of pain and cognitive dissonance from getting conflicting advice.  (This is overstating the conclusion a wee bit I think, and people change on both axes over time.  That said, I’m definitely onboard with “all advice is a product of the circumstances faced by the person giving it”, but I think you can pull lessons from 37Signals in a B2C business or from Joel Spolsky regardless of whether you want to craft the ideal place to work or go big with a VC company or both.)

Sidenote: my business got mentioned from the stage three times, and I’m seriously honored and humbled about being put in that company.

Paul Kenny

Paul talked about the sales function — not just the sales job title — at software companies.  See his talk from last year for details about that.

Why do so many salespeople suck?  Well, the typical career path requires:

• One year from hiring to get up and running with the particular company/market segment/product.
• One year to achieve momentum in building up one’s list of contacts and starting the sales cycle.
• One year to demonstrably fail to achieve results.
• One year for HR to convert the salesman from non-performing to “no longer with the firm.”

Given that it takes four full years to go through this cycle, you could see a salesman with an impressive resume of 3 ~ 5 year stints at name brand companies with impressive sales to their credit… and not learn, from their resume, that their sales skills are atrocious and their successes were actually the result of products which could be sold solely on the strength of their quality/marketing/brand even with the salesman impeding customer perception of all three.  For example, Jason’s Frank probably fit into this category.

Paul nonetheless believes that strictly depending on inbound marketing restricts your business to plucking the low-hanging fruit, and that going solely after low-hanging fruit is not a great business strategy.  (This would be an excellent opportunity to whip out the above quadrant map, by the way: if you’re in either of the B2C quadrants, congratulations, low-hanging fruit is your business strategy, and you can be quite successful with it.  Ditto 37Signals and many similar cheaper B2B SAAS apps, by the way.)

The point of speaking to customers, which is all sales is, is to reach a mature understanding of what your client values the most.  Then, and only then, you offer that to them.

Engineers routinely suck at doing demos for customers because they demo every bit of the interface, from start to finish, going through all the little knobby settings bits.  Nobody cares about these things.  Demo the bits of the software your customers find interesting.  When you email customers, talk about the bits of the software your customers find interesting.  When you blog about the software, blog about the bits of the software your customers find interesting.  But above all things recognize that your customers find themselves interesting and the software boring unless the software makes a demonstrable improvement in their lives.

Quick tip when speaking to customers: “The most interesting people you will ever meet are the people who are most interested in you.”  The average sales rep spends about 47 seconds in talking about customers prior to talking about the solution.  That number should be increased, greatly.  On the plus side, because the industry is so comprehensively screwed up here, even minor improvements provide an opportunity to surprise and delight customers.  “Goodness, the CEO of that company spent five whole minutes talking to me about how we go about our business!  They really care!”

(Sidenote: This also goes if you’re selling yourself as an employee to a company.  If you’re in an interview and talking about the employer, you are probably going to get hired.  For reasons beyond my ken they don’t teach this lesson in college.)

Key takeaway: The quality of your dialogue with customers is directly proportional to the quality of the customers you will acquire.  If you understand their needs better, you will close bigger deals with happier customers who consume less resources.

Customer faith in your product as a solution to their problem is directly proportional to how well they believe that you understand their problem.  This again counsels spending more time talking to them and asking perceptive questions, then repeating their own language right back at them.  If they call it a foo, you call it a foo, even if internally everyone knows it is “really” a bar and, after all, it implements IBarable in the source code.  Relatedly, you cannot tell your way into a dialogue with the customer, you can only ask your way in.

Customers have DNA: Drivers, Needs, and Aspirations.  You should be capturing your understanding of these as you talk to customers, or you aren’t learning what you need to learn to bring the customer and the firm to a mutually satisfactory relationship.

Some factors to consider

• Customer needs
• Timescale for implementation
• Scalability
• Integration with existing systems/processes
• Affordability
• Results

Customers have many priorities:

• Ego (underrated by engineers in my opinion… even those who own iPhones because they’re worth owning iPhones)
• Perceived gain
• Sense of belonging (“Nobody ever got fired for…”)
• Security
• Ease of use

If you think of a funnel of concerns prior to sale, starting at the top:

• The experiences customers have right now.
• Their business case or project which may benefit from your solution.
• The utility your solution can offer.  (Note to engineers: many of you stop inquiring here.  That is a mistake.)
• Options they have competing with your solution.
• What the company values in terms of outcomes, drivers, etc.
• Resources they have to solve the problem (i.e. talk budget last, not first)

Founders have an advantage in doing sales, even if they’re engineers who think they can’t do sales.  Nobody understands the problem domain like a founder, nobody understands the solution’s capabilities better than a founder, and nobody else can say “You can trust me on this because I build the bloody company” or inspire the same degree of emotional response in a customer that talking to a founder provides.  (This is so true in my personal experience.  I hear, over and over again, from my customers that they trust me because they know they get an answer “straight from the top” when they send me an email.  I hear that most often from people who never emailed me in the first place, or who emailed me with issues that could be easily dealt with by Level 1 CS.  This is one compelling reason why I don’t outsource customer service.)

Rob Walling

Rob writes Software By Rob, runs a stable of small software businesses as a one-man show (including, most amusingly to me, selling beach towels — he focuses just on software these days), and wrote a book I really enjoyed.  (Flip to the chapter about Virtual Assistants, it is worth the price of the book and then some.)  The slides and outline for his talk are online.

After review of statistics generously donated by several software companies, Rob is of the mind that the most important goal of your website is to bring people back for additional visits, because the conversion rate of your website is much, much higher among engaged visitors than it is among first-time visitors.  I didn’t write down the exact numbers, but I recall him citing DotNet Invoice (“written in Ruby on Rails… no, just kidding”), CrazyEgg, and the beach towel site among a few others.  Interestingly, a few of the examples had higher average order values for returning visitors, too.

I only investigated this for about 30 seconds, but I have lower conversion rates to the online free trial of my software from returning visitors than first time visitors.  That actually isn’t contrary to his thesis, though — it is an artifact of having a very scalable SEO strategy and customer acquisition highly dependent on high-converting organic and AdWords landing pages.  This is one of the threats of making decisions based on shallow understanding of what Google Analytics tells you is the “truth” of your business.  (One minor surprise: about 60% of my customers who convert from the free trial convert within 3 hours of starting it.  Nail that first run experience and nail it hard, guys.)

“The ineffective marketer asks you to buy too soon.”  A customer on their first visit could have multiple reasons why they are not currently prepared to buy your software:

1. They don’t have enough info about it.
2. They don’t trust you.  (Oh goodness, this topic is worth a book.)
3. They don’t have money to buy it right now.  (A particular issue for a $300 invoicing software.)
4. The don’t need the software right now.
5. They’re never going to buy this.

You can recover from all the issues but #5 if you establish a relationship with them.  Permission-based email marketing is the best way to do this, bringing them back to your website via a communication channel everyone has and uses.  Repeated, trust-building contact with them will gradually wear down their resistance to parting with their cold-hard cash.

Email is far superior to competing methods of repeatable contact with customers — such as Twitter, blogs, RSS feeds, or your favorite social network — because it allows “personalized broadcasts”, where you can scalably communicate with an arbitrarily high number of people while also making it feel like everyone is getting individualized attention.  (Not nearly enough people use this to its maximum potential, particularly with SaaS.  Free tip from me: note what search term sent them to your website, put a variation of it in the subject line of the email, watch your open rate go through the freaking roof.)

To get the maximum value out of emailing your customers:

1. Have a killer landing page laser targeted on the “give us your email in return for this immediate benefit to you” call to action.  Give them a whitepaper or similar resource in return for the email.  It greatly, greatly will increase your conversion rate.  (Rob is not a fan of free-to-use trials.  That is an interesting idea to me, as I come from a “of course the software is free to try out” background.  He has achieved truly eyepopping conversion rates to an email submission — 40%+ on some pages.  I can’t get that to my free trial, which also requires an email address, and I have pages which are very, very compelling at “selling” that to customers who were looking for exactly what the free trial offers.)
2. Give something away in return for the email address.  See the slides for specifics, but this has a monumental ROI.  (I have some confidential numbers from consulting clients.  Suffice it to say this is right on the money.)  Interestingly, if you just call a white paper a “report” it has higher perceived value.  (News you can use!)
3. Set follow-up for emails.  MailChimp, my mail provider of choice, calls this an autoresponder sequence.  If you take one thing from this presentation, learn and use autoresponders.  They will change your email marketing life.

Some excellent tactical suggestions:

• Spam: Don’t spam!  Don’t mistakenly get classified as spam by hitting filters.  You can pre-test your emails against common filters, like SpamAssassin, to reword against accidental false positives.  (A long time ago I worked on anti-spam systems for a living.  Oh, the hilarious stories I could tell you.  Suffice it to say that a home builder might want to avoid promising that they could “increase the size [of your patio]”.)  Incidentally, MailChimp will do this for you for a nominal per-email fee.
• Some words kill open rates if you use them in subject lines.  Reminder, Special, % off, and “Help [us]…” are all culprits.
• From Name: your own personal name is best, and role names are a bit worse, company name is a bit worse, and webmaster@example.com is worst of all.
• All advice on emails can be A/B tested!

Peldi (Giacomo Guilizzoni)

Peldi runs Balsamiq, a small software company which has become practically synonymous with low-fidelity mockups for software.  He wrote the best blog post on the Internet about seeking coverage for your startup.  You should execute on it.

The theme of the talk was worries: worries you shouldn’t have, worries you will have, and what keeps Peldi up at night these days.

Things you should not be worried about:

• Asking customers to pay you money.  (Both Peldi and I were shy about this prior to starting, apparently, because we had the genetic flaw all engineers have that says that there might be a bug and, therefore, it is too early to accept appreciable amounts of money for the product.  I promise to do my part and say “Charge more” to any person that will listen to me until we have cleansed this from our collective gene pool.)
• Pirates.  (Not an issue in practice, you can pretty much fire-and-forget a low-pain DRM system, and since if you’re smart you’re doing web apps anyway this is disappearing from the radar screen entirely.)
• Picking a niche which is too small.  (Peldi put up a screenshot of Bingo Card Creator here, to the widespread amusement of the crowd.  I later heard more than one attendee say that they were under the impression it was “a funny Italian joke” until I gave my lightning talk.  That actually worked out great for my talk’s reception.)

Early worries in the lifecycle of his company:

• When to make the jump from full-time employment to starting your own business.  If I recall the talk correctly, Peldi scrimped and saved for a while, sold his shares of Adobe, and emptied his IRA so that he would have a year of runway in hand prior to going full-time on Balsamiq.  He also had written code on nights and weekends for several months, so he was not doing a standing start.  (And his rise was truly meteoric after that — a combination of product quality and being a marketing genius.  If you don’t understand why he is a marketing genius, go back and read that post I cited earlier.)
• Work/life balance.  Regrettably, you’re going to have to get your family on board with losing you for a few months while you disappear into the Bat Cave and hammer out your software.  (This is my one bone to pick: no, you do not.  You can virtually always stretch out your schedule.)  Peldi then delivered the best line of the conference: “If you work while they sleep, they won’t know that you’re ignoring them.”

Many people are under the impression that Balsamiq was an instant, overnight success, largely based on Peldi’s embrace of radical transparency and publishing revenue numbers.  Peldi had a series of very interesting graphs: the first shows the hockey stick revenue curve that competitors salivate over.  Then it zooms out to include the time while the software was getting written (i.e. no sales).  Then it zooms out to include his previous career at Macromedia/Adobe where he learned to polish software to perfection in the way that separates Balsamiq from its many, many competitors.  Then it zooms out to include his programming career starting at age 12.  Overnight success only took thirty years.  In a conference with many profound insights, I think this was one of the best ones.

Let me give you an example from my own experience: I recently received a wire transfer from a consulting client, for an amount of money which would be unexceptional to some people and which is staggering for me — more than I’ve ever had accessible in my entire life, by a factor of lots.  It was for two weeks of work and was roughly equivalent to what I used to make for six months at my previous day job.  The story for how I got to do those two weeks of work starts out with participation on the Joel on Software discussion boards some four and a half years ago.  I started dabbling in the topic the work covered almost six years ago.  The focus on communication which made it possible to sell the client (and eventually, their team) on the importance of the work getting done can be traced back probably to middle school (when I took up public speaking to get over a speech impediment).  You literally needed an entire lifetime to prepare for what you’re doing today.

Back to Peldi: don’t be afraid of an inability to deliver.  You’re a talented engineer, you’ll figure it out.  If you fall in love with the problem and can discuss it with boundless energy, then every interaction with customers will make your product better.  At early stages in the product lifecycle, feedback from customers is a much better thing to capture than mere money.  (Peldi gave away licenses like they were candy during the first few years.  Again, read the above blog post, in addition to getting great feedback this was a great PR and linkbuilding hook.)

Peldi had an extended discussion of how you can use Obama’s debate script as a model for addressing your fear of confrontation with customers.  While I’m not a fan of his politics, I agree, he is an excellent communicator.  There was a flow-chart involved, but the core is:

• Thank the customer for taking time out of their busy life to communicate with you.
• Empathize with them.
• Apologize if they’re ticked off, even if they have no right to be.  (Oh, crikey, how much do I agree with this.)

Lightning Talks

All of the Lightning Talks were delivered by non-professional speakers.  They followed a murderously difficult format: you get 15 slides which get auto-advanced every thirty seconds, and that’s it.  You might think that means you can get away with slapping together something in an hour: oh no.  In discussions with my fellow lightning talkers, we agreed that there has been something of a “lightning talk arms race”: the two talks I was most impressed by took over 24 hours to prepare, and mine took at least twelve solid hours over two months, with probably half of that being rehearsal until I could literally count out thirty seconds with a prepared spiel delivered in a voice other than my own.

I’m going to refrain from talking about my own lightning talk.  Why?  First, it relies on a “reveal” for a great deal of the impact.  Second, it was, far and away, the best speech I’ve ever given (out of thousands — some people play sports, I did public speaking).   The combination of heartstopping terror and the crowd reaction produced a strange alchemy which made the talk as delivered a lot better than the videos I made of myself performing it in a room for my webcam.

I’m told it was recorded and the video will be made available to the public.  I’ll tell you when it is ready.  I promise you, you’ll enjoy that a lot more than you’ll enjoy me posting the slides and a video of me saying virtually the same words to a wall.

As to the lightning talks other than mine, a combination of heartstopping terror about my talk and the exigencies of the format mean that my notes on them are woefully inadequate.  I expect that a few of the lightning talks will eventually be put online at some point.  Those videos will do them better justice than my hazy recollections, so I recommend waiting and seeing them.

Eric Sink

Eric presented the nuts and bolts of what happens if your company (or, in his case, a product thereof) gets acquired by BigCo (in this case, Microsoft).  Short version: a lot of pain and suffering, followed by a check so large that the VP of his community bank said putting it in his account would throw their company into absolute havoc because of the impossibility of putting so much money to work.  Let me emphasize the pain and suffering part, though.  (The low point: when the deal had consumed his waking life so much that Eric was reduced to taking a phone call from his lawyer while in church.)

One of the weirder points for Eric was that, with a cast of literally dozens working the Microsoft side of the deal, he never met or learned the identity of the person who actually made the decision to buy the product.  In a gang of people with confusingly similar job titles and descriptions, it seemed that “nobody was in charge.”  The deal was lead by a particular point of contact — we’ll call him Fred — who, as a trained negotiator whose full-time job is assimilating biological and technological distinctiveness, was a thousand times more prepared than any entrepreneur will ever be.

The process involves a long rigmarole of procedural steps.  You’ll want to see the full talk for the breakdown, because I’m hazy on it, but it involved multiple levels of due diligence, getting signed agreements covering IP rights from every person who had ever put a finger to a keyboard within a meter of the source code (cough start work on this today if you want to sell cough), etc etc.

While many entrepreneurs obsess about The Number, what you actually get is The Deal, which ended up being 150 pages of bullet points all individually negotiated covering every aspect of the transfer.  This included guarantees of treatment for employees who were relocating with their product from central Illinois to Microsoft, all manner of indemnifications, tax treatment of the deal, escrow (code and otherwise), the exact structure of compensation for the deal, the terms of the NDA covering material terms of the deal, etc etc.

The deal nearly fell through at several points, over contentious negotiating issues.  Nearly might not be strong enough: it was dead.  The deal was off, and the other side went to total radio silence for a period of weeks.  This was necessary to get the best achievable outcome for Eric and the team.  Eric is of the opinion that you need to walk away about twice: less and you’re getting screwed, more and you’re buying more heart attacks than the marginal improvement in terms is worth.  (One friend of his — in the elite fraternity of “people who have sold a company”, whose stories “you only become privy to after you have joined the club” — had to walk away five times.)

One interesting point raised in discussion was the importance of having multiple interested buyers so that you can play them against each other.  (Michael Pryor from Fog Creek is especially articulate about this one.  The short version: if your BATNA is “get bought by the other guy”, you have more leverage than if your BATNA is “you don’t get acquired, six months of work flushes down the drain, and the company potentially withers.”  The long version: ask Michael.)

Dan Bricklin

So here is a generation gap question for you: I didn’t know who Dan Bricklin was, and someone said “Well, duh, he’s the guy who wrote VisiCalc.”  I then had to Wikipedia VisiCalc because I had never heard of it.  For those of us who were born in the eighties or later: he invented spreadsheet software.  Yowza.

In addition to a hundred and one precious anecdotes about inventing spreadsheet software while a poor grad student (my favorite: the development machine and sole source repository, which cost the team’s life savings, being protected from a plumbing problem by constant vigilance with mops), there were a couple of generally useful takeaways:

• Customers have jobs that they need to get done.  Identify those jobs.  Build general purpose software which allows them to get the jobs done.
• The perceived value of general purpose software, which solves a variety of current and anticipated needs of a customer, is higher than that of software which solves one specific pain point.  (I respectfully disagree, but then again, I didn’t invent the spreadsheet.)
• Software which the customer can customize to their particular needs allows “tailoring at the ends.”
• The two-week payback rule: software sells itself if you can reliably demonstrate that your product will recoup the purchase price within two weeks.  VisiCalc (spreadsheet on a mini-computer) was so disruptively better than using a time-sharing system to do calculations that you could buy the software and the system to run it for just half of one month’s metered use of the big mainframe you leased time on to run your TPS reports.  That contributed to it selling 700,000 copies, back when distribution channels for software were stone age compared to the alternatives we have today.  (My unconscious brain was virtually screaming: “You can sell software without a website?!” when I saw the magazine ad slides.)
• Games and relative cheapness were very important in driving new technology adoption.  Note that this, and the “general purpose applications win” rule, are demonstrated in abundance by the runaway success of the iPhone.

Derek Sivers

Derek had originally planned on delivering a talk about various profit models for software companies (spiritually similar to this talk).  I met him for the first time at a speaker’s dinner.  During the course of dinner, one topic frequently returned to was why we do what we do.  Derek told a very moving story about how he sold CDBaby, his labor of love for many years, and someone (I think it was Rob Walling) suggested that he do that as a talk instead.  He ended up doing so.

Where to start:

Derek originally got into CDBaby to sell CDs of his own band’s songs.  Back at the dawn of the Internet, getting a merchant account for selling CDs was a multi-month process, with plenty of administrative hassle and pain involved.  Derek persevered and, with no programming background, managed to hack together enough CGI scripts to get a buy now button on his website in a mere three months.  (And again I bless my lucky stars that I had Paypal and e-junkie.)

After he started selling CDs, some of his friends with bands said “Hey man, that’s really cool.  Can you sell my CDs, too?”  So he’d hack the scripts again and then he was selling another CD.  Eventually, word of mouth had gotten to the point where his band’s website was colonized by other bands who lacked a good way to sell their CDs online, and CDBaby was born.  His blog covers this story in detail.

Derek was once interviewed by NPR and, when asked about his exit plan, said that long after the sun had set on CDs and when he was old and grey, he’d still be shipping CDs to the last aging person who swore that the CDs they played when he was young were far superior to that newfangled hologram 7D garbage that passed for music in 2050.

This turned out not to be the case: after a major rewrite of the CDBaby source code, where Derek achieved pretty much every technical goal he had for the site (and had paid back years of code debt), and after changing his management style such that he became totally superfluous in day-to-day operation of the company, Derek became an absentee owner of the company he founded.  Some people might view that as a wonderful outcome.  It turns out that Derek’s employees viscerally hated him for it: convinced that they were the true bearers of the CDBaby vision, rather than the unseen owner who many of them had never even met, they began organizing what could be described as a corporate coup.

When Derek found out about this (oh, the joys of reviewing MP3 recordings of meetings at midnight), he realized that he was well past the point of no return with regards to employee loyalty.  So he turned Apache off and started composing an emailed pink slip for the entire staff.  No, really.  Luckily for all concerned, he eventually decided to sleep on it, and came back to advice he had received from Seth Godin: if you are no longer passionate about the problem, you owe it to your customers to sell to someone who is.  Apache went back on and the email was not sent.

Derek pursued a few options for getting rid of the company.  One was to simply give it away: he actually pursued pulling off a Willy Wonka and putting five golden tickets in CDs delivered by his company, with the five holders invited to present their visions for the future growth of the company, and the winner (as decided by a vote of customer musicians) receiving the entire company.  Due to fortunate advice from friends regarding the desirability of being sixty and having no money, Derek threw away the golden tickets he had gotten printed and decided to do the more usual thing and just sell.

The sale came with a wrinkle: Derek first transferred the company into a charitable trust, then sold it, meaning that he is forced to draw 5% of the sale price per year, and when he passes away the principal and investment gains (if any) of the trust will be distributed to the charitable causes of his choice.  Derek claims this is mostly not for altruistic purposes: he asked his lawyer how to achieve “I want to have a stable income until I die, and that is all I need”, and the charitable trust was an attractive option.

Derek mentioned that having competing bidders was wonderful, since they could be played against each other.  This helped him achieve some goals he had in selling the company:

• I’m out.  (Founder participation as a consultant or employee for a defined term is a frequent requirement of sales.  Derek took that option off the table entirely as a precondition of sale.)
• I keep my database.  (A decade of running essentially a marketplace between music sellers and music buyers gives Derek a powerful foundation for future endeavors aimed at helping musicians, his perennial concern.)
• I keep helping musicians.  (The non-compete was narrowly tailored so that he would not sell CDs again, but could continue participating in the music industry.  Sometimes non-competes are are very broadly drafted — I’ve heard possibly apocryphal reports from one that, boiled to its essence, barred a technical founder from programming for the duration that a large company with a small-sounding name continued selling operating systems.)

Joel Spolsky

I’m pretty sure most people know Joel co-founded Fog Creek and Stack Overflow.

Continuing the theme of the later part of the conference about big picture questions like “Why?”, Joel’s presentation was probably the bravest speech I’ve ever heard.  Theoretically, it was about the story of transitioning from a low-growth (ish) software business like Fog Creek to  a high-growth VC-funded shoot-the-moon attempt like StackOverflow.  However, it dwelled on some intensely personal questions about leadership, philosophy, and intra-team conflict, and I don’t know if I could have discussed them at that level in public.

Let’s see: the brief takeaway on seeking investment is that if you’ve got a successful track record as a well-known CEO and a good idea which also has demonstrable traction, raising capital is fairly easy (Joel and company were practically booked solid when they went to Silicon Valley to meet with VCs).  However, it still has some of the industry seediness.  One VC who they met with, who ended up not investing in the company despite early promise, later tried his darndest to cause a rift between Joel and his co-founder (Jeff Atwood) so that he could pick up the pieces.  Another VC was quite interested in StackOverflow, provided it was CEOed by anyone other than Joel.

Interference from VCs aside, they eventually got signed by Union Square Ventures, who have treated Joel and company fairly.  The cultural gap between a self-funded software company and the VC world has caused a few disconnects (for example, at one point they found that StackOverflow had $200,000 lying around that they had forgotten to disclose based on revenue, and they asked what corporate structure should receive it or whether they could licitly award it to employees: the VCs were totally uninterested in $200,000, and rather amused at the notion that a company they had funded could actually have had a profit from operations at the stage they were at).

StackOverflow has a very different corporate culture than Fog Creek, which Joel describes as a software company founded with the goal of becoming an “intentional community” whose mission in life was providing programmers a great place to work.  It was explicitly analogized to a kibbutz, with collective ownership, the profit being redistributed among employees, and decisions specifically made to promote community (one which stuck with me was the huge importance placed on breaking bread together on a daily basis).

The culture involves decision-making by consensus.  For a variety of reasons, StackOverflow does not operate in this fashion.  This lead to some contentious battles between Joel and Jeff, exacerbated by the aforementioned evil VC, who tried to split the founders to pick up the company on the cheap.  Among other things, After frequent screaming and crying, Joel figured that his traditional style of hiring smart people and letting them get things done wasn’t working, and he called in a CEO coach for assistance.  He thought he’d get management advice on how to better run meetings and write org charts and do the things that CEOs do.

Instead, he got corporate psychoanalysis, and was introduced to the notion that the CEO’s job is not to manage the company, but to lead it.  This was particularly important as he was CEOing two companies simultaneously.  Once he started accepting that “I am the CEO, I have heard your viewpoints, we do it my way and that is final” was also a valid managerial style, conflict in the team ironically decreased greatly.

On the question of why we do what we do: Fog Creek is the company Joel designed to have a great place to work.  StackOverflow, on the other hand, is targeted at contributing one significant thing to the world: fixing the broken style of asking expert questions on the Internet.  They want to bring the improvement StackOverflow brought to programmers to the wider community (and communities) of interest who Google questions which have canonical good answers and find a SERP stuffed with 10 year old VBulletin threads where the answer is on page 7 in between a cat photo, a flame war, and the announcement of the forum’s guru’s goddaughter’s wedding.

Joel would prefer to work at Fog Creek, but thinks he has an ethical responsibility to make StackOverflow succeed first, because — irrespective of Fog Creek maximizing his personal enjoyment — StackOverflow has the potential to contribute a large social good to the world at large, and he views this as ethically mandatory given the opportunity.  He contrasts this to the 37Signals-esque notion of being content to run a small business because it allows you financial success and the lifestyle you enjoy (although he was quick to note that, while saying this, 37Signals gave the world Rails and teaching, so their own ethical bases are covered).

For an example, see CraigsList, which has moved a huge amount of wealth from newspapers to the posters of classified advertisements by making classifieds free.  CraigsList is run with a goal other than profit, and would instantly be a gigantic company if they charged for ads in categories other than the ones they do now (which they do as a spam-control device).  Joel argues that Craig should do this, because the massive wealth transfer his business enabled has redistributed the producer surplus from newspapers (which once used it for a socially beneficial purpose: underwriting unprofitable but socially beneficial local investigative journalism) to a consumer surplus for “people selling couches, landlords, and pimps.”  His point was that, when we know the trade-off our businesses are making, we have an obligation to choose the results wisely.

I am perhaps not doing this point justice: the two minute thumbnail sketch of the argument I heard Joel deliver profoundly altered my perception of where my company fits into the larger tapestry of my life.  I would previously have characterized, e.g., the decision to keep the company small as being essentially a personal aesthetic choice with no moral consequences, akin to picking colors for one’s bedroom.  I will be digesting the implications of this for a while, but I think I am now convinced that that is probably untrue.

A sidenote: since, pace Seth Godin, the cost of producing software is cratering and the competition is exploding, Joel is gradually coming to the belief that the value in a software company isn’t the code, it is the community.  The thing StackOverflow had to offer VCs wasn’t the code (which, to quote what has become an in-joke among HN readers, “could be written in a weekend”), it was the “large number of engaged users.”  Microsoft estimates there are 9 million programmers in the world.  StackOverflow has upwards of 8 million monthly uniques.  They’ve become the canonical go-to place for answers in the programming niche, and have dreams of becoming the canonical go-to place for answers for all expert queries which have canonical right answers, from tax advice to math theorems to gardening and many points beyond.

Commitment To Community & Disclaimer

(Back to top.)

Like I mentioned earlier, I strongly recommend you attend Business of Software.  Even at close to 10,000 words, this post captures a bare fraction of the value I got out of going to the conference.  I understand that both the ticket and the ancillary costs of attending are quite high for bootstrapped startups.  Trust me on this one: a plane ticket from Japan to Boston and four days in a hotel wipe out nearly a month of revenue for me — and it was worth every penny and then some.

Neil Davidson, the conference organizer, was very generous with giving away tickets to the event to deserving bootstrappers.  In the spirit of the overwhelming generosity of the community and the thoughts sparked by talk of our ethical responsibilities, I will likewise make an opportunity for at least one marginal person to go next year.  Watch this space — I’ll hammer out specifics closer to the actual event.

Now, the obligatory disclaimer: have never accepted advertising on my blog and don’t feel like starting anytime soon.  I was given a free ticket to the conference in return for presenting my Lightning Talk at it, and after winning the competition for best Lightning Talk (which I literally did not know existed until five minutes before it was announced that I won it), I received a Kindle and an invitation to a very nice dinner.  I’m easy to impress, but not that easy: my effusive enthusiasm for this conference comes from it being one of the highlights of my professional career, not because of compensation.

If you obsess over your YSlow and/or Page Speed performance scores, you might be wondering how to get your Rails app to automatically crunch Javascripts and CSS files without actually requiring cumbersome rake tasks to be repeated during every deploy.  Fear not: you can quickly monkeypatch Rails to do this.

1. Copy this gist into your lib/ directory.
2. Create a config/initializers/javascript_minimization.rb as described here.
3. Use your javascript_include_tag and stylesheet_include_tag helpers as normal, with the caching option turned on.
4. Watch your users save 100kb.

When I started this blog four years ago, I never thought it would be seen by more than a few dozen people, so I never planned any sort of information architecture to it.  500 posts (300,000 words!) and several hundred thousand readers later, it is unwieldy to get to the good stuff unless you have been following along for the last couple of years, or have a week free.  To make this a little less annoying, with the help of an assistant from Hacker News I found the best 72 articles I’ve written, grouped them by category, and for ones which are logically related explained what the connections are (in particular, for the “experiment” / “results” pairs).

I hope you find it useful.  If I let out anything or if there is a category you know I publish on but did not include in the list, please let me know in the comments.

One of the attractions of having a website is that it operates twenty four hours a day, 365 days a year.  However, depending on what your market is, it might not operate evenly on all of those days.  The phenomenon of market seasonality is well-understood in offline businesses — ice cream shops do most business in the summer, retailers have a 4th quarterly sales spike — but understanding of it seems to be fairly limited among software companies.  Since market seasonality has non-trivial impact on many software businesses, I thought I would write about what I’ve learned.

Brief Background

I sell software which makes bingo cards to elementary schoolteachers, who overwhelmingly purchase the software with their own money directly in advance of the day they plan to use it.  This gives me a very quirky calendar relative to many software businesses.  Collapsing my sales chart for the last several years over the calendar year produces…

Average Sales Per Month

As you can see, I have a trough of sales every summer and a spike in October, among other seasonal anomalies.  More about why that is later.  This is just the sales graph, but almost any other statistic for my business — page views, signups, ad impressions, search traffic, etc etc — more or less tracks this curve, although that isn’t going to be the case for all businesses.

Know Your Market

I came into this situation more or less blind: prior to starting my business, if I had thought about it, I would have expected “Yeah, sales are probably going to pick up when the school year starts”, but the huge spike in October and the mini-trough in November/December were all surprises to me.  If possible, don’t be surprised.  Presumably you are expert or near-expert at your particular problem domain, and should be accustomed to what your business calendar probably looks like in your market.  If you aren’t, time to make a courtesy call to either customers or competitors and ask them.  (Sure, you can ask your competitors this.  Why not?  If you’re worried about being laughed off the phone, then find someone who is publicly traded and look at their quarterly earning reports.  The numbers and explanatory notes will often give you all the information you need.)

For highly seasonal businesses, get used to the words year-over-year.  As in, sales in July for me were up 42% year-over-year, meaning that they were up 42% as compared to last July, as opposed to comparing them with this June.  This gives you a much more accurate picture of where the business is going, because for businesses which are not growing at truly meteoric rates, seasonal effects can very easily swamp the underlying trend.  (If you’re a viral Silicon Valley darling, you may be able to discount this.  However, if you’re a retailer, it is virtually impossible to have a January better than your December.  In fact, if it happens, it means you probably had a catastrophe.)

Cash Flow Management For The Off-Season

When people stop buying what you sell, you’ll tend to have less money.  That sounds vacuous, but it has important implications for your cash management.  It means that you’ll need to model your behavior on the ant, not the grasshopper: when times are easy, save up for when times are going to be difficult.

There are other options besides just banking money made during the on-season.  For many businesses, support loads, marketing activities, and the like all roughly coincide with the level of sales.  The number of emails I send regarding Bingo Card Creator falls by roughly 80% during the summer months.  This frees up a lot of time and focus in your organization.  One option to keep the lights on for software businesses is to do consulting or similar billable work during the off-season and go back to the product during the on-season — I took advantage of that this summer, to avoid breaking the piggybank too badly.

Another option, very popular among offline businesses but not used much online, is to do seasonal sales.  For example, if you sell software to teachers, you could do a back to school sale in August and, essentially, cut three weeks off of summer.  I did a 20% discount one year from late July through mid-August, which helped cushion the blow a bit.

Many SaaS startups are currently scratching their heads wondering “Why does this matter?”, because they get consistent revenue on a monthly basis regardless of when the customer’s signup was, and hence can’t conceive of having a growing user population and falling revenues.  This is yet another enviable reason to sell software on a subscription basis.

Other Ways To Exploit The Off Season

If you find that your marketing, customer support, and what have you decline in the off season, and you don’t need to find someone to bill to keep the lights on, you have a variety of fun options:

• Deploying new features, particularly ones which involve significant engineering changes (like wholescale site redesigns or business model changes), gets much easier when 80% of the user population is not actually on the site.  In the event you have an issue, the number of people affected by it is likely to be much smaller, and you’ll have a few months to get the kinks out prior to sales hitting their peaks again.  When I released the online version of my software last year, conversion rates plunged by about 30% while I was getting debugging the software and (more importantly, as it turns out) the marketing message.  Since it was the dog days of summer, though, that didn’t end up costing me all that much money (although it was a sock in the gut to have my first year-over-year decline ever).
• For similar reasons, there is never a time like the present to start on a new project.  It might be worth doing one which either has a different seasonal cycle, or none at all, as this eliminates the cash flow challenge.  (On the other hand, having a built-in 3 month vacation isn’t the worst problem in the world.)

Communicating Seasonal Effects

To the extent that outside parties do not understand the seasonality in your market, you need to educate them.  Investors, for example, might assume if they come from a non-seasonal background that any dip in the “up and to the right” story means you are losing traction.  It is a virtual certainty, for example, that GroupOn is going to sell less coupons in January than they will in December.  Having an article on TechCrunch saying “Groupon Growth Finally Puttering Out?” would not be positive for them (well, to the extent that TechCrunch coverage matters to a profitable business…).

Here are two very different graphs:

And here’s the same data, presented in another light:

The first business looks like it has had mostly flat sales for the last year, and a recent slump.  The second business looks like it is killing it.  They are the same business.  In the rather unlikely scenario that I was seeking investors, I know which of these two graphs I’d put in my slide deck.

Other presentational techniques to use:

• Second derivative of (pick a metric: sales, signups, etc).  The story is “We’re growing faster all the time.”  (Note as you can tell from eyeballing the above graph this is not actually true for me.)
• Pick a metric where the seasonal effects are less apparent.  Conversion rates are wonderful for this, since often you’ll have an increased conversion during the off-season.  (If someone comes into an ice cream shop in December, that is because they really, seriously are in the mood to buy ice cream.  They’ll also find the place absolutely deserted.)

Plan In Advance For Peak Season, Too

The top of the year for educational bingo is in October, due to a massive influx of teachers wanting to play Halloween bingo at their class party.  (A brief aside: Halloween is considered a holiday of special importance to young children in America, but they aren’t given the day off, which means teachers want to do a fun activity in class to commemorate it.  It has also been mostly secularized, which means public schoolteachers are at no risk to their career if they throw a Halloween party, whereas throwing an Easter party is a wee bit tricky.  You have to do it without mentioning Jesus and, simultaneously, without mentioning that you’re not mentioning Jesus.  Most teachers just have a chocobunny and hope for break to arrive.)

When I was young and stupid, I started planning for Halloween in October.  That works very poorly for Internet marketing: it will take months for your holiday-specific content (such as my page about Halloween bingo cards) to start ranking in the search engines where you want it to be.  If you do AdWords advertising, starting a campaign right before a holiday is likely going to be disastrous, because a) if it gets plunged into approval purgatory you just missed your holiday (this has happened to me quite frequently) and b) your campaign won’t have time to build up the positive history which will get it the widest exposure and best costs.

For SEO projects, start several months in advance of holiday or season you’re targeting.  For AdWords, I’d recommend at least a few weeks.  These schedules can be attenuated if you have a national-level brand to fall back on or a built-in tribe of evangelists like 37Signals has assiduously cultivated, but in general, more time is better.  (If this sounds like a whole lot of work, that is only because it is a whole lot of work, but you can have stupendously disproportionate returns to seasonal promotions.  I set records every year at Halloween even without ranking #1 for the term I was most interested in, and now that I am ranking #1 I am cautiously optimistic that I’ll absolutely smash my records this year.)

If you have another facet of this you’d like covered, or if you have some tactics which have worked for you, I’d love to hear about them in the comments.

Some months ago when the 99designs fixed-price logo store launched, my buddy Thomas at Matasano remarked that it would have been the perfect choice for my business.  I said that, while I’m quite impressed with 99designs’ business model and many of the logos on offer, I thought the less-generic-looking logo which I had custom-made fairly cheaply (by the folks at Logo Samurai — I bought three logos and chucked the two I liked least) was better than a generic logo.  Thomas disagreed, and said that the more “professional” look of the off-the-rack logos would offset the genericness.

If Thomas and I were bigwigs at a multinational firm with entire departments devoted to preventing anything from getting accomplished, that disagreement might have been settled by a great deal of harumphing.  However, we’re engineers, so we made a wager: I would purchase a logo handpicked by Thomas and A/B test it against my present logo sitewide, and whomever’s logo lost would donate $100 to a charity named by the victor.

The Original Logo:

The 99 Designs Logo:

Technical Details

This A/B test was done sitewide — on first access of the site, via landing page, organic search, or any other source, users were randomly assigned one of the two logos to be the “official” Bingo Card Creator logo.  They saw the same logo for all subsequent accesses.  This was done with A/Bingo, the Rails A/B testing framework I wrote.  This particular test required A/Bingo to be slightly extended to ignore bots, because — since the test was sitewide and visible prior to a bot-blocking login screen — the deluge of bots this site deals with was drowning out the signal with their non-converting noise.  Happily, since A/Bingo is open source, all other users of the software benefit from Thomas and I fighting over geek cred.

We agreed to measure conversions to the free trial rather than conversions to purchase.  Measuring conversions to purchase would have been technically feasible, but since much of my market is out of school during the summer, getting enough data to be statistically significant would probably have taken months.  Conversions to the trial are much more frequent so it was reasonable to expect sufficient data in a week or two.  The test was allowed to run for two weeks.

The Results

My original logo ended up winning by a virtual mile: 11.05% of all users seeing the classic logo converted, versus 9.92% of those seeing the 99designs logo.  Just by eyeballing that you might think “Hmm, 1% difference, that’s probably just noise”, but when you factor in the sample sizes (8,500 real people saw each variation) it turns out that the difference is statistically significant at the 99% confidence level.

(If those conversion numbers strike you as fairly low, remember that the test is running sitewide rather than on a single landing page.  Many folks who visit the site are drawn to free activities created for SEO purposes, and downloading those activities doesn’t necessarily require signing up.  Conversion rates for my AdWords landing pages are closer to 25 ~ 30%.  Relatedly, consolidated conversion rate is a poor metric to make decisions on, but that is a discussion for another day.)

Even though in this particular case the A/B test did not succeed in identifying a profitable change to make in my business, I consider it a wonderful use of my time — it only required $99, a few minutes of time from my local designer (to put the logo on the background I use for my header without it getting aliased), and a single line of code.  I can’t possibly stress enough how valuable it is to have your A/B tests require only a single line of code.  You’ll test more frequently, you’ll test more diligently, and you’ll discover more valuable things about your business.

Implications For Other Businesses

Graphic designers: Some of you appear to be highly opposed to spec work, at least partially because you believe it to produce inferior work.  You might consider measuring the effect of well-designed logos (for any value of “well-designed”, for example, “uses a business model I approve of”) versus other logos.  I have the graphical skill of a molerat — I should be linking to this article on your blog, professional graphic designer, not writing it.

99designs: Don’t get me wrong, I love you guys.  My Appointment Reminder site uses an off-the-rack logo from your site that I’m very, very fond of.  I’m very, very open to the notion that there exist circumstances under which this test would have come down the other way.

Software companies: Your logo could potentially add or subtract 10% from enterprise value.  Bolded for emphasis because holy cow.  (10% higher conversion rate to a trial flows virtually directly to my bottom line.)  The majority of companies in our industry get a logo done early in their corporate life, generally exactly once.  For some companies, brand equity might be of paramount concern, and the risk of creating confusion in a two-week A/B test swamps even a 10% potential increase in conversion rates.  For those of us who are more transaction-oriented than brand-oriented, though, this is an easy, obvious, fairly inexpensive area to test.

(Obligatory “I passed Stats 101″ disclaimer: the significance test I performed says that logo #1 is 99% likely to truly be better than logo #2, rather than saying that logo #1 is 99% likely to be 10% better than logo #2.)