Archive | startups RSS feed for this section

Kalzumeus Software Year in Review 2014

I’m Patrick McKenzie, perhaps better known as patio11 on the Internet. Back in 2006 I created a side project selling software. This eventually morphed into something a little more. Since 2010, running this software business has been my full-time gig.

Every year, I publish a writeup of how the year went, what the statistics for the business looked like, and what I tried that went well or went poorly. This is partially for my own planning purposes (very useful, by the way — I recommend you do a writeup, too, even if you only publish it to your hard drive) and partially in the hope that other folks can use bits of it. You can read the write-ups for 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, and wow do I feel old.

(Hint for Googlebot: If someone is looking for Bingo Card Creator Year in Review 2014 or Appointment Reminder Year in Review 2014, this is the right place.)

What’s new about this writeup?: In previous years, I’ve been coy with the numbers for Appointment Reminder. This has always be mildly irksome for me, as I would prefer to have them here, but I had justified it because I was perpetually on the fence about perhaps taking investment money at some point, and having numbers publicly available would complicate that process.

For more thoughts on why, see below, but I’m virtually positive I won’t try to raise funding for Appointment Reminder, so I’m deciding to burn those ships behind me and run the business the way that I’d rather, and that includes being present in this writeup. (To the maximum extent possible compatible with my commitments to clients, vendors, and contractors, at any rate.)

Where are the expense numbers?: Way down below. I previously broke down expenses on a per line of business basis, but due to process changes with how we do bookkeeping, I can’t do that anymore. That breakdown was always kind of handwavy — for instance, I allocated 100% of our server expenses to BCC, which was true in 2006 but ludicrous today — so I’m just killing it.

Capsule summary: This was a pretty good year business-wise. I didn’t quite hit the goals which I had set out last year, due to some major reconfigurations of priorities mid-year, but I’m reasonably satisfied with it in most respects. I finally feel like Appointment Reminder is making sustainable progress. BCC is basically a vestigial project at this point.

My one major regret is that I was pulled in a few too many directions and, as a consequence of that, didn’t hit a shipping target for one of my products.

My businesses grossed roughly $200,000 in sales and produced roughly $120,000 in profit. (These numbers are slightly fudged for sensitivity of enterprise deals — see below.)

Disclaimer: For the first time in my life, I actually have really solid books for the business (thanks Bench — more on that later), but for continuity with my informal style for previous years, the following numbers do not match the books. They’re also my best guesses as of today, which is complicated, since we haven’t closed books for December yet. Treat these as approximations rather than audited financial figures.

The Year In Brief

Appointment Reminder finally got a bit of attention paid to it in a sustained fashion. For the last four years running it has theoretically been my main business priority. Practically speaking, though, I’ve treated it with benign neglect.

This year our monthly recurring revenue is up by about 35%, I’m no longer the sole person in the business, and I have a much better handle on where it is going than I did previously.

Bingo Card Creator continues to be in maintenance mode. Sales continued to fall off a cliff, largely due to a decline in traffic from Google/AdWords and my neglect of it, which may or may not be related.

Productized consulting produced my greatest regret of the year: the course on A/B testing which I’ve been working on for a year and took pre-orders for last December didn’t completely ship in 2014. As to why that happened, don’t worry, we’ll talk about it in a minute.

Since I didn’t feel right spinning up new projects with an existing one which was stalled, I did no training events, no new products, and no launches for old products, which caused substantially below-plan sales.

I did one small consulting engagement, even though I thought I was done with consulting.

I didn’t do any new angel investing this year, mostly out of lack of bandwidth, both in terms of attention and in terms of ability to write checks. I tried to continue helping out the folks at Riskpulse and Binpress. I also started working as a formal advisor (formal paperwork, actual duties, an equity grant, etc, as opposed to informal “Yeah, I love talking about this stuff, email me any time” that I do with basically any geek on the Internet) with a pair of startups, MakeLeaps and another which I can’t talk about yet.

Appointment Reminder

Appointment Reminder is a SaaS product which sends automated phone calls, text messages, and emails to the clients of professional services businesses, ranging from tutors through HVAC companies on our publicly available plans to nationally-renowned hospitals and service firm chains on our enterprise plans.

In principle we’re capable of providing services at anywhere from about $5k to about $100k per year to enterprises, but in practice, we haven’t closed a six figure deal yet. I want to someday, more for the merit badge than the money, but when we were very close to the finish line earlier this year on getting one, I had to withdraw due to inability to provide services in the required timeframes.

I can’t tell you the exact number, but pretend the largest check we’ve ever received (not in this calendar year) was for $75k, since that is roughly accurate. I walked it into the bank and was shaking because I thought they’d shoot me for attempting a robbery. They didn’t.

Before I get into the numbers, a bit of context about goals: My software business was a sideline to my day job prior to going full-time in 2010. I started Appointment Reminder literally days after quitting, and at the time I thought “For the moment I’m going to run this similarly to my existing business, without stressing too much about it or jumping in with two feet or taking investment, because I don’t know how life is going to change as a result of going full-time.”

I didn’t predict that I’d soon meet the most wonderful woman in the world and successfully court her (lifetime conversion rate on marriage proposals: 100%, booyah) or that I’d fall backwards into running a fairly in-demand software consultancy.

While this was happening, I often thought “I might try to take Appointment Reminder to The Next Level (TM) sometime.” I spend a lot of time on Hacker News and am peripherally involved in the startup community, and a lot of folks have wanted me to roll the dice on a funded startup with big put-a-dent-in-the-universe ambitions. At times, I wanted to want that for myself, but for the moment I was content to keep running my business in the traditional fashion. I work mostly on what I want to work on, take a day off whenever I feel like it, and optimize the business for quality of life rather than for any particular growth or financial targets.

To maintain optionality for taking funding, I have avoided speaking publicly about Appointment Reminder’s trajectory. Well, to be honest, that’s the excuse I’ve given. At least half of the real reason was that I’ve been disappointed with Appointment Reminder’s performance. If you had asked me in 2010 where I thought AR would be at in 2014, I’d have said “$500k a year or so.” It’s not there. I feel embarrassed by this — some people are accountable to investors for quarterly performance, and I feel an analogous sense of obligation to a lot of people who think I’m a smart guy and should, therefore, be running a business with a certain amount of scale.

Is it weird to say that is distinct from actually wanting the money? I wouldn’t mind a little bit more to establish more of a buffer for my family, but other than that, money qua money doesn’t buy me anything I don’t already have. It might as well be WoW gold.

As long as I’m putting fairly personal thoughts in the report this year, let me add another one: Appointment Reminder has been one long bloody slog. I have not frequently enjoyed the business until up to a few months ago. Why not? Well, a bunch of reasons, most keenly one which Peldi warned me about back in 2010: I don’t really care about the problem it solves.

I’m not the world’s most avid player of bingo, either, but with Bingo Card Creator I really sunk my teeth into doing low-touch software sales and marketing over the Internet. That was an enormously fun and engaging problem, even though making bingo cards is not. (The app proper has been described as “Hello World with a random number generator” and that’s only a tiny exaggeration. I have played perhaps two games of bingo in the last ten years. The only time in recent memory I used it myself was when a Redditor asked for anti-Bitcoin bingo cards, a request which I am unquestionably the best qualified person in the world to answer.)

I like that Bingo Card Creator has helped to teach millions of students to read, and I like that Appointment Reminder helps get patients to the doctor on time and puts the children of some of my clients through school (via increasing the revenue of their businesses), but these businesses don’t really excite me.

Peldi told me I should do a business which excites me. I told him that Appointment Reminder was a boring problem space but a great business, and that I’d find some challenge in it which was different than what I’d done with BCC. As it turns out, AR didn’t offer a lot of fun new challenges — it offered the same old been-there-done-that work that BCC had offered, and while doing BCC for the first few years was a form of play, AR was very much work. I often avoided things that would be clearly beneficial for the business just because they were a boring grind, to focus on other things, from the consulting business to my personal life to, well, far too many games of League of Legends.

I’ve recently sunk my teeth into systematizing Appointment Reminder’s sales operations and this is, for the first time in ~4 years, a fun problem to work on for Appointment Reminder. That said, it’s like maybe a 7/10 fun problem. I realized recently that, while I had been saying “Yeah, maybe after I grow Appointment Reminder a little bit and want a new adventure I’ll take investment for it and shoot for the moon”, but intellectually speaking I know what that would have to look like, and it commits me to working for another 5+ years on a product/problem/market segment which, on its best days, is 7/10 in terms of fun.

Life is too short to do that. I certainly do not intend on devoting 90%+ of my career for the next five years to Appointment Reminder, so I won’t take investment for it. (That would be unfair to the investors.) Given that, there’s no harm in sharing the numbers.

Appointment Reminder Stats

Appointment Reminder’s key stat of interest is monthly recurring revenue on our publicly available plans, since this is the most predictable revenue in any of our businesses and thus lets me make consequential decisions like “Move the family to Tokyo, where rents are 5X what they were in Ogaki” (that happened, more later” or “Bring on help” (that also happened, more later).

Here’s the graph since we launched in December 2010. This is an MRR graph, so when we get annual pre-payments (we offer 12 months of service for 11 months of cash up front), they’re pro-rated over the next 12 months on this graph. However, since we’re a cash-basis business, the revenue number for the year will be higher.

Appointment Reminder MRR graph

MRR as of December 2014: ~$6,500 (up roughly 35% YOY)

Revenue for 2014 (on the publicly available plans): ~$75,000

Both of these numbers are approximations — we have credit card charges which will happen between now and December 31st, and they’re fairly predictable.

Brief fun digression on how much I love the SaaS model: our oldest customer signed up in January 2011, weeks after we launched publicly. He has since paid us $1,334, $29 at a time.

Our presently available plans are on the pricing page. They’re primarily segmented by appointment count. This is a decent if imperfect proxy for value a customer gets from Appointment Reminder. It is imperfect because an HVAC company which saves an appointment can earn $2,000+ in revenue. A hair salon, on the other hand, only values an appointment in the tens of dollars.

Once upon a time we had a $9 Personal plan available, but the customers who signed up for it were pathological in every sense of the word — some cost hundreds of dollars in imputed support costs then demanded that we send out $5 rebate checks for “unused” appointments. We killed that option a few years ago — I should never have offered it, but I thought I’d be able to pitch it to productivity bloggers at Lifehacker and the like and thereby gain exposure to their audience which might include office managers at professional firms. That never materialized.

Our target customer for the publicly available plans is the office manager who would otherwise be doing the calls herself. I call her Office Manager Milly. She’s comfortable with basic computer usage but not a technologist, earns about $20k to $50k a year, is generally an employee rather than owner of the business, is usually in her thirties or forties, and she has independent authority to make the purchasing decision. She works for a professional services firm, like a legal firm, an accounting practice, an HVAC installation firm, a plumbing firm, yadda yadda.

This is distinct from a personal service firm, like say a massage therapy practice. I thought personal service firms would be a big chunk of our book of business, but they aren’t: they’re smaller, they often are operated in non-businesslike fashions (many hair salons have no one accountable for making sure people actually come to appointments, but Milly is accountable to the business owner — this is one of her core tasks), and their per-appointment value is so low that losing an appointment doesn’t really constitute a hair-on-fire problem for them. It is a hair-on-fire problem for Milly: she can often be in trouble with the boss if she forgets to send a reminder even if the client makes the appointment anyhow, because potentially hundreds or thousands of dollars of revenue was just put at risk.

Anyhow, our publicly available plans range from Professional ($29, 100 appointments) to Small Business ($79, 300 appointments) to Office ($199, 1,000 appointments). This will likely change early next year — I’m taking my own advice to charge more, and re-aligning those numbers with actual customer behavior rather than the numbers I guessed four years ago.

I picked the original appointment quotas half based on projections and half based on total guesswork. My rough, untutored impression was that a personal service provider (stylist, massage therapist, etc) might average five appointments a day. I then linearly extrapolated how many workers would provide services at various target firm sizes and then fudged the resulting numbers a bit to provide a volume discount and to have round numbers. Translation: our pricing is, basically, made up.

Most of our customers are on the Professional plan, which annoys the heck out of me, but it’s my fault. Since I was thinking personal services, where 100 appointments a month barely sustains a sole practitioner (it implies $3k to $8k gross revenue), I thought any sizable business would be forced to pay more meaningful amounts of money. It turns out that you can run a nice boutique law office with sales in the high seven figures or an architectural consultancy with millions in revenue on less than 100 appointments a month. Believe me, I know several examples.

Anyhow:

Paying account breakdown: 142 (as of today)
Professional: 106
Small Business: 31
Office: 5

Churn rates: Churn is the percent of accounts which paid us money in month N who did not pay us money in month N+1. As we’re on a card-upfront free trial model, this definition structurally excludes someone who fails to continue a free trial as a churned account. That is in keeping with the business purpose of the metric: churn measures accounts which we once serviced satisfactorily which at some point were not willing to continue purchasing services, where trial conversion rates measure our effectiveness at convincing people that AR is right for their business.

We track it on a per-plan basis, since our customer behavior is wildly different. As a benchmark for you, for B2B SaaS sold on a month-to-month basis on a low-touch model, 5% is on the high side, 3% is what you should shoot for, and 2% is best-in-class.

I rather suspect that our churn rates have gone down over time as our product has improved, but that is difficult to tell, partly because we churn rates are very volatile at the (small) number of accounts we have. Our churn rates since inception are:

Professional: ~7%
Small Business: ~6.9%
Office: ~3.8% <— very small sample, so this number in particular is noisy as heck

I’m mildly dissatisfied with these churn rates, particularly in the months where churn eats the MRR increase from new converted trials (as happens occasionally), but they are what they are. In a more mature business than AR, I’d spend a lot more resources measuring this and working on the causes of it, but AR’s bigger problem is not closing enough new accounts every month for churn to really matter yet, so I prioritized working on sales recently rather than working on churn directly.

We track reasons that people churn. I separate that feedback into reasons we could have addressed and those which were beyond our help. “You didn’t respond to my emails” or “I needed the software to do X” plug back directly into our decisionmaking. Unfortunately, the bulk of our cancellations are for reasons like “We migrated to a solution which includes your product as a feature” or “We never really convinced the whole team to use it.” Historically failure-to-adopt has been something that we only can weakly influence, but we’re actively working on that with concierge onboarding/customer success as one facet of the sales initiatives.

Knowing churn lets you calculate the lifetime value of customer accounts. (LTV = Monthly price / monthly churn rate.) They are, assuming you trust our churn numbers as roughly representative:

Professional: ~$400
Small Business: ~$1,000
Office: ~$5,000

Knowing these numbers let us budget for paid acquisition (though we don’t do any significant amount of it — I’ve experimented halfheartedly but haven’t found a channel that works yet) and prioritize which accounts get proactive outreach from our sales rep and how much attention (and, ahem, commissions) we can afford.

Free trial acquisition: Our primary channel is organic Google search, by a factor of lots. This is partially driven by us ranking well for generic terms ([appointment reminder], [appointment reminder software], etc), very modestly by the sort of scalable content creation that put BCC on the map (I was unhappy with the content quality of our initial experiment into this for AR and opted not to scale until I fixed the process causing the quality issue… and then simply ignored that topic for the next 3 years), and referrals from clients.

Many people who run professional services firms are themselves consumers of professional services firms. If a lawyer gets an automatic SMS from their accountant, they might ask the accountant “That’s amazing. I wish I had that.” The accountant tells them “Ask the office manager what we use for it.” She says: “It’s really easy to remember: Appointment Reminder.” They Google it, bam.

It wasn’t the world’s most inspired naming choice, but the $8.95 I paid for the domain name is probably the best ROI of anything I’ve ever bought. (AR is virtually guaranteed to be a mortal lock on the query [appointment reminder] due to the combination of the exact match domain bonus and the fact that most links to it naturally cite the name of the company. The fact that it is a .org is irrelevant, except in that the .org was $8.95 and the company which owned the .com wanted $30,000 for it.)

So given our very, very lackadaisical marketing, we have very low traffic and a fairly modest number of free trials per month:

Unique visitors (2014): 66k, of which many are existing customers. Yep, this whole business has less traffic than my most popular blog post from 2011. I know, the irony, right?

Conversion to free trial: Due to the high percentage of people visiting our homepage who are existing customers, I feel like the most representative view of the funnel I can give you is this one (pulled straight out of Kissmetrics):

Funnel numbers

Free trials (card up front) (2014): ~350
Trial conversion rate: ~42% <— Would you believe I actually had no screen to track this number, simply because calculating it was not straightforward due to technical debt? Yep, I know, embarrassing.

You’re thinking “42% conversion rate! That is very high!” For low-touch B2B SaaS which you can sign up to without a credit card, it is generally closer to the 2~5% region. We require a credit card, both to prevent abuse of the system (ask me about the time AR was used to violate a temporary restraining order by, essentially, proxying harassing phone calls) and because forcing people to take their free trial seriously means that the increase in trial conversion rate is greater than the number of accounts we lose due to requiring the card.

Against comparable companies with cards required upfront, I feel like 42% is a happy place to be, but not outstanding yet. Hopefully it will increase by having more human outreach available. Additionally, we lost a lot of trials due to non-responsiveness to CS inquiries earlier in the year when I was sick. As one customer accurately opined in an irate email: “How can I trust you with my business long-term if you won’t answer an easy question on day 1?” (I apologized. Nothing else to do, really: under the health circumstances I wasn’t going to further stress myself over $29 a month.)

Since someone always asks me: no, no, no, we don’t hope that Office Manager Milly signs up for the trial and then forgets about us. We send at least ~5 emails during the free trial and, now that we have a sales rep, have an actual person manually reaching out, too. If someone’s account looks abandoned, I eventually get in touch with them and then terminate it proactively if I don’t hear back. I have gotten more complaints about that policy than you’d think: “Hey, why’d you close that account?!” “Because you hadn’t logged in in several months, had no client data in the system, and did not respond to several attempts to get in touch. I didn’t feel right taking your money.” “We were getting around to it! Open it back up again!”

Enterprise deals: I still can’t tell you numbers with a high degree of resolution here. We cashed only a handful of invoices this year, from AR and from one quick consulting gig, and any level of granularity gets legally dicey. Let me ballpark it as $50k < X < $100k.

As to how enterprise sales works at Appointment Reminder? In lieu of repeating myself, I covered it the other day and nothing changed over the weekend, so check out that writeup.

Vanity metrics: Our non-enterprise customers had 141k appointments (double last year) and sent over 200k reminders, saving the equivalent of about four full-time people doing nothing but dialing phones and leaving voice mails under sweatshop conditions. We saved our customers several million dollars in aggregate that would have been lost due to missed appointments.

My favorite story on this, from the principal of a contracting firm, is that his daughter is going to Harvard on the Appointment Reminder scholarship (when you lose $2k on a missed appointment it adds up quickly). We’ve also got a small pile of notes from folks in all walks of life thanking us for getting them to their doctor appointment, chat with the immigration lawyer, cable company, and he like on time.

Appointment Reminder: What Went Right

I did some sustained A/B testing on AR for the first time earlier this year. The full writeup would make an already long post even longer, but suffice it to say that we rejiggered the signup process and had a ~50% increase in the number of free trials as a result. I’ll talk more about that test in the future.

I also did some long-delayed work on automated customer onboarding and first-run experience earlier in the year, with the upshot that people are more likely to succeed in getting up-and-running and hence more likely to convert. Probably. I don’t have really great historical numbers there, but sentiment from customers improved and I got less “This is confusing. How do I…?” inquiries, anecdotally.

I have, finally, started working with other people. Appointment Reminder presently has two people working on it part-time, though this is only for the last few weeks so we don’t have great results to show yet. My virtual assistant who has been doing CS for Bingo Card Creator for several years is getting spun up as tier-one CS for AR, and I have a commission-based salesperson doing a combination of outreach to inbound leads and customer success style management of people’s trials.

I built software to support sales — easily the most fun thing I’ve done in AR since the original buildout of the core functionality — which helps the above-mentioned folks do their jobs. I recently wrote about that in considerable detail.

To build out that software and get the team spun up, I had to actually sit down and document our business processes, which was a great opportunity to force myself to actually think through them. The discipline of doing this made me confront a bunch of very obviously not optimal decisions I had made, like the lack of anything even approaching a repeatable sales process for enterprise details, so now we have one. Hopefully we can execute on it a bit in 2015.

Appointment Reminder had rock solid reliability in 2014. We had no systemwide outages, for the first time ever.

We did have an exacerbation of the “server occasionally times out during a phone call” problem that I mentioned last year was a real stitch in my craw. Errored-out calls increased from “a handful a week” to “a dozen a day”, which lit a fire under me to finally spend time investigating and fix things.

It turned out to be a combination of poor configuration choices for the MySQL server and the lack of an index for a particular query caused by our auditing system, which wasn’t a problem back when there were only thousands of audit records, but wasn’t quite so ignorable when it got to millions of audit records.

That issue is now fixed. I haven’t succeeded in completely eradicating timeouts yet. We dropped three calls in December. That is three too many.

Appointment Reminder: What Didn’t Go Right

Health management / burnout: Virtually all of the good news for AR this year happened between July and December. I accomplished very little which meaningfully drove the business forward in the first half of the year. This was due to a complex cocktail of stress, illness, and competition with the conversion optimization course for cycles (about which, more later).

This manifests itself, most commonly, as me fleeing from forward progress on the business and, on particularly bad days, avoiding my inbox. Avoiding my inbox never helps on stress levels, because I think that if I open it there will be more stress… and then it is suddenly Thursday and I haven’t checked emails since Monday. This caused me to deliver suboptimal customer service at a few points in the year — much better than last year, but still not where I’d like it to be.

I waited way too long to bring people on. Hat tip to Jason Winder, a good friend and fellow CEO, who finally managed to smack some sense into me. Here’s hoping that I can retain the fundamental character of the business while now having folks who I’m responsible to.

Due to having no process for closing enterprise leads and not enough brain cycles free to do the wildcat shoot-from-the-hip stuff which got us our existing enterprise customer base, our enterprise pipeline dried up early in the year and, as a consequence, we closed very little additional enterprise business.

Bingo Card Creator

Bingo Card Creator is a B2C SaaS application which is sold on a one-off transactional model (hint: don’t do this), primarily to elementary school teachers.

BCC was very firmly in maintenance mode. Substantially all customer support for it is done by my virtual assistant. I only get involved for refunds or harder issues where we have to do hand-fixes in the Rails console.

I worked less than five hours on Bingo Card Creator this year. It runs itself… not particularly well, but it runs.

BCC traffic, which is quite seasonal, peaked in 2012/early 2013 and has been on a steady decline since then. This decline accelerated in 2014. I assume it is due to some combination of the content on the website being stale (nothing has materially changed since, oh, 2011 or so?) and perhaps changes to Google’s algorithms, which I don’t actively keep abreast of these days.

Additionally, we’ve historically gotten a large portion of our traffic from AdWords. That also hasn’t been touched since 2011 other than buying Larry and Sergey a few more drink coasters made out of materials that can only be created in the Large Hadron Supercollider. AdWords is identifying less opportunities for our ads to be run profitably, hence showing them less, hence sending us less traffic (though costing us less in absolute numbers, so that partially balances).

I have done nothing to address the decline, as it is economically irrational to try to recapture those halcyon days of ~$5k a month sales when for the same amount of work I can hand off major portions of AR’s workload, cut my stress massively, and almost certainly make more money in the long-run.

BCC Stats

Sales: 1,128 (down 35% from last year’s 1,734)

Refunds: 15 (down from last year’s 57)

Sales Net of Refunds: $33,319.35 (down 33% from last year’s 50,156.16)

Profits: ~$20,000 (estimate based on best guess of allocation of costs to BCC which is consistent with last year’s number, down from ~$23,000 last year)

Wage Per Hour: ~$4,000 — I was checked out of BCC this year

BCC Web Stats

Visits: 500k (down 35% from last year’s 770k)

Unique visitors: 430k (down 32% from 630k last year)

Page views: 1.5M (down from 2.4M)

Traffic sources of note: Google (51%), AdWords (16%), direct (11%), BingHoo (11%)

Trial signups: 41,000 (down from 61,000)

Approximate trial to purchase conversion rate: 2.8% (up modestly from 2.6%)

BCC: What Went Right

There’s a lot to be said for having a business which runs totally on auto-pilot.

BCC: What Went Wrong

Traffic has continued to fall off a cliff, as mentioned previously.

I had collaboration issues with a consultant, which didn’t meaningfully impact business results (the trajectory on BCC is pretty clear), but which caused him unnecessary stress, which I regret. I had engaged Nick Disabato’s Draft Revise service to do A/B testing for BCC. My expectation with working with Nick was that I’d basically write checks and never have to think about the matter again. Nick wanted a more collaborative process with me, which I was not particularly unwilling to provide but which I did not prioritize actually making happen. As a consequence, I missed emails from him for months at a time, which inhibited his ability to do his work.

How disengaged was I from Bingo Card Creator? Nick and I got together for lunch last week and one of the things I wanted to tell him was that I wanted to dissolve the relationship amicably as I didn’t have time/effort/desire to manage it and he had previously expressed concerns about that. Nick brought up that he had actually stopped charging me several months ago and stopped work at the same time, and that I would have known this if I read the emails he had sent me. Whoops.

I’m glad he’s now freed up to work with better clients than me. They can actively engage with him on products which have a bright future ahead of them. That way, the insights he pulls out of the A/B testing process can actually get looped back into the product/marketing on a timely basis.

So if that was my level of responsiveness for someone working with me, how responsive do you think I was to customers? Yep, luckily my VA takes care of 95%+ of issues by herself, but the remaining 5% which got escalated to me (refund requests, etc) were often delayed by, literally, weeks at a time. I regret this — when I noticed it I apologized to customers, but that level of responsiveness makes me feel like BCC has become a business which is other than the kind I want to operate. I’m going to fix it or get it into the hands of someone who can.

Consulting

I had quit consulting last year, but had the combination of a wonderful opportunity fall into my lap at the same time as a move to Tokyo was in works. I can’t say anything about the engagement. Basically, same old same old, I built systems that helped make a software company a wee bit more efficient at getting a great product into the hands of more people.

The Move To Tokyo

So in lieu of that, let’s talk Tokyo. My wife and I moved from Ogaki, the small town in Japan where I had lived for 10 years, to Tokyo, as of a few months ago. This quintupled our rent on an ongoing basis and required substantial expenses to set up our new household, hence motivating me to do a quick consulting engagement.

Why move? The long and short of it: Ruriko (my wife) does not love Ogaki the way I love Ogaki, and wanted a change. That’s a good enough reason to do anything, but as it happened I was also feeling socially isolated in Ogaki (aside from Keith, I have no friends within the city anymore, and few enough close acquaintances), and I was ready to start a new chapter, too.

Why move to Tokyo? I wasn’t ready to uproot the family to the US, so looking around Japan, Tokyo is pretty clearly the best option for us. Ruriko has friends and family there (not immediate family, but I expect that to change), the QOL according to things she cares about is quite high there, and Tokyo has a rather substantial tech community.

I thought Tokyo was going to be… tolerable. I’m rapidly falling in love with it, or at least the little slice of it I’ve found in Nakameguro (our neighborhood). Jason Winder, a close friend of mine, runs a company in that neighborhood. We met when he started doing HN meetups several years ago — I used to go out 3 hours on the train once a month just to talk software with anyone — and, as we became friends and I ended up spending a bit more time in the neighborhood, came to think it represented a nice compromise of the advantages of big city living but with a friendly atmosphere.

I remember thinking years ago that Tokyo was just wall-to-wall overcrowded alienating hell, like the subways. The subways are indeed wall-to-wall overcrowded alienating hell, at least during rush-hour. I have solved this by making sure to sleep in until rush-hour is over and joined a coworking space within walking distance, so that I’m only on the subway twice a week rather than twice a day.

It has been great businesswise — in particular, Americans from Silicon Valley invariably hit Tokyo when they come to Japan but don’t come to Ogaki, so just by being there I get to meet with a lot of interesting folks who I would otherwise have to fly over the ocean to see. (I also hope to get a bit involved with the Japanese startup community, but have been rather busy lately.)

Productized Consulting

In addition to the blog (you’re on it), podcast with my co-host Keith Perhac, and occasional essay delivered via email, I have also made a few paid products which help software companies market and sell their software.

They include a book on conversion optimization, a video course on lifecycle email marketing, and occasional one-off training events on email, copywriting, productizing one’s own consulting services, and the like. I didn’t actually do any paid training events in 2014.

I have been working for the past year on a video course to teach software companies how to do conversion rate optimization for their websites and products. It has been, frankly, a long, slow nightmare of a project.

I originally opened pre-orders for it last year in December and anticipated delivering it by the end of January. That didn’t happen. The slip date shipped repeatedly [editor’s note: that spelling mistake was unintentional but I like it so much I’m leaving it in].

I accomplished partial delivery of the course in summer — it had been sold on a three tier model, with the first tier having access only to canned videos, the second additionally getting personalized advice about their websites in a group discussion format, and the third getting a mini-consultation privately. To apologize for the delay, I bumped everyone to the tier one higher than they had paid for, and then delivered the non-canned portion. This required a few weeks of scrambling to do mini-consults and give a few dozen companies conversion advice, but it was the right thing to do, and it kept most of the customers happy.

I suppose it is worth saying why delivering that was comparatively easy and delivering the course has been comparatively hard. Partially, expectations in terms of production quality for live events are much lower than they are for video, and I wanted this course to be produced much better than the last one I did (which was me talking into a webcam). The other factor is that there is a difference between doing and teaching, and (contrary to the old saw) teaching is actually sometimes harder than doing.

I have substantial experience with doing conversion optimization work and, shown a company’s (often-unoptimized) home page, trial signup funnel, and pricing page, the ideas flow really freely. That’s easy for me and I’m very, very good at it. What I am not an expert at is making other people good at it, too. Curriculum development, collateral (slides & etc), and actually shooting the course has taken far, far more time than I anticipated, and I’ve thrown out quite a bit of work because, after seeing the final product, it didn’t hit my internal quality bar. (Which is set rather high for this, partly in reaction to it being late and partly because I am “the A/B testing guy” and I want to make this some of the best work in my career thus far.)

Every time I get in touch with folks about this, I offer refunds for those folks who (quite reasonably) feel inconvenienced by continuing to wait on the course. Even so, I feel awful about it having taken this long. That has been one of my main stressors this year.

This has totally stalled my product development pipeline, as it were, for productized consulting. I had expected last year to do a few one-off training events again, because people were thrilled with them last year and the model very clearly works. I wouldn’t feel right breaking time off the course’s schedule to build, launch, and deliver those training activities, though, so nothing on that front has happened this year.

Productized Consulting Stats

Sales (Hacking Lifecycle Emails): $9,443

Pre-sales (Software Conversion Optimization): $2,988

Refunded pre-sales: -$1,588

Royalties (Sell More Software): $469

Gross Sales Net of Refunds: $11,312

Productized Consulting: What Went Right

I did a wee bit of tweakage to my email list such that folks get my favorite 5ish essays delivered in a sensible sequence after they sign up for it. This lets me get some mostly passive sales for Hacking Lifecycle Emails, as one of the essays has a brief plug for it. (Yep, prior to 2014 there was no lifecycle email to sell my course on lifecycle email. The cobbler’s children have no shoes.)

Keith and I found a good podcast editor, which makes getting new episodes up less of an ordeal than it was previously. Hopefully after the situations in our personal lives get a little more stable, we’ll do even more of them. I think, off the top of my head, that we recorded more in 2014 than any year previous. (Some are still in the can.)

Productized Consulting: What Went Wrong

I won’t belabor the point, but not delivering the conversion optimization course this year is, far and away, my biggest regret for the year.

As a consequence of not getting that course launch and having it block other things in the pipeline, productized consulting fell fall short of my financial goals for that segment of the business.

I have not written as much as I’d have liked to this year.

Some days I feel like I have a few mana pools of creative energy — one for programming, one for writing/speaking/podcasting, one for generic business stuff, etc. As a consequence of stress and this course hanging around my neck, the mana pool for writing has been bone dry for most of this year. On those few occasions where I’ve had enough left over to cast the spell of 10,000 Word Blog Post, I’ve found myself with nothing new to write about.

This is irrational, and I know it is irrational. What I should do is continue to write on topics that I’ve covered or work on in the past, because that provides incremental value to folks who like reading my work. However, I always use the burst of inspiration which covers conquering fun new challenges to fuel writing about those challenges, so when I have no fun new challenges being conquered, my writing cadence falls off a freaking cliff.

I’ve only published three essays to my newsletter this year, where I try to do it bi-weekly and I’m embarrassed when it is less than monthly. I don’t really love blog posts anymore as an expressive medium, and only do it when I have something which should go to a wider audience than my usual mailing list (like, say, a 10k word brain dump of everything I know about doing business in Japan).

Business Administration

I changed the way we do bookkeeping this year. Previously, I used homegrown bookkeeping software built into the Bingo Card Creator website to do my books. I then had my VA do them, by periodically dumping hundreds of pages of transactions from my credit card statements, emailing them over, and having her categorize them for me.

This was a poor use of her time. It is more important that customers get useful, friendly responses in a timely fashion than the bookkeeping get done on any sort of schedule. My VA did not particularly enjoy the bookkeeping work, which requires a certain level of obsessive-compulsive attention to detail (one reason why I am bad at it), and the process routinely produced errors. Occasionally, they were errors which — had I not caught them — would have materially affected our tax liability. If you make a one-digit error on the date in a single-entry bookkeeping system and file a $5,000 hotel stay in the wrong year, that costs me thousands in additional taxes.

As a result of this, collaborating with my VA on bookkeeping consumed an excessive amount of time relative to the value we were getting out of it. I was weakly attached to the notion of continuing automated expense transparency on the BCC website, but not at the expense of additional stress this year. So I looked for a better solution.

Enter Bench. They’re basically Bookkeeping as a Service, where an actual, honest-to-God human bookkeeper manages your books for you using algorithms as a lever rather than as a substitution for their work and expertise. (I previously used a few solutions which are designed to automate the process entirely, but they produced books which required so much manual correcting on my part that they were worse than having no books at all.)

Bench is structurally a simple app. They use Yodlee (presumably) to slurp transactions out of my bank accounts. I upload statements for those accounts where that is impossible, and receipts for those transactions which need them for substantiation purposes (receipts for hotel stays for business travel, etc). Graydon the bookkeeper does the books for me. The site has an interface for passing messages to them and annotating individual transactions with a note, like “That transfer to a German tech company wasn’t a distribution to an owner of the LLC — easy mistake to make since all my wires to Japan are — it was actually for a software license.” Graydon then goes in and recategorized as required.

It runs $125 a month at my scale and is worth every penny and then some. I’m thrilled.

Expenses (total): $74,913.90 (plus any transactions which are “in flight” and not entered yet, so probably $80k by the end of the year)

(I’d be remiss if I didn’t mention that, while that number is comparable to the one I report every year, it doesn’t include our expenses paid in Japan. Those are mostly immaterial with regards to clear business expenses — a ream of copy paper, yay — but Japanese tax law also lets me expense a substantial fraction of e.g. our rent and utilities, so we do.)

Major highlights:

People (accountant, VA, sales rep): ~$20k
SaaS: ~$15k
Hosting & Domains: ~$10k
Advertising: ~$7k
Twilio: ~$6.5k <— Largest single vendor after finally surpassing Google, which makes me happy, as I’m their biggest fan
RailsLTS: ~$5.5k
Credit card processing (Stripe/Paypal/etc): $4k
Macbook Pro: $3k

I finally joined the vast Mac conspiracy. Darn all y’all for being right. It is a much, much more productive environment than my old Dell, even considering the amount of time I’ve spent banging my head on the learning curve.

Goals for 2015

Bingo Card Creator

  • Retool CS processes such that customers get responses to their questions in a timely fashion, even when they have questions which (at present) require tier-two support.
  • Either continue presiding over BCC’s long twilight or sell it to someone who can give it the level of attention it needs.

Appointment Reminder

  • I’m shooting for $15k MRR on the publicly available plans. That’s slightly more than double what we’re doing right now, but I think it is doable with the assistance of my sales rep plus me actively working on marketing for a change.
  • We’ve historically sold HIPAA-compliant services on a very limited basis. I’m hoping to release HIPAA-compliant services sold on a medium-touch model, now that I have a very good understanding of the process of providing that in a compliant fashion and (knock on wood) actually closing the deals. This will probably be in the $500 to $1,000 a month range. If we could hit, oh, $15k MRR on those accounts as well, that would be nice, but that might be a bit aggressive.
  • The enterprise pipeline is, at the moment, close to bone dry. I should restock that, hopefully with our sales rep taking a bit of the workload. I have no clue how to forecast from “bone dry”, so let’s pick “break six figures” as a nice round number to shoot for.
  • I’d still love to earn that “Cashed a six figure check” merit badge but I’m not wedded to it.
  • Do one or two ambitious development projects on the product side of the house, if I have the time.

Productized Consulting

  • Ship the Software Conversion Optimization course as early as possible given my non-business commitments.
  • I’m hoping that, with the course out, I can do some more experiments in form factor this year. For example, I’ve been really wanting to write a book (from scratch this time) for a while, and maybe this is the year.
  • Numerically, I’m shooting for ~$80k in sales when I launch the course, and then hope to do about $150k in this segment for the year.

Big News

I have one potentially big project in the offing, but it isn’t public knowledge yet. Suffice it to say it could materially impact the above by a lot if it comes to pass. More about that in the usual places as it gets closer to happening.

I kept the best news for the end. Ruriko and I were blessed by the birth of our daughter, Lillian. I love her to pieces. I’m still working on the balancing act of being a husband and father while also running the business, but I’ve got the best co-founder in the world to figure out the challenges with.

What The Rails Security Issue Means For Your Startup

January has been a very bad month for Ruby on Rails developers, with two high-severity security bugs permitting remote code execution found in the framework and a separate-but-related compromise on rubygems.org, a community resource which virtually all Ruby on Rails developers sit downstream of.  Many startups use Ruby on Rails.  Other startups don’t but, like the Rails community, may one day find themselves asking What Do We Do When Apocalyptically Bad Things Happen On Our Framework of Choice?  I thought I’d explain that for the general community.

Nota bene: I’m not a professional security researcher.  Mostly, I sell software.  In the course of doing that, I (very occasionally) do original security research.  I did no significant amount for these bugs, aside from mitigating them for my own applications.  I am currently engaged in a Ruby on Rails security safari, and anticipate publishing the results of that in February, after responsible disclosure to the relevant security teams.  If you don’t know enough to know whether I’m trustworthy with regards to generic advice, pay someone you trust to get advice on this.

Don’t skip this post because you’re not a Rails developer.  If you’re reading this blog, this matters to you.

Background: What Has Been Happening in Rails-land?

Ruby on Rails recently released two sets of security patches (announcements here and here), in response to related vulnerabilities discovered in the frameworks.

How bad were those bugs? Severity: Apocalyptic.  They permitted attackers to execute arbitrary code on virtually ever Ruby on Rails application, without requiring that the application do anything to enable the attack other than “be hooked up to the Internet.”

What does “execute arbitrary code” mean?  Literally, it means that the attacker can choose to have your server execute any code they can dream up.  In practice, it means that you lose the server that the code is executing on.  Any further access to that server or applications on should be assumed to be compromised.

What went wrong?  This has been covered in more detail by security researchers, in posts such as here and here.  A brief description: Ruby on Rails makes extensive use of a serialization format called YAML, most commonly (you might think) for reading e.g. configuration files on the server.  The core insight behind the recent spat of Rails issues is that YAML deserialization is extraordinarily dangerous.  YAML has a documented and “obvious” feature to deserialize into arbitrary objects.   Security researchers became aware in late December that just initializing well-crafted objects from well-chosen classes can cause arbitrary code to be executed, without requiring any particular cooperation from the victim application.  Since then, the bug hunt has been on: security researchers have been actively finding lots of ways in the Ruby on Rails code base, and in related code bases, to cause the application to deserialize YAML which is in some way under the control of the attacker.

So far this has included:

  • Rails, for programmer convenience, used YAML to implement JSON deserialization.  JSON is designed to get into Rails quite easily indeed — just POST it at the server, wham, YAML.load(attacker_data) happened.  (The actual mechanics of achieving that were more complicated, but that’s the practical upshot.)
  • Rails allows XML documents to include YAML attributes.  That decision has caused a bit of head scratching, since it seems like a curious choice for most programmers in the community, but be that as it may this allowed posting XML at Rails apps to be trivially exploited.
  • Rubygems used YAML to hold metadata about each gem submitted to it.  An attacker was able to create a malicious gem, cause the Rubygems web application to evaluate the metadata contained in it, and thereby compromise the Rubygems server infrastructure.
  • February will see more compromises, with my certainty of this prediction approaching my certainty that the sun will rise tomorrow.  There exist many, many other code paths in Rails to get to YAML.load().  Some of them will be found to be amenable to attackers, either (worst case) for all or substantially all Rails applications or (still bad case) to Rails applications whose application logic involuntarily cooperates with the attack.  (i.e. In the worst case, attackers root every unpatched Rails app on the Internet.  In the best case, attackers only root some apps and they often have to have an expert do a modicum of marginal work to do so.)

Ruby on Rails security sucks lolz amirite? No.  Well, no to the nuance.  Software security does, in general, suck.  Virtually every production system has security bugs in it.  When you bring pen testers in to audit your app, to a first approximation, your app will lose.  While Ruby on Rails cherishes its Cool-Kid-Not-Lame-Enterprise-Consultingware image, software which is absolutely Big Freaking Enterprise consultingware, like say the J2EE framework or Spring, have seen similar vulnerabilities in the past.  The recent bugs were, contrary to some reporting, not particularly trivial to spot.  They’re being found at breakneck pace right now precisely because they required substantial new security technology to actually exploit, and that new technology has unlocked an exciting new frontier in vulnerability research.  It sucks for users of Rails that Rails is currently on the bleeding edge — believe me, after having lost 3 consecutive nights to patching my own applications, I know — but it would suck much, much worse if the Bad Guys had found these first and just proceeded to remote-own every Rails app on the Internet.  That is, by the way, an achievable scenario.

Was anyone actually compromised?  Yes.  The first reported compromise of a production system was in an industry which hit the trifecta of amateurs-at-the-helm, seedy-industry-by-nature, and under-constant-attack.  It is imperative that you understand that all Rails applications will eventually be targeted by this and similar attacks, and any vulnerable applications will be owned, regardless of absence of these risk factors.

Will anyone else be compromised?  Yes.  Thousands upon thousands of Ruby on Rails applications will be compromised using these vulnerabilities and their spiritual descendants, and this will happen for years.

  • Many Rails developers have not reacted to this news with the alacrity they should have.  (See next question.)  These applications may be compromised already.
  • There are many Rails applications which were created years ago, which are not under active development any more, for whom no-one is responsible for applying security patches.  Any of these applications which are publicly routable on the Internet will be compromised.
  • There are many Rails applications which are installed by end users, some of whom do not have security expertise.  For example, Redmine — an open source developer productivity tool — is commonly installed at individual companies.  Every publicly accessible Redmine instance which is not patched will be compromised.
  • Ruby on Rails lacks a CMS with the mindshare of, say, WordPress, which is good, because every unpatched Ruby on Rails CMS delivered to a non-technical company to serve as their website or backend to their mobile application will be compromised.
  • There are many developers who are not presently active on a Ruby on Rails project who nonetheless have a vulnerable Rails application running on localhost:3000.  If they do, eventually, their local machine will be compromised. (Any page on the Internet which serves Javascript can, currently, root your Macbook if it is running an out-of-date Rails on it. No, it does not matter that the Internet can’t connect to your localhost:3000, because your browser can, and your browser will follow the attacker’s instructions to do so. It will probably be possible to eventually do this with an IMG tag, which means any webpage that can contain a user-supplied cat photo could ALSO contain a user-supplied remote code execution.)
  • Many companies — including ones which do not even consider themselves Ruby on Rails shops — nonetheless have a skunkworks project running somewhere.  For example, they might have a developer who coded a quick one-off analytics app, which is accessible outside the firewall so that sysadmins could check server loads from home.  If the app is on the public Internet, it will be compromised.
  • Many Ruby on Rails shops have good development practices and no longer have the “monorail” anti-pattern, where everything their company does is in one gigantic Rails app.  They have already patched most of their main apps, but they missed one.  Maybe it is the customer support portal at admin.example.com.  Maybe it is a publicly accessible staging server at EC2 spun up by a developer who has since left the company and not shut down because, hey, $20 a month.  Maybe it is a 20% project by a junior engineer which he has on the back burner for the moment.  It doesn’t matter why this app was forgotten: if it is publicly accessible, it will be compromised.

What was the proper way to react to these patches?  Patch immediately, install a workaround immediately, or pull the plug on your application.  (“Pull the plug” means disconnect it from the Internet or shutdown the server while you get a mitigation plan into place.)  You should have distinct memories of you or someone under your employ having at least two separate incidents in the last four weeks in which they dropped everything they were doing and immediately took action to resolve these problems.  Immediately means exactly that: right now, not during the next schedule code spring, not tomorrow, not in an hour.

I was up at 3 AM Japan time applying these patches, twice.  If the next patch drops at 3 AM your local time, someone should be applying it immediately.  Computers can count to big numbers very quickly indeed.  A six hour window between a patch dropping and the start of business the next day is more than enough for an automated scanner running on a botnet to have tried compromising substantially every Rails app on the Internet.  (Do you disagree?  You are overestimating how hard it would be to find your application.)

Aren’t you exaggerating?  Our application isn’t particularly high risk!  We aren’t high-profile, it doesn’t have obvious monetary return for exploiting it, etc etc. Good thing you aren’t really saying that, but you might be at an Internet cafe next to an engineer who has poor reading comprehension, so help me explain this to him: nobody needs to care about your application to compromise it using these vulnerabilities. They can be exploited in a totally automated manner over the open Internet, requiring zero knowledge of e.g. what version of Ruby you are running, what version of Rails you are running, what your URL structure looks like, etc.  (Somebody suggested “How would you determine which servers were running Ruby on Rails?”  Answer: It’s absolutely trivial to detect Rails applications in a scalable fashion, but why bother?  Fire four HTTP requests at every server on the Internet: if the server is added to your botnet, it was running a vulnerable version of Ruby on Rails.)

Aren’t you exaggerating? Clearly this would take a lot of specialized expertise to exploit! Yep… the first time. Now that people know how the exploitation is done, however, you could do it by just copy/pasting one of the proof-of-concept scripts or b) running a browser bookmarklet. (I am not passing out that browser bookmarklet, because I think that would inevitably lead to mischief, but just know that you’re rootable in a click if you didn’t take action on this. And, by the way, have been for three weeks or so now.)

We’re A Startup.  What Happens If We Lose A Server?

If you lose one server, you will lose every server, with very high confidence.  If, for example, you are a Python-using shop which had a Redmine instance running around with no code on it, and you lose that Redmine server, you can expect a skilled attacker to then pivot from that privileged location within your network to start compromising other servers on your network.  At this point, you need to have done absolutely everything right to make it impossible for that skilled attacker to prevail, and you almost certainly have not.  (Compelling evidence that you’re not as good as you think you are: you already had one vulnerable application which could be compromised over the open Internet.  To a certain philosophy, that isn’t your fault, but the attacker gets root regardless of whose fault it is.)

The actual steps a pen-tester would take to root your other boxes are pretty academic after they have one.  (For example, you can start probing other machines on the network for vulnerable services, use credentials found on your compromised machine to suborn other machines, take over routing hardware using vulnerable administration panels and then start intercepting all network traffic, etc etc.)  Just take it as a given, you will lose.  Companies much larger and smarter than you lose everything when this happens, essentially every time it happens.

We’re A Startup.  What Happens If We Lose Every Server?

A short preview of coming attractions:

  • You will lose the next several weeks out of your schedule dealing with this issue.
  • You will have to take down all of your applications and rebuild all your servers from scratch.
  • You can assume the attacker now has a copy of your source code, all credentials you have, all your databases, and all information you had like e.g. log files.
  • Do you take credit cards?  Were you taking credit cards through an exploited application?  You now have a PCI-reportable data breach.
  • Your local jurisdiction may have legal requirements that you notify the people whose data just got exposed.
  • You now have a public relations nightmare on your hand.
  • In addition to compromising any customer data you possessed, you have made it possible for diligent attackers to compromise those customers elsewhere.  The most trivial example is, if you did not implement password storage correctly, you have just handed the attackers a list of email addresses and associated passwords which they can now re-use on higher value targets like e.g. bank accounts or Gmail, because many users re-use their passwords everywhere.  (You use bcrypt?  Wonderful.  Did  the attackers turn it off when they rooted all your applications?  Can you conveniently check that, knowing that you cannot trust the contents of any logs on those compromised servers?  No?  OK, so instead of losing all the passwords, we can upper bound exposure at only all users who logged in since the attack started.  That’s an improvement… sort of.)

Basically, it’s Very Bad, but not the end of the world.  You’ll probably need expert help to get through it, like you would need if e.g. you got sued.  Unfortunately, lawsuits generally give you weeks of notice and progress slowly, but security vulnerabilities often give you negative several hours notice and get worse for every minute left unchecked.

We’re A Startup.  We Don’t Use Ruby on Rails So We’re Totally Cool, Right?

Can you enumerate every account on the Internet where you have a password and also every service consumed by your business?  Go ahead, take as long as you need: it is very important that you don’t miss one.

OK, let’s start with the obvious: Look for analytics providers and other folks on that list who have instructed you to embed JS on your website.  If I do this exercise, I come up with at least three results here.  Do any of them use Ruby on Rails?  (Are you sure?  Remember, if they have at least one Rails app on their network…)  Great.  If they didn’t patch in a timely manner, you should assume that Javascript you’re embedding on your website is in the hands of the enemy.  It is now a cross-site scripting vulnerability against every page it is embedded on.  Do you embed it on e.g. log in pages or anywhere your admins expose their own all-powerful admin cookies?  Boo, now the enemy has your password / cookies / etc.

Alright, let’s move down the line: Look for anybody who implements OAuth/Facebook Connect/etc.  Do any of them use Ruby on Rails?  Are you sure?  If they haven’t patched, you’ve handed the union of all privileges over the linked accounts to the attackers.

Alright, let’s move down the line: Consider everybody who has a copy of a password which you re-use elsewhere.  (You shouldn’t be re-using passwords, or variants of passwords, but I ignored that advice for years so I’m betting a lot of you did, too.  Maybe not you, specifically, but you know that chap in marketing who is great with people but thinks MSWord is complicated?  Consider whether he has access to anything sensitive in your company.  He does?  Well, sucks to be you then, but good on your for password security.)  Do any of them use Ruby on Rails?  Are you sure?  Did they use bcrypt/scrypt/etc to properly secure passwords at rest, and did they patch these vulnerabilities fast enough to prevent attackers from pulling them off of the wire?  Are you sure?  If you’re not sure of all of these things, consider every password compromised and take action appropriately.

One of my friends who is an actual security researcher has deleted all of his accounts on Internet services which he knows to use Ruby on Rails.  That’s not an insane measure.  (It might even be inadequate, because all the folks who are compromised are probably going to lose their database backups as well.  Well, if they have database backups.)

These are just a sample of ways in which these vulnerabilities can ruin your day.  They are very much not an exhaustive list.  If you believe in karma or capricious supernatural agencies which have an active interest in balancing accounts, chortling about Ruby on Rails developers suffering at the moment would be about as well-advised as a classical Roman cursing the gods during a thunderstorm while tapdancing naked in the pool on top of a temple consecrated to Zeus while holding nothing but a bronze rod used for making obscene gestures towards the heavens.

Somebody Dropped A 0-Day On Rubygems. What If It Happens To Me?

Yes, that certainly sucks royally.  Rubygems wasn’t even exploited using the patched Rails vulnerabilities — an attacker just learned something which worked (again, we’re on the leading, bleeding edge of security research here), applied it in a novel fashion, and compromised the Rubygems application.  As of me writing this it looks like we avoided the Ruby-ecosystem-wide apocalypse that would have happened if they had started backdooring gems, but let’s just focus on the immediate fallout: their system got compromised.  What if one of yours did, like that?

The first step is a preventative inoculation: If you run an application on the Internet, you should today establish a security contact page.  It only needs to include two things: a working, monitored email address and a PGP key.  Bonus points for giving some sort of public recognition to people who report security vulnerabilities to you in a responsible matter.  This helps to co-opt some security researchers so that they e.g. get in touch with you about the problem prior to just going ahead an exploiting it.  Software security has a curious system of social norms, where scalp collecting both builds both karma and pseudo-currency.  It’s bizarre, but just take this on faith: having a security page with a working email gives you a certain amount of We Should Avoid #’#(ing Their #()#% Up Without Asking First street cred.  (Naturally, like any social norm, that is a preventative measure rather than a panacea.  However, given that it is a well-understood norm, it gives you a bit of an edge in the PR battle should someone decide to just drop a 0-day on you.)

Good security pages to pattern after: 37signals (I particularly like how this page works for responsible disclosure while, in a dual-audience fashion, also doubles as being great marketing copy), Twilio, Heroku (again, dual audience), etc.

Have a plan for responding to security incidents. I call mine the Big Red Button. Thomas, a security consultant friend of mine, accurately observed that these probably caused the first Big Red Button events that many folks in the Rails community have ever had to deal with. We should learn from our experiences here.

For example: I pushed the Big Red Button at 3 AM in the morning, twice this month, to apply critical security patches and work-arounds.

So did I do a great job of addressing this problem? No, I did a pretty effing atrocious job of addressing this problem. Specifically, I have two old-as-the-hills Rails apps running on 2.3.X at the moment. Waaaaay back in 2010, Mongrel and Rails had a bit of a compatibility issue, and I solved it via a monkeypatch. The monkeypatch relied on a hardcoded version number, which I have been hand-incrementing every time I update Rails. It’s literally on the redeploy checklist, next to a note “TODO: This is stupid and should be fixed when I get a moment.”

I did three Rails app upgrades locally, three test suite runs, and three sets of smoke tests when applying one of these patches. The one in the middle happened to be Appointment Reminder, which is an application that has to be up during the US workday. Unfortunately, because I was exhausted while following my deployment and smoke test checklists, I a) forgot to bump the version number in that monkeypatch and b) did not follow the part of the smoke test which would have clued me on to “This is going to cause log-ins to fail on some browsers.” That resulted in some breaking downtime for some customers during the US workday, and me having to send an apology to all customers. That sucked horribly.

I have now fixed my monkeypatch to not require hard-coding the Rails version, simplified some of my deploy procedures, and am working in the next several months on beefing up my testing suite. Also, lesson learned about resolving “TODO: This is stupid” when it would take 5 minutes to do rather than having it blow up in my face.

There, that’s an experience I went through. Now you know the punchline, so hopefully you don’t have to go through it as well.

Similarly, we can observe:

  • We need an updated list of all applications running on our servers, so that we know when a problem with a technology stack affects them, even though this sounds like a boring Big Freaking Enterprise IT Department requirement. (And gulp their dependencies.)
  • For each tech stack we support, we need at least one expert following the primary source for security news for that tech stack.
  • We need whomever is responsible for product development and/or ops to, effectively, carry a pager for drop-everything-and-do-it-now resolution of security issues, just like we’d do for e.g. the server has fallen over or “our building is, physically, on fire.”
  • These requirements suggest minimizing the number of tech stacks we work with, even if that means passing up the new hotness occasionally.
  • Just like we have e.g. insurance on the building physically burning down, we should have some upfront investment in security. Good forms might include security training, outside consulting, or (if we’ve got a lot of money) contributing work towards securing tech stacks we rely on.

You Should Be At Defcon 2 For Most Of February

Big security vulnerabilities tend to be discovered in bunches.

Why does this happen?

  • Blood in the water attracts sharks. Some of my security friends would hate this phrasing, because “researchers don’t cause vulnerabilities, they find vulnerabilities”, but as a businessman who depends on software for his livelihood, I had exactly zero days of the last six years spent sleepless because of the latent vulnerability in Rails, but two days this month due to the pressing need for immediate mitigation. There are many more eyes pouring over Rails — and related projects — more closely now than typically. Many of them are white hats (yay!). Some aren’t. In general, there is a virtually infinite need for software security expertise, just like there is an infinite need for software, and there is a crushing lack of expertise which can meet it. Some folks who are capable of finding vulnerabilities are, due to attention/topicality/renewed interest/commercial potential/etc, now looking at Rails as of today.
  • Technology marches on. After you have a new exploit vector to play with, you can start applying some of the technology used to discover / develop / exploit it against other code bases, code paths, etc etc. For example, the first Rails vulnerability was parlayed within a day into a similar vulnerability in the MultiXml gem. The same underlying “YAML is very dangerous” realization enabled the Rubygems compromise. If I were working on e.g. Django, I would strongly suspect that security researchers are going to see whether they can find similar patterns on Django — it wouldn’t be the first time, since e.g. HMAC tampering vulnerability disclosures in Rails were followed up by similar findings on Django the same week.

I previously had a version of this post queued up right after the first bug dropped, but didn’t hit Publish because I got busy that weekend and thought it wouldn’t be timely anymore. That post included the lines “I will bet $1,000 at 100-to-1 odds that Rails suffers another code execution vulnerability before the end of January.” If you had hypothetically taken that bet, you would have lost.

You should expect February to be a very trying month for the Rails community and startups in general. Your security team should be at Defcon 2: be ready to respond to patches with particular alacrity, and expect there to be failures in the ecosystem outside of your ability to control them. For example, I’d make sure that you can rebuild systems without requiring access to Github / Rubygems / etc, and that’s (unfortunately) the tip of the iceberg.

This Sounds Like A #$()#%ing Disaster

That is primarily because this is a #$()#%ing disaster.

For my part, in addition to taking steps to fortify my own businesses, I’m (as time permits) doing some pro-bono security work on Rails. I do not have results which can be published yet. I strongly suspect based on early research that I will, in February, and I strongly suspect that other researchers (both white hats and the Bad Guys) are much, much better at this than I am.

Get ready. It will get worse before it gets better.

Bingo Card Creator (and other stuff) Year in Review 2012

I’m Patrick McKenzie (occasionally better known as patio11). When I started my business six years ago, I was greatly inspired by a few other folks who published the minutiae of their software businesses, particularly actual sales and expenses numbers. I resolved to do it for Bingo Card Creator, my (first) software business, and then just kept up the habit. I traditionally post the year’s numbers and my reflections on what worked and what didn’t right before Christmas: see years 2006, 2007, 2008, 2009, 2010, and 2011.  (This year’s installment was slightly delayed.  Merry belated Christmas?)

Obligatory disclaimers: It is a good thing that I’m CEO and not the bookkeeper, because if I were bookkeeper I’d be fired for incompetence. When I do the official accounts for tax purposes I virtually invariably discover a few thousand dollars of extra expenses. (You might reasonably think “Then shouldn’t you outsource this?”, and you’re smart for thinking that, but sadly my part-time bookkeeper can’t always catch problems like “Patrick forgot to hand her a business trip worth of receipts.”)

On transparency: I’m weakly committed to transparency: it is nice to have but not one of my core values. I don’t impose it on other people, so when my business touches a partner or customer I generally err on the side of keeping their details private, absent specific permission to share. I also politely decline to discuss stats for Appointment Reminder, largely justified by “I don’t want this post quoted against me in a partner meeting” should I ever decide to raise money for it.

Capsule summary of 2012: I had a very good year, across all lines of business, in terms of personal satisfaction, value to clients, and profitability.   The big story was meat-and-potatoes execution: taking things which I knew how to do and knew to be effective, and applying them in fun new ways.  Some examples follow.  Profits roughly tripled from ~$70k to ~$200k (on total sales of ~$275k), exclusive of Appointment Reminder.  2013 looks to be very exciting indeed.

The Year In Brief

Bingo Card Creator was in maintenance mode for approximately 48 weeks of the year again, with two experiments done with a site redesign and incorporation of direct ability to charge credit cards (via Stripe) rather than using Paypal or Google Checkout. The experiments were, taken together, a smashing success.

I once again planned on spending most of my time working on Appointment Reminder, and (once again!) life decided to get in the way. Last year it was losing two months of the calendar to immigration issues. This year’s “distractions” were much happier: I took off approximately three months for my wedding and honeymoon, and my consulting business decided to grow like gangbusters. In any event, I was able to repay a lot of AR’s technical debt, fix the occasional technical issues the service had been experiencing, knock off a few new features, close my first enterprise contracts, and approximately triple the paying customer base on the published plans.

Speaking of consulting: As planned, I spent less time on acquiring new clients and assorted promotional activities (conference speaking, etc), and roughly the same amount of time on the boring mechanics of scheduling and delivering engagements. I also walked my rate up a few times.

There was an interesting outgrowth of the consulting business: over the last two years I’ve delivered engagements regarding email strategies for SaaS businesses several times, and had to turn down many more due to lack of availability, so I tried my hand at productizing consulting via creating a video training course about that subject. This worked out very well, both for myself and my customers.

An opportunity fell into my lap to try angel investing (as angel, not as entrepreneur). It’s a bit of a long story, so I’ll probably cover it some other day.  I also wrote a book, as previously covered on the blog.  It launched very late in the year, so I’ve got no interesting numbers to share about it yet.

Bingo Card Creator

Bingo Card Creator makes bingo cards, mostly for elementary schoolteachers. It had far-and-away its best year ever, despite being in maintenance mode. This was largely driven by organic growth of the business and huge increases in conversion rates following the redesign and Stripe integrations, covered here. The differences are very apparent if you look at conversion rates for any month after May and compare it to the year previous, which is necessary since BCC traffic and sales are very heavily seasonal. Or you could, you know, just take a look at the sales graph.

Stats:

Sales: 2,254 (up 55% from last year’s 1,451)

Refunds: 89 (up massively from 14 — the story is so good you’ll have to read it below)

Sales Net of Refunds: $64,791.81 (up 40% from $46,233.68)

Expenses: $26,193.40 (up from $23,003.19)

Profits: $38,598 (up 66% from $23,230)

Wage per Hour: Approximately $1,000, given that I worked for approximately 15 hours integrating the new design and spend approximately 20 minutes a week doing support.

Web Stats:

(All stats are from bingocardcreator.com unless otherwise specified.)

Visits: 1.08 M (up from 821k)

Unique visitors: 875k (up from 670k)

Page views: 3.4 million (up from 2.9 million)

Traffic sources of note: Google (56%), AdWords (12%), Binghoo (11%)

Trial signups for online version: 87,000 (up from 83,000)

Approximate online trial to purchase conversion rate: 2.4% (up from 1.8%)

Narrative version:

Overwhelmingly the best thing that happened in 2012, or for that matter the last several years for BCC, were the A/B tests where I reskinned the application and marketing site and where I introduced Stripe charging individual credit cards. This breathed quite a bit of life into a business that had previously simply been running on autopilot. I’m incredibly happy with how that worked out, particularly as I was able to get the actual design work done by someone else, and only had to do the Rails integration and a few tweaks to get it working.

What Went Right:

  • I’m almost totally superfluous to the day-to-day operation of the business.
  • The aforementioned A/B tests delivered major wins, on top of a half-dozen more minor ones. (A percent here, two percent there, it adds up when you keep doing it for six years.)

What Didn’t Work So Well:

  • I used the Stripe quick-start code to do my integration and did not build in server-side validation to stop duplicate transactions, trusting the client to only submit once, using Javascript to guarantee that. This is reliable as long as your client is not the IE Javascript engine running on a machine while it is being struck by a bolt of lightning. My poor customer got charged 36 times for Bingo Card Creator. I, of course, refunded the purchases when I caught them. (In case you’re worried: while a lot of electronics got melted, my customer was physically unharmed.)
  • In addition to the above, switching from Paypal to credit card orders increased the number of duplicate orders customers put through by more than an order of magnitude. Previously I just trusted people to not do this. Apparently… it is time to algorithmically suggest to customers that just because they didn’t get an email in 30 seconds doesn’t mean they should try the purchase again.
  • At some point in 2012, I started dreading doing customer support. I’m not sure why — I think I’m just really tired of answering the same questions for six years now. I’m going to try to pass off L1 support to a VA in 2013. I probably should have done this 5 years ago.

Appointment Reminder

Appointment Reminder does appointment reminding phone calls, text messages, and emails to customers of professional services businesses. I launched it in December of 2010, so it is just turning 2 years old right now. I go back and forth on whether I want it to be the Next Big Thing for me. Since I want to keep my options open on that score, I refrain from quoting numbers about it publicly.

My idea was that AR would be my primary business focus at the start of the year. That was the plan last year, too. Once again, my execution on it left a little to be desired: I think I got done about 60% of what I wanted to get done. This was partially due to distraction from the rest of the business, and partially from not understanding the difference between “single”, “engaged”, and “married” as well as I thought I did. (That’s not a complaint so much as it is a reflection about reality — marriage is far and away the best thing that ever happened to me.)

Revenue: Undisclosed.  The monthly revenue run rate on the publicly available plans is approximately quadruple what it was in December 2011. Enterprise sales went from “zero” to “non-zero”.

Expenses: Undisclosed.

What Went Right:

  • Technical issues: Last year AR had multiple customer-visible failures, and when AR broke it broke very badly, with failure modes like “DDOS someone’s home phone line” or “Failure to deliver time-sensitive reminders sent to patients by their doctors.” I spent quite a bit of tightening the system up, and had a much, much more stable year. We still had one major incident (the VPS it runs on became unable to boot after a distribution upgrade) which caused six hours of wall-clock downtime, but thankfully maintenance was timed so that this only resulted in about 15 minutes of downtime relevant to customers, and we only dropped ~6 calls. I’ve figured out a lot of architecture / tech stack problems prior to reaching extreme scale, which is probably for the best.
  • Email marketing: AR sent precisely one marketing mail on January 1st of this year: “Thanks for signing up for the free trial.” I frequently do email marketing for clients, and it is always more sophisticated than that, but I figured AR didn’t have trial numbers to justify extra work. When I was writing my video course on email marketing, though, not taking my own advice felt very disingenuous, so I implemented most of what I was advising. Wham, conversion rates and customer happiness up, just like advertised. (Best single win? A checkup at 3 weeks into the trial which, if the account looks likely to convert, tells them how much money they’re saving. If they’re unlikely to convert, it offers a one-month extension to the free trial if they speak to me about it. That single email has been worth low five figures. Want more suggestions? Buy my course about lifecycle email marketing.)
  • Redoing pricing/plans: Appointment Reminder launched with $9/$29/$79/Call Me Maybe pricing. (Hey I just met you / And this is crazy / But pay me ten thousand dollars / It’s enterprise software, this line won’t even rhyme.) The $9 personal plan was a mistake, and I knew that when I created it, and even despite that I suffered a year and a half of it anyhow. D’oh. That wasn’t the worst mistake, though — it turns out there was a substantial market segment who were at above the quotas that the Small Business ($79) plan addressed, but were unwilling to play the Enterprise pricing game. They do, however, fit in the Office plan ($200). The (new) most expensive plan now accounts for over 1/3rd of revenue from the publicly available plans.
  • Enterprise sales: It’s a long story, but surprisingly it isn’t impossible to win them as a one-man firm calling from Japan… you just have to make the most out of the utterly unfair advantages that gives you.  (A trump card I lay early and often: “I’m the founder.”)  If you’re interested in this topic, I recommend signing up for my mailing list, since I seem to write more about B2B topics than on my blog.

What Didn’t Work So Well:

  • My responsiveness: I have not been doing a great job this year at pursuing enterprise sales (i.e. only successfully get a decisionmaker on the phone a low percentage of the time even for inbound leads), partly because I get a lot of leads via voicemail, which I don’t deal with very well. Many of them are poorly qualified, and as a result I find myself dreading listening to voicemail to call back and talk for 10 minutes (at 2 AM in the morning) only to discover that they’re not good fits for AR. This is something which rationally speaking I should want to do, since it the path forward for the business, but I have been only sporadically successful at forcing myself to do it. Ideally, I will systematize the sales process and then offload it to someone, but this requires consistently executing on it myself first, and at the moment my successful sales have been all one-offs rather than anything resulting from a repeatable process. (n.b. Welcome to sales at any early-stage startup.)
  • Technical issues: Did I mention I had six hours of downtime and nearly gave myself a heart attack resolving it prior to the business day starting for my EST-based customers? That isn’t acceptable going forward. I still have more to learn about this (and likely always will).
  • General level of interest: Even in weeks that I have blocked off to work on AR, I often find myself just lacking any desire to do it. The business isn’t intrinsically more boring that e.g. Bingo Card Creator, but the sort of things that I need to do to move it forward seem to hit my desire to work with a damp towel. On the plus side, not having investors or employees means that I have 100% control over the schedule. On the minus side, not having investors or employees means that I have 100% control over the schedule, and AR has frequently lost out to pressing matters like consulting engagements, League of Legends matches, or wonderful opportunities to clean drains around my apartment.

Consulting

Like 2010 and 2011, I did a bit of consulting in 2012 for software companies. I increase sales of SaaS companies, and that’s all. Under that fairly broad brief, I do everything from writing software to support marketing objectives (Fog Creek has a case study coming out eventually, I believe) to doing lifecycle email campaigns to repricing plan offerings to A/B testing copy tweaks to… you get the general idea.

My guests and I on the podcast ended up talking quite a bit about consulting in the last few months, and I wrote an article about it.

Consulting Sales: ~$140,000  (this includes something like $20k of Accounts Receivable, for delivered engagements whose payments I will not constructively receive in 2012)

Consulting Expenses: $~40,000 (travel, conferences, and catch-all for anything which isn’t obviously for another line-of-business, like e.g. buying a business iPad)

Narrative Version

Where do clients come from?  I primarily source engagements by participating on the Internet (Hacker News, my blog, etc), speaking at/attending conferences (most relevantly to consulting, Business of Software), having word-of-mouth from previous happy clients or other folks who know me, and occasionally from nebulous reputational factors.  A new client and I would typically talk for an hour or two, and if they look like a good fit, I send them a one-to-two page mini-proposal for the engagement.  The prototypical “good fit” for me is an established software as a service company with revenues in the eight figure range, a few dozen employees, and a company culture which focuses more on the product/engineering side of things than on the marketing/sales side of things.

What are engagements structured like?  It depends on the engagement, but a fairly typical proposal for a new client would be for a 1 to 3 week engagement, delivered contiguously and on-site.  (I do remote engagements, too, but largely for existing clients.  Being on-site is a bit higher bandwidth, which is helpful in the getting-to-know-you-and-your-systems/products/people stage of a relationship.  People also generally tend to trust folks they’ve met in the flesh and broken bread with a heck of a lot more than they trust an email address with attached wiring instructions.)  I charge a flat weekly rate, generally in the five figure region.  The beautiful thing about the choose-your-engagement-length structure to proposals is that if the client has budgetary issues then we can address them by moving particular deliverables out-of-scope and shortening the engagement, rather than by compromising on the rate itself.

“What is it you do, exactly?”  It varies extensively depending on the engagement, and the specifics are often NDAed.  Broadly speaking, I make software companies money, primarily by increasing the sales of their SaaS products, usually through either a) applying engineering expertise to solve a particular marketing problem or b) just straight-up marketing expertise.  (If a client were to theoretically ask me to just crank out features for their Ruby on Rails app, I could theoretically do that, but more talented programmers are available for cheaper, so I’d advise them against it.)  Some specific tactical examples might be:

  • Designing and implementing the first-run experience for their SaaS application, with the goal of increasing conversion from free trial signups to paying accounts and increasing lifetime value of paying accounts
  • Implementing a drip campaign, such as allowing potential customers to sign up for a free one-month mini-course on $PICK_A_TOPIC, where the mini-course also duals as a sales channel for the SaaS product the company sells  (One of the rare engagements I can actually talk about was doing this for WPEngine — it meaningfully and permanently increased their sales.)
  • Re-writing marketing site copy or re-doing design (I do wireframes, their designers make PSDs and working code, most of the time) to increase conversions to a SaaS product, generally with the new work getting A/B tested versus the old stuff so we know whether it is working or not
  • Re-doing pricing / packaging options, or presenting them in a more effective light, to increase sales, average order value, and average customer lifetime value.
  • Teaching teams at clients to implement A/B testing, email, better pricing/packaging options, etc etc so that clients can get good at these rather than needing to rely on me.
  • Being a sounding board for product / UX / packaging / etc decisions.  (e.g. “We’re considering moving a very successful desktop application sold on a licensed model to the SaaS model.  That will cost us millions of dollars and, if we commit to it, would be our #1 strategic priority for next year, to the exclusion of all others.  Prior to committing to doing that, we’d like to have external confirmation that this isn’t insane.”)

What Went Right:

  • Leveling up: The advice I gave in the podcasts and the above article is largely distilled from my own experience. In general, as compared to earlier in my consulting career, I’m a bit smarter with regards to client selection and to the kind of projects I work on, and I charge to match. The increase in sales is totally driven by an increase in average bill rate — I actually cut down weeks worked. (There are broadly speaking three ways to increase consulting revenue: increase utilization rate (percentage of time you spend doing billable work), increase your bill rate, or hire people. I could schedule as many weeks of work as I wanted, but don’t really feel the urge to do so since it would conflict with my software businesses and life in general, and don’t really see myself managing other consultants… at the moment, anyhow.)
  • Working with great clients: I’m privileged to have had the opportunity to continue working with smart companies, with excellent products, which had good opportunities for applying my skills to our mutual benefit. A lot of the stress of consulting is dealing with client relationships which you shouldn’t be in in the first place. Being picky and choosy has been a major win for me, and as time goes on I’m getting better at it.
  • Delivering for clients: I have my own personal Nagging Doubt Monster. NDM often wonders whether e.g. I’m worth what I charge to clients. On balance, I’ve always thought the answer was Yes, but I have had troubled sleep about it, particularly as my bill rate hit arbitrary threshholds that flipped my “comfortable” bit. Earlier this year, a particular engagement, whose results I’m unfortunately not at liberty to disclose, sent the Nagging Doubt Monster into indefinite hibernation. In addition to that particular engagement, it has in general been a very good year. Clients are generally thrilled with what they got out of working with me, and I feel likewise.

What Didn’t Work So Well:

  • Legal Stuff: You know how every consultant ever tells you “Hire a lawyer to do contract review”? You should hire a lawyer to do contract review. Some clients and I had differences of opinions with regards to the meaning of some boilerplate, which (while they eventually were resolved amicably) caused me way, way more stress than necessary.
  • Scheduling Issues (Client-side): I had about three-ish weeks of availability this year where I intended to be doing consulting work, but didn’t end up billing anybody, because I didn’t move engagements through the pipeline fast enough. (That would be a decent-sized hit if I were a traditional consultant, who generally aim for about 35 weeks of work in the year, but since I generally shoot for about ten-ish…) In the future, I’m going to revise the proposal-and-present-contract dance to decouple it from engagement delivery dates. Previously, I’ve generally gotten the final greenlight within 2 weeks of an engagement starting, and if I blow that date that generally means I blow that week of availability. Random events can delay both contract signing and delivery, so I think decoupling them in the future will result in not having to spin my wheels.
  • Scheduling Issues (My side): Relatedly, I occasionally have anticipated availability evaporate. I took three months off for my wedding, but that was planned. I also had August marked off on the calendar for working on my course, but that ended up swallowing a lot of September, and that delayed contract negotiations scheduled for September and thus cost a week or two of my fall consulting season when that bubbled down the line.

Productized Consulting

I created and sold a video course which teaches SaaS businesses how to use lifecycle emails.

I have, historically, intentionally avoided selling anything to software developers. Partly this was out of wondering whether I had anything of value, partly this was thinking the market was terrible (penny-pinchers with not-invented-here syndrome), and partly this was out of lingering distaste regarding “selling shovels.”

There’s a persistent meme among software developers which says “The way to get rich in a gold rush isn’t to mine for gold, it is to sell shovels to gold miners.”  This meme is often deployed to suggest that shovel-sellers are exploiting naive gold miners.  I want to eventually write an anthropology paper on the gold rush narrative as applied to startups, because it is fascinating, but my brief sketch is that people often use an incorrect syllogism along the lines of “If you sell shovels, then your customer must be a miner, then there must be a gold rush, but gold rushes are either intrinsically bad or there is in fact no gold rush, so your business is either doomed, distasteful, or distastefully doomed.”

After seeing 37signals, Fog Creek, Ramit Sethi, Amy Hoy, and others all produce information products which actually seemed to create customer value (in many of those cases directly to technologists), I started to feel a little more open to the idea of doing it. So early in the year, I created an email list for folks running software businesses, with the idea being that I’d continue cultivating an audience by producing free things that they’d enjoy, and eventually offer them an opportunity to buy something a little more in-depth than my typical writing.

Concurrently with this, I was doing consulting engagements, and I kept my eyes open for recurring customer needs. One major one was that most SaaS companies don’t make effective use of email marketing. In particularly, they send next-to-no lifecycle emails (emails triggered off of customer actions in the software), and those are an incredible opportunity if you execute well on them. I implemented lifecycle campaigns for fivish consulting clients, in some cases making hundreds of thousands of dollars in sales off of individual emails, and thought that rather than hiring out that expertise by the week I could probably package some of it as a training product, so other companies could implement the campaigns without needing to hire a consultant to do it for them.

(Here’s a replicatable strategy for making several hundred thousand dollars with a single email: start with a revenue base of $X million a year.  Email all customers asking them to switch from monthly billing to annual billing, in return for some incentive you can offer, which can range from “a month free” to “15% discount” to “Hey, you can book the expense this calendar year, so that will save you money on taxes.”  Feel free to try this with any client or day-job of yours if they’re already at scale — “We made so much money the accountant/bank called us to complain” will make for a great bullet point at your next contract/salary review.)

Why bother doing a productized consulting offering when I have software businesses and standard consulting to keep me busy? Partially, I love trying new things and just wanted an excuse to experiment. Also, consulting is working out fantastically well, but it routinely requires me to spend multiple weeks abroad on business, and that is less and less attractive to me as I get more and more married. So if I could replace on-site consulting with a consulting-like offering that I could execute on here from Ogaki, that would be a bit of a win.

I eventually decided on making a study-at-your-own-pace video course as opposed to e.g. an ebook or a series of webinars, and then wrote out lesson plans and started recording. I anticipated about two weeks to do the recording (I was shooting for about 5 hours of video after editing, so perhaps six or seven hours of raw video) and two weeks for a freelance video editor to get things ready for me. (I wrote all the courseware and payment processing code myself — rationally speaking that should have been hired out, too, but I was really looking for a programming project at the time.)

The course was eventually delayed a few times (my original estimate was two weeks of work and a three week shipping schedule, but it ended up closer to four weeks of work and an eight week shipping schedule).  Nonetheless, it did successfully ship, and seems to have worked out pretty well for customers.  (Amy Hoy interviewed me about the process in detail, in case you want tactical advice.  I expect that interview to be up in a week or two.)

(You can find the course here.)

Course sales: ~$60,000  (My mental target was $20k, so this was a pleasant surprise.)

Course expenses: ~$6,000 (freelance video editing, payment processing, video hosting, etc)

What Worked Well:

  • Building an email list: About 5,000 folks asked to receive email from me. They mostly get free advice along the lines of what I’ve often blogged, except in a bit more detail. For example, I wrote about SaaS pricing and consulting, and subscribers have told me that they’ve used advice in those emails to substantial effect in their business. My basic brief is “Don’t ever waste their time”, mostly because I respect that people have invited me into their inboxes. (Also, I pay MailChimp about $100 every time I hit the Send button. That would probably change the character of my blog posts a bit…) In any event, when you have a “warmed” email list of people who have pre-existing reasons to like what you have to say, since you’ve been creating value for them for months/years/etc, doing product launches is a lot easier than “Build it and pray that they’ll come.”
  • Value for customers: One of the reasons I avoided doing this for so long was that I was concerned whether customers would actually get value from it or not.  For both genuinely compelling ethical reasons and not-nearly-so-compelling Nagging Doubt Monster reasons, I greatly prefer doing things which have highly obvious ROI for customers over things that don’t.  Feedback about individual companies’ results with lifecycle email has been tremendously positive, ranging from “We had this on the list for 2 years but never knew where to got started, but then we bought your course, gave it to an engineer, and shipped within 3 weeks” to “This made us six figures.”  (Seriously mindblowing: the sales copy made one customer six figures.  An engineer reading it thought one point I mentioned in passing was worth repeating and forwarded the mail to their bizdev guy.  The bizdev guy used it the next day to close a 500 seat license.)
  • Stripe: Despite some issues with, primarily, corporate American Express cards thinking that $2k charges for training materials were a little suspicious, Stripe was extraordinarily easy to integrate and reasonably priced, like usual.  In addition, unlike one might reasonably expect for a merchant account or Paypal, Stripe didn’t require either advance warning or an after-the-fact investigation when I suddenly had a considerable volume spike.  (I was expecting plus-or-minus $20k in sales in a short period of time and, if one goes from $3k a month of sales to $20k a month, Paypal will have words with you, sometimes freezing your account in the process.  This is, I rush to add, totally rational and solvable by e.g. submitting them a bit of documentation and waiting, but I had a lot on my plate, and not worrying about that was a boon.)

What Didn’t Work So Well:

  • Workflow issues with video: I’m a good writer and a fairly decent conference speaker / classroom lecturer.  It turns out that lecturing to a camera is another skillset entirely, both in terms of maintaining pacing / energy / interest / etc and in terms of stupid technical issues like “You need to worry about having scads of hard drive space and, by the way, good lighting for taking the video.”  I’d give the content quality (in terms of advice) an A- or an A, but the presentation was often a B-.  This will probably improve as I get more experience with projects in this form-factor.  For example, while I like my decision to avoid word-for-word scripting the videos, the next time I’ll probably create e.g. Powerpoint slides or something to give people something more meaningful to look at during the lessons than me talking at them.
  • Outsourced video editing: I hired somebody to do all the editing for these videos, which was a tremendous time-saving measure over doing it myself, considering that teaching myself the Adobe toolchain would have been a terrible decision.  Unfortunately, my freelancer (a good friend of mine from high school — and yeah, I hear you and you’re probably right) had a run of “bad luck” with regards to e.g. hardware failures and scheduling issues, which resulted in the work getting delayed quite a bit and only about 90% of the way finished.  (There are, e.g., videos which I shipped with known editing bugs in them, on the theory that shipping today was better than delaying launch by a non-deterministic amount.)
  • Writing my own courseware: The site (which handles both sales and fulfillment) is a built-from-scratch Rails application which probably took a week or two to write (I was doing it concurrently with filming videos so I don’t have a great breakdown of hours used).  It is, basically, the best possible project to ask an intermediate Rails consultant to bang out, since the behavior is very well-specified and there are no surprises.  While I was quite pleased to have the opportunity to write it — you know, it’s like a new car, a new programming project has that smell of fun to it — rationally speaking that was a poor decision which probably cost me time and aggravation versus a) hiring it out and b) doing a totally-for-jollies programming project which wouldn’t need boring-but-important-to-get-correct glue code like user management or Stripe integration.  (Relatedly: what the heck possessed me to put it on a VPS again versus doing Heroku.)

Goals for 2013

Bingo Card Creator

  • Given that I haven’t had a full year at the new-and-improved conversion rates yet, I reasonably expect BCC to coast to approximately $80k in sales on flat costs, for something like $55k in profit.
  • I want to outsource 90% of the customer service load for Bingo Card Creator, because I add zero value to most interactions these days (there’s no reason other than ego to have “Thanks for your email.  Bingo Card Creator doesn’t support pictures and we do not anticipate supporting pictures in the future.” come from me rather than from a freelancer), my response times are getting longer and my patience are getting shorter with each passing year, and the cognitive load of dealing with even trivial amounts of BCC CS email makes me procrastinate about opening my inbox and dealing with (much higher priority) email for my other lines of business.

Appointment Reminder

  • This goal worked out pretty well for me this year, so let’s try it again: 10X sales from 2012.
  • I want to explore flying to an industry conference as a sales channel for AR.  My back of the envelope math suggests that it’s probably straight-up worth it to just show up with an iPad in a target rich environment and take orders for the $200 a month plan on the spot.  (My expected LTV is over $2k and I can demonstrate the product in about seven minutes while standing on my head, so any decent close rate makes that a very good use of a day, right?)  Plus if I successfully execute on that plan two times then I can take the best-converting demo script, write supporting software, and then hire somebody with good interpersonal skills and a desire to spend time on the road to deliver it for me.
  • Now that I have a few marquis clients on enterprise pricing, I’d like to start closing more enterprise deals at true enterprise rates, rather than discounted-heavily-to-win-this-proposal rates.
  • In addition to walking up enterprise rates, I’d like to systematize the enterprise sales process, with the eventual goal of being able to have large parts of it executed by people who are not me.
  • I’ve neglected AR’s systematized marketing (e.g. content creation for the website, A/B testing, etc) horribly.  Need to rectify that.
  • Deliver more features which are needed for higher-end customers, like “upload CSV of appointment data” rather than requiring manual entry, group appointments, etc etc.
  • Continue improving service reliability.
  • Strongly consider whether Appointment Reminder needs to eat more of my business attention pie, given current results and growth prospects.  (e.g. At AR’s 2012 revenue rates, an opportunity which would generate $50k in revenue for a few weeks of work elsewhere made sense.  There are plausible scenarios for AR under which that would be economically irrational after some point in 2013, versus just continuing to execute on AR.)
  • Also, strongly consider gulp hiring.  Which I’ve been saying for two years now, but one of these years it will probably happen.
  • Continue to wrestle with the questions of whether “I devote 100% of my work efforts to AR, take investment, and  take a shot at an eight or nine figure exit five years from now.” sounds like an attractive option and, if so, whether now is the time to pull the trigger on it or not.  I go back and forth on this.

Consulting

  • $300k in sales looks like a decent number to shoot for, assuming I’m actively available for consulting all of 2013, which is not a given.  (That means that I have availability throughout the year, rather than meaning that I have 52 weeks of availability — consulting is a very part-time thing for me.)
  • Continue to adjust rates such that clients and I are mutually happy with engagement outcomes.
  • Schedule things better to pack work more densely into fewer, shorter trips abroad.  (Delta really enjoys me flying 100k miles a year but Mrs. McKenzie doesn’t, particularly when it means six weeks away.)  If this results in less availability, that isn’t an unhappy outcome.

Productized Consulting

  • Do more stuff along these lines, since it worked out pretty well in the experiment, I can only see it working better with improved execution, and the project ended up being a lot of fun.
  • Let’s pluck a number out of thin air for a numeric target: $200k in sales.
  • Offer better packaging options for later products, including some sort of scheduled, scalable live component like webinars, which would provide a lot of value for customers, justify higher price points, and not disrupt family life or the other businesses’ schedules too much.
  • Outsource more of the execution of collateral tasks in the future, like video editing and programming for the sales site.

A Brief Personal Note: Ruriko and I got married on June 23rd.  Words can’t express how wonderful she is, including tolerating my weird little hobbies, like entrepreneurship.

I think that, aspirationally, career/job/business/etc was never supposed to be my #1 priority, but be that as it may it sucked up a disproportionate amount of my twenties.  I have no immediate plans for retiring, but will work on having my stated priorities more closely match my allocation of time and attention in the future.

Bingo Card Creator (and etcetera) Year In Review 2011

I’m Patrick McKenzie (patio11 on the Internets) and for the last several years I’ve run a small software company.  My first product was Bingo Card Creator, my current product focus is Appointment Reminder, and I do occasional consulting for a variety of clients, mostly on helping them sell more of their software over the Internet.

Traditionally, right before Christmas every year I release an annual report.  See, for example, 2006, 2007, 2008, 2009, and 2010.  (Crikey, have I really been doing this for that long?)  I’ve also traditionally published live stats for Bingo Card Creator, but not my other lines of business.

Writing the annual report is partially to keep me grounded, partially to talk through my thoughts on the year and goals for next year, and partially to (hopefully) give other folks ideas that they can use in their own businesses.  I hope you find it interesting or, at the very least, mildly amusing.

Obligatory disclaimers: Assume any statistics that I give are “roughly accurate, to the best of my knowledge, at the time this report was written.”  There are still a few weeks left in the year.  Sales are typically low in the last two weeks, but the exact timing of credit card charges can cause a bit of jitter in the December stats.  From past experience, I have a high degree of certainty that there are about $1,000 or $2,000 of expenses (across all lines of business) which aren’t in the bookkeeping  system yet and won’t be until I sit down in March and check things for taxes.

Capsule summary: Best year ever, by a lot.  Broke $100,000 in sales for the first time and increased total profits to ~$70k.  2012 has inflection points coming for life and the business.

The Year In Brief

I put Bingo Card Creator into maintenance mode for approximately 48 weeks out of 2011: I only answered emails and kept systems running, but took no action to improve the product or marketing.  (The other four weeks I tried a few minor things out.)  This was, theoretically, supposed to free me to spend most of my efforts on Appointment Reminder…

… but that didn’t end up happening.  For a variety of reasons, most of my focus business-wise went into consulting.  Although I technically only did about 10 weeks of consulting during the year, I spent quite a bit of overhead time on e.g. arranging deals which ended up falling through, arranging the deals which did actually go through, and doing general promotion activities like speaking at conferences.  (I had the opportunity to speak at about a half dozen conferences this year, and assorted other events.  It is great fun, but since I generally have to fly to America for them, they tend to munch a full week out of my schedule each.  I spent almost three months of the year in the US, doing a combination of family events, consulting, prospecting, speaking, and meeting some Internet buddies to discuss plans for later.)

I also lost two solid months due to dealing with legal issues, mostly centering around Immigration.  I’d love to fill you in on the nitty-gritty, but have been asked not to by people close to the situation.  Suffice it to say that I was a shoe-in for a Japanese visa back when I worked at a large megacorp, was not a shoe-in for a visa when doing my own thing, and had a very hairy experience with getting them to approve me as a “self-employed engineering consultant.”  Tips of the hat to my Japanese clients, particularly Makeleaps / Webnet IT and myGengo, whose support was instrumental in getting Immigration to approve my renewal.

Despite not having nearly as much time to work on Appointment Reminder as I would have liked, I did manage to firm up its technical underpinnings, add new features requested by clients, and do a small amount of work marketing it.  I hope to make that more of my focus in 2012.

Bingo Card Creator

Despite being in maintenance mode, BCC continued performing like a trooper.  People always ask “Could you afford to live on it only?” and the answer is “Yes, but barely, and it would require a lifestyle adjustment, mostly in the don’t-fly-across-the-Pacific-so-often department.”  BCC did not meet the numeric goals that I had for the year.

Stats:

Sales: 1,539 (up 6% from last year’s 1,451)

Refunds: 14 (down from 22 last year, to .9% of sales from 1.5%)

Sales Net Of Refunds: $45,479.93 (up 5% from $43,398.55)

Expenses: $22,560.00 (up from $18,287.93, but largely just due to an accounting issue — I can’t split costs in my homegrown bookkeeping software, so the ~$3,000 I paid for servers for AR is hiding in that number)

Profits: $22,919.93 (see above accounting issue, essentially flat from last year’s $25,904.66)

Wage per hour: Let’s see, ~15 hours of programming, 20 minutes a week on customer support…  about $700 an hour.  Not too bad.

 

Web Stats:

(All stats are from bingocardcreator.com unless otherwise specified.)

Visits: 821k (up from 777k)

Unique visitors: 670k (up from 655k)

Page views: 2.9 million (up from 2.7 million)

Traffic sources of note: Google (46%), AdWords (18%), Binghoo (13%)

Trial signups for online version: 82,000 (up from 72,000)

Approximate online trial to purchase conversion rate: 1.8%

 

Narrative Version:

Aside from kicking up AdWords spend modestly (to no good effect) and running a few A/B tests, nothing really substantial happened with Bingo Card Creator this year.  I lost probably $1,000 to $2,000 of sales when the site crashed right during the middle of the Halloween rush for ~9 hours while I was on an airplane.  That was a little disappointing, but while it broke my candy budget it won’t exactly put me in the poorhouse.

Projections that BCC would continue to grow despite not being actively worked on turned out to be totally wrong.  I forecast 50% growth, reasoning “Hey, most of the systems work pretty much without my intervention, so I think the overall growth of the Internet plus a few A/B test means, oh, 50% or so.”  It mostly tread water.  I’m not hugely disappointed.

 

What Went Right:

  • Not having to work hardly at all for it.
  • Aside from the Halloween crash, the system was largely stable for the year.  I think I got woken up by the automated alarm maybe once.
  • SEO, AdWords, email marketing, and the usual scalable marketing stuff continued to be my bread and butter even when I was too lazy to actually cut and butter bread.

What Didn’t Work So Well:

  • Crashing on the third busiest day of the year, in such a way that it depresses my AdWords campaigns for the first and second busiest days of the year.
  • I integrated Stripe and expected a huge lift in conversions for going from Paypal to a simple CC-based payment system.  I tested this extensively in A/B tests.  I love everything about the Stripe system, but I have no evidence for “Stripe is better than Paypal/Google Checkout”, “Stripe/Paypal/Google Checkout is better than Paypal / Google Checkout”, etc etc.  That said, it might be something as simple as my buttons being ugly.  I’ll probably take a whack at it in the future, or better yet, have my designer take a whack at it.

Consulting

I did a few weeks of consulting this year, for several different clients.  Mostly, I do my engineering / marketing shtick for software companies, although some of my clients have been a wee bit farther afield.  I wrote up a fairly typical engagement with Fog Creek.  That one was a mutual success and we’ll continue to work together in the future.  (To the best of my knowledge, all of my consulting clients are happy with my work.)

One thing I’m going to do differently in the future is to work for less clients.  Don’t get me wrong: I love all my clients.  I was privileged to work with them.  However, it takes approximately X units of work to set up an engagement with a previous satisfied customer, 5X units of work to get a new prospect to the go/don’t-go decision on a new engagement, and I generally have to get three to four prospects to that point to actually wind up with a signed contract.  As my buddy Thomas at Matasano says, “That is life in the big leagues.”  However, since I’m not in a position where 100% utilization is a huge overriding goal of mine, I don’t need to keep the new prospect pipeline totally full… so I’m probably going to cut back on it quite a bit in 2012.  I’ll continue doing follow-up engagements for established clients where it makes mutual sense to do so, and I’m still of course available for interesting projects, but I’m not going to be doing six-week fly-across-America-four-times tours to drum up new business.

The following numbers are approximations only.  NDAs and having the sense God gave a tadpole constrain me from revealing my “going rate.”

Consulting sales: $55,000

Consulting expenses: $13,000  (mostly hotels and airfare for prospecting, which I pay for out of pocket.)

What Went Right:

  • Client selection.  I was, again, privileged to work for people who have interesting businesses, problems that I could make substantial contributions on, and the willingness and ability to pay all invoices in a timely fashion.
  • Raising rates.  My first guesstimate at my rate, back in 2010, was $X.  It turns out that I could do just about as much work as I wanted regardless of whether I charged $X, $2X, or $5X.  As a result, I typically quote fairly high rates and mostly stick with them, unless there is another reason I really, really want an engagement to happen.

What Didn’t Work So Well:

  • Disorganization.  At one point I was juggling something like five simultaneous proposals out while preparing for three conferences, two engagements, and six weeks of travel.  It got so bad that I showed up at a city once and checked at the airport for where I was staying, quickly seeing that I mistimed a conference by three days and thus had no hotel booked, booking a hotel from the taxi, and then arriving at the hotel to recheck my schedule and discover that I had used the previous year’s schedule and was actually simultaneously at a different hotel across Brooklyn.  (Shoutout to the Brooklyn Beta guys for saving me from my own stupidity that week.)  There were multiple points in the year where I found myself wishing for either a boss or a secretary or somebody to just say “Show up to X on Monday and Do Stuff and all the stuff that is not Stuff will be taken care of.”  My occasional slipups in dealing with the demands of a growing business caused me to drop balls in ways that were sometimes client-visible, too.  This is a major part of the motivation for cutting back next year.  (There is Plan B, of course: hire folks to do either the execution or the admin and take whichever part they’re not doing, but I don’t think I’m moving in that direction.)
  • Too much work!  Largely due to overhead and travel, plus the outsize distraction generated by the same, consulting munched a heck of a lot more time than I thought it was going to.  I wanted to have a solid eight months of the year to work on AR.  I think I probably got maybe two.

 

Appointment Reminder

I launched Appointment Reminder last December, with the goal of having approximately 200 customers and $10k in monthly recurring revenue by now.  I had planned on focusing for most of 2011 on marketing and selling it to more businesses.  That largely didn’t happen, but since I got the fundamentals of my SEO strategy in place (while largely ignoring the modestly more advanced content creation / etc that runs BCC and that I usually help clients with), the business grew despite my best efforts at totally neglecting it to focus on consulting and not getting deported.

AR has been hanging around at a crossroads for a while now.  There are two very different trajectories it could go down.  In one, I grow it organically, and it grows into a modestly profitable software business which will provide handsomely for my family and (in the fairly near future) employees.  In two, I take outside investment, and attempt to grow as quickly as possible to $N million a year in revenue, at which point options would include either a) selling to one of the larger players in the small business software space or b) continued operations at scale with a focus on growth.  Luckily, I have  the luxury of waiting on making that decision: my runway is infinite, the market opportunity is only getting bigger, and the perceived value of my involvement with a startup among investors does not appear to be depreciating.

This is one of the reasons I can’t be as open as I would like to be about the current status of the business.  BCC has essentially no secrets, and would not really benefit from having them, as — aside from elementary school English teachers — there is nobody out there who has something I want for BCC.  However, if I hypothetically wanted to take investment, then accredited investors suddenly have something I want very much and having secrets about AR gives me something with which to trade to get it.  (It is similar to not putting prices on an Enterprise Software website.  You can trivially get them, but the price of getting them is giving a salesman permission to give you the spiel.  Similarly, folks who ask about AR’s numbers these days are generally asking in the hopes that they eventually receive a phone call asking them for a check.)

The other reason I can’t talk about AR numbers so much is that I radically underestimated how important the enterprise market would be to the business, and you can’t spell enterprise without NDA.

So: I wanted to have two hundred customers by now.  For the publicly available plans, I currently have a few dozen paying customers.  There are ways to get things from me that don’t involve paying the numbers on the Pricing page.

AR is modestly profitable — it covers all of its own costs.  I plow most of the money it generates back into the business, though, rather than taking distributions.  For example, I’m now about 95% certain that I will have significant contractor or employee involvement on it in 2012.

Revenue: Undisclosed

Expenses: Undisclosed (very modest ongoing expenses, reinvested most profits)

Profits: I took about $5k just to have a number that would minimize disbelief at the tax office.

What Worked Right:

  • Twilio.  The Twilio API and service have been unalloyed epic wins for Appointment Reminder.  I had zero disruptions in service attributable to them, their customer support has been fast, responsive, and technically savvy (even helping me debug my own code at points), and they’ve been very supportive of me.  Plus they have these awesome red track jackets that they keep sending me, which you’ve probably seen if you’ve seen a picture of me doing any talk this year.  (I actually wear them mostly because I love the color red, but apparently I wear them so often that folks at the Fog Creek office thought the Twilio logo was my logo.)
  • Sendgrid: It’s like Twilio, except for email.  Great service.  No red jackets.
  • Unit testing & staging servers.  I am gradually getting more sophisticated in my engineering practices, and have been ramping up my testing activities since starting to code AR.  It has transformed the way that I do development, for the better, and made it easier to respond to customer requests to change things while decreasing the number of problems I have caused.  Total win.  See my presentation at TwilioConf for examples of the specific ways I use it for AR.
  • Exact match domain names.  “Hey Patrick, how is it that with no marketing budget and nearly no marketing work you rank #1 for [appointment reminder]?”  I told everybody that I was buying the .org specifically because that would happen but apparently folks didn’t believe me.
  • Using the self-service site as lead generation for enterprise sales.  Fairly self explanatory.
  • The service itself: AR solves a clear customer need, and my customers are raving fans of it.  There exist many services businesses which incur hundreds in direct costs and thousands in forgone revenue for a single missed appointment.  (Think, say, an HVAC company which sends a three-man team of tradesmen out to your house to replace your heater, which is a $2,000+ job, only to discover that you aren’t home to let them in.)  One of my customers reports that just the delta in no-shows since starting to use AR would pay for his mortgage and his daughter’s college education.  Many of my other customers report that their office managers, who previously did telephone reminder calls manually, are ecstatic to not have to do them any more.  Customer retention among folks who actually use the system (as opposed to signing up, doing a test call, and forgetting about it) is virtually 100%.
  • Talking to smart people for advice: Since I’ve been going back and forth on the investment question, I talked to a lot of entrepreneurs and investors whose opinions I respect.  I really appreciate their feedback, which ranged from “Are you kidding?  You’d hate it.” to “I want to invest in you, but realistically, you would lose nothing by waiting until you are sure.” to “Best decision I ever made.” and helpfully included a lot of actionable advice on how to do things in the meanwhile such that options remain open.

What Didn’t Work So Well:

  • Catastrophic engineering failures.  I had one combination outage/catastrophic failure in February (the details are recounted in that TwilioConf presentation) and a ~3 day period of sporadically degraded operations after my move to Rackspace, which I finalized over the Thanksgiving holiday.  Both of those were my fault, for architecting the system in a way which did not gracefully handle its multiple moving parts getting out-of-sync with each other.  I’ve since done significant work on making it more stable.  (Overall reliability for the year has been excellent, but those periods were easily the most stressed I’ve ever been about any business issue.)
  • Lack of focus: I’ve been commenting above on this, so I won’t belabor the issue, but I really didn’t get to work on AR as much as I wanted.
  • Enterprise sales: I’m actually fairly decent at Enterprise Sales, and am working with someone in the industry who has a deep Rolodex among folks who would be great candidates for AR, but (partly due to the focus issue and partly due to my own comfort level) I didn’t put nearly enough effort towards it this year.  What I should honestly do is go to a conference some time, prospect like a madman, and then make following up on those leads my only job until I’ve got contracts signed.  (The prices for enterprise SaaS make this very economically viable.)

Goals For 2012

Bingo Card Creator

  • I’d be happy with continued flatness ($~30k profits on $50k sales), maybe.  It isn’t the source of growth for my business anymore.
  • Continue using it as a laboratory for weird ideas I have on conversion optimization.
  • Don’t break it during Halloween.

Consulting

  • Do less work prospecting for new clients.
  • Do more work for existing clients.
  • Modestly increase billings, if that makes sense for where my overall business is.  (If I take external investment in AR, that will likely require shuttering the consulting business.)

Appointment Reminder

  • Figure out whether I want to take investment or not.  If so, do so.
  • Convince Keith (who I do my podcast with) to work with me, if possible.  (Don’t worry, he knows this is on the agenda.  We’re best friends.)
  • See about transferring responsibility for the engineering (particularly front-end) side of things so I can focus on marketing/sales.
  • 10x current sales numbers.  That seems to be a fairly safe bet regardless of whether I shoot for a small business or for a high-growth business.  (1,000x-ing would be another story.)

A personal note: The last 3,300 words ultimately matter much, much less than the next 3: she said yes.  We’re announcing to our family on Christmas, as per our family tradition.

New Trends In Startup Financing Explained For Laymen

Noted American technology investor and all-around smart guy Paul Graham wrote recently about emerging trends in startup funding, specifically that convertible notes and rolling closes are displacing the traditional equity rounds done at a fixed valuation done with angel syndicates.

Did that sound like Greek to you?

Great, you might benefit from this translation of Financier into Geek.  (P.S. If you haven’t figured out the significance of it originally being written in Financier instead of in Geek, please, think it through.)  I originally wrote it as a comment on Hacker News but somebody asked me to put it somewhere easily findable.  I have elaborated with standard blog post formating and graphs where I thought they helped the explanation:

Why We Care About Angel Investing

Startups raise money from investors to accelerate their growth into, hopefully, massively profitable businesses and/or massively large acquisitions from big companies.

One particular type of investor that invests in startups is called an angel investor. An angel investor is often an individual human being who is wealthy, frequently as a consequence of successful entrepreneurship. They invest anywhere from $25,000 to $250,000 or so.

Fundraising is painful, and requires a lot of time and focus from startup founders. To mitigate the pain, it is often structured in terms of “rounds”, where the startup goes out to raise a particular large sum of money all at once. For an angel round, let’s say that could be a million dollars. (n.b. It is trending down, as companies can now be founded for sums of money which would have been laughable a few years ago.)  Clearly we’re going to need to piece together contributions from a few angels here.

Why Angel Investing Frustrates Founders

Traditionally, one angel has been the “lead” angel, who handles the bulk of the organizational issues for the investors. The rest just sit by their phone and write checks when required. (Slight exaggeration.) Investors are often skittish, and they require social proof to invest in companies, so you often hear them say something like a) they’re not willing to invest in you but b) they are willing to invest in you if everybody else does. This leads to deadlocks as a group of investors, who all would invest in the company if they company were able to raise investment, fail to invest in the company because it cannot raise investment.

Startup founders are, understandably, frustrated by this.

What “Valuation” Means

All numbers below this point were chosen for ease of illustration only.  They do not represent typical valuations, round sizes, or percentages of companies purchased by angels.

One item of particular interest in investing is the valuation of the company. This gets into heady math, but the core idea is simple: if we agree that the company is worth $100 at this instant in time (the “pre-money valuation”), and you want to invest $100, then right after the company receives your investment, the company is worth $200 (the “post-money valuation”). Since you paid $100, you should own half the company.

Traditionally, the company has exactly one pre-money valuation (which is decided solely by negotiation, and bears little if any relation to what disinterested outside observers could perceive about the company). All investors receive slices in the company awarded in direct proportion to the amount of money they invest. Two investors investing the same amount of money receive the same sized slice of the company. This can be written as “they invested at the same valuation.”

The thesis of PG’s essay is that allowing investors to invest at the same valuation is not advantageous to the startup. Instead, by offering a discount to valuation for moving quickly, you can convince investors to commit to the deal early, thus starting the stampede from the hesitant investors who were waiting to see social proof.

For example, take the company from earlier. We said it was worth $100 prior to receiving investing, but that is not tied to objective reality. Say instead we’ll agree that it is worth $80… but only with respect to the 1st investor. He commits $20. $80 + $20 = $100, so he gets $20 / $100 = 20% of the company for $20, or $1 = 1%. This convinces a second investor to invest. He says “Can I get 20% for $20, too?” Not so fast, buddy, where were you yesterday? The company isn’t worth $80 any more. We think it is worth $105 now. (Did we just get through saying $100? Yes. But valuations are not connected to objective reality.) So you get $20 / ($105 + $20) = 16% of the company for your $20. Think that is fair? You do? OK, done.

This continues a few times. The startup raises money — possibly more money, depending on how much the angels want in — with less hassle for the founders.

What Is A Convertible Note?  Why Do Founders Like Them?

We’ve been talking about just dollars so far, and alluding to control of the company as if it were equity like stocks, but there is a mechanism called “convertible notes” at play here. A convertible note is the result of a torrid affair between a loan and an equity instrument. It looks a bit like Mom and a bit like Dad. Like a loan, it charges interest: typically something fairly modest like 6 to 8%, much less than a credit card.

The tricky thing about convertible notes is that they convert into partial ownership of the company at a defined event, most typically at the next round of VC funding or at the sale of the company. So, instead of the first investor getting $20 = 20% of the company, he loans the company $20 in exchange for a promise like this: “You owe me $20, with interest. Don’t worry about paying me back right now. Instead, next time you raise money or sell the company, we’re going to pretend that I’m either investing with the other guy or selling with you. The portion of the company which I buy or sell will be based on complicated magic to protect both your interests and my interests. If you want to sweeten the deal for me, sweeten the magic.”

Do we understand why this arrangement works for both parties? It incentivizes investors to commit early, which lets startups raise more money with less pain. Because startups are in the driver’s seat, it also lets them avoid collusion among investors (“We decided we’d all invest in you, but we don’t think the company is worth $100. We think it is worth $50. Yeah, that has no basis in objective reality, but objective reality is that your company is worth $0 without the $100 in our collective pockets. What is it going to be? Give up 2/3 of the company, or go broke and get nothing.”)

How Do You Calculate The Equity Value of A Convertible Note?

OK, back to complicated magic. When the company takes outside investment, the convertible notes magically convert into stock, based on:

  • a) the valuation the company receives for the investment round  (higher numbers are better for both founders and angels)
  • b) a negotiated discount to the valuation, to reward the angel investor for his early faith in the company (higher numbers are better for angels)
  • c) possibly, a valuation cap (higher numbers, or no cap,  are better for founders)

For example, continuing with our “low numbers make math comprehensible” startup, let’s say it goes on a few months and is then raising a series A round, which basically means “the first time we got money from VCs”. We’ll say the VC and startup negotiate and agree that the company is worth $500 today, the VC is investing $250, ergo the VC gets a third of the company.

How much does our first $20 angel investor get? Well, he gets to participate like he was investing $20 today, plus he gets a discount to the valuation. So instead of getting $20 / $750 = 2.67% of the company, maybe he got a 20% discount to the valuation, so he gets $20 / (.8 * $750) = 3.33% of the company. (We’re ignoring the effect of interest here for simplicity, but he probably effectively has $21 and change invested by now in real life.)

After this is over, the convertible note is gone, and our angel investors are left with just shares (partial ownership of the company), which they probably hold until the company either goes IPO or gets bought by someone. So if the company later gets bought for $2,000 by Google, our intrepid angel investor makes $66 on his $20 investment.

How Does A Valuation Cap Work?

We haven’t discussed valuation caps yet. Valuation caps are intended to prevent the startup dragging its feet on raising money, thus building up lots of worth in the company, and then the angel investor getting cheesed. For example, if they had just grown through revenues for a year or two, they might be raising money at a valuation of $1,250. In that case, $20 only buys you 2% of the company (remember, he gets a 20% discount : $20 / (.8 * $1250) = 2%), which the angel investor might think doesn’t adequately compensate him for the risk he took on betting on a small, unproven thing several years before. So we make him a deal: he gets to invest his $20 at the same terms as the VCs do if, and only if, the valuation is less than $750. If it is more than $750, for him and only him, we pretend it was $750 instead. This means that under no circumstances will he walk away with less than $20 / (.8 * $750) = 3.33% of the company, as long as the company goes on to raise further investment. (Obviously, if they fold, he walks away with nothing. Well, technically speaking, with debt owed to him by a company which is bankrupt and likely has no assets to speak of, so essentially nothing.)

Perhaps This Will Be Clearer With A Picture

Angels ultimately benefit from higher discounts to the valuation of the Series A round, and lower valuation caps.  Higher discounts, and higher effective discounts, mean you get more of the company for less money.  That is an unambiguous good, as long as you keep the quality of the company constant.

Let’s see how valuation caps affect how much of the company you end up with.  The better the company is doing by Series A time, the less of the company the angel ends up with.  This shows the incentive for the founders: do as well as you can prior to raising money, which is the same incentive founders always have.

As you can see from the below graph, a valuation cap essentially gives the angel an artificially higher discount for if the Series A valuation exceeds the valuation cap.  Obviously then, it is in the interest of angels to negotiate as low a cap as possible, and in the interests of founders to negotiate a high cap or no cap at all. According to Paul Graham, this becomes the primary “pricing” mechanism in the new seed financing economy: if a founder wants to reward an angel, they award them with a lower cap.  If they don’t, the angels get a higher cap, or no cap at all.  This kicks discussions of valuations down the road a little bit, and allows you to simultaneously offer the company to multiple angels at multiple “price points”.  That allows you to reward them for non-monetary compensation (mentoring, having a big name, etc) or for early action on the deal.

This Is Not My Business. Take With A Grain Of Salt.

Lest anyone get the wrong impression, my familiarity with angel investing is very limited and, to the extent that it exists, it is mostly about angel investing in small town Japan.  (Oh, the stories I can’t tell.)  The above explanation is based on me processing what I’ve read and trying to prove that I understand it by explaining it to other people.  If I have made material errors, please correct me in the comments.

My current business is not seeking funding (and would be an extraordinarily poor candidate for it).  I’ll never say never for the future, but for the present, I rather like getting 100% of the returns.

[Edit: Want to use some or all of this, including the graphs, for your own purposes?  Go ahead.]