I never touched Unix in my life before going to college, but it was the development environment of choice for most of the upper-level coursework at school (lower level was Windows + Emacs… not a combination I would recommend given the excellent Java IDEs which exist right now, incidentally), and so I’ve gotten very used to having my good friends ls, grep, wc, and gawk available on my machine.  I use Cygwin to have them available on my Windows box, because while Linux is a lovely operating system in many ways I rather enjoy having iTunes and games available.

Anyhow, I had a task yesterday: I have a local copy of my website.  All of the pages have a very similar structure because the navigational elements are embedded in static HTML.  However, this didn’t use an official templating feature because NVU doesn’t support any, which means updating the navigational elements means updating approximately twenty pages by hand.  There was one particular snippet of HTML that I wanted to change to another, globally.

Normally, I would do this in a gawk script (you can do just about anything in a gawk script), but it gets a little messy:

BEGIN {

startTag = “<a href=\”guarantee.htm\”>Money-back Guarantee”

replacementTag = “<a href=\”guarantee.htm\”><img src=\”images/money-back-guarantee.htm\” alt=\”Money-back Guarantee\” \\>”

}

($0 !~ startTag { print $0}

($0 ~ startTag) {

gsub(startTag, replacementTag, $0);

print $0;

} 

Anyhow, that is the way to do it in gawk.  Define your regular expression to search for, print out every line that doesn’t match it and do some surgery on the ones that do.  Note that you’ll have to run this on every file in your web directory, with the easiest way probably being a bash script, and output the results.  However, if you output to the file directly it will generally end up empty for reasons that I have never bothered really understanding, so you probably want to have another quick shell script to output it to a temp file and then copy over the original.  Something like, oh,

for file in *.htm do

  gawk -f myscript $file > temp.htm

  mv temp.htm $file

done

But that is two scripts, and two scripts is too complicated!  Enter sed, a handy little stream processing utility.  Sed is perfect when you just need to do a quick substitution for a regular express:

sed -e “/myregularexpression/myreplacement/”

There, you’re done.  Or in my case, you have to finangle the regular expression into a string, which involves quite a bit of escape characters, but it still only took about 30 seconds.  This saves about half of your typing time versus doing it in gawk, and you can’t accidentally bork the matching logic (something which is easy to do in gawk — forget the ! before the ~, or execute the (string ~ regex) block before the (string !~ regex) block, and you’re screwed.

Of course, you could always write it in Perl.  That would probably take a single line and look like your cat had just tap danced across your keyboard.  But if you want a simple, intuitive way to do fairly powerful data processing, you’ll see sed is your man…  err, I mean, you should see man sed.

… perhaps I haven’t overdefined my target market.  On the one hand, one of the nice things about making Bingo Card Creator pretty much built around the assumption that my user is a teacher is that its very easy to consider possible features and marketing campaigns in that light.  I can make assumptions, like assuming that she (I can assume she is a she!  Sort of a rarity in software development, I would think) is not as comfortable using her computer as I am using mine.  I know how she spends her day (she visits Bingo Card Creator during planning periods, lunch, or when lesson planning at home at about 9:00 pm), I know what her key problems are, I know that she absolutely melts when I say my favorite character in Harry Potter is Hermione (totally true, incidentally), and I more or less speak her language (I know what a manipulative is, for example, to pull a random word out of my search queries for this week).  All in all, I’m in a good place to be with regards to my target market.

But my target market isn’t the whole of people who could use Bingo Cards.  I sell a few copies (impossible to know how many, of course — I can only guess based on customer information provided, email addresses, support requests, that sort of thing) to folks who are not teachers but nonetheless click through 100% teacher-oriented advertising copy to buy anyhow.  I have had repeated purchases from folks organizing corporate events, where my icebreaker bingo idea is a natural fit.  And today I got, lets see if I can anonymize this a little… OK, pretend there is a National Association of Skateboard Punks.   There isn’t, to my knowledge (apologizes to the NASP if there is), but a customer about as non-elementary-schoolteacher as NASP wanted Bingo Card Creator for some advertising needs.  Now, I had certainly been of the opinion BCC was a one-use program (it makes bingo cards, and does nothing else, by design), but I know of folks who are currently using it to make worksheets, track patient progress in rehabilitation, make customized placemats (print the card, mount it, laminate — apparently crafty-types use computers now), and now advertise SkatePunkz2k6, the annual get together for the discerning skate punk.

I am considering, after my project calms down at work and my life returns to some semblance of normal, retooling the website a bit to accomodate these multiple user groups.  Minimally, I would like to have one “neutral” presentation which doesn’t assume teacher-hood for the program, and probably one pitch for corporate-types planning for the annual retreat (Synergize your best-of-breed team-building paradigm with Bingo Card Creator and achieve six sigmas of internal customer empowerment!).   Hopefully it will lead to a bit of natural market growth without losing my main focus on the educational market.

(Well, I suppose that technically all of my sales are international, since I’ve never sold a copy in Japan.  Be that as it may, the business is headquartered in the US even if I’m not, or at least thats my story and I’m sticking to it when it comes to taxes and banking matters.)

Bingo is popular throughout the English speaking world (and in some points beyond, too), so its not too shocking that I have a few customers from outside the US.  Probably, oh, off the top of my head 10% or so.  I was unsatisfied with my previous level of service for them, chiefly because I had the prices listed only in dollars and this both confused the Australians (hey, I could just have easily have been an Aussie uISV, and there is nothing in my website to tell you differently — doh) and costed all of my customers currency exchange fees when dealing with Paypal.  I hate currency exchange fees with a burning passion in my soul, having been burned by them plenty (Bank of America will happily ding you for 3% a transaction).  So I decided to price my product natively.  There are a couple of issues here.

VAT: Ahh, I love the smell of overly-complicated growth-killing Eurocrat mandated regulations in the morning.  VAT, for value added tax, is a freaking gigantic consumption-esque tax you have to worry about if you sell in Europe.  My position on VAT is simple: I’m a small businessman, I have never set foot in Europe, and I do not want to deal with the hassle of collecting this tax.  I give my customers one method (using eSellerate), which has them pay the tax (something like 20% of the purchase price!) and one method (Paypal) which doesn’t.  They can worry about their own local laws.  Incidentally, with respect to my local laws, I’m scrupulous about not actively selling in Japan (because I don’t have the legal structure required to remit Japanese sales taxes yet — thats 5%, incidentally), and I will pay sales tax on all of my orders in Illinois.  I don’t know if Illinois will be very happy that I don’t ask for people’s address unless they ask for a CD, but I’m highly unlikely to pass the minimum dollar amount for tax reporting anyhow.

What to charge folks: I got out Google’s currency converter, plugged in $24.95 in the various currencies, rounded to a convinient number in whatever direction was required, and then chopped off 5/100 of the local currency.  My guiding principle was to be mostly fair without sacrificing the convinience of round numbers, which as I recall results in the Brits paying a wee bit more and the Aussies paying a wee bit less relative to the USD than other customers.  However, I’m pretty sure all of my international customers benefit in that I absorb their currency conversion costs through Paypal rather than them absorbing it through their banks.  (This is a matter for me of a few dimes worth of profit per sale, less than nothing really, but its a small thing you can do to really endear yourself to a fraction of your customers.  I’m all about that.)

I’m thinking of officially incorporating myself next year, but there are some non-trivial issues with incorporating in Japan (“Proof of 3 million yen worth of business capital…  hmm, well, can I count all the money I spent on college as human capital?  Because thats the only way I’ve had nearly $30,000 at any point in my life.”)

I am not an accomplished Perl hacker and something has been frustrating me at work for the last several weeks.  If you know of an easy way to solve this, I will be forever in your debt:

We are using Perl to run a HTTP server (HTTP::Daemon, incidentally, works like a charm) to receive information from a rich client application.  The client has to upload text files via HTTP post.  Everything is A-OK on the client’s side of things, but I have no clue how to go about extracting the uploaded text file from the HTTP request on the server side of things.  My experiments have been a bunch of failures, I can’t seem to find any CPAN Module which will do this cleanly, and there are other deliverables up in the air which are getting held up by the server being only 98% complete (that last pesky 2% is critical to testing the client application, sadly).  Does anyone have any suggestions?  Perl’s documentation is, well…  up to its usual standard of lucidity.

Steph of LandLordMax fame recently posted about how to deal with email spam.  This just happens to be the focus of my day job (we’re busy exploiting the synergies of a new paradigm of… actually, no, I don’t work in Dilbert’s office).  I use spam filtering religiously on all my email accounts (4 at last count) and it saves me roughly 2,000 spam a day (my business email address is forwarded to by my college email address, which was widely circulated back when I was young and stupid).

Here is everything you need to know about spam prevention in two sentences sentence: If you are comfortable administering your own email server (and already do), use SpamAssassin.  If you are not, use Popfile.  There, you’re done and didn’t even have to pay me my company’s consulting rate.

SpamAssassin takes a multi-tiered approach to dealing with spam: it has rules written by the administrator (or, if you’re a good outsourcer like any uISV should be, you just download them from the Rules du Jour website) which are handwritten regular expressions which look for spam signs, it has RBL checking, and it has a Bayes statistical engine.  Popfile just does Bayes.  You can find descriptions of exactly how Bayes works elsewhere on the Internet (Paul Graham was one of the early pioneers of the idea, incidentally: see his seminal article A Plan for Spam, with the caveat that the field has improved significantly since then and that you should consider this more of a historical piece than a tech blueprint).

Both SpamAssassin and Popfile, when trained properly, will produce well over 99% accuracy (both in not improperly labeling good mail as spam, and in not missing spam).  Exactly how you do the training depends on your installation.  If you are a mail administrator, start reading about SpamAssassin and understand you will be spending much of your free time feeding the monster.  If you aren’t, can I say for the second time, Popfile will save your life.

Popfile works as an email proxy, which means instead of having your email client (Outlook, Thunderbird, whatever) dial up your server when it wants mail, you have it dial up a port on your local computer instead.  There POPfile says “Hold on chap, just one second” and runs to fetch your mail from the actual server.  Popfile then scans the mail, uses statistical witchcraft to determine whether it is spammy or not, and marks it appropriately (via one or both of adding another message header to it or changing the subject line to read “[spam] Do 1+ 4 yur l0v3d 0n3″).  You then either just scan those subjects yourself or, if you’re smart, configure your mail client to automatically move everything tagged as spam into a special folder, which you periodically run through (look for missclassified emails and then purge).

Classifying emails is a bit of a nuisance: you have to visit a special web page (on your local machine), which you can do with a click of an icon in your system tray.  Then you have to locate the proper email (easy with a search-by-subject or sender feature) and click the appropriate classification (Popfile calls it “bucket”) and click reclassify. 

POPfile isn’t just for spam versus ham — you can have as many buckets as you want.  For example, I use Thunderbird at home to download from both my ISP email account and my bingocardcreator.com account(s).  I have POPfile configured to automatically tag any emails which are autogenerated (Amazon sales receipts, bounce mails, subscription confirmations, and the like), from family, regarding Bingo Card Creator, spam, from 2 particular mailing lists, and the like.  After you classify a shockingly small number of emails (generally less than a day’s worth in my experience, your mileage WILL vary), POPfile gets the hang of things very quickly: it “knows” that a genuine message from Paypal is invariably about Bingo Card Creator (despite it not saying “bingo” anywhere) and it “knows” that a fake message from Paypal (which it is very good at sniffing out) is spam.  My accuracy hovers in the 99.5% range.

The main problem with ANY anti-spam system is dropping legitimate emails.  I lost 3 in the last week, although two were identical (my friend hit the send button twice, apparently).  On a scale of criticality from 1-10 the email that was duplicated was like a 9.2, which was unfortunate (luckily I noticed it in a periodic scan of my spam folder).  The other one was a subscription confirmation (no big deal, since I was expecting it a search found it instantly).  For folks you absolutely want to get mails from, you can set up “magnets”, which are basically hand-written rules which override the statistics.  For example, I’ve got a magnet for my mother — anything she sends goes into family.  I’ve had another magnet for any email which quotes one of my own receipts, because that is what most of my customer support emails look like.  I stopped using that magnet because after the first 3 times that sort of mail happened POPfile was on it like white on rice.

OK, enough plugging POPFile (same story as always: opinions are my true feelings and totally my own, and nobody is sliding me money under the table from them, although you can reasonably expect that since my day job is directly related to this I am not an unbiased observer).

 Steph mentioned another method in his post: challenge/response.  Basically, if someone who you don’t know sends you an email, you send them an automated message saying “Hey, prove you’re a human by responding to this CAPCHA and I will deliver your message”.  DO NOT USE THESE.  The potential problems are numerous and they get even more numerous the more people who use them — and, since if you’re reading this you are a business, you should know that end-users HATE THEM with a burning passion.

… I am extremely busy at the day job at the moment.  Today was a deliverable for the project I am working on.  There are at least two deliverables to go, and final acceptance testing was supposed to start on, oh, August 31st?  Yep, you could say we’re under a weeeee bit of pressure to get things finished.  As a result my time on my uISV consists of responding to an email or three a week, and the odd transfer from Paypal to my bank account.  I hope to get “caught up” with it after we’re out of the woods at work.  I have the data from Oct. 1st through 15th stored for folks who are interested in stats, and will try to get that posted tonight or tomorrow night.

I’ve mentioned a couple of times that I like Direct Access, and use it on a daily basis (and by “daily” I mean “I type in a Direct Access shortcut about every 30 seconds to five minutes while working, but I do this every day, so thats daily”).  You can see my full review of the program earlier on my blog.  Actually, perhaps review is the wrong word to use: that implies objectivity, and I’ve got none here: I love the program like I love chocolate.  Anyhow, the reason I’m mentioning it out of the blue is that another site I rather like, Bits Du Jour, is running a one day 50% off sale on it.  So, if for some reason you’re still fence-sitting, you can get it for $20.

As usual, when I plug something here I’m not paid to do it.

I spent quite a bit of time in the last two weeks getting my personal finances in order (I generally do this once a year around tax time, and since I have an extension for living abroad tax time is October for me).  My situation, which is already a wee bit of a mind-bender as a result of having accounts scattered in Japan and the US, is going to be made a wee bit more difficult next year as a result of the uISV I set up.  Here is what I am doing to make this as easy as possible:

I got myself a credit card.   I have always been a cash-only person in my dealing with American companies (interest!  ick!), but I am starting to see the value of having a paid-in-full-monthly credit card.  For example, I have dreams of eventually owning a home, and that is going to require some demonstration that I am a responsible credit risk.  From the perspective of the American consumer finance industry my financial health is like that of a corpse, since the only activity for the last few years has been student loan payments.  Anyhow, I got myself a nice little Visa to help that out, and since all of my business expenses are dollar-denominated I’ll have something to charge every month.  Additionally, and crucially, with all of my business expenses and nothing else on one card it becomes very easy to file my Schedule C-EZ next year.  Schedule C is the form American small business owners report their taxes on, and it essentially asks you for your sales and expenses (capped at $5,000).  You pay taxes on the difference, at your individual income tax rate.

Its crucial that you be able to account for every dime you spent on your business, so that you can properly claim every dime worth of business expenses.  Back when I started out I paid for my business expenses on my Japanese credit card, my American debit card, and through Paypal, and as a result my record-keeping was a mess.  (I will seriously be reading my own blog when filling out my Schedule C, thats how bad it is to reconcile my credit card statements.)  With everything on one card, all you have to do is pull your statements online and sum.  This ensures that you don’t miss a single expense, because any expense you miss costs you money in taxes you pay but don’t owe.  For example, if I forgot come next October that the previous October I had paid $29.95 for stock icons (clearly a deductible business expense for me), then I would end up paying my tax rate (call it 25%) on $30 of income that those stock icons should have offset.  Thats $7.50 worth of idiot tax for forgetting a single transaction.  Don’t pay the idiot tax, keep copious records in an easy-to-access place!

The credit card also allows you to smooth out cash flow issues.  I get paid at odd times throughout the month, whenever someone buys my program and I transfer the money out of Paypal.  My bills, however, come both at set times during the month (web hosting, e-junkie), variable times during the month (Goooogle), and periodically (software/services purchases).  Having a credit card around lets me essentially reschedule those bills to come after I have had the sales to pay for them.  (N.B. I have a day job, and the day job can easily cover all my business expenses without blinking an eye.  However, the day job and the day job’s paycheck are on the wrong side of the ocean, and if my American checking account is dry when the Google AdWords bill comes that would be less than fun.  I would eat an insufficient funds fee, a week of below-minimum charges on my checking account, and a bank transfer, wire transfer, and currency conversion fee for fixing that.  So my credit card gives me a lot of piece of mind.)

I’m keeping paper files, too:  I have always been a direct-to-digital person when it comes to dealing with my finances.  However, to keep myself organized, I got myself a folder marked “BINGO TAXES” and have started feeding it three pieces of paper a month on the 31st: my copy of e-junkie’s sale tracking page, my copy of my Paypal account statement, and my copy of my credit card statement.  That allows me to reconstruct income and expenses in about 3 minutes per month.  Even if I were selling 10 times what I am now (wouldn’t that be nice), it would still be trivial.  I also keep PDF printouts of these documents on my hard drive (side note: if you don’t have a virtual printer which exports to PDF, can I recommend PDFCreator, it saves me from having to have a printer at all) and will soon be backing them up somewhere else in event of a catastrophe at my residence.

Roughly 80% of my customers do not buy software.  I don’t mean this in the sense that they are pirates, who grab software which they want without paying for it.  I mean this in the sense that their relationship to their computer is not primarily mediated through applications.  Many of them do not particularly like using their computers, and use only the programs which do what they need to do, rather than seeking out new, exciting things that they can do with their machine.  And what they need to do is generally simple: they need to create documents, they need to write emails to their nephews and coworkers, and they need to check the status of that lovely knickknack they are secretly hoping to snipe on eBay.  (Side note: about half of my customers have used eBay.  Yep.  It is that mainstream.)

Now, if I do say so myself, I’m pretty decent at selling software to these people who don’t particularly buy software.  You might be, too, if you’re wandering off the well-trodden “I am going to sell to software developers like myself” uISV path.  (Not that there is anything wrong with that, by the way, but it has its own challenges: you’ll be forever worrying about OSS coming along and eating your lunch, and your customers are in general loathe to spend money to accomplish something they think they can just whip up themselves… if they had a couple months free.)  Here are some of the things I have learned over the last few months of this.

The look of your program is critical.  Since release, I have upgraded my program with precisely three new features worthy of bullet point: a font selector, word wrap, and stock icons.  Before the upgrade (which, off the top of my head, I think was v1.03), I was getting two emails a week asking whether there was any way to do font selection, and perhaps one every two weeks regarding word wrap.  Do you know how many people ever mailed me saying “Your program looks kind of drab”?  Not a single one.  Do you know how many compliments I get in the average week now that my program looks bright, inviting, and the perfect thing for Grandma the bingo enthusiast who runs Windows98 and types her emails in ALL CAPS because that shift key is too difficult for her?  More than I have gotten for every feature request, ever.  The amount of space download sites allocate to screenshots is tiny, and mine having a splash of color in them instead of looking like a spreadsheet probably gets me some clicks I would not otherwise have gotten.

They want to know someone is home.   A certain segment of my customers needs a little hand-holding before they are willing to part with their credit card information over the Internet.  Typically, what they do is send me a single email asking a question about features: “Can I print a call list?”  “Yes, ma’am, go to the File Menu and select Print Call List.”  And then I go to sleep for the evening and wake up to You’ve Got Money.  My theory on this is that people want to know that the program is being actively maintained by a real live human being, and they want to know that if there is a problem somebody will be on the case.  Which is why I have my email address plastered to nearly every page on my website and in the program itself.  Some people say that if you’re selling at my price point ($24.95) you cannot afford to offer personalized customer support.  I say you can’t afford not to: your skittish prospective customers seeking reassurance will run away.  (I also write reassurances into my web site copy: your credit card information is safe, full money-back guarantee, upgrades available forever, free support from someone fanatically dedicated to making you happy, etc).

The ordering pathway needs to be failure proof:  I have taken significant steps towards making my ordering pathway failure proof: it is the same for PC and Mac, there are extremely few options and I bury all the ones except the most common (single copy of the software available via download), I ask for no additional information besides what I need to process the order (telephone numbers and shipping addresses?  Why?), and all roads lead to Rome from my purchasing.htm page: practically anything you click within the main body of the page will take you directly to the ordering page.  (Side note: purchasing.htm is NOT the ordering page because people who hit it typically have come from a link in my application.  Many of them are just interested in seeing what the link does or checking out the price.  As a direct consequence, I use purchasing.htm to make my “Close the deal” sales pitch to wavering prospects while simultaneously allowing folks who have their credit card out to consumate the purchase in 15 seconds or less.  My average is actually 42 seconds, most of it spent waiting for Paypal’s site to load by my guess.)