Well, I flagged a return to failposts in my last failure, and not a moment too soon. Things have been a bit shaky since it took so much effort to catch up recently, and when I put off other work to do it the other work tends to return with a vengance later. I think the other part is that I'm still trying to establish a habit of writing in the morning rather than the evening. Previously, I would just stay up late to write if I had to, but I've come to see that as a form of atonement so I'm trying to avoid it. Sometimes that means coming to realise that I've put my writing off for too long and I've run out of time that day.
In positive news, though, this is a much shorter turnaround to get back to caught up than previously, and it hasn't felt like the same level of enormous effort, so I think the strategy I've moved towards is a good one. I'm also trying a bit harder to stick to a red-green-refactor system for writing. Although I can write without it, RGR is faster and I've noticed that taking a long time to write tends to make it easy to fall behind and harder to fix when I do.
So all told I'm feeling fairly positive about this failure. I would prefer not to have failed, but given that I did I'm glad I could recover more quickly than I have been.
I've been thinking recently about different reasons to do prototypes and how I can better categorise them. Some, like recently, are a good way to do incremental or experimental work on a larger project. Others are functional, like these link checking prototypes; you just want to get something done with the smallest unit of code you can. And perhaps my favourite (yet most dangerous) prototypes are ones that explore an idea. I like these, but they often explode into day-long (or even multi-day) projects.
There's one other kind of prototype that I've found surprisingly useful: tire kickers. A tire kicker is when you make a prototype where the goal isn't the prototype itself, but evaluating the environment you make it in. For example, this Floyd-Steinberg prototype used AWS Lambda, but otherwise it was identical to my earlier Floyd-Steinberg prototypes. I didn't have any kind of burning need to dither Yao Ming's face at cloud scale, I just wanted to use Lambda for something and see how it was. Ultimately, I found out the answer: good, but rough around the edges.
Once I started thinking about tire kickers as a distinct category, I realised that's what a lot of my prototypes were. I tried out PureScript, Pony, Haskell v Rust, Elm, and of course Lambda. In every case I learned a lot, and in fact when I haven't done any prototypes for a while the thing I've missed the most is the regular exposure to new tools and techniques. However, even though I've gotten a lot out of them, I think I can do better.
The thing is that you don't want too many points of articulation: if you are working on a new idea in a new language, you're not going to know whether the idea or the language is responsible for the outcome. When I worked on the Haskell and Rust protoypes for the Beyoncé numbers, I was solving the exact same problem in two different programming languages and it made it clear just how good Rust is at mutating stuff quickly. On the other hand, the Pony project really didn't work at all, partly because I was trying to figure out how I wanted it to work and figure out Pony at the same time.
So this is the critical feature of a tire kicker: it's a problem you've solved before and are very comfortable with. I've heard a similar thing about evaluating sound equipment, that it's a good idea to have a few songs you've listened to a bunch of times and know really well. Of course, you could just randomly shuffle through your music, but you'll get much better information if you're comparing the same thing each time. By analogy, the most useful thing if you want to evaluate new tools or programming languages is have a small library of tire kickers that you're really familiar with.
I think Floyd-Steinberg is a pretty good one, as is the hosts file mangler, and my friends over at Prismatik have a nice one involving flaky webservices. That said, I expect the choice of tire kicker is going to vary a lot. Depending on what kind of work you want to do, a webservice tire kicker might be amazingly relevant or totally useless. Beyond that, the kind of problems you find easy will probably be down to your individual skills and preferences. Much like a music library, a tire kicker library is a personal thing.
Meanwhile, I've got some valuable new ideas to bring to my prototyping process: clearly, the times when I try to kick the tires on some new tools and do something interesting have been way more difficult and less useful than doing either individually. From now on it's one or the other for me.
I find it strange to hear people on the internet describing their ownership of an audience, or sometimes a platform. Those words aren't necessarily incorrect, but the way they're used comes off pretty creepy. "I've built a large audience that I'm figuring out how to monetise", or "I have a large platform that I use to bring attention to important issues". It gives the impression that the people who are interested in your work are best represented as inert goods, like a field full of turnips that you've carefully grown and are now ready to pull out of the ground and take to the market.
The current wave of big-growth internet businesses all have the same strategy: build and leverage some piece of infrastructure like a messaging system, a social network, or a two-sided marketplace. In phase 1, you concentrate exclusively on growth, just getting more people in the door at any cost. You keep that up until you've hit some crucial network effect point or until you start running out of investor money. Then comes phase 2: exploitation. You've done the hard work to build your platform, now you can direct that resource to whatever ends you need. Usually, but not always, that means restructuring your platform to be friendlier for advertising.
Many of these sites struggle in the second phase. Twitter, for example, has been struggling for many years to balance its identity as a piece of internet infrastructure with its goals as a user monetisation engine. They famously came down on third-party clients, and have recently been struggling to get users to accept a non-chronologicaltimeline. These changes, like those made at Facebook and Youtube, are designed to give the service more control over what you see so that they can more effectively turn your attention into money.
Reddit has had the opposite problem; to some extent, they seem pot-committed to being a government-like utility. While this is attractive from a user perspective, it puts them in a tricky situation as far as commercialisation. For example, they'd love to commecialise AMAs, except that some of them go hilariously wrong, largely because the behaviour of the users can't be controlled, and changing the scoring algorithm so that inconvenient questions are hidden would be against Reddit's ethos. Similar problems prevent corporate-run subreddits, Facebook-style ranking changes, and targeted advertising (though the recent link tracking changes indicate some movement on that front).
Perhaps most interesting is the progression of the image host Imgur, which started as a Reddit user's homegrown alternative to sites like Imageshack. The problem with sites like Imageshack is that they tried so hard to monetise their users that they became really user-hostile and everyone hated them. Imgur was the opposite; it just wanted to host images, and thus became enormously popular. Inevitably, the question came: how do we leverage this? How do we go from being a piece of dumb infrastructure to a platform where we can direct and monetise our audience? And now Imgur is aggressively trying to move people onto its mobile app and social network. You either die a hero, or live long enough to see yourself become Imageshack.
To some extent, the issue is that, well, what would you do differently? Would you rather be 2010 Imgur or 2016 Imgur? It's hard to argue that Reddit is doing better by any commercial metric than Facebook, Youtube or even Twitter. The build-and-leverage system exists because it works. And, bringing it back to an individual audience, it works there too. Kim Kardashian can build an audience of people who care about her Instagram pics and use them to sell gummy vitamins. Maybe I too can grow an audience of people who like one thing I'm doing and, when the time is ripe, use them to drive sales of my ebook, get attention for a new project, or direct them to some cause I find worthy (or get paid to find worthy).
I'm not saying it wouldn't work – it certainly seems to. I just feel that with time the ownership metaphor breaks down. People aren't turnips, even if they act like them in large groups. It's not your audience that you own, but a small slice of their attention that they've chosen to give to you in exchange for the things you do. Just as quickly, they could decide that what you're offering is no longer worth it and walk away. Their attention is a resource you have to continually earn, not something you can buy once and take for granted. A platform, yes, but one that's easy to burn if you try too hard to exploit it.
And as for the big growth internet platform leverage bonanza? I don't think it can last forever. Internet companies are offering value in exchange for what they extract from the platform, but these exchanges are happening at monopoly prices, not competitive ones. What happens when there are plenty of other infrastructure-for-attention trades available? Unlike a personal audience, the people on these platforms are locked in by various means, but they can and will walk, however slowly, for a better deal.
I suspect the eventual stabilisation point looks more like 2010 Imgur, 2016 Reddit, or 2018 Twitter than it does 2016 Facebook. Infrastructure has bad margins, and as much as today's internet giants would like to pretend they're doing something transcendental, that's all a platform is: leveraged infrastructure. But time and competition wears down that leverage, and I can't think of many infrastructure companies worth 350 billion dollars.
I'd like to return to optimisation order, the idea that when you're optimising for multiple things, the one that you start with gets the best result. In the Gale-Shapley algorithm, used among other things in matching medical students with hospitals, the students apply in rounds to their preferred hospitals who accept or reject their applications, retracting existing offers if necessary. Although this might seem to put hospitals in the stronger position, in fact this system is known as "applicant-optimal". If there are multiple solutions, students will get the best acceptable match, whereas hospitals will get the worst. The key observation is that a student's preferences are evaluated first (when they apply) and a hospital's preferences are evaluated last (when there's a conflict).
Thinking in terms of optimisation order makes you aware of distinctions that are otherwise invisible. For example, when talking about features to add to some software program. "Oh, can't you just add encryption to it? What's so hard about adding encryption?" On paper, perhaps, nothing. The problem isn't this individual decision, it's the question about where this decision is coming from and whether you make this decision before or after you make others. If your goal is to be a secure messenger first, and an easy-to-use messenger second, then sure. But what if your goal is to be an easy-to-use messenger first and a secure messenger second?
Sure, yes, both, of course we want both. But let's say that you have to make some minor compromises along the way to do that. Maybe adding encryption means you can't store messages on the server, so people who get new computers lose their message history. Maybe adding encryption means the app has to do some verification song and dance every when you add a new contact and it just confuses people. Hell, maybe it just makes the settings page marginally harder to understand. The point is there's always a cost, always some tradeoff where you have to start saying, yes, some of these encryption priorities need to come before your usability priorities. So now you're not easy-to-use first, and someone else who is will beat you if people care more about that.
The difficult thing here is that, while individual actions are visible, optimisation order isn't. Let's say you want the best job you can get. You go through every job ad, rank them carefully, apply in order, prepare well, interview well, and in the end you get your first preference. You win, right? Nope, you lose. The problem is that you interviewed for the first company that posted a job ad. In other words, the companies get the first pass of optimisation when they decide whether to hire and what to hire for. You then get the second pass when you decide which of those companies is your favourite. If you want the best possible outcome, you need to start with your list of dream jobs without considering whether they're hiring, and then work down the list trying to get hired at each one anyway.
This is also part of the reason why sales situations are so easy to get turned around in. Salespeople set the bounds of the decision, they say "we could give you option A with cost W and benefit X, or option B with cost Y and benefit Z", and you get to choose between the options that have already passed the good-for-them filter. If you're good at working that situation, you can get the best option they will offer you, but you'll never get the best option they wouldn't offer you. That would require you to become the salesperson and start trying to convince other people to accept your terms instead.
One last thought on this, which is that, okay, you may end up paying too much for something, or get your 5th most preferential job instead of your 3rd, but that might not be so bad. It's just a TV, or a job, or whatever. But no matter who you are, there's one area where you can't afford to forget about optimisation order: your own decisions about what you want. Are you running your life starting with your preferences, or starting with someone else's and figuring yours out afterwards? Are you trying to find a way to make the world work with how you want to be, or are you taking on the world's expectations and dealing in leftovers?
Who's on first? To my mind, there's only one acceptable answer.
Last week I posted the first half of my prototypes for Real life vs Internet, and this is the second half. Most of the fiddly part was last week, this week was mainly minor adjustments and some utility stuff as the project went on. The notable exception is the final web interface, which ended up pretty substantial bit of work. Although the individual parts are here and in the last prototype wrapup, you can also find the whole thing in its own repo.
I eventually realised that it was way more fun watching the display update in real time, but to do that I needed to move to websockets. This was a rewrite of the code for the ESP chip handling the network communication for the Arduino. I was initially devastated to realise that NodeMCU doesn't support websockets, until I remembered that I could just use, y'know, regular sockets. This was actually much neater than the previous code, and I felt like Lua kinda came into its own in this part. Handling reconnections and all the messages flying around from Arduino/C++ code probably would have been a bit more cumbersome.
This was the final version of the client and server code. A lot of different stuff went into this, including websocket support rather than polling, a pretty graphical cooldown on the button, and a sweet 7-segment font. The one thing that's kind of unfortunate is that because it evolved from polling to websocket based code, it ended up a bit spaghetti-ish. By the end I was dealing with 5 different kinds of connection: database, polling web clients, websocket clients, the raw socket for the display, and the polling for the display (still left as a backup). Would have been nice to find a way to neaten that all up, but that's weekend projects for you.
This last bit was pretty simple. At the end I wanted to pull the raw data for each hour out of redis before I took the server down, so I figured a CSV file would be a good option. I threw together a little Coffeescript thing to generate it, and committed the CSV data as well in case that's useful for some reason.