Two small photography-related scripts

I just added a couple of small photography-related scripts to Github – one for photo organisation, and one for generating preview galleries. Every photographer has their own workflow, so these are more useful as starting points for your own tools rather than something to use out of the box. They're easier to adapt with some backstory.

How I sort my photos

When I finish a photoshoot1, I copy the photos into a directory with a datestamp and a title. For instance:

  • 20190617 - Raptors victory parade
  • 20190728 - fire spinners
  • 201911 - Thailand

Next, I pick the photos I want to edit. I use macOS' filesystem tags to organise these into two categories – "definitely" (for my best shots) and "maybe" (for shots with potential). Tags are great:

  • They're part of the OS – so you can use them in searches, in file → open dialogs, and so on. One click in the Finder sidebar shows me my edit queue.
  • You're not locked into one app for organisation, making it easier to change apps or use multiple editors.
  • Your tags get backed up as part of your regular Time Machine backups.
  • Tags are attached to individual files, so if you copy photos to an external drive the tags are copied too.
  • You can interact with tags programatically, so you can build your own tooling on top of them.

The Finder's built-in tools are enough to sort through my photos from small photoshoots. Open a Finder window, switch it to thumbnail mode, maximise the zoom, and resize to get two photos per row. Then select a file, and press space to open Quick Look to check out the photo. If you like it, right-click → click a tag to add.

A screenshot of a Finder window, showing the maximum zoom for filtering images

This doesn't work so well for larger photoshoots. It isn't easy to winnow down your selections2, and you move between the keyboard and the mouse a lot.

I use Affinity Photo as an image editor, and I really like it. But I recently picked up a copy of Exposure X5, mostly for its many film presets/colour grades (including some passable imitations of my camera's built-in presets). Affinity Photo's similar to Photoshop, whereas Exposure's more like Lightroom – which means it's also capable of organising your images and faster winnowing. Load a directory of images, sort through them applying flags/ratings, then filter down to the flagged/rated images and upgrade/remove the ratings to suit. There's keyboard shortcuts for these, so it's potentially faster – though Exposure seems to do a poor job of pregenerating RAW previews, so I've found some frustrating delays.

If you did this in Lightroom, those ratings would be stored in Lightroom's central catalog. Exposure, however, stores all its metadata alongside your files. This makes it much easier to work with programatically! I wrote a small script, tagsfromexposure, which reads Exposure data and creates filesystem tags. Flagging an image as "pick" in Exposure translates to "definitely"; 2 stars or above translates to "maybe". This uses the tag command line utility for tagging, as it looked complicated in standard Ruby.

Generating preview galleries

Once I've picked my photos, I generate a preview gallery for online sharing. If it was a "proper" photoshoot, this lets my model pick their favourites for me to edit; if it was a day out or a trip abroad, I can share the good shots with family & friends. I dislike image editing and I'm a slow editor, so most of my photos would never be seen if I didn't publish these previews.

albumfromtags uses Sigal to generate these galleries based on macOS tags. Anything with a "maybe" or "definitely" tag gets included. This script also uses the tag utility to read filesystem tags. It copies the images to a new directory, uses Sigal to generate the gallery, then removes the temporary directory. You'll need to customise the file to add your name for the gallery's copyright notice, save it somewhere permanent, and edit the albumfromtags file to point to the config.


One thing I like about these scripts & this workflow is that it's very modular. I've taken several unrelated tools, and used a small amount of code to glue them together so they work for me. They're easy to extend and change. There's no custom data formats involved – just the operating system's features. These are all unixy approaches to building tools – an approach as useful in 2019 as it was in 1989.

  1. This can be an actual photoshoot, or a day out, or a trip abroad, or whatever. I also keep monthly "miscellaneous Toronto" directories for my handful of day-to-day shots around the city.  ↩

  2. For instance, if you have 3 good portraits of someone in a particular pose, there's a lot of clicks and filters needed to tag all 3 as "maybe" and then later upgrade one to "definitely".  ↩

Four Cool URLs

One of the things I've always found elegant about programming is how small, simple elements can be combined to accomplish complicated tasks. Sometimes this is a compound elegance (like stringing unrelated UNIX commands together to achieve your goal); sometimes it's an atomic elegance (like someone using something simple in a way you never imagined).

URLs have been around for more than 20 years. They're a method of "identifying a resource", which means "unambiguously pointing to something on the internet." That's how they're normally used: URLs point to websites, articles, images, songs, videos, downloads – everything. Most people don't give them much thought, in part because web browsers increasingly hide URLs away. But some URLs have special powers. Here are some that knock my socks off.

Goodhertz – Vulf Compressor preset

Goodhertz is a company that makes audio processing plugins1. Presets let people save their favourite sounds, share them with others, and keep them safe so they can revisit the settings later. They're also a great way to learn: if you want to learn how someone got a great sound, they can show you the settings.

This URL is cool for a couple of reasons: first up, it's a preset in itself. Pass it to the audio plugin, and the plugin uses the settings from the URL. But it's also a valid web page2 that shows the settings used, in a more human-readable format. This is great for learning, and if you find yourself in a niche situation3 it might make your life easier.

Reddit - /r/redditlaqueristas

There's nothing specific about the Laqueristas subreddit that made me select it, beyond it being a great example of an online interest-specific community. I could have picked any Reddit URL for this demo – and indeed, that's what makes this URL cool. Take any Reddit URL, append .json to it, and you'll get back a JSON version:

This is a really user-friendly way of exposing an API! It makes it easy for programmers to get started – no need to register an API key, set up oAuth, or anything like that. Those barriers are often counterproductive (as people will just scrape the web pages rather than jump through those hurdles). It's not just subreddits – if you want to retrieve user details, or the contents of a post, it works there as well. Removing a .json can be useful too: if you're trying to debug some code that interacts with Reddit, you can take the relevant link and turn it back into a form designed for human consumption rather than trying to read through the JSON.

The concept of content negotiation has been around for a while, where you can do this via the HTTP Accept header, but I'd really like to see more of it. Wouldn't it be neat to get a PDF version of your receipt by appending .pdf to the URL? Websites can use <link rel="alternate"> to expose these other versions to the world, though sadly most browsers don't communicate those alternate versions to users.

It seems the idea of an open music web is pretty much dead. It's all closed silos these days: there's no interoperability between Spotify, YouTube, Apple Music, Soundcloud, and so on. If you're sharing a track with a friend, what do you do? Generally, the most practical solution is to send a YouTube link. That tends to work OK, but isn't ideal4. You could send them a link to your preferred service, but they might use a different one. And you probably don't keep track of which friend uses which service.

Enter You hand it a link to something on a music service:

And it gives you links to that music on all the services it can:

Now your friend can click through to their music provider and play what you shared. That's a cool service, but look at the URL: it's pretty similar to the Spotify one.

This means you don't need to use to construct a URL – you can do it yourself, or write a script to handle it. If the service disappears or changes, you haven't lost a link to your music – the original ID is intact. is a site that provides "accessible UK train timetables". It's an unofficial site, but I vastly prefer it to National Rail Enquiries. It's got a simple visual design and works great without JavaScript – which means it's a tiny web page. Tiny web pages are quick to load, and work well even on poor 3G coverage. Just what you need if you're in the middle of nowhere trying to figure out a route home.

The URL is cool because it's designed to be human editable. Let's look at trains from London to Brighton:

Do you happen to know the official 3 letter code for London Bridge, and want to travel from that station specifically? You can drop that in:

Do you want to leave a couple of hours later? That's also an easy change:

Are you actually planning for tomorrow? Change the last part.

This is all documented on the front page of the site, but you don't need to know these details to use the service. There's a friendly form to start your search, and the results page lets you navigate to earlier or later trains (along with the return trip). But a user interface that tried to cater for every use case would be cluttered, and supporting URL editing doesn't cost anything. If you use the site often enough to be familiar with it the URLs, it's often quicker to edit the URL rather than perform a fresh search.

We can see some general principles at work in these URLs:

  • A URL points to a thing, but it can also be the thing itself. In the Goodhertz case, the preset was the URL (and the URL was the preset). A page of search results is a more prosaic example: the results displayed match the parameters for the search in the URL.
  • URLs can be for both human and machine consumption. The Goodhertz URL is primarily for consumption by software – the plugin that uses the preset. The Traintimes URL is designed to be easy for a human to manipulate. The Reddit page is a hybrid: easy for a human to transform into a computer-friendly format by adding .json to the end.
  • URLs can be robust. Even if the service fails or dies, you can easily return to the original links.
  • URLs can be predictable. You don't need API documentation to get started with Reddit's API. You don't need to involve to generate a valid link. If you learn the format of the Traintimes links, you could type one in by hand.
  • Let power users edit your URLs. Most Reddit or Traintimes users aren't going to retrieve JSON or explore times by editing the URL, but power users can do so. There's no extra cost to that apart from the initial design, and your users benefit from having the option.
  • Good URLs are descriptive. The Traintimes URL describes the results you'll see. Even if you've never visited the site, you can make a reasonable guess about what would show you.

These principles aren't applicable to every scenario. Sometimes a link is just a pointer to a particular document online. A news website would find it hard to let users edit their links in a meaningful way. And sometimes you want your URLs to be hard to predict (for instance, you don't want people to guess the links to your Google docs).

URLs are consumed by machines, but they should be designed for humans. If your URL thinking stops at "uniquely identifies a page" and "good for SEO", you're missing out.

  1. Software that changes sound. Want to add an echo effect? Make your vocals sound like a robot? Make your audio sound like it's underwater? These plugins are one way of accomplishing that.  ↩

  2. Some software constructs strings that look like URLs to store info, and the software can use the URL, but it doesn't lead to a working web page. Those always make me sad.  ↩

  3. "My laptop doesn't have an internet connection in the studio, but I only have this preset on my phone. Do I have to type it all out by hand? Oh cool, I can see the settings on my phone & use the interface to set them all."  ↩

  4. The quality can be iffy, it might get taken down due to copyright claims, it might not be available in their territory. They can't add it to a playlist, save it to their music library, or listen in the background when on a mobile device. Plus it's video rather than audio, so it's not mobile-data-friendly.  ↩

A review of Beyerdynamic's Byron headphones

I bought a pair of Beyerdynamic Byrons1 this year, and I hate them. But before I explain why, I'm afraid we must explore my history of headphones.

I've used and loved a pair of Sennheiser HD280s at home & work for over a decade. Two pairs, in fact; one pair broke after a respectable 8 years of daily usage. They're great: amazing sound, solid build quality, spare parts are available, people nearby can't hear your music, and they smother enough ambient noise to make open-plan offices more peaceful. But they are indoor headphones: they look ridiculous if worn outside, and they're too bulky for that anyway. So the Sennheisers live on my desk, and I use earbud headphones when out in the world.

The particular choice of earbuds has varied through the years. I used some Sennheisers for a while, and they were fine. When I got an iPhone the convenience of the remote/microphone made me switch to the included EarPods. And I liked them a lot! The sound was acceptable, the remote/mic was useful2, and the build quality was good. Well, the build quality was OK. Kind of OK. Not... not that OK.

The earpieces themselves are solid, and the remote's always worked fine. But I've owned several pairs of EarPods, and they all end up like this:

The EarPods, with heatshrink

Designers generally include strain relief when a cable meets a plug. This prevents the wire from being pulled into sharp angles and getting damaged. The EarPods have this too, but it's just a small piece of rubber. Over time, this gets broken and splits off. Then the headphones start to cut out in one ear, and the remote goes wild when you're not touching it. It's easy enough to add some heatshrink to compensate for this if you have a nearby Hackspace, and this fixes things for a while, but you're now on borrowed time. The cable will fail, and your headphones will crackle or be silent.

I've gone through at least 3 sets of EarPods since 2012, and when my latest pair started to fail I decided enough was enough. Surely I could spend a little more, and get something that doesn't consistently fail in the same way? I knew I didn't want Apple's AirPods3, so I looked around, read some reviews, and settled on Beyerdynamic's Byron headphones. Around $65 from Amazon, compared to $35 for the EarPods.

When looking around, here's what I thought my priorities were:

  1. They've got to sound good. They don't have to be amazing4, but it's got to be good. At least as good as the EarPods.
  2. They've got to have a remote.
  3. They've got to have decent build quality. If I'm spending a bit more money, they should last longer than the EarPods.
  4. They've got to be wired. I don't need another gadget to charge in my life, and can deal with trailing cables.
Beyerdynamic's Byron headphones

Let's start with their good points. They've got a slightly longer cable, which was more convenient than I expected. The build quality seems stronger, particularly around the plug. Just look at that strain relief! It's going to last way longer than the EarPods. They cut down a lot of outside noise – great for plane/coach trips – and they're more comfortable to sleep with. Finally, the sound quality can be great. There's been a couple of times when I've listened to music I knew well, and I heard something new – something I'd missed even on my big Sennheisers. That's strong praise.

But despite these upsides, I cannot recommend them. More than that: I feel that people must be warned. It turns out I had a greater priority, one I never realised. One I never thought to enumerate:

  1. They've got to stay in your ears.

And they don't! They just don't. Whenever I go for a run they fall out constantly, and I look like a newscaster who thinks he's lost contact with the studio. These headphones are pushed into the ear canal, yet still fall out constantly. The EarPods just sat on your ear-shelf, but never fell out. It's genuinely baffling, and it fills me with anger. The Byron's earpieces are made from grippy rubber, they fit tightly, and have more surface area in contact with the ear. Yet they're not as secure as the EarPods? How is that possible? It's not just running, either: it happens when walking around the city too. Not as much, but it's still a notable flaw.

But let's put that to one side, and return to the sound quality. I said before it can be great - but it can also be terrible. The Byrons will sound great when sitting at exactly the right point in your ears. Too deep, and they will be thuddy and dull and overbearing. And if they're loosening – which they will be – they're just empty, with nothing but high frequencies. But that's not all: if the cable moves at all you'll get an annoying constant sub-bass rumble. This is annoying on a treadmill, and a bit irritating at other times. I'm not convinced this is a problem specific to the Byrons – I suspect it's a problem with all in-canal headphones – but it's still a reason not to buy them.

Beyerdynamic themselves recommend that you could buy some alternative ear pieces for sports. I briefly considered this, before realising that it was completely bananas to spend another $20 in the hope that these headphones were not a lost cause. You get 3 sizes of ear pieces in the box and I have tried them all. None of them seem to make any difference.

The remote is another problem – though this one's my fault. The Byrons are designed for Android phones. This really means that a couple of the connections in the TRRS jack are transposed, so the remote/mic won't work with Apple devices. I should have read the description more closely before buying, and I could live with a non-functional remote. But the remote works! To different levels on every device!

  • On my iPhone, the remote acts completely normally. Yaaaaay.
  • On my iPad, the volume buttons don't work but the play/pause button does.
  • On my laptop, the volume buttons move the volume in the correct direction, but don't stop moving it. So I have a "Turn sound off" or "Destroy my ears" buttons.

As I say: my fault. But if it's going to work weirdly, could it at least be consistent? Or maybe include an adapter in the box, if Android/Apple compatibility is just a case of transposing two connections?

Finally, let's take a closer look at the headphones after a month or so of use.

Byron headphones, looking into the drivers

Look closely at the driver on the top. No mesh! Somehow, the mesh has fallen out somewhere. I don't know how I'll try to clean this when it gets gunky. I've also managed to lose two of the rubber cups somewhere – so now both ears fit poorly and differently. An asymmetrical annoyance.

I bought a pair of Beyerdynamic Byrons this year, and I hate them.

  1. This is an affiliate link, as other the other Amazon links in this article. If you follow it to the Byrons, I recommend you buy something else.  ↩

  2. Extremely useful after I switched to a MacBook Pro. I always had the EarPods on me, so I always had an acceptable Skype headset. Perfect for surprise calls wherever I was.  ↩

  3. Which discombobulates me, as they have astonishingly good reviews, but I don't think they'd work for me. I worry about them falling out when running/cycling, I would miss the remote, and I really don't want to own something that needs to be charged several times a day.  ↩

  4. Sound quality is pretty closely tied to the size of the driver, so in-ear headphones will almost always sound worse than over-ear headphones.  ↩

Exploring weird maths with code

Sometimes, while reading an innocuous-seeming article, I stumble across an aside that makes me sit bolt upright and mutter something incredulous. Asides like this one:

A counterintuitive property of coin-tossing: If Alice tosses a coin until she sees a head followed by a tail, and Bob tosses a coin until he sees two heads in a row, then on average, Alice will require four tosses while Bob will require six tosses (try this at home!), even though head-tail and head-head have an equal chance of appearing after two coin tosses.


This was a surprise! The four possible outcomes of two tosses are equally likely, so it seems weird that a heads-tails outcome would take longer to reach than a heads-heads. Weird enough to try it at home – at least by programming. Let's write some Ruby and see if we get the same result. (I recommend opening irb and exploring these examples for yourself if you want to fully understand them.)

Checking some assumptions

First of all, let's agree to toss a coin by picking a random symbol from an array1:

def coin_toss
  %i(heads tails).sample #=> :heads or :tails. 

And let's confirm that this is close enough to 50/50, by counting the result of tossing a coin 100,000 times:

results = {heads: 0, tails: 0}
1000000.times { results[coin_toss] += 1 }

puts "After 100000 tosses we saw #{results[:heads]} heads and #{results[:tails]} tails."

.sample chooses an element at random, so the result will be a little different each time. I ran this program 10 times, and got these results:

After 100000 tosses we saw 50131 heads and 49869 tails.
After 100000 tosses we saw 49845 heads and 50155 tails.
After 100000 tosses we saw 50094 heads and 49906 tails.
After 100000 tosses we saw 49672 heads and 50328 tails.
After 100000 tosses we saw 50062 heads and 49938 tails.
After 100000 tosses we saw 50046 heads and 49954 tails.
After 100000 tosses we saw 50003 heads and 49997 tails.
After 100000 tosses we saw 50094 heads and 49906 tails.
After 100000 tosses we saw 50124 heads and 49876 tails.
After 100000 tosses we saw 49838 heads and 50162 tails.

I think these results look OK, but the next thing I tried was busting out some statistics and checking the standard deviation. You can think of it as a measure of how closely-clustered our results are – we'd expect to get a low standard deviation if .sample is fair. Calculating the standard deviation is a little bit complicated, so I used the descriptive_statistics gem to make it easier. Let's calculate the standard deviation of the number of heads in each run:

require 'descriptive_statistics'
[50131, 49845, 50094, 49672, 50062, 50046, 50003, 50094, 50124, 49838].standard_deviation #=> 146.014

But is 146.014 low or not? I have no idea! This is where my statistics knowledge runs out. For now, let's presume our eyeballs are correct and our coin tosses are fair.

Back to the question

If we can toss a coin fairly, we can return to our original question: how many tosses, on average, does it take to reach a given combination?

We'll need a target combination, we need to toss at least twice, and we want to toss until we hit the target:

target = [:heads, :heads]
tosses = [coin_toss, coin_toss]

until tosses.last(2) == target
  tosses << coin_toss

I ran this in irb and I got [:tails, :tails, :heads, :tails, :heads, :heads]. It works! Let's turn this into a method so we can reuse it:

def tosses_until(target)
  tosses = [coin_toss, coin_toss]
  until tosses.last(2) == target
    tosses << coin_toss

Running the experiment repeatedly will make our result more reliable. If something weird happens once it could be a fluke, but you can't fluke something thousands of times. We could use the .times method again, and build up an array of results like we built the array of tosses:

experiments = []
100000.times { experiments << tosses_until([:heads, :heads]) }

Or we can make this shorter by using Ruby's .map method. .map applies a method to every element in a list. It's normally used to modify an existing list:

["cat", "dog", "avocado"].map { |t| t.upcase } #=> ["CAT", "DOG", "AVOCADO"]
(1..4).map { |n| n * 3 } #=> [3, 6, 9, 12]

But it doesn't matter if we throw the original elements away instead. You can try this in the console, but beware! It's going to print out all 100,000 results.

experiments = (0..100000).map { tosses_until([:heads, :heads]) }

It's not really relevant to our experiment, but I wondered what the shortest and longest sequence until our target was. You might expect that we can use experiments.min and experiments.max to find out:

experiments.min #=> [:heads, :heads]
experiments.min.length #=> 2
experiments.max #=> [:tails, :tails, :tails, :tails, :tails, :tails, :tails, :tails, :tails, :tails, :tails, :tails, :tails, :tails, :tails, :heads, :tails, :heads, :tails, :heads, :heads]
experiments.max.length #=> 21

But that's not quite right2 for the maximum case. It looks right, though – a handy reminder that verifying data by eye can lead you astray. Instead, we need to use .max_by to explicitly look at the length of the array:

experiments.max_by { |e| e.length }

This pattern – calling a method on the value passed into the block – is common, so Ruby provides a shorthand for this:

experiments.max_by(&:length) #=> [:heads, :tails, :tails, :tails, :heads, :tails, :tails, :tails, :heads, :tails, :tails, :tails, :tails, :tails, :heads, :tails, :tails, :heads, :tails, :heads, :tails, :tails, :tails, :heads, :tails, :tails, :heads, :tails, :tails, :heads, :tails, :tails, :heads, :tails, :tails, :heads, :tails, :heads, :tails, :tails, :heads, :tails, :tails, :heads, :tails, :tails, :heads, :tails, :tails, :heads, :tails, :tails, :tails, :tails, :tails, :tails, :tails, :tails, :tails, :heads, :heads]
experiments.max_by(&:length).length #=> 61

Let's put all this together in one place, and add some output about our results:

def coin_toss
  %i(heads tails).sample #=> :heads or :tails. 

def tosses_until(target)
  tosses = [coin_toss, coin_toss]
  until tosses.last(2) == target
    tosses << coin_toss

experiments = (0..100000).map { tosses_until([:heads, :heads]) }
average_toss_count = experiments.reduce(0) { |sum, n| sum + n.length } / experiments.length.to_f # We'll talk about this line below.

puts "Our shortest sequence was #{experiments.min_by(&:length)}"
puts "Our longest sequence was #{experiments.max_by(&:length)}"
puts "On average, we had to toss #{average_toss_count} times before (heads, heads) came up."

.reduce is a close cousin of .map. .map does something to every element in a list; .reduce takes two elements from a list and boils them down into one. It does that repeatedly to produce a final value:

[1, 2].reduce { |a, b| a + b } #=> 3
[1, 2, 3].reduce { |a, b| a + b } #=> 6: [1, 2, 3] → [3, 3] → 6.
[1, 2, 3, 4].reduce { |a, b| a + b } #=> 10: [1, 2, 3, 4] → [3, 3, 4] → [6, 4] → 10.

You can also give .reduce a starting value, which is what we did in our program:

[1, 2].reduce(10) { |sum, a| sum + a } #=> 13: 10 + 1 = 11 then 11 + 2 = 13.
[1, 2, 3].reduce(10) { |total, a| total + (a * 2) } #=> 22.

We started our toss count at 0, then added the length of each run to that total. Finally, we divided it by the total number of runs to get an average. The .to_f on the end converts the length to a floating point number, because we'd like to see the decimal places in the result.

9 / 2 #=> 4; really "4 remainder 1", but Ruby throws the remainder away
9 / 2.to_f #=> 4.5

Simplifying our code

This works, but is more complicated than it needs to be. Our goal was to find out how many tosses, on average, it takes to hit our target – we don't care about the sequence of tosses to get there. Let's change our tosses_until method to return the number of tosses instead of the sequence itself:

def tosses_until(target)
  tosses = [coin_toss, coin_toss]
  until tosses.last(2) == target
    tosses << coin_toss

This lets us make our trial run code simpler. We could build an array of the sequence counts, then add it up:

experiments = (0..100000).map { tosses_until([:heads, :heads]) }
average_toss_count = experiments.reduce(&:+) / experiments.length.to_f

We could skip the array entirely, and just maintain a total:

total_experiments = 100000
total_tosses = 0
total_experiments.times { total_tosses += tosses_until([:heads, :heads]) }
average_toss_count = total_tosses / total_experiments.to_f

Or we could use reduce again:

total_experiments = 100000
total_tosses = (0..total_experiments).reduce(0) { |sum, _| tosses_until([:heads, heads]) }
average_toss_count = total_tosses / total_experiments.to_f

The "best" version is a matter of taste, but personally I prefer the first version. It uses more memory, but that doesn't matter in experiments like these. It's the shortest code, we can find the longest run of tosses, and it's reasonably clear how it works once you get your head around .reduce.

Let's put the first version into a method that runs the experiment and reports the outcome for a given target:

def coin_toss
  %i(heads tails).sample

def tosses_until(target)
  tosses = [coin_toss, coin_toss]
  until tosses.last(2) == target
    tosses << coin_toss

def average_toss_count(target, num_experiments)
  experiments = (0..num_experiments).map { tosses_until(target) }
  average_toss_count = experiments.reduce(&:+) / experiments.length.to_f

  # sprintf formats the average so it prints to two decimal places only.
  puts "On average, we had to toss #{sprintf('%.2f', average_toss_count)} times before #{target.inspect} came up. Our longest run was #{experiments.max} tosses."

The other cases

Now we have all the building blocks to run the experiment for each of the four possible outcomes:

targets = [[:heads, :heads], [:heads, :tails], [:tails, :heads], [:tails, :tails]]
targets.each { |target| average_toss_count(target, 100000) }

Which produces:

On average, we had to toss 5.98 times before [:heads, :heads] came up. Our longest run was 52 tosses.
On average, we had to toss 4.00 times before [:heads, :tails] came up. Our longest run was 22 tosses.
On average, we had to toss 3.99 times before [:tails, :heads] came up. Our longest run was 20 tosses.
On average, we had to toss 6.00 times before [:tails, :tails] came up. Our longest run was 55 tosses.

Sure enough, it takes longer on average to hit [:heads, :heads] or [:tails, :tails] than [:heads, :tails] or [:tails, :heads], even though each outcome has an equal probability. It's still weird, but now I'm satisfied it's true.

Why does this happen?

Let's go back to Alice and Bob, who are targeting [:heads, :tails] and [:heads, :heads] respectively:

Player Target
Alice H T
Bob H H

Let's presume they both win their first toss – they both get a result they're looking for:

Player Target Result 1
Alice H T H
Bob H H H

Then, presume they lose their second toss:

Player Target Result 1 Result 2
Alice H T H H
Bob H H H T

There's now a major difference between the two players: Alice can hit her target on toss 3, but Bob can't until toss 4. Bob must start over after losing on toss 2; Alice's loss can be part of a win if she gets a tails on turn 3.


If you'd like to explore this some more, here's some suggestions for things to try:

  1. Change the program so it runs the experiment a million times instead of 100,000.
  2. If we toss three coins, there's eight possible outcomes. How long does it take, on average, to hit each combination? Are there some sequences that take longer than others?
  3. We left our proof of a fair coin toss at "Yeah, that looks OK." Can you do better? How would you satisfy yourself that it's producing fair results?

  1. %i() is Ruby shorthand that generates an array of symbols. %i(foo bar baz) means the same as [:foo, :bar, :baz].  ↩

  2. But why doesn't this work? When we call .min, Ruby uses the <=> comparison operator to find the smallest value in the list. experiments is an array of arrays; <=> for arrays calls <=> on each of the elements of the list in turn until it finds a difference. In this case, our list elements are symbols. Symbols get converted to strings before comparison, and "heads" < "tails" because "h" < "t". So the upshot of this is that experiments.max returns the result with the longest initial streak of tails.

    Yes, I had to look this up in the documentation.  ↩

The EU Referendum: A Retrospective

I have tried for days to write about the referendum, but I keep getting overwhelmed by the immensity of it. Will we actually leave, or prevaricate forever? Can we negotiate reasonable trade deals, or will the EU make an example out of us? Will companies still open offices in the UK now we're no longer a gateway to Europe? Will Scotland become independent? What happens next in Ireland? Will our most deprived regions keep their funding? Will workers' rights be protected? Any one of these would be Pandora's box; we have opened many at once.

All of these issues are important, and all of these are beyond my control. They're also beyond my foresight: I have no idea what happens next. The stock market is suffering and the pound is at a thirty year low. These falls came from the decision to leave, but the fluctuation comes from the uncertainty. Uncertainty is the UK's greatest national resource now. We can certainly export that to the world.

A collage of anti-EU, anti-migrant front pages from UK newspapers.
Nothing says tolerance, compassion, and decency like calling people "Ethnics". Collage via @gameoldgirl.

Everyone promptly found out that the "leave" campaign was a Potemkin village, but its shoddy foundations were laid over the previous decades. The tabloid press constantly pumped out anti-EU & anti-immigrant froth, and nobody found a way to combat it effectively. Politicians found they could use these fears to their advantage, so why try to dispel them? Besides, it would invite the wrath of the press.

Without this backdrop – a nation flooded by freeloaders, powerless to prevent pointless meddling from Brussels – the UK would never vote to leave. It would have sounded preposterous. It was our government alone that failed to invest in the NHS, to build houses and schools, to make sure our post-industrial regions weren't dependent on grants, and allowed employment to become more precarious. Nothing to do with Europe. But it's no surprise that the people on the losing end of rising inequality would vote against the status quo.

As a user, given that I have a time machine...

I've grappled with two questions since Thursday night: "What should I1 have done differently?" and "What should I1 do now?". I'd kept my own counsel in previous elections but I spoke up a little this time. Some of that was amongst friends, but I also made a small website that laid out the benefits of European co-operation. I tried to back up all my claims, but my goal was to change people's feelings – not their minds. I wanted undecided people to see this long list and think "Wow, I never realised that the EU had a part in all this". People in the UK think of the EU as faceless, ineffectual, meddling bureaucrats who force legislation upon us; I hoped to replace that with some affection.

The site was a small success. It reached a couple of thousand people, and sparked some discussion showing it reached folk who weren't voting "remain" already. But I can't shake the feeling that my aim was off. Older people are more likely to vote, and more likely to vote "leave" – but they're harder to reach through the internet, and I don't have a voice in traditional media. People outside of large cities were more likely to vote "leave" – but they're harder to reach as my social circle is very urban. What could I have done differently to reach those groups? What medium should I have used? Would a different message have resonated more?

This campaign seriously impressed me. It's so simple, but appeals directly to the viewer's sense of identity. Just three words and a picture of Churchill speak volumes about persevering through tough times and standing with our neighbours.

Or is this the wrong question? Instead of asking how to reach a different audience, perhaps it's better to convince my audience they need to vote. My gut says that's a harder problem – people have been trying to motivate the younger generation to participate in politics unsuccessfully for years. Transforming online activism into real-world action is Herculean. I don't know what I can do as an individual, but Facebook's "I voted" feature is the strongest encouragement I've seen online.

What do we do now?

I doubt we'll see a second referendum. We'd need to negotiate a new deal with the EU – one different enough to merit putting it to the vote again. But Europe wants us out and doesn't need to negotiate with a gun to its head. We already had many exceptions to EU rules but voted to leave anyway. So Europe has no motivation to offer us a deal, and no pro-EU politician will want to risk a second "leave" outcome. We might hope for a stalemate – the UK never invoking Article 50, the EU not finding a way to force us out – but I expect some combination of economic uncertainty & European resentment will result in Britain leaving the EU.

Journalists and politicians will try to identify the effects of leaving, but conclusive evidence will be scarce. You can't see the corporate headquarters that gets built in France instead, nor can you see the uncreated jobs from a lack of economic growth. Businesses don't fail for one reason alone. Infrastructure takes at least a decade to become obviously dated; too slow to recognise and attribute.

Individuals can't change the UK's situation, but we can make our communities better. I have four concrete suggestions:

  • Stand up for others when you see abuse and prejudice.
  • Talk with your friends and neighbours about your beliefs. Don't proselytise; just listen to what they say, and gently try to move their opinions a little. Be compassionate and polite. You're trying to show people that there's a huge range of perspectives in the world, and to dispel myths & fears.
  • Lobby your MP to focus on their constituency instead of party politics. MPs need to support job security and job creation. They need to protect worker's rights and the social safety net. Let them know you expect this of them.
  • Hold the people who got us into this mess to account. They convinced us to leave, but don't want the responsibility of figuring out the details or standing by their pledges. And don't forget the disgusting parts either.

I also have an idea for another project. Something that makes it easier for people to engage with the politics that affects them, not the Westminster soap opera. I don't know if it will see the light of day, but I'm trying to use my anxiety about the future to propel it forward. It might not help after all, but anything's better than just looking on in horror.

  1. "I" really means "we": "what should an individual citizen, acting in their country's best interest, have done differently?"  ↩

How to Expand an OSX FileVault partition on El Capitan

I switched to OSX as my primary operating system around a year ago, after a lifetime of running Linux on the desktop. Using Ubuntu on a Macbook Pro is surprisingly straightforward and didn't require any low-level finagling, but it did come with some annoyances; annoyances that led me to try OSX as my primary OS. I kept the dual boot, but over time I wanted more disk space available to OSX. I had to piece the process together every time; here's what worked for me.

Step 0: take a good backup.

Editing partitions carries a risk of losing all your data. Backup everything! In both operating systems! These steps worked for me, but might not work for you. And while we're talking precautions: choose a time when you don't have important deadlines, meetings, or other computer-centric tasks.

Step 1: make space on the drive.

You need free space for your OSX partition to expand into. The disk utility in OSX is limited and can't resize Linux partitions, but GParted can. I downloaded Ubuntu and made a bootable USB key. Plug it in, then reboot while holding down the option ⌥ key. You can then choose a device to boot from.

Once in Linux, my process was:

  1. Clear some space on the Linux partition beforehand, then shrink it in GParted.
  2. Move the now-smaller partition to the end of the drive.
  3. Move any other partitions (eg. OS X recovery partitions) towards the end of the drive, so there's unallocated space after the partition you want to expand. Something like this1:
A screenshot of gparted, showing some unallocated space.

GParted will let you queue up these changes and try to apply them all in one go, but that gave me some (apparently harmless) error messages. I'd recommend making the changes one at a time.

Step 2: reboot into OSX and turn off CoreStorage.

OSX uses a volume manager called CoreStorage that acts as an intermediary between the operating system and the hardware. It's a requirement for FileVault encryption, but we can't expand drives while it's enabled. First, let's see all the CoreStorage volumes using diskutil cs list on the terminal:

CoreStorage logical volume groups (2 found)
+-- Logical Volume Group UUID 9559695B-73C6-40ED-B6EB-F3DE8767058A
|   =========================================================
|   Name:         Macintosh HD
|   Status:       Online
|   Size:         249222377472 B (249.2 GB)
|   Free Space:   0 B (0 B)
|   |
|   +-< Physical Volume UUID A76BF102-C0CF-41C4-9D88-27F8BB9A180E
|   |   ----------------------------------------------------
|   |   Index:    0
|   |   Disk:     disk0s2
|   |   Status:   Online
|   |   Size:     249222377472 B (249.2 GB)
|   |
|   +-> Logical Volume Family UUID EDA455C5-3FD0-444E-B00C-F9F8F2EF88EC
|       ----------------------------------------------------------
|       Encryption Type:         AES-XTS
|       Encryption Status:       Unlocked
|       Conversion Status:       Complete
|       High Level Queries:      Fully Secure
|       |                        Passphrase Required
|       |                        Accepts New Users
|       |                        Has Visible Users
|       |                        Has Volume Key
|       |
|       +-> Logical Volume UUID 4F3C168A-F0BB-40B6-B3FF-CE94D38506AD
|           ---------------------------------------------------
|           Disk:                  disk1
|           Status:                Online
|           Size (Total):          248873222144 B (248.9 GB)
|           Revertible:            Yes (unlock and decryption required)
|           Revert Status:         Reboot required
|           LV Name:               Macintosh HD
|           Volume Name:           Macintosh HD
|           Content Hint:          Apple_HFS

The most nested entry is the logical volume with a UUID of 4F3C168A-F0BB-40B6-B3FF-CE94D38506AD. Copy that UUID and use it in diskutil cs revert <UUID>:

diskutil cs revert 4F3C168A-F0BB-40B6-B3FF-CE94D38506AD

Reverting back to a regular volume takes some time, but you can check on the progress by running diskutil cs list until it shows as complete.

Step 3: Reboot, then expand the partition.

Your drive is no longer encrypted and doesn't use CoreStorage any more, so you can use Apple's Disk Utility to expand it. The 'partition' section has a pie chart with handles you can drag. Something like this:

A screenshot of Apple's disk utility, showing the pie chart with handles.

Step 4: Reboot again, then convert your drive back to a CoreStorage partition.

Run diskutil list to see all the partitions in your Mac. The one you want to convert is probably called "Macintosh HD". Let's re-enable CoreStorage:

diskutil cs convert "Macintosh HD"

Step 5: Reboot, then re-enable FileVault.

You can re-enable this from System Preferences → Security & Privacy → FileVault. This will also prompt you to reboot, for the last time.

Getting out of trouble

Everything broke with this final reboot. On startup, the Apple logo & a progress bar appeared before being replaced with a "no entry" logo (🚫) around ⅔ of the way through. This is how a Mac says "I found something that looked like OSX, but didn't contain a valid system folder."

The short-term fix was to reboot while holding the option ⌥ key. There was only one option ("Macintosh HD") in the list, which booted fine. The permanent fix was to use "System Preferences" → "Startup disks" and ensure that "Macintosh HD" was selected.

  1. This screenshot isn't from my system, so don't worry about the lack of OSX partitions here. It's just to show the unallocated space after the first partition on the drive.  ↩

Using structs in Arduino projects

I had some trouble getting structs to work in my Arduino project. This is how I fixed my code.

My project's ultimate goal is to replace the innards of a fibre optic lamp with a custom lightshow, but it's also a chance to play around with low-level circuitry & coding1. So far, I've designed and prototyped a hardware LED controller that's driven by an Arduino. The Arduino pumps out binary to the controller; this determines which LEDs light up.

A close-up of the circuit board

Each of the 4 LEDs you see on the board is an RGB LED, meaning it's actually a package of individual red, green, and blue LEDs. My old code used numbers to choose which colour to display, so it had function signatures like these:

void turnOnLED(int colour, int led);
void bounceColour(int colour); // A 'chase' pattern across all 4 LEDs.
void fadeBetween(int startingColour, int endingColour, int duration); // Fade between two colours in `duration` milliseconds

That's fine for pure colours, but it doesn't allow for compound colours (mixes of red, green, and blue) because the controller can only turn LEDs on and off. If you want purple, for instance, you turn on the red LED for a few milliseconds, then turn it off & turn blue on for a few milliseconds. Repeat this over & over and persistence of vision does the rest.

I could have used an integer for each colour, but using a struct keeps all the information in a single variable. I also created some constants for common colours using my new struct:

struct Colour {
    byte red;
    byte green;
    byte blue;

const Colour C_RED = {1, 0, 0};
const Colour C_BLUE = {1, 0, 1};
const Colour C_PURPLE = {1, 0, 2}; // 2 parts blue to 1 part red.
const Colour C_COLOURS[] = {C_RED, C_BLUE, C_PURPLE};

Next, I updated my functions to take a Colour parameter instead of an int. I also changed my parameters so they were pointers to the Colour instead; I couldn't get my code to compile without this.

void turnOnLED(Colour* colour, int led);
void bounceColour(Colour* colour);
void fadeBetween(Colour* start, Colour* ending, int duration);

The -> operator is used to access the values from the Colour* arguments2:

void turnOnLED(Colour* colour, int led) {
    float total_time = 3500;
    float total_colours = colour->red + colour->green + colour->blue;
    float timeRed = total_time * (colour->red / total_colours);
    float timeGreen = total_time * (colour->green / total_colours);
    float timeBlue = total_time * (colour->blue / total_colours);

    // Rest of function that rapidly changes between red, green, and blue removed for brevity.

Now the functions take a pointer to a Colour, I can change the calls to them to pass the address of a colour (using the & operator):

void loop() {
    bounceColour(&C_COLOURS[random(0, 3)]);

The final piece of the puzzle is to work around some limitations in the Arduino IDE. The IDE preprocesses your code before passing it to the compiler. One of its transformations is to generate function prototypes for your code – but it doesn't get it right for functions that use custom types, so you have to define them youself. The docs recommend you add it to a header file, but if you've only got a few then you can add them directly to your sketch. I added my function signatures below the struct Colour definition.

To summarise:

  1. Declare your struct at the top of your file.
  2. Update or override your functions so they take a pointer to your new struct. Remember to use the & operator when calling the new functions!
  3. Add function prototypes immediately after your struct definition for functions that take your struct as a parameter (or return the struct).
  4. Use the -> operator to access the properties in your struct.

An aside on understanding pointers

Pointers are a straightforward concept – "a variable that holds the address of a value, rather than holding the value directly," – but are really challenging to fully understand and use.

One trick that helped me was pronouncing the * in variable declarations as 'a pointer to'3, and pronouncing & as "the address of". So turnOnLED(Colour* colour, int led) is read aloud as "a function turn-on-LED that takes a pointer to a Colour and an integer". Or, consider the call to bounceColour here:

void loop() {

I say this as "call bounce-colour, and pass it the address of C_RED."

Another thing that helped was grasping that * means different things in declarations & usage. In declarations, * means "this is a pointer":

int* avocado; // Define a variable 'avocado' containing a pointer to an integer

But when using a variables, * means dereference: follow this pointer and use the thing it's pointing at.

// Define a couple of numbers, and a pointer to an integer
int x = 3;
int y = 8;
int* num;
num = &x; // "num equals the address of x"; it's now a pointer to x.
*num = *num + 4; // "Follow num and use the value of what it's pointing to, add 5 to it, and then store it back in the slot pointed to by num."

After this code, x is equal to 7, and *num is equal to 7 – they're both ways to access the same section of memory. std::cout << num will print the memory address (eg. "0x1234ABCD"); std::cout << *num will print the value "7".

This difference is why I write declarations as int* foo (and not int *foo). Keeping the asterisk next to the type emphasises that it's part of the type, not part of the name. foo is a pointer to an integer; there's no variable named *foo in this program. I find it a useful reminder (as does Bjarne Stroustrup), but it can trip you up if you declare multiple variables on one line:

int* bell, book, candle; // Declares one pointer to an int, and 2 ints
int *blinky, inky, clyde; // Also declares one pointer to an int, and 2 ints

int* bell, *book, *candle; // 3 pointers, but looks messy
int *blinky, *inky, *clyde; // Also 3 pointers, with a consistent style

Personally, I think declaring multiple variables on one line is best avoided. It's hard to tell if it's clever code or a subtle bug with declarations like that. But like all code formatting choices, it's down to individual taste.

  1. There are much better ways to make your own lighting projects if you don't have an interest in the low-level details. Neopixels, for instance, are reasonably priced, easier to extend, and easier to code for.  ↩

  2. -> is used because the argument is a pointer. If we were passing a Colour directly, we'd use the . operator instead (eg.  ↩

  3. I preferred the word 'reference' instead of 'pointer' when I was learning C, but C++ has a references feature that makes that confusing now.  ↩

The Friend Multiplier

The second in an occasional series about product design heuristics. The first article was about mutual benefits.

Another heuristic that's useful in product design is something I call "The Friend Multiplier". There's two parts to it, and both are requirements:

  1. Your product must be useful on its own.
  2. Your product must be n times more useful if a user's friends use it too.

It has a powerful effect when your product is inordinately more useful (or more compelling, fun, deep, etc) when a user's friends also use it. Done right, it feels like an entirely new dimension opening up: a new world of possibilities, not a couple of new features1. It's important that the benefits are genuine (there's nothing magical about an artificial barrier of "Wait 4 hours, or invite 5 friends to continue"), and that the friends are a user's real friends2 – people who would have a coffee with your user, given the chance.

But don't overlook the first part: few products can get away with being useless for solo users. Users are more likely to stick around if it's easy for them to start using your product3, and everybody wants to try something before recommending it to their friends. And what if they're not your ideal user? What if they don't live in a big city, or lack early-adopter friends? What if they've got a 4 year old phone or a 10 year old laptop? Will your product still work well for them? Millions of people fall into this demographic and they're often neglected, especially by tech startups. They might not be your primary audience, but if you improve their experience then your product will be better for everybody.

But the multiplier is the magic part; the reason this heuristic indicates your product is on-track, and that word-of-mouth growth is plausible.

Some examples of friend multiplier magic

I really like FourSquare as an example of the friend multiplier, despite their recent disoriented product decisions. You use FourSquare to log where you've been4 and it recommends new places to you. That's mildly useful & fun on its own, but the day FourSquare really clicked for me is burnt into my brain. I was in a café that I'd never visited before and I checked in on FourSquare. FourSquare popped up a 2-year-old note from a friend, giving me the wifi password. Magical! My friend hadn't left it for me; it was a generic tip. But FourSquare knew we were friends, and knew that my friend had left a note here, so it showed me the note when I checked in. There's no wizardry in the technology powering it, but the effect was profound. I was now aware of hidden traces left by my friends; traces I could stumble upon, seek out, or leave for others.

LiveJournal is a simpler example. LiveJournal users post diary entries to the web, either publicly or privately. Which is useful in itself, if you want to keep a diary. But when your friends are on LiveJournal too, you get an intimate window into their lives. LJ's fine-grained privacy settings mean your friends have control over what they're sharing – but you'll know more about how they feel, what's on their mind, and what they're doing. I never had a LiveJournal, but a lot of my university friends did. I always had a sense that there was an in circle – a deeper, mutually supportive, more emotionally bonded group – that I was outside of. Again, there was no groundbreaking technology involved. But the social impact on individuals was huge.

It won't last forever

It's important to have a target number in mind for your ideal user's friends. The magical feeling normally ebbs away after a (normally low) threshold. Twitter's a rare example with a high threshold: it's better when you follow a few hundred people. Most products have more in common with a shared calendar; more than a handful of people using one results in a deluge of events, making it hard to find ones relevant to you & making the calendar feel like a mess. So pick a low number & design with that number in mind.

Some counterexamples

This heuristic isn't a law. Lots of products thrive without any kind of social features, and (as before) there's lots of products where it's best avoided. Sometimes that's immediately apparent: I don't want my friends involved in my banking or my to-do list. I think that it's best to keep them away from dating & relationships, too; Tinder's "friends you have in common" feature always felt like a warning about potential future awkwardness, rather than an endorsement.

There's also fuzzier cases. Features that seem like a good idea can fall apart like wet cake once the complications of real life get involved. Windows 10 has a feature where your friends can log on automatically to your wifi. Which sounds great – your computer already knows who your friends are, so why not let them onto your wifi without having to ask for the password? Ah, but the details: your computer doesn't know about your friends. It knows about your contacts. Contacts contain friends, family, colleagues, ex-lovers, stalkers, businesses, abusers, toxic people you've cut out of your life. A whole lot of people you'd want kept away from your wifi network.

The friend multiplier is a simple but effective guideline. It's not a requirement – if it doesn't make sense for your product, don't worry about it. But in an increasingly connected world, it makes sense to think about how that can make your products much more compelling.

  1. A "solo" vs "with friends" feature comparison would be missing the point, anyway: product design is about what your product lets users accomplish, not the bullet points you can list.  ↩

  2. "Real" isn't the same as "real life"; internet-only friends count for this too. It's about genuine, two-way friendships – not people followed because a user likes their jokes, or stuff they make.  ↩

  3. Or service – especially services. If it's easy for someone to start using your service it's more likely they'll find value in it, so it's less likely they'll cancel. Fewer cancellations mean a lower churn rate, and a lower churn rate makes it easier to build a viable business.  ↩

  4. Which is interesting in itself; "Wow, have I really not been here for a year and a half? Really?"  ↩

Why your multipart emails show up as plaintext

While building Electric Handshakes, a tool that connects job-hunters to employers via email, I spent days debugging an email problem. My code sent valid multipart emails – emails with both HTML and plaintext parts – but Gmail and showed the plaintext. Why?

I found my answer buried on page 35 of an RFC12. The order of the parts matters. Later parts override earlier ones. Your mail client will display the final part it understands. I’d placed my plaintext version after the HTML version. Swapping those sections fixed the problem.

Other common causes of errors include:

  • Sending an invalid multipart message. There’s a few online validators. Both of these declared my message to be invalid, but didn’t point out the ordering error above.
    • In particular, make sure you send a valid Content-Type header. The capitalisation matters, and the value should be multipart/alternative.
  • Sending invalid HTML. I expected a complete HTML document would be required - one with <html>, <head>, and <body> tags rather than just formatting. But there isn’t consensus on that. Microsoft Outlook sends full HTML documents, as does Yahoo Mail, Hotmail, and Apple’s But Gmail sends formatting only, with no document structure. I think such an email is technically invalid3, but every mail client renders it anyway. Gmail can send & store smaller emails by omitting the document structure.
    • Even without a fully-structured document, markup errors can still prevent your message from being displayed. Make sure all your tags are closed, you’ve closed them in the right order, and so on.

If you’re using Rails, you can take some email pain away by using the letter_opener gem to preview your mails in your browser, and the premailer-rails gem to inline your CSS. It can also generate your plaintext parts automatically, and generally does a decent job of it.

  1. “In general, user agents that compose multipart/alternative entities should place the body parts in increasing order of preference, that is, with the preferred format last. For fancy text, the sending user agent should put the plainest format first and the richest format last. Receiving user agents should pick and display the last format they are capable of displaying.”  ↩

  2. When you find yourself on page 35 of a standards document praying for answers, your day’s already blown.

  3. It’s invalid because the HTML part is declared as text/html, and you need the structural tags for your document to be valid HTML.  ↩

Don't call it magic

I recently wrapped up 6 months of teaching at General Assembly. I was teaching WDI - a 12 week course that turns beginner programmers into employable junior web developers. It taught me a lot about teaching. One subtle thing I picked up is that we shouldn't talk about magic when we talk to novices.

Novice programmers like to ask questions, but those questions can't always be answered immediately. Either the explanation would be too confusing, or take up too much time, or be a useless distraction. So when the beginner asks, "But how did it know to do that?" we reply "Magic." We say "Magic" because developer culture is tied up with D&D, fantasy novels, and the idea that programmers are mystical and inscrutable. But what we really mean is "Don't worry about it." Rails isn't actually magic. You know that. I know that. The novices know that too1.

But the beginners don't always hear "Don't worry about it right now." Sometimes, they hear "It's beyond everybody's understanding, and we're all OK with that." These brackets and semicolons are confusing to everyone. Just repeat it back phonetically and everything will be fine. You can fix your bugs via trial & error. Nobody understands it anyway.

And that's a dangerous untruth. Programming is entirely understandable: every line, every word, every symbol. Some of the most productive teaching sessions have come from not allowing a student to handwave past some code and insisting they explain how it works, atom by atom. What's this line do? It assigns something to a variable. What's in that variable - what's its type? OK, what did we set it to? That second half, after the plus - what's it doing? Good. Next line.

One half of an instructor's role is imparting knowledge & understanding. The knowledge will come in time if you immerse the students in it, but the understanding will be slower to arrive if students aren't looking out for it. There's a difference between "We don't have to look behind the curtain right now," and "The Wizard is real". the language teachers use should reflect that.

The other half of the job is instilling confidence & self-belief. Programming is hard, technical, confusing work. It gnaws at almost everyone. You have to reinforce that yes, the students are programmers. What they are doing is programming - no matter how much they have to Google for answers and ask for help.

Again, the language you use can reinforce this message. It's better to ask a student "What do you know about [subject]?" instead of "Do you know about [subject]?". The former question is an open-ended chance for the student to show off their knowledge. The latter is often heard as "I have set an arbitrary high standard for your knowledge about [subject]. Do you dare assert that you meet it?". Beginners will always say "No," even if they have encountered it before. You'll quickly find out they do know about [subject]2, but it's still a tiny drag-down instead of a tiny build-up. You can't give people a giant boulder of confidence in one go. You can only give them tiny grains of positive feedback, and trust that one day it will outweigh their self-doubt.

  1. The Ruby source code might not know that.  ↩

  2. "Okay, the thing about frobbing the whatsit is..." "Oh, I know about that, I used that last week."  ↩