The Algorithms

And suspicious abundance

How MergeSort works: split them up, then merge them back together.

When writing about technical topics (finance, biochemistry, statistics, etc.) for an audience that’s intelligent but may not be educated on the topic in question, most writers rely on some kind of familiar analogy about cakes or cocktail parties or moving trains or some such. So I’ll start there, and forgive me if it reads as a bit condescending:

Imagine you want to organize your bookshelf so that the books are in chronological order according to when they were written. There are a few ways of going about it: you could find pairs of books in the “wrong” order and swap their positions until all of them are in the right place. You could pull all the contents of your bookshelf onto the floor in a heap, then out of that heap, find the book with the earliest date and set it on the leftmost spot on the shelf, repeating until there are no more books on the floor. Or you could pull a book off the floor from the top of the pile, find the right spot for it, and then shift the others over to make room. Or you could drop them into a few floor piles based on your rough idea of what era they belong to, put them back on the shelf in blocks, and then go through them in order and correct any errors by swapping them. Which method is the fastest? That depends on how many books you have, and whether they started out being mostly in order or completely jumbled. Each system has advantages and disadvantages. Some of them require a staging area, some of them will be very tedious, and some of them will require shoving books around on the shelf. The system you choose, as long as you follow it consistently, is your algorithm. This isn’t even really a metaphor — a step-by-step system for accomplishing a task is just what an algorithm is. People have been coming up with them for centuries to organize books, do math problems, train athletes, and so on. Systems for winning at gambling (regardless if they produce the desired results) must be some of the oldest.

Twenty years ago, I rarely heard the word “algorithm” outside of explicitly technical contexts, but now it’s a word everyone knows. Colloquially, we use it now (with the definite article: “The Algorithm”) to describe the mysterious process by which computers decide which posts or consumer products or search results to show at the tops of our screens. When used this way, it’s often to suggest something mysterious and sinister, a malevolent and manipulative force to which we are in thrall.

One way in which the field of Computer Science is like the humanities: it has a canon, a set of algorithms that everyone is expected to know to be considered “educated,” regardless of whether they’re likely to ever make practical use of them.¹ In the hierarchy of tech workplaces, the more prestigious the company, the more it will aim to hire people with a thorough grounding in the canon. Computer Science students complain about having to learn these algorithms just like high schoolers complain about having to read the Odyssey. The students are correct that they’ll probably never have to actually know MergeSort on the job. The canon-defenders argue that a solid grounding in the classics better equips the mind to understand and approach contemporary work. They’re right, too.

But despite the way they’re talked about, these canonical algorithms aren’t like mathematical axioms or the properties of molecules — they don’t represent immutable laws of the universe; each one of them is a system that some guy came up with and published in a paper, sometimes naming it after himself. A few ways of sorting things, a few ways of finding things, a few ways of arranging or combining them for some purpose or other. And, frankly, most of the canonical algorithms aren’t very sophisticated — they can usually be described in plain language, the way I’ve described bookshelf-organizing above. As far as human achievements go, I’m much more impressed by the Odyssey. The closest that the algorithms canon gets to the mysteries of the universe is that class of problems for which no good algorithm exists, to our knowledge: NP-Hard problems, exemplified most famously by the Traveling Salesman Problem.

One of the strange parts about working for Google Maps was having people tell me, with complete confidence, their theories about how the routing algorithm worked. The things people asserted included: that advertisers pay to route people past their retail locations, that they pay to route people past their billboards, that Maps does “load balancing” to try to alter macro traffic patterns in cities, that it deliberately suggests strange and out-of-the-way routes to collect more diverse traffic data, and so on (none of these are true, or at least they weren’t when I worked there). I can’t entirely blame them — when there’s not a lot of information about what’s actually happening, people’s imagination fills in the blanks. The reality, though, is so much more boring. I worked on a couple of features where an outsider might imagine there was sophisticated machine learning or personalization going on behind the scenes, when actually the logic was almost childishly simple. The complex algorithms mostly had to do with things that users don’t notice much except when they’re not working: choosing where and when to label the name of a street, for example.

When people shake their fists at The Algorithm, usually the algorithms they’re talking about are those in the Collaborative Filtering family: the ones that power countless “people who liked this movie/song/t-shirt also liked…” engines, the ones that try to predict what we are likely to respond to and then serve it up to us. Their mechanisms are well-understood. These are the algorithms held responsible for radicalizing us, homogenizing our tastes, conditioning us to respond to click-bait and rage-bait, and creating echo chambers.

“Algorithm” discourse bothers me for a couple of reasons. People often make claims that “we don’t know how they work” (yes we do) and follow this by ascribing all kinds of anthropomorphized qualities to “the algorithm:” reasoning, psychological insight, malevolent intent, or sometimes near-omniscience. For people of a conspiratorial bent who fear shadowy sources of influence, it makes a convenient villain: everything can be ascribed to it, and it’s easy to claim to find secret codes and hidden manipulations. Tech companies themselves often encourage this by making outsized claims about what their code is doing, making it sound much more sophisticated than it actually is. A lot of the “smart digital assistant” products of the last decade (Siri and her sisters) are actually pretty dumb under the hood, which became obvious after the first hype wave wore off. I promised I wouldn’t write about Chat GPT again, but you can read my thoughts about it here. I wish that people would be way less credulous in general when it comes to claims about what “the algorithm” is doing.

Not that I don’t think the influence of Collaborative Filtering algorithms has had negative effects on society. But it’s important to remember that Collaborative Filtering is a system, with mechanisms easy to describe, that some guy(s) came up with in the aughts for inferring someone’s taste based on the taste of their peers. The complexity of the algorithm pales in comparison to the mechanisms of human psychology: that so many of us have been willing to outsource our taste and education to an algorithm; that we did it because it was convenient and provided a temporary sense of abundance; that we devalued what appeared to be abundant; that reading someone’s opinion about a book or movie was faster and easier than reading the book or seeing the movie; that using our phones to alleviate our boredom or discomfort was easier and more convenient than every other way of doing so; that our capacity for addiction can exercise itself on anything that alleviates boredom or discomfort; that we instinctively trust people who speak authoritatively and seem similar to us; that such people became abundant and their abundance was rousing.

Algorithms have transformed us, but I think they’re often a red herring. The main thing they do is speed things up, not alter the contents of our souls. But all this speed does have its own effects. I am learning to distrust abundance, even to be repelled by it.

1. These are the algorithms I consider “canonical:” Fibonacci (ancient), Bubble Sort (dark ages), Quick Sort, Merge Sort, Binary Search, Depth-First Search, Breadth-First Search, Dijkstra (neoclassical), A*, Red-Black Tree (romantic), MapReduce (modernist).