On Autonomous Software

Rule of Law

Algorithms don’t care about race, creed, origin, gender, height, looks, wealth, fame, or any of the other factors that bias human judgement… You cannot coerce, threaten, or bargain with an algorithm. Perhaps even more importantly, they never get tired, hungry, or frustrated, and are no more or less likely to be lenient early in the day or just after a lunch break. In this respect… it’s tempting to think of an algorithm as the perfect judge (and, perhaps, jury and executioner as well).

Autonomous algorithms tend to make things like transparency, accountability, and auditability hard. We can cause an algorithm to log every decision made, every action taken, etc., but in many cases it may be difficult or impossible for an algorithm to “explain” the rationale behind such a decision: the answer may have high complexity and therefore not be reducible to anything less than all of the code of the algorithm itself, and all its input data (which, for various reasons, might not be made public). As a result, in many cases we may not actually know why an algorithm made the decision that it did. […]

I don’t know about you, but I tend not to trust a system that isn’t transparent, accountable, and auditable. After all, what other guarantees do I have that a system is doing what its creators say it does, what it’s intended to do?

Algorithms necessarily reduce the wonderful, complex, multi-dimensionality of the world around us down to cold, hard, inflexible ones and zeros. In so doing, much that makes the analog world a wonderful place is lost. […] If you give me the choice, I would rather face a human jury and be sentenced by a human judge — any human judge — than by the fairest, most efficient, most robust digital court imaginable. […]

There will doubtless be many future examples of autonomous systems that no one but the system’s creators fully understand. This has the frightening potential to create a tiny, privileged elite who understand and can possibly even exploit such systems to their own advantage. For this reason it’s absolutely essential that autonomous systems be made as simple as possible, that they be based on common design patterns as much as possible, and that their full specifications, codebases, and documentation be made available for public scrutiny.

Social Structure

All software has bugs. […] Even a hypothetical, bug-free piece of software still has to be upgraded for reasons of maintenance, optimization, and ongoing development. What’s more, this upgrade process must by definition be initiated from outside. For this reason, every software-based system must have some mechanism for fixing or upgrading the system — effectively, an emergency “stop” button or an escape hatch that allows the system to be shut down, at least temporarily, in case of malfunction or need of upgrade. […]

But if this is the case, then by definition there is some person or set of people who have “god-mode” control over the system — who, acting from outside the system, can exercise unilateral authority over the system. Can such a system ever truly be called autonomous? (To put the question another way: if some alien had the power to press a button and “pause” or “reboot” or unilaterally alter the governance of the United States, is the United States still a sovereign nation?)




Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store