It looks like I may be getting a traffic citation in the mail within the next few days. A few nights ago, while making a left into my neighborhood, I was slowed by a car that made a creeping right ahead of me onto the same street. As I finally completed my turn, I saw a bright flash go off behind me. While I’ve noted the proliferation of traffic light cameras around town with mildly disconcerted interest, I hadn’t yet noticed the camera on this rather inconsequential intersection. A day or two later at the same spot, I found myself coming to an abrupt stop once the light hit yellow to ensure that I wasn’t caught completing my turn as the light turned red. Automated surveillance had done its job; I had internalized the gaze of the unblinking eye.

For some time now I’ve been unsettled by the proliferation of traffic light cameras, but I’ve not yet been able to articulate why exactly. While Big Brother fears may be part of the concern, I’m mostly troubled by how the introduction of automated surveillance and ticketing seems to encourage the replacement of human judgment, erroneous as it may often be, by unthinking, habituated behavior.

The traffic light camera knows only that you have crossed or not crossed a certain point at a certain time. Its logic is binary: you are either out of the intersection or in it. Context matters not at all; there is no room for deliberation. If we can imagine a limited set of valid reasons for proceeding through a red light, automated ticketing cannot entertain them. While the intermittently monitored yellow light invited judgment and practical wisdom; the unceasingly monitored yellow light tolerates only unwavering compliance.

In this way, it hints at a certain pattern in the relationship between human beings and the complex technological systems we create. Take the work of getting from here to there as an example. Our baseline is walking. We can walk from here to there in just about any way that the terrain will allow and at whatever rate our needs dictate. And while the journey may have its perils, they are not inherent in walking itself. After all, it would be a strange accident indeed if I were incapacitated as a result of bumping into someone or even stumbling on a stone.

But walking is not necessarily the fastest or most efficient way of getting from here to there, especially if I have a sizable load to bear. Horse-drawn conveyances relieve me of the work of walking and potentially increase my rate of speed without, it seems to me, radically increasing the risks. But they also tend to limit my freedom of motion, a decently kept road being rather more of a necessity than it would’ve been for the walker. Desire paths illustrate this point neatly. The walker may make his own path, and frequently does so.

Then the train comes along. The train radically increased the rate of speed at which human beings may travel, but it also elevated risks–a derailment, after all, is not quite the same thing as a stumble–and restricted freedom of motion to the tracks laid out for it. It’s worth noting that the railway system was one of the first expansive technological systems requiring, for its efficient and safe operation, rigidly regimented and coordinated action. We even owe the time zones to the systematizing demands of the railroads. The railway system, then, was a massive feat of system building, and would-be travelers were integrated into this system.

The automobile, too, is powerful and potentially dangerous, and must also be carefully managed. Consequently, we created an elaborate system of roads and rules to govern how we use this powerful machine; we created a mechanistic environment to manage the machine. Interestingly, the car allows for a bit more freedom of action than the train, illustrated nicely by the off-roading ideal, which is a fantasy of liberation. But, for the most part, our driving, in order to be safe and efficient, is rationalized and systematized. Apart from this over-arching systematization, the off-roading fantasy would have little appeal. All of this is, of course, a “good thing.” Safety is important, etc., etc. The clip below, filmed at an intersection in Addis Ababa, illustrates what driving looks like in the absence of such rationalization and regimentation.

In an ideal world, one in which all rules and practical guidelines are punctiliously obeyed, the traffic flows freely and safely. Of course, this is far from an ideal world; accidents happen, and they are a leading source of inefficiency, expense, and harm. When driving is conceived of as an engineering problem solved by the fabrication of elaborate systems, accidents are human glitches in the machinery of automobile transportation. So, lanes, signals, signs, traffic lights, etc.–all of it is designed to discipline our driving so that it may resemble the smooth operation of machines that follow rules flawlessly. The more machine-like our driving, the more efficient the system.

As an illustration of the basic principle, take UPS’s deployment of Orion, a complex algorithm designed to plot out the best delivery route for drivers. “Driver reaction to Orion is mixed,” according to a WSJ piece on the software,

“The experience can be frustrating for some who might not want to give up a degree of autonomy, or who might not follow Orion’s logic. For example, some drivers don’t understand why it makes sense to deliver a package in one neighborhood in the morning, and come back to the same area later in the day for another delivery. But Orion often can see a payoff, measured in small amounts of time and money that the average person might not see.”

Commenting on this story at Marginal Revolution, Alex Taborrok added, “Human drivers think Orion is illogical because they can’t grok Orion’s super-logic. Perhaps any sufficiently advanced logic is indistinguishable from stupidity.” However we might frame the matter, it remains the case that, given the logic of the system, the driver’s judgment is the glitch that needs to be eradicated to achieve the best results.

Let’s consider the traffic light camera from this angle. Setting aside the not insignificant function of raising municipal funds through increased ticketing and fines, traffic light cameras are designed to mitigate pesky and erratic human judgment. Always-on surveillance ensures that my actions are ever more strictly synchronized with the vast technological system that orders automobile traffic. The traffic light camera assures that I am ever more fully assimilated into the logic of the system, to put it a bit too grimly perhaps. The upside, of course, is the promise of ever greater efficiency and safety–the only values technological systems can recognize, of course.

Ultimately, however, we don’t make very good machines. We get drowsy and angry and drunk; we are easily distracted and, even at our best, we can only take in a limited slice of the environment around us. Enter self-driving cars and the promise of eliminating human error from the complex automotive transportation system.

The trajectory that leads to self-driving cars was already envisioned years before the modern highway system was built. In the film that accompanied GM’s Futurama exhibit at the 1939 New York World’s Fair, we see a model highway system of the future (1960), and we are told, beginning around the 14:30 mark, “Traffic moves at an unreduced rates of speed. Safe distance between cars is maintained by automatic radio control …. The keynote of this motorway? Safety. Safety with increased speed.”

Most pitches I’ve heard for self-driving cars trade on the same idea: increased safety through automation, i.e., the elimination of human error. That, and increased screen time, because if you’re not having to pay attention to the road, then you’re free to dive into your device of choice. Or look at the scenery, if you’re quaint that way, but let’s be serious. Take, for example, the self-driving car Mercedes-Benz displayed at this year’s CES, the F015: “For interaction within the vehicle, the passengers rely on six display screens located around the cabin. They also interact with the vehicle through gestures, eye-tracking or by touching the high-resolution screens.”

But setting the ubiquity of screens aside, we can extract a general principle from the trajectory I’ve just sketched out.

In a system that works best the more machine-like we become, the human component becomes expendable as soon as a machine can outperform it. Or to put it another way, any system that encourages machine-like behavior from its human components, is a system poised to eventually eliminate the human element altogether. To give it another turn, we might frame it as a paradox of complexity. As human beings create powerful and complex technologies, they must design complex systemic environments to ensure their safe operation. These environments sustain further complexity by disciplining human actors to abide by the necessary parameters. Complexity is achieved by reducing human action to the patterns of the system; consequently, there comes a point when further complexity can only be achieved by discarding the human element altogether. When we design systems that work best the more machine-like we become, we shouldn’t be surprised when the machines ultimately render us superfluous.

Of course, it should be noted that, as per usual, the hype surrounding self-driving cars is just that. Writing for Fortune, Nicholas Carr, cited Ford’s chief engineer, Raj Nair, who, following his boss’s promise of automated cars rolling off the assembly line in five years time, “explained that ‘full automation’ would be possible only in limited circumstances, particularly ‘where high definition mapping is available along with favorable environmental conditions for the vehicle’s sensors.’” Carr added,

“While it may be relatively straightforward to design a car that can drive itself down a limited-access highway in good weather, programming it to navigate chaotic city or suburban streets or to make its way through a snowstorm or a downpour poses much harder challenges. Many engineers and automation experts believe it will take decades of further development to build a completely autonomous car, and some warn that it may never happen, at least not without a massive and very expensive overhaul of our road system.”

That said, the dream of full automation will probably direct research and development for years to come, and we will continue to see incremental steps in that direction. In retrospect, one of those steps will have been the advent of traffic light cameras, not because it advanced the technology of self-driving cars, but because it prepared us to assent to the assumption that we would be ultimately expendable. The point, then, of this rambling post might be put this way: Our attitude toward new technologies may be less a matter of conscious thought than of tacit assumptions internalized through practices and habits.