Prison Pod Cities and Killer Robots

We already live under the cyborg devouring mother

Mary Harrington

Image credit: Vanessa GF

According to a report out today from intergovernmental watchdog International IDEA, half the world’s democracies showed signs of decline over the last five years. Well, no sh** Sherlock - most of those democracies spent a significant portion of that half-decade forbidden to associate freely, in some cases compelled to undergo medical interventions, and subject to fines or even imprisonment if we breached strict rules about being out of the house for anything except a narrowly defined set of reasons. It’s not clear whether International IDEA will have anything to say about Covid measures, of course, but faced with the propagation of authoritarian pandemic policies throughout most advanced democracies, any watchdog measuring the state of ‘civil liberties’ would surely struggle to find anything positive to say.

Reasonable people remain divided, of course, over whether lockdown measures were justified in the name of disease control. But this is precisely the point: whatever print-era democracy enjoyers may think, for people habituated to the emergent-systems mindset native to the digital era individual agency is just less important than the overall health of the system. And this comprises most people under 40 in the developed world. For this demographic has, by and large, grown up with online communities as a key normative template for legitimate use of power. And online communities are rarely ‘democratic’ all the way up, in the print-era sense. Rather, while such platforms make a big song and dance about how they enable everyone to ‘have their say’, at scale rules and sanctions are doled out without accountability. And action that threatens the system overall legitimates near-limitless top-down control.

A generation raised to view the autocratic power of online moderators as normal will not find it particularly strange if real-world authorities also begin functioning in much the same way. In this context, it should not surprise anyone that young people are increasingly authoritarian. But perhaps the most salient feature of online communities as a governing metaphor for legitimate use of power is the backstage fusion it enables between human and machine authority.

Most social media platforms don’t have named moderators, but have the worst content violations managed by a rotating cast of (routinely traumatised, usually precaritised and increasingly offshored) humans. These are not wielders of power so much as fleshbots tasked with enacting the real source authority, which is a set of procedures underwritten by technologies of control. The work of these fleshbots is, across numerous platforms, seamlessly supplemented by algorithms that identify problematic content, amplify or throttle others, supply ‘context’ to posts that trigger algorithmic concerns about ‘misinformation’, and otherwise curate the overall shape of the memetic swarm. This algorithmic level of governance is an increasingly key field of political struggle, as illustrated by recent denunciations of Elon Musk’s decision to roll back Twitter’s Covid censorship regime.

So the mode of governance normalised by digital platforms is a combination of human implementation, supplemented by AI, in the service of impersonal “community guidelines” (which is to say unaccountably-enforced proceduralism). Let’s call this post-democratic order ‘swarm governance’. And the characteristic forms of this swarm governance now percolate well beyond the digital world, to structure the offline one too.

The positive case for this fusion of human and machine rule was set out, shortly after the pandemic, by the high priest of swarm governance, Benjamin Bratton. In his paean to post-Covid ‘positive biopolitics’, The Revenge of the Real, Bratton denounces in disgusted tones those eccentrics and reactionaries still wedded to obsolete ideas of “the destructive (and self-destructive) nature of libertarian individualism as the basis of sovereignty”.

The way forward, in his view, is in a merging of human and machine intelligences into collectivised hybrid super-intelligence at a planetary level: “A positive biopolitics,” Bratton declares, “is to be found in the agency of the non-subjective, in abstraction, in externalization, and in different relay points within these”. And this, by definition, necessitates sacrificing some measure of individualism: for the system to work, it’s not enough to engage with humans on their own chosen terms, or as he puts it “at an origin point of anthropomorphic sovereignty.”

Bratton conceives of this imagined order almost ecstatically, as a kind of algorithmic analogue to maternal attunement, able to care for the needs of any point in its aggregate body via ‘sensing layers’ with both human and machine components. History tells us, though, that utopian practice often falls short of utopian ideals. And Bratton has little to say on what this all-enveloping cyborg swarm-mama does when some component of its collective misbehaves. Based on performance to date, then, what does ‘positive biopolitics’ look like in practice - and how does it discipline rogue elements of the collective?

We can garner a few clues from the live-action cautionary tale now playing out in China , where Xi Jinping’s Zero Covid regime is now into its third year. There, Chinese citizens burn to death inside buildings they can’t escape because of lockdown confinement, and must engage with the ‘sensing layer’ - which is to say, take a PCR test at their own expense - every 48 hours in order to use public transport, go shopping, or go to work. Meanwhile, vast pod-cities are under construction for ‘quarantine’ - internment nominally used only for hygiene but rapidly extending to discipline. For with access to public services and even apartment doors now digitally controlled via smartphone, citizens discover that failure to cooperate with the all-enveloping ‘sensing layer’, or who otherwise commit infractions against the regime, may result in their Covid code being remotely set to Red - which means two weeks in Covid quarantine, at their own expense. In a sense this isn’t dissimilar to receiving a temporary social media ban for violating Terms and Conditions. Except it’s your real-world life, and your access to food, shelter and basic public services.

But surely this is something that could only happen Over There, in places where Our Democracy doesn’t prevail. Well, let’s not forget that as Thomas Fazi noted yesterday, the G20 recently announced a resolution to strengthen global digital health infrastructure including surveillance capacity at borders, global standards on proof of vaccination, and a commitment to “capitalise and build on the success of the existing standards and digital Covid-19 certificates.”

Meanwhile, large-scale efforts are under way to transform (and digitise) children’s education via virtual platforms, AI, ‘gamification’ and the monitoring of ‘social-emotional learning’ metrics, while critics worry that this will amount, over time, to the introduction of social credit scores in the West by the back door. Elsewhere, too, a number of countries are test-running Central Bank Digital Currencies. These are entirely digital forms of currency, whose use would render citizens’ every transaction visible to the issuing state.

And a bit like Twitter mods being able to read your DMs, this isn’t really a problem until you find yourself politically at loggerheads with those mods for some as yet unforeseen reason. Let’s not forget, after all, that when the trucker protests against Canadian Covid authoritarianism threatened to get embarrassing, Justin Trudeau (who is now praising Chinese protests against Zero Covid) ended the standoff by cutting off protesters’ bank accounts.

Put all these pieces together, and it’s clear that once applied to governance, even in the West, the digital transition and resulting shift toward swarm governance does an end-run round many of the core principles of government by consent that structured liberal democracy. The best-case argument for doing so is the belief that it will deliver a ‘positive biopolitics’, as Bratton calls it, which promises a kind of impersonal, omniscient nurture: an all-enveloping, touchless, hygienic cyborg mother. To do so, the post-democratic digital order merges technology and biomedical surveillance with faceless, procedural authoritarianism: governance via “Community Guidelines”, enforced “for your own good”.

But few enthusiasts for this cyborg mother want to think about what happens when she makes mistakes, or gets angry. In this light, we can all point and shudder at the drones Over There spraying chemicals to ‘disinfect’ the air over Chinese cities, or barking mechanised orders to stay indoors. But in truth Bratton’s ‘positive biopolitics’, underwritten and extended by robot intelligence, is increasingly backed up in the West as well as in China by the ultimate in impersonal coercion: lethal machines. No one seriously believes Boston Dynamics’ promise that they’ll never put a gun on Big Dog. And only yesterday, San Francisco granted permission for its police force to deploy killer robots in certain circumstances. (I doubt this is what ‘defund the police’ advocates had in mind as a replacement, but here we are.)

The difference between the relatively soft-pedalled and opt-in nature of cyborg technocracy in the West, compared to its full-bore top-down rollout in China, is mostly down to the fact that Chinese culture doesn’t emphasise the individual as is the case in the West. Un-softened by the Western cultural imperative to reassure those who still believe in individual freedom, the Chinese order looks more nakedly, coercively dystopian than the one now emerging in the West. But it’s the same picture.

And it’s not just coming. It’s already here. You can expect the majority of Western young people to welcome it. For we’ve been forming young people into the ideal citizens for a distributed, unaccountable, AI-supported procedural technocracy since the early 2000s, by acclimatising them to the governance norms of social media. That demographic is now well into adulthood, and have by and large accepted the transition to digital post-democracy without a murmur.

If I have cause for optimism, it rests on a very qualified foundation: the manifest ineptitude of states where large-scale tech projects are concerned. While poor implementation is itself a likely source of countless as-yet-unimaginable cruelties of algorithmic bureaucracy, it does at least hold out the bleak hope that cracks will remain in the cyborg mother’s omniscience. That is: as long as governments continue to be useless at IT, there remains some hope that pockets of eccentricity will be able to persist, where our characteristically human friction, untidiness, and resistance to standardisation continue to flourish despite every technocratic effort to nurture them out of existence.

Source: Reactionary Feminist