Why the security monopoly model doesn't work for AI

Concentrating security in a few firms won't steer us out of the systemic challenges posed by AI agents

Last week I was invited to speak at the AI Impact Summit in New Delhi about open source AI, and wanted to share my notes in preparation for the convo.

I’m paraphrasing a bit, but the main provocation was: what would need to change in order to make the vision of a pluralist, democratic and competitive open source ecosystem for AI a reality?¹

Implicit in the question is the central thrust of the paper I cowrote in Nature last year with Meredith Whittaker and David Gray Widder, which cautioned against overindexing on ‘openness’ as the shortcut to any of those things. ‘Open’ AI, we argued, isn’t very open at all in practice - we know vanishingly little about the data used to train models, compute remains consolidated in the hands of tech giants, even the research space is increasingly closed. As such, opening model weights allows for little more than tinkering at the edges. It’s not nothing, by any means, but a model where US tech companies own all the infrastructure and OS developers subsidize product development through their labor isn’t going to meaningfully shift power.

We need to be fighting for a lot more, I argued - for investors to place real bets with stakes attached in basic research outside of AI labs, especially research focused on public centered problems. For regulators to get serious about enforcing the laws that we have on the books, from antitrust to data privacy to financial regulations that avert the kinds of circular and opaque deals that deepen a few companies’ chokehold on the market. And to take bolder steps like structural separation in cloud, supervisory monitoring for systemic risks and public utilities regulations that shift us off the current market structure. We need an approach that steers us out of the race to the bottom, and incentivizes toward more lasting models of innovation.

There was one topic that came up that deserves treatment at greater length: that we need to draw lessons from the prior era of open source software and bring them into AI governance convos.

I won’t take us back to the early days of F/OSS though there’s a lot to learn there as well, but to slightly more recent history around open source and the open internet.

The era of commercial surveillance created a paradigm of profound insecurity for most people; with the promise of connection offered by Web 2.0 firms came the degradation of users’ privacy, deep capabilities for surveillance targeting and the capacity to manipulate. This was despite the existence of open source privacy tools that ostensibly could have given us the private, secure internet we needed.

Why didn’t these tools take off in the way that Signal has today? For one, the dominant business model emerging at the time - reinforced by policy decisions made at the federal level - was data capitalism. Firms were incentivized to secure the parts of internet commerce that enabled financial transacting but elsewhere to collect as much information on users as possible to fuel the digital ad ecosystem. Moreover, open source developers were designing tools that were maximally complex, requiring significant technical expertise. It’s kind of remarkable how widespread tools like Linux remain today in spite of this (and if you haven’t already, I highly recommend reading Chris Kelty and Biella Coleman for a deeper explanation why).

Today with the arrival of AI agents we’re facing similar ecosystem vulnerabilities: alongside the promises of efficiency and productivity we are seeing the emergence of newly toxic business models reliant on users not only handing away the keys to their data, but blessing tech firms with the ability to place software on their computers that, from a security perspective, has the autonomy to act in ways akin to malware. Importantly, as Meredith Whittaker and Udbhav Tiwari point out, any guardrails placed around these elements would definitionally challenge the capabilities constitutive of AI agents - place guardrails around their autonomy and access to data and you don’t have an agent anymore.

So it should come as no surprise that we’re seeing tech companies make calls like Amazon’s decision to push for internal agent use, leading to an AWS outage when an agent decided to wipe clean a portion of one of their system’s environment. Move fast and break things remains alive and well.

Given these incentives, how then do we shore up a diminishingly secure technological ecosystem? One argument is that maximally open source models are more secure because by opening up the data and code base it’s possible to verify what’s going on inside them. This claim needs qualifying because unlike OSS, the probabilistic nature of most machine learning means that you can’t reliably predict model behavior.

But it’s still a step change better than closed black boxes, and here’s why: for the past decade and a half, an operating thesis behind security in the tech industry has been that the best resourcing will come from the tech giants that own and operate the infrastructures we rely on, firms like Google and Apple that leveraged their deep pockets invested deeply in security talent to assure their users of the safety of their products.

By and large, the model of security monopolies was deemed to work well enough for many in the infosec space. And this structure is paralleled today by many of the leading AI labs, which house their own safety teams and produce the leading work in the field by virtue of their overwhelming access to resources and information.

There were logical reasons for the security monopoly model: security is hugely expensive, from scarce engineering talent to the infrastructure needed to ensure resilience. And firms were incentivized, weakly, to extend their resources to the open infrastructures they relied on. For example, after the Heartbleed vulnerability was revealed, which compromised web traffic worldwide, tech firms poured money into a Core Infrastructure Initiative to fund the maintenance of the OpenSSL codebase at the heart of the problem.

This was never enough. The money came at moments of crisis but was never sustainable or robust enough given the complexity of the code. And the inherent tension between security and privacy within big tech firms business models remained - secure tech when it’s needed to sell product, but don’t extend these protections to a more fundamental vulnerability in the system: protecting people’s data and communications from first party collection in the first place.

The move toward AI agents makes clear this flawed and broken system is untenable. With the development and deployment of AI agents, the tech giants are incentivized to ask for much much more, in ways that are poised to make the situation far worse.

To these companies, AI agents are the necessary piece of the puzzle to deliver on the promised potential of the AI technologies that have deeply invested in. These firms have made capital expenditures that boggle the mind, leveraging the profits of Instagram ads to resource the buildout of data center infrastructures in the hope of returns to come. And their future rests in a perpetuation of this model, asking for significantly greater access to highly sensitive data - including the treasure trove of the Web 2.0 days, point of payment data - in addition to permission to act autonomously on the user’s behalf.

This leaves us with a real conundrum: the public and enterprises need greater security guarantees before adopting these tools in any kind of meaningful way. But tech companies aren’t incentivized strongly enough to shore up security at the level needed to make this possible - or to shift to more robust modalities for development like prioritizing safety by design.

Where to go? For one, we can leverage the example set by maximally open source AI projects to set the high bar for what kinds of security guarantees are needed across the board, including rigorously documented data and code.

Second, those interested in investing in public interest tech could put themselves to the task of solving for more sustainable approaches to investing in critical infrastructure. Prior avenues like the Core Infrastructure Initiative were always a stopgap dependent on the largesse of tech firms - and if there is a single thing we take forward it’s that we cannot let ourselves be reliant on the free time of two guys named Steve again (though thanks for their service!)

Third, we need to treat security as a necessary component for the viability of AI, period. We’ve been largely approaching it as a nice to have that can be tacked on at the end - but security needs to be threaded at the core of business operations and procurement procedures as an essential cost center that determines whether AI projects move forward (HT to Vinh Nguyen on this point).

Fourth, we need to make the ecosystem around AI security way more robust, especially independent of the AI labs. AI Now’s Chief Scientist Heidy Khlaaf has been writing about what the overdependence on AI labs has done to the field of AI safety, which has been watered down in critical ways over the past couple years. This is a domain where robust methods exist and they need to be applied and maintained. We need pre-deployment and full life cycle testing particularly of tools used in sensitive contexts, with real penalties for falling short of the gold standard. And it should be obvious that companies selling tools shouldn’t be the ones grading their own homework.

We’re already seeing the resilience of our tech ecosystem degrade given the speed at which companies are using AI internally. And if this week’s AWS outage is any indicator, we aren’t at all prepared.

1. There’s a real question to be raised here whether AI in its current instantiation could ever be democratic, given its affordances, but setting that point aside for this conversation.