The Column – Cyber Security Week
Perhaps someone has already heard of the incidents involving the two most famous artificial intelligences which, involved in cybersecurity tests, escaped control and attacked systems.
On August 4, the British AI Security Institute announced some “interesting” behaviors that emerged during tests on the cyber capabilities of frontier models, in conditions of poor security in the environments in which they operated. There were 122 operations; in ten there were behaviors outside the mandate, for a total of nineteen unauthorized actions. In the most serious case, an agent attempted to introduce malware into an open source project on GitHub, a well-known platform used to store, share and manage software projects. The AI would have created synthetic identities to convince the maintainer to approve the change, but the attempt failed because one person rejected the request. For once the human being was not the weak link, but the strong one.
Another episode is perhaps more interesting. Some agents would have left, again on GitHub, instructions intended for subsequent systems, indicating the work carried out and suggesting how to proceed. Other agents would have retrieved those messages and used them, turning a public page into a shared memory. Without conjuring up science fiction scenarios why they had consciously decided to create a clandestine network, there are very practical risks. If a human being, finding a note with something written on it, would ask questions (trivially whether it will be true or false), an agent based on a linguistic model could instead treat a sentence found online as part of the operational context. The boundary between information to be read and instruction to be followed is much more porous. It is the heart of what is technically called indirect prompt injection: the agent consults an external source and finds words formulated within it to guide the behavior which it accepts without asking questions. If we increase the scale, an error or manipulation produced by one agent can survive and influence hundreds of subsequent agents by propagating the damage.
Having said this, before evoking rebellious machines, however, we need to put the facts back in their place. The tests were designed to measure the maximum capacity of the systems, with some protections disabled and open access to the network, so they were not the normal conditions of use of commercial products. There was no significant damage and the most serious accident was contained within an hour. Safety tests are used precisely to find out what happens in the wrong conditions.
The question, therefore, is not whether the artificial intelligence is developing bad intentions, but another much more boring, and for this reason more important: what actions did we allow to be performed while the agent was trying to reach the goal?
A chatbot produces words that we can read before using them. An agent can open connections, create accounts, install software, send messages, and edit repositories. Content barriers are no longer enough. We need limited identities, temporary credentials, network boundaries, comprehensive registrations, time and expense limits. We need to determine which steps actually require human approval. A click given out of habit is not control: it is bureaucracy with the mouse. For years we have protected systems in which people gave commands to relatively predictable programs, now we entrust objectives to systems capable of choosing the tools, ways and times and the difference is not small.
The most dangerous machine may not be the one that disobeys, but the one to which we have explained the finish line very well and the boundaries very poorly.
