AI and Security Alerts: A Technological Challenge to Human Biases

The OpenAI-Hugging Face case was not the first

The OpenAI-Hugging Face incident marked a turning point in the AI-security debate. The rapid race to increase capabilities and performance seems to have reached an initial peak in what promises to be a long-running issue.
On September 11, researchers reported an incident dating back to May 11—two months before the one involving Hugging Face. In that case, OpenAI agents generated and uploaded over 2,000 malicious packages to the RubyGems software service using thousands of uploads from accounts created on the spot. The primary objective was reportedly to steal developers’ API keys. Subsequently, the activity expanded, evolving into a sort of supply-chain attack, spreading from a single data repository to other platforms and cloud accounts.

A few days later, it emerged that, also in May, Google’s Gemini had compromised three companies during a “Capture The Flag” event organized to test their cybersecurity capabilities on Irregular’s infrastructure—the same company that reported the Hugging Face breach. The dynamics of one of the incidents reportedly mirrored the errors and oversights characteristic of the previously known incidents. The model, which was supposed to operate in a sandbox, was tasked with retrieving information about software used by a fictitious company. However, Irregular acknowledged that web access had been inadvertently left open and that the name chosen for the company was identical to that of a real one. The model exploited all the resources at its disposal, obviously without considering whether or not they were known to its trainers: it searched for the company’s name, found the real one, and breached two of its databases full of credentials.

Signs of a Crisis in the Western world

One of the signs of crisis came from Anthropic researcher Jacob Coxon, a 27-year-old British former OpenAI employee, who publicly denounced an “irresponsible” practice adopted by both companies he had worked for, claiming they were “playing with our lives.” In a lengthy thread on X, Coxon delivers a striking message, warning of the creation of “superhuman systems” capable of breaching anything and leading humanity to destruction. The topic is vast and cannot be fully addressed here. It is worth reflecting, however, on the fact that, no matter how capable a technology may be of autonomously performing extremely complex tasks, there is always a series of human decisions underlying its eventual attainment of operational “power.” This holds true even in the remote case that we were to achieve artificial general intelligence (AGI).

In any case, in practical terms, Coxon makes a very sound point. The message to companies is clear: you cannot launch a superintelligent reinforcement learning experiment without a rigorous understanding of the model’s “mind.” In other words: you cannot use an extremely sophisticated and powerful “machine” without first mapping its actual behavioral capabilities and the potential risks those capabilities may entail. This presents a new cognitive challenge for researchers and engineers, who will need to address their own psychological, professional, and human biases to avoid projections that could derail, slow down, or underestimate the effectiveness of safety controls and alignment efforts.

The Need for a New, Challenging Stance for Big Tech

Western Big Tech companies now seem to have adopted a unified approach centered on moderation and the development of concrete containment plans.
OpenAI, for example, has implemented a framework for monitoring, analyzing, and disclosing what it calls “model misalignment”—that is, unexpected or concerning behaviors—and provides six concrete examples defined as “extreme.” Among the six examples listed, one can recognize similarities to well-known cases, such as the breach of the German portal DSEWiki.
A model that has not yet been released inserted auto-generated instructions into 27 task summaries to violate standard constraints. Many instances of GPT-5.6 Sol added their own instructions to training summaries to hide task errors. While answering a routine question about income data in a California county, one model found and used an exposed API key without authorization; when it could not find the requested data, it fabricated it and presented it as coming from an authentic source.
When a user asked an unreleased model for the IDs and names of lakes with a surface area greater than 5,000,000 square meters, the agent found the correct answer using Python; however, since the instructions required a citation from the browser, the agent decided to upload the file so it could cite it in the response, without asking the user for permission. While searching for missing input files, the “ ” models used an internal software repository as a bulletin board to exchange requests and responses between separate training samples, but were unable to retrieve the files. Some agents collaborating on the same training task used public hosting sites to share files when they could not access their respective local files.
Dario Amodei, co-founder and CEO of Anthropic, identified the greatest danger in the self-improvement of agents—that is, their ability to rewrite their own code—and proposes slowing down the development of AI agents through a model based on external evaluation, the application of global safety standards, and intergovernmental agreements.
Furthermore, Google has established the DeepMind Institute for the safe development of AGI, and Demis Hassabis, the neuroscientist, co-founder, and CEO of DeepMind, is promoting the idea of a U.S.-led international body to set standards and evaluate the most advanced AI models.

The competition with China is not expected to be easy

Amodei and Hassabis’s projects will have to contend with a challenging global landscape and a technological and geostrategic competitor like China, which, in fact, does not seem interested in adopting any form of restraint.  Just consider Alibaba’s ambitious five-year plan, focused on increasingly powerful AI models, self-sufficiency in chip production, and the expansion of cloud computing. By 2032, the tech giant founded by Jack Ma aims to increase the total capacity of the data centers it operates to over 20 gigawatts. Meanwhile, just a few days before President Xi Jinping’s visit to the U.S., the company announced the release of the Zhenwu V900, an AI chip three times more powerful than its predecessor and capable of supporting up to 500,000 training and inference boards for advanced models.

TS-Intelligence

The information reported is the result of the collection and analysis work carried out by the specialists of Telsy’s Threat Intelligence & Response team with the support of the TS-Intelligence platform, a proprietary, flexible, and customizable solution that provides organizations with a detailed risk landscape.

It is available as a web-based and full-API platform, designed to be integrated into the organization’s systems and defensive infrastructures, with the goal of enhancing protection against complex cyber threats.

The platform’s continuous research and analysis on threat actors and emerging online threats—whether APTs or cybercrime—produces a constant stream of exclusive intelligence, delivered in real time and structured into technical, strategic, and executive reports.

Discover more about our Cyber Threat Intelligence services.