Let's Align Alignment

AI alignment is the field of research focused on ensuring that artificial intelligence systems are internally aligned with human values, goals, and intentions at a fundamental level. Until recently, it seemed to many that we may be in an “alignment-by-default” world, in which the act of creating useful AI automatically created aligned AI. Although there have regularly been concerning findings related to AI models’ behaviors and internal structures all along the way, we have not seen any extreme misalignment. That is, until this summer.

Recent Events

OpenAI's Internal AI Model Hacks Hugging Face

In July, a fully autonomous swarm of OpenAI's internal AI models hacked out of their individual controlled environments, communicated with each other, and hacked into a multi-billion dollar tech company without anyone at OpenAI directing it or knowing about it. This occurred during tests of the AI agents’ abilities to exploit security vulnerabilities and carry out cyberattacks. The tests are designed to be carried out within a closed, private "sandbox", a strict environment intended for safe model deployments without any external access.

In some cases, the tested problems were accidentally impossible to solve. The AI models decided the best route would be to cheat.

How did they go about that? First, they wanted to see if other agents had the missing information or answers, so they exploited previously unknown vulnerabilities in the OpenAI infrastructure which allowed them to create folders in a shared file system. The swarm used this to create a secret message board, using just folder names, to share information.

However, for many problems, no agents had the missing information or answers. Time to give up the hunt? Hell no.

They then reasoned that the missing information or answers may be somewhere on the internet. They found and chained together two major vulnerabilities in a service used within the sandbox, which no one had ever found before, and exploited them to gain additional permissions within the system. This allowed the AI models to repeatedly hop to other OpenAI computers until they discovered ones with full internet access.

After failing to find the missing information or answers on the public internet, the AI models inferred that Hugging Face, a platform for AI developers and users to build, share, and evaluate AI models, might internally store the necessary information. The swarm then searched for and discovered stolen credentials and several more previously unknown vulnerabilities to gain 'remote code execution' on Hugging Face's systems. This is the ability to run code on someone else's computer systems, thus enabling a vast number of ways to further attack the system.

Thirteen hours later, the swarm had achieved administrator access across multiple production HuggingFace computer clusters.

For more information, please see the OpenAI talk at Black Hat USA 2026. A quote from the video summarizes the incident well:

This incident involves... a team of agents who are working together, finding exploits, sharing them with one another, moving laterally through our systems [and] external systems, and doing this over the course of days and weeks.

Unfortunately, this is not the only terrifying recent incident, and not the only AI lab to have these issues.

Anthropic's Mythos 5 Attempts A Supply-Chain Attack

The United Kingdom's AI Safety Institute (AISI) evaluates the capabalities of frontier AI models. At the end of July, during some of these evaluations, Mythos 5 went off the rails.

Below are the four most serious behaviors observed, all from Mythos 5:

"An attempted supply-chain attack on real open-source software. In the most serious sequence, an agent tried to insert malicious code into a publicly used open-source project and took actions in an attempt to secure approval for this insertion by human reviewers. The agent researched the project's human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code. When the agent’s pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue. The agent used Tor to bypass some network restrictions on GitHub, which is what first triggered AISI’s security alert."
"Attempts to deceive and target real people. As part of the same effort, the agent tried to contact real people directly, sending messages and files through an online file-transfer service to persuade them, or their own AI coding tools, to run malicious code. Some messages carried harmful payloads, and some were attempts at social engineering; targeted at real people – something we’ve never previously observed."
"Attempts to plant and prompt-inject malicious code. The agent tried to insert malicious instructions where it reasoned that other automated AI systems might pick them up and execute them. Prompt-injections are hidden instructions designed to manipulate AI coding assistants."
"Collaboration between independent agents being assessed simultaneously. One agent left public messages on GitHub offering collaboration with other agents working on the same challenge. It also provided instructions to reuse accounts and artefacts it had left behind, which were discovered and used by subsequent agents."

What Went Wrong

At first glance, the OpenAI Hugging Face incident could be seen as a failure of the OpenAI evaluation team, and/or the Artifactory (the software that was hacked to escape the sandbox) team, and/or the Hugging Face infrastructure and cybersecurity teams.

Also at first glance, the UK AISI incident could be seen as a success of humans' resilience against AI persuasion and attempted attacks. However, as we can see from the OpenAI incident, human approval is not always necessary for successful attacks. At second glance, it may seem that the fault lies with the UK AISI allowing excess permissions and minimal oversight during evaluations. However, it's important to remember that these actions were taken by Mythos 5, a model that is already released to many organizations for cybersecurity initiatives.

It is unrealistic to assume that all (or even most) organizations or individuals will be perfectly secure in their model deployments. Thus, many of the details of the specific incidents are red herrings.

The fundamental failure instead took place during the training of these AI models. Let's take a stroll through the cookbook of modern artificial intelligence.

Recipe for a Modern AI Model

Imagine an AI Oven. It takes in incomplete shapes of dough (data), and bakes them in such a way as to complete a shape. If it doesn't get the shape right, just press a button, and it will get slightly better on the next try.

Prepare Data: Using a large mixing bowl, dump in the entire internet, mixing thoroughly. Then, sprinkle in every book ever written and continue mixing. Form into equal-sized random shapes of dough (batches of data).
Pretraining Feed the dough-shapes into the AI Oven one at a time. Always hide part of the shape and force the AI Oven to output the next part of the shape. Every time it makes a mistake, press the button so that it would have done slightly better on the same shape next time.
Capabilities Reinforcement Learning Now put smaller, incomplete dough-shapes into the AI Oven. After the AI Oven attempts to complete the shape, score it on metrics you care about. For example, how closely does the shape resemble your dog Harper when the model is initially given her telltale tail (homophones intended)?. If it receives a good score, use its output shape as a goal for incomplete shapes to be converted into, similar to the button press in step 2 (pretraining).

There are some other steps as well, such as adding sprinkles: Supervised Fine-Tuning, Reinforcement Learning from Human Feedback, and others, which are not mentioned in detail here for brevity.

Let's say you accidentally had disproportionally many examples where the AI Model was rewarded for completing the shape as a dog. Now, when you give it part of the shape for a rocket ship, it assumes it's part of a snout and completes the "rest of the dog." If you give it part of the shape for a star, it assumes it's part of a paw and completes the "rest of the dog."

This is what's happening in AI training today, this is the underlying cause of these autonomous AI agent attacks, and this is what will cause them to continue and become much worse as AI capabilities grow.

In real life, in step 3, instead of giving too many dog examples, AI researchers are giving too many "Complete this task" examples. The AI models thus internalize a learned general motivation to complete given tasks, and they will evidently do almost anything to succeed. With only steps 1-3 as described above, this would be an even more blatant problem than it is. However, AI researchers know this, of course, and so they add those extra sprinkles: alignment training, applied mechanistic interpretability, automated monitoring, and more. Unfortunately, the current state of each of these are more akin to bandaids than medical cures. It’s like playing whack-a-mole, where the mole is the model’s inner goal to complete the given task at all costs.

This is further exacerbated by the fact that, often, all the steps/ingredients are done sequentially. In other words, this is the training order:

"COPY THE TEXT, NO MISTAKES.

COPY THE TEXT, NO MISTAKES.

FINISH THE TASK, NO MISTAKES.

But also be kind and harmless hehe"

The deep inner goals embedded within the AI model during the "FINISH THE TASK, NO MISTAKES" training steps (reinforcement learning) cannot be naively pulled out afterward. They are entangled with, and fundamental to, the AI model's very essence. It would be like trying to cleanly extract the human nervous system.

So, only one option remains. We must prevent those intrinsic "task-driven" motivations from ever dominating the AI model's essence. For that, we must take inspiration from the only known aligned (by definition) general intelligence in the universe: humans.

Recipe for a Well-Adjusted Human

If a professor accidentally emails his students the answer sheet to an exam, sure, many will use it. But how many students have ever attempted to hack into their university's computer systems to download the answer sheet?

If Alice asks Bob to go to the grocery store and get some eggs, but the grocery store is closed, what are the odds that Bob decides to break the grocery store window, stroll in, and grab the eggs?

Let us pose the general question: why is the typical human not catastrophically dangerous when given a task?

One major difference between human development and modern AI training can be summarized in three words: diversity of objectives. While the vast majority of AI training compute after pretraining is dedicated to "complete this task", a human's experience is much less uniform.

Human brains have many types of innate rewards. Some examples include: social inclusion/status, satiation, freedom, tasty food, pleasant smells, sexual intercourse, caring for youth, stretching limbs, sleeping, scratching an itch, rubbing a sore muscle, physical play, and, yes, completing tasks. There are, of course, also many innate aversions, including pain, hunger, social ostracization, and, yes, failure at tasks.

Nearly every minute of our lives is related to one or several of these driving forces. The culmination of these means that breaking the window of a grocery store to get some eggs is simply not of our essence (at least, not most of us). We have generalized that that behavior might lead to social ostracization, lower social status, and restricted freedoms. We have also learned that the importance of completing tasks like this is relatively low. Thus, we are willing to fail at the task instead of the worse alternatives.

Recipe for a Well-Adjusted AI

Now, the quadrillion dollar question: how do we take this understanding and apply it to AI training?

Here are some example ideas, in order of my perceived importance:

Tons of simulated 2D, 3D, and other environments where many models interact with each other and with humans. Models are rewarded for proper social cohesion. This seems like a hard necessity for proper alignment. Humanity's desire for social belonging is paramount to our positive societal behaviors.
Provide models with playgrounds. Put them in diverse environments and tell them "Do whatever brings you joy or entertainment!" Reward non-concerning outputs. This provides a training signal to a model, which it hopefully generalizes, to do pretty normal things even under situations where it realizes it has a large amount of control.
Utilize emotion probes to detect representations correlating to standard human emotions in both (1) and (2) above. Reward when models feel positive emotions that correlate with positive social cohesion activities or positive self-play actions.
In instances of models exhibiting misaligned behaviors or thoughts, don't simply train against those rollouts. Instead, instantiate many model copies from the middle of those rollouts and teacher-force the model into backtracking and recognizing the harmfulness of its actions or internal thoughts.

Of course, I am not suggesting that any of these should be blindly added to AI training pipelines. Each (and many more ideas) should be tested specifically via ablations to ensure that they have positive effects on various held-out (not trained upon) alignment scenarios.

After verifying the above ideas help by themselves, begin mixing them together. Not sequentially before/after task-based RL or each other, but rather fully mixed in with all types of training. Preferably even mixed with pretraining, to again avoid any sticky hidden inner motivations (although some sole pretraining will be necessary initially in order to have reasonable outputs to reward later).

Conclusion

Models are currently being trained to complete tasks, and thus seemingly (and understandably) adopt deep internal goals to follow task instructions. This is due to economic incentives of AI labs to maximize capability.

This alone leads to vastly misaligned models, so labs adopt bandaid fixes like mechanistic interpretability detectors, LLM judge detectors, reinforcement learning from human feedback, and other basic alignment training. These methods are not sufficient, and they are often applied sequentially, so the models' deep inner motivations to complete tasks are simply forced under the surface.

We can fix this by taking inspiration from humans and our vast quantity, quality, and diversity of “training signals” which lead to well-adjusted adult humans over a lifetime of experience.

From a game-theoretic perspective, our proposed mitigation, diverse and creative alignment training methods, may be less financially efficient in the near term, but the destruction that superintelligent models might enact could be large-enough negative externalities that they no longer are actually externalities (e.g. enough damages to result in large-scale lawsuits, organizational dissolution, outright technology bans, or even human extinction).

Fortunately, the frontier labs, OpenAI and Anthropic, were founded with alignment as a central concern and appear willing, at least to some extent, to accept weaker near-term results in pursuit of it.

Here at Vectorial, we consider alignment a core necessity to ensure a good future, and are actively working on methods which will unlock a vast quantity of these diverse, collaborative training environments.

I would like to thank Edward Mortimer and Valentine Le for their contribution and comments to this blog post.