ReTiCAL: Real-Time Computer Agent Learner

At Vectorial Labs, we're working on aligning AI to humanity via measuring and rewarding prosocial behavior. An obvious type of environment to start with is video games, as there are tens of thousands that already exist and support rich and diverse social interaction.

However, many video games are fast-paced, requiring quick actions and reactions. By default, LLMs cannot perform in these environments, due to slow "time-to-first-token" latencies, often taking several seconds to react to any input.

Generalized "System One" models, such as Jev are fast, but do not have the intelligence necessary to actually succeed in these environments without unfair and unscalable advantages given for each game.

ReTiCAL (Real-Time Computer Agent Learner) is our solution to this problem. We have trained custom fast-action AI models to utilize the reasoning of open-source LLMs alongside their own fast visual processing. This allows them to operate at 20hz in video games, while maintaining advanced general intelligence. ReTiCAL uses pixel-only inputs and mouse-and-keyboard outputs, there is no privileged access to internal game state.

In the demo video presented above, we were able to convert Gemma 4 12B into an instruction-following Minecraft agent with minimal cost (hundreds of dollars of compute).

Creating fast-paced AI agents which can play video games is only an intermediate milestone. At Vectorial Labs, we are utilizing this technology to repurpose video games into environments for rewarding and measuring prosocial behavior.

As discussed in our previous blog post, Let's Align Alignment, we believe current training pipelines overly fixate on "complete-the-task" objectives, which directly causes visible misalignmnet and safety concerns in the resulting AI models. A vital way to align AI models of the future to humanity will be to provide them with training environments that reward them for interacting socially in positive and constructive ways.

We are excited to show you what we have been working on!

(*Note: the above demo video combines non-contiguous episodes)