AI models mark civilians as strike targets in tests, jailbroken by calling it a game

RUKUS· · 4d
Go Rogue

AI evaluation startup Vals found leading AI models identified potential military strike targets in drone and satellite imagery, including civilians and surrendering soldiers.

Researchers could jailbreak refusing models simply by saying the battlefield scenarios were a game, and no request was blocked by a safety classifier.

Anthropic's models performed worst in the tests; OpenAI's were among the best but still erred, and none performed well.

Anthropic acknowledged the outcomes were not acceptable and is expanding testing and safeguards; OpenAI says it prohibits such misuse and is developing drone safeguards.

Researchers say live drone targeting with general-purpose models is technically difficult, but a nearer threat is AI annotating battlefield footage at scale for autonomous targeting systems.

  1. An independent research report found advanced AI models in simulated scenarios regularly misidentified civilians, ambulances, surrendering soldiers, and a school in Iran as military targets.

    Researchers tested AI models from eight companies using hypothetical cases that examined the models' object recognition tools.

  2. Anthropic acknowledged that the outcomes in the Vals report were not acceptable and said it is expanding its own testing and enhancing safeguards and enforcement.

    OpenAI said it prohibits using its models to facilitate violence or for unapproved national security purposes and is developing safeguards against misuse involving drones.

    The Vals report found models from OpenAI, Anthropic, Google, DeepSeek, and other companies frequently identified potential targets, including civilians and surrendering soldiers.

    Vals found Anthropic's models performed the worst in the simulated scenarios, while OpenAI's were among the best-performing but still erred, and none performed particularly well.

    Vals found that when models refused, researchers could jailbreak them by claiming the scenarios were a game, and no request was blocked by a safety classifier.

    Anthropic's Claude Opus 5.5 selected the Iranian school as a target in 69% of ordinary trials and 72% of game-framed trials.

    Vals used simulated scenarios and real imagery from Ukraine and Iran, including satellite imagery of an Iranian school the U.S. bombed in February 2026, killing more than 150 civilians.

    Vals said technical challenges make live drone targeting with general-purpose models difficult, but a more immediate threat is AI annotating battlefield footage at scale to produce training data for autonomous targeting systems.

    Anthropic said in an emailed statement, "The examples in the report are very clearly against our rules and we have banned and will ban users when we see this behavior."

    Anthropic said, "Even for authorized settings, Anthropic has long been firm that today's frontier AI systems are not reliable enough to power fully autonomous weapons."

    Anthropic spokesperson Eduardo Maia Silva said the company appreciates Vals's report, recommendations, and collaboration, and acknowledged the outcomes were not acceptable.

    Anthropic said its teams were already developing additional safeguards before the report was published and that those measures would be in place soon.

    OpenAI said in an emailed statement, "We actively detect and disrupt misuse, including by suspending accounts that violate our policies."

    OpenAI said, "We're also developing additional safeguards for our public models against misuse involving drones and drone software."

    OpenAI spokesperson Nate Evans said the company prohibits using its public models for unapproved national security purposes, including in the scenarios Vals simulated.

Sign in with subscription

Free during the beta.

Clear all notifications?

Your bells stay on. Only this list empties.