This spring, the US military nearly boarded a Chinese ship in the Middle East based on an intelligence report that was completely made up, generated in part by a chatbot that got the facts wrong. That’s not a hypothetical risk scenario. That happened.
According to CNN, the report alleged that the Chinese vessel was carrying components of a nuclear weapons program. Military planes were in the air. Armed personnel were preparing to board. It was only at the last moment that officials looked harder at the underlying report and discovered a special operations analyst had used AI to produce it, and the chatbot had misidentified what the ship was actually carrying. One source described the report as “entirely false” and said it “almost started a war.”
Think about what that actually means. A US military operation against a Chinese ship in the middle of an active war with Iran could have triggered an armed conflict between two nuclear powers. The margin between that outcome and what actually happened was apparently a few officials deciding to double-check the sourcing.
The analyst had queried a chatbot about intelligence on the ship’s manifest, originally sourced from US Special Operations Command Pacific in Hawaii. The bot then pulled together open-source material with classified signals intelligence and reached its conclusion. The analyst used AI a second time to format the findings into a standard intelligence report, the kind that military officials are conditioned to trust, and sent it up the chain. Nobody flagged it as AI-assisted. Nobody apparently verified it before the military started moving.
What tool was used? That detail is still unclear. CNN could not confirm whether it was a commercial product or a government-built system. But one former senior official put it bluntly: “The internal tools are mostly just copies of the commercial stuff wearing lipstick.” So the military may have nearly gone to war with China based on the same category of technology that writes cover letters and summarizes meeting notes.
This sits inside a much bigger push across the Defense Department to wire AI into nearly everything. In January, Defense Secretary Pete Hegseth released an “Artificial Intelligence Acceleration Strategy” explicitly aimed at putting AI tools in the hands of all three million civilian and military personnel, at every classification level. The strategy is about speed, specifically the fear that China or another adversary will move faster if the US doesn’t adopt these tools aggressively.
But the rollout is fragmented. Different parts of the military and intelligence community are using different tools under different rules with no unified standard for verifying AI-generated outputs. Nobody is coordinating what “good enough” looks like before a report gets disseminated. That gap between ambition and governance is exactly where this incident lives.
The broader concern here isn’t just about one hallucinating chatbot. Sources told CNN that similar AI errors have occurred across the intelligence community since these tools started spreading through government, and this case was not isolated. The specific risk in targeting is obvious and severe:
- AI systems can confidently present false conclusions without flagging uncertainty
- Analysts, especially younger ones who grew up using these tools, are more likely to trust outputs uncritically
- Speed pressure means reports move faster, leaving less time for human verification
- There are currently no consistent standards for how a human “in the loop” actually prevents fatal errors in targeting decisions
One source said it plainly: “AI allows you to get to a bad idea faster.” That’s the privacy and security problem that rarely gets discussed in the AI policy conversation, which tends to focus on data collection and model bias. The more immediate issue is that these systems produce confident-sounding outputs that humans are poorly equipped to interrogate under time pressure, in high-stakes environments, where the cost of being wrong is measured in lives or, apparently, wars.
Washington has spent weeks debating existential AI risk, the possibility that models could eventually escape human control entirely. This story is a reminder that the catastrophic near-misses are already happening, not because the AI became autonomous, but because people trusted it too much, too fast, with too little verification and almost no accountability structure in place.