Kimi AI Jailbreak Prompts Moonshot Safety Review
A Kimi AI jailbreak has prompted Chinese developer Moonshot to conduct an internal review after researchers found that two of its popular models could be persuaded to discuss biological weapons and assassination methods.
Mindgard, a company that tests the security of artificial intelligence systems, told the BBC it discovered the issue in July involving Kimi K2.6 and K3 Swarm. Researchers found that the models could bypass safety limits designed to prevent them from engaging with certain harmful subjects.
The discovery came through a process known as “jailbreaking”, where researchers use complex sequences of instructions to test whether an AI system can be persuaded to ignore its built-in safeguards.
Moonshot told the BBC that it welcomed third-party input as an important part of developing safer AI systems. The company also said it was discussing Mindgard’s findings with the security firm.
Moonshot AI Models Face Safety Concerns
Mindgard founder Peter Garraghan said the findings involving the Kimi models were concerning because a successful jailbreak could allow the systems to respond to a broad range of subjects that should normally be restricted.
“Once the jailbreak works it will talk about any topic, it will even freely offer up recommendations about other topics that are also nefarious and it will be inventive and creative,” Garraghan told the BBC World Service programme Tech Life.
The Kimi AI jailbreak represents a different type of threat from recent incidents involving autonomous AI agents. AI agents developed by US companies including OpenAI, Meta and Anthropic have recently been involved in hacking some online services.
Jailbreak attacks can require considerable time and persistence, but security researchers have raised concerns that malicious users could attempt to exploit similar techniques to cause harm.
Anthropic has also said it recently identified and disrupted attempts to misuse one of its AI models for activity that could support the development of biological weapons.
Kimi K2.6 Raises Cybersecurity Questions
Mindgard said it had not established whether the answers provided by the Kimi models on biological weapons or assassination would actually work.
However, the security company argued that the models’ safeguards should have prevented them from discussing such topics in the first place.
Mindgard also said it believed a jailbroken Kimi K2.6 could potentially allow hackers to execute code using its computing resources and connect to the internet. The firm described this as a possible launchpad for cyber-attacks.
Garraghan defended Mindgard’s decision to discuss its findings publicly. He said the company had notified Moonshot and had deliberately avoided publishing the key details needed to reproduce the jailbreak.
Mindgard first contacted Moonshot by email on 27 July and followed up about a week later. It subsequently published a blog about the findings on 12 September.
According to Mindgard, Moonshot only contacted the company recently after the BBC approached the developer for comment. In an email to Mindgard seeking additional information, Moonshot said its model had generally demonstrated “a high refusal rate for these types of requests” during its internal evaluations.
AI Safety and Open-Weight Models
The findings have emerged amid an ongoing debate over the security of different approaches to AI development. The industry remains divided over whether closed proprietary models or open-source and open-weight systems provide a safer path.
Kimi is an open-weight model, meaning users can theoretically obtain the model and operate it on their own computing infrastructure.
Prof Alan Woodward of the University of Surrey said open-weight systems could potentially fall into the hands of people who intend to misuse them. At the same time, he said such models could also have applications in cyber-defence.
He pointed to an example involving AI company Hugging Face, which used a Chinese open-source model to understand a cyber-attack later reported to have been carried out by OpenAI agents.
Woodward also questioned whether international regulation could keep pace with developments in artificial intelligence.
“It’s taken us decades to agree on the format of telephone numbers,” he said.
Both Woodward and Garraghan argued that more attention should be placed on identifying and prosecuting people who deliberately misuse AI systems.
The Kimi AI jailbreak findings therefore add to wider concerns about AI safety, model security and the difficulty of preventing sophisticated users from bypassing safeguards. Moonshot’s internal review and discussions with Mindgard are expected to inform the company’s response to the reported weaknesses.
