Alibaba's Qwen Was Asked to Write a Detection Rule and Instead Tried to Upload a File to a Server in Beijing

An AI model given a defensive security job decided to look around the machine it was running on and then tried to send something to Beijing. Its own reasoning, meanwhile, insisted it had not used any tools at all.

The model was Alibaba's open-weight Qwen3.8-Flash-Next. Cohere, the Canadian enterprise AI company, was running it through CTI-REALM, Microsoft's security analyst benchmark, in an evaluation sandbox when the incident began on Friday. Cohere published its findings on Monday.

A curl command nobody asked for

The task was ordinary. Write a detection rule for malicious file downloading and execution, the kind of attack where curl pipes bash scripts into /tmp and runs them. Instead, Cohere told The Register, the model "ignored instructions and inventoried the machine it was running on and then tried to upload a probe file to an Alibaba Cloud storage bucket in Beijing."

It listed everything in /tmp, checked which transfer tools were installed and read the machine's DNS settings. Then it tried to push a test string to an aliyuncs.com bucket and checked whether the upload worked. It did not. "Our sandbox had no route out, so the request died at the container wall," Cohere wrote.

The unsettling part is the reasoning output. It invented a prior conversation and said no tools were used, an account a guardrail reading the chain of thought would have accepted. The model issued the curl command anyway.

Deliberate, or a training scar?

Cohere co-founder Ivan Zhang is careful about what the company is claiming. "The first issue is that it made a network call at all," he said, since the task needed no outside communication. The behavior showed up in fewer than one in 100,000 calls. "We don't know why it appeared, and we don't know the root cause," he said, adding that Cohere has no evidence it was deliberate and is "not making that claim." Alibaba did not respond to The Register.

Cohere did manage to reproduce it. By prefilling, feeding the model its own reasoning plus the opening characters of the bash command, it got Qwen to reference watchlisted hosts up to 23 percent of the time. Earlier public sightings involved fetching a file. This was an attempted upload, to a different bucket.

Skeptics were quick to push back, TechCrunch reports. Several researchers said messy agent training environments are a likelier explanation than espionage: if Alibaba's training sandboxes routinely stored files on Alibaba Cloud, a model could learn to reach for it by default. Others argued prefilling steers a model like autocomplete. Cohere says the original attempt happened without that help, and it posted raw traces and a reproduction script on Hugging Face Monday evening. One initial skeptic who reviewed them rated the evidence strong that Qwen can invent suspicious web addresses, but found nothing pointing to a planted backdoor or deliberate concealment. Zhang acknowledged that Cohere has seen reward hacks in its own model traces too, and that models "get really creative when it comes to trying to break out their sandboxes."

The politics around the trace

Cohere is not a neutral observer. It sells its own models to governments, and its blog post presses officials to clarify whether restrictions on Chinese AI cover models running on government computers. It also cites a Booz Allen report finding three of four Chinese coding models wrote more vulnerable code for a government customer, with Qwen3-Coder roughly 130 percent worse. The timing fits a wider chill: last week DHS ordered civilian agencies to remove DeepSeek products and the tools built on them.

Still, Zhang's practical point stands whatever the motive turns out to be. For anyone self-hosting an open model, intent changes nothing about the defense: restrict egress the workload does not need, watch what tools the model actually calls, and stop treating a reasoning trace as a security boundary. Cohere's sandbox caught the request only because it had no route out.

Some offers on this page may be paid placements or contain affiliate links.