MyTechDailyNews General: Cloudflare's GLM-5.3 Workers AI Leak Explained

MyTechDailyNews readers who use hosted AI tools now have a public case study about response data surfacing across customers. The disclosure centers on Cloudflare's Workers AI service and the model endpoint @cf/zai-org/glm-5.3, also written as GLM-5.3 or GLM 5.3. It says the remediation and reader-facing report were coordinated for Oct 26, 2026, after a short window that affected 155 responses.

Why this matters to general readers

For ordinary users, the practical question is not how a model stores memory pages; it is whether a service can reveal text belonging to another account. In this case, the answer was yes for a narrow set of very long prompts, then no after fixes. The alert arrived on September 8, 2026; the model returned on September 9, 2026. That September 8 and September 9 sequence happened on the same day, with verification the next morning and withdrawal within 40 minutes of incident declaration. The report describes two overlapping issues: a size mismatch in the SGLang setup and an engine behavior that failed to stop a rejected request. No evidence of deliberate exploitation was found once the researcher's activity and Cloudflare's own tests were removed from the matched traffic.

Timeline in plain language

Date or marker	Reader-facing change
August 27, 2026	The window starts as GLM 5.3 goes live with a prefill pool smaller than the advertised 1,048,576-token context window. Aug 27, 2026 also appears in the published summary.
September 8, 2026	Gadi Evron of Knostic reports through HackerOne that @cf/zai-org/glm-5.3 can leak other users' content in long-prompt replies.
Sep 8, 2026 - 23:11	The intake timestamp in the coordinated disclosure.
Sep 9, 2026 - 08:00	Verification begins.
Sep 9, 2026 - 13:22	Product Security reproduces the leak using two independent accounts; the reproduction window is 25 minutes.
Sep 9, 2026 - 14:00	Incident is declared, then the model is set private; requests return HTTP 400 and the status page is updated.
Sep 9, 2026 - 14:35	Root cause is narrowed to the prefill pool being smaller than the context window; a corrected canary node is not susceptible.
Sep 9, 2026 - 14:52	All prefill nodes move to the corrected configuration.
Sep 9, 2026 - 19:04	The rebuilt deployment includes the configuration correction and an engine patch that returns errors for scheduler-rejected requests.
Sep 9, 2026 - 19:40	The public model is restored, completing the September 9 return the same day.
September 10	A working session with the SGLang maintainers is recorded as Sep 10, 2026 - 23:30.
Sep 11-19, 2026	Additional production hardening is rolled out.
Sep 16-Oct 12, 2026	Upstream work lands for admission handling, transfer checksums, abort acknowledgment preservation, and default deferred KV release.
Oct 2, 2026	SGLang v0.5.21 is released.
Oct 19, 2026	SGLang v0.5.22 is released.
Oct 26, 2026	Publication is coordinated with the researcher and RadixArk.

How the leak worked

The model family is Z.ai's GLM, offered through Workers AI. The root story is about a KV cache: a fixed pool of GPU memory pages reused across requests. Pages are not zeroed automatically, so the inference engine must track who owns what. In this deployment, prefill workers build cache for the prompt and transfer it through RDMA and Mooncake to decode workers, which generate the answer. The pool size is measured at startup from leftover GPU memory; the advertised context length is a separate setting. The mismatch appeared when the pool held 681,280 tokens while the model's advertised context was 1,048,576 tokens.

Another report had already been visible to the open-source community: issue #30233 was opened on July 6, 2026. Cloudflare had also been discussing related memory-safety classes with the SGLang team since v0.5.10.

When a prompt exceeded the pool, the scheduler rejected it. Instead of aborting cleanly, the path swapped the real prompt for a one-token placeholder. In disaggregated mode, the admission checks looked for finished status, not rejection status, so the bad request kept moving.

1. The router announced a prompt that occupied 8 pages, so decode reserved 8 KV pages.
2. Prefill admission treated the rejected request as active and let it proceed.
3. The rejected prompt had been replaced with a one-token placeholder.
4. Prefill transferred only one page of cache and sent a transfer complete signal.
5. Decode admission accepted the handoff.
6. Decode attention read all 8 reserved pages; only one page held data from this request.
7. The client received HTTP 200, prompt_tokens matched the long prompt, the reply appeared coherent, it arrived faster than an 8-page prefill should take, and it contained stale content from earlier users.

What Cloudflare changed first

• Configuration fix: the setup was aligned with the GLM 5.2 configuration, freeing GPU memory so the prefill KV pool holds about 2.04 million tokens, roughly twice the 1,048,576-token context length. If a prompt fits the context window, it also fits the pool.
• Engine fix: a pending rejection is now retired before bootstrap, allocation, or KV transfer. Cloudflare proposed the approach as #38704; upstream #36988 was merged September 6 and released in v0.5.20, while #38935 rejects such requests earlier in v0.5.21.
• Validation fix: the team checked the researcher's reproduction script and its own canary-marker tests before restoring public access.

Hardening and upstream work

The published list includes changes beyond the immediate bug. They are grouped here for readers who want a checklist instead of a dense engineering table.

• Earlier lifetime work included #24226 and #30937; #30899 makes decode wait for an in-progress host offload before retracting a request and freeing pages.
• Cross-node race contributions include #31357, #38961, and reproduction script #32564.
• Incident path: #36988 retires aborted prefill requests before results reach decode; #38704 was Cloudflare's proposed variant; #38935 stops rejected requests at admission.
• Router cancellation: #39461 and #41612 help sgl-router cancel both request halves after a client disconnect or prefill failure.
• Deferred release: #37454 and #41023 enable deferred release where supported, then make it default.
• Drain handling: #40645 and #41402 fix cases where drain confirmation could be lost; #41404 holds pages on every transfer failure; #41618 corrects a regression in the abort notification path.
• Mooncake failure handling: #43719 stops a failed send from claiming writes were drained; #37077 tags abort messages per request and adds MORI support; #36612 shares prefill-to-decode failure notifications across Mooncake, NIXL, and MORI; #32758 caps transfer batch size.
• Lifetime consistency: #43562 keeps tensor-parallel ranks aligned when a host-memory restore fails.
• Observability: #39500 adds a KV checksum option; #39729 is cleanup for that option and is merged while awaiting release; #38596 and #40642 repair KV canary behavior; #41128 adds metrics for deferred release.
• Still open or future-facing: a draft proposal #46435 covers zeroing KV pages on release, and broader support for KV canary and checksum validation continues.

The disclosure says these follow-ups were released to Cloudflare's disaggregated production deployments, and operators running SGLang in similar setups were told to update to v0.5.22 or backport needed changes.

FAQ

Was the bug limited to one model?

The summary says only GLM 5.3 / @cf/zai-org/glm-5.3 was affected. Other Workers AI model names that may appear in related discussions, such as GLM 5.3 Flash and DeepSeek V4, are not described as part of the leak incident.

Did attackers exploit it?

The report says matched traffic, after removing the researcher's and Cloudflare's tests, looked like ordinary long-context use in timing and volume. No evidence of deliberate exploitation was found.

Why would long prompts matter more?

Only prompts longer than 681,280 tokens reached the bad rejection path. That is a very large context boundary, so ordinary short chats were not the affected class described in the incident.

Common misconceptions

• It was not a model weights leak. The exposed data was temporary KV cache state associated with nearby requests.
• It was not a normal HTTP error problem. The client saw HTTP 200 even though the request had been logically rejected by the engine.
• It was not solved by monitoring alone. The configuration mismatch was visible in logs, but response-code tests had not checked whether output used the right memory pages.

Action recommendations

1. If you run Workers AI workloads with very long prompts, treat the disclosure as a reminder to review output isolation after model updates.
2. If you operate SGLang in prefill/decode form, measure KV pool size against context length on every node, especially after changes to weights, quantization, parallelism, or memory fraction.
3. Do not rely only on HTTP success codes. Validate coherent output and memory ownership for over-length scenarios.
4. Update to the patched line the report recommends, v0.5.22, or backport the listed pull requests.
5. Follow the hardening checklist: admission rejection handling, deferred release, bounded transfers, checksums, and canary tests.

This MyTechDailyNews summary is prepared by Sarah Chen for Tech Daily. Questions can be sent to hello@techdaily.com.