A reporting failure kills the daemon and every job it is running #25

New Issue

Closed

opened 2026-08-04 06:25:50 +00:00 by day01 · 0 comments

day01 commented

2026-08-04 06:25:50 +00:00

Owner

A job whose final log update is rejected takes the whole runner down with it.

Observed on the production runner three times on 2026-08-03 (16:05:16, 19:08:10,
20:32:23):

error: reporting failed: control-plane reporting failed: UpdateLog returned
HTTP 500 Internal Server Error: unknown: rpc error: code = AlreadyExists

ExitCode=0, OOMKilled=false, RestartCount=3 — the daemon ends itself and
compose brings it back. The release build of 0.2.1 died with it: job publish
of run 293 was marked failed at 20:32:21, two seconds before the restart, after
it had already pushed runner:main, latest and the commit tag. Its log simply
stops; nothing reports a cause, because the process that would have reported it
was gone.

The path is direct. crates/runner-agent/src/agent/run.rs:208 calls
finish_job(result)?, and finish_job (run.rs:222-231) does
result.map_err(AgentError::TaskJoin)?? — the inner ? propagates
ReporterError out of the agent loop, so run() returns Err and the daemon
exits. A per-job reporting problem is handled as a fatal daemon condition.

Expected: a failed report fails that job and is logged. The daemon keeps
serving, and unrelated jobs on the same runner survive.

Worth pairing with the forge side: UpdateLog answering 500 with
AlreadyExists on what looks like a retried or duplicated final chunk is its
own bug. But even with the forge fixed, one rejected request must not be able to
kill a runner mid-build.

A job whose final log update is rejected takes the whole runner down with it. Observed on the production runner three times on 2026-08-03 (16:05:16, 19:08:10, 20:32:23): error: reporting failed: control-plane reporting failed: UpdateLog returned HTTP 500 Internal Server Error: unknown: rpc error: code = AlreadyExists `ExitCode=0`, `OOMKilled=false`, `RestartCount=3` — the daemon ends itself and compose brings it back. The release build of 0.2.1 died with it: job `publish` of run 293 was marked failed at 20:32:21, two seconds before the restart, after it had already pushed `runner:main`, `latest` and the commit tag. Its log simply stops; nothing reports a cause, because the process that would have reported it was gone. The path is direct. `crates/runner-agent/src/agent/run.rs:208` calls `finish_job(result)?`, and `finish_job` (`run.rs:222-231`) does `result.map_err(AgentError::TaskJoin)??` — the inner `?` propagates `ReporterError` out of the agent loop, so `run()` returns `Err` and the daemon exits. A per-job reporting problem is handled as a fatal daemon condition. Expected: a failed report fails that job and is logged. The daemon keeps serving, and unrelated jobs on the same runner survive. Worth pairing with the forge side: `UpdateLog` answering 500 with `AlreadyExists` on what looks like a retried or duplicated final chunk is its own bug. But even with the forge fixed, one rejected request must not be able to kill a runner mid-build.