fix: keep the daemon alive when a job report fails #28

Merged

day01 merged %!d() commits from %!s() into %!s()

2026-08-04 06:43:47 +00:00

day01 commented

2026-08-04 06:31:45 +00:00

Owner

Closes #25.

A rejected log update ended the whole runner. finish_job propagated
ReporterError out of the agent loop with ??, so run() returned Err, the
daemon exited and compose restarted it — killing every other job the runner had
in flight.

This is not theoretical. On the production runner it fired three times on
2026-08-03 (16:05:16, 19:08:10, 20:32:23):

error: reporting failed: control-plane reporting failed: UpdateLog returned
HTTP 500 Internal Server Error: unknown: rpc error: code = AlreadyExists

ExitCode=0, OOMKilled=false, RestartCount=3. The last one killed the 0.2.1
release build two seconds after the job was marked failed, once it had already
pushed runner:main, latest and the commit tag but before the job image was
built. The build log simply stops, because the process that would have explained
it was gone.

Reporting now fails the job it belongs to and goes to stderr, exactly as a
failed poll already does (run.rs:194). A join failure stays fatal: it means
the task never produced a result at all, which is a different class of problem.

finish_job moved into its own module so the tests fit under the 250-line file
limit. Four of them: a control-plane failure and an unacknowledged final log
both leave the agent running, a clean finish is not an error, and an aborted
task still ends it.

Ran locally: cargo fmt --check, cargo clippy --workspace --all-targets --all-features -- -D warnings, cargo test -p syncode-runner-agent (10
passed), scripts/check-architecture.sh, scripts/check-rust-loc.sh.

Note the other half: UpdateLog answering 500 with AlreadyExists is a forge
bug and still needs its own fix. This change makes it survivable rather than
fatal.

Closes #25. A rejected log update ended the whole runner. `finish_job` propagated `ReporterError` out of the agent loop with `??`, so `run()` returned `Err`, the daemon exited and compose restarted it — killing every other job the runner had in flight. This is not theoretical. On the production runner it fired three times on 2026-08-03 (16:05:16, 19:08:10, 20:32:23): error: reporting failed: control-plane reporting failed: UpdateLog returned HTTP 500 Internal Server Error: unknown: rpc error: code = AlreadyExists `ExitCode=0`, `OOMKilled=false`, `RestartCount=3`. The last one killed the 0.2.1 release build two seconds after the job was marked failed, once it had already pushed `runner:main`, `latest` and the commit tag but before the job image was built. The build log simply stops, because the process that would have explained it was gone. Reporting now fails the job it belongs to and goes to stderr, exactly as a failed poll already does (`run.rs:194`). A join failure stays fatal: it means the task never produced a result at all, which is a different class of problem. `finish_job` moved into its own module so the tests fit under the 250-line file limit. Four of them: a control-plane failure and an unacknowledged final log both leave the agent running, a clean finish is not an error, and an aborted task still ends it. Ran locally: `cargo fmt --check`, `cargo clippy --workspace --all-targets --all-features -- -D warnings`, `cargo test -p syncode-runner-agent` (10 passed), `scripts/check-architecture.sh`, `scripts/check-rust-loc.sh`. Note the other half: `UpdateLog` answering 500 with `AlreadyExists` is a forge bug and still needs its own fix. This change makes it survivable rather than fatal.

day01 added 1 commit 2026-08-04 06:31:46 +00:00

fix: keep the daemon alive when a job report fails

		A rejected log update ended the whole runner. `finish_job` propagated
`ReporterError` out of the agent loop, so `run()` returned `Err`, the process
exited and compose restarted it — taking every other job on the runner with it.

On the production runner this happened three times on 2026-08-03 and killed the
0.2.1 release build after it had already pushed the runner image.

Reporting now fails the job it belongs to and is written to stderr, the way a
failed poll already is. A join failure stays fatal: it means the task itself
never produced a result.