The control plane compiles a workflow, keeps the run as an aggregate advanced
by commands, and pushes the versioned plan down a bidirectional gRPC stream.
The node says hello and is given work; nothing polls, and a report only counts
from the node that holds the run.
The stream handler converted every failure at its call site. It now has one
error type with From impls and a single conversion to Status at the boundary.
control-domain said nothing about its contents. The runs the control plane
holds, the aggregate that decides about them and the queue in front of them are
one crate; the gRPC stream a node opens is the other.
dtolnay/rust-toolchain publishes versions as branches whose heads move, so a
commit pinned today is unreachable from any ref once the branch advances and a
fresh mirror no longer carries it. The branch is what a mirror can serve.
The runner on production cannot resolve an action reference named like a
version, so dtolnay/rust-toolchain is unusable there until it ships the fix.
The toolchain is pinned by image digest instead, which the job container
resolves without any action at all.
A run is handed out under a lease with a term and a grant number. Everything
the node says about that run states the grant, and a message stating an older
one is refused, so a report from a node that has already lost the run cannot
land. A lease that stops being renewed is reclaimed and the run goes back in the
queue under a new grant.
Heartbeat renews every lease it names; the runs it does not name lapse.
A node exchanges an enrolment token for an identity and a short-lived
credential. The token is spent on use, rotation replaces the way in rather than
adding one, and a revoked node is refused on the next thing it says. Silence
past the deadline marks a node offline.
Capabilities are typed — architecture, system, container runtime and version,
cores, memory — with labels kept beside them as the compatibility projection.
A node that has not said what it can do, or how much room it has, cannot be
given work.
A session now starts with enrolment: the node spends a token, gets an identity
and a credential, and opens the session with it. Hello carries typed
capabilities and capacity, so a node that has not said what it can do or how
much room it has cannot be given anything. Heartbeat refreshes capacity as well
as leases, which is what makes the answer current rather than what was true at
registration.
A node below the free space floor is passed over instead of being handed a job
that runs out of disk halfway through, and it is told the reason rather than
left waiting for work that will never arrive. The same sentence answers an
operator asking why a job is not starting.
The aggregate expired leases correctly and nothing ever asked it to. A sweep now
runs on a timer: nodes that stopped speaking are marked offline once, and runs
whose lease lapsed go back in the queue under a new grant.
Triggering compiles the workflow once, expands its matrix and queues a run per
combination, each carrying its own plan. A workflow that does not compile is
refused when the run is triggered and queues nothing, instead of failing halfway
through a job.
Triggering read a workflow and queued it. It now asks the workflow whether it
wants this event first: a workflow that does not declare it is neither an error
nor a run, and queues nothing. What the workflow does declare still becomes one
run per matrix combination.
A run has to be compiled from what a commit says, not from what the branch says
by the time the control plane gets round to it, so the contents are read at a
stated commit. Both directories the dialect allows are read, in the order the
forge reads them.
A repository without workflows is not a failure; a forge that refuses is, and
says so rather than passing for a repository with nothing in it.
A workflow that calls another contributes the called workflow's runs, resolved
from the same commit, so the whole run is compiled from one state of the
repository. A cycle is refused, a call this commit does not contain is refused,
and calling another repository is refused with the reason rather than treated as
missing.
Inputs and secrets are not passed yet, so a call that states them is refused.
Running the called workflow without them would run the wrong thing quietly.
The pieces that turn an event into runs were all here and none of them was
connected: trigger, the workflow sources and the reusable resolver lived in the
library with their own tests, while the binary started a gRPC server over an
empty registry. Nothing in a running process ever called queue, so a node could
enrol, say hello and wait forever for work that had no way to exist.
The forge now delivers events over a signed endpoint. A push is read into the
shape deciding needs, the workflows at that commit are fetched, and every
workflow that declares the event is compiled into runs. A node holding a session
is handed the plan the push produced.
Deliveries the control plane does not act on — an event it does not model, a
deleted branch, a reference that is not a branch — are accepted rather than
refused, because answering an error would make the forge retry something that
can never succeed. A workflow that declares the event and does not compile stops
the intake instead: a run that silently fails to exist is worse than a delivery
the forge sees refused.
The test drives the whole path the way production does, over both protocols: a
signed push arrives on the endpoint, and a node that enrols against the same
control plane receives the compiled plan.
The control plane held every run in memory. A restart lost the queue, the
leases and the fences with it, which is worse than losing work: a node that
reconnected still believed it held a run, and the control plane had no grant to
recognise the report against. The one mechanism that stops a job running twice
only works if the grant outlives the process that issued it.
The aggregate was already event sourced — it decides, produces an event and
appends it to a log with a monotonic sequence — so the run's history was
already the truth and only had nowhere to live. It lives in Postgres now.
Deciding and recording are separated so a decision can be written down before
it is believed. A run that took an event in memory and failed to store it would
hand out a lease nobody could later prove. Because the write happens under the
same lock as the decision, the registry became async, and that reaches through
trigger, intake, the sweep and the node session.
The schema has no column for a run's state. State is what replaying the events
produces, so there is nothing stored that could disagree with them. A run and
the event that opened it are written in one transaction: a run with no history
is not something anything could rebuild.
RunLog is a port in the domain and Postgres implements it in a crate of its
own, so the aggregate is still exercised without a database and the boundary
check still holds. Forgotten is the in-memory log tests drive; it is nowhere
near the binary, because forgetting every lease on restart should be a choice
and not something arrived at by omission.
Runs survived a restart; the nodes doing them did not. Every identity, every
credential and every unspent enrolment token lived in memory, so a restart
locked out nodes that had done nothing wrong: they held a credential the
control plane no longer recognised, and the only way back in was a token an
operator had to issue by hand.
A node is not a history the way a run is — it is a current fact: who it is,
what it may do, and when it was last heard from. So this stores state and
overwrites it rather than appending events, and the crate says why.
Every change is written down before the registry hands the result back. A node
is changed on a copy, the copy is stored, and only then does it become what the
registry believes; a change the store refused never happened. That makes the
node registry async too, which reaches through the session and the sweep.
Unspent enrolment tokens are stored as well. A token an operator handed out and
a restart quietly invalidated would be indistinguishable, to whoever holds it,
from a token that never worked. A spent one stays spent: were it to come back,
a leaked token would enrol a second node after every restart.
Two gaps left the control plane unable to stand in for the forge.
Log lines were parsed and dropped, so a run was unobservable: the aggregate
knew a job had finished and nobody could see what it did. They are stored now,
keyed by the offset the node reported, which makes a resent batch the same fact
twice rather than new output. A node retrying after a broken connection no
longer doubles the log.
Only pushes triggered runs. Pull requests do now, filtered by the branch they
are aimed at and compiled at the head commit, with closing a pull request
starting nothing.
That second part carried a trap worth naming. A path filter with nothing to
admit does not fire, and a pull request webhook does not say which files it
touches — so a workflow filtering on `paths:` would have silently never run,
with nothing anywhere saying why. The files are asked for through a port of
their own instead of being assumed empty.
Output travels beside the aggregate rather than through it: a job that prints
nothing is not less valid, and a line arriving late does not move the run
anywhere.
A node cannot join a control plane it has no token for, and there was no way to
get one: issue_token was reachable from code and from tests, and from nothing an
operator can run. The control plane was therefore impossible to enrol against,
which made the whole node protocol unreachable in practice.
The token is stored before it is printed, so one that reaches an operator is one
the control plane will honour. The scope is stated as an operator thinks of it —
instance, organisation:<name>, repository:<owner>/<name> — and parsing that
shape lives in the binary rather than the domain, because it is a convenience of
the command line and not something the control plane means.
There was no way to run this anywhere but from a checkout: no Dockerfile, and
the only workflow was the one that validates a pull request. A control plane
that cannot be deployed cannot replace anything.
The image follows the runner's: a validation stage, a release build, and a slim
runtime carrying only the binary and the certificates it needs. It is tagged by
the digest of the sources it was built from, so rebuilding the same tree reuses
the published image rather than producing a second one differing only in
timestamps.
The test suite is deliberately not part of the validation stage. Its store tests
refuse to pass without a Postgres to run against and an image build has none;
running the rest and calling that validation would say "verified" about exactly
the part that was not. The full suite, database included, runs in
pull-request.yml.
The binary listens on the loopback by default, which is right on someone's own
machine and useless in a container, where the network is the only way in. The
image therefore states every interface in its command, and the container's
isolation is what keeps that private.
A node credential lasts fifteen minutes and nothing ever replaced it.
`Nodes::rotate` existed but had no caller and no message to carry its
result, so every node was locked out a quarter of an hour after enrolling
and only a fresh enrolment token got it back. That is what took dev down.
The control plane now pushes `Rotated` on the open session whenever the
node speaks and the credential it holds is within the renewal margin of
expiry. It is pushed rather than requested, and offered again on every
heartbeat, so a lost message costs nothing.
An enrolled node that never spoke now settles as offline as well. It was
reported as newly gone on every sweep forever, which is what buried the
real failure in the log.
A run opened before the control plane wrote down where a run came from has
no origin, cannot be read back, and took the whole service down with it: the
restore refused the row, the process exited, and it did so again on the next
start and every one after. One unusable row was an outage with no way out but
editing the database by hand. Dev spent this morning that way.
Applying the schema now drops such a run and makes both columns mandatory, so
no later one can look like it. Inventing an origin is still off the table --
what changed is the cost of refusing. A run with no origin can never be
assigned, because a node cannot be told what it is building, so keeping it
bought nothing and cost everything.
With the columns mandatory there is no longer a row shaped like that to read,
so the case and its error go away.
The runner and this service must name the same revision, because a plan
compiled here is executed there. This is the revision that reads `on.schedule`
as the sequence it is written as.
Deciding which runs an event asks for is a function of the workflow files at a
commit and the event itself. It needs no database, no node and no deployment,
so it is measured here rather than by standing a second control plane beside
the forge and waiting for pushes to arrive: the runs arise in memory, through
the same `trigger` the service uses, and are thrown away once counted.
Against the five SynCode repositories, three agree with the forge exactly and
two produce nothing at all, both for one reason: an execution plan holds a
single job, and their workflows have four and five. That is now a measurement
rather than a suspicion, and it is the only divergence there is.
A run also names the workflow file it was compiled from. One commit can answer
one event with several workflows, so without the file there is nothing to tell
their runs apart, and nothing to line any of them up against on the forge. An
origin that cannot name its file is dropped when the schema is applied, for the
same reason an origin-less run is.
A workflow is as many plans as it has jobs, so it is as many runs. Until now
anything with a second job was refused outright and produced nothing at all.
Measured against the five SynCode repositories, at the level of the jobs a node
would actually be handed, both sides now decide the same work: five jobs for
the runner, four for the control plane, one each for meta and the forge, none
where none. The comparison reads those jobs off the queue rather than off the
compilation, so what is counted is what would have been offered.
What this does not yet do is order them. Every plan carries the `needs` its job
declared, and nothing reads them: the queue still offers whatever is at the
front. So this control plane must not be pointed at a forge until it does, or
it would start a job before what it waits for has finished.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
nauron
Promote the verified SynCode 0.3 implementation to the release branch.
Approved for the verified SynCode 0.3.0 release.