Skip to main content
A run is one execution. Everything else — metrics, files, spans, code — hangs off it.

​
Opening one

The CLI never creates. probe run start opens a run inside an experiment that already exists, or directly under a project. A slug that is a near-miss of an existing one is refused, not created — on the CLI the slug is hand-typed every time, and that is where typos come from. Use probe project create / probe experiment create explicitly.The SDK does create, because a training script should not fail on a missing row.

​
Naming

Omit --name and the server names the run after its slug, or after the petname it mints when there is none — then replaces that with a generated title once the run reaches a terminal status. A name you supply is yours and is never overwritten.

​
Project-direct runs

Omit --experiment entirely. The run hangs off the project with no experiment in between. That is the honest home for work that has no hypothesis, and it is the shape a Weights & Biases import lands in.

​
Relationships

child

A run descended from this one: fork, resume, retry or branch. The relation is required.

fork

A new run continuing the source from a checkpoint step. The source stays untouched.

group

A peer in a sweep, ensemble or distributed job.
A run that consumed another’s output — an eval of its checkpoint, an SFT run over its rollouts — is not its child. Record that as a use (probe checkpoint use ... --role eval, see datasets) or a lineage edge (probe edge add --relation evaluates_on).

​
Wrapping a command

probe exec opens the run itself, holds it with a heartbeat for as long as the child lives, and closes it from the child’s real exit code. It hands the child PROBE_RUN_ID and PROBE_RUN_EPOCH, and probe.init() inside the job reads them and joins that same run — so wrapping a script that already calls probe.init() produces one run, not two.
sbatch, ray job submit and modal deploy are detected. Those commands return as soon as the job is submitted, so recording the submitter’s exit code as the job’s outcome would be a lie. Probe opens the run awaiting attach instead, and the job claims it when it starts.

​
Closing one

probe run end is the delivery barrier: it blocks until everything queued for that run has actually been delivered, unless you pass --async. That is what makes “the run is closed” mean “the data is there”. From the SDK, probe.finish() does the same. A script that exits without calling it is still closed at exit as completed, failed or canceled.

​
When the process dies

SDK-created runs heartbeat every 60 seconds, so the server can reap a run whose process vanished rather than leaving it running forever. Tune with PROBE_HEARTBEAT_SECONDS; <=0 disables it. When a run crashes or fails, Probe emails a postmortem: a timeline of what the run’s telemetry showed before it broke — step time, GPU temperature, loss spikes, memory pressure, learning-rate trouble — where in the execution it stopped (training/epoch-7 > step 41300) and the signal that ended it (killed by SIGKILL).

​
Reading runs back

​
Amending

A description says what the run is. --notes says what a later reader should distrust about it, and is usually learned after the run finished; it replaces the run’s notes (see notes to append instead). A run has no Overview page of its own: its experiment’s summary is the write-up. --status corrects a finished run’s record and keeps its start and end times; failed emails the run’s launcher. To close a run that is still running, use probe run end. probe link attaches foreign keys under metadata.foreign_keys — how a Probe run stays joinable to the scheduler, tracker or ticket it came from.

​
Relaunching from a checkpoint

Rewind discards the incumbent run’s record from that step onward and rewrites it, unlocking a completed incumbent to do so. Without the flag nothing is ever deleted — a re-push against an existing external id is rejected rather than allowed to overwrite history quietly.