Designing a Durable Scheduler

The unit that must survive

Most agent frameworks make the conversation, or the agent loop, the primary object. That works until a process dies. Once it restarts, everything you were counting on — the half-finished task, the leases, the ordering — is gone. In ARGUS 里我们做了一个不同的决定：持久的单位是 project，而不是 conversation。

A project has an identity, a queue of tasks, and a status. An agent is a worker that drains the queue. If the worker disappears, the project remains, and a new worker can resume exactly where the old one stopped. That reframing is the whole design.

The restore path

On boot, the scheduler loads every project that is not terminal, re-leases its in-flight tasks, and re-enqueues anything whose lease lapsed. The critical invariant is that a lease is a claim, not a reservation: if the holder never renews it, the work becomes eligible for someone else.

export async function restore(projects: Project[]): Promise<void> {
  for (const project of projects) {
    if (isTerminal(project.status)) continue;
    const lapsed = await claimStore.findLapsedLeases(project.id, now());
    for (const task of lapsed) {
      await queue.enqueue({ taskId: task.id, key: task.idempotencyKey });
    }
    await scheduler.resume(project.id);
  }
}

The enqueue is keyed by task.idempotencyKey, so a restore that runs twice is safe. The second run finds no lapsed leases and enqueues nothing.

Measured before and after

We ran a restore against a mixed backlog after killing the worker mid-drain. These are internal harness numbers, not production SLAs.

How the pieces fit

A coarse view of the runtime, with the queue deliberately kept out of the source-of-truth role.

  API ──▶ Store (projects, tasks, claims)   ← source of truth
              │
              ▼
        Scheduler ──▶ Queue (Redis) ──▶ Workers
              ▲                              │
              └────── lease renew / heartbeat

Timeline

What I’d keep, what I’d change

Durable projects are the right primitive and I’d keep them without hesitation. The lease model took two rewrites to get right; a counter from the start would have saved weeks of chasing a bug that only appeared under load and skew. The counterargument — that a single per-process timer is enough — holds only while you run one node. The moment you run two, the lease has to be a shared, monotonic fact.