lea · Agents at work

CPUs, heavy jobs and host pressure

CPUs #

Test suites, static analysis and builds start one worker per CPU they see. lea gives each agent a window of LEA_AGENT_CPUS CPUs (4), the ones fewest agents share. The agent, its tools and the containers it starts in the nested Podman run there: nproc, PHPStan, php-cs-fixer, Go, pytest-xdist and Node's os.availableParallelism() see that many CPUs, without anything in your projects. (lea runs the nested Podman's crun through a wrapper that pins an agent's containers, podman exec included.) Tools that count /proc/cpuinfo still see every CPU. The window is not a hard limit: the lea container's LEA_CPUS is. lea doctor lists the windows.

Heavy jobs #

Test suites, static analysis, builds and browsers start one worker per core of the host: a few agents running them at once take every core. Agents run them through lea heavy:

lea heavy -- vendor/bin/phpstan analyse
lea heavy -- go test ./...

lea heavy waits for one of the container's LEA_HEAVY_SLOTS slots, in order, saying who holds them, then runs the command on LEA_HEAVY_WORKERS CPUs of its own (tools that count the CPUs they may use, such as nproc, see that many) with GOMAXPROCS, MAKEFLAGS=-jN, CMAKE_BUILD_PARALLEL_LEVEL, CARGO_BUILD_JOBS, PYTEST_XDIST_AUTO_NUM_WORKERS and LEA_HEAVY_WORKERS set. A slot whose job was killed frees itself within three minutes; a job holds its slot two hours at most. lea board lists the jobs that run and wait.

Host pressure #

The lea daemon samples the host and each lea container every 5 seconds: PSI (cpu some, memory full, io full, avg10), the memory in use against the container's limit (page cache the kernel drops first left out), and the host's load. It reads /proc and the container's cgroup v2, found from its process, so it works the same with Docker and rootless Podman.

Above a threshold a lead's agents_spawn is refused with the reason, for example under pressure (container cpu PSI 76% > 70%, memory 92% of 24.0 GiB > 85%); 14 agents run, 2 heavy slots busy. With queue: true the spawn waits instead, up to 30 minutes, and the lead gets a message when it starts; one calm sample starts one queued agent at most. The queue lives in the daemon's memory: a daemon restart drops it without a message. agents_request_spawn is not refused: a request starts nothing, the lead's agents_spawn is the gate. Meanwhile each container gets half its heavy slots (at least one): jobs that run keep theirs, new ones wait.

When a container (or the host) stays critical for LEA_SHED_AFTER seconds (30: memory at 95 % of the limit, memory PSI 30 %, or cpu PSI 80 %), lea pauses its lowest-priority agent together with the containers it started in the nested Podman (SIGSTOP: the nested Podman has no cgroup freezer). A frozen process keeps its memory, so when memory is what is critical lea first stops those containers, and starts them again when it resumes the agent. It pauses another one every LEA_SHED_AFTER only while each pause makes the reading better, and at most half the agents it may pause: no cascade when what weighs is elsewhere. Idle agents go first, then the newest dev, reviewers after devs; never a lead, never an agent that holds a heavy slot. A busy agent is paused only once no idle one is left: it carries on where it stopped when resumed. Its lead gets a message, and lea board lists it under "Paused under pressure". Once the pressure stays under the admission thresholds for LEA_SHED_RESUME seconds (60), lea resumes them one by one, the last paused first; one you resumed yourself (the deck's p) is left alone. The list survives a daemon restart.

lea doctor checks that PSI is readable, that each running lea container's cgroup is, and that the shedding thresholds sit above the admission ones.

lea board shows the host's gauge and each container's (CPU 14.8/16 · PSI 22% · mem 18.1/24.0 GiB · heavy 3/4 (+2 queued)); the deck shows the container's on its last line, amber near a threshold, red over one.