Eliminating Zombie Internal Jobs with Heartbeats and Run Leases

Why “zombie” internal jobs happen in modern workflow engines

A “zombie” internal job is a workflow step that is no longer making progress but still appears active to the scheduler. It can hold locks, keep downstream steps waiting, consume worker slots, and distort operational metrics. These are rarely caused by a single bug; they emerge from the messy edges of distributed systems: transient network partitions, worker node restarts, process OOM kills, container evictions, dependency timeouts, or external services that never return.

Teams often discover zombies only after user-facing symptoms (stale dashboards, missing exports, delayed approvals) or after someone notices unusually long runtimes. The fix is usually manual: cancel and rerun, or patch a one-off. That approach doesn’t scale once you have dozens of DAGs, mixed languages, and a blend of scheduled and event-driven triggers.

A deterministic solution combines two primitives: heartbeats (proof of life/progress) and run-leases (a time-bounded right to execute). Together they let you detect stuck steps, quarantine them to prevent further damage, and auto-heal safely without turning every glitch into duplicate work.

The deterministic pattern in one sentence

Each running step must periodically renew a lease and emit a heartbeat; if renewal or heartbeats stop beyond a known threshold, the engine marks the step as suspect, quarantines it, and either reclaims the lease for a retry or escalates with enough state to recover idempotently.

Core building blocks

1) A run-lease with an explicit owner

A run-lease is a record in a durable store (typically the workflow database) that says: “Run R, step S is owned by worker W until time T.” The critical properties are:

• Time-bounded: ownership expires if not renewed.
• Exclusive: only one active owner at a time (enforced with unique constraints or conditional updates).
• Renewable: the worker extends T while it is healthy and progressing.

This solves the classic problem where a crashed worker leaves the scheduler uncertain whether it should re-run a step. With leases, the scheduler has a deterministic rule: if the lease expired, the step is eligible to be reclaimed.

2) Heartbeats that signal progress, not just liveness

A naive heartbeat (“I’m alive”) can keep a zombie looking healthy even when it is stuck in a tight loop or blocked on an I/O call. Better heartbeats include progress signals:

• Step phase (e.g., “fetching”, “transforming”, “writing”).
• Monotonic counters (rows processed, bytes uploaded, pages fetched).
• Last external checkpoint (e.g., last message offset, last file chunk).
• Optional span/trace correlation IDs for observability.

Now you can detect not just “silent” failures, but also “loud” stalls where a worker is alive yet not moving.

3) A clear stuckness model with two thresholds

To avoid flapping, define two time thresholds per step type:

• Heartbeat timeout: if no heartbeat arrives within this window, the step becomes suspect.
• Lease expiration: if the lease is not renewed, the step becomes reclaimable.

In practice, you often set the heartbeat timeout shorter than lease duration. That lets you quarantine quickly (prevent downstream harm) while still leaving a buffer before you retry.

Detection algorithm that stays deterministic

The scheduler/monitor loop can be simple and reliable:

1. Read active steps with their lease expiry and last heartbeat timestamps.
2. Mark suspect when now - last_heartbeat > heartbeat_timeout.
3. Quarantine suspect steps by preventing them from scheduling dependents and by pausing side-effecting actions (see below).
4. Reclaim when now > lease_expiry using a conditional update: only the scheduler can assign a new owner if the lease is expired.
5. Retry or escalate depending on retry policy, idempotency guarantees, and failure history.

The determinism comes from using time and state stored in one place (lease + heartbeat records) rather than inferring health from logs or worker connectivity.

Quarantining without causing more damage

Quarantine is the difference between “we detected a zombie” and “we made it safe.” The goal is to stop propagation while preserving forensic context.

• Block downstream scheduling: dependents remain pending until the quarantined step is resolved.
• Freeze external side effects where possible: for example, pause writes by gating them behind a “still owner” check (the step must prove it holds the current lease before committing).
• Preserve artifacts: last progress heartbeat, last log offset, retry count, and any trace IDs.

This is also where code-defined DAGs shine because you can standardize policies (timeouts, retries, compensation) across workflows rather than re-implementing them per cron job. If your team is moving from ad-hoc schedules to explicit graphs, the approach aligns well with the discipline described in Migrating Cron Sprawl to Code-Defined DAGs With OpenTelemetry Traceability.

Auto-healing strategies that won’t duplicate work

Once a step is quarantined and its lease is reclaimable, auto-healing typically follows one of three paths.

1) Safe retry with idempotency keys

If the step is designed to be idempotent, the scheduler can rerun it immediately. Typical techniques:

• Write results to a content-addressed path (hash-based) then atomically “publish.”
• Use database upserts keyed by (run_id, step_id).
• Include an idempotency key in external API calls.

The key is that the step can be executed twice without producing two side effects.

2) Resume from a checkpoint

For long-running work (large exports, batch transforms), heartbeats should carry checkpoint state (e.g., “processed through row 8,200,000”). A retried step then resumes instead of restarting. This reduces cost and makes the system tolerant of spot interruptions and rolling deploys.

3) Compensate then retry

When idempotency isn’t possible, define compensation steps (e.g., delete the partially created artifact, revert a status transition) before retrying. This requires careful modeling, but it is still more reliable than manual cleanup under pressure.

Practical implementation details teams miss

Lease renewal should be conditional

Workers should renew only if they are still the owner. Use compare-and-swap semantics: “extend lease where owner=W and lease_id matches.” This prevents a stale worker from “resurrecting” itself after the scheduler has reassigned the step.

Heartbeats should be cheap and independent of logging

Don’t rely on log volume as a proxy for heartbeat. Heartbeats should be a small write on a fixed cadence, ideally resilient to log pipeline backpressure.

Time budget must account for queueing

A step can look “silent” if it’s queued rather than running. Track states separately: queued, starting, running. Apply heartbeat expectations only once the worker has actually begun executing.

Where Windmill fits naturally in this model

In internal automation, it helps when the workflow engine already treats execution as a first-class, observable unit: runs, steps, logs, retries, and scheduling rules. A code-first platform like windmill.dev is a natural home for the heartbeat + lease pattern because workflows are explicit DAGs, scripts are real code, and operational signals (logs, alerts, telemetry exports) can be standardized across teams instead of reinvented in each service.

As you implement stuck-step controls, align them with your broader state discipline. If you also build internal UIs around long-running processes, the same “explicit state + transitions” approach carries through to the frontend, similar to the idea in State Machine UI Pattern for No-Code Apps That Scale Multi-Step Journeys.

A minimal checklist to eliminate zombies

• Every running step holds a renewable lease with a single owner.
• Every step emits progress heartbeats on a predictable cadence.
• Two thresholds exist: heartbeat timeout (suspect) and lease expiry (reclaim).
• Quarantine blocks downstream work and gates side effects behind lease ownership checks.
• Auto-heal uses idempotent retry, checkpoint resume, or compensation.
• All events (lease changes, quarantine, retries) are observable and alertable.

With these in place, “zombie jobs” stop being a mysterious operational tax and become a bounded, testable failure mode.