←All posts

A Genealogy of Task Scheduling — From FIFO to Fair Queueing and LLM Inference

The right answer to “what should run next?” depends on whether mean delay, responsiveness, deadlines, or fairness comes first. This guide traces FCFS, SJF, Round Robin, EDF, CFS, WFQ, SFQ, DRR, and LLM-oriented VTC by objective, mechanism, and assumptions, and shows with a simulator why SFQ suits LLM request admission when output length is unknown.

AlgorithmsSchedulingOperating SystemsLLMInteractive

There is no single correct answer to “what should run next?” for a shared queue. The right algorithm depends on whether the objective is to reduce mean waiting time, respond to interaction quickly, meet deadlines, or divide resources fairly among users.

CPU scheduling, network fair queueing, and LLM inference serving can be traced as one genealogy by asking what each algorithm tries to optimize and what it changes from its predecessors. For the order in which LLM requests are admitted into a server pool, the “select, then charge” structure of Start-time Fair Queueing (SFQ) fits well: even when output length is unknown, a router can admit a request first and charge its estimated cost afterward. VTC (Virtual Token Counter), used inside an inference engine, differs by charging tokens as they are actually generated.

Watch the video explainer4:13Synthesized narration · burned-in captions
Show transcript

For the same queue, there is no single correct answer to what should run next. From CPU scheduling to networks and LLM inference, let's trace how scheduling branches by goal.

The first question is what “good” means: shorter averages, faster first response, or something else. Meeting deadlines, sharing fairly, using all capacity, and lowering P99 often pull in different directions.

FCFS runs jobs in arrival order until completion, but short jobs can pile up behind a long one: the convoy effect. SJF chooses the shortest known job first and minimizes mean waiting time when all jobs are present at once. SRTF switches to the shortest remaining job; ideally it minimizes mean flow time, but long jobs can starve. Round Robin lets every task run soon, at the cost of switching overhead; MLFQ infers short, interactive work from past behavior.

On the deadline branch, EDF selects the ready job with the earliest absolute deadline. With independent jobs, one preemptive CPU, and negligible switching cost, if any schedule can meet the deadlines, EDF can too. RM gives shorter-period tasks higher fixed priority. This branch is about feasibility, not equal service.

In the proportional-share branch, schedulers favor the principal with the least normalized service so far. Lottery uses randomness; Stride advances a deterministic pass more slowly for larger weights, so those tasks are picked more often. As of 2026, mainline Linux uses EEVDF for normal tasks. Among tasks not ahead of their fair share, it picks the earliest virtual deadline.

Networks add a twist: once a packet starts transmitting, it normally cannot be split and interleaved. GPS, Generalized Processor Sharing, is an ideal fluid model serving backlogged flows simultaneously in proportion to their weights. WFQ compares virtual finish tags, while Start-time Fair Queueing, SFQ, compares start tags. That timing shift matters for LLM routers. DRR skips tags: it visits flows in turn, grants a quantum, carries unused credit as deficit, and aims for constant time per packet.

For LLMs, input length is known, but output length is not final until generation ends. max_tokens is only a cap, so SJF or WFQ would need a length predictor. Prefill is parallel and compute-heavy; decode generates one token at a time while reading the KV cache. So a request can be indivisible at the router, yet engine continuous batching can regroup it at every decode iteration.

In front of unmodified servers, SFQ with per-tenant FIFO queues and meters is a simple router policy. Select the tenant with the smallest meter. The current request's cost never decides whether it passes now. It only affects how long that tenant waits for its next turn. When an idle tenant returns, its meter is raised to at least the current baseline. Unused share cannot be banked. The guarantee is admission fairness over estimated cost, not measured GPU time.

If you control the engine, VTC, the Virtual Token Counter from OSDI 2024, gives a different answer. VTC charges each client's counter for input tokens. It adds actual output tokens every decode iteration, and favors the client with the smallest counter. When prefix locality also matters, DLPM comes from a 2025 preprint. It favors prefix matches while a per-client quantum lasts, relaxing fair order in a controlled way. The 2025 Equinox preprint argues that one token counter cannot capture latency, throughput, and GPU utilization.

The article includes interactive CPU and LLM demos, so you can run the policies yourself and see the differences.

1. Define a good schedule first

Represent task ii by arrival time aia_i, processing demand pip_i, completion time CiC_i, deadline did_i, and weight wiw_i. Common objectives include:

ObjectiveMetricTypical question
Finish work earlyTurnaround Ci−aiC_i-a_i, mean flow timeHow can jobs finish sooner on average?
React earlyResponse time (first run time −ai-a_i)How can an interactive request respond immediately?
Meet deadlinesDeadline misses, maximum latenessCan control, audio, or video work finish on time?
Divide fairlyDifference in normalized service, max-min fairnessCan a noisy neighbor be isolated?
Use all capacityThroughput, utilization, work conservationIs usable capacity being left idle?
Be predictableTail latency, maximum wait, jitterCan P99 or a worst case be bounded?

These objectives often conflict. Prioritizing short jobs reduces mean wait but can starve long jobs. Switching in small increments improves responsiveness but adds context switches and hurts cache locality.

Before choosing an algorithm, settle at least the following assumptions:

  • Preemptibility: can work stop and resume partway through?
  • Size information: is pip_i exact, estimated, or unknown until completion?
  • Resource dimensionality: is CPU the only resource, or do GPU memory, bandwidth, and KV cache contend too?
  • Fairness principal: is fairness among processes, users, organizations, API keys, or models?
  • Decision layer: is the decision admission, replica placement, dispatch order, or an in-flight token iteration?

2. Genealogy: branches by objective

The following diagram groups the algorithms by objective and selection logic. It does not show a chronology or a strict inheritance tree. Solid edges connect related designs by classification or contrast. Dotted edges show Linux's adoption transition or DLPM relaxing VTC's strict order for locality.

Newer algorithms do not dominate older ones; which branch fits depends on the objective and constraints. DRR's amortized O(1)O(1) work is attractive for tens of thousands of flows at line rate. A gateway with a few dozen LLM tenants can afford a tag-based algorithm with O(log⁡n)O(\log n) selection.

3. The completion-time and responsiveness branch

3.1 FCFS / FIFO: arrival order is easy to explain

First-Come, First-Served (FCFS) selects the head of an arrival-ordered queue and normally runs it to completion.

ready = queue ordered by arrival
if the CPU is free:
    running = ready.pop_front()
run running until completion

Each enqueue/dequeue and next-task decision is O(1)O(1), deterministic, and easy to explain; constructing the full schedule for nn jobs is O(n)O(n). Its weakness is the convoy effect: when one long job is followed by many short jobs, every short job waits behind the long one. Mean waiting time rises, and interactive work responds later.

3.2 SJF / SRTF: prefer short work when size is known

Shortest Job First (SJF) chooses the arrived task with minimum pip_i whenever the CPU becomes free. Under the classical assumptions that all jobs are available together, lengths are known, there is one CPU, and execution is non-preemptive, SJF minimizes mean waiting time.

Shortest Remaining Time First (SRTF, also called SRPT or STCF) is the preemptive counterpart. At each decision point it selects the task with the smallest remaining time rir_i. For jobs with known release times and processing times on one unit-speed machine, with cost-free preemption, SRTF minimizes total—and therefore mean—flow time. It does so at a cost:

  • long tasks can starve while short tasks keep arriving;
  • the policy is sensitive to size-prediction error;
  • every preemption adds state-save and restore work;
  • a user benefits by declaring a job shorter, so untrusted self-reported size is unsafe.

3.3 Round Robin: prioritize “run once soon” over completion order

Round Robin (RR) gives each task at most a time quantum qq, then moves unfinished work to the tail.

while ready is not empty:
    task = ready.pop_front()
    run min(q, task.remaining)
    if task is unfinished:
        ready.push_back(task)

As q→∞q\to\infty, RR approaches FCFS. Reducing qq lets every runnable task respond sooner, but making it too small increases context-switch and cache-miss costs on real hardware. RR equalizes CPU time. A request-count round robin, discussed later, does not equalize resources when one request can be much heavier than another.

3.4 MLFQ: infer short and interactive work without knowing the future

A Multi-Level Feedback Queue (MLFQ) maintains several priority queues. New tasks start at high priority; tasks that consume their CPU allotment are demoted; interactive tasks that block or yield quickly remain high. Periodically moving every task back to the top—a global priority boost—limits starvation. Aging, which raises an individual task's priority as it waits, is another related mechanism.

MLFQ effectively replaces the “predict total run time” input that SJF requires with online inference from observed behavior. If the rules reward yielding just before a quantum expires, however, a task can game the policy, so implementations account for cumulative CPU usage and related behavior.

Whereas MLFQ adjusts priorities from observed behavior, fixed-priority scheduling always chooses the highest-priority runnable task. In a simple implementation, low-priority tasks can starve, so aging or bounded priority boosts are needed. Priority inversion is a different failure mode: a low-priority task holding a shared lock can block a high-priority task while medium-priority work preempts the lock holder. Priority inheritance temporarily boosts the lock holder to the waiting task's priority. Priority-ceiling protocols instead use each resource's ceiling priority to restrict or raise lock holders, limiting inversion and deadlock.

4. The deadline and proportional-share branches

4.1 EDF and Rate Monotonic: deadline feasibility before fairness

Earliest Deadline First (EDF) selects the ready job with the earliest absolute deadline did_i and preempts when an earlier deadline arrives. Under the standard assumptions of independent jobs, one preemptive CPU, and negligible switching cost, EDF is feasibility-optimal: if any schedule can meet the deadlines, EDF can as well.

Rate Monotonic (RM) gives a shorter-period periodic task a higher fixed priority. Under the Liu–Layland assumptions of independent periodic tasks, implicit deadlines Di=TiD_i=T_i, one preemptive CPU, fixed worst-case execution time eie_i, and no blocking or switching overhead, the famous sufficient utilization condition for nn tasks is

U=∑i=1neiTi≤n(21/n−1).U=\sum_{i=1}^{n}\frac{e_i}{T_i}\le n\left(2^{1/n}-1\right).

In the same independent periodic-task, implicit-deadline model with idealized preemption and switching cost, EDF is schedulable exactly up to total utilization U≤1U\le1, i.e. 100% of one CPU.

EDF and RM reorder work to meet deadlines rather than to give everybody equal service.

4.2 Lottery / Stride: convert weights into CPU shares

Lottery scheduling gives each task tickets proportional to its weight and draws which task receives the next fixed-size allocation quantum. It provides proportional shares in long-run expectation but has short-term variance.

Stride scheduling makes the process deterministic. For a large constant LL, define

stride⁡i=Lwi.\operatorname{stride}_i=\frac{L}{w_i}.

Give each task a counter, pass. Choose the task with the smallest pass, grant the same-size quantum, then update pass += stride. If allocations vary in size, advance proportionally with pass += service_used × L / w_i. A high-weight task advances more slowly per unit service and is selected more often. The same skeleton—choose the principal with the least accumulated normalized service—reappears in CFS, SFQ, and VTC.

4.3 From CFS to EEVDF

Linux's Completely Fair Scheduler (CFS) approximates an ideal weighted CPU that runs all runnable tasks simultaneously. On a real CPU it selects the task with the smallest normalized execution time, vruntime. Running that task advances its vruntime, and eventually another task has the smallest vruntime and is selected.

Earliest Eligible Virtual Deadline First (EEVDF) was proposed in 1995, and Linux began moving to Peter Zijlstra's 2023 EEVDF design in version 6.6. As of 2026, the default selection rule for mainline Linux's normal fair class is therefore EEVDF, which keeps CFS's virtual-time approach, rather than CFS's “pick minimum vruntime” rule. EEVDF expresses how far a task is behind its fair share as lag; among tasks with lag >= 0, it chooses the earliest virtual deadline. Making eligibility and deadline explicit in the selection rule preserves proportional sharing while making latency-sensitive tasks with shorter requested slices easier to express.

5. The fair-queueing branch that grew up in networks

CPU time can be preempted finely. A network packet normally cannot be interleaved with another flow's packet once transmission begins. Research on dividing such indivisible work fairly among flows leads directly to LLM request scheduling.

5.1 GPS: an ideal model that defines fairness

Generalized Processor Sharing (GPS) treats every backlogged flow as infinitely divisible fluid and shares one link simultaneously in proportion to weights wiw_i. Real packet schedulers approximate GPS completion order or service discrepancy using indivisible packets.

5.2 WFQ: choose the smallest virtual finish time

Assign start and finish tags to packet kk from flow ii:

Sik=max⁡(V(aik),Fik−1),Fik=Sik+Likwi.S_i^k=\max\left(V(a_i^k),F_i^{k-1}\right),\qquad F_i^k=S_i^k+\frac{L_i^k}{w_i}.

Weighted Fair Queueing (WFQ) broadly selects the packet with the smallest FikF_i^k, approximating GPS completion order. Packet length LikL_i^k must be known before selection. WFQ computes the virtual time V(t)V(t) by simulating GPS.

5.3 SFQ: choose by start tag, not finish tag

Start-time Fair Queueing (SFQ) compares the FIFO head packet of every backlogged flow and selects the one with the smallest SikS_i^k. For the virtual time VV, it uses the start tag of the packet in service rather than simulating GPS. The selected packet's length does not decide whether that packet is selected now. Instead, it sets the finish tag—and therefore affects the next selection from that flow.

That shift in time suits an LLM router that knows request size only as an estimate. Let a request pass first, record its estimated cost on the tenant's meter, and let the other tenants catch up before selecting that tenant again—a pay-after, or deferred-charge, structure.

5.4 DRR: replace tag arithmetic with a quantum and deficit

Deficit Round Robin (DRR) keeps a deficit counter DiD_i for each flow and adds quantum QiQ_i whenever it visits an active flow. It repeatedly transmits head packets while they fit, subtracting each length, and carries an unused deficit to the next round only while the flow remains backlogged. When the queue drains, it resets Di=0D_i=0 so idle credit cannot accumulate. Weighted DRR uses Qi∝wiQ_i\propto w_i. If Qi≥Lmax⁡Q_i\ge L_{\max}, every visit can advance at least one packet, supporting the usual amortized-O(1)O(1) argument.

DRR hands out credit before use, whereas SFQ lets work pass first and records the debt after selection. A network MTU gives a stable basis for choosing a quantum. Maximum LLM request cost changes substantially with the model and workload, so a quantum is harder to choose.

5.5 DRF: consider multiple resources together

In a cluster, some jobs use mostly CPU and others mostly memory. Dominant Resource Fairness (DRF) takes each user's maximum allocation fraction across resources as its dominant share and favors users with smaller dominant shares. This branch defines max-min fairness over multiple resources instead of forcing every cost into one processing-time scalar.

6. Run the algorithms yourself

The lab below has two modes.

  1. CPU: change arrival time, burst length, and RR quantum; compare FCFS, SJF, SRTF, and RR timelines and delays.
  2. LLM: change every tenant's estimated cost, request count, and weight, plus tenant C's join time; compare the admission order under Global FIFO, Tenant Round Robin, Shortest request first, and SFQ.

In CPU mode, first observe short C waiting behind long A, then switch to SRTF. In LLM mode, set A's cost to 8 and B's to 2. Count admissions in the sequence under Tenant Round Robin and SFQ, then compare them with the normalized-cost bars for the common-backlog interval: Tenant Round Robin equalizes request counts, while SFQ equalizes normalized cost.

Scheduling Lab

Switch between a CPU timeline and LLM-router admission, tune parameters, and step through one stage at a time.

Runs the earliest arrival to completion. With the defaults, short C waits behind long A—the convoy effect.

Task A
Task B
Task C
Task D

Execution timeline

t = 0
A
B
C
D
015

Ready queue (waiting runnable tasks)

A

Remaining time

A
7
B
4
C
1
D
3
Initial state
No task ran in the preceding interval
1 / 16

Final metrics

Final metrics
TaskCompletionTurnaroundWaitingResponse
A7700
B111066
C121099
D151188
Average5.755.75

This teaching model uses one CPU, integer ticks, and zero switch cost. SJF/SRTF assume known bursts. Each stage shows the preceding one-tick transition and the resulting state.

The simulator is a discrete teaching model. In particular, LLM mode models only admission order into a server pool, not GPU execution time. Because finite queues eventually expose different total demand, observe fairness while both tenants remain backlogged rather than only at the final state.

In LLM mode, each tenant's configured requests arrive as one batch at its join time, and logical time advances by one per admission. While queues remain active, a boundary join occurs before that time's admission. If the preceding admission drained every queue, SFQ's idle reset occurs before a join at the same boundary. If all queues remain empty until a future arrival, time jumps there and reset / join appears as a separate non-admission stage. With no future arrival, the final reset is annotated on the last admission.

7. What is special about an LLM request?

LLM requests have properties that are missed if they are treated as differently sized HTTP requests.

7.1 Output length is unknown at selection time

Input token count is known, but output is not determined until generation terminates—by an end-of-sequence (EOS) token, an output cap, a stop sequence, cancellation, or error. max_tokens is a limit, not the generated length. Policies such as SJF and WFQ that rank the current job using an exact length cannot be applied directly without a predictor.

7.2 Prefill and decode have different cost structures

Prefill processes prompt tokens in parallel and tends to be compute-heavy. Decode generates tokens autoregressively, one iteration at a time, while repeatedly reading the KV cache. A token stresses different bottlenecks depending on phase, batch composition, sequence length, and hardware. Token count is a useful service proxy, not a synonym for GPU seconds.

7.3 A request is indivisible outside but iterative inside

After a load balancer dispatches a request, returning it safely and moving it to another server is generally expensive. Treating it as non-preemptive at the outer router is therefore natural.

Inside an engine, Orca proposed iteration-level scheduling with selective batching. In what is now commonly called continuous or in-flight batching, an engine can rebuild a batch at decode-iteration boundaries. Some implementations can also preempt with KV-cache swap or recomputation. Whether an LLM request can be preempted therefore depends on which layer's scheduler is making the decision.

7.4 Batching, KV cache, and prefix locality interact

Co-locating requests with a shared prefix can reuse KV cache. Strict fairness order can scatter requests from one tenant and destroy that locality. When the head request does not fit available KV blocks, skipping it improves work conservation but weakens the ability to explain order using a fairness counter alone.

7.5 Request count is the wrong fairness unit

Giving 100 short chats and 100 long generations equal request counts does not equalize resource consumption. If the fairness principal is the tenant, service needs at least a positive estimated cost such as

c(r)=αninput+βn^output.c(r)=\alpha n_{\text{input}}+\beta\widehat{n}_{\text{output}}.

If model, prefix-cache hit, LoRA adapter, or SLO tier matters, cc becomes a profile-calibrated policy.

8. SFQ fits the router layer

Assume one scheduler stands in front of a pool serving the same model. Every tenant queue is FIFO, and whenever a slot is available, the scheduler decides only which single request to admit next.

8.1 Collapse tags into one variable per tenant

Keep meter MiM_i, the next request's start tag, for every tenant and one virtual time VV. With equal weights, one decision is four lines:

select: i = argmin(active tenants) M_i   # tenant ID breaks ties
clock:  V = M_i
admit:  send tenant i's FIFO head r to the server pool
charge: M_i = M_i + c(r)

With weights, the last line becomes Mi←Mi+c(r)/wiM_i\leftarrow M_i+c(r)/w_i. The key is that current request cost c(r)c(r) is not part of argmin. Estimation error does not decide whether this request passes now; it changes the rest period after admission.

8.2 Leave and rejoin without banking idle credit

A tenant with an empty queue leaves the candidate set. When it becomes non-empty again, join the current baseline with

Mi←max⁡(Mi,V).M_i\leftarrow\max(M_i,V).

A tenant cannot save an unused share while idle and monopolize a later burst.

When every tenant's waiting queue drains, close the busy period without carrying old debt into the next one:

V←max⁡(V,max⁡iMi).V\leftarrow\max\left(V,\max_i M_i\right).

The interactive lab exposes both tenant C's late join and the final reset of V in its step display.

8.3 Why not retroactively correct every meter with actual cost?

In this router-level SFQ variant, c(r)c(r) is fixed at admission and is not surcharged or refunded after completion. Meters only move forward, and no race arises over the order in which measured completion costs are applied. Completion still determines when capacity opens relative to arrivals, so it can change the candidate set. Deterministic replay requires the same totally ordered stream of arrival, cancellation, and dispatch-capacity events, serialized decisions, and deterministic tie-breaking—not merely the same arrivals. Systematic error is addressed by slowly recalibrating α\alpha, β\beta, or the output-length predictor for future requests rather than rolling back individual meters.

Other designs reconcile predicted and actual output, such as VTC's length-prediction variant discussed below. They need an analysis different from the simple fixed-charge SFQ invariant.

8.4 What does it guarantee?

Use the tags from the original SFQ paper with the discrete router interpretation.

Assume equal weights, one-at-a-time admission, and a positive cost fixed at admission. Let cˉ\bar c be either a configured uniform bound on every request or the largest charge from the start of the busy period containing an observation interval through that interval's end. The maximum inside the observation interval alone cannot cover debt created before the interval. With this definition, meters for waiting tenants always lie in the following band at decision boundaries:

V≤Mi≤V+cˉ.V\le M_i\le V+\bar c.

For two tenants i,ji,j that remain backlogged throughout an interval, let Wi,WjW_i,W_j be the estimated cost charged in that interval. Bounding the meter difference at both endpoints yields

∣Wi−Wj∣≤2cˉ.|W_i-W_j|\le 2\bar c.

For weighted tenants, define normalized charge q(r)=c(r)/wowner⁡(r)q(r)=c(r)/w_{\operatorname{owner}(r)} and a busy-period-covering bound qˉ\bar q. For fixed weights, the corresponding statement is

∣Wiwi−Wjwj∣≤2qˉ.\left|\frac{W_i}{w_i}-\frac{W_j}{w_j}\right|\le 2\bar q.

Because one indivisible request can move a tenant ahead by one whole request immediately after admission, granularity on the order of cˉ\bar c (or qˉ\bar q with weights) is unavoidable.

The guarantee covers admission fairness over the estimated cost charged. It does not guarantee measured GPU seconds, KV-cache occupancy, per-request deadlines, or fairness across independent scheduler replicas. Nor does it transplant the original SFQ paper's single-link delay and throughput bounds unchanged to a parallel server pool.

8.5 Implementation cost

For TT active tenants, scanning every meter is O(T)O(T). A heap or ordered set keyed by (meter, tenant_id) gives O(log⁡T)O(\log T) updates and selection. For tens or hundreds of tenants, a linear scan is practical and easier to audit.

Implementation details that matter:

  • preserve tenant-local FIFO and use tenant ID to break equal-meter ties deterministically;
  • use integer or fixed-point cost units to avoid ambiguous floating-point ties;
  • only Mi−VM_i-V matters, so values can be rebased atomically with Mi←Mi−VM_i\leftarrow M_i-V for every tenant and V←0V\leftarrow0;
  • under the fixed-charge model, do not refund admission cost on cancellation or failure;
  • keep admission control, replica placement, and autoscaling in separate layers.

9. Use VTC inside the engine, and DLPM when locality matters

9.1 VTC: charge unknown output as tokens appear

Virtual Token Counter (VTC), defined in an OSDI 2024 paper, is an engine-level fairness mechanism. It integrates with a continuous-batching scheduler and maintains a counter per client. Adding a new request charges its input-token service. Every decode iteration adds the service of output tokens actually generated. When capacity opens in the batch, requests from the client with the smallest counter are favored.

VTC also lifts a returning client's counter to the active baseline so that idle time cannot accumulate credit. Its biggest difference from router-level SFQ is that unknown output need not be predicted: service is observed and charged while generation progresses.

AspectRouter-level SFQEngine-level VTC
DecisionAdmission into a server poolRequests added to a running batch
When cost is chargedFixed estimate immediately after admissionKnown input-token service at selection/prefill; generated output-token service after each decode step
Output lengthEstimated and fixedGenerated service observed before completion
Engine modificationNot requiredScheduler and counter updates must be integrated
Main guarantee targetDispatch fairness over estimated costExecution fairness over defined token service
RelationshipUpstreamDownstream; both can be combined

VTC's formal bound applies to clients continuously backlogged in the waiting queue under its single-engine, non-preemptive continuous-batching model. It bounds the defined token-service difference and depends on maximum input cost and the running batch's token capacity.

The VTC paper says unknown output length prevents classical SFQ from being applied directly, yet this does not conflict with recommending SFQ here, because the two guarantee different quantities. VTC targets measured token-service fairness inside the engine, while router SFQ narrows the problem to ordering fairness over cost fixed at admission.

VTC's length-prediction variant pre-charges predicted output cost when selecting a request, adds any underprediction during decoding, and refunds overprediction at completion. Experiments show that it can reduce average service discrepancy, but it does not improve the paper's worst-case lower bound. Prediction error and completion events also enter its counter updates, which makes it a different mechanism from fixed-charge router SFQ.

9.2 DLPM / D²LPM: relax fair order to reuse prefixes

VTC does not consider prefix locality. Deficit Longest Prefix Match (DLPM), proposed in a 2025 arXiv preprint, addresses the conflict between VTC's strict order and prefix locality. DLPM uses a DRR-inspired client quantum QuQ_u and deficit: while credit permits, it favors longest-prefix matches; when credit is exhausted, it yields to other clients. Double Deficit LPM (D²LPM) adds a per-client/per-worker quantum QwQ_w to control stickiness to a worker, bringing distributed prefix locality and load balance into the decision.

Increasing QuQ_u gives locality more freedom but also loosens DLPM's bound on the service difference. QwQ_w primarily controls worker stickiness and the locality/load-balance trade-off. DLPM-family policies expose the conflict between fairness and efficiency—one order cannot maximize both—as these explicit knobs.

9.3 Equinox: the argument that one token counter is insufficient

Equinox, a 2025 preprint, takes a different direction that predicts and combines multiple metrics. It argues that prefill/decode asymmetry prevents token count alone from representing latency, throughput, and GPU utilization simultaneously, and it combines predicted user-side and resource-side counters. Because it needs predictors and several policy weights, the meaning of a policy is harder to follow than with SFQ or VTC, and there is more to validate.

10. Choose from the objective and layer backward

SituationFirst candidateReason / caution
Reduce batch mean completion time with known sizesSJF / SRTFNeeds starvation control and handling of prediction error
Let interactive CPU tasks run quickly onceRR / MLFQQuantum versus context-switch cost
Hard or soft deadlines dominateEDF / RMSpecify the task model and analyze feasibility first
Divide CPU share by weightsStride / CFS / EEVDF familyAccount for sleep/wakeup and granularity
Share packet service rapidly across many flowsDRRBest when Lmax⁡L_{\max} and a quantum can be designed
Allocate multiple cluster resources to usersDRFPlacement can remain a separate problem
Fairly order admission into unmodified LLM serversEstimated cost + SFQGuarantee stops at one scheduler and estimated-cost order
Handle unknown output inside continuous batchingVTCRequires engine integration and a cost function
Preserve LLM prefix locality tooDLPM / D²LPMQuantum controls the fairness-locality trade-off
Optimize SLO/latency and GPU efficiency togetherProfile- or predictor-based schedulerContinuously validate prediction error and policy weights

A production design is usually clearer when responsibilities are layered instead of assigned to one algorithm:

Rate limits block abuse and enforce budget ceilings. A fair scheduler divides capacity during congestion. Placement handles load and locality. The engine scheduler controls actual token service. Separating these responsibilities by layer makes “why did this request wait?” easier to explain.

11. Pre-deployment validation checklist

  1. Objective: which comes first—mean, P99, deadlines, or tenant isolation?
  2. Backlog definition: one item in a queue, or a demand level at which another admission cannot improve throughput?
  3. Cost function: how are input/output tokens, profiled GPU time, and cache hits counted?
  4. Weight semantics: contract tier or urgency? Does w=2w=2 mean twice the service?
  5. Idle/rejoin rule: is unused share forbidden from accumulating, or intentionally exposed as burst credit?
  6. Indivisibility: where is preemption possible, and where is dispatch irreversible?
  7. Work conservation: can a fairness rule leave a GPU idle despite schedulable demand?
  8. Adversarial traces: test heavy tails, ON/OFF traffic, one-request rejoining, prediction error, and cancellation.
  9. Distributed state: does splitting counters across replicas destroy global fairness?
  10. Observability: record per-tenant wait, normalized service, counters, cache hits, TTFT (time to first token), and TPOT (time per output token) together.

Jain's fairness index

J(x1,…,xn)=(∑ixi)2n∑ixi2J(x_1,\ldots,x_n)=\frac{(\sum_i x_i)^2}{n\sum_i x_i^2}

summarizes how evenly service is shared in a single number, but its meaning changes depending on whether xix_i is request count, token service, or GPU seconds. It should not be used to demand equal service for a tenant with little demand, and it is undefined when all xi=0x_i=0. The lab resets its baseline whenever the comparison cohort changes and computes the displayed value only from increments during an interval in which that same cohort shares backlog. Production evaluation should likewise combine it with backlogged intervals, SLOs, and work conservation.

12. Summary

  • A scheduler combines an objective, assumptions, charging unit, and decision layer; the rule for choosing the next item is only one part.
  • FCFS → SJF/SRTF targets mean delay; RR/MLFQ targets responsiveness; EDF/RM targets deadlines; Lottery/Stride/CFS/EEVDF and fair queueing pursue proportional allocation on separate branches.
  • Approximating GPS with indivisible packets produced two implementation directions: WFQ/SFQ select minimum tags, whereas DRR avoids a global argmin by using cyclic visits and deficits.
  • Unknown output, prefill/decode, continuous batching, KV cache, and prefix locality mean request-count round robin is insufficient for LLM serving fairness.
  • In front of an unmodified server pool, SFQ is simple and explainable when it charges a fixed estimated cost after selection. Its guarantee is dispatch fairness over that estimate—not measured GPU fairness.
  • When the engine can be controlled, VTC's incremental charge for actual token progress better matches unknown output. DLPM-family policies become relevant when prefix locality must also be preserved.

Do not choose an algorithm by name. Define the fairness principal and the unit of service, identify the layer where a decision can actually be enforced, and then compare selection rules under those conditions. That design order is the same from classical operating systems to LLM serving.

References

  1. Remzi H. Arpaci-Dusseau, Andrea C. Arpaci-Dusseau, Operating Systems: Three Easy Pieces — Scheduling: Introduction / Multi-Level Feedback Queue.
  2. Linus E. Schrage, "A Proof of the Optimality of the Shortest Remaining Processing Time Discipline", Operations Research, 1968.
  3. C. L. Liu, James W. Layland, "Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment", JACM, 1973.
  4. Carl A. Waldspurger, William E. Weihl, "Stride Scheduling: Deterministic Proportional-Share Resource Management", MIT/LCS/TM-528, 1995.
  5. Linux Kernel documentation, CFS Scheduler / EEVDF Scheduler.
  6. Alan Demers, Srinivasan Keshav, Scott Shenker, "Analysis and Simulation of a Fair Queueing Algorithm", SIGCOMM, 1989.
  7. Abhay K. Parekh, Robert G. Gallager, "A Generalized Processor Sharing Approach to Flow Control in Integrated Services Networks", 1993.
  8. Pawan Goyal, Harrick M. Vin, Haichen Cheng, "Start-time Fair Queueing", SIGCOMM, 1996.
  9. M. Shreedhar, George Varghese, "Efficient Fair Queuing Using Deficit Round-Robin", 1996.
  10. Ali Ghodsi et al., "Dominant Resource Fairness: Fair Allocation of Multiple Resource Types", NSDI, 2011.
  11. Gyeong-In Yu et al., "Orca: A Distributed Serving System for Transformer-Based Generative Models", OSDI, 2022.
  12. Woosuk Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP, 2023.
  13. Ying Sheng et al., "Fairness in Serving Large Language Models", OSDI, 2024.
  14. Shiyi Cao et al., "Locality-aware Fair Scheduling in LLM Serving", 2025.
  15. Zhixiang Wei et al., "Equinox: Holistic Fair Scheduling in Serving Large Language Models", preprint, 2025.
  16. Carl A. Waldspurger, William E. Weihl, "Lottery Scheduling: Flexible Proportional-Share Resource Management", OSDI, 1994.
  17. Raj Jain, Dah-Ming Chiu, William Hawe, "A Quantitative Measure of Fairness and Discrimination for Resource Allocation in Shared Computer Systems", DEC TR-301, 1984.
  18. Kentaro Imajo, "Start-time Fair Queueing: Pay-after fair scheduling", 2026.