A Genealogy of Task Scheduling — From FIFO to Fair Queueing and LLM Inference
The right answer to “what should run next?” depends on whether mean delay, responsiveness, deadlines, or fairness comes first. This guide traces FCFS, SJF, Round Robin, EDF, CFS, WFQ, SFQ, DRR, and LLM-oriented VTC by objective, mechanism, and assumptions, and shows with a simulator why SFQ suits LLM request admission when output length is unknown.
There is no single correct answer to “what should run next?” for a shared queue. The right algorithm depends on whether the objective is to reduce mean waiting time, respond to interaction quickly, meet deadlines, or divide resources fairly among users.
CPU scheduling, network fair queueing, and LLM inference serving can be traced as one genealogy by asking what each algorithm tries to optimize and what it changes from its predecessors. For the order in which LLM requests are admitted into a server pool, the “select, then charge” structure of Start-time Fair Queueing (SFQ) fits well: even when output length is unknown, a router can admit a request first and charge its estimated cost afterward. VTC (Virtual Token Counter), used inside an inference engine, differs by charging tokens as they are actually generated.
Show transcript
For the same queue, there is no single correct answer to what should run next. From CPU scheduling to networks and LLM inference, let's trace how scheduling branches by goal.
The first question is what “good” means: shorter averages, faster first response, or something else. Meeting deadlines, sharing fairly, using all capacity, and lowering P99 often pull in different directions.
FCFS runs jobs in arrival order until completion, but short jobs can pile up behind a long one: the convoy effect. SJF chooses the shortest known job first and minimizes mean waiting time when all jobs are present at once. SRTF switches to the shortest remaining job; ideally it minimizes mean flow time, but long jobs can starve. Round Robin lets every task run soon, at the cost of switching overhead; MLFQ infers short, interactive work from past behavior.
On the deadline branch, EDF selects the ready job with the earliest absolute deadline. With independent jobs, one preemptive CPU, and negligible switching cost, if any schedule can meet the deadlines, EDF can too. RM gives shorter-period tasks higher fixed priority. This branch is about feasibility, not equal service.
In the proportional-share branch, schedulers favor the principal with the least normalized service so far. Lottery uses randomness; Stride advances a deterministic pass more slowly for larger weights, so those tasks are picked more often. As of 2026, mainline Linux uses EEVDF for normal tasks. Among tasks not ahead of their fair share, it picks the earliest virtual deadline.
Networks add a twist: once a packet starts transmitting, it normally cannot be split and interleaved. GPS, Generalized Processor Sharing, is an ideal fluid model serving backlogged flows simultaneously in proportion to their weights. WFQ compares virtual finish tags, while Start-time Fair Queueing, SFQ, compares start tags. That timing shift matters for LLM routers. DRR skips tags: it visits flows in turn, grants a quantum, carries unused credit as deficit, and aims for constant time per packet.
For LLMs, input length is known, but output length is not final until generation ends. max_tokens is only a cap, so SJF or WFQ would need a length predictor. Prefill is parallel and compute-heavy; decode generates one token at a time while reading the KV cache. So a request can be indivisible at the router, yet engine continuous batching can regroup it at every decode iteration.
In front of unmodified servers, SFQ with per-tenant FIFO queues and meters is a simple router policy. Select the tenant with the smallest meter. The current request's cost never decides whether it passes now. It only affects how long that tenant waits for its next turn. When an idle tenant returns, its meter is raised to at least the current baseline. Unused share cannot be banked. The guarantee is admission fairness over estimated cost, not measured GPU time.
If you control the engine, VTC, the Virtual Token Counter from OSDI 2024, gives a different answer. VTC charges each client's counter for input tokens. It adds actual output tokens every decode iteration, and favors the client with the smallest counter. When prefix locality also matters, DLPM comes from a 2025 preprint. It favors prefix matches while a per-client quantum lasts, relaxing fair order in a controlled way. The 2025 Equinox preprint argues that one token counter cannot capture latency, throughput, and GPU utilization.
The article includes interactive CPU and LLM demos, so you can run the policies yourself and see the differences.
1. Define a good schedule first
Represent task by arrival time , processing demand , completion time , deadline , and weight . Common objectives include:
| Objective | Metric | Typical question |
|---|---|---|
| Finish work early | Turnaround , mean flow time | How can jobs finish sooner on average? |
| React early | Response time (first run time ) | How can an interactive request respond immediately? |
| Meet deadlines | Deadline misses, maximum lateness | Can control, audio, or video work finish on time? |
| Divide fairly | Difference in normalized service, max-min fairness | Can a noisy neighbor be isolated? |
| Use all capacity | Throughput, utilization, work conservation | Is usable capacity being left idle? |
| Be predictable | Tail latency, maximum wait, jitter | Can P99 or a worst case be bounded? |
These objectives often conflict. Prioritizing short jobs reduces mean wait but can starve long jobs. Switching in small increments improves responsiveness but adds context switches and hurts cache locality.
Before choosing an algorithm, settle at least the following assumptions:
- Preemptibility: can work stop and resume partway through?
- Size information: is exact, estimated, or unknown until completion?
- Resource dimensionality: is CPU the only resource, or do GPU memory, bandwidth, and KV cache contend too?
- Fairness principal: is fairness among processes, users, organizations, API keys, or models?
- Decision layer: is the decision admission, replica placement, dispatch order, or an in-flight token iteration?
2. Genealogy: branches by objective
The following diagram groups the algorithms by objective and selection logic. It does not show a chronology or a strict inheritance tree. Solid edges connect related designs by classification or contrast. Dotted edges show Linux's adoption transition or DLPM relaxing VTC's strict order for locality.
Newer algorithms do not dominate older ones; which branch fits depends on the objective and constraints. DRR's amortized work is attractive for tens of thousands of flows at line rate. A gateway with a few dozen LLM tenants can afford a tag-based algorithm with selection.
3. The completion-time and responsiveness branch
3.1 FCFS / FIFO: arrival order is easy to explain
First-Come, First-Served (FCFS) selects the head of an arrival-ordered queue and normally runs it to completion.
ready = queue ordered by arrival
if the CPU is free:
running = ready.pop_front()
run running until completionEach enqueue/dequeue and next-task decision is , deterministic, and easy to explain; constructing the full schedule for jobs is . Its weakness is the convoy effect: when one long job is followed by many short jobs, every short job waits behind the long one. Mean waiting time rises, and interactive work responds later.
3.2 SJF / SRTF: prefer short work when size is known
Shortest Job First (SJF) chooses the arrived task with minimum whenever the CPU becomes free. Under the classical assumptions that all jobs are available together, lengths are known, there is one CPU, and execution is non-preemptive, SJF minimizes mean waiting time.
Shortest Remaining Time First (SRTF, also called SRPT or STCF) is the preemptive counterpart. At each decision point it selects the task with the smallest remaining time . For jobs with known release times and processing times on one unit-speed machine, with cost-free preemption, SRTF minimizes total—and therefore mean—flow time. It does so at a cost:
- long tasks can starve while short tasks keep arriving;
- the policy is sensitive to size-prediction error;
- every preemption adds state-save and restore work;
- a user benefits by declaring a job shorter, so untrusted self-reported size is unsafe.
3.3 Round Robin: prioritize “run once soon” over completion order
Round Robin (RR) gives each task at most a time quantum , then moves unfinished work to the tail.
while ready is not empty:
task = ready.pop_front()
run min(q, task.remaining)
if task is unfinished:
ready.push_back(task)As , RR approaches FCFS. Reducing lets every runnable task respond sooner, but making it too small increases context-switch and cache-miss costs on real hardware. RR equalizes CPU time. A request-count round robin, discussed later, does not equalize resources when one request can be much heavier than another.
3.4 MLFQ: infer short and interactive work without knowing the future
A Multi-Level Feedback Queue (MLFQ) maintains several priority queues. New tasks start at high priority; tasks that consume their CPU allotment are demoted; interactive tasks that block or yield quickly remain high. Periodically moving every task back to the top—a global priority boost—limits starvation. Aging, which raises an individual task's priority as it waits, is another related mechanism.
MLFQ effectively replaces the “predict total run time” input that SJF requires with online inference from observed behavior. If the rules reward yielding just before a quantum expires, however, a task can game the policy, so implementations account for cumulative CPU usage and related behavior.
Whereas MLFQ adjusts priorities from observed behavior, fixed-priority scheduling always chooses the highest-priority runnable task. In a simple implementation, low-priority tasks can starve, so aging or bounded priority boosts are needed. Priority inversion is a different failure mode: a low-priority task holding a shared lock can block a high-priority task while medium-priority work preempts the lock holder. Priority inheritance temporarily boosts the lock holder to the waiting task's priority. Priority-ceiling protocols instead use each resource's ceiling priority to restrict or raise lock holders, limiting inversion and deadlock.
4. The deadline and proportional-share branches
4.1 EDF and Rate Monotonic: deadline feasibility before fairness
Earliest Deadline First (EDF) selects the ready job with the earliest absolute deadline and preempts when an earlier deadline arrives. Under the standard assumptions of independent jobs, one preemptive CPU, and negligible switching cost, EDF is feasibility-optimal: if any schedule can meet the deadlines, EDF can as well.
Rate Monotonic (RM) gives a shorter-period periodic task a higher fixed priority. Under the Liu–Layland assumptions of independent periodic tasks, implicit deadlines , one preemptive CPU, fixed worst-case execution time , and no blocking or switching overhead, the famous sufficient utilization condition for tasks is
In the same independent periodic-task, implicit-deadline model with idealized preemption and switching cost, EDF is schedulable exactly up to total utilization , i.e. 100% of one CPU.
EDF and RM reorder work to meet deadlines rather than to give everybody equal service.
4.2 Lottery / Stride: convert weights into CPU shares
Lottery scheduling gives each task tickets proportional to its weight and draws which task receives the next fixed-size allocation quantum. It provides proportional shares in long-run expectation but has short-term variance.
Stride scheduling makes the process deterministic. For a large constant , define
Give each task a counter, pass. Choose the task with the smallest pass, grant the same-size quantum, then update pass += stride. If allocations vary in size, advance proportionally with pass += service_used × L / w_i. A high-weight task advances more slowly per unit service and is selected more often. The same skeleton—choose the principal with the least accumulated normalized service—reappears in CFS, SFQ, and VTC.
4.3 From CFS to EEVDF
Linux's Completely Fair Scheduler (CFS) approximates an ideal weighted CPU that runs all runnable tasks simultaneously. On a real CPU it selects the task with the smallest normalized execution time, vruntime. Running that task advances its vruntime, and eventually another task has the smallest vruntime and is selected.
Earliest Eligible Virtual Deadline First (EEVDF) was proposed in 1995, and Linux began moving to Peter Zijlstra's 2023 EEVDF design in version 6.6. As of 2026, the default selection rule for mainline Linux's normal fair class is therefore EEVDF, which keeps CFS's virtual-time approach, rather than CFS's “pick minimum vruntime” rule. EEVDF expresses how far a task is behind its fair share as lag; among tasks with lag >= 0, it chooses the earliest virtual deadline. Making eligibility and deadline explicit in the selection rule preserves proportional sharing while making latency-sensitive tasks with shorter requested slices easier to express.
5. The fair-queueing branch that grew up in networks
CPU time can be preempted finely. A network packet normally cannot be interleaved with another flow's packet once transmission begins. Research on dividing such indivisible work fairly among flows leads directly to LLM request scheduling.
5.1 GPS: an ideal model that defines fairness
Generalized Processor Sharing (GPS) treats every backlogged flow as infinitely divisible fluid and shares one link simultaneously in proportion to weights . Real packet schedulers approximate GPS completion order or service discrepancy using indivisible packets.
5.2 WFQ: choose the smallest virtual finish time
Assign start and finish tags to packet from flow :
Weighted Fair Queueing (WFQ) broadly selects the packet with the smallest , approximating GPS completion order. Packet length must be known before selection. WFQ computes the virtual time by simulating GPS.
5.3 SFQ: choose by start tag, not finish tag
Start-time Fair Queueing (SFQ) compares the FIFO head packet of every backlogged flow and selects the one with the smallest . For the virtual time , it uses the start tag of the packet in service rather than simulating GPS. The selected packet's length does not decide whether that packet is selected now. Instead, it sets the finish tag—and therefore affects the next selection from that flow.
That shift in time suits an LLM router that knows request size only as an estimate. Let a request pass first, record its estimated cost on the tenant's meter, and let the other tenants catch up before selecting that tenant again—a pay-after, or deferred-charge, structure.
5.4 DRR: replace tag arithmetic with a quantum and deficit
Deficit Round Robin (DRR) keeps a deficit counter for each flow and adds quantum whenever it visits an active flow. It repeatedly transmits head packets while they fit, subtracting each length, and carries an unused deficit to the next round only while the flow remains backlogged. When the queue drains, it resets so idle credit cannot accumulate. Weighted DRR uses . If , every visit can advance at least one packet, supporting the usual amortized- argument.
DRR hands out credit before use, whereas SFQ lets work pass first and records the debt after selection. A network MTU gives a stable basis for choosing a quantum. Maximum LLM request cost changes substantially with the model and workload, so a quantum is harder to choose.
5.5 DRF: consider multiple resources together
In a cluster, some jobs use mostly CPU and others mostly memory. Dominant Resource Fairness (DRF) takes each user's maximum allocation fraction across resources as its dominant share and favors users with smaller dominant shares. This branch defines max-min fairness over multiple resources instead of forcing every cost into one processing-time scalar.
6. Run the algorithms yourself
The lab below has two modes.
- CPU: change arrival time, burst length, and RR quantum; compare FCFS, SJF, SRTF, and RR timelines and delays.
- LLM: change every tenant's estimated cost, request count, and weight, plus tenant C's join time; compare the admission order under Global FIFO, Tenant Round Robin, Shortest request first, and SFQ.
In CPU mode, first observe short C waiting behind long A, then switch to SRTF. In LLM mode, set A's cost to 8 and B's to 2. Count admissions in the sequence under Tenant Round Robin and SFQ, then compare them with the normalized-cost bars for the common-backlog interval: Tenant Round Robin equalizes request counts, while SFQ equalizes normalized cost.
Scheduling Lab
Switch between a CPU timeline and LLM-router admission, tune parameters, and step through one stage at a time.
Runs the earliest arrival to completion. With the defaults, short C waits behind long A—the convoy effect.
Execution timeline
t = 0Ready queue (waiting runnable tasks)
Remaining time
Final metrics
| Task | Completion | Turnaround | Waiting | Response |
|---|---|---|---|---|
| A | 7 | 7 | 0 | 0 |
| B | 11 | 10 | 6 | 6 |
| C | 12 | 10 | 9 | 9 |
| D | 15 | 11 | 8 | 8 |
| Average | 5.75 | 5.75 |
This teaching model uses one CPU, integer ticks, and zero switch cost. SJF/SRTF assume known bursts. Each stage shows the preceding one-tick transition and the resulting state.
The simulator is a discrete teaching model. In particular, LLM mode models only admission order into a server pool, not GPU execution time. Because finite queues eventually expose different total demand, observe fairness while both tenants remain backlogged rather than only at the final state.
In LLM mode, each tenant's configured requests arrive as one batch at its join time, and logical time advances by one per admission. While queues remain active, a boundary join occurs before that time's admission. If the preceding admission drained every queue, SFQ's idle reset occurs before a join at the same boundary. If all queues remain empty until a future arrival, time jumps there and reset / join appears as a separate non-admission stage. With no future arrival, the final reset is annotated on the last admission.
7. What is special about an LLM request?
LLM requests have properties that are missed if they are treated as differently sized HTTP requests.
7.1 Output length is unknown at selection time
Input token count is known, but output is not determined until generation terminates—by an end-of-sequence (EOS) token, an output cap, a stop sequence, cancellation, or error. max_tokens is a limit, not the generated length. Policies such as SJF and WFQ that rank the current job using an exact length cannot be applied directly without a predictor.
7.2 Prefill and decode have different cost structures
Prefill processes prompt tokens in parallel and tends to be compute-heavy. Decode generates tokens autoregressively, one iteration at a time, while repeatedly reading the KV cache. A token stresses different bottlenecks depending on phase, batch composition, sequence length, and hardware. Token count is a useful service proxy, not a synonym for GPU seconds.
7.3 A request is indivisible outside but iterative inside
After a load balancer dispatches a request, returning it safely and moving it to another server is generally expensive. Treating it as non-preemptive at the outer router is therefore natural.
Inside an engine, Orca proposed iteration-level scheduling with selective batching. In what is now commonly called continuous or in-flight batching, an engine can rebuild a batch at decode-iteration boundaries. Some implementations can also preempt with KV-cache swap or recomputation. Whether an LLM request can be preempted therefore depends on which layer's scheduler is making the decision.
7.4 Batching, KV cache, and prefix locality interact
Co-locating requests with a shared prefix can reuse KV cache. Strict fairness order can scatter requests from one tenant and destroy that locality. When the head request does not fit available KV blocks, skipping it improves work conservation but weakens the ability to explain order using a fairness counter alone.
7.5 Request count is the wrong fairness unit
Giving 100 short chats and 100 long generations equal request counts does not equalize resource consumption. If the fairness principal is the tenant, service needs at least a positive estimated cost such as
If model, prefix-cache hit, LoRA adapter, or SLO tier matters, becomes a profile-calibrated policy.
8. SFQ fits the router layer
Assume one scheduler stands in front of a pool serving the same model. Every tenant queue is FIFO, and whenever a slot is available, the scheduler decides only which single request to admit next.
8.1 Collapse tags into one variable per tenant
Keep meter , the next request's start tag, for every tenant and one virtual time . With equal weights, one decision is four lines:
select: i = argmin(active tenants) M_i # tenant ID breaks ties
clock: V = M_i
admit: send tenant i's FIFO head r to the server pool
charge: M_i = M_i + c(r)With weights, the last line becomes . The key is that current request cost is not part of argmin. Estimation error does not decide whether this request passes now; it changes the rest period after admission.
8.2 Leave and rejoin without banking idle credit
A tenant with an empty queue leaves the candidate set. When it becomes non-empty again, join the current baseline with
A tenant cannot save an unused share while idle and monopolize a later burst.
When every tenant's waiting queue drains, close the busy period without carrying old debt into the next one:
The interactive lab exposes both tenant C's late join and the final reset of V in its step display.
8.3 Why not retroactively correct every meter with actual cost?
In this router-level SFQ variant, is fixed at admission and is not surcharged or refunded after completion. Meters only move forward, and no race arises over the order in which measured completion costs are applied. Completion still determines when capacity opens relative to arrivals, so it can change the candidate set. Deterministic replay requires the same totally ordered stream of arrival, cancellation, and dispatch-capacity events, serialized decisions, and deterministic tie-breaking—not merely the same arrivals. Systematic error is addressed by slowly recalibrating , , or the output-length predictor for future requests rather than rolling back individual meters.
Other designs reconcile predicted and actual output, such as VTC's length-prediction variant discussed below. They need an analysis different from the simple fixed-charge SFQ invariant.
8.4 What does it guarantee?
Use the tags from the original SFQ paper with the discrete router interpretation.
Assume equal weights, one-at-a-time admission, and a positive cost fixed at admission. Let be either a configured uniform bound on every request or the largest charge from the start of the busy period containing an observation interval through that interval's end. The maximum inside the observation interval alone cannot cover debt created before the interval. With this definition, meters for waiting tenants always lie in the following band at decision boundaries:
For two tenants that remain backlogged throughout an interval, let be the estimated cost charged in that interval. Bounding the meter difference at both endpoints yields
For weighted tenants, define normalized charge and a busy-period-covering bound . For fixed weights, the corresponding statement is
Because one indivisible request can move a tenant ahead by one whole request immediately after admission, granularity on the order of (or with weights) is unavoidable.
The guarantee covers admission fairness over the estimated cost charged. It does not guarantee measured GPU seconds, KV-cache occupancy, per-request deadlines, or fairness across independent scheduler replicas. Nor does it transplant the original SFQ paper's single-link delay and throughput bounds unchanged to a parallel server pool.
8.5 Implementation cost
For active tenants, scanning every meter is . A heap or ordered set keyed by (meter, tenant_id) gives updates and selection. For tens or hundreds of tenants, a linear scan is practical and easier to audit.
Implementation details that matter:
- preserve tenant-local FIFO and use tenant ID to break equal-meter ties deterministically;
- use integer or fixed-point cost units to avoid ambiguous floating-point ties;
- only matters, so values can be rebased atomically with for every tenant and ;
- under the fixed-charge model, do not refund admission cost on cancellation or failure;
- keep admission control, replica placement, and autoscaling in separate layers.
9. Use VTC inside the engine, and DLPM when locality matters
9.1 VTC: charge unknown output as tokens appear
Virtual Token Counter (VTC), defined in an OSDI 2024 paper, is an engine-level fairness mechanism. It integrates with a continuous-batching scheduler and maintains a counter per client. Adding a new request charges its input-token service. Every decode iteration adds the service of output tokens actually generated. When capacity opens in the batch, requests from the client with the smallest counter are favored.
VTC also lifts a returning client's counter to the active baseline so that idle time cannot accumulate credit. Its biggest difference from router-level SFQ is that unknown output need not be predicted: service is observed and charged while generation progresses.
| Aspect | Router-level SFQ | Engine-level VTC |
|---|---|---|
| Decision | Admission into a server pool | Requests added to a running batch |
| When cost is charged | Fixed estimate immediately after admission | Known input-token service at selection/prefill; generated output-token service after each decode step |
| Output length | Estimated and fixed | Generated service observed before completion |
| Engine modification | Not required | Scheduler and counter updates must be integrated |
| Main guarantee target | Dispatch fairness over estimated cost | Execution fairness over defined token service |
| Relationship | Upstream | Downstream; both can be combined |
VTC's formal bound applies to clients continuously backlogged in the waiting queue under its single-engine, non-preemptive continuous-batching model. It bounds the defined token-service difference and depends on maximum input cost and the running batch's token capacity.
The VTC paper says unknown output length prevents classical SFQ from being applied directly, yet this does not conflict with recommending SFQ here, because the two guarantee different quantities. VTC targets measured token-service fairness inside the engine, while router SFQ narrows the problem to ordering fairness over cost fixed at admission.
VTC's length-prediction variant pre-charges predicted output cost when selecting a request, adds any underprediction during decoding, and refunds overprediction at completion. Experiments show that it can reduce average service discrepancy, but it does not improve the paper's worst-case lower bound. Prediction error and completion events also enter its counter updates, which makes it a different mechanism from fixed-charge router SFQ.
9.2 DLPM / D²LPM: relax fair order to reuse prefixes
VTC does not consider prefix locality. Deficit Longest Prefix Match (DLPM), proposed in a 2025 arXiv preprint, addresses the conflict between VTC's strict order and prefix locality. DLPM uses a DRR-inspired client quantum and deficit: while credit permits, it favors longest-prefix matches; when credit is exhausted, it yields to other clients. Double Deficit LPM (D²LPM) adds a per-client/per-worker quantum to control stickiness to a worker, bringing distributed prefix locality and load balance into the decision.
Increasing gives locality more freedom but also loosens DLPM's bound on the service difference. primarily controls worker stickiness and the locality/load-balance trade-off. DLPM-family policies expose the conflict between fairness and efficiency—one order cannot maximize both—as these explicit knobs.
9.3 Equinox: the argument that one token counter is insufficient
Equinox, a 2025 preprint, takes a different direction that predicts and combines multiple metrics. It argues that prefill/decode asymmetry prevents token count alone from representing latency, throughput, and GPU utilization simultaneously, and it combines predicted user-side and resource-side counters. Because it needs predictors and several policy weights, the meaning of a policy is harder to follow than with SFQ or VTC, and there is more to validate.
10. Choose from the objective and layer backward
| Situation | First candidate | Reason / caution |
|---|---|---|
| Reduce batch mean completion time with known sizes | SJF / SRTF | Needs starvation control and handling of prediction error |
| Let interactive CPU tasks run quickly once | RR / MLFQ | Quantum versus context-switch cost |
| Hard or soft deadlines dominate | EDF / RM | Specify the task model and analyze feasibility first |
| Divide CPU share by weights | Stride / CFS / EEVDF family | Account for sleep/wakeup and granularity |
| Share packet service rapidly across many flows | DRR | Best when and a quantum can be designed |
| Allocate multiple cluster resources to users | DRF | Placement can remain a separate problem |
| Fairly order admission into unmodified LLM servers | Estimated cost + SFQ | Guarantee stops at one scheduler and estimated-cost order |
| Handle unknown output inside continuous batching | VTC | Requires engine integration and a cost function |
| Preserve LLM prefix locality too | DLPM / D²LPM | Quantum controls the fairness-locality trade-off |
| Optimize SLO/latency and GPU efficiency together | Profile- or predictor-based scheduler | Continuously validate prediction error and policy weights |
A production design is usually clearer when responsibilities are layered instead of assigned to one algorithm:
Rate limits block abuse and enforce budget ceilings. A fair scheduler divides capacity during congestion. Placement handles load and locality. The engine scheduler controls actual token service. Separating these responsibilities by layer makes “why did this request wait?” easier to explain.
11. Pre-deployment validation checklist
- Objective: which comes first—mean, P99, deadlines, or tenant isolation?
- Backlog definition: one item in a queue, or a demand level at which another admission cannot improve throughput?
- Cost function: how are input/output tokens, profiled GPU time, and cache hits counted?
- Weight semantics: contract tier or urgency? Does mean twice the service?
- Idle/rejoin rule: is unused share forbidden from accumulating, or intentionally exposed as burst credit?
- Indivisibility: where is preemption possible, and where is dispatch irreversible?
- Work conservation: can a fairness rule leave a GPU idle despite schedulable demand?
- Adversarial traces: test heavy tails, ON/OFF traffic, one-request rejoining, prediction error, and cancellation.
- Distributed state: does splitting counters across replicas destroy global fairness?
- Observability: record per-tenant wait, normalized service, counters, cache hits, TTFT (time to first token), and TPOT (time per output token) together.
Jain's fairness index
summarizes how evenly service is shared in a single number, but its meaning changes depending on whether is request count, token service, or GPU seconds. It should not be used to demand equal service for a tenant with little demand, and it is undefined when all . The lab resets its baseline whenever the comparison cohort changes and computes the displayed value only from increments during an interval in which that same cohort shares backlog. Production evaluation should likewise combine it with backlogged intervals, SLOs, and work conservation.
12. Summary
- A scheduler combines an objective, assumptions, charging unit, and decision layer; the rule for choosing the next item is only one part.
- FCFS → SJF/SRTF targets mean delay; RR/MLFQ targets responsiveness; EDF/RM targets deadlines; Lottery/Stride/CFS/EEVDF and fair queueing pursue proportional allocation on separate branches.
- Approximating GPS with indivisible packets produced two implementation directions: WFQ/SFQ select minimum tags, whereas DRR avoids a global
argminby using cyclic visits and deficits. - Unknown output, prefill/decode, continuous batching, KV cache, and prefix locality mean request-count round robin is insufficient for LLM serving fairness.
- In front of an unmodified server pool, SFQ is simple and explainable when it charges a fixed estimated cost after selection. Its guarantee is dispatch fairness over that estimate—not measured GPU fairness.
- When the engine can be controlled, VTC's incremental charge for actual token progress better matches unknown output. DLPM-family policies become relevant when prefix locality must also be preserved.
Do not choose an algorithm by name. Define the fairness principal and the unit of service, identify the layer where a decision can actually be enforced, and then compare selection rules under those conditions. That design order is the same from classical operating systems to LLM serving.
References
- Remzi H. Arpaci-Dusseau, Andrea C. Arpaci-Dusseau, Operating Systems: Three Easy Pieces — Scheduling: Introduction / Multi-Level Feedback Queue.
- Linus E. Schrage, "A Proof of the Optimality of the Shortest Remaining Processing Time Discipline", Operations Research, 1968.
- C. L. Liu, James W. Layland, "Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment", JACM, 1973.
- Carl A. Waldspurger, William E. Weihl, "Stride Scheduling: Deterministic Proportional-Share Resource Management", MIT/LCS/TM-528, 1995.
- Linux Kernel documentation, CFS Scheduler / EEVDF Scheduler.
- Alan Demers, Srinivasan Keshav, Scott Shenker, "Analysis and Simulation of a Fair Queueing Algorithm", SIGCOMM, 1989.
- Abhay K. Parekh, Robert G. Gallager, "A Generalized Processor Sharing Approach to Flow Control in Integrated Services Networks", 1993.
- Pawan Goyal, Harrick M. Vin, Haichen Cheng, "Start-time Fair Queueing", SIGCOMM, 1996.
- M. Shreedhar, George Varghese, "Efficient Fair Queuing Using Deficit Round-Robin", 1996.
- Ali Ghodsi et al., "Dominant Resource Fairness: Fair Allocation of Multiple Resource Types", NSDI, 2011.
- Gyeong-In Yu et al., "Orca: A Distributed Serving System for Transformer-Based Generative Models", OSDI, 2022.
- Woosuk Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP, 2023.
- Ying Sheng et al., "Fairness in Serving Large Language Models", OSDI, 2024.
- Shiyi Cao et al., "Locality-aware Fair Scheduling in LLM Serving", 2025.
- Zhixiang Wei et al., "Equinox: Holistic Fair Scheduling in Serving Large Language Models", preprint, 2025.
- Carl A. Waldspurger, William E. Weihl, "Lottery Scheduling: Flexible Proportional-Share Resource Management", OSDI, 1994.
- Raj Jain, Dah-Ming Chiu, William Hawe, "A Quantitative Measure of Fairness and Discrimination for Resource Allocation in Shared Computer Systems", DEC TR-301, 1984.
- Kentaro Imajo, "Start-time Fair Queueing: Pay-after fair scheduling", 2026.