โ†All posts

How Go's Scheduler Works โ€” Understanding the G, M, P Model Visually

Go's runtime scheduler puts goroutines (G) in run queues owned by logical processors (P) and runs them on OS threads (M) that hold a P. The runtime source and an interactive animation show each step.

GoConcurrencyRuntimeInteractive

The Go runtime scheduler maps goroutines onto OS threads using the G-M-P model. A G (goroutine) waits in a queue owned by a P (the right to run Go code), and an M (OS thread) holding that P takes G's from the queue and runs them. I trace this flow through the structs and functions in the runtime package, and an animation shows the same steps.

Watch the video explainer3:57Synthesized narration ยท burned-in captions
Show transcript

Go's lightweight concurrency is powered by the runtime scheduler. Let's see how goroutines move across OS threads through the three roles called G, M, and P.

A G is the goroutine that wants to run, an M is an OS thread, and a P is the right to run user Go code. An M can execute user goroutines only while it is attached to a P. With this model, Go multiplexes many Gs onto fewer Ms.

The runtime.g struct carries an initial 2 KB stack and a gobuf used to resume execution. The runtime.m struct has g0, the scheduler goroutine, and curg, the user G currently running. The runtime.p struct is the execution token itself, with a local run queue and runnext slot.

Each P's runq is a fixed 256-entry ring buffer. A newly created G can go into the one-slot runnext cache, ahead of the queue. Besides each P's local queue, there is also a global queue shared by all Ps.

An M calls the schedule function on the g0 stack to find the next G to run. Every 61st tick checks the global queue, then runnext and the local queue are tried. If nothing is found yet, the search widens.

findRunnable tries runtime work such as GC and finalizers, then the global queue and netpoll. If it still finds nothing, it steals work from another randomly chosen P. At the end it waits for timers or netpoll, so the CPU is not spun forever.

When a G enters a syscall, its state becomes _Gsyscall, and its M is blocked in the OS. If the P stayed stuck too, the Gs queued on it could not run, so the runtime monitor sysmon reclaims the P with retake. If a P in syscall has other work, handoffp detaches it and gives it to another M.

A G that runs too long needs preemption. With cooperative preemption, sysmon spots a G running for more than 10 milliseconds. sysmon writes a special value into stackguard0, so the G drops back to runnable at its next function call. Since Go 1.14, signal-based asynchronous preemption can also interrupt loops that make no function calls.

Network I/O is made asynchronous through the netpoller. A G waiting in conn.Read is parked, while epoll or a similar OS facility watches its fd. When its I/O completes, the G is found by netpoll and put back into a run queue.

A G moves from runnable state, _Grunnable, into _Grunning on an M. When it waits on channels, mutexes, or I/O, it is _Gwaiting; inside a syscall, it is _Gsyscall. After completion, the G becomes _Gdead, and its struct goes onto the gFree list for reuse.

A goroutine switch just saves a few registers, such as the program counter and stack pointer, in gobuf. The article gives roughly 100 to 200 nanoseconds for a G switch. An OS thread switch is roughly 1 to 10 microseconds, with a 1 to 8 MB stack.

The number of Ps comes from GOMAXPROCS, but the number of Ms can be higher. With 4 Ps, 10 more Ms blocked in syscalls can bring the total to 14 Ms. Since Go 1.25, the default can account for Linux cgroup CPU limits. Once GOMAXPROCS is set manually, that automatic behavior is disabled.

Go's scheduler combines several mechanisms. Local queues, work stealing, hand-off, the netpoller, and preemption all work together. In the article, you can step through the G, M, P model yourself with an interactive demo.

G-M-P Model Overview

Internal Structure of G, M, P

G (Goroutine) โ€” runtime.g struct

A goroutine is represented by the runtime.g struct defined in runtime/runtime2.go:

type g struct {
    stack       stack   // lo, hi โ€” current stack bounds
    stackguard0 uintptr // used for preemption checks
    m           *m      // M currently running this G
    sched       gobuf   // saved context for context switch
    atomicstatus atomic.Uint32 // G's state flag
    goid         uint64 // goroutine ID
    // ... many more fields
}

Key fields:

  • stack โ€” Each G starts with a 2 KB stack (OS threads get 1โ€“8 MB); since Go 1.19, the runtime may allocate a larger initial stack based on average stack usage. When it runs out, runtime.morestack() grows it.
  • sched (gobuf) โ€” Stores PC (Program Counter), SP (Stack Pointer), and BP (Base Pointer) for context switches. It is read and written every time a G is suspended or resumed.
  • stackguard0 โ€” Normally set near the stack lower bound. Set to stackPreempt (0xfffffade) when preemption is requested.
  • atomicstatus โ€” The G lifecycle: _Gidle โ†’ _Grunnable โ†’ _Grunning โ†’ _Gsyscall / _Gwaiting โ†’ _Gdead

M (Machine) โ€” runtime.m struct

An M is a 1:1 wrapper around an OS thread (runtime/runtime2.go):

type m struct {
    g0      *g     // scheduling goroutine (large stack)
    curg    *g     // currently running user G
    p       *p     // attached P
    nextp   *p     // P to attach on wakeup
    spinning bool  // looking for work to steal
    // ...
}
  • g0 โ€” A special goroutine that exists on every M. Runtime functions like schedule(), findRunnable(), and GC code run on g0's stack. mcall() switches from a user G to g0.
  • curg โ€” The user goroutine currently running on this M. Set by execute(), cleared on goexit() or preemption.
  • spinning โ€” Indicates the M is actively searching for runnable work to steal. The runtime caps the number of spinning M's so idle threads don't burn CPU.

P (Processor) โ€” runtime.p struct

A P represents "the right to execute user Go code" โ€” a logical resource (runtime/runtime2.go):

type p struct {
    status    uint32 // _Pidle, _Prunning, _Pgcstop, _Pdead (_Psyscall was retired in Go 1.26)
    runqhead  uint32 // local queue head index
    runqtail  uint32 // local queue tail index
    runq      [256]guintptr // local run queue (ring buffer)
    runnext   guintptr      // next G to run (higher priority than queue)
    gFree     struct { ... } // free list of reusable G's
    // ...
}
  • runq โ€” A 256-entry fixed-size ring buffer that is used without locks. Only the owning P appends at the tail; both the owner and stealers take from the head by advancing it with an atomic CAS.
  • runnext โ€” A one-slot holder for the G that the current G just made runnable (for example, one it just created). It runs before anything in the queue, so it can use data that is still in cache.
  • The number of P's follows the current GOMAXPROCS value. It can change during execution through runtime.GOMAXPROCS, and on Go 1.25+ the default can also be re-evaluated automatically unless you set it manually.

The schedule() Loop in Detail

All scheduling decisions happen in runtime.schedule() in runtime/proc.go. Each M runs this loop on its g0 stack:

func schedule() {
    // Fairness: check global queue every 61st tick
    if schedtick%61 == 0 {
        G = globrunqget(P)    // fetch from global queue
    }
    if G == nil {
        G = runqget(P)        // โ‘  runnext โ†’ โ‘ก local queue
    }
    if G == nil {
        G = findRunnable(P)   // โ‘ข blocking search
        // โ†’ globrunqget โ†’ netpoll โ†’ steal from random P
    }
    execute(G)  // restore gobuf โ†’ run user code
}

If neither the periodic global-queue check in schedule() nor the local runqget() finds a G, findRunnable() searches more widely: trace-reader and GC-worker work, another periodic global-queue check, finalizer/cleanup goroutines, the local run queue, a batch from the global queue (min(len/GOMAXPROCS+1, len, 128) G's, where 128 is half the local queue's 256 slots), non-blocking netpoll, work stealing, and idle GC/trace work, in that order, before finally blocking on timers or netpoll. The exact order changes across releases, so treat this as a sketch of Go 1.27.

Stepping Through the Scheduler

The visualization below steps through these mechanics one at a time. Each step names the runtime functions called at that point.

Go Scheduler: G, M, P in Action

Step through how Go's runtime scheduler maps goroutines (G) onto OS threads (M) via processors (P).

P1
M1
OS thread
Running
โ€”
Local queue
empty
P2
M2
OS thread
Running
โ€”
Local queue
empty
Global run queue
empty
Initial: Entering the runtime.schedule() loop. Each M runs schedule() on its g0 stack, searching for the next G to run. P1-M1 and P2-M2 are bound.
1 / 14

Preemption Internals

Cooperative Preemption (Go 1.2+)

The Go compiler inserts a stack check prologue at the beginning of most functions:

// Function prologue (pseudocode)
MOV  AX, [G.stackguard0]
CMP  AX, SP
JBE  morestack          // stack growth or preemption

When sysmon's retake() detects a G running for more than 10 ms, it sets stackguard0 to stackPreempt (0xfffffade). At the next function call, the prologue detects this value and calls morestack() โ†’ gopreempt_m(), which sets the G back to _Grunnable.

Tight loops such as for {} contain no function calls, so they never reach the prologue and cannot be preempted this way.

Asynchronous Preemption (Go 1.14+)

Go 1.14 introduced signal-based asynchronous preemption:

  1. sysmon calls preemptone()
  2. On Unix-like platforms, sends SIGURG to the target M (other OSes use platform-specific preemption mechanisms)
  3. Signal handler doSigPreempt() saves the current PC/SP
  4. An asyncPreempt frame is injected onto the G's stack
  5. G is suspended and control returns to schedule()
// runtime/signal_unix.go โ€” https://github.com/golang/go/blob/master/src/runtime/signal_unix.go
func doSigPreempt(gp *g, ctxt *sigctxt) {
    if wantAsyncPreempt(gp) && isAsyncSafePoint(gp, ctxt.sigpc(), ctxt.sigsp(), ctxt.siglr()) {
        ctxt.pushCall(abi.FuncPCABI0(asyncPreempt), ctxt.rip())
    }
}

isAsyncSafePoint() only allows preemption at points where GC can accurately enumerate the root set.

sysmon โ€” The Watchdog Daemon

sysmon is the runtime's system-monitor function running on its own M/OS thread without a P (runtime/proc.go):

func sysmon() {
    for {
        usleep(delay) // 20ฮผs to 10ms (adaptive)
        // 1. Harvest netpoller results
        netpoll(0)
        // 2. Hand off P's stuck in long syscalls
        retake(now)
        // 3. Preempt long-running G's
        // 4. Signal for GC STW if needed
        // 5. Force GC if none in 2+ minutes
    }
}

retake() Decision Logic

func retake(now int64) uint32 {
    for _, pp := range allp {
        if pp.status != _Prunning {
            continue // since Go 1.26, a P in a syscall stays _Prunning
        }
        // Running >10ms โ†’ preemptone()
        if pd.schedwhen + 10ms <= now {
            preemptone(pp)
        }
        if inSyscall(pp) { // the P's G is _Gsyscall
            // In syscall >20ฮผs & there's other work โ†’ handoffp()
            if runqempty(pp) && sched.nmspinning + sched.npidle > 0 {
                continue // don't hand off yet
            }
            handoffp(pp) // detach P from M
        }
    }
}

Netpoller โ€” Async I/O Integration

All network I/O in Go (net.Conn.Read(), etc.) is made asynchronous through the netpoller:

  1. A connection's fd is registered with epoll/kqueue/IOCP when it is created
  2. G starts network I/O; if the fd is not ready, runtime.pollWait() calls gopark() and G becomes _Gwaiting
  3. findRunnable() and sysmon periodically call netpoll()
  4. I/O-completed G's are injected back into run queues via injectglist()
// Internal flow of network I/O
conn.Read(buf)
  โ†’ internal/poll.FD.Read()
    โ†’ runtime.pollWait()
      โ†’ gopark(netpollblockcommit) // G โ†’ _Gwaiting
// ... I/O completion detected via epoll_wait ...
runtime.netpoll()
  โ†’ G โ†’ _Grunnable
  โ†’ injectglist() into run queues

Unlike syscalls (read(2) etc.), the netpoller does not block the M, so net/http servers can handle thousands of connections with very few OS threads.

G Lifecycle โ€” State Transitions

_Gidle โ†’ _Gdead โ†’ _Grunnable โ†’ _Grunning โ†’ _Gdead
                       โ†‘            โ†“
                       โ† โ†  preempt โ† โ†
                       โ†‘            โ†“
                       โ† _Gwaiting โ†  (chan/mutex/IO)
                       โ†‘            โ†“
                       โ† _Gsyscall โ†  (syscall)
  • _Grunnable โ€” Ready to run but not yet assigned to an M
  • _Grunning โ€” Currently executing on an M
  • _Gwaiting โ€” Blocked on channel receive, mutex, I/O, etc.
  • _Gsyscall โ€” In a syscall (M is also blocked)
  • _Gdead โ€” Completed. The struct is pooled in gFree for reuse.

Context Switch Cost

Switching goroutines costs roughly an order of magnitude less than switching OS threads. The times in the table are published measurements and vary with the CPU and OS.

Go goroutineOS thread
What's switchedPC, SP, BP, a few registers (gobuf)More register/kernel scheduler state; possible cache/TLB disruption (page-table switches are mainly process switches)
Switch cost~170 ns (Bendersky's measurement)~1โ€“2 ฮผs (same measurement; direct cost, excluding cache effects)
Stack sizeInitial 2 KB or more (grows dynamically)Fixed 1โ€“8 MB
Creation cost~200 ns (Go runtime benchmark result)~5 ฮผs (pthread_create measurement)

The gobuf structure:

type gobuf struct {
    sp   uintptr // stack pointer
    pc   uintptr // program counter
    g    guintptr
    ctxt unsafe.Pointer
    ret  uintptr
    lr   uintptr // link register (ARM)
    bp   uintptr // base pointer (frame pointer)
}

GOMAXPROCS vs Actual Thread Count

The current number of P's is set by GOMAXPROCS, but the actual number of M's (OS threads) can be higher.

  • M's blocked in syscalls release their P, and new M's are created to run that P
  • The default maximum number of M's is 10,000 (runtime/debug.SetMaxThreads)
  • Idle M's are not destroyed immediately โ€” they're pooled in midle
runtime.GOMAXPROCS(4)  // P = 4
// But M count can be > 4
// e.g., 4 P's + 10 M's blocked in syscalls = 14 total M's

Container-Aware Defaults (Go 1.25+)

For a long time GOMAXPROCS defaulted to the number of logical CPUs at startup (runtime.NumCPU()). In containers this was a common cause of performance problems: run a Pod with a 2-core CPU limit on a 64-core node and the Go runtime would set GOMAXPROCS=64 and create 64 P's. The cgroup CPU limit would then throttle the process, and the extra context switches would make latency worse. The usual workaround was an external library such as automaxprocs.

Go 1.25 moved this adjustment into the runtime itself:

  • On Linux the runtime accounts for the cgroup CPU bandwidth limit โ€” if the limit is lower than the number of logical CPUs, that lower value becomes the GOMAXPROCS default. This corresponds to the Kubernetes CPU limit; CPU requests are not considered.
  • On all OSes the value is re-evaluated periodically โ€” GOMAXPROCS follows changes in available logical CPUs or in the cgroup limit.

Both behaviors switch off automatically as soon as you set GOMAXPROCS manually, via the environment variable or runtime.GOMAXPROCS() (they can also be disabled individually with the GODEBUG settings containermaxprocs=0 and updatemaxprocs=0). If you have set it manually and want to return to the default behavior, use the new runtime.SetDefaultGOMAXPROCS().

Summary

Go's G-M-P scheduler is built from these seven mechanisms:

  1. M:N scheduling โ€” Multiplexes many G's onto a few M's
  2. Lock-free local queues โ€” A 256-entry ring buffer per P, so G's go in and out without taking a lock
  3. Work Stealing โ€” A P with nothing to run takes G's from another P's queue to spread the load
  4. Hand-off โ€” When an M blocks in a syscall, its P moves to another M and keeps running
  5. Netpoller โ€” Network I/O handled asynchronously via epoll/kqueue without blocking M's
  6. Cooperative + async preemption โ€” Function-prologue checks plus platform-specific async preemption keep one G from holding the CPU indefinitely
  7. sysmon โ€” A dedicated thread without a P that detects long syscalls and long-running G's and acts on them