Where Go Concurrency Helps in Practice — When to Choose Channels, and Production Patterns

Write go func() and a goroutine starts. Send with ch <- v, receive with <-ch. Use select to wait on several channels at once. As syntax, this takes thirty minutes to learn. The textbook examples make sense too.

And yet, when you write code at work, you cannot find anywhere to use a channel. Or you force one in, and the result is harder to read and easier to break than what you had before.

The cause is that textbooks teach the mechanism of concurrency but not the motivation for it. "Sum 1 to 100 using three goroutines" explains the machinery correctly, but it has nothing to do with why production code uses concurrency. Real work requires decisions: should this function be concurrent? Is this a channel or a mutex? What buffer size, and why?

This article explains Go concurrency starting from those decisions.

Three misconceptions to clear up first

Three common misconceptions make the patterns below look like an unrelated collection of tricks, so I correct them first.

In production server applications, the main reasons to introduce concurrency are overlapping wait time and controlling lifecycle, not saturating the CPU. Profile a web application and most of the wall clock is spent waiting on a database, an external API, or storage. CPU parallelism matters only for CPU-bound work such as image processing or cryptography.

The net/http server starts a goroutine per connection (net/http/server.go calls go c.serve(connCtx) from Server.Serve; with HTTP/1.1 keep-alive, requests on one connection are handled sequentially inside that goroutine, and HTTP/2 spawns a goroutine per stream). Either way, handlers for different clients run at the same time. So the moment you write a Go web application, you are already writing a concurrent program. Every global variable, shared cache, and connection pool your handler touches is accessed concurrently. Whether to use concurrency is already decided; the question is where to add structure to a program that is already concurrent.

That is true, but it misstates the proportions. In production code, channels carry timing (when may I proceed?) and ownership (which goroutine may touch this value right now?) far more often than they carry data. <-ctx.Done() carries no data at all, and it is the channel operation written most often in practice.

The rest of the article is organized as follows:

1. A decision framework (Chapter 1) — should this be concurrent, and is this a channel or a mutex?
2. Six production patterns (Chapters 2–7) — fan-out, bounded concurrency and backpressure, lifecycle management, ownership serialization, batching, and pub/sub
3. A catalogue of failures (Chapter 8) — bugs that occur in production, and how to detect them as of Go 1.27
4. The toolbox (Chapter 9) — choosing among standard and quasi-standard libraries
5. A case study (Chapter 10) — building a webhook delivery worker step by step
6. Testing and observability (Chapter 11)

All code here is verified on Go 1.27 (it passes go vet and go test -race). This article covers only how application code uses concurrency. For runtime internals (the G/M/P model, work stealing, the netpoller), see the companion article "How Go's Scheduler Works".

Chapter 1: A decision framework

1.1 The four motivations for concurrency

Observe any real codebase and concurrency shows up for roughly four reasons. Different motivations call for different structures, so name the one you are after before you start writing code.

Parallelizing CPU-bound computation is a fifth motivation, but it is rare in a typical web backend. When it does apply, it usually collapses into "split the work across GOMAXPROCS workers", and the backpressure and lifecycle discussions later in this article matter much less.

① and ② are different goals. ① (latency) is achieved by splitting one job into pieces dispatched simultaneously; ② (throughput) is achieved by keeping many jobs flowing without clogging. Starting unbounded goroutines to get ① breaks ② (Chapter 3).

1.2 Channel or mutex?

Go has a famous slogan:

Don't communicate by sharing memory; share memory by communicating.

The slogan states Go's design philosophy, but it does not mean "always use a channel." The official Go Wiki: Use a sync.Mutex or a channel? quotes the slogan and answers which to use:

Use whichever is most expressive and/or most simple.

That page lists channels as suited to "passing ownership of data, distributing units of work, communicating async results", and mutexes as suited to "caches, state". It also warns explicitly:

A common Go newbie mistake is to over-use channels and goroutines just because it's possible, and/or because it's fun.

Translated into working criteria:

• what you are protecting is the data structure itself (a map, a counter, a config value)
• the critical section is short and performs no I/O
• operations reduce to "read" and "write", and order carries no meaning

• you are transferring ownership of a value to another goroutine (after which the sender no longer touches it)
• you need to communicate timing (done, cancelled, time elapsed)
• you need to regulate flow by making the producer wait (backpressure)
• you need to join several wait conditions with select

Only channels can do the last of these. select is the only construct that safely expresses "react to whichever of these event sources fires first". A mutex cannot express waiting simultaneously on "the result arrived", "we timed out", and "we were cancelled".

1.3 Measuring the cost — channels are not cheap

To put numbers on the decision, here is the same "increment a counter" operation implemented three ways.

type mutexCounter struct {
    mu sync.Mutex
    n  int64
}
 
func (c *mutexCounter) Inc() {
    c.mu.Lock()
    c.n++
    c.mu.Unlock()
}
 
type atomicCounter struct{ n atomic.Int64 }
 
func (c *atomicCounter) Inc() { c.n.Add(1) }
 
// The "one goroutine owns the state" actor approach
type chanCounter struct {
    inc  chan struct{}
    stop chan struct{}
    n    int64
}
 
func newChanCounter() *chanCounter {
    c := &chanCounter{inc: make(chan struct{}), stop: make(chan struct{})}
    go func() {
        for {
            select {
            case <-c.inc:
                c.n++
            case <-c.stop:
                return
            }
        }
    }()
    return c
}
 
func (c *chanCounter) Inc() { c.inc <- struct{}{} }
 
// Note: this benchmark harness never closes `stop` — deliberately, since its
// lifetime is the measuring process. Production code should follow 4.2 rule 2.

Contended via b.RunParallel:

goos: windows, goarch: amd64, go1.27
cpu: 12th Gen Intel(R) Core(TM) i7-12700K
GOMAXPROCS=8, -benchtime=2s -count=5 (median)
 
BenchmarkCounterAtomic-8      14.3 ns/op
BenchmarkCounterMutex-8       50.2 ns/op
BenchmarkCounterChannel-8    441.9 ns/op

Absolute values vary a lot by environment, and this measurement is itself noisy (five runs on the same machine spread by roughly ±30%), so read only the orders of magnitude. For a simple state update, the channel-actor approach is roughly an order of magnitude slower than a mutex and about 30× slower than an atomic. Sends and receives on an unbuffered channel park and unpark goroutines (handing control to the scheduler), so this cost comes with the structure and cannot be optimized away.

That does not mean "channels are slow, avoid them." Use channels to express structures worth a few hundred nanoseconds per operation, and do not pay that cost for plain mutual exclusion.

If you use a channel once per request, 441 ns is irrelevant: network I/O is measured in milliseconds, three to four orders of magnitude larger. Conversely, if you are updating a counter millions of times per second inside a hot loop, choose an atomic over a mutex.

1.4 The decision flow

The bottom two nodes are checks for every path that starts goroutines: did you thread ctx through, and does every goroutine have a way out? Every goroutine-leak cause in 8.1 comes from code that skipped one of these two checks.

Chapter 2: Pattern ① fan-out / fan-in — overlapping wait time

2.1 Where it shows up

Overlapping wait time (motivation ①) is the most common reason for concurrency in production code and the easiest to recognize. The typical case is a BFF (Backend for Frontend) or an aggregation API.

To render one screen you need user info, order history, recommendations, and banners from four different services. Each takes 100 ms. Serially that is 400 ms; concurrently it is a little over 100 ms.

// Serial: 400ms
func (c *Clients) BuildPageSerial(ctx context.Context, uid string) (*Page, error) {
    u, err := c.User.Get(ctx, uid)
    if err != nil {
        return nil, err
    }
    o, err := c.Order.List(ctx, uid)
    if err != nil {
        return nil, err
    }
    r, err := c.Reco.For(ctx, uid)
    if err != nil {
        return nil, err
    }
    b, err := c.Banner.Fetch(ctx)
    if err != nil {
        return nil, err
    }
    return &Page{User: u, Orders: o, Recos: r, Banners: b}, nil
}

The four calls can run concurrently only because there is no data dependency among them. If fetching o requires u.TenantID, those two must stay serial; the depth of the dependency graph sets a lower bound on latency.

2.2 The naive version and its four holes

Most people write this first:

// Problematic
func (c *Clients) BuildPageNaive(ctx context.Context, uid string) (*Page, error) {
    var (
        wg   sync.WaitGroup
        page Page
        mu   sync.Mutex
        errs []error
    )
    wg.Go(func() {
        u, err := c.User.Get(ctx, uid)
        mu.Lock()
        defer mu.Unlock()
        if err != nil {
            errs = append(errs, err)
            return
        }
        page.User = u
    })
    wg.Go(func() {
        o, err := c.Order.List(ctx, uid)
        mu.Lock()
        defer mu.Unlock()
        if err != nil {
            errs = append(errs, err)
            return
        }
        page.Orders = o
    })
    wg.Wait()
    return &page, errors.Join(errs...)
}

(wg.Go is the method added in Go 1.25. It rules out the classic bug of mismatched wg.Add(1) / defer wg.Done() calls, so new code should always prefer it.)

The code works, but it is not ready for production, for four reasons.

1. Cancellation does not propagate. If the user fetch fails at 500 ms, the order fetch still runs to completion. The user will never see the result, but the load on downstream services continues.
2. Error meaning is lost. Bundling everything with errors.Join leaves the caller unable to distinguish required data from optional data.
3. A panic kills the process. Only a recover inside the panicking goroutine can stop it; the caller's defer recover() cannot catch it.
4. The lock is guarding the wrong thing. page.User and page.Orders are distinct memory locations, so the page fields need no lock. What mu actually protects here is the append to errs (see 2.4).

2.3 Doing it with errgroup

golang.org/x/sync/errgroup is the quasi-standard library for this. Its API is five functions:

func WithContext(ctx context.Context) (*Group, context.Context)
func (g *Group) Go(f func() error)
func (g *Group) TryGo(f func() error) bool
func (g *Group) SetLimit(n int)
func (g *Group) Wait() error

The ctx returned by errgroup.WithContext is cancelled the first time a function passed to Go returns a non-nil error, or the first time Wait returns, whichever comes first. Cancellation on the first error is what the naive version lacked. Cancellation when Wait returns has a consequence too: reusing that ctx after g.Wait() yields context.Canceled even when every task succeeded. The derived context lives only as long as the group.

func (c *Clients) BuildPage(ctx context.Context, uid string) (*Page, error) {
    g, ctx := errgroup.WithContext(ctx) // use this ctx from here on
    var page Page
 
    // Required: if either fails, there is no page to render
    g.Go(func() error {
        u, err := c.User.Get(ctx, uid)
        if err != nil {
            return err
        }
        page.User = u
        return nil
    })
    g.Go(func() error {
        o, err := c.Order.List(ctx, uid)
        if err != nil {
            return err
        }
        page.Orders = o
        return nil
    })
 
    // Optional: failure must not sink the whole page (return nil)
    g.Go(func() error {
        rctx, cancel := context.WithTimeout(ctx, 80*time.Millisecond)
        defer cancel()
        r, err := c.Reco.For(rctx, uid)
        if err != nil {
            slog.WarnContext(ctx, "recommendation degraded", "err", err)
            return nil // this is the design decision
        }
        page.Recos = r
        return nil
    })
 
    if err := g.Wait(); err != nil {
        return nil, err
    }
    return &page, nil
}

Overwriting the original ctx prevents accidentally using the non-cancelling one further down. If you do have work that must not be swept up in the group's cancellation, keep the original under a different name.

errgroup's semantics are "if one fails, abandon everything." That is right for required data and too strong for optional data, and "show the page even if recommendations are unavailable" is a common requirement. In that case, swallow the error inside the goroutine (but log it) and return nil.

The error you return to errgroup means "this subtask's failure is a reason to abort the whole group", not "an error occurred". Confuse the two and a transient failure in an optional dependency turns the whole page into a 500.

A tighter deadline than the required path's prevents nice-to-have data from holding up everything else. Above, recommendations are abandoned after 80 ms.

Even errgroup leaves hole 3 from 2.2 (panic) open. It deliberately does not propagate panics from its goroutines, so a panicking g.Go function still takes the process down. The reasoning and the remedy are in 3.2 and 8.7.

2.4 Why page needs no lock

Different goroutines write page.User and page.Orders, and that is not a race. Under the Go memory model, distinct struct fields are distinct memory locations, and concurrent writes to distinct memory locations do not conflict. The reader only touches page after g.Wait(), whose synchronization edge makes all those writes visible.

This reasoning holds only under these conditions:

• each goroutine writes a disjoint set of fields
• the fields do not share a slice or map (e.g. page.Orders and page.Recos must not alias the same backing array)
• reads happen strictly after Wait()

By contrast, multiple goroutines appending to the same slice is a data race (append reads and writes both the length and the backing array). When you need to collect results, either write to fixed indices or aggregate through a channel.

// OK: each goroutine writes only its own index
results := make([]Result, len(inputs))
g, ctx := errgroup.WithContext(ctx)
for i, in := range inputs {
    g.Go(func() error {
        r, err := fetch(ctx, in)
        if err != nil {
            return err
        }
        results[i] = r // distinct index = distinct memory location
        return nil
    })
}

(This loop is safe to write this way because Go 1.22 made loop variables per-iteration. On Go 1.21 and earlier, i, in := i, in was mandatory, and modules whose go.mod declares go 1.21 or earlier still need it (8.6).)

2.5 Fan-out amplifies tail latency

I described fan-out as "400 ms → 100 ms", but production demands a more pessimistic estimate. The join point waits for the slowest of the four calls, so if each downstream has a p99 of 500 ms, the p99 of four concurrent calls is worse than 500 ms. One slow dependency caps the benefit of the whole fan-out.

If four independent calls are each slow 1% of the time, the probability that at least one is slow is 1 - 0.99^4 ≈ 3.9%. This amplification is why the per-call timeout in 2.3 matters.

Countermeasures, in order of preference:

1. Reduce the required set — must you really have all of it before responding?
2. Give optional work a short timeout — enforce the ceiling structurally
3. Cache away the downstream call entirely — most effective, but the most expensive to build
4. Send hedged requests (fire a second attempt after a delay) — effective, but always cap the fraction, since it adds downstream load

2.6 errgroup versus sync.WaitGroup

Go 1.25's WaitGroup.Go made the two look alike in code, but the rule is simple: use errgroup for tasks that return errors and WaitGroup for tasks that do not.

Hand-rolling error aggregation onto WaitGroup to avoid a dependency rarely pays off; golang.org/x/sync is maintained by the Go team.

Chapter 3: Pattern ② bounded concurrency and backpressure

3.1 The most common concurrency incident

The fan-out in Chapter 2 was safe because "four" is a fixed, small number. This chapter is about the case where the count is determined by the input data.

// The code that takes down production
func ProcessAllUnbounded(ctx context.Context, items []Item) {
    var wg sync.WaitGroup
    for _, it := range items {
        wg.Go(func() { _ = process(ctx, it) })
    }
    wg.Wait()
}

Ten items is fine. A hundred thousand takes production down. Goroutines themselves are cheap: the initial stack is 2 KB, and since Go 1.19 the runtime may start goroutines with a larger stack based on the average stack usage it has observed. The trouble is the resources each one holds.

• Connection pool exhaustion — with SetMaxOpenConns(20), 100,000 arrivals leave 99,980 blocked waiting for a connection. That wait eventually exceeds your application timeout and every request errors
• Unintentional DoS on downstream services — even if you survive, the callee dies under 100,000 concurrent connections, and the failure cascades
• Memory — 1 MB of buffer per goroutine is 100 GB. Goroutines are light; payloads are not
• File descriptor exhaustion — too many open files disables the whole process

What makes this dangerous is that the code never fails in development or staging, because the test fixture has ten rows. It fails only in production.

3.2 Three ways to bound it

func ProcessAllErrgroup(ctx context.Context, items []Item, limit int) error {
    g, ctx := errgroup.WithContext(ctx)
    g.SetLimit(limit)
    for _, it := range items {
        // Blocks here once `limit` goroutines are in flight
        g.Go(func() error { return process(ctx, it) })
    }
    return g.Wait()
}

After SetLimit(n), g.Go blocks while n goroutines are running. You get error aggregation, cancellation, and bounding together, so this covers most cases.

There is a caveat. As the implementation shows (errgroup.go), g.Go waits on a plain channel send:

func (g *Group) Go(f func() error) {
    if g.sem != nil {
        g.sem <- token{} // not a select — ctx is not consulted
    }
    ...
}

So if ctx is cancelled while g.Go is blocked, g.Go does not return until a slot frees up. If you are feeding a large number of long-running tasks and need responsive cancellation, use TryGo (returns false immediately when full) or make the wait explicit with select, as in Option 2.

As noted in 2.3, errgroup does not propagate panics from its goroutines; the source comments give the reasons (delaying a panic makes bugs harder to find, it hides the stack from crash-monitoring tools, and so on). If the function you hand to g.Go panics, the process dies. Mitigation is in 8.7.

SetLimit also has a precondition: the limit must not change while goroutines in the group are active (changing an existing limit then panics). Always set it before the first Go.

func ProcessAllSemaphoreChan(ctx context.Context, items []Item, limit int) error {
    sem := make(chan struct{}, limit)
    var (
        wg   sync.WaitGroup
        mu   sync.Mutex
        errs []error
    )
    for _, it := range items {
        select {
        case sem <- struct{}{}: // take a slot (blocks when full)
        case <-ctx.Done():
            wg.Wait()
            return ctx.Err()
        }
        wg.Go(func() {
            defer func() { <-sem }() // return the slot
            if err := process(ctx, it); err != nil {
                mu.Lock()
                errs = append(errs, err)
                mu.Unlock()
            }
        })
    }
    wg.Wait()
    return errors.Join(errs...)
}

The number of values make(chan struct{}, limit) can hold is the concurrency limit. Unlike Option 1, this version uses select to wait for "a slot" and "cancellation" at the same time.

This idiom also shows Misconception 3 in practice: the struct{}{} being sent carries no data. What travels is permission to proceed.

func ProcessWeighted(ctx context.Context, items []Item, budgetMB int64) error {
    sem := semaphore.NewWeighted(budgetMB)
    g, ctx := errgroup.WithContext(ctx)
    for _, it := range items {
        // A weight above budgetMB makes Acquire wait forever (see below) — always clamp
        cost := min(estimateMB(it), budgetMB)
        if err := sem.Acquire(ctx, cost); err != nil {
            break // ctx cancelled
        }
        g.Go(func() error {
            defer sem.Release(cost)
            return process(ctx, it)
        })
    }
    return g.Wait()
}

Use this when the constraint is "at most 2 GB in flight" rather than "at most 10 at once" — video transcoding, batch aggregation, any workload where per-task cost varies by orders of magnitude.

The min clamp is a required defence. As semaphore.go shows, Acquire(ctx, n) with an n larger than the semaphore's total capacity does not return an error. It blocks on <-ctx.Done(), since the request can never be satisfied even when the semaphore is empty. Submit a 3 GB item against a 2 GB budget and the loop stalls until cancellation, so the caller must clamp or reject oversized items up front.

3.3 Choosing the limit — do not guess

A lot of code sets limit to "10, I guess" or to the number of CPUs. The limit can instead be derived.

Cap at runtime.GOMAXPROCS(0). Beyond that you only add context switches. Since Go 1.25 the default GOMAXPROCS accounts for Linux cgroup CPU limits, so the value is trustworthy in containers too (see "How Go's Scheduler Works").

This is by far the more common case. Your concurrency ceiling is set by what the downstream can absorb, not by your CPU. Use Little's Law:

L = λ × W
 
L : number of jobs in the system at once (= required concurrency)
λ : throughput (items/sec)
W : residence time per item (sec)

For example, suppose the downstream API is contractually capped at 200 req/s and its mean response time is 50 ms.

L = 200 × 0.05 = 10

Concurrency 10 saturates exactly 200 req/s. Raising it to 100 just gets you rate-limited: no extra throughput, only errors and retries. Dropping it to 5 gets you 100 req/s.

(Strictly, this conversion assumes W is constant. Overshooting the contracted rate when the downstream gets faster is covered in 3.3.1.)

This calculation derives the concurrency setting from a contract you must honour, so when the downstream allowance changes, you know the concurrency should change with it.

Another common constraint is the database connection pool:

MaxOpenConns per app instance = 20
connections held by one task    = 1
→ DB-touching workers must stay below 20 concurrent
   (remember the HTTP handlers share the same pool)

A classic incident: batch workers and online requests share one pool, nobody notices, and the API goes down the moment a batch runs.

3.3.1 Concurrency limits and rate limits are different things

Many implementations conflate these two. Look at Little's Law again:

L = λ × W

• A semaphore (SetLimit, semaphore.Weighted) bounds L, the number of jobs in flight at once
• A rate limiter bounds λ, the number of jobs you start per unit time

A semaphore alone cannot honour a limit on λ. The calculation in 3.3 (concurrency 10 ⇒ 200 req/s) holds only while W stays at 50 ms. If the downstream gets faster and W drops to 5 ms, the same concurrency of 10 emits 2,000 req/s and hits the rate limit.

// golang.org/x/time/rate
 
// 200 req/s sustained, bursts of up to 20
limiter := rate.NewLimiter(rate.Limit(200), 20)
 
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(10) // L: in-flight count (pool and memory constraints)
 
for _, it := range items {
    g.Go(func() error {
        // λ: start rate. Blocks until a token is available (ctx-interruptible)
        if err := limiter.Wait(ctx); err != nil {
            return err
        }
        return process(ctx, it)
    })
}

rate.Limiter's three methods map directly onto the three strategies in 3.5:

Reserve() needs care: it consumes the token immediately. If you inspect the returned *Reservation's Delay() and decide not to wait, you must call r.Cancel() to hand the token back, or your effective rate quietly degrades. (ReserveN(t, n) with n above the burst returns a reservation whose OK() is false.)

The burst argument (20 above) is a design decision of the same kind as buffer size in 3.4: it declares how many calls may go out back-to-back in an instant.

In practice you usually need both. A rate limiter alone lets in-flight work pile up without bound when the downstream stalls (λ is honoured while L diverges). A semaphore alone exceeds the contracted rate whenever the downstream speeds up. If you see an implementation with only one, check whether the other constraint really is absent.

3.4 What backpressure actually is

Backpressure stays invisible as long as you think of a channel as "an async queue".

An unbuffered channel means complete backpressure.

ch := make(chan Item) // unbuffered
ch <- item            // blocks until a receiver takes it

The sender cannot proceed until a receiver accepts, so the producer is forced to keep to the consumer's pace. When production outpaces consumption, the queue cannot grow without bound and exhaust memory.

A buffered channel declares that you will absorb a burst of N items.

ch := make(chan Item, 100) // sender doesn't block for the first 100

A frequent misconception is "bigger buffer, faster." A buffer can raise the rate you actually attain, but it does not raise the ceiling. The ceiling is the consumer's service rate μ.

A buffer buys two things:

1. absorbing transient bursts — accepting a momentary inflow above μ on the promise of digesting it later
2. decoupling producer and consumer scheduling — amortizing the goroutine park/unpark paid at every rendezvous (the few hundred ns measured in 1.3) so the two overlap

(2) is a measurable win in a pipeline where both sides do real work. But if production persistently outruns consumption, the buffer fills sooner or later, and from then on the channel behaves exactly like an unbuffered one.

Buffer size is therefore a design decision about how much backlog you will tolerate, not a performance knob.

In practice, derive buffer size from how many seconds of backlog you accept. If an item takes 10 ms and you accept up to one second of delay, the buffer is 100. That number can be explained; "100 seems fine" cannot.

3.5 What to do when it is full — three strategies

Designing backpressure means deciding explicitly what happens when the queue is full. Code that has not decided has implicitly chosen "block".

// Strategy A: block (push backpressure upstream)
select {
case queue <- item:
case <-ctx.Done():
    return ctx.Err()
}
 
// Strategy B: drop (load shedding)
select {
case queue <- item:
default:
    metrics.Dropped.Inc()
    return ErrOverloaded
}
 
// Strategy C: wait with a deadline, then give up
timer := time.NewTimer(50 * time.Millisecond)
defer timer.Stop()
select {
case queue <- item:
case <-timer.C:
    metrics.Dropped.Inc()
    return ErrOverloaded
case <-ctx.Done():
    return ctx.Err()
}

The choice follows from the nature of the data.

Dropping (strategy B) can look like giving up, but it is a deliberate design choice. Blocking unconditionally when full causes upstream request handlers to pile up, goroutine count and memory to climb, and eventually the whole service to stop responding: the textbook cascading failure. Shedding load with an early 503 keeps the system as a whole healthier.

Strategy B's default clause means "give up immediately", so putting it in a for loop produces a busy loop. If you want to retry, always give it something to wait on, as in strategy C.

3.6 Making the queue observable

You can only tell whether a backpressure design was right at runtime. Exporting len(ch) and cap(ch) as metrics is a cheap way to find out. (The metrics below refers to your application's metrics package, e.g. a Prometheus client.)

func (d *Dispatcher) observeQueues(ctx context.Context, interval time.Duration) {
    t := time.NewTicker(interval)
    defer t.Stop()
    for {
        select {
        case <-ctx.Done():
            return
        case <-t.C:
            for i, lane := range d.lanes {
                metrics.QueueDepth.WithLabelValues(strconv.Itoa(i)).
                    Set(float64(len(lane)))
            }
        }
    }
}

How to read it:

In the third case, enlarging the buffer only hides the problem and worsens latency. Backlogged items are also stale items whose data has lost value while waiting.

Go 1.26 added a per-state goroutine breakdown to runtime/metrics, so runtime saturation is directly readable too. (This metrics is the standard library's runtime/metrics.)

// runtime/metrics
 
samples := []metrics.Sample{
    {Name: "/sched/goroutines:goroutines"},          // total (Go 1.16+)
    {Name: "/sched/goroutines/runnable:goroutines"}, // ready but not running (Go 1.26+)
    {Name: "/sched/goroutines/running:goroutines"},  // running (Go 1.26+)
    {Name: "/sched/goroutines/waiting:goroutines"},  // waiting on I/O or sync (Go 1.26+)
    {Name: "/sched/latencies:seconds"},              // scheduler wait distribution (Go 1.17+)
}
metrics.Read(samples)

Reading guide:

• chronically high runnable → you are out of CPU. Adding concurrency makes it worse
• waiting high but flat, runnable low, CPU idle → I/O-dominated. This is where adding concurrency helps
• waiting growing without bound → a leak (8.1) or a stalled downstream. Adding concurrency makes it worse; read the stacks in the goroutine profile
• a lengthening tail on /sched/latencies:seconds → goroutines are being made to wait by the scheduler

With these, you can answer "should I raise the concurrency?" with numbers instead of intuition.

Chapter 4: Pattern ③ lifecycle management

4.1 <-ctx.Done() is the channel you actually use

I opened by saying that you cannot find anywhere to use a channel in work code. In fact most people already use one every day: context.Context's Done().

type Context interface {
    Deadline() (deadline time.Time, ok bool)
    Done() <-chan struct{}   // this is a channel
    Err() error
    Value(key any) any
}

The design of context relies on two channel properties.

This property makes one-to-many broadcast possible. When cancel() runs, it executes close(done), and every goroutine waiting on <-ctx.Done() is released at once. A sent value would reach only one receiver, so close is the only correct mechanism here.

The standard library (context/context.go) does exactly that:

// excerpt from cancelCtx.cancel
func (c *cancelCtx) cancel(removeFromParent bool, err, cause error) {
    // ...
    c.mu.Lock()
    // ...
    d, _ := c.done.Load().(chan struct{})
    if d == nil {
        c.done.Store(closedchan)
    } else {
        close(d)          // wake every waiter at once
    }
    for child := range c.children {
        child.cancel(false, err, cause) // propagate recursively
    }
    // ...
}

context.Background()'s Done() returns nil. A select arm waiting on a nil channel is never chosen, so "a context that is never cancelled" needs no special case. This property reappears as a production idiom in 8.4.

4.2 Three rules for cancellation

Code that accepts a ctx and fails to hand it to downstream calls has severed the chain. The contextcheck linter finds these mechanically (go vet's lostcancel covers only Rule 3: it reports an uncalled cancel, not a missing propagation).

A function that launches a goroutine is responsible for giving that goroutine a path by which it reliably terminates. A go statement whose end you cannot explain is a bug.

Go 1.21 added a tool for this rule. A watchdog goroutine that exists only to clean up when ctx is cancelled is easy to leak:

// Commonly written, easily leaked
go func() {
    select {
    case <-ctx.Done():
        conn.Close()
    case <-done: // forget to close this and the goroutine stays
    }
}()

context.AfterFunc replaces it:

stop := context.AfterFunc(ctx, func() { conn.Close() })
defer stop() // deregister

With AfterFunc, no goroutine sits parked waiting on ctx. f is only registered in the parent's children set and gets its own goroutine at cancellation time (context.go's afterFuncCtx.cancel calls a.once.Do(func() { go a.f() })), so there is no waiting goroutine to leak.

Read stop()'s return value conservatively. The documentation says it returns true if it stopped f from running, and that false means "either the context is canceled and f has been started in its own goroutine, or f was already stopped". It also says that stop does not wait for f to complete. So true tells you the cleanup had not yet started; false tells you nothing about whether it finished. If you need to know, coordinate with f explicitly.

If you never call stop(), f and everything its closure captures stay alive for as long as the parent context lives. This is the same kind of trap as a missing defer cancel() under Rule 3.

ctx, cancel := context.WithTimeout(parent, 3*time.Second)
defer cancel() // call it even when you finish early

If you skip cancel(), the child context stays registered in the parent's children map, and its memory is not released until the parent is cancelled or the child's own timeout fires. It leaks until then. go vet catches this.

4.3 Layering timeouts

Timeouts nest:

// 3 seconds for the whole handler
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
defer cancel()
 
// 1 second for the database within it
dbCtx, dbCancel := context.WithTimeout(ctx, 1*time.Second)
defer dbCancel()

context.WithTimeout cannot extend a parent's deadline. If the parent has 3 seconds, a child given 10 seconds still expires at 3, so the outermost timeout is always a hard ceiling for the whole call tree.

Go 1.21's WithTimeoutCause is useful here:

var ErrDBSlow = errors.New("db call exceeded budget")
 
dbCtx, dbCancel := context.WithTimeoutCause(ctx, time.Second, ErrDBSlow)
defer dbCancel()
 
if err := db.QueryRowContext(dbCtx, q).Scan(&v); err != nil {
    // ctx.Err() is always context.DeadlineExceeded,
    // but context.Cause(dbCtx) returns ErrDBSlow
    if errors.Is(context.Cause(dbCtx), ErrDBSlow) {
        // distinguish "the DB was slow" from "the overall budget expired"
    }
}

ctx.Err() only ever says context.DeadlineExceeded, so it cannot tell you which layer expired. Cause can, which makes incident analysis much easier.

4.3.1 A cancellation error has four different meanings

The sections above covered how to propagate cancellation. In production, interpreting a cancellation you observe is harder. When a handler sees context.Canceled or context.DeadlineExceeded, there are at least four possible causes, and each needs a different response.

Test ④ before ①. In 4.5's wiring, BaseContext supplies workerCtx, so stopWorkers() also cancels the r.Context() of every in-flight handler. That makes r.Context().Err() != nil true for both ① and ④, so testing ① first misclassifies a shutdown as a client disconnect and skips the 503 you owe the caller.

Confusing ① with ② and ③ is the most common mistake in practice. A user closing their browser gets counted as a server error, and the error-rate graph appears to spike on every deploy. Some of the "we implemented graceful shutdown but 5xx didn't drop" reports come down to this. A client-side disconnect is not your failure in the first place.

Case ② can be distinguished because of how errgroup is implemented. In errgroup.go, WithContext builds its context with context.WithCancelCause and calls g.cancel(g.err) on failure, so a sibling goroutine can learn which error stopped the group via context.Cause(ctx).

g, ctx := errgroup.WithContext(context.Background())
 
g.Go(func() error {
    return errBoom // somebody fails
})
g.Go(func() error {
    <-ctx.Done()
    // ctx.Err()          → context.Canceled (no information)
    // context.Cause(ctx) → errBoom (the real cause)
    return fmt.Errorf("aborted: %w", context.Cause(ctx))
})

Running this on Go 1.27 with x/sync v0.23.0 yields ctx.Err()=context canceled alongside context.Cause(ctx)=boom. When the fan-out in 2.3 fails and you want the subtask side to log why the page could not be built, that one line does it.

On the sending side, context.WithCancelCause lets you attach a reason:

ctx, cancel := context.WithCancelCause(parent)
defer cancel(nil)
 
// ... when something goes wrong
cancel(fmt.Errorf("upstream quota exhausted"))
// downstream reads it back with context.Cause(ctx)

Where 4.3's WithTimeoutCause records why a deadline was set, WithCancelCause records why someone cancelled. With both, you are no longer limited to the bare context.Canceled / DeadlineExceeded, which carry no information about the cause.

4.4 Work that must outlive the request

Writing fire-and-forget work like this is a very common bug:

// Bug: ctx is cancelled the instant the handler returns
func handler(w http.ResponseWriter, r *http.Request) {
    doMainWork(r.Context())
    go auditLog(r.Context(), "action performed") // almost certainly context canceled
    w.WriteHeader(http.StatusOK)
}

r.Context() is cancelled when the handler returns, so the audit write usually fails with context canceled. Go 1.21's context.WithoutCancel exists for this.

func handler(w http.ResponseWriter, r *http.Request) {
    doMainWork(r.Context())
 
    // keep the values (trace ID, etc.), detach only cancellation
    bg := context.WithoutCancel(r.Context())
    go func() {
        ctx, cancel := context.WithTimeout(bg, 5*time.Second) // always add a deadline
        defer cancel()
        if err := auditLog(ctx, "action performed"); err != nil {
            slog.ErrorContext(ctx, "audit log failed", "err", err)
        }
    }()
 
    w.WriteHeader(http.StatusOK)
}

WithoutCancel preserves Values such as trace IDs while detaching cancellation and deadline. Always set a fresh timeout; otherwise each request leaves behind a goroutine with no deadline, which leads to the leaks in Chapter 8.

This go func() is lost when the process stops. If the data must not be lost, put it in a durable queue (or the worker set assembled in 4.5) rather than a bare goroutine. Fire-and-forget is only for work you can afford to lose.

4.5 Graceful shutdown — step order and time budget

On Kubernetes, pod termination sends SIGTERM and then SIGKILLs after terminationGracePeriodSeconds (30 by default). What must happen in between is well defined, and it comes down to the order of the steps.

Reversing the order causes incidents. Stop the workers first and the requests you are still accepting cannot be served. Close the database first and every in-flight request dies.

func RunServer(handler http.Handler, workers []*Worker) error {
    // cancelled on SIGTERM / SIGINT
    rootCtx, stop := signal.NotifyContext(
        context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()
 
    // Workers must stop *after* the entrance closes, so give them
    // a separate lineage from rootCtx.
    workerCtx, stopWorkers := context.WithCancel(context.Background())
    defer stopWorkers()
 
    srv := &http.Server{
        Addr:              ":8080",
        Handler:           handler,
        ReadHeaderTimeout: 5 * time.Second,
        BaseContext:       func(net.Listener) context.Context { return workerCtx },
    }
 
    g, gctx := errgroup.WithContext(rootCtx)
 
    g.Go(func() error {
        if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
            return err
        }
        return nil
    })
 
    for _, w := range workers {
        g.Go(func() error { return w.Run(workerCtx) })
    }
 
    // the shutdown goroutine
    g.Go(func() error {
        <-gctx.Done() // a signal, or any task failing
 
        // steps 1-2: close the entrance, wait for in-flight work.
        // The timeout comes from the whole-sequence budget (Decision 2)
        shutCtx, cancel := context.WithTimeout(context.Background(), 7*time.Second)
        defer cancel()
        if err := srv.Shutdown(shutCtx); err != nil {
            slog.Error("graceful shutdown timed out", "err", err)
            _ = srv.Close() // give up and force-close
        }
 
        // step 3: stop workers only after the entrance is closed
        stopWorkers()
        return nil
    })
 
    if err := g.Wait(); err != nil && !errors.Is(err, context.Canceled) {
        return err
    }
    return nil
}

Decision 1: use signal.NotifyContext. It is shorter than hand-rolling signal.Notify with a channel, and it hands downstream code a ctx.

You must finish before SIGKILL, and the grace period has to cover the whole sequence, not just Shutdown. Spend 25 s in Shutdown and then 15 s draining workers and you are at 40 s against a 30 s grace period: SIGKILLed mid-drain, losing exactly the data the drain existed to save.

terminationGracePeriodSeconds = 30s
  LB drain delay             5s   (4.6)
+ srv.Shutdown               7s   (4.5)
+ worker drain              10s   (10.2's drain)
+ post-budget cleanup        3s   (10.2's abandon)
+ headroom                   5s
= 30s

(10.2's lanes drain concurrently, so the 10 s and 3 s are a maximum across lanes, not a sum over them.)

Derive each stage's timeout from that budget. The code above uses 7 s for shutCtx. If Shutdown times out, Close() force-drops connections so that the process at least terminates on time.

Decision 3: srv.Shutdown does not cancel in-flight request contexts. Many people assume it does. Shutdown stops accepting, closes idle connections, and waits for active ones, leaving each running handler's r.Context() uncancelled (net/http/server.go's Shutdown closes listeners and idle connections, then polls until the connections go quiescent). So if a handler is stuck in an infinite loop, Shutdown itself keeps waiting, and in the code above only the 7-second shutCtx gets you out. That is why BaseContext supplies workerCtx: step 3's stopWorkers() then stops the background workers and also signals every in-flight handler, as a last resort when Shutdown does not return in time.

Decision 4: do not derive the worker context from rootCtx. If workers received rootCtx, they would stop the instant SIGTERM arrives, while HTTP requests are still in flight and may still enqueue jobs. Workers must stop after the entrance has fully closed, hence a separate lineage and a manual stopWorkers().

Decision 5: ignore context.Canceled from g.Wait(). A normal signal shutdown cancels gctx; treating that as an error yields exit code 1 and Kubernetes records an abnormal termination.

4.6 Do not forget to leave the load balancer

The load balancer adds one more step. If you close the listener immediately on SIGTERM, the load balancer may still be sending you traffic (Endpoints updates are asynchronous with process shutdown), so some requests get connection-refused.

The standard practice is to fail the readiness probe first, wait a few seconds, and only then start draining. Replace 4.5's shutdown goroutine with this:

// hardStop closes on a *second* SIGTERM / Ctrl-C. signal.NotifyContext
// swallows repeat signals until stop() is called, so an operator's second
// Ctrl-C does nothing unless you receive it yourself. This goroutine
// intentionally lives for the lifetime of the process.
hardStop := make(chan struct{})
go func() {
    sig := make(chan os.Signal, 2)
    signal.Notify(sig, os.Interrupt, syscall.SIGTERM)
    defer signal.Stop(sig)
    <-sig // first (shared with NotifyContext)
    <-sig // second = "don't wait, go down now"
    close(hardStop)
}()
 
g.Go(func() error {
    <-gctx.Done()
 
    // 1. fail readiness (leave the LB rotation)
    ready.Store(false)
 
    // 2. wait for the LB to notice (probe interval + margin)
    select {
    case <-time.After(5 * time.Second):
    case <-hardStop:
    }
 
    // 3. now drain (per Decision 2's budget)
    shutCtx, cancel := context.WithTimeout(context.Background(), 7*time.Second)
    defer cancel()
    if err := srv.Shutdown(shutCtx); err != nil {
        slog.Error("graceful shutdown timed out", "err", err)
        _ = srv.Close()
    }
 
    stopWorkers()
    return nil // as in Decision 5, a clean stop is not an error
})

Most reports of "we implemented graceful shutdown but still get 5xx on every deploy" trace back to this missing delay.

Chapter 5: Pattern ④ ownership serialization

5.1 Three signs a mutex has stopped fitting

Section 1.2 recommended a mutex for plain state protection. In practice, the following three signs tell you when to switch to a channel.

// Dangerous
func (c *Cache) Refresh(ctx context.Context, key string) error {
    c.mu.Lock()
    defer c.mu.Unlock()
 
    v, err := c.origin.Fetch(ctx, key) // hundreds of ms of network I/O under the lock
    if err != nil {
        return err
    }
    c.m[key] = v
    return nil
}

This works until the origin slows down, at which point every request serializes. Doing network I/O while holding a lock is one of the most expensive anti-patterns in production Go.

The first remedy for Sign 1 is to move the I/O out of the critical section, not to switch to an actor:

func (c *Cache) Refresh(ctx context.Context, key string) error {
    v, err := c.origin.Fetch(ctx, key) // I/O outside the lock
    if err != nil {
        return err
    }
    c.mu.Lock()
    c.m[key] = v // the lock covers only the assignment
    c.mu.Unlock()
    return nil
}

That resolves the serialization (add 9.1's singleflight on top if duplicate fetches for the same key concern you). Switching to an actor would not fix it, because I/O inside the owner loop stalls it in the same way (5.4). Actors are worth using when, after the I/O has been moved out, you still need to enforce the ordering of state transitions (Sign 3 below).

This is where deadlocks come from. A lock-ordering convention maintained by documentation always breaks down as a codebase grows.

Rules like "a connection only moves connecting → connected → draining → closed" or "writes after close are ignored" become state checks scattered across the top of every method when expressed with a mutex.

When Sign 2 or Sign 3 appears, or when Sign 1 survives after you have already moved the I/O out of the lock, switch to a structure where one goroutine owns the state and everyone else touches it only through a channel: the actor model.

5.2 Command channels and reply channels

The skeleton:

type entry struct {
    val       string
    expiresAt time.Time
}
 
// Commands as types
type cacheCmd interface{ isCacheCmd() }
 
type getCmd struct {
    key   string
    reply chan getResult // a dedicated reply channel
}
 
func (getCmd) isCacheCmd() {}
 
type getResult struct {
    val string
    ok  bool
}
 
type setCmd struct {
    key string
    val string
    ttl time.Duration
}
 
func (setCmd) isCacheCmd() {}
 
type Cache struct {
    cmds chan cacheCmd
    done chan struct{} // announces the owner goroutine has exited
}
 
func NewCache(ctx context.Context) *Cache {
    c := &Cache{
        cmds: make(chan cacheCmd),
        done: make(chan struct{}),
    }
    go c.loop(ctx)
    return c
}
 
// Done exposes "the owner has fully stopped" as public API.
// As 11.1 shows, testability requires it.
func (c *Cache) Done() <-chan struct{} { return c.done }

The core is a single for-select loop. Only this goroutine touches m, so no locks are needed.

func (c *Cache) loop(ctx context.Context) {
    defer close(c.done) // broadcast termination to all waiters
 
    m := make(map[string]entry)
    gc := time.NewTicker(time.Minute)
    defer gc.Stop()
 
    for {
        select {
        case <-ctx.Done():
            return
 
        case cmd := <-c.cmds:
            switch cmd := cmd.(type) {
            case setCmd:
                m[cmd.key] = entry{cmd.val, time.Now().Add(cmd.ttl)}
            case getCmd:
                e, ok := m[cmd.key]
                if ok && time.Now().After(e.expiresAt) {
                    delete(m, cmd.key)
                    ok = false
                }
                if !ok {
                    e.val = "" // never hand back an expired value
                }
                cmd.reply <- getResult{e.val, ok}
            }
 
        case now := <-gc.C:
            // periodic sweep; the mutex version would lock the whole map here
            for k, e := range m {
                if now.After(e.expiresAt) {
                    delete(m, k)
                }
            }
        }
    }
}

In the final case now := <-gc.C:, a time-driven chore sits in the same loop as state access. The mutex version would need a separate goroutine that locks the whole structure to sweep it, stalling the entire API meanwhile. Combining timers and input in one loop is something only channels and select can do.

5.3 The caller — always have three exits

At the public API, wrap every channel operation in a select with three escape routes.

var ErrCacheClosed = errors.New("cache: closed")
 
func (c *Cache) Get(ctx context.Context, key string) (string, bool, error) {
    reply := make(chan getResult, 1) // the buffer of 1 matters (below)
 
    // (1) send the command
    select {
    case c.cmds <- getCmd{key: key, reply: reply}:
    case <-c.done:
        return "", false, ErrCacheClosed
    case <-ctx.Done():
        return "", false, ctx.Err()
    }
 
    // (2) await the reply — a reply that already arrived wins over a concurrent shutdown
    select {
    case r := <-reply:
        return r.val, r.ok, nil
    default:
    }
 
    select {
    case r := <-reply:
        return r.val, r.ok, nil
    case <-c.done:
        return "", false, ErrCacheClosed
    case <-ctx.Done():
        return "", false, ctx.Err()
    }
}

This is the actor pattern's biggest trap. Running c.cmds <- cmd after the owner goroutine has exited blocks forever, since no one will ever receive. Without a timeout on ctx, that goroutine never comes back. A channel announcing the owner's liveness (here, done) is mandatory.

Omit it and, when shutdown races with a reply, you discard a correctly computed result and return ErrCacheClosed about half the time. The owner can complete cmd.reply <- ... (non-blocking, buffer 1), loop, take <-ctx.Done(), return, and fire defer close(c.done); this is an ordinary interleaving. At that instant the caller's select sees both reply and c.done ready and, per 8.3, picks one uniformly at random.

In 400 trials on my machine, exactly 200 returned a bogus ErrCacheClosed. Adding the non-blocking select first brings that to 0. This is 8.3 applied to the actor's reply path: the done arm must not win over a reply that has already been delivered.

Without the buffer in make(chan getResult, 1), if the caller abandons step (2) via ctx.Done() and the owner then executes cmd.reply <- ..., the owner loop blocks forever with no receiver. One stalled actor stalls every API call, a complete outage. With a buffer of 1 the owner always completes the send, and the value is simply garbage-collected unread.

As in 3.4, the buffer size states a design decision: the 1 guarantees that the owner never blocks on a reply.

5.4 Actor or mutex?

For a read-dominated cache, sync.RWMutex is better. Actors are the better fit when the specification is mainly a state machine: connection state machines, token buckets, job queues with ordering guarantees.

Slow work inside the actor also becomes the bottleneck of the whole system. Keep the owner loop to state updates, and dispatch I/O to another goroutine.

Chapter 6: Pattern ⑤ batching

6.1 Where it shows up

"Send when N items accumulate or T seconds pass, whichever comes first." This requirement is very common in production:

• shipping metrics and audit logs (hitting HTTP once per item kills the downstream)
• bulk INSERT into a database (per-row round trips dominate)
• external bulk endpoints (max 500 items per request)
• search index updates

It is also a typical case of work that only channels express cleanly. You must join two independent event sources, "input arrived" and "time elapsed", and select is the only natural way to write that.

6.2 Implementation

type Batcher struct {
    in       chan Event
    done     chan struct{}
    sink     Sink
    maxSize  int
    maxDelay time.Duration
}
 
func NewBatcher(sink Sink, queue, maxSize int, maxDelay time.Duration) *Batcher {
    return &Batcher{
        in:       make(chan Event, queue), // queue = the backlog you tolerate (3.4)
        done:     make(chan struct{}),
        sink:     sink,
        maxSize:  maxSize,
        maxDelay: maxDelay,
    }
}
 
func (b *Batcher) Run(ctx context.Context) error {
    defer close(b.done)
 
    batch := make([]Event, 0, b.maxSize)
    timer := time.NewTimer(b.maxDelay)
    defer timer.Stop()
 
    flush := func(reason string) {
        if len(batch) == 0 {
            return
        }
        // detach cancellation so we can still write during shutdown
        fctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
        defer cancel()
        if err := b.sink.WriteBatch(fctx, batch); err != nil {
            slog.Error("batch write failed", "reason", reason, "n", len(batch), "err", err)
        }
        batch = batch[:0]
    }
 
    for {
        select {
        case <-ctx.Done():
            // on stop: drain what is queued, then flush
            for {
                select {
                case e := <-b.in:
                    batch = append(batch, e)
                    if len(batch) >= b.maxSize {
                        flush("drain")
                    }
                    continue
                default:
                }
                break
            }
            flush("shutdown")
            return nil
 
        case e := <-b.in:
            batch = append(batch, e)
            if len(batch) == 1 {
                // measure max delay from when the *first* item arrived.
                // On Go 1.23+ no stale value survives Stop(), so the
                // <-timer.C drain must NOT be written (see 6.4)
                timer.Stop()
                timer.Reset(b.maxDelay)
            }
            if len(batch) >= b.maxSize {
                timer.Stop()
                flush("size")
            }
 
        case <-timer.C:
            flush("time")
        }
    }
}

6.3 The decisions embedded here

That is the len(batch) == 1 condition. Naively flushing "every 100 ms" means an event arriving at the 99 ms mark is shipped alone 1 ms later. Measuring from the batch's first item still ships every event within T of its arrival, and gives each batch the full T to fill.

The case <-ctx.Done(): branch drains the remainder. Without it, every deploy silently loses the in-flight batch (up to maxSize - 1) plus everything still buffered in b.in (up to cap(b.in)). This is the usual cause of "logs occasionally go missing".

This drain loop assumes the entrance is already closed. It exits via default: the first instant b.in is empty. If producers keep calling Add, they keep feeding it, and every maxSize boundary costs up to 5 s in flush, so stop time becomes unbounded. Follow 4.5's ordering (entrance → in-flight → workers) and this cannot happen; in any other arrangement, impose an explicit budget the way 10.2's drain does.

That is what context.WithoutCancel(ctx) does. During the shutdown flush, ctx is already cancelled, so a write using it fails instantly. Always pair WithoutCancel with WithTimeout, though, because a shutdown that never finishes just gets SIGKILLed.

var ErrBatcherClosed = errors.New("batcher: closed")
 
func (b *Batcher) Add(ctx context.Context, e Event) error {
    select {
    case b.in <- e:
        return nil
    case <-b.done: // 5.3 Point 1: always include the owner's liveness
        return ErrBatcherClosed
    case <-ctx.Done():
        return ctx.Err()
    }
}

b.in's buffer size is the backlog you tolerate (3.4). For droppable data such as metrics, a default: clause that discards is a legitimate design.

Omit the <-b.done arm and an Add after Run has returned either silently drops values into a buffer nobody drains (up to cap(b.in) of them) or blocks until the caller's own ctx expires. The rule imposed on the actor in 5.3 applies here for the same reason.

time.NewTimer(b.maxDelay) starts immediately, so if no event arrives within maxDelay of startup you get exactly one case <-timer.C: firing on an empty batch. It is harmless, since flush returns early when len(batch) == 0. If you want to avoid it, call timer.Stop() right after construction.

6.4 time.Timer etiquette — what changed in Go 1.23

The implementation in 6.2 just calls timer.Stop() and never drains an already-delivered value with <-timer.C. A great deal of existing code contains the classic if !t.Stop() { <-t.C } idiom, but Go 1.23 changed time.Timer's implementation in a way that makes that idiom wrong.

The Go 1.23 release notes list two changes:

1. Timers and Tickers that are no longer referenced become eligible for GC immediately, even without Stop. Previously an un-stopped Timer was not collected until it fired, and an un-stopped Ticker was never collected
2. Timer channels are now unbuffered (capacity 0). Go now guarantees that no stale value prepared before a Reset or Stop call is sent or received after it

Practical consequences:

• time.After inside a loop no longer leaks memory (Go 1.23+). Before that, time.After(time.Hour) in a for-loop select kept an hour's worth of timers alive
• the cost of allocating a timer each iteration remains, so hot loops still favour reusing a time.Timer or using context.WithTimeout
• on Go 1.23+, do not write the <-timer.C drain after Stop at all

The last point needs explanation, because it is most dangerous for people copying from older articles. On Go 1.23+, Stop() removes an already-delivered value from the channel if there is one, and returns true when it does. So Stop() returns false only when the channel is guaranteed to be empty. The classic

if !t.Stop() {
    <-t.C // On Go 1.23+, reaching here always deadlocks
}

therefore deadlocks whenever Stop() returns false (time/sleep.go states that "any receive from t.C after Stop has returned is guaranteed to block").

Adding a guard such as timerActive does not make it correct either, because the condition that makes the guard false is not the condition that makes the drain safe. Call timer.Stop() and write no drain.

Verified on Go 1.27:

tm := time.NewTimer(1 * time.Millisecond)
time.Sleep(20 * time.Millisecond) // make sure it fires
 
tm.Stop() // → true (it removed the delivered value)
// after this, <-tm.C blocks forever

The asynctimerchan GODEBUG (which restored the old behaviour) was removed entirely in Go 1.27. On current Go, timer channels are always unbuffered.

Chapter 7: Pattern ⑥ pub/sub and broadcast

7.1 Channels are one-to-one

One value sent to one channel reaches exactly one receiver, however many goroutines receive from that channel. (The runtime keeps waiters in a FIFO queue called recvq, so in practice the longest-waiting goroutine is woken; the language spec leaves the order unspecified, so do not depend on it.) This is correct behaviour for work distribution.

So "notify everyone" needs something else. Two options:

7.2 Broadcast via close

For an event that happens exactly once, this is the cheapest and most reliable mechanism.

type Signal struct {
    once sync.Once
    ch   chan struct{}
}
 
func NewSignal() *Signal { return &Signal{ch: make(chan struct{})} }
 
func (s *Signal) Done() <-chan struct{} { return s.ch }
 
// safe to call any number of times (double-close would panic)
func (s *Signal) Fire() { s.once.Do(func() { close(s.ch) }) }

The sync.Once wrapper is needed because closing an already-closed channel panics. In real code where shutdown can be triggered from several paths, a close without Once will eventually crash.

For shutdown, context already does this, so you rarely write it yourself. You need it for other one-shot events, such as initialization completing or leadership being acquired.

7.3 A subscription hub — SSE / WebSocket

For recurring events delivered to N subscribers, give each subscriber its own channel.

type Hub struct {
    mu   sync.RWMutex
    subs map[*subscriber]struct{}
 
    // RLock is a *shared* lock, so several Publish calls run concurrently.
    // A plain int64 here is a race that -race will report.
    dropped atomic.Int64
}
 
type subscriber struct {
    ch    chan Message
    unsub func()
}
 
func NewHub() *Hub {
    return &Hub{subs: make(map[*subscriber]struct{})}
}
 
func (h *Hub) Subscribe(buffer int) (<-chan Message, func()) {
    if buffer < 1 {
        // With buffer 0, a non-blocking send only succeeds while a receiver
        // is already parked — Publish would drop nearly everything.
        buffer = 1
    }
    s := &subscriber{ch: make(chan Message, buffer)}
 
    var once sync.Once
    s.unsub = func() {
        once.Do(func() {
            h.mu.Lock()
            delete(h.subs, s)
            h.mu.Unlock()
            close(s.ch) // only the sender closes
        })
    }
 
    h.mu.Lock()
    h.subs[s] = struct{}{}
    h.mu.Unlock()
 
    return s.ch, s.unsub
}
 
// Publish never blocks; slow subscribers lose messages.
func (h *Hub) Publish(m Message) {
    h.mu.RLock()
    defer h.mu.RUnlock()
    for s := range h.subs {
        select {
        case s.ch <- m:
        default:
            h.dropped.Add(1) // make it observable
        }
    }
}
 
// Always expose the drop count (7.4)
func (h *Hub) Dropped() int64 { return h.dropped.Load() }

This design combines channels and a mutex. A mutex protects the subscriber set (a map), and each subscriber receives through its own channel. This follows the split in 1.2: guarding a map is mutex work, and handing off a message is channel work. A design can use both.

dropped is an atomic.Int64 because RLock is a shared lock, so multiple Publish calls execute this loop simultaneously. A bare h.dropped++ is a read-modify-write, and a shared lock does not protect it. This is a common case where "I'm holding a lock, so I'm safe" fails, and go test -race reports it immediately. Always check that the kind of lock (exclusive versus shared) matches the operations performed under it.

Subscribe returns <-chan Message, a receive-only channel, so subscribers can neither send nor close. The type system, rather than documentation, enforces the convention that only the sender closes a channel.

7.4 The slow-consumer problem

The default: in Publish is the most consequential design decision in this code. What should happen when one subscriber is slow (congested network, heavy processing)?

The disconnect strategy has a trap. Calling unsub() directly inside Publish's default: branch always deadlocks. Publish holds h.mu.RLock(), and unsub asks for Lock() on the same RWMutex. Go's sync.RWMutex is neither reentrant nor upgradeable. Collect the subscribers to disconnect and call their unsub functions after RUnlock.

func (h *Hub) PublishDisconnectSlow(m Message) {
    var doomed []func()
 
    h.mu.RLock()
    for s := range h.subs {
        select {
        case s.ch <- m:
        default:
            doomed = append(doomed, s.unsub) // collect, do not call
        }
    }
    h.mu.RUnlock()
 
    for _, unsub := range doomed {
        unsub() // needs the exclusive lock — must run after RUnlock
    }
}

In practice, drop or disconnect are the first choices. Blocking ties the caller of Publish, usually the core of your business logic, to the pace of the slowest subscriber. For a publicly exposed API, letting one slow client stall the whole system is especially dangerous.

Blocking can also deadlock. If you rewrite Publish with a plain s.ch <- m, it parks while holding h.mu.RLock(). If that subscriber's reader decides it is done and calls unsub(), unsub blocks waiting for h.mu.Lock() and stops draining s.ch. Nobody will ever drain s.ch, so Publish never returns, so RUnlock never runs, so unsub never returns: a mutual deadlock. Worse, Go's RWMutex stops admitting new readers once a writer is waiting, so every subsequent Publish and Subscribe wedges behind it.

Under the blocking strategy, then, the recovery path of cutting the slow subscriber loose cannot run.

Sending outside the lock trades the deadlock for a panic. Holding RLock across the send is what keeps it mutually exclusive with unsub's close(s.ch). If you snapshot the subscriber set and send outside the lock, that exclusion is gone: a publisher holding a stale *subscriber sends into a channel unsub just closed and dies with send on closed channel. With concurrent Publish calls, s.ch is a multi-sender channel, which is the case 8.5 rule 2 tells you not to close.

If you take the snapshot route you must stop closing s.ch: signal unsubscription with a per-subscriber done channel in the select and let the channel be garbage-collected instead. If that is not worth it, do not put data you cannot drop through a Hub; put it in a durable queue.

Implementing "drop oldest" looks like this:

select {
case s.ch <- m:
default:
    select {
    case <-s.ch: // discard the oldest
    default:
    }
    select {
    case s.ch <- m: // put the newest in the freed slot
    default:
    }
}

With concurrent Publish calls this is not a strict "latest N". If you need strictness, give each subscriber a ring buffer guarded by a mutex.

Whatever you choose, always export the drop count as a metric. Silent data loss is the most dangerous failure mode in this structure, because nothing else reveals it.

Chapter 8: A catalogue of production failures

8.1 The four causes of goroutine leaks

Goroutine leaks are the most common form of memory leak in Go. Until a goroutine ends, its stack and everything reachable from it stay alive.

The most common pattern.

// Leaks
func LeakySearch(ctx context.Context, q string) (string, error) {
    backends := []string{"a", "b", "c"}
    results := make(chan string) // unbuffered
 
    for _, backend := range backends {
        go func() {
            results <- search(backend, q) // all but the first block forever
        }()
    }
 
    select {
    case r := <-results:
        return r, nil // returning here leaks the other two forever
    case <-ctx.Done():
        return "", ctx.Err() // returning here leaks all three
    }
}

"Use only the fastest result" is a reasonable design; the bug is that it does not release the remaining senders. The fix is one expression:

results := make(chan string, len(backends)) // senders can never block

This is the rule from 3.4 again. len(backends) states the invariant that every sender can complete its send even with no receiver, and has nothing to do with performance.

for v := range ch { // blocks forever if nobody closes ch
    process(v)
}

This happens when no one is responsible for closing the channel, or when an error return skips the closing path. The rule is that the sender closes the channel, with defer close(ch).

// Cannot be stopped
go func() {
    for {
        time.Sleep(time.Minute)
        refreshCache()
    }
}()

It runs as long as the process lives. Start it in a test and it outlives the test.

// Correct
go func() {
    t := time.NewTicker(time.Minute)
    defer t.Stop()
    for {
        select {
        case <-ctx.Done():
            return
        case <-t.C:
            refreshCache(ctx)
        }
    }
}()

for v := range ch {
    if v.IsBad() {
        return errors.New("bad value") // the sender blocks on the rest
    }
}

If early return is possible, the sender needs a ctx-based escape too.

8.2 Go 1.27's goroutine leak profile

Until now, the runtime could not detect those four patterns, so catching them was largely left to careful review. Go 1.27 changes that:

A new profile type that reports leaked goroutines, previously available as an experiment in Go 1.26, is now generally available. The new profile type, named goroutineleak, is supported in the runtime/pprof package. It is also available as the net/http/pprof endpoint /debug/pprof/goroutineleak.

The profile reuses the garbage collector's reachability analysis. If goroutine G is blocked on synchronization primitive P, and P is unreachable from any runnable goroutine (or any goroutine those could unblock), then P can never be unblocked, so G is leaked.

Here it is on a small program that leaks three goroutines:

func leak() {
    ch := make(chan int) // local; nothing references it once leak() returns
    go func() {
        ch <- 1 // blocks forever
    }()
}
 
func main() {
    for range 3 {
        leak()
    }
    time.Sleep(100 * time.Millisecond)
 
    p := pprof.Lookup("goroutineleak")
    fmt.Printf("goroutines alive: %d\n", runtime.NumGoroutine())
    fmt.Printf("Count() before write: %d\n", p.Count())
    _ = p.WriteTo(os.Stdout, 1)
    fmt.Printf("Count() after write: %d\n", p.Count())
}

Output on Go 1.27:

goroutines alive: 4
Count() before write: 0
---- profile (debug=1) ----
goroutineleak profile: total 3
3 @ 0x7ff77731c56a 0x7ff7772b0f5c 0x7ff7772b0b57 0x7ff77738829e 0x7ff777322661
#	0x7ff77738829d	main.leak.func1+0x1d	.../leakdemo/main.go:15
 
Count() after write: 3

It reports all three leaks with the exact source line. The traditional goroutine profile lists every blocked goroutine and cannot tell a healthy wait from a leak; goroutineleak makes that distinction automatically.

Two ways to use it:

// 1. Expose it over HTTP (just import net/http/pprof)
//    go tool pprof http://localhost:6060/debug/pprof/goroutineleak
 
// 2. Assert on it at the end of a test
func TestNoLeaks(t *testing.T) {
    // ... test body ...
 
    p := pprof.Lookup("goroutineleak")
    var buf bytes.Buffer
    _ = p.WriteTo(&buf, 1) // detection runs here; Count() is 0 before it
    if n := p.Count(); n > 0 {
        t.Errorf("leaked %d goroutines:\n%s", n, buf.String())
    }
}

Two caveats:

• Count() returns 0 until the first WriteTo (see the output above). Leak detection runs at write time, so code that checks only Count() always sees zero. In tests, call WriteTo first
• as the release notes state, leaks blocked on primitives reachable from global variables or from the locals of runnable goroutines are not detected. That is an inherent limit of the reachability approach

Because the profile cannot see every leak, pair it with 11.1's goleak and goroutine-count monitoring.

8.3 select randomness and priority

When several cases are ready, select picks one uniformly at random (Go spec). That prevents starvation and gets in the way when you want priority.

// Does not do what it looks like
select {
case <-ctx.Done():   // "cancellation should win"
    return ctx.Err()
case job := <-jobs:
    process(job)
}

When ctx is already cancelled and jobs has data, the job is processed 50% of the time. If the requirement is "never start new work after cancellation", use two stages:

// Strictly prioritise cancellation
select {
case <-ctx.Done():
    return ctx.Err()
default:
}
 
select {
case <-ctx.Done():
    return ctx.Err()
case job := <-jobs:
    process(job)
}

Conversely, when you want to finish the remaining work at shutdown, the priority inverts (the drain loop in 6.2 does this). Decide explicitly which requirement you have.

The two-stage form guarantees only that no new work starts after cancellation has been observed. If cancellation lands between the two selects, the second one is still a coin flip. The window is narrow but nonzero. If the work cannot afford even one stray item, re-check ctx.Err() at the top of the work itself, and on cancellation do not process the item: move it onto the drain budget or hand it back as unprocessed. 10.2 applies this to deliver.

8.4 nil channels — a practical idiom

Sending to or receiving from a nil channel blocks forever. That sounds like nothing but a source of bugs, but it lets you enable and disable select arms dynamically.

func Pump(ctx context.Context, in <-chan Result, out chan<- Result) error {
    var pending *Result
    for {
        // while there is nothing to send, nil out the send arm
        var send chan<- Result
        var val Result
        if pending != nil {
            send = out
            val = *pending
        }
 
        // while holding an item, nil out the receive arm
        var recv <-chan Result
        if pending == nil {
            recv = in
        }
 
        select {
        case <-ctx.Done():
            return ctx.Err()
        case v, ok := <-recv:
            if !ok {
                return nil
            }
            pending = &v
        case send <- val:
            pending = nil
        }
    }
}

Writing this without nil channels means two separate select statements for "holding an item" and "not holding", duplicating the ctx.Done() handling.

Pump is a single stage, and real pipelines chain stages: generate → transform → sink. The code above never closes out because Pump does not own it. In a pipeline, the convention is that each stage owns its output channel and closes it with defer close(out) (8.5 rule 1).

// Stage 1: generate. It owns out, so it closes out.
func gen(ctx context.Context, items []Item) <-chan Item {
    out := make(chan Item)
    go func() {
        defer close(out) // the owner closes
        for _, it := range items {
            select {
            case out <- it:
            case <-ctx.Done():
                return // escapable even if the downstream disappears
            }
        }
    }()
    return out
}
 
// Stage 2: transform
func transform(ctx context.Context, in <-chan Item) <-chan Result {
    out := make(chan Result)
    go func() {
        defer close(out)
        for it := range in { // ends when the upstream closes
            v := convert(it) // evaluate outside the select (see below)
            select {
            case out <- v:
            case <-ctx.Done():
                return
            }
        }
    }()
    return out
}
 
// Stage 3: sink. Draining here is what lets the whole chain finish.
func sink(ctx context.Context, in <-chan Result) error {
    for r := range in {
        if err := write(ctx, r); err != nil {
            return err
        }
    }
    // Distinguishes "the upstream was interrupted" from "it closed normally"
    return ctx.Err()
}
 
// Composition. Owning the ctx here is the whole point.
func RunPipeline(parent context.Context, items []Item) error {
    ctx, cancel := context.WithCancel(parent)
    defer cancel() // if sink returns early, release stages 1 and 2
 
    return sink(ctx, transform(ctx, gen(ctx, items)))
}

convert(it) is evaluated outside the select because the Go spec says the channel and right-hand-side expressions of send statements are evaluated exactly once, upon entering the select. So case out <- convert(it): runs convert even when the ctx.Done() arm is the one chosen. That is harmless for a pure function, but wasteful or wrong if convert is expensive or has side effects.

In a pipeline, it is the send that leaks. Many people stop worrying once the receive is written as for v := range in, but if a downstream stage dies, out <- v has no receiver and blocks forever. That is why every send must be wrapped in a select with a ctx.Done() arm. This is 8.1's Cause 1 in pipeline form.

Also, if sink returns early (the failing write above), the upstream stages jam because nobody reads in. To prevent this, RunPipeline, the function doing the composition, owns the ctx. Only when its defer cancel() fires can stages 1 and 2 escape through <-ctx.Done(). sink cannot cancel the context it was handed, so this responsibility has to sit with the caller.

You can check this. Remove defer cancel() from RunPipeline, make sink fail on the third item, and goleak reports gen's goroutine as still alive. Put it back and the report goes away.

Finally, a pipeline's throughput is set by the slowest stage's μ (3.4). Adding stages cannot make it faster than that stage.

The same nil trick removes closed channels from a select:

for a != nil || b != nil {
    select {
    case v, ok := <-a:
        if !ok {
            a = nil // a closed channel is always ready; take it out
            continue
        }
        handle(v)
    case v, ok := <-b:
        if !ok {
            b = nil
            continue
        }
        handle(v)
    }
}

A closed channel is always ready to receive. Without swapping in nil, that case is selected endlessly and you have a busy loop. You need this whenever you write fan-in by hand.

8.5 The rules of close

Three rules cover it.

1. The sender closes. The receiver never does
2. Never close a channel with multiple senders. If you must, signal every sender via a separate done channel (or ctx), wg.Wait() for all of them to exit, and only then close. Wrapping the close in sync.Once does not solve it: Once only prevents the double-close panic, not "send on closed channel", and the latter is the actual multi-sender hazard (a sender parked in ch <- v when the close runs panics regardless of Once)
3. Sending on a closed channel panics; double-close panics. Receiving from a closed channel first returns any values still buffered, and only then yields the zero value with ok == false

The buffered-values detail in rule 3 applies to 7.3. When unsub calls close(s.ch), there may still be buffered messages. They are not lost; the receiver reads them all before ok goes false.

6.3 is different. There, Add accepts up to cap(b.in) items simply because the buffer has room, but nobody ever closes b.in, so this rule does not apply. That is why the <-b.done arm is needed to stop accepting work nobody will drain.

In 7.3, Subscribe returns <-chan Message so that the type system enforces rule 1. At API boundaries, use directional channel types (chan<- / <-chan). That is far more reliable than writing the convention in a doc comment.

8.6 Data races and the memory model

For catching data races, nothing beats running -race in CI. The detector reports only races that actually occurred, but it has no false positives: if it reports a race, it is a bug.

go test -race ./...

The race detector documentation puts the cost for a typical program at 2–20× the execution time and 5–10× the memory, so keep it out of production binaries. Besides CI, a staging canary is a good place to run it.

The Go memory model guarantees four things about channels (the Go 1.19 revision restructured the model around synchronized before; happens before is still defined, in terms of synchronized before plus program order):

1. a send on a channel is synchronized before the completion of the corresponding receive
2. closing a channel is synchronized before a receive that returns the zero value because of the close
3. a receive from an unbuffered channel is synchronized before the completion of the corresponding send
4. the kth receive on a channel of capacity C is synchronized before the completion of the k+Cth send

The third is the formal statement that unbuffered channels are a rendezvous. Visibility comes from the first rule: every write the sender performs before the send, including the writes that build the value, is visible to the receiver.

The fourth looks obscure, but the memory model states its purpose directly:

This rule generalizes the previous rule to buffered channels. It allows a counting semaphore to be modeled by a buffered channel: the number of items in the channel corresponds to the number of active uses, the capacity of the channel corresponds to the maximum number of simultaneous uses, sending an item acquires the semaphore, and receiving an item releases the semaphore.

So the "buffered channel as a semaphore" pattern from 3.2 is a use the memory model explicitly provides for.

Through Go 1.21, the loop variable was shared across all iterations.

// Go 1.21 and earlier: every goroutine sees the same i
for i := 0; i < 3; i++ {
    go func() { fmt.Println(i) }()
}
 
// The workaround required before Go 1.22
for i := 0; i < 3; i++ {
    i := i // redeclare
    go func() { fmt.Println(i) }()
}

Go 1.22 made loop variables per-iteration, which removes the problem. The behaviour depends on the go directive in go.mod, though: a module that still declares go 1.21 keeps the old semantics. When copying from older articles, check your go.mod version, not whether the code has i := i.

8.7 A panic in a goroutine kills the process

recover is per goroutine. A caller's defer recover() cannot catch a panic in a different goroutine.

func handler(w http.ResponseWriter, r *http.Request) {
    defer func() {
        if v := recover(); v != nil { // never reached for the goroutine below
            w.WriteHeader(500)
        }
    }()
    go doWork() // a panic here kills the whole process
}

net/http recovers panics in the handler itself and closes just that connection, but it cannot catch panics in goroutines the handler started. As shown in 3.2, errgroup deliberately does not propagate them either.

The remedy is to wrap worker launches:

func Go(ctx context.Context, name string, f func()) {
    go func() {
        defer func() {
            if v := recover(); v != nil {
                slog.ErrorContext(ctx, "goroutine panicked",
                    "name", name, "panic", v,
                    "stack", string(debug.Stack()))
                metrics.PanicTotal.WithLabelValues(name).Inc()
            }
        }()
        f()
    }()
}

Do not recover unconditionally, though. If state may be corrupted, crashing and restarting is safer than carrying on. Ask whether this goroutine's failure affects the correctness of other requests. A worker that handles one request in isolation can recover and continue; a goroutine midway through updating shared state should be allowed to crash.

Chapter 9: The toolbox

Before assembling channels by hand, check whether something already exists.

9.1 singleflight — the highest-value quasi-standard library

A cache entry expires and a thousand requests for the same key arrive at once. All of them hit the origin (a database, an external API). This is a cache stampede (thundering herd), and it happens routinely in production.

golang.org/x/sync/singleflight does one thing: collapse concurrent calls for the same key into one and hand the result to everyone. A thousand requests become one origin call.

func (l *Loader) Get(ctx context.Context, key string) (string, error) {
    if v, ok, err := l.cache.Get(ctx, key); err == nil && ok {
        return v, nil
    }
 
    ch := l.sf.DoChan(key, func() (any, error) {
        // do NOT use the caller's ctx directly (see below)
        octx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 3*time.Second)
        defer cancel()
 
        v, err := l.origin(octx, key)
        if err != nil {
            return nil, err
        }
        _ = l.cache.Set(octx, key, v, time.Minute)
        return v, nil
    })
 
    select {
    case r := <-ch:
        if r.Err != nil {
            return "", fmt.Errorf("load %q: %w", key, r.Err)
        }
        return r.Val.(string), nil
    case <-ctx.Done():
        return "", ctx.Err() // leave alone; other waiters are unaffected
    }
}

Do blocks, so it keeps waiting even after the caller's ctx is cancelled. DoChan returns a channel, so a select can wait for the result and for the caller's own cancellation at the same time. A caller can leave without affecting the others sharing the call.

Leaving is safe because of how DoChan is implemented. In singleflight.go, it creates a buffered make(chan Result, 1) for each waiter, so even if you leave via ctx.Done(), the sender completes its send without blocking. That buffer of 1 is why this does not become 8.1's "send after the receiver left" (the docs also state that the returned channel is never closed).

You only see this trap by reading the source. Do runs g.doCall(...) synchronously on the calling goroutine; DoChan runs go g.doCall(...) in a new one. The location differs, but in both cases the closure that runs is the one built by whichever caller arrived first for that key. The functions passed by the second and later callers are never executed at all.

So if that function captures its caller's ctx, the origin call is aborted the moment the first caller times out, and every caller sharing the call fails with it. Worse, "the first caller" changes from run to run, so the failure shows up as sporadic, unreproducible errors. Detach with context.WithoutCancel and give it an independent timeout.

singleflight has three known caveats:

1. Errors fan out to everyone. One transient origin failure becomes a thousand errors. Pair it with retries or a fallback (serving stale cache). Forget(key) removes an in-flight call from sharing
2. A panic inside the work function cannot be recovered on the DoChan path. When there are channel waiters, doCall re-raises the captured panic from a detached goroutine (go panic(e)). This follows the same design as errgroup in 3.2. (A plain Do with no channel waiters re-panics on the caller's own goroutine, where recover does work.) See 8.7
3. Deduplication is per process. With N replicas you get N origin calls: "1000 → N", not "1000 → 1". 10.4 makes the same boundary argument for channels

One more caveat: the ctx detached with context.WithoutCancel still carries the first caller's request-scoped values. Trace IDs are among them, so a shared call made on behalf of a thousand callers gets attributed to whichever single request happened to arrive first. If observability matters, strip the values or open a new span linked to all the waiters.

9.2 Copy-on-write with atomic.Pointer

For data that is read constantly and updated rarely, such as configuration or routing tables, there is a better option than a mutex.

type Config struct {
    Timeout  time.Duration
    Features map[string]bool
}
 
type Store struct {
    cfg atomic.Pointer[Config]
}
 
// Read: lock-free, a few ns
func (s *Store) Get() *Config { return s.cfg.Load() }
 
// Update: build a new instance and swap it in
func (s *Store) Update(c *Config) { s.cfg.Store(c) }

Readers take no lock at all. The catch is that the contents of a stored *Config must never be mutated. Updates always construct a fresh Config and Store it. If a holder of the returned *Config writes into the Features map, that is a race.

If your team cannot keep to that convention, an RWMutex is the safer choice.

9.3 When not to use sync.Map

sync.Map is not a fast concurrent map for every situation. The documentation says it is optimized for two cases:

1. a key is written once and thereafter only read (append-only caches)
2. multiple goroutines read and write disjoint sets of keys

Go 1.24 replaced sync.Map's internals with a HashTrieMap-based implementation, substantially improving write-heavy, delete-heavy, and highly contended workloads. On Go 1.24+, the old rule of thumb that write-heavy workloads always favour map + RWMutex no longer holds as stated.

The order of operations is unchanged, though: start with map + RWMutex, and reconsider only once profiling shows lock contention. Outside the two documented use cases, benchmark sync.Map on your own workload before choosing it for performance reasons.

Chapter 10: Case study — a webhook delivery worker

This chapter combines the patterns into one system. The requirements are typical of a real product.

1. Deliver events to external URLs via HTTP POST
2. Preserve ordering per destination (event order is meaningful)
3. Retry with exponential backoff; past a limit, send to a dead-letter sink
4. Bound concurrency so downstreams are not DoSed
5. Lose nothing during graceful shutdown
6. Make backlog and failures observable

10.1 Design: reconciling ordering with concurrency

Requirements 2 and 4 look contradictory: ordering demands sequential execution, and sequential execution severely limits throughput.

Sharding by destination resolves this. Hash the destination to pick a lane, and run one worker per lane. Within a lane, execution is sequential, so per-destination order holds; across lanes it is concurrent, so throughput holds.

The lane count is the maximum concurrency. Use the value derived from Little's Law in 3.3.

10.2 Implementation

type Delivery struct {
    Endpoint string // destination; ordering is preserved per endpoint
    Payload  []byte
    Attempt  int
}
 
type DispatcherConfig struct {
    Shards       int           // lanes = effective max concurrency
    QueuePerLane int           // per-lane buffer = tolerated backlog
    MaxAttempts  int
    BaseBackoff  time.Duration
    MaxBackoff   time.Duration
    SendTimeout  time.Duration
}
 
type Dispatcher struct {
    cfg    DispatcherConfig
    sender Sender
    lanes  []chan Delivery
    dead   func(context.Context, Delivery, error)
    done   chan struct{} // 5.3 Point 1: announces the owner's liveness
 
    enqueued atomic.Int64
    rejected atomic.Int64
    sent     atomic.Int64
    failed   atomic.Int64
}
 
func NewDispatcher(cfg DispatcherConfig, sender Sender,
    dead func(context.Context, Delivery, error)) *Dispatcher {
 
    if cfg.Shards < 1 {
        cfg.Shards = 1 // 0 would make lane()'s modulus a divide-by-zero panic
    }
    lanes := make([]chan Delivery, cfg.Shards)
    for i := range lanes {
        lanes[i] = make(chan Delivery, cfg.QueuePerLane)
    }
    return &Dispatcher{
        cfg: cfg, sender: sender, lanes: lanes, dead: dead,
        done: make(chan struct{}), // omit this and 10.2's liveness guard is dead code
    }
}
 
// Accessor for observability (per 7.4 / 10.3, always export these)
func (d *Dispatcher) Stats() (enqueued, rejected, sent, failed int64) {
    return d.enqueued.Load(), d.rejected.Load(), d.sent.Load(), d.failed.Load()
}
 
func (d *Dispatcher) lane(endpoint string) chan Delivery {
    h := fnv.New32a()
    _, _ = h.Write([]byte(endpoint))
    // Take the modulus in uint32: int(h.Sum32()) can be negative on 32-bit
    return d.lanes[h.Sum32()%uint32(len(d.lanes))]
}

When a lane is full, the producer gives up immediately (strategy B from 3.5):

var ErrQueueFull = errors.New("dispatcher: queue full")
var ErrDispatcherClosed = errors.New("dispatcher: closed")
 
func (d *Dispatcher) Enqueue(ctx context.Context, del Delivery) error {
    // The two-stage guard from 8.3. A select with `default` never blocks,
    // so mixing in `case <-ctx.Done()` would enqueue anyway ~50% of the time
    // when the context is already cancelled and the lane has room.
    if err := ctx.Err(); err != nil {
        return err
    }
 
    // Same rule as 5.3 Point 1: refuse once the workers are gone
    select {
    case <-d.done:
        return ErrDispatcherClosed
    default:
    }
 
    select {
    case d.lane(del.Endpoint) <- del:
        d.enqueued.Add(1)
        return nil
    default:
        d.rejected.Add(1)
        return fmt.Errorf("%w: %s", ErrQueueFull, del.Endpoint)
    }
}

8.3's priority problem applies directly to Enqueue. A select with default never blocks, so adding a case <-ctx.Done(): arm is actively harmful. When the context is cancelled and there is room, both arms are ready, so cancellation is ignored half the time. For non-blocking sends, check ctx before the select.

The <-d.done pre-check is also mandatory. After Run returns (say Shutdown timed out and stopWorkers() fired), the lanes still have room. Without this check, Enqueue reports success and increments enqueued, but that delivery is never processed and never dead-lettered. You would be creating the "events sometimes disappear" failure that 10.4 warns about. The rule imposed on the actor in 5.3 and on Batcher in 6.3 applies here too.

The check does not close the window between the test and the send, though. If close(d.done) lands right after the default:, that one delivery still vanishes. As in 8.3, the two-stage guard makes the loss orders of magnitude rarer but not impossible. If you truly cannot lose an item, you need the durable queue of 10.4.

Make the error a sentinel. A bare fmt.Errorf cannot be matched with errors.Is, so the caller cannot implement 3.5's strategy B ("return 503 when full"). Wrap with %w so ErrQueueFull propagates.

One worker per lane:

func (d *Dispatcher) Run(ctx context.Context) error {
    defer close(d.done) // tell Enqueue we have stopped
 
    g := &errgroup.Group{}
    for i := range d.lanes {
        g.Go(func() error {
            d.runLane(ctx, d.lanes[i])
            return nil
        })
    }
    return g.Wait()
}
 
func (d *Dispatcher) runLane(ctx context.Context, lane chan Delivery) {
    for {
        // two-stage guard from 8.3: never start a new delivery after cancellation
        select {
        case <-ctx.Done():
            d.drain(lane)
            return
        default:
        }
 
        select {
        case <-ctx.Done():
            d.drain(lane)
            return
        case del := <-lane:
            d.deliver(ctx, del)
        }
    }
}
 
func (d *Dispatcher) drain(lane chan Delivery) {
    // bounded grace for shutdown (same reasoning as decision 2 in 4.5)
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()
    for {
        select {
        case <-ctx.Done():
            d.abandon(lane) // budget spent: stop sending, just record
            return
        default:
        }
 
        select {
        case del := <-lane:
            d.deliver(ctx, del)
        default:
            return
        }
    }
}
 
// abandon cleans up after the budget is gone. It attempts no sends and caps
// the dead-letter writes as a whole, so stop time cannot grow with cap(lane).
func (d *Dispatcher) abandon(lane chan Delivery) {
    ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
    defer cancel()
    for {
        select {
        case del := <-lane:
            d.failed.Add(1)
            d.dead(ctx, del, context.DeadlineExceeded)
        default:
            return
        }
    }
}

The two-stage guard in runLane is mandatory. With a single select, 8.3's priority problem appears in its worst form. If the lane is non-empty when cancellation arrives, both arms are ready and <-lane wins half the time. deliver then builds sctx from an already-cancelled ctx, the send fails instantly, and the retry backoff falls into <-ctx.Done(), so the item goes to the dead-letter sink without a single delivery attempt. On average, one item per lane fails for no reason on every deploy.

(Even with the leading non-blocking select, a window remains if cancellation lands between the two selects. If you need strictness, check ctx.Err() at the top of deliver too, per 8.3, and route those items through the drain budget.)

drain hands off to abandon to keep stop time bounded. A naive version that keeps calling deliver after the budget expires makes stop time grow with cap(lane). The dead ctx does make each send fail instantly, but every item still costs one Send attempt, one log line, and one dead-letter write. Each d.dead call is capped at 2 s, but nothing caps them in aggregate. With QueuePerLane = 1000 and a 20 ms dead-letter write, you spend another 20 s after the 10 s budget. That is the failure 4.5's Decision 2 warns about: losing the very data the drain existed to save.

So once the budget is gone, abandon stops sending and gives the dead-letter writes their own 3-second ceiling.

The 3-second guarantee has two caveats. First, d.dead must honour the deadline it is handed; an implementation that ignores ctx silently exceeds the budget. Second, anything still in the lane when the 3 s expires is neither sent nor recorded.

Delivery itself, with retries and backoff:

func (d *Dispatcher) deliver(ctx context.Context, del Delivery) {
    for {
        sctx, cancel := context.WithTimeout(ctx, d.cfg.SendTimeout)
        err := d.sender.Send(sctx, del)
        cancel() // not deferred — this is a loop, release each iteration
 
        if err == nil {
            d.sent.Add(1)
            return
        }
 
        del.Attempt++
        permanent := errors.Is(err, ErrPermanent) // retrying a 4xx is pointless
        exhausted := del.Attempt >= d.cfg.MaxAttempts
 
        if permanent || exhausted {
            d.failed.Add(1)
            // The dead-letter write must land even mid-shutdown — but not forever
            dctx, dcancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Second)
            d.dead(dctx, del, err)
            dcancel()
            return
        }
 
        wait := backoff(d.cfg.BaseBackoff, d.cfg.MaxBackoff, del.Attempt)
        slog.WarnContext(ctx, "webhook retry",
            "endpoint", del.Endpoint, "attempt", del.Attempt,
            "wait", wait, "err", err)
 
        select {
        case <-time.After(wait):
        case <-ctx.Done():
            // on shutdown, hand the unsent item to the dead-letter sink
            d.failed.Add(1)
            dctx, dcancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Second)
            d.dead(dctx, del, err)
            dcancel()
            return
        }
    }
}
 
// Exponential backoff with full jitter; the result is in [1, maxWait].
// rand is math/rand/v2 (v1 has no Int64N).
func backoff(base, maxWait time.Duration, attempt int) time.Duration {
    if maxWait <= 0 {
        return 1 // rand.Int64N(0) panics
    }
    shift := attempt - 1
    if shift < 0 {
        shift = 0 // a negative shift count panics at run time
    }
    d := base << min(shift, 16) // cap the shift to avoid overflow
    if d > maxWait || d <= 0 {
        d = maxWait
    }
    return time.Duration(rand.Int64N(int64(d)) + 1)
}

10.3 The decisions, indexed

The code is short, but it embeds almost every decision from the earlier chapters:

On not using defer cancel(). deliver is a loop, so deferring would stack one un-cancelled context per retry until the function returns. Ten retries leave ten unreleased contexts. Inside a loop, call cancel() in place. go vet's lostcancel check does not catch this leak.

On WithoutCancel plus a deadline around d.dead. The dead-letter write is the record you least want to lose, because it says "we gave up on delivering this". Both call sites are reachable mid-shutdown with an already-cancelled ctx, so both detach cancellation. But the context context.WithoutCancel returns has no deadline (Deadline() reports ok == false). Pass that straight through and your shutdown ceiling disappears, so always layer a WithTimeout on top.

On full jitter. rand.Int64N(int64(d)) + 1 returns a value in [1, d]. Without jitter, all retries arrive at the same moment, so a downstream that has just recovered is knocked over again. Jitter spreads retries across time.

10.4 The remaining limits — in-process queues

This dispatcher has three limits.

• If the process is SIGKILLed, the queue contents are gone. drain only rescues graceful stops
• Horizontal scaling breaks the ordering guarantee, since events for one destination can land on different instances. For the same reason, changing Shards reshuffles the hash % N assignment and breaks ordering for anything in flight
• One dead destination takes down every healthy destination sharing its lane. deliver waits out its retries inline, stalling the whole lane for up to MaxAttempts × MaxBackoff (say 5 × 30 s). The lane then fills, and Enqueue starts rejecting destinations that have nothing wrong with them. The design itself creates this cascading failure. Fix it with a per-endpoint circuit breaker that fails fast to dead-letter, or park retries in a delay queue so the lane keeps moving

If "never lose an item" or "strict ordering" is a requirement, an in-process channel is the wrong queue. Use a durable queue (PostgreSQL SELECT ... FOR UPDATE SKIP LOCKED, Redis Streams, Kafka) and restrict channels to distributing jobs pulled from it to in-process workers.

A channel is an excellent in-process concurrency mechanism, but it is neither a durable queue nor a distributed one. In practice, recognizing that boundary may be the most important judgment of all. A channel's contents vanish when the process dies and cannot be shared across processes. Miss that boundary and you inherit an unreproducible "events sometimes disappear" incident.

Chapter 11: Testing and observability

11.1 -race is mandatory, goleak is strongly recommended

go test -race ./...

go.uber.org/goleak detects goroutines still alive at the end of a test. 8.2's goroutineleak profile detects only goroutines that are provably unreachable, so the two complement each other.

func TestCacheNoLeak(t *testing.T) {
    defer goleak.VerifyNone(t)
 
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel() // stop it even if we exit via t.Fatal
    c := NewCache(ctx)
    // ... test body ...
 
    cancel()
    <-c.Done() // wait for the owner goroutine to exit before checking
}

Do not omit <-c.Done(). goleak.VerifyNone does retry, by default up to 20 times with backoff (options.go), so it will not fail on a goroutine that lingers for a microsecond after cancel(). Without <-c.Done(), though, the test's outcome depends on whether shutdown fits inside goleak's retry budget. Add a drain and a final flush and it will not fit, and you have a flaky test.

More generally, signalling a stop and waiting for the stop are different operations. Any testable component must offer the second as public API. That is why Cache has a Done() method in 5.2 (a test poking at c.done directly only compiles inside the same package).

To apply it package-wide, use TestMain:

func TestMain(m *testing.M) {
    goleak.VerifyTestMain(m)
}

11.2 testing/synctest — deterministic time-dependent tests

This package was introduced experimentally in Go 1.24 and stabilized in Go 1.25. It addresses the most troublesome problem in concurrent testing: flaky tests built around time.Sleep.

Check your Go version. Go 1.25 shipped only Test and Wait; the Sleep used in the example below was added in Go 1.27 (api/go1.25.txt lists Test / Wait, api/go1.27.txt lists Sleep). On Go 1.25 and 1.26, write time.Sleep(d) followed by synctest.Wait() instead; Sleep is equivalent to those two calls.

synctest.Test runs the function it is given in an isolated bubble, and time inside the bubble runs on a virtual clock. When every goroutine in the bubble is durably blocked (unblockable only by another goroutine in the same bubble), time jumps instantly to the next event.

Testing the batcher from 6.2:

func TestBatcherFlushesOnTimeout(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        sink := &fakeSink{}
        b := NewBatcher(sink, 10, 100, 250*time.Millisecond) // queue, maxSize, maxDelay
 
        ctx, cancel := context.WithCancel(context.Background())
        defer cancel()
        go func() { _ = b.Run(ctx) }()
 
        if err := b.Add(ctx, Event{ID: "e1"}); err != nil {
            t.Fatal(err)
        }
        synctest.Wait() // let everything settle
 
        if got := sink.snapshot(); len(got) != 0 {
            t.Fatalf("flushed too early: %v", got)
        }
 
        synctest.Sleep(250 * time.Millisecond) // advance virtual time (Go 1.27+)
        // On Go 1.25 / 1.26:
        //   time.Sleep(250 * time.Millisecond)
        //   synctest.Wait()
 
        if got := sink.snapshot(); len(got) != 1 || len(got[0]) != 1 {
            t.Fatalf("want 1 batch of 1, got %v", got)
        }
    })
}