Inside DuckDB — Vectorized Execution, Morsel Parallelism and a Single-File Store, Read from the Code

pip install duckdb, and you can run SELECT as soon as the install finishes. No server to start. No config file. And yet, in a 2022 benchmark by the DuckDB developers, it grouped and aggregated a hundred million rows of a pandas DataFrame in under three seconds on a laptop (a 2021 MacBook Pro with an M1 Max). The first time, that speed is almost unsettling; it comes from design decisions at every layer, from the execution engine down to the storage format.

This article looks at how DuckDB is built rather than how to use it, chiefly the data representation of its vectorized execution engine, its push-based pipelines, parallelism without exchange operators, a columnar store in a single file, HyPer-style MVCC, and the ART index. For each, I trace the work down to the struct and file in duckdb/duckdb that does it.

In kitchen terms: if a row-oriented OLTP database is a counter that cooks one dish per order, DuckDB is a kitchen that preps 2048 plates of the same dish at once.

• Vectorization: instead of asking "is this an int? is it NULL?" again for every row, decide once for 2048 values of the same type.
• Push-based: the customer (the parent operator) does not call for "the next plate"; the kitchen (the child operator) pushes finished plates onto the pass.
• Morsel-driven: split the prep work into small units (row groups) and let any free cook grab the next one.
• Columnar + compression: box the ingredients by ingredient, not by dish. Identical things sit together, so whole boxes compress.

Together, these four make analytical queries on a single machine much faster.

1. Where DuckDB Sits

DuckDB is an embedded analytical (OLAP) database started by Mark Raasveldt and Hannes Mühleisen at CWI (Centrum Wiskunde & Informatica). Two axes place it:

That is where "the SQLite of OLAP" comes from. The resemblance is in how the two are deployed; their internals are completely different. SQLite is row-oriented, B-tree based, with a tuple-at-a-time bytecode VM. DuckDB is columnar, vectorized and push-based. They share only their placement: a library linked into the host process.

The biggest consequence of that placement is that no wire protocol and no serialization sit in the path. To hand a hundred million result rows to Python, PostgreSQL encodes them into its wire protocol, sends them over TCP, and the client decodes them again. DuckDB keeps the data on one process heap throughout.

"Zero copy" means different things in each direction, though. Inbound, it is literal when numeric columns are contiguous in memory: DuckDB points at the caller's buffer instead of copying it (§10.2 covers the separate pandas and Arrow routes). Outbound, it is not. fetchdf() and fetchnumpy() allocate a fresh NumPy array per column and convert into it, and to_arrow_table() goes through ArrowAppender, which writes into its own ArrowBuffers. A layout conversion is unavoidable because a Vector may still be dictionary- or FSST-encoded and strings are string_t (§2.6). Serialization and IPC go away; the memory copy does not.

2. Vectorized Execution and Data Representation

2.1 Three execution models

Query engines fall into three groups by how many rows each operator call processes.

Vector-at-a-time sits between the other two. The idea comes from Boncz, Zukowski and Nes, "MonetDB/X100: Hyper-Pipelining Query Execution" (CIDR 2005), and DuckDB descends directly from it.

2.2 Why 2048?

DuckDB's vector size lives in src/include/duckdb/common/vector_size.hpp.

// duckdb/common/vector_size.hpp
//! The default standard vector size
#define DEFAULT_STANDARD_VECTOR_SIZE 2048U
 
//! The vector size used in the execution engine
#ifndef STANDARD_VECTOR_SIZE
#define STANDARD_VECTOR_SIZE DEFAULT_STANDARD_VECTOR_SIZE
#endif
 
#if (STANDARD_VECTOR_SIZE & (STANDARD_VECTOR_SIZE - 1) != 0)
#error The vector size must be a power of two
#endif

The vector size is a compile-time constant, not a runtime setting. Because it is a macro, it can appear in template arguments and in fixed-size array declarations, and stack arrays like row_t row_data[STANDARD_VECTOR_SIZE]; appear throughout the engine.

The value 2048 keeps intermediates inside the CPU cache.

For an 8-byte type, one vector takes 2048 × 8 = 16 KB. A typical L1 data cache is 32–48 KB and L2 runs from a few hundred KB to a few MB, so an operator's input, output and scratch space still fit in L2. At the same time, 2048 rows amortize the per-call overhead — virtual dispatch, type switching, state loading — down to 1/2048 per row.

DuckDB can also be built with STANDARD_VECTOR_SIZE=512 or even =2, and the test suite runs at tiny vector sizes; that is why you occasionally see #if STANDARD_VECTOR_SIZE > 2 guards in the code.

2.3 DataChunk — the "2048 rows × N columns" box

The unit that flows between operators is a DataChunk. It is a collection of columns, not of rows.

// duckdb/common/types/data_chunk.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/common/types/data_chunk.hpp
class DataChunk {
public:
	//! The vectors owned by the DataChunk.
	vector<Vector> data;
	// ...
};

It is a thin box holding a vector<Vector> plus a cardinality; the interesting work happens in Vector.

DataChunk::Slice() shows how DuckDB avoids copying. When a filter "removes" rows, no values move: DuckDB builds a SelectionVector listing the surviving indices and layers it on top of each Vector.

2.4 Vector — several physical encodings of the same logical data

A Vector is not necessarily "an array of 2048 values". It can switch between several physical encodings of the same logical data, and much of DuckDB's code is easier to read once that is clear.

// duckdb/common/enums/vector_type.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/common/enums/vector_type.hpp
enum class VectorType : uint8_t {
	FLAT_VECTOR,       // Flat vectors represent a standard uncompressed vector
	FSST_VECTOR,       // Contains string data compressed with FSST
	CONSTANT_VECTOR,   // Constant vector represents a single constant
	DICTIONARY_VECTOR, // Dictionary vector represents a selection vector on top of another vector
	SEQUENCE_VECTOR,   // Sequence vector represents a sequence with a start point and an increment
	SHREDDED_VECTOR    // Shredded variant vector
};

Take SELECT * FROM t WHERE country = 'JP'.

• The literal 'JP' is a CONSTANT_VECTOR: one materialized value, so a comparison operator can compute once and broadcast the result over 2048 rows.
• If the country column is dictionary-compressed, the scan hands back a DICTIONARY_VECTOR as-is; only the dictionary entries exist as strings.
• An FSST-compressed string column reaches the operator as an FSST_VECTOR and is not decompressed until something actually needs the bytes.
• range(0, 1000000) and rowid columns are SEQUENCE_VECTORs: two integers, start and increment (plus the row count).

The playground below switches an 8-row Vector between four of these encodings. Every tab holds eight logical rows, but the physical buffer holds all eight values, a single value, a dictionary plus a list of indices, or just two integers, a start and an increment.

2.5 UnifiedVectorFormat — the compromise

Operators do not branch six ways on these encodings. UnifiedVectorFormat gives them a common read view.

// duckdb/common/vector/unified_vector_format.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/common/vector/unified_vector_format.hpp
struct UnifiedVectorFormat {
	const SelectionVector *sel;
	const_data_ptr_t data;
	ValidityMask validity;
	SelectionVector owned_sel;
	PhysicalType physical_type;
	// ...
};

Code that reads through UnifiedVectorFormat always takes this shape:

UnifiedVectorFormat vdata;
input.ToUnifiedFormat(vdata);
auto values = UnifiedVectorFormat::GetData<int32_t>(vdata);
 
for (idx_t i = 0; i < count; i++) {
	auto idx = vdata.sel->get_index(i);   // logical row i -> index into data
	if (!vdata.validity.RowIsValid(idx)) {
		continue;                          // NULL
	}
	// values[idx] holds the value
}

• For FLAT, sel is the identity (FlatVector::IncrementalSelectionVector()).
• For CONSTANT, sel is an all-zeros vector (ConstantVector::ZeroSelectionVector()), so everything resolves to data[0].
• For DICTIONARY, sel is the dictionary index itself.

The view is a compromise because ToUnifiedFormat() does not cover every encoding:

// src/common/types/vector.cpp — https://github.com/duckdb/duckdb/blob/main/src/common/types/vector.cpp
void Vector::ToUnifiedFormat(UnifiedVectorFormat &format) const {
	format.physical_type = GetType().InternalType();
	auto vtype = GetVectorType();
	if (vtype != VectorType::FLAT_VECTOR && vtype != VectorType::CONSTANT_VECTOR &&
	    vtype != VectorType::DICTIONARY_VECTOR) {
		// FSST/SEQUENCE/SHREDDED: flatten first so the buffer can provide unified format
		Flatten();
	}
	Buffer().ToUnifiedFormat(format);
}

FSST and SEQUENCE cannot be reduced to sel + data, so they are flattened into 2048 materialized values first, and calling ToUnifiedFormat() on them gives up the benefit of the compact encoding. CONSTANT and DICTIONARY avoid flattening, but the generic path still computes row by row, as in the loop above, so it cannot turn their few distinct values into less work. DuckDB's hot operators — comparison, hashing, aggregation — therefore carry dedicated fast paths that detect CONSTANT and DICTIONARY alongside the generic path.

2.6 string_t — design decisions packed into 16 bytes

The string representation lives in src/include/duckdb/common/types/string_type.hpp.

// duckdb/common/types/string_type.hpp
struct string_t {
	static constexpr idx_t PREFIX_BYTES = 4 * sizeof(char);
	static constexpr idx_t INLINE_BYTES = 12 * sizeof(char);
	// ...
private:
	union {
		struct {
			uint32_t length;
			char prefix[4];
			char *ptr;
		} pointer;
		struct {
			uint32_t length;
			char inlined[12];
		} inlined;
	} value;
};

Strings of twelve bytes or fewer fit entirely inside the struct, so reading them needs no heap access. Longer strings still keep their first four bytes in the struct as a prefix, which speeds up equality checks:

// duckdb/common/types/string_type.hpp — inside string_t::StringComparisonOperators
struct StringComparisonOperators {
	static inline bool Equals(const string_t &a, const string_t &b) {
		// (the DUCKDB_DEBUG_NO_INLINE branch is elided)
		uint64_t a_bulk_comp = Load<uint64_t>(const_data_ptr_cast(&a));
		uint64_t b_bulk_comp = Load<uint64_t>(const_data_ptr_cast(&b));
		if (a_bulk_comp != b_bulk_comp) {
			// Either length or prefix are different -> not equal
			return false;
		}
		// they have the same length and same prefix!
		a_bulk_comp = Load<uint64_t>(const_data_ptr_cast(&a) + 8u);
		b_bulk_comp = Load<uint64_t>(const_data_ptr_cast(&b) + 8u);
		if (a_bulk_comp == b_bulk_comp) {
			// either they are both inlined (so compare equal) or point to the
			// same string (so compare equal)
			return true;
		}
		// ...
	}
	// ...
};

The first eight bytes (4-byte length + 4-byte prefix) are compared as a single uint64_t. If the lengths or the first four characters differ, the function returns false immediately, and in real workloads that one comparison settles most mismatches.

The next eight bytes are compared the same way. If those match too, the pair is either "both inlined with identical contents" or "two pointers to the same string", so equality on strings of 12 bytes or fewer resolves in four loads and two comparisons. The pointer chase — and its cache miss — only happens for long strings whose length and prefix already matched.

Ordering comparisons use the same trick, plus a byte swap:

uint32_t a_prefix = Load<uint32_t>(const_data_ptr_cast(left.GetPrefix()));
uint32_t b_prefix = Load<uint32_t>(const_data_ptr_cast(right.GetPrefix()));
if (a_prefix != b_prefix) {
	return BSwapIfLE(a_prefix) > BSwapIfLE(b_prefix);
}

Reading four bytes as a uint32_t on a little-endian machine reverses their order, so BSwapIfLE puts them back before comparing. A four-character lexicographic comparison thus becomes one integer comparison.

2.7 SelectionVector and ValidityMask

A SelectionVector is a thin wrapper over sel_t* (uint32_t*), yet the execution model relies on it throughout.

• Filters write the indices of surviving rows.
• Hash joins express matches as two selection vectors, one into the probe side and one into the build side.

(Sorting is the one exception: it reorders the sort keys themselves rather than a selection vector. §6.6 has the details.)

ValidityMask is the NULL bitmap: one bit per row, and a set bit means valid (not NULL). When a vector has no NULLs at all, no mask is allocated (GetData() returns nullptr), and if CannotHaveNull() (formerly AllValid()) holds, the NULL-checking loop can be skipped entirely. Many real columns contain no NULLs, so this shortcut applies often.

2.8 Value versus Vector — representations inside and outside the engine

Open the source and you meet a type called Value almost immediately, and it is not "one element of a Vector". Vector is the execution engine's representation; Value is the representation everywhere else. Constant folding, min/max statistics, SET option values, catalog defaults, and the runtime-filter bounds of §8.4 are all Values: a simple but heavyweight object carrying one type tag and one value.

A fixed set of functions crosses the boundary. Vector::GetValue(i) boxes one row into a Value; the other direction is Vector::Reference(const Value &, count_t) or turning the vector into a ConstantVector. Calling GetValue() in a hot loop is slow, and DuckDB's own code uses it only in places such as finalize, where a single value is pulled out.

Casting likewise comes in two flavors. On a Value, value.DefaultTryCastAs(type) returns a failable result. On a Vector, BoundCastExpression::AddCastToType() splices a cast operator into the expression tree, and at runtime a function resolved from CastFunctionSet runs vector-at-a-time. Treating casts as first-class, failable operations also shapes runtime filter pushdown, which is one reason it only passes through integral casts and records, via RuntimeFilterCastMode { DEFAULT_CAST, TRY_CAST }, whether each cast could fail.

3. The Push-Based Execution Model

The execution model decides how the vectors of §2 move between operators.

3.1 From pull to push

In the classic Volcano model, the root operator calls next() and the call recurses down to the leaves. A parent pulls from its children.

DuckDB runs the other way. Data is pushed upward, and every operator plays one of three roles:

An Operator may keep state, but never an unbounded amount. Among operators whose output row count differs from their input, four filter and join operators derive from CachingPhysicalOperator: PhysicalFilter, PhysicalJoin, PhysicalCrossProduct and PhysicalRecursiveCTEKeyJoin. They coalesce small output chunks in a cached_chunk until they approach 2048 rows (CanCacheType() excludes LIST / MAP / ARRAY / VARIANT). Rows can therefore still be sitting inside an operator after the source is exhausted, so a pipeline has a fourth entry point beyond Source / Operator / Sink: FinalExecute(), which PipelineExecutor::TryFlushCachingOperators() drains at the end.

The same physical operator can play different roles in different pipelines. PhysicalHashJoin is a Sink in the build pipeline, an Operator in the probe pipeline, and a Source both when the join type propagates the build side (PropagatesBuildSide: RIGHT / FULL OUTER / RIGHT_SEMI / RIGHT_ANTI) and when it spills and becomes an external join. The code does not separate the three statically: IsSource() returns true unconditionally, and GetDataInternal() short-circuits to FINISHED when !sink.external && !PropagatesBuildSide(join_type).

3.2 Pipelines and pipeline breakers

Pipelines are cut at pipeline breakers: operators that cannot emit a single row until they have seen all their input. Hash-join build sides, hash aggregation, sorting and window functions all qualify.

A set of pipelines sharing one sink is a MetaPipeline.

// duckdb/parallel/meta_pipeline.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/parallel/meta_pipeline.hpp
//! MetaPipeline represents a set of pipelines that all have the same sink
class MetaPipeline : public enable_shared_from_this<MetaPipeline> {
	// ...
};

The two branches of a UNION ALL, for instance, are separate pipelines that pour into the same sink, so they belong to one MetaPipeline (PhysicalUnion::BuildPipelines calls meta_pipeline.CreateUnionPipeline()). Crossing a sink is the other case — CreateChildMetaPipeline() starts a new one — so the four pipelines in the diagram above are four MetaPipelines linked parent-to-child.

3.3 HAVE_MORE_OUTPUT — when output expands

Push-based execution has a problem pull-based execution does not: one input chunk can produce more than 2048 output rows. If each input row of a hash join matches a hundred rows, a 2048-row input becomes about 200,000 output rows.

DuckDB handles this with OperatorResultType:

// duckdb/common/enums/operator_result_type.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/common/enums/operator_result_type.hpp
// Upstream declares the enumerators on one line, with the semantics in a block
// comment above the enum. The per-value comments below are the author's summary.
enum class OperatorResultType : uint8_t {
	NEED_MORE_INPUT,   // done with this input, give me the next
	HAVE_MORE_OUTPUT,  // more output from the same input, call me again
	FINISHED,          // the whole pipeline is done: neither this nor any other operator in it is called again (LIMIT satisfied, etc.)
	BLOCKED            // waiting asynchronously, resume me later (intermediate operators do not emit this — see §3.4)
};

On HAVE_MORE_OUTPUT, the PipelineExecutor pushes that operator onto an in_process_operators stack.

// duckdb/parallel/pipeline_executor.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/parallel/pipeline_executor.hpp
//! If the stack of in_process_operators is empty, we fetch from the source instead
stack<idx_t> in_process_operators;

While the stack is non-empty, the executor does not return to the source: it re-enters the same operator with the same input chunk. This works even with several expanding operators in one pipeline, because the stack records how far back to rewind. The push model resumes in the middle of a chunk through this one stack.

3.4 BLOCKED — waiting for I/O without wasting a thread

BLOCKED is a relatively recent addition for queries that involve network I/O, such as reading Parquet from S3.

In a pull model the worker thread would simply block on the socket and sit idle. In DuckDB, when a source or sink returns BLOCKED, the PipelineExecutor propagates INTERRUPTED, and the PipelineTask returns TASK_BLOCKED, handing the thread back to the scheduler. When the I/O completes, the callback held in InterruptState fires and the task is rescheduled.

// src/parallel/pipeline_executor.cpp — https://github.com/duckdb/duckdb/blob/main/src/parallel/pipeline_executor.cpp
// SINK INTERRUPT
if (result == OperatorResultType::BLOCKED) {
	remaining_sink_chunk = true;
	return PipelineExecuteResult::INTERRUPTED;
}

This re-entrancy is why PipelineExecutor is full of flags like remaining_sink_chunk: all progress must be recorded in explicit members so an interrupted pipeline resumes at precisely the right point. The result is a hand-written state machine that is hard to read, the price of asynchrony without coroutines.

4. Morsel-Driven Parallelism — Throwing Away the Exchange Operator

4.1 What is wrong with the traditional approach

Traditional parallel databases parallelize plans with exchange operators (Volcano parallelism): insert operators that scatter data into N partitions and run the downstream N ways.

That brings three problems:

1. Degree of parallelism is fixed at plan time, so CPUs that become idle mid-query cannot be put to work.
2. It is fragile under skew: one heavy partition slows the whole query.
3. Scattering is costly, because the data must be materialized.

4.2 Morsels

DuckDB follows Leis et al., "Morsel-Driven Parallelism" (SIGMOD 2014):

• Split the input into morsels. For a table scan a morsel is one row group — 122,880 rows by default.
• Launch N identical tasks per pipeline.
• Each task advances a global cursor (under a short lock) to claim its next morsel.

In code, RowGroupCollection::NextParallelScan() is what advances that cursor, and DataTable::MaxThreads() caps the number of threads a scan can use:

// src/storage/data_table.cpp — https://github.com/duckdb/duckdb/blob/main/src/storage/data_table.cpp
idx_t DataTable::MaxThreads(ClientContext &context) const {
	idx_t row_group_size = GetRowGroupSize();
	idx_t parallel_scan_vector_count = row_group_size / STANDARD_VECTOR_SIZE;
	if (ClientConfig::GetConfig(context).verify_parallelism) {
		parallel_scan_vector_count = 1;
	}
	idx_t parallel_scan_tuple_count = STANDARD_VECTOR_SIZE * parallel_scan_vector_count;
	return GetTotalRows() / parallel_scan_tuple_count + 1;
}

Normally that is "total rows / row group size + 1". Turning on the verify_parallelism debug setting shrinks the morsel to a single vector, but that mode exists only to stress-test parallelism in the test suite.

No exchange operator appears in the plan; Pipeline::GetMaxThreads() decides parallelism dynamically.

Row groups suit both the execution engine and storage as a morsel size. A row group is the smallest unit that owns its own per-column ColumnSegments and min/max statistics, so aligning morsels to row groups makes segment-level pruning (§5.3) line up exactly with task assignment. And 122,880 rows is 60 vectors' worth of work per cursor advance, which keeps the lock around the global cursor from becoming a bottleneck. The downside is that for a table smaller than one row group, MaxThreads() returns 1 and the scan is effectively single-threaded.

The visualization below runs twelve morsels with deliberately skewed costs on four threads and compares a static split with morsel-driven scheduling. While one thread is busy with a heavy morsel, the idle threads take the remaining morsels, which eases the skew problem from §4.1.

4.3 Global versus local state

Morsel-driven scheduling only works because operator state is cleanly split into global and local halves.

// duckdb/execution/physical_operator.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/execution/physical_operator.hpp
// (source and sink declarations live in separate sections; collected here for contrast)
virtual unique_ptr<LocalSourceState> GetLocalSourceState(ExecutionContext &context, GlobalSourceState &gstate) const;
virtual unique_ptr<GlobalSourceState> GetGlobalSourceState(ClientContext &context) const;
// ...
virtual unique_ptr<LocalSinkState> GetLocalSinkState(ExecutionContext &context) const;
virtual unique_ptr<GlobalSinkState> GetGlobalSinkState(ClientContext &context) const;

• Local state belongs to one thread and needs no locks: a thread-local hash table for aggregation, the current scan position for a table scan.
• Global state is shared and guarded by a lock or atomics: the "next row group to read" cursor.

When a thread runs out of work it merges its local state into the global one via Combine(), and the last one calls Finalize().

thread 0 ─ Sink() × n ─┐
thread 1 ─ Sink() × n ─┼─ Combine() ─→ GlobalSinkState ─ Finalize() ─→ next pipeline
thread 2 ─ Sink() × n ─┘

Sink() runs per chunk, so it must avoid locks. Combine() runs once per thread, so a lock is fine. Finalize() runs once, so anything goes. Matching the locking to these three levels of granularity keeps lock contention close to zero.

4.4 Task granularity

To stop a single task from processing every morsel, DuckDB has a partial-execution mode.

// duckdb/parallel/pipeline.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/parallel/pipeline.hpp
class PipelineTask : public ExecutorTask {
	static constexpr const idx_t PARTIAL_CHUNK_COUNT = 50;
	// ...
};

Under TaskExecutionMode::PROCESS_PARTIAL, PipelineExecutor::Execute(50) returns PipelineExecuteResult::NOT_FINISHED after 50 chunks (up to 102,400 rows), and PipelineTask::ExecuteTask() translates that into TaskExecutionResult::TASK_NOT_FINISHED to yield to the scheduler. That preserves cancellation responsiveness and fairness with other queries.

5. Storage — a Columnar Store Sealed in One File

On disk, the data is laid out in units derived directly from the execution engine's 2048.

5.1 File layout

A DuckDB database is a single file, with fixed-size headers up front.

+---------------------------+  offset 0
| MainHeader (4096 B)       |  [0..8) checksum, magic "DUCK" at offset 8, library version / git hash
+---------------------------+  offset 4096
| DatabaseHeader #0 (4096 B)|  iteration, meta_block, free_list, block_count, ...
+---------------------------+  offset 8192
| DatabaseHeader #1 (4096 B)|
+---------------------------+  offset 12288  (SingleFileBlockManager::BLOCK_START)
| Block 0 (256 KB)          |
| Block 1 (256 KB)          |
| ...                       |
+---------------------------+

The file has two DatabaseHeaders.

// duckdb/storage/storage_info.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/storage/storage_info.hpp
//! The DatabaseHeader contains information about the current state of the database. Every storage file has two
//! DatabaseHeaders. On startup, the DatabaseHeader with the highest iteration count is used as the active header. When
//! a checkpoint is performed, the active DatabaseHeader is switched by increasing the iteration count of the
//! DatabaseHeader.
struct DatabaseHeader {
	//! The iteration count, increases by 1 every time the storage is checkpointed.
	uint64_t iteration = 0;
	idx_t meta_block = 0;
	idx_t free_list = 0;
	uint64_t block_count = 0;
	idx_t block_alloc_size = 0;
	idx_t vector_size = 0;
	StorageVersion storage_compatibility = StorageVersion::INVALID;
	// ...
};

A checkpoint writes the new state into whichever header is not currently active and bumps iteration. Each header is 4096 bytes (Storage::FILE_HEADER_SIZE, matched to the page size; the identically-valued Storage::SECTOR_SIZE is a separate constant used for direct I/O). The source only claims the disk writes that size "more or less atomically", so the safety comes from the write ordering around it.

// src/storage/single_file_block_manager.cpp — https://github.com/duckdb/duckdb/blob/main/src/storage/single_file_block_manager.cpp
// We need to fsync BEFORE we write the header to ensure that all the previous blocks are written as well
handle->Sync();
// ...
auto location = active_header == 1 ? Storage::FILE_HEADER_SIZE : Storage::FILE_HEADER_SIZE * 2;
ChecksumAndWrite(context, header_buffer, location);
// switch active header to the other header
active_header = 1 - active_header;
//! Ensure the header write ends up on disk
handle->Sync();

DuckDB fsyncs the data blocks, writes into the header slot that is not active, switches active_header, and fsyncs again. If the process dies mid-write, the older valid header is untouched: this is atomic checkpointing via shadow paging. Each header also carries an 8-byte checksum, so CheckChecksum() catches a torn write at open time and raises DataCorruptionException instead of reading garbage.

The header also stores vector_size and block_alloc_size, so a binary built with a different STANDARD_VECTOR_SIZE cannot open the file by accident.

5.2 RowGroup and ColumnSegment

A table is a RowGroupCollection, whose contents are a sequence of row groups.

// duckdb/storage/storage_info.hpp
//! The standard row group size
#define DEFAULT_ROW_GROUP_SIZE 122880ULL

122,880 = 60 × 2048, so one row group is exactly 60 vectors. The compiler checks that the row group size is a multiple of the vector size:

// duckdb/storage/storage_info.hpp
#if (DEFAULT_ROW_GROUP_SIZE % STANDARD_VECTOR_SIZE != 0)
#error The row group size must be a multiple of the vector size
#endif

Within a row group, each column has a ColumnData, whose contents are a sequence of ColumnSegments.

Each ColumnSegment lives in a block and picks its own compression scheme independently. A block is 256 KB by default (DEFAULT_BLOCK_ALLOC_SIZE 262144ULL); its first 8 bytes are a checksum, leaving 262,136 bytes of payload, and 256 KB is also the maximum, MAX_BLOCK_ALLOC_SIZE. The same column can be compressed differently in different row groups, for example with RLE in a row group where one value repeats in long runs and with bit packing in one where the values rise almost monotonically.

5.3 Zone maps — reading nothing at all

Every segment carries min/max statistics (BaseStatistics). For WHERE l_shipdate > DATE '1998-01-01', any segment whose max(l_shipdate) <= '1998-01-01' can be skipped without reading a single byte.

These statistics form a zone map (a min-max index), but DuckDB never builds it as a separate step: the statistics are collected as a by-product of writing. Every compression scheme's compress state embeds a StatsWriter<T> that updates min/max as the values go by, and so do the uncompressed path (string_uncompressed.hpp) and in-place updates to existing rows (update_segment.cpp). For range filters on naturally sorted columns, such as time series, this alone removes most of the I/O.

5.4 Compression is decided by a contest

In DuckDB, candidate compression schemes compete for each segment.

At checkpoint time, ColumnDataCheckpointer runs three phases for every candidate that supports the physical type (DBConfig::GetCompressionFunctions(physical_type)):

1. InitAnalyze — create per-candidate analysis state
2. Analyze(vector) — stream every vector of the segment through it (returning false drops the candidate)
3. FinalAnalyze — return an estimated byte count

The smallest estimate wins. The losers are discarded, and only the winner goes through InitCompression → Compress → FinalizeCompress. The design is expensive, because the data is scanned twice, but in analytical workloads data written once is read many times, so the extra scan pays for itself.

Change the input pattern in the lab below to see which scheme wins and where the intuition "sequential → RLE, random → bit packing" stops holding.

The main schemes, from src/storage/compression/:

FSST (Fast Static Symbol Table) comes from Boncz, Neumann and Leis (VLDB 2020), and its defining property is that you can search the data while it is still compressed. LZ4 or Zstd must decompress a whole block before you can read one row; FSST decodes each string independently. That is what lets an FSST-compressed column travel as an FSST_VECTOR and stay compressed until an operator needs the bytes (§2.4).

ALP (Adaptive Lossless floating-Point, SIGMOD 2024) starts from the observation that many real-world DOUBLEs, such as prices and measurements, are really decimals. 938.71 is 93871 × 10^-2. ALP picks an exponent e and factor f per 1024 values, rounds value × 10^e to an integer, verifies the round trip, and on success stores it as FOR plus bit packing. Outliers go into an exception list.

// duckdb/storage/compression/alp/alp_constants.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/storage/compression/alp/alp_constants.hpp
class AlpConstants {
public:
	static constexpr uint32_t ALP_VECTOR_SIZE = 1024;
	static constexpr uint32_t RG_SAMPLES = 8;
	static constexpr uint16_t SAMPLES_PER_VECTOR = 32;
	// We calculate how many equidistant vector we must jump within a rowgroup
	static constexpr uint32_t RG_SAMPLES_DUCKDB_JUMP = (DEFAULT_ROW_GROUP_SIZE / RG_SAMPLES) / STANDARD_VECTOR_SIZE;
	// ...
};

As RG_SAMPLES = 8 and SAMPLES_PER_VECTOR = 32 indicate, ALP does not scan a whole row group: it picks eight equidistant vectors and samples 32 values from each. Those 8 × 32 = 256 values only drive FindTopKCombinations, which shortlists exponent/factor pairs; the final size estimate then actually compresses the eight complete sampled vectors (up to 1024 values each). Sampling vectors from the row group, then values from each vector, balances analysis cost against representativeness.

5.5 The buffer manager

DuckDB can process data larger than memory, and BufferManager makes that possible.

• Blocks are managed as BlockHandles; Pin() brings one into physical memory and Unpin() makes it an eviction candidate.
• Eviction starts above the memory limit (by default 80% of physical memory, cgroup-aware). It is not a true LRU but a FIFO concurrent queue of unpin events: unpinning appends an entry, re-pinning and unpinning again appends a newer one, and the stale entry is skipped as a "dead node" because its sequence number no longer matches. This approximates LRU without locks; the comment in EvictionQueue::Purge() says we want to keep the LRU characteristic alive.
• The queues come in three tiers, ordered by eviction cost: BLOCK / EXTERNAL_FILE first (cheap, just free), then MANAGED_BUFFER (has to be written to storage), and TINY_BUFFER last.
• Persistent blocks are already in the database file, so evicting one simply drops it from memory; temporary blocks — hash tables, sort buffers — are spilled to temp_directory.

MemoryTag (BASE_TABLE, HASH_TABLE, ORDER_BY, TRANSACTION and others) classifies memory use, and the duckdb_memory() table function shows the breakdown.

6. The Heavy Operators — Aggregation, Join, Sort, Window

Hash aggregation, hash join, sorting and window functions are pipeline breakers. None of them can emit a row until they have seen all their input, so each has to decide for itself what to do when memory runs out, which makes them some of the most intricate parts of the engine. Two pieces they share are covered along the way: the hash-table entry representation (§6.2) and the memory arbiter (§6.4).

6.1 Radix-partitioned hash aggregation

GROUP BY is implemented in src/execution/radix_partitioned_hashtable.cpp. As the name says, the partition comes from bits of the group-key hash — but not the topmost ones.

// duckdb/common/radix_partitioning.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/common/radix_partitioning.hpp
//! 4096 partitions ought to be enough to go out-of-core properly
static constexpr const idx_t MAX_RADIX_BITS = 12;
 
//! Radix bits begin after uint16_t because these bits are used as salt in the aggregate HT
static inline constexpr idx_t Shift(idx_t radix_bits) {
	return (sizeof(hash_t) - sizeof(uint16_t)) * 8 - radix_bits;   // = 48 - radix_bits
}

The partition index is therefore taken from the radix_bits bits immediately below the top 16. Those top 16 bits are left alone because they are the hash-table salt of §6.2; if the two overlapped, every entry in a partition would share those salt bits, and the salt pre-filter would lose much of its power.

MAX_RADIX_BITS = 12 (4096 partitions) is the ceiling of the radix-partitioning framework as a whole, though; hash aggregation uses far fewer. Its own cap is RadixHTConfig::MAXIMUM_FINAL_SINK_RADIX_BITS = 8 (256 partitions), trimmed further for wide rows — one bit at or above ROW_WIDTH_THRESHOLD_ONE = 32 bytes, two at or above ROW_WIDTH_THRESHOLD_TWO = 64 — because otherwise, as the comment says, cache misses get out of hand.

The partitioning serves two purposes:

1. Parallelism: a given group key always lands in one partition, so partitions are fully independent and the finalize step needs no cross-partition synchronization.
2. A spilling unit: under memory pressure you can pin one partition at a time and leave the rest on disk.

Pressure is detected by MaybeRepartition():

// src/execution/radix_partitioned_hashtable.cpp
const auto total_size =
    aggregate_allocator_size + ht.GetPartitionedData().SizeInBytes() + ht.Capacity() * sizeof(ht_entry_t);
if (total_size > gstate.GetThreadLimit()) {
	// We're over the thread memory limit
	// ...
}

Above the per-thread limit, SetRadixBitsToExternal() flips the operator into external mode and Repartition() increases the partition count, shrinking each one until it reliably fits in memory.

Newer DuckDB versions add spilling of the aggregate states themselves (AggregateStateSpilling). For aggregates whose state grows large, such as string_agg and list, spilling the row data still leaves the arena allocator resident. DuckDB therefore "exports" the states into a ColumnDataCollection so the arena can be freed, and "imports" them back during finalize.

The hash aggregation is also adaptive. Each thread starts with a deliberately small hash table, estimates distinct keys with HyperLogLog as it goes, and after 1,048,576 rows (RadixHTLocalSinkState::ADAPTIVITY_THRESHOLD) calls DecideAdaptation():

• If the estimated unique ratio exceeds 95% (SKIP_LOOKUP_UNIQUE_PERCENTAGE_THRESHOLD), meaning almost nothing is being aggregated, it stops probing the hash table and only appends, deferring all deduplication to the GetData phase (ht.SkipLookups()).
• If it finds that a larger table would have deduplicated more than twice as well, it grows the capacity and rebuilds.

Both decisions are made after looking at the data, so the execution engine can recover on its own when cardinality estimation (§8.3) is wrong. The GetThreadLimit() above is not a constant either; it follows the amount the TemporaryMemoryManager (§6.4) has handed out.

(This adaptation only applies above GROW_STRATEGY_THREAD_THRESHOLD = 2 threads. At one or two threads DuckDB takes the "grow" strategy instead: SinkCapacity() starts at 262,144 rather than the adaptive path's 32,768, and DecideAdaptation() is never called.)

6.2 ht_entry_t — using every spare bit of a pointer

A hash-table entry is a single 64-bit word.

// duckdb/execution/ht_entry.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/execution/ht_entry.hpp
struct ht_entry_t {
public:
	//! Upper 16 bits are salt, lower 48 bits are the pointer
	static constexpr const hash_t SALT_MASK = 0xFFFF000000000000;
	static constexpr const hash_t POINTER_MASK = 0x0000FFFFFFFFFFFF;
	// ...
private:
	hash_t value;
};

x86-64 user-space pointers effectively use only 48 bits, so the top 16 bits are free. DuckDB stores part of the hash, a salt, in them.

When linear probing inspects a slot, it compares the salt first. On a mismatch it moves on without dereferencing the pointer. The most expensive step in hash-table probing is the cache miss behind that dereference, so a 16-bit pre-filter that cuts false positives to 1 in 65,536 saves a lot.

(On Android the optimization is disabled because of the virtual address space layout; DUCKDB_DISABLE_POINTER_SALT switches to salt-free entries. The source comment there includes some grumbling about it.)

In the join hash table, the salt comparison also only pays off once the table is big enough.

// duckdb/execution/join_hashtable.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/execution/join_hashtable.hpp
//! only compare salts with the ht entries if the capacity is larger than 8192 so that it does not fit into the CPU
//! cache
static constexpr const idx_t USE_SALT_THRESHOLD = 8192;

If the table already fits in cache, dereferencing is cheap and the pre-filter only adds work, so JoinHashTable::UseSalt() returns false and the salt comparison is skipped. The code thus acknowledges that even an optimization has a break-even point.

ht_entry.hpp also contains a branch-free modulo helper:

// uses an AND operation to apply the modulo operation instead of an if condition that could be branch mispredicted
inline void IncrementAndWrap(idx_t &offset, const uint64_t &capacity_mask) {
	++offset &= capacity_mask;
}

Capacities are kept at powers of two so that this one line works.

The join hash table sizes itself with a load factor that is the inverse of the usual ratio (entries ÷ capacity): it sets the minimum capacity as a multiple of the row count.

//! For a LOAD_FACTOR of 2.0, the HT is between 25% and 50% full
static constexpr double DEFAULT_LOAD_FACTOR = 2.0;

Capacity is at least twice the row count, rounded up to a power of two, so actual occupancy sits between 25% and 50%. The table is kept sparse on purpose to keep probe sequences short.

6.3 External hash joins and ProbeSpill

When the build side does not fit in memory, DuckDB radix-partitions it and spills probe rows that do not hit a currently resident partition.

// duckdb/execution/join_hashtable.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/execution/join_hashtable.hpp
//! ProbeSpill represents materialized probe-side data that could not be probed during PhysicalHashJoin::Execute
//! because the HashTable did not fit in memory. The ProbeSpill is not partitioned if the remaining data can be
//! dealt with in just 1 more round of probing, otherwise it is radix partitioned in the same way as the HashTable
struct ProbeSpill {
	// ...
};

The split during probing is one call to RadixPartitioning::Select(), which writes matching rows and spilling rows into two separate selection vectors:

// src/execution/join_hashtable.cpp — https://github.com/duckdb/duckdb/blob/main/src/execution/join_hashtable.cpp
const auto true_count =
    RadixPartitioning::Select(hashes, FlatVector::IncrementalSelectionVector(), probe_keys.size(), radix_bits,
                              current_partitions, &true_sel, &false_sel);

As the comment says, if one more round will finish the job, the spilled data is not partitioned at all, which avoids unnecessary partitioning work.

6.4 TemporaryMemoryManager

When several operators compete for memory, a fixed per-operator cap wastes memory: if a hash join and a sort run concurrently and one needs little memory, the other should be able to use the rest.

TemporaryMemoryManager is the global arbiter. Each operator registers a TemporaryMemoryState and declares "I need at least this much" (SetMinimumReservation) and "I would like this much more" (SetRemainingSizeAndUpdateReservation). The manager looks at total demand, hands out reservations, and each operator decides from its reservation whether to fall back to external mode.

6.5 The external hash join's state machine

The Source-side stages show how the ProbeSpill from §6.3 is used:

// src/execution/operator/join/physical_hash_join.cpp — https://github.com/duckdb/duckdb/blob/main/src/execution/operator/join/physical_hash_join.cpp
enum class HashJoinSourceStage : uint8_t { INIT, BUILD, PROBE, SCAN_HT, DONE };

An external join walks these once per round of resident partitions:

1. BUILD — construct the pointer table for the partitions that fit in memory this round (ExternalBuild).
2. PROBE — pull the spilled probe rows back chunk by chunk from probe_spill and probe them (ExternalProbe).
3. SCAN_HT — when PropagatesBuildSide holds (RIGHT / FULL OUTER / RIGHT_SEMI / RIGHT_ANTI), emit the remaining build-side rows (ExternalScanHT → ScanFullOuter).
4. Back to PrepareBuild(): BUILD again if more partitions fit, otherwise DONE.

The Source enters at PROBE, not BUILD. The first round's pointer table was already built by the Sink's Finalize() (PrepareExternalFinalize() → ScheduleFinalize()), so Initialize() sets global_stage straight to PROBE; BUILD only runs from the second round onwards.

Waiting for a stage transition reuses the BLOCKED machinery from §3.4 unchanged:

// src/execution/operator/join/physical_hash_join.cpp
if (gstate.TryPrepareNextStage(sink) || gstate.global_stage == HashJoinSourceStage::DONE) {
	gstate.UnblockTasks();
} else {
	return gstate.BlockSource(input.interrupt_state);
}

"Wait until the other threads finish this stage" uses the same mechanism as "wait for an S3 read": the thread goes back to the scheduler and the task is rescheduled later. So BLOCKED is not specific to async I/O; it is the general way to express any wait inside a pipeline.

6.6 Sorting — byte-comparable keys first, radix sort second

ORDER BY is one of the parts of DuckDB that were recently rewritten wholesale. It lives in src/common/sort/, while its headers are under src/include/duckdb/common/sorting/.

The sort starts by doing without a comparator. Sort holds an expression for a scalar function called create_sort_key, runs it on every sink chunk, and folds all the ORDER BY columns into one byte-comparable byte string. After that the comparison itself never looks at column types, ASC/DESC or NULLS FIRST/LAST again: a memcmp-style comparison already yields the right order. (The type and the sort specifier are needed again only to decode a key back into values — Sort::Sort binds a matching decode_sort_key expression right next to create_sort_key.)

The physical shape of that byte string is a SortKeyType:

// duckdb/common/sorting/sort_key.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/common/sorting/sort_key.hpp
enum class SortKeyType : uint8_t {
	INVALID = 0,
	//! Without payload
	NO_PAYLOAD_FIXED_8 = 1,
	NO_PAYLOAD_FIXED_16 = 2,
	NO_PAYLOAD_FIXED_24 = 3,
	NO_PAYLOAD_FIXED_32 = 4,
	NO_PAYLOAD_VARIABLE_32 = 5,
	//! With payload (requires row pointer in key)
	PAYLOAD_FIXED_16 = 6,
	PAYLOAD_FIXED_24 = 7,
	PAYLOAD_FIXED_32 = 8,
	PAYLOAD_VARIABLE_32 = 9,
};

Each width is pinned by a static_assert (sizeof(SortKey<NO_PAYLOAD_FIXED_8>) == 8, and so on). The PAYLOAD variants carry a pointer to their payload row inside the key struct (SetPayload / GetPayload), so sorting moves only 8-to-32-byte structs. The row data does not move at all, except when the sort goes external and SortedRun::Finalize(external) reorders it physically ("Sorts the data (physically reorder data if external)").

Yet the key is not actually stored in byte-comparable order:

// duckdb/common/sorting/sort_key.hpp — FixedSortKey::Construct
// IMPORTANT NOTE: We don't actually store the data in byte-comparable order!
// This allows us to do int64_t comparisons, yielding better performance.
// This means we have to ByteSwap once more when decoding the keys later.
ByteSwap();

ByteSwap() is BSwapIfLE, the same helper §2.6 uses for string_t prefix comparison. The bytes are read as an array of uint64_ts and compared lexicographically by SortKeyLessThan<PARTS>. This applies the §2.6 technique of turning a four-character lexicographic comparison into one integer comparison to the entire sort key.

DuckDB does not write its own sort algorithm here: it uses vergesort, with ska_sort (a radix sort) as the fallback.

// src/common/sort/sorted_run.cpp — https://github.com/duckdb/duckdb/blob/main/src/common/sort/sorted_run.cpp
const auto fallback = [ska_extract_key](const BLOCK_ITERATOR &fb_begin, const BLOCK_ITERATOR &fb_end) {
	duckdb_ska_sort::ska_sort(fb_begin, fb_end, ska_extract_key);
};
duckdb_vergesort::vergesort(begin, end, std::less<SORT_KEY>(), fallback);

The per-thread sorted runs are then merged by SortedRunMerger, again without an exchange operator. The output is cut into partitions, and each thread independently computes where its slice starts and ends inside every run using K-way Merge Path (COMPUTE_BOUNDARIES → ACQUIRE_BOUNDARIES → MERGE_PARTITION → SCAN_PARTITION). Once the boundaries are known the partitions are fully independent, so the merge parallelizes.

Inside a partition, DuckDB does not merge at all; it sorts again (except in special cases such as a single remaining run):

// src/common/sort/sorted_run_merger.cpp — https://github.com/duckdb/duckdb/blob/main/src/common/sort/sorted_run_merger.cpp
// Seems counter-intuitive to re-sort instead of merging, but modern sorting algorithms detect and merge
static const auto fallback = [](SORT_KEY *begin, SORT_KEY *end) {
	duckdb_pdqsort::pdqsort_branchless(begin, end);
};

Here too the actual call is duckdb_vergesort::vergesort(..., std::less<SORT_KEY>(), fallback), with pdqsort_branchless as the fallback, the same shape as the run sort. As the comment says, modern sorting algorithms already detect and merge pre-sorted subsequences, and the developers judged this to be faster than a hand-written K-way merge.

When the sort exceeds memory, BlockIteratorStateType flips from IN_MEMORY to EXTERNAL and the same templated code starts reading keys through the buffer manager instead.

Sort is also what window functions run on. SortStrategy::Factory() returns one of three implementations:

• FullSort: for OVER (ORDER BY ...).
• HashedSort: for OVER (PARTITION BY ...). It hashes the partition keys, radix-partitions the rows into hash groups, and gives each group its own sort state (sort_global / sort_source). All groups share a single Sort, which the header calls the "Common sort description".
• NaturalSort: for OVER (); it does no sorting at all.

HashedSort reuses the "hash-partition, then process each partition independently" structure from §6.1.

6.7 Window functions — hash groups and segment trees

PhysicalWindow in src/execution/operator/aggregate/physical_window.cpp is, as its comment says, an operator that assumes all functions share a common partitioning and ordering. Its Sink feeds straight into whichever strategy SortStrategy::Factory() from §6.6 picked (HashedSort when there is a PARTITION BY); its Source runs one independent state machine per hash group.

// src/execution/operator/aggregate/physical_window.cpp
enum WindowGroupStage : uint8_t { SORT, MATERIALIZE, MASK, SINK, FINALIZE, GETDATA, DONE };

A WindowHashGroup owns one group's sorted data, its boundary masks, the per-function global states and the per-thread local states. The MASK stage fills partition_mask and order_masks, both ValidityMasks. Because "the partition changes at this row" and "the ORDER BY peer group changes at this row" are recorded as single bits, every later frame-boundary computation is bit arithmetic.

Task assignment extends the morsel approach: groups are ordered largest-first, and a single group is split by block across several threads, so parallelism holds up even with fewer groups than threads. Waiting for a stage transition uses TryPrepareNextStage() / UnblockTasks() / BlockSource, the same BLOCKED machinery as the external hash join in §6.5.

Frame aggregation itself is WindowSegmentTree. Computing SUM(x) OVER (ROWS BETWEEN 100 PRECEDING AND CURRENT ROW) naively costs 100 updates per row; with a segment tree any range costs $O(\log n)$.

// src/function/window/window_segment_tree.cpp — https://github.com/duckdb/duckdb/blob/main/src/function/window/window_segment_tree.cpp
//! The actual window segment tree: an array of aggregate states that represent all the intermediate nodes
WindowAggregateStates levels_flat_native;
//! For each level, the starting location in the levels_flat_native array
vector<idx_t> levels_flat_start;
//! The level being built (read)
std::atomic<idx_t> build_level;

The tree's nodes are aggregate states themselves (raw states produced by WindowAggregateStates::Initialize()), and all levels are packed into one flat array, with levels_flat_start recording where each level begins. build_level is atomic because the tree is built level by level across several threads. If the aggregate is order-insensitive, the upper levels can all be combined and the leaves handled in one FramePart::FULL pass. If it is not, the leaves are evaluated LEFT → upper levels → RIGHT, with the upper levels' right-hand ranges cached and drained in reverse. (Splitting the frame itself into a left and a right half is a separate axis, driven by the EXCLUDE clause.)

Functions that need a rank within the frame, such as median and quantile, and value functions with an argument ORDER BY, such as first_value(x ORDER BY y), take a different route via MergeSortTree (WindowMergeSortTree / WindowIndexTree). It is an index that can answer SelectNth(frames, n), and it owns a Sort from §6.6 outright (sort = make_uniq<Sort>(...)). Window functions use sorting, hash partitioning, segment trees and BLOCKED, nearly every part covered so far, which is why they are the most intricate operator in the engine.

7. MVCC — Never Blocking a Reader

Everything so far has been about reading. Once someone writes, MVCC decides who can see what.

7.1 Where version information lives

DuckDB's MVCC follows Neumann, Mühlbauer and Kemper, "Fast Serializable Multi-Version Concurrency Control for Main-Memory Database Systems" (SIGMOD 2015), the HyPer design.

Row visibility is managed per row group by a RowVersionManager, whose contents are one ChunkVectorInfo per 2048 rows.

// src/storage/table/row_version_manager.cpp — https://github.com/duckdb/duckdb/blob/main/src/storage/table/row_version_manager.cpp
RowVersionManager::RowVersionManager(BufferManager &buffer_manager_p) noexcept
    : allocator(STANDARD_VECTOR_SIZE * sizeof(transaction_t), buffer_manager_p.GetTemporaryBlockManager(),
                MemoryTag::BASE_TABLE) {
}

ChunkVectorInfo records, per row, when it was inserted and when it was deleted, both as transaction_t. The storage form varies: when every row shares one id they collapse into constants (constant_insert_id / constant_delete_id); otherwise they live in per-row id arrays reached through IndexPointers (inserted_data / deleted_data) in a FixedSizeAllocator. The test is:

// conceptually
bool UseVersion(TransactionData transaction, transaction_t id) {
	return id < transaction.start_time || id == transaction.transaction_id;
}
bool visible = UseVersion(transaction, inserted[row]) && !UseVersion(transaction, deleted[row]);

The trick is in the value range of transaction_t.

// src/common/constants.cpp — https://github.com/duckdb/duckdb/blob/main/src/common/constants.cpp
const transaction_t TRANSACTION_ID_START = 4611686018427388000ULL;                // 2^62
const transaction_t MAX_TRANSACTION_ID = NumericLimits<transaction_t>::Maximum(); // 2^63
const transaction_t NOT_DELETED_ID = NumericLimits<transaction_t>::Maximum() - 1; // 2^64 - 1

• Below TRANSACTION_ID_START → a commit timestamp
• At or above it → an uncommitted transaction id

One field carries two meanings, and a single comparison tells them apart. Uncommitted ids are always enormous, so id < start_time is false and other transactions cannot see them. On commit, CommitState::CommitEntry overwrites the id with the commit timestamp, which makes all of the changes visible at once.

NOT_DELETED_ID is MAX - 1 for the same reason: "not deleted" is treated as "deleted infinitely far in the future".

(The values in the source comments are not exact. transaction_t is a uint64_t, so MAX_TRANSACTION_ID is 2^64 − 1, NOT_DELETED_ID is 2^64 − 2, and TRANSACTION_ID_START is 2^62 + 96.)

7.2 Compressing version ids

Stored naively, that costs two transaction_ts (16 bytes) per row, but in practice almost every row was committed long ago and never deleted. CompressVersionIds() exploits that.

// duckdb/storage/table/row_version_manager.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/storage/table/row_version_manager.hpp
//! Attempts to compress the per-row insert/delete ids of each vector into constants
//! This is possible when the ids behave identically for all transactions with a start time of at least
//! lowest_active_start (i.e. all active and future transactions)
//! ...
void CompressVersionIds(transaction_t lowest_active_start);

If every current and future transaction would get the same answer, an array of 2048 ids collapses into one constant. Deletions have three levels in DeleteIdState: CONSTANT, MASKED and ARRAY, meaning "same for all rows", "a bitmask plus one delete id", and "different per row".

7.3 The UndoBuffer

Everything needed to undo a transaction's changes is appended linearly to an UndoBuffer.

// src/transaction/undo_buffer.cpp — https://github.com/duckdb/duckdb/blob/main/src/transaction/undo_buffer.cpp
constexpr uint32_t UNDO_ENTRY_HEADER_SIZE = sizeof(UndoFlags) + sizeof(uint32_t);
 
UndoBufferReference UndoBuffer::CreateEntry(UndoFlags type, idx_t len) {
	idx_t alloc_len = AlignValue<idx_t>(len + UNDO_ENTRY_HEADER_SIZE);
	auto handle = allocator.Allocate(alloc_len);
	auto data = handle.GetDataMutable();
	// write the undo entry metadata
	Store<UndoFlags>(type, data);
	data += sizeof(UndoFlags);
	Store<uint32_t>(UnsafeNumericCast<uint32_t>(alloc_len - UNDO_ENTRY_HEADER_SIZE), data);
	// increment the position of the header past the undo entry metadata
	handle.position += UNDO_ENTRY_HEADER_SIZE;
	return handle;
}

The buffer is a hand-written byte stream of (type tag, length, payload) entries, and the same buffer is walked once for each purpose:

Only rollback needs the reverse direction, and the source says only that: // rollback needs to be performed in reverse. On the commit side each entry is essentially independent (UPDATE_TUPLE is a concurrency check followed by the single line info->version_number = commit_id;; INSERT_TUPLE calls CommitAppend). Insertion order matters in two places. One is catalog changes: entries touching the same object, such as a CREATE followed by an ALTER, must be stamped with the commit timestamp in the order they were appended. The other is the WAL write, which walks the same buffer in insertion order, because a log only replays correctly if its operations are in the order they happened.

UndoBufferAllocator allocates through the BufferManager, so undo information for a huge transaction can spill to disk too. Updating a billion rows in one transaction therefore does not run out of memory because of undo data.

7.4 WAL, optimistic writing and checkpoints

The write path looks like this:

During a bulk insert, DuckDB uses optimistic writing: it writes completed row groups straight into the database file without waiting for the commit.

• If the transaction commits, the WAL only needs pointers to those blocks, so the payload is never written twice.
• If it rolls back, the blocks return to the free list.

DuckTransaction::PreFlushOptimisticBlocks() also runs before the commit locks are taken, which keeps slow operations like fsync out of the critical section.

Checkpoints run automatically once the WAL exceeds the public checkpoint_threshold setting (16 MiB by default; internally stored as checkpoint_wal_size), and RowGroupCollection::Checkpoint() flushes the row groups to disk. Unchanged row groups reuse their existing metadata and are skipped (RowGroupWriteAction::REUSE_EXISTING_ROW_GROUP_METADATA), so updating one row does not rewrite the whole table.

The flush is also column-at-a-time:

// src/storage/table/row_group.cpp — https://github.com/duckdb/duckdb/blob/main/src/storage/table/row_group.cpp
// In order to co-locate columns across different row groups, we write column-at-a-time
// i.e. we first write column #0 of all row groups, then column #1, ...

This places data for the same column contiguously on disk, which minimizes seek distance for a single-column scan.

8. The Optimizer

The execution engine described so far runs a given physical plan efficiently. The optimizer decides which plan that is.

8.1 A sequence of passes

DuckDB's optimizer is a sequence of transformations over the logical plan (src/optimizer/), including expression rewriting, filter pushdown, unused-column removal, statistics propagation, join ordering, common subexpression elimination and regex range extraction (regex_range).

For example, when regex_range finds a two-argument regexp_full_match with a constant pattern, it asks RE2's PossibleMatchRange for bounds, and if it gets them it inserts col >= min AND col <= max as an extra filter below the original. The regex is still evaluated, but the zone maps of §5.3 can prune first.

Individual passes can be turned off with a comma-separated list (SET disabled_optimizers='join_order'), and SELECT * FROM duckdb_optimizers(); lists every valid name, so you can measure what each pass contributes.

8.2 Join ordering — DPhyp

Join ordering lives in src/optimizer/join_order/plan_enumerator.cpp, and the comment names its source:

// the plan enumeration is a straight implementation of the paper "Dynamic Programming Strikes Back" by Guido
// Moerkotte and Thomas Neumannn, see that paper for additional info/documentation bonus slides:
// https://db.in.tum.de/teaching/ws1415/queryopt/chapter3.pdf?lang=de
bool PlanEnumerator::SolveJoinOrder() {
	// ...
}

The strings "DPccp" and "DPhyp" appear nowhere in the code, but the function names (EmitCSG, EnumerateCSGRecursive, EnumerateCmpRecursive) match the paper's, so the code can be read alongside it. They identify the algorithm as the CSG-CMP-pair enumeration introduced by DPccp (VLDB 2006) and extended to hypergraphs as DPhyp in "Dynamic Programming Strikes Back". Either way it is far cheaper than a DP that tries every subset, because it enumerates only connected subgraphs of the query graph. For a query whose relations form a chain, only a small fraction of the $2^n$ subsets is ever examined.

The enumeration also has a fallback:

auto swap_to_approximate_threshold = Settings::Get<ApproximateJoinOrderThresholdSetting>(query_graph_manager.context);
// first try to solve the join order exactly
bool solved;
if (query_graph_manager.relation_manager.NumRelations() >= swap_to_approximate_threshold) {
	solved = SolveJoinOrderApproximately();
} else {
	auto completed_exactly = SolveJoinOrderExactly();
	// ...
}

When the relation count reaches a threshold, or once the budget of examined pairs (pairs) runs out, it switches to a greedy algorithm (SolveJoinOrderApproximately), giving up optimality to keep optimization time bounded.

8.3 Cardinality estimation

CardinalityEstimator supplies the intermediate sizes the cost model needs, using a numerator/denominator model:

• Numerator: the product of the base cardinalities of the participating relations
• Denominator: the product of the total domain (estimated distinct count of the equi-join column) for each join predicate

This generalizes the classic "equi-join selectivity = 1 / max(distinct)" to subgraphs. The implementation builds equivalence classes (if a.x = b.x and b.x = c.x, all three share one total domain) and uses EquivalenceGroupApplied to avoid double-counting transitively redundant predicates.

DuckDB keeps HyperLogLog distinct-count estimates on join columns, which feed total domains as a DistinctCountSource. When the source is neither HLL nor EXACT, that is, when IsReliableDistinctCount returns false, it falls back to a precomputed, conservative fallback_distinct_count.

8.4 Runtime filter propagation

DuckDB also builds filters while the query runs. Planning only decides where a filter will later be injected (which column of which LogicalGet) and leaves the slot empty until the build finishes. The reserved slot is a DynamicTableFilterSet; what goes into it is specified by a JoinFilterPushdownInfo.

// duckdb/execution/operator/join/join_filter_pushdown.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/execution/operator/join/join_filter_pushdown.hpp
struct JoinFilterPushdownInfo {
	//! The join condition indexes for which we compute the min/max aggregates
	vector<idx_t> join_condition;
	//! The probes to push the filter into
	vector<JoinFilterPushdownFilter> probe_info;
	//! Min/Max aggregates
	vector<unique_ptr<Expression>> min_max_aggregates;
	//! Whether the build side has a filter -> we might be able to push down a bloom filter into the probe side
	bool build_side_has_filter;
};

min_max_aggregates holds real aggregate expressions. The min/max is not computed after the build finishes; it is fed into a thread-local LocalUngroupedAggregateState inside the build's Sink() and merged globally in Combine(), reusing the three levels of granularity from §4.3.

At finalize time DuckDB pushes more than min/max. FinalizeFilters() inspects the build side and selectively injects up to four kinds of filter:

Bloom and prefix-range filters add cost on the build side, so the gates are strict. CanUseBloomFilter() gives the reason in a comment: building the bloom filter is costly on the build to make probing faster, so only use it if there are less build tuples than probing tuples.

Bloom and prefix-range filters are also always wrapped as selectivity-optional filters, and min/max filters are too for non-equality comparisons. After the scan starts, if the first few vectors show the filter is not rejecting much, it switches itself off and is no longer evaluated. A wrong planning-time estimate therefore has a bounded cost rather than one that grows with the data. (IN lists are never wrapped: they hold the actual values, so their selectivity is already known.)

For queries shaped like "small dimension table × huge fact table", such as TPC-H Q3, a narrow build-side range means most of the fact table's row groups can be rejected on min/max alone.

9. Indexes — Why ART?

9.1 Does an analytical database need indexes?

With columnar storage and zone maps, most analytical queries need no index at all. DuckDB still has one, primarily to enforce constraints (PRIMARY KEY / UNIQUE / FOREIGN KEY) and to serve highly selective point lookups.

DuckDB uses an ART (Adaptive Radix Tree) rather than a B+ tree, following Leis, Kemper and Neumann, "The Adaptive Radix Tree: ARTful Indexing for Main-Memory Databases" (ICDE 2013).

ART is designed for main memory. B+ tree lookups binary-search inside each node, so every comparison risks straddling a cache line. A radix tree (trie) indexes directly by key byte without comparisons, and its height depends only on key length. A plain radix tree, however, stores 256 pointers per node, which wastes most of the space in sparse trees.

ART solves this by changing the physical node representation according to the number of children.

9.2 Adaptive nodes

DuckDB's NType in src/include/duckdb/execution/index/art/node.hpp lists every variant:

// duckdb/execution/index/art/node.hpp
enum class NType : uint8_t {
	PREFIX = 1,
	LEAF = 2,
	NODE_4 = 3,
	NODE_16 = 4,
	NODE_48 = 5,
	NODE_256 = 6,
	LEAF_INLINED = 7,
	NODE_7_LEAF = 8,
	NODE_15_LEAF = 9,
	NODE_256_LEAF = 10,
};

Alongside the paper's Node4 / Node16 / Node48 / Node256, several DuckDB-specific kinds appear:

• PREFIX: path compression, where a run of non-branching bytes collapses into one node. The prefix length is configurable per ART instance (ART::PrefixCount()).
• LEAF_INLINED: with exactly one row id, no leaf node is allocated; the row id is embedded in the pointer. In a unique index this is the common case.
• NODE_7_LEAF / NODE_15_LEAF / NODE_256_LEAF: nodes that hold only the present bytes and no child pointers.

Add children one at a time in the visualization below to watch the node switch Node4 → Node16 → Node48 → Node256.

Node48 shows the memory saving over a plain trie most clearly:

// duckdb/execution/index/art/node48.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/execution/index/art/node48.hpp
//! Node48 holds up to 48 children. The child_index array is indexed by the key byte.
//! It contains the position of the child node in the children array.
class Node48 {
	// ...
	static constexpr uint8_t CAPACITY = 48;
	static constexpr uint8_t EMPTY_MARKER = 48;
	static constexpr uint8_t SHRINK_THRESHOLD = 12;
private:
	uint8_t count;
	uint8_t child_index[Node256::CAPACITY];
	NodePtr children[CAPACITY];
};

Even with a 256-byte index array, sizeof(Node48) is 648 bytes once NodePtr's 8-byte alignment is accounted for, against 2056 for a Node256: under a third of the footprint for up to 48 children. The one-byte indirection table is what lets most of the eight-byte pointer array go.

Shrinking is not the mirror image of growth. Node256::SHRINK_THRESHOLD = 36 and Node48::SHRINK_THRESHOLD = 12 sit well below the growth thresholds of 48 and 16. This deliberate hysteresis keeps repeated inserts and deletes near a boundary from switching the node type back and forth.

9.3 Gates — a tree inside the tree

In a non-unique index, one key maps to many row ids. DuckDB represents them not as a list but as a nested ART keyed by row id, and marks the boundary with a gate.

// duckdb/execution/index/art/node.hpp
//! A gate sets the leftmost bit of the metadata, binary: 1000-0000.
static constexpr uint8_t AND_GATE = 0x80;
static constexpr idx_t AND_ROW_ID = 0x00FFFFFFFFFFFFFF;

Setting the top bit of the NodePtr metadata byte marks everything below as a nested ART. Like ht_entry_t, this uses the spare bits of a pointer.

The leaf-specialized nodes such as NODE_7_LEAF exist for the inside of that nested tree. Row ids are a fixed eight bytes, so the bottom level only needs the set of final bytes and no child pointers. Node256Leaf is therefore just a 256-bit mask:

// duckdb/execution/index/art/node256_leaf.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/execution/index/art/node256_leaf.hpp
//! Node256Leaf is a bitmask containing 256 bits.
class Node256Leaf {
	// ...
	uint16_t count;
	validity_t mask[CAPACITY / sizeof(validity_t)];
};

2048 bytes of child pointers collapse into a 256-bit mask. (Only 256 bits, or 32 bytes, are logically needed, but the declaration validity_t mask[CAPACITY / sizeof(validity_t)] reserves 32 × 8 = 256 bytes, so sizeof(Node256Leaf) is 264. v2.0.0 changed the declaration to mask[CAPACITY / ValidityMask::BITS_PER_VALUE], which shrinks the mask to 32 bytes.) The same ART framework thus applies different optimizations to the outer tree and the nested one.

9.4 Where ART nodes are allocated

ART nodes are not allocated with new; they come from a per-node-type FixedSizeAllocator.

// duckdb/execution/index/art/art.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/execution/index/art/art.hpp
//! Fixed-size allocators holding the ART nodes.
shared_ptr<array<unsafe_unique_ptr<FixedSizeAllocator>, ALLOCATOR_COUNT>> allocators;

NodePtr is not a raw pointer but an IndexPointer, a 64-bit value meaning "buffer X, slot Y". Combined with the per-type allocators, this indirection has three benefits:

• Nodes of the same type sit contiguously, improving cache behavior.
• The whole index can live under the buffer manager, so indexes larger than memory work.
• Serializing to disk needs no pointer rewriting.

This indirection is what allows an ART to be persisted.

10. Extensibility — a Small Core plus Extensions

10.1 Extensions

DuckDB keeps its core small. Parquet, JSON, httpfs (S3/HTTP access), spatial, iceberg, delta, fts (full-text search), vss (vector search) and more are separated out as core extensions, DuckDB's term for extensions maintained by the DuckDB team and shipped from the official repository, as opposed to third-party community extensions.

Some of them still ship inside the binary: in the standard Python and CLI distributions, parquet, json and some others are statically linked into the binary and loaded through DBConfig::linked_extensions.

// duckdb/main/extension/linked_extension_registry.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/main/extension/linked_extension_registry.hpp
//! An extension compiled into this binary, as a name and the callable that loads it.
struct LinkedExtension {
	string name;
	//! empty when what was registered is not an extension
	std::function<void(DuckDB &)> load;
	// ...
};

That is why SELECT * FROM 'x.parquet' works without INSTALL parquet. Only extensions outside that set are loaded dynamically.

INSTALL httpfs;
LOAD httpfs;
SELECT count(*) FROM 's3://bucket/data/*.parquet';

A dynamically loaded extension is a shared library (.duckdb_extension) that registers scalar functions, table functions, types, optimizer passes and storage backends into the catalog at load time. To keep ABI compatibility across versions, extension binaries are built per core version and platform, and their signatures are verified.

Function resolution is FunctionBinder's job: given the candidate set for a name, it picks the best overload by implicit-cast cost. Scalar functions registered by extensions go through the same path as built-ins, so they join the same overload-resolution candidate set.

DBConfig carries the hooks that receive them:

// duckdb/main/config.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/main/config.hpp
//! Replacement table scans are automatically attempted when a table name cannot be found in the schema
vector<ReplacementScan> replacement_scans;

10.2 Replacement scans — why SELECT * FROM 'file.parquet' works

The comment above describes the mechanism. When the binder cannot resolve a table name, DuckDB tries the registered replacement scans in order before reporting an error.

• Name ends in .parquet: rewrite to parquet_scan('...') (an alias of the same function set as read_parquet). The parquet extension's own LoadInternal() registers this hook into config.replacement_scans, so it only exists once that extension is loaded.
• Name ends in .csv or .tsv: rewrite to read_csv_auto('...'). A .gz / .zst suffix is stripped first, so data.csv.gz works too.
• In the Python client, look through the caller's local variables for a pandas DataFrame or Arrow table and rewrite to a function that scans it. (PyArrow and Polars objects become an arrow scan; a DuckDBPyRelation becomes a subquery, not a table function. By default only the immediate calling frame is searched; python_scan_all_frames widens it to ancestor frames.)

import duckdb, pandas as pd
df = pd.DataFrame({"a": [1, 2, 3]})
duckdb.sql("SELECT sum(a) FROM df")   # df is not a table, yet this works

This behavior comes from one design decision: make name-resolution failure hookable. Because the rewrite happens at the logical plan level, the substituted table function still receives the usual optimizations, such as filter pushdown and projection pruning. That is why SELECT a FROM 'huge.parquet' WHERE b = 1 can consult the Parquet footer and skip the row groups and columns it does not need.

The inbound zero copy mentioned in §1 also goes through this mechanism. On the pandas route, ScanNumpyColumn / ScanNumpyFpColumn hand the NumPy buffer straight to FlatVector::SetData only when stride == sizeof(T); a strided array falls into a gather loop, and object columns are always converted (that code lives in the separate duckdb/duckdb-python repository, src/numpy/numpy_scan.cpp). On the Arrow route, DirectConversion in src/function/table/arrow_conversion.cpp calls FlatVector::SetData with no stride test at all, because for fixed-width primitives the Arrow layout is already one contiguous values buffer and only an offset is needed. That applies to fixed-width numerics and some temporal types only: BOOLEAN is bit-unpacked, and TIME and nanosecond timestamps are converted row by row.

10.3 Table functions

TableFunction is the standard socket for plugging external data into DuckDB. An implementer writes bind (decide column names and types), init_global / init_local (create parallel scan state) and function (fill one chunk).

The hooks for optimizations come in two forms, which are easy to confuse: boolean capability flags and callbacks.

// duckdb/function/table_function.hpp — https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/function/table_function.hpp
//! Whether or not the table function supports projection pushdown. If not supported a projection will be added
//! that filters out unused columns.
bool projection_pushdown;
//! Whether or not the table function supports filter pushdown. If not supported a filter will be added
//! that applies the table filter directly.
bool filter_pushdown;

Set projection_pushdown / filter_pushdown to true and the required columns and pushed-down filters arrive directly on the TableFunctionInitInput handed to init_global / init_local, as column_indexes and filters (leave them false and DuckDB adds a Projection / Filter operator on top of the scan instead). Implement the optional callbacks as well (cardinality, table_scan_progress, and pushdown_complex_filter, which receives arbitrary expressions, not just comparisons against constants) and a table function gets the same optimizations as a built-in table. Internally, the Parquet and CSV readers are table functions too.

10.4 The CSV reader — what "just a table function" hides

The CSV reader in src/execution/operator/csv_scanner/ is more than an ordinary table function, though. It tackles two hard problems: guessing the dialect and reading in parallel.

Dialect detection comes first. read_csv can guess the delimiter, types and header because it runs a staged detection process:

// duckdb/execution/operator/csv_scanner/sniffer/csv_sniffer.hpp
//! CSV Sniffing consists of five steps:
//! 1. Dialect Detection: Generate the CSV Options (delimiter, quote, escape, etc.)
//! 2. Type Detection: Figures out the types of the columns (For one chunk)
//! 3. Type Refinement: Refines the types of the columns for the remaining chunks
//! 4. Header Detection: Figures out if  the CSV file has a header and produces the names of the columns
//! 5. Type Replacement: Replaces the types of the columns if the user specified them

It uses the same kind of contest as the compression choice in §5.4. DetectDialect() expands the cartesian product of delimiter × comment character × (quote, escape) candidates, builds one state machine per combination, and actually parses the first chunk with each. The candidate whose per-row column count is most consistent wins, and RefineCandidates() keeps running the survivors over further chunks to break ties. The state machines are cached in CSVStateMachineCache; each one is CSVState state_machine[NUM_TRANSITIONS][NUM_STATES], a 256-character × 19-state transition table. The source comment reads "The Transition Array is a Finite State Machine" / "It holds the transitions of all states, on all 256 possible different characters".

Parallel reading comes next. CSV has no natural boundary like a Parquet row group, so DuckDB cuts the file into byte ranges and finds the line start afterwards:

// duckdb/execution/operator/csv_scanner/scanner_boundary.hpp
//! As in real life, much like my childhood country, the rules are not really enforced.
//! So the end boundaries of a Scanner Boundary can and will be pushed.
//! In practice this means that a scanner is tolerated to read one line over it's end.

CSVGlobalState::Next() only hands out the next byte range via CSVIterator::Next(), and the width is in bytes, not rows. The cut may land inside a quoted field: a newline inside a,"foo ... bar",c is not a record separator, and treating it as one misaligns every following row. StringValueScanner::SetStart() tries up to three interpretations and keeps the one that yields a valid row (the source comment says "At this point we have 3 options").

• TryRow(CSVState::STANDARD_NEWLINE, ...): assume the position is outside quotes, run to the next record separator, and check with IsRowValid() that exactly one row parses cleanly.
• If a quote character is configured, TryRow(CSVState::QUOTED, ...): assume the position is inside quotes and do the same.
• If neither worked, TryRow(CSVState::ESCAPE, ...): assume the position is inside an escaped value and do the same.

For this check it builds a separate strict-mode state machine (state_machine_strict), because lenient parsing accepts too many false line starts.

Because the line start is a guess, it can be wrong. When reading in parallel with neither ignore_errors nor store_rejects set, a validator runs at the end:

// src/execution/operator/csv_scanner/table_function/global_csv_state.cpp
// If we are running multithreaded and not ignoring errors, we must run the validator
scan.validator.Verify();

Every scanner registers the first line it read, and the validator checks that it lines up with the previous scanner's last line, failing the query if it does not. The reader guesses fast and verifies afterwards, like the selectivity-optional filters in §8.4.

Parallel reading has a limit: with null_padding enabled, a quoted newline makes the line start impossible to determine in parallel, so DuckDB raises CSVError::NullPaddingFail.

11. Decomposing the Speed

DuckDB's speed comes from decisions at every layer.

These decisions share each other's assumptions. STANDARD_VECTOR_SIZE was chosen for the execution engine, but DEFAULT_ROW_GROUP_SIZE must be a multiple of it, MVCC's ChunkVectorInfo is cut at the same granularity, and compression analysis runs in the same unit.

Many of the decisions also give something up. ToUnifiedFormat() flattening FSST trades speed for a simpler abstraction; cutting off join-order search with a budget gives up optimality. DuckDB aims to be fast in almost every case rather than optimal in all of them.

12. A Map for Reading the Source

A practical way into the code is to look at the internal state through the duckdb_ table functions first, then read the corresponding code. If you open only one file, open src/include/duckdb/common/types/vector.hpp; almost everything in this article branches off from there.

-- plan plus measured timings
EXPLAIN ANALYZE SELECT ...;
 
-- compression scheme and statistics per column segment
SELECT * FROM pragma_storage_info('my_table');
 
-- memory usage broken down by MemoryTag
SELECT * FROM duckdb_memory();
 
-- each buffer eviction queue with its tier (queue_type) and size
SELECT * FROM duckdb_eviction_queues();
 
-- every valid optimizer pass name
SELECT * FROM duckdb_optimizers();
 
-- extensions, with whether each is installed and loaded
SELECT * FROM duckdb_extensions();

When pragma_storage_info shows RLE and FSST in the compression column, you are seeing the result of the per-segment compression contest from §5.4.

The table below lists where each topic lives in the source.

(For sorting, the implementation lives in src/common/sort/ while the headers live in src/include/duckdb/common/sorting/; the directory names do not match.)

References

• Peter Boncz, Marcin Zukowski, Niels Nes. "MonetDB/X100: Hyper-Pipelining Query Execution." CIDR 2005.
• Viktor Leis, Peter Boncz, Alfons Kemper, Thomas Neumann. "Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Framework for the Many-Core Age." SIGMOD 2014.
• Viktor Leis, Alfons Kemper, Thomas Neumann. "The Adaptive Radix Tree: ARTful Indexing for Main-Memory Databases." ICDE 2013.
• Thomas Neumann, Tobias Mühlbauer, Alfons Kemper. "Fast Serializable Multi-Version Concurrency Control for Main-Memory Database Systems." SIGMOD 2015.
• Guido Moerkotte, Thomas Neumann. "Analysis of Two Existing and One New Dynamic Programming Algorithm for the Generation of Optimal Bushy Join Trees without Cross Products." VLDB 2006 (DPccp).
• Guido Moerkotte, Thomas Neumann. "Dynamic Programming Strikes Back." SIGMOD 2008 (DPhyp).
• Peter Boncz, Thomas Neumann, Viktor Leis. "FSST: Fast Random Access String Compression." VLDB 2020.
• Azim Afroozeh, Leonardo X. Kuffo, Peter Boncz. "ALP: Adaptive Lossless floating-Point Compression." Proc. ACM Manag. Data 1(4), Article 230, 2023 (presented at SIGMOD 2024).
• Mark Raasveldt, Hannes Mühleisen. "DuckDB: an Embeddable Analytical Database." SIGMOD 2019 (Demo).
• Mark Raasveldt, Hannes Mühleisen. "Data Management for Data Science — Towards Embedded Analytics." CIDR 2020.
• duckdb/duckdb — all code quoted here is from the main branch as of writing (August 2026, the v1.5 series). Some of the quoted code, such as the Node256Leaf mask declaration, changed in v2.0.0 (October 2026).