Skip to content

Size VCF header dictionaries by their contents - #2147

Open
tfenne wants to merge 2 commits into
samtools:developfrom
tfenne:tf/shrink-hdr-dicts
Open

tfenne wants to merge 2 commits into
samtools:developfrom
tfenne:tf/shrink-hdr-dicts

Conversation

@tfenne

@tfenne tfenne commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

This PR is an attempt to keep the vast majority of #1644's VCF parsing speed-up while reducing the additional memory used, thereby freeing up to ~2.2 MB from every parsed header.

Since #1644 (1.19), bcf_hdr_init() pre-sizes the ID, contig and sample dictionaries to 16384/16384/2048 buckets: about 2.2 MB before any header line is read. That bought a real VCF parsing speed-up. But frequently most of the space is unused, because typical headers have tens of tags, hundreds of contigs and a single sample (or no samples!).

This matters for programs that hold many headers at once. A joint caller reading thousands of single-sample gVCF/BCF files holds a header per input. At 10,000 inputs that is 20–40 GB of mostly empty tables by heap, of which 5–24 GB is actually resident. bcftools merge over many inputs pays the same cost per input.

Nearly all of the speed comes from the ID dictionary (FILTER, INFO and FORMAT), which is searched for every key of every VCF record. The contig dictionary holds most of the memory but is looked up only once per record. So this PR:

  • keeps the ID dictionary at most 1/32 full in bcf_hdr_sync(), capped at the previous 16384 buckets, so it is never larger than today. Parsed, duplicated, merged and programmatically built headers all get this.
  • leaves the contig and sample dictionaries to khash's normal growth.

This builds on #2146, because smaller dictionaries rehash more often when header lines are added. #2146's commit is the first of the two here, so please review only the second. I'll rebase if/when #2146 is merged.

Memory per parsed header (heap on macOS; resident memory falls by less, 0.5–1.2 MB per header, because untouched pages are never resident):

Header Before After
15 KB header, 27 tags 2.27 MB 0.10 MB
CanFam4 gVCF, 2,199 contigs 3.05 MB 1.15 MB
GIAB HG002 v4.2.1 2.32 MB 0.18 MB
gnomAD v4.1 sites, 244 tags 2.36 MB 0.65 MB
100,000 contigs 36.5 MB 35.3 MB

bcftools merge of 100 GRCh38 single-sample gBCFs: peak RSS falls from 321 to 221 MB, and the output is identical.

Read speed (median of 15 interleaved paired rounds, separate builds, Apple M4 Max, clang 21 -O2):

Input Change
VCF, gnomAD v4.1 sites (INFO-heavy) +0.9%
VCF, gnomAD HGDP+1KG (4,151 samples) +0.1%
VCF, GIAB HG002 +0.5%
BCF, GIAB with 6 tag lookups by name per record +0.3%
BCF, GIAB plain (no dictionary lookups; the noise floor for separate builds) +0.3%

That is under 1% against the 23–53% from #1644. Small headers parse 39% faster. A 2,199-contig header parses 7% (≈75 µs) slower, because the contig table grows instead of starting large.

How the sizing was chosen:

  • No pre-size at all costs 5–8% on these inputs, under either X31 or FNV-1a. So FNV-1a (Use FNV1a for string hashing #1806) didn't make the pre-size redundant: table load still matters.
  • Contigs can grow naturally for free. Keeping develop's ID dictionary but letting contigs grow naturally costs nothing measurable.
  • The ID dictionary needs to stay sparse. At 1/4 load it costs up to 4%. At 1/8 to 1/32 it is within about 1% of develop. 1/32 costs at most 64 KB for typical headers.
  • The remaining ~1% on single-sample VCF can be cut to +0.1–0.5% by giving the ID dictionary a 4,096-bucket floor, at about 190 KB more per header.

I only measured on Apple silicon. Given the CPU sensitivity noted in vcf.c, if you're otherwise approving of this PR and would like benchmarks on Intel and/or AMD I'm happy to do so.

Suggested NEWS entry: Reduced the memory used by each VCF/BCF header by up to ~2 MB, by sizing the header dictionaries from their contents rather than pre-allocating them. This helps programs that hold many headers open at once.

Assisted-by: Claude:claude-opus-5-5

bcf_hdr_t.id[type][i].val points into the vdict hash tables.  When
bcf_hdr_add_hrec() puts a new key into a dictionary that has reached its
upper bound, khash rehashes it and moves every entry, leaving the id
pointers dangling until the next bcf_hdr_sync().

Callers commonly add header lines, e.g. with bcf_hdr_printf(), and then
carry on using the header without syncing it.  `bcftools norm
--old-rec-tag` does this, then reads INFO fields through the stale
pointers.  Since 1.19 bcf_hdr_init() pre-sizes the dictionaries to 16384
buckets, which hides the problem until a header has about 12,600 tags or
contigs, but smaller dictionaries expose it in bcftools' own tests under
AddressSanitizer.

Re-point the affected dictionary's id entries whenever adding a header
line rehashes it.  Rehashes are rare (the table doubles each time), so
this adds O(1) amortised work per line.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Tim Fennell <tfenne@tfenne.com>
Since 1.19 bcf_hdr_init() has pre-sized the three header dictionaries to
16384, 16384 and 2048 buckets: about 2.2 MB per header before any line
is parsed.  That made VCF parsing faster, but a typical header has tens
of tags, a few hundred contigs and one sample, so most of the space is
never used.  Programs holding many headers at once pay for it, e.g.
bcftools merge over many inputs, or joint callers with thousands of open
readers.

Nearly all of the speed comes from the ID dictionary (FILTER, INFO and
FORMAT keys), which is searched for every key of every VCF record.  The
changes here keep that one at most 1/32 full, capped at the previous
16384 buckets, and size it in bcf_hdr_sync() so that parsed, duplicated,
merged and programmatically built headers all get it.  Contigs and samples
are looked up far less often, and leaving them to khash's normal growth has
no measurable cost that I can see.

Heap per parsed header (macOS):
  15 KB header, 27 tags                 2.27 MB -> 0.10 MB
  CanFam4 gVCF header, 2,199 contigs        3.05 MB -> 1.15 MB
  GIAB HG002 v4.2.1 header              2.32 MB -> 0.18 MB
  gnomAD v4.1 sites header, 244 tags    2.36 MB -> 0.65 MB

Read speed is close to within run-to-run noise (median of 15 paired
rounds, Apple M4 Max): VCF gnomAD v4.1 sites +0.9%, gnomAD HGDP+1KG
(4,151 samples) +0.1%, GIAB HG002 +0.5%; BCF with tag lookups by name
+0.3%.  Removing the pre-size entirely instead costs 5-8% on the same
inputs, with either the X31 or the FNV-1a string hash.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Tim Fennell <tfenne@tfenne.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant