Repository navigation
Conversation
bcf_hdr_t.id[type][i].val points into the vdict hash tables. When bcf_hdr_add_hrec() puts a new key into a dictionary that has reached its upper bound, khash rehashes it and moves every entry, leaving the id pointers dangling until the next bcf_hdr_sync(). Callers commonly add header lines, e.g. with bcf_hdr_printf(), and then carry on using the header without syncing it. `bcftools norm --old-rec-tag` does this, then reads INFO fields through the stale pointers. Since 1.19 bcf_hdr_init() pre-sizes the dictionaries to 16384 buckets, which hides the problem until a header has about 12,600 tags or contigs, but smaller dictionaries expose it in bcftools' own tests under AddressSanitizer. Re-point the affected dictionary's id entries whenever adding a header line rehashes it. Rehashes are rare (the table doubles each time), so this adds O(1) amortised work per line. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Tim Fennell <tfenne@tfenne.com>
Since 1.19 bcf_hdr_init() has pre-sized the three header dictionaries to 16384, 16384 and 2048 buckets: about 2.2 MB per header before any line is parsed. That made VCF parsing faster, but a typical header has tens of tags, a few hundred contigs and one sample, so most of the space is never used. Programs holding many headers at once pay for it, e.g. bcftools merge over many inputs, or joint callers with thousands of open readers. Nearly all of the speed comes from the ID dictionary (FILTER, INFO and FORMAT keys), which is searched for every key of every VCF record. The changes here keep that one at most 1/32 full, capped at the previous 16384 buckets, and size it in bcf_hdr_sync() so that parsed, duplicated, merged and programmatically built headers all get it. Contigs and samples are looked up far less often, and leaving them to khash's normal growth has no measurable cost that I can see. Heap per parsed header (macOS): 15 KB header, 27 tags 2.27 MB -> 0.10 MB CanFam4 gVCF header, 2,199 contigs 3.05 MB -> 1.15 MB GIAB HG002 v4.2.1 header 2.32 MB -> 0.18 MB gnomAD v4.1 sites header, 244 tags 2.36 MB -> 0.65 MB Read speed is close to within run-to-run noise (median of 15 paired rounds, Apple M4 Max): VCF gnomAD v4.1 sites +0.9%, gnomAD HGDP+1KG (4,151 samples) +0.1%, GIAB HG002 +0.5%; BCF with tag lookups by name +0.3%. Removing the pre-size entirely instead costs 5-8% on the same inputs, with either the X31 or the FNV-1a string hash. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Tim Fennell <tfenne@tfenne.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is an attempt to keep the vast majority of #1644's VCF parsing speed-up while reducing the additional memory used, thereby freeing up to ~2.2 MB from every parsed header.
Since #1644 (1.19),
bcf_hdr_init()pre-sizes the ID, contig and sample dictionaries to 16384/16384/2048 buckets: about 2.2 MB before any header line is read. That bought a real VCF parsing speed-up. But frequently most of the space is unused, because typical headers have tens of tags, hundreds of contigs and a single sample (or no samples!).This matters for programs that hold many headers at once. A joint caller reading thousands of single-sample gVCF/BCF files holds a header per input. At 10,000 inputs that is 20–40 GB of mostly empty tables by heap, of which 5–24 GB is actually resident.
bcftools mergeover many inputs pays the same cost per input.Nearly all of the speed comes from the ID dictionary (FILTER, INFO and FORMAT), which is searched for every key of every VCF record. The contig dictionary holds most of the memory but is looked up only once per record. So this PR:
bcf_hdr_sync(), capped at the previous 16384 buckets, so it is never larger than today. Parsed, duplicated, merged and programmatically built headers all get this.This builds on #2146, because smaller dictionaries rehash more often when header lines are added. #2146's commit is the first of the two here, so please review only the second. I'll rebase if/when #2146 is merged.
Memory per parsed header (heap on macOS; resident memory falls by less, 0.5–1.2 MB per header, because untouched pages are never resident):
bcftools mergeof 100 GRCh38 single-sample gBCFs: peak RSS falls from 321 to 221 MB, and the output is identical.Read speed (median of 15 interleaved paired rounds, separate builds, Apple M4 Max, clang 21
-O2):That is under 1% against the 23–53% from #1644. Small headers parse 39% faster. A 2,199-contig header parses 7% (≈75 µs) slower, because the contig table grows instead of starting large.
How the sizing was chosen:
I only measured on Apple silicon. Given the CPU sensitivity noted in vcf.c, if you're otherwise approving of this PR and would like benchmarks on Intel and/or AMD I'm happy to do so.
Suggested NEWS entry: Reduced the memory used by each VCF/BCF header by up to ~2 MB, by sizing the header dictionaries from their contents rather than pre-allocating them. This helps programs that hold many headers open at once.
Assisted-by: Claude:claude-opus-5-5