Skip to content

Keep header id pointers valid when adding lines grows a dictionary - #2146

Open
tfenne wants to merge 1 commit into
samtools:developfrom
tfenne:tf/hdr-id-resync
Open

tfenne wants to merge 1 commit into
samtools:developfrom
tfenne:tf/hdr-id-resync

Conversation

@tfenne

@tfenne tfenne commented Nov 13, 2026

Copy link
Copy Markdown
Member

bcf_hdr_t.id[type][i].val points into the vdict hash tables. When bcf_hdr_add_hrec() puts a new key into a dictionary that has reached its upper bound, khash rehashes it and moves every entry, leaving the id pointers dangling until the next bcf_hdr_sync().

Callers commonly add header lines, e.g. with bcf_hdr_printf(), and then carry on using the header without syncing it. bcftools norm --old-rec-tag does this, then reads INFO fields through the stale pointers. Since 1.19 bcf_hdr_init() pre-sizes the dictionaries to 16384 buckets, which hides the problem until a header has about 12,600 tags or contigs, but smaller dictionaries expose it in bcftools' own tests under AddressSanitizer.

Re-point the affected dictionary's id entries whenever adding a header line rehashes it. Rehashes are rare (the table doubles each time), so this adds O(1) amortised work per line.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Tim Fennell tfenne@tfenne.com

bcf_hdr_t.id[type][i].val points into the vdict hash tables.  When
bcf_hdr_add_hrec() puts a new key into a dictionary that has reached its
upper bound, khash rehashes it and moves every entry, leaving the id
pointers dangling until the next bcf_hdr_sync().

Callers commonly add header lines, e.g. with bcf_hdr_printf(), and then
carry on using the header without syncing it.  `bcftools norm
--old-rec-tag` does this, then reads INFO fields through the stale
pointers.  Since 1.19 bcf_hdr_init() pre-sizes the dictionaries to 16384
buckets, which hides the problem until a header has about 12,600 tags or
contigs, but smaller dictionaries expose it in bcftools' own tests under
AddressSanitizer.

Re-point the affected dictionary's id entries whenever adding a header
line rehashes it.  Rehashes are rare (the table doubles each time), so
this adds O(1) amortised work per line.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Tim Fennell <tfenne@tfenne.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant