Skip to content

call: let groups of a site's walks choose its genotype under --read-likelihood, and refuse --top-down there - #5083

Open
benedictpaten wants to merge 19 commits into
vgteam:masterfrom
benedictpaten:group-candidates
Open

benedictpaten wants to merge 19 commits into
vgteam:masterfrom
benedictpaten:group-candidates

Conversation

@benedictpaten

@benedictpaten benedictpaten commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor
Commit
refuse --top-down and -Y with --read-likelihood The read-likelihood caller's top-down calling is deleted. Under --read-likelihood it wrote nested sites more than once (155,958 records for 128,733 sites on chr20 short reads), wrote 107 alleles reversed, and could crash on copies sharing a record key. The support-based --top-down and -Y are unchanged.
add --group-candidates Groups of a site's walks choose its genotype, and the walks choose within the chosen groups.
turn --group-candidates on by default --no-group-candidates turns it off, which at this commit gives output byte-identical to the base.
keep two TAP tests of grouped walks Two t/18 tests on reads where grouping changes the call.
test a child chain that one strand of a grouped parent crosses A unit test of nested calling under a grouped parent.
re-genotype each read-linked run in its own frame Re-genotyping chooses a heterozygote's phase from the reads of one run of read-linked sites, and averages over the orientation of each other run's reads.
weigh the panel's phase by its fitted error rates The panel's vote on the phase of two sites falls with the distance between them, by switch and flip rates fitted to the reads. Re-genotyping weighs another run's reads by how surely their run was oriented.
count each read once in a hung site's vote A read that spans several of the phase-chain sites voting on a hung site votes once for each run of them, not once for each site.
document mapping without the reference The docs say how to map reads to a copy of the graph without the reference paths, made with vg gbwt -R, so that the reference does not draw reads away from the sampled haplotypes, and how to call on the graph that has them.
read the grouped-walk test records with cut A t/18 fix from review.
anchor file version 10, with a kind column and ranked pin sources Each A row of the --anchors-out file ends with a kind column. Where two anchors would hold one read point, an anchor of a site of the graph keeps it first, then one of an off-panel site, then a fill anchor.
fill anchors at evenly spaced points inside unbranched node runs --anchors-fill N, 1000 by default and used only with --anchors-out, adds anchors at evenly spaced points about N bp apart inside each run of nodes with no branch that is longer than N bp.
document fill anchors, and test them on the command line The Fill anchors part of the docs, and t/18 tests that fill anchors leave the VCF unchanged, that no read point is in two anchors, that every A row ends with its kind, and that the anchor file does not depend on the thread count.
fill anchors take their slot from the innermost genotyped site A fill anchor's slot and kind come from the called walks of the innermost genotyped site that contains its point, and a read is placed at a fill point only where its alignment crosses the point.
tally the fill anchors' fetches apart, and test fill anchors further The fill anchors' read fetches are logged apart from the site queries'. More t/18 tests.
say why fill anchors in repeats hold more misplaced reads than site anchors Docs.
build each site's anchors over the alleles its line wrote The render writes a site's line before it collects the site's anchors, which then use the VCF alleles the line wrote: two called traversals that spell one sequence share one allele, so the anchors' kind and gqn agree with the line's GT and GQN.
place the fill anchors' reads from the windows the direct pass fetches The read source passes each window of reads it fetches whole during the direct pass to the fill anchors, which place the reads crossing that window's points while it is in memory. Only the windows the pass never fetched whole are fetched afterwards.

What grouping does

Walks of a site are in the same group when they differ only inside the site's child chains: outside them they take the same steps, and they cross the same child chains in the same order and direction. Nested calling already calls what each copy does inside a child chain, so the parent only needs to choose groups. Without grouping, a copy that is a mosaic (one walk in one child, another walk in the next) fits no walk, and its reads split among the walks that spell parts of it.

For a genotype G whose groups form the multiset X:

own(G)     = Σ_r ln[ (1 − e_r) Σ_i π_i rel(r, g_i) + e_r ] + D(G)              the usual likelihood
shared(G)  = the same, with rel(r, g_i) replaced by the best rel(r, w) over the walks w in g_i's group
Δ(X)       = shared_R(X) − own_R(X)                                             the gain, without the depth term D
c(X)       = max(0, max over groups γ of X with two or more walks of [Δ(X) − Δ(X ⊕ γ)])
grouped(G) = shared(X) − c(X) − [own(X) − own(G)]

own(X) and shared(X) are the best scores over the genotypes with groups X, and X ⊕ γ is X with one more copy of γ, at ploidy + 1. The charge c stops one copy of a group from absorbing the reads of two haplotypes: one more copy takes those reads over, so the gain drops. grouped(G) feeds GL, GQ, GP, QUAL, the linkage pass and re-genotyping. The full description is in artifacts/doc/read-likelihood-direct-genotyping.md.

What the read-phasing commits do

Read phasing (--read-phasing) and re-genotyping (--regenotype) are off by default and on under --preset ont and --preset hifi. These three commits change output only when they are on. In vg call, a strand is one of the sample's haplotypes at a site, not a strand of DNA.

vg call first phases each heterozygous site from the graph's haplotypes, the panel. Read phasing then decides the phases from the reads. The sites whose alleles the reads tell apart reliably form the phase chain, and a read that spans two of its sites links them. A run is a stretch of the chain within which the reads fix every site's phase relative to the others. A new run starts at each break in the chain that no read links across, or where the panel's vote outweighed the reads'.

A heterozygous site off the chain is hung: the reads it shares with its nearest chain sites vote on its phase, and so does the panel. Re-genotyping then scores every site's genotypes again. Each read leans towards the strand that its alleles at the phase set's other heterozygous sites place it on, so a heterozygous genotype that agrees with the phase gains likelihood.

  • Runs in re-genotyping. Reads fix the strands of one run's sites relative to each other, but not to another run's. So a site's reads are divided by the run their strands come from. The frame is the site's own run if any of the reads' strands come from it, and otherwise the run most of them come from. Each heterozygous genotype is scored under both assignments of its alleles to the frame's strands, and the better is kept. Within one assignment, the reads of each other run add ln(p e^F + (1 − p) e^R). F is their score with their strands as read phasing set them, R with the strands swapped, and p the probability that read phasing oriented their run correctly relative to the frame's. The heterozygous slot in the --anchors-out file uses the same weights. Scored in one frame, the reads of a run that the panel had oriented the wrong way could outvote every heterozygous assignment, so a true heterozygote could lose to a homozygote.
  • The panel's errors. The panel is wrong at switches, which fall at a rate λ per base, and at single flipped sites, with probability f each. It phases two sites d bases apart correctly with probability p(d) = (1 + (1 − 2f)² e^(−2λd))/2. Its vote is log10(p / (1 − p)), in the log10-odds units of a read's link. Each round of read phasing fits λ and f by maximum likelihood. The fit uses consecutive panel-phased sites inside stretches of the chain where reads alone fix the phase, pooled over all phase sets. The vote is added to the reads' at each break, between the panel-phased sites nearest it, and at each hung site the panel phases. With too few pairs to fit, reads alone decide the breaks, and hung sites weigh the panel by --phase-prior. Across runs, p = (1 + Π c)/2, over the correlations c = 2q − 1 of the breaks between them. Here q = 1/(1 + 10^(−|S|)) is the probability that the break's summed vote S decided it correctly, and c is 0 at a break nothing decided.
  • One vote per read at a hung site. A hung site is voted on by its nearest chain sites, up to --phase-hang/2 + 1 on each side. For each run among them, a read's alleles at that run's sites give the likelihood of each of its strands, and its allele at the hung site then favours keeping or reversing the hung site's phase. The read votes once for each run, however many of the run's sites it spans, and a run's vote counts as a link when it is not 0. Adding the read's link to each voting site instead would count its allele at the hung site once per site: an ONT read spanning six would count it six times and outweigh the panel.

The full description is in artifacts/doc/read-likelihood-read-phasing.md.

What the fill-anchor commits do

With --anchors-out, vg call writes the reads at points of the graph, so that an assembler can align the reads to each other there. Each point is an anchor. The file has one A row for each anchor and slot, followed by one R row for each read placed there, which gives where in the read the point falls. A slot is a strand of the site's genotype: slot 0 holds the reads of the strand of the first allele in GT, and slot 1 those of the second. A homozygous site puts all its reads in slot 0.

Before these commits, anchors lay only at the ends of sites, where each site meets the sequence around it. Sequence that no site varies, and the inside of a site with long alleles, could lie far from every anchor. In a graph that keeps the centromere, the alpha-satellite arrays lie inside one site several megabases long, and each haplotype's array is a run of nodes with no branch.

A fill anchor lies at a point chosen from the graph alone. A unitig is a longest run of nodes joined end to end with no branch. Inside each unitig longer than N bp, --anchors-fill N places ceil(L/N) − 1 evenly spaced points, where L is the unitig's length, counted from the end whose node has the smaller ID, so every run on one graph uses the same points.

During the direct pass, the first pass over the sites and the only one that fetches reads, the read source passes each window of reads it fetches whole to the fill anchors, which place the reads crossing that window's points. The windows the pass never fetched whole are fetched after it. A read is placed at a point where its alignment crosses the point. Placing reads records nothing that genotyping uses, so the VCF, the --mosaic-out file and the reads at each site's anchors do not change. Only the R rows' read IDs shift, because the file's table of read names gains the fill anchors' reads.

  • Slot. A fill anchor takes its slot from its deciding site: the innermost site that contains its point and has a called genotype, passing over a nested chain whose strand is not known. Where only strand s of that site's call crosses the point, the anchor is fill-strand, in slot s. Where both strands cross it, where the site is in a region of ploidy 1, or where no site contains the point, it is fill-shared, with every read in slot 0. Where no called walk crosses it, or no site containing it has a called genotype, it is fill-off-call, also in slot 0.
  • One anchor per read point. A read point is the place in one read where it crosses a point of the graph. Where two anchors would hold one read point, an anchor of a site of the graph keeps it first, then one of an off-panel site (a site vg call adds where the reads show a change that no panel haplotype carries), then a fill anchor. A fill anchor left with fewer than --anchors-reads reads is not written.
  • Format. The anchor file becomes version 10. Each A row ends with a kind column: het, hom, hom-split or haploid for the anchors of sites, and fill-shared, fill-strand or fill-off-call for fill anchors. A fill anchor writes . for the site's fields (allele, gqn, explained and reliability), and #filters records the spacing.

The definitions are in the Assembly anchors section of artifacts/doc/read-likelihood-genotyping.md.

Results

HG002 chr20, chr6 and chr17, short reads and ONT, against the T2T Q100 v1.1 truth. All runs used --anchors-out.

Grouping. Each cell compares grouping on with grouping off, both before the read-phasing commits.

Cell Net small-variant errors Refined SV F1 Switches / flips
chr20 short −3 0.7054 → 0.7144 1316/297 → 1316/295
chr6 short −2 0.7864 → 0.7932 2634/551 → 2639/550
chr17 short −21 0.6791 → 0.6933 1604/380 → 1592/382
chr20 ONT −9 0.8317 → 0.8318 53/187 → 59/187
chr6 ONT +6 0.8876 → 0.8873 23/314 → 23/315
chr17 ONT −18 0.7998 → 0.7967 80/201 → 76/202
  • Pooled over the three short-read contigs, refined SV F1 rises +0.0094 (block bootstrap 95% CI +0.0024 to +0.0171). ONT is flat (−0.0009, CI −0.0047 to +0.0018).
  • On two chr20 centromere test graphs (ONT), the array call goes from two paternal walks to the right maternal/paternal pair. The share of the maternal array's 31-mers found in the call goes from 0.249 to 0.909, and from 0.250 to 0.951.
  • Wall time is within 2% of --no-group-candidates.

Read phasing. Each cell compares "test a child chain that one strand of a grouped parent crosses" (the last grouping commit) with the head. Short-read cells ran with --read-phasing --regenotype --mismap-max 0.95, and ONT cells with --preset ont. Phase error is whatshap's switch-error rate: switches plus twice the flips, over the pairs of sites it assesses.

Cell Small-variant errors (FP + FN) Refined SV F1 Switches / flips Phase error
chr20 short 3,445 → 3,456 0.7144 → 0.7205 1312/299 → 1060/204 3.12% → 2.40%
chr6 short 8,410 → 8,406 0.7932 → 0.7971 2641/549 → 2119/356 2.23% → 1.69%
chr20 ONT 3,896 → 3,898 0.8318 → 0.8318 59/187 → 55/137 0.69% → 0.53%
chr6 ONT 9,173 → 9,168 0.8873 → 0.8973 23/315 → 21/216 0.38% → 0.27%
  • The grouping table's short-read runs held --regeno-temper at the value fitted without grouping, and these runs fit it, so their "before" switches and flips differ slightly (chr20 1316/295, chr6 2639/550).
  • SV F1 before refinement moves by −0.0009 to +0.0011. truvari refine compares the phased haplotypes region by region, so better phasing alone raises refined SV F1.
  • We also called six chr20 test graphs that keep the centromere's satellite array, which the graph above clips out. Each holds CHM13, GRCh38 and 32 haplotypes that vg haplotypes sampled from HG002's short-read k-mers. In them the array is one site several megabases long, whose reads can take their strands from several runs, because no read links across some stretches of the array.
    • On four of the six, the ONT call of the array was homozygous (1|1) and is now heterozygous. The share of the maternal array's 31-mers in the call rises from 0.27–0.28 to 0.91.
    • On three of these four, the old call was one 6.4 Mb record that hid the nested records beneath it, including truth regions beside the array. F1 over all small variants rises there by 0.0076–0.0078 (0.9718 → 0.9795 on one). On the fourth, the old call was split into smaller records, and F1 is unchanged.
    • With ONT, phase error falls by 31–34% on the six, and by 24% on the graph above with its reads mapped afresh. With short reads, it falls by 21–25% on three of them and on the graph above.
  • On the four cells of the table, CPU time changes by −1% to +4%, and peak memory by at most +0.6 GB, from peaks of 10.6 to 26.4 GB. In two back-to-back runs of each build on one ONT test graph, CPU time rises by 2.3% (664 and 666 s → 681 and 678 s), all of it from "weigh the panel's phase by its fitted error rates", and peak memory does not change (21.5 and 21.1 GB → 21.6 and 21.3 GB).

Fill anchors. Fill anchors were measured on two chr20 graphs that keep the centromere, sampled from HG002's short-read k-mers by vg haplotypes --preset model (#5112), one with 32 haplotypes and one with 2. ONT reads were mapped to a copy of each graph without the CHM13 paths, as the new docs describe, and called on the graph itself with --preset ont. Each "→" compares the anchors without fill anchors with those with them, at the default spacing.

  • At spacings of 0, 500, 1000 and 2000 bp, the VCF records, the --mosaic-out file and the reads at every site's anchors are the same as before these commits. Each site's anchor was compared as its A row without the kind column, and its R rows with read names in place of read IDs; a planted change shows the comparison can fail. The kind column agrees with the genotype vg call held for each site when it wrote the anchors.

  • On the 2-haplotype graph, with reads of MAPQ 5 or more:

    Region Read bases > 1 kb from an anchor Reads with no anchor
    alpha-satellite arrays 53.9% → 3.4% 811 → 7
    rest of the centromere (CHM13 25.82–33.06 Mb) 14.0% → 1.6% 185 → 8
    rest of chr20 22.5% → 1.1% 4,834 → 253

    On the 32-haplotype graph, the centromere's figures are within a quarter of a percentage point and 13 reads of these, and the share elsewhere goes from 13.1% to 1.1%.

  • Fill anchors add 17% to the R rows on the 2-haplotype graph and 10% on the 32-haplotype graph. On the 2-haplotype graph, wall time rises by 7 s, from 105 s, and CPU time by 6%; peak memory does not change.

  • A spacing of 500 bp adds 45% and 30% to the R rows. It moves the share of read bases more than 1 kb from an anchor by under half a point, and cuts the reads outside the centromere with no anchor from 253 to 75 (250 to 80 on the 32-haplotype graph). A spacing of 2000 bp raises those reads to 1,167 (954).

  • Linking the anchors into runs through the reads they share, and orienting each anchor by the parent of its reads, gives the same breaks between runs, switches and flips with fill anchors as without them: 3 switches and 8 flips on the 2-haplotype graph, and 4 and 11 on the 32-haplotype graph. A read's parent comes from its alleles at heterozygous SNVs that match the truth, so only about 1,040 of the 8,760 fill-strand anchors can be oriented: most lie in the arrays, where reads cross too few such SNVs.

  • We aligned the reads to HG002's maternal and paternal truth assemblies and judged each placement whose read aligns there with MAPQ 20 or more. Outside the centromere and the last 110 kb of chr20, fill anchors place 99.99% of these reads correctly, against 99.2% for site anchors. In the rest of the centromere outside the arrays, 95% against 99%. Within the same 10 kb stretches of the arrays, 95% against 96% on the paternal array, and 75% against 83% on the maternal one. The misplaced reads come from other copies of the repeat. A site's anchor leaves some of them out, because their best allele is not one the site was called with, but a fill anchor keeps every read that crosses it. In the last 110 kb of chr20, reads from paralogs on chr6 and chr9 map with MAPQ 60, and fill and site anchors alike are wrong there about 60% of the time.

  • On chr20 short reads, with --read-phasing --regenotype --mismap-max 0.95, the VCF records and the --mosaic-out file are the same as before these commits with --anchors-out and without it, and with --anchors-fill 0. So are the reads at every site's anchors. Fill anchors add 5.6% to the R rows there, about 1% to the wall time and 2% to the CPU time.

Known limitations

  • The charge can drop a true het deletion beside a mosaic. The read-by-walk matrix cannot tell "one copy is a mosaic and the other deletes the chain" from "two copies, with a deletion falsely called"; only depth and the deletion's junction reads differ. On fixtures with known truth this happens at MAPQ 5–7 (13 of 468 sweep fixtures). A charge that weighs depth and junction reads is the next step.
  • The ploidy + 1 scores come from a short list of genotypes, which can undercharge: it leaves a false 1,982 bp het deletion at chr6:70.745 Mb.
  • The explained-share discount of GQ and GQN counts walks, not groups, so a grouped mosaic call gets lower GQ and GQN.
  • Reads at hung sites are overconfident. On chr20 ONT the panel's vote at a hung site is at most about 2.6, so reads whose summed vote passes it decide the site, yet against the truth the reads beat the panel only above a vote of about 4. Hung sites hold about 70% of the remaining flips. Scaling the reads' vote by about 0.35 would cut the truth-assessable errors at hung sites from 103 to about 64. Fitted against the panel instead of the truth, the scale comes out at 0.12, because the panel is wrong more often at hung sites than elsewhere, so it is not applied.
  • Short-read indels lose a little: indel F1 falls by 0.0005 and 0.0004 on two of the centromere test graphs. On chr20 short reads, small-variant errors rise by 11 of 3,445.
  • A fill-strand slot names a strand only through its deciding site, which the A row does not name. A reader must find that site in the snarl tree, it may have no VCF record, and the slot names a haplotype only if the site is phased. Where the anchor's reads also cross heterozygous sites, their slots there can tie it to a haplotype, but in a long unitig many of its reads cross none.
  • Fill anchors in repeats hold more misplaced reads than site anchors, as the results show. A graph or haplotype sample with fewer wrong copies of the repeat would reduce this.

Not measured

The default configuration without --anchors-out on real data; --preset hifi and --enumerate-support with grouping; --no-nested, --no-atomize-blocks, -a and -A on real data; haploid regions on real data; the read-phasing commits on chr17, or with --preset hifi, which turns them on; fill anchors on contigs other than chr20, on graphs that clip the centromere, or with HiFi reads.

Tests

At the head, on macOS:

  • unit tests pass (1,097 cases); make lint and .claude/skills/intelligible-comments/lint_prose.py --base f8957177e are clean; each commit builds;
  • test/t/18_vg_call.t passes; artifacts/test/vg_call_read_likelihood.t passes 302 of 304, and tests 135–136, which need the external gaf-base tool, fail the same way at f895717 on this machine;
  • at the last grouping commit, on chr20 short reads and ONT, --no-group-candidates writes the same VCF records, --mosaic-out file and --anchors-out file as 0ce5825, call: add opt-in read-level genotype likelihoods (--read-likelihood) #4990's head before its last 47 commits, and the default writes the same as this series did before it was rebased onto f895717;
  • merging master (4d48b28) changes no VCF record, and no line of the --anchors-out or --mosaic-out files, in the chr20 short-read and ONT calls.
  • with fill anchors on and off, chr20 short-read and ONT calls at the head write the same VCF records and --mosaic-out file as at 298ecfe, and every site's anchors have a kind and gqn that agree with its line's GT and GQN;
  • "place the fill anchors' reads from the windows the direct pass fetches" changes no VCF record and no line of the --anchors-out or --mosaic-out files in those calls.

🤖 Generated with Claude Code

benedictpaten and others added 5 commits October 29, 2026 21:50
--top-down and -Y/--star-allele are options of the support-based caller's nested mode, which keeps
them unchanged. With --read-likelihood they are now refused, and the read-likelihood caller's
top-down calling is removed: it kept nested calling on as well, so on chr20 short reads each nested
site was genotyped once per level and written several times (155,958 records for 128,733 sites); it
took a child's walks in the orientation of the site overlay while the candidate finder turns the
child to run along the reference, so 107 sites wrote a reversed allele; and copies of a site that
share a record key could crash the caller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…choose its genotype

Two walks of a site share a group when their symbolic alleles over every child chain of the site
are equal: they take one route through the site and differ only inside child chains. Each multiset
of groups scores its best genotype with every read's relative likelihood for an allele replaced by
its best over the allele's group, less a charge where one more copy of a group takes over reads
that one copy carried for two haplotypes. The walks then choose the genotype within the chosen
groups. The option is off by default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndidates

Under --read-likelihood, groups of a site's walks choose its genotype unless --no-group-candidates
is given, which chooses each site's genotype over its walks alone. MultiPassCaller and
TreeGenotyper keep grouping off unless it is set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… changes the call

The five TAP tests of grouped walks gave the same genotypes with grouping on and off, so they could
not catch a change to it. Two tests replace them, on reads where grouping changes the call: the
default call, and the call with --no-group-candidates.

The candidate_groups unit test covers walks that differ inside an inverted chain, inside a chain the
reference does not cross and inside a chain crossed twice, and the read_likelihood_caller unit tests
cover the scores the groups choose by. The calls of nested SNPs under a grouped parent, and phase
carried across two grouped parents, are no longer tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The walks through the chain differ only inside its three SNP children, so
they form one group. One strand of the sample crosses the chain with all three
alternative bases, which no one walk carries, and the other strand deletes
the chain. The group explains the crossing strand, so the parent is
heterozygous for the deletion, and each child is written at ploidy 1. With
grouping off, the walks alone call both strands through the chain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@adamnovak adamnovak added the inference Theme/team for genome inference (calling, assembly, etc.) label Nov 2, 2026
@adamnovak

Copy link
Copy Markdown
Member

@benedictpaten This needs to be rebased or back-merged or something, so the Github diffs only shows the differences since #4990, before it can be reviewed.

benedictpaten and others added 4 commits November 5, 2026 01:11
Read phasing joins the phase chain into runs where reads link pieces, and orients each run
relative to the others only by the panel. Re-genotyping summed every read's strand log-odds over
the whole phase set and scored a heterozygote under one strand assignment for all of a site's
reads. At a site whose reads come from two runs the panel framed in opposite orientations, every
assignment conflicted with one run's reads, so the homozygote won. On the noclip C_n0 graph this
turned the 6.5 Mb chr20 centromere site from het to 1|1 and swallowed its nested records.

Read phasing now numbers the runs and gives each site the run its order is relative to. Each
read's log-odds are kept per run, and a read whose other sites lie in two runs keeps the part of
one. Re-genotyping groups a site's reads by run: the site's own run, or else its largest, takes
the better assignment, and every other group's assignment is averaged over. A nested haploid
chain, and the anchor slot of a het site, use only strands from their own run. A site whose reads
with a strand come from one run, none of them from two, is scored exactly as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read phasing now fits how often the panel's phase is wrong, from the pieces of the phase chain that
reads alone order. The panel errs by switches, at a rate per base, and by flips of single sites, so
it phases two sites d bases apart as the reads would with probability
p(d) = (1 + (1 - 2f)^2 e^(-2 lambda d)) / 2. The rates are fitted by maximum likelihood on a grid,
over all phase sets, before any is phased. The flip rate is at least 1e-4, which keeps the panel's
vote finite. With too few pairs nothing is fitted and phasing is as before.

Where the rates are fitted:
- At a break of the phase chain, the panel's vote for its order of the nearest sites it orders
  either side joins the read links across the break. A break no read crosses is then oriented
  against its neighbour at the break, rather than by each run's first site the panel orders.
- A site hung off the chain gets the panel's vote at the distance to its nearest chain site the
  panel orders, instead of the fixed --phase-prior.

A break starts a new run unless reads link across it and it is oriented as their links alone say.
Each run records its correlation with the run before it, 2p - 1 for the probability p that the
decision at the break was right. Re-genotyping and the anchor file's het-slot tilt use the product
of run correlations as the probability that two runs' strands agree, instead of 1/2.

The fit is reported as a progress line, with a table of observed and predicted agreement by
distance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A site off the phase chain was phased by summing its links to its nearest
phase-chain sites, so a read spanning all of them counted its allele at the
site once per site. Long reads span them all, and the inflated sums outvoted
the panel's fitted vote where the reads were wrong. Each read now votes once
for each run among those sites: its alleles there give the strand it came
from, and its allele at the site then favours one order. With one voting
site, this is the link between the two.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@adamnovak adamnovak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the stuff that isn't in artifacts/.

I think we should change one of the test comments, and if possible make the parse-a-VCF-in-AWK test actually be a C++ unit test to avoid doing that.

I pulled this in on the group-candidates branch so CI can run.

Comment thread test/t/18_vg_call.t Outdated
# chains share a group. The sample carries the insertion >2>10..13>5 on both strands, which differ
# inside it at node 11 or 12, and the SNP at node 7 or 8 on one strand. The walks through the
# insertion share a group, though the reference does not cross its chain, and so do the walks that
# differ only at the SNP. The reads have no MAPQ, so each may be mismapped.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not what having no MAPQ means. Reads can be mismapped if MAPQ is specified, or if it isn't.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 298ecfe: vg sim leaves the reads at MAPQ 0, so each gets the mismapping probability --mismap-max; the walks alone miscall the site only when that is at least 0.95, so the test now sets it.

Comment thread test/t/18_vg_call.t Outdated
Comment on lines +279 to +291
# Each record as POS:number of ALTs:GT, with the GT's two alleles sorted, since either strand order
# is the same call.
calls_of() {
grep -v '^#' "$1" | awk -F'\t' '{split($10, a, ":"); split(a[1], g, "|");
printf "%s:%s:%s ", $2, split($5, b, ","), (g[1] < g[2] ? g[1] "|" g[2] : g[2] "|" g[1])}'
}
vg call nest_off.gbz --read-likelihood --gam nest_off.gam -t 1 -s samp 2>/dev/null > nest_off.vcf
is "$(calls_of nest_off.vcf)" "47:2:1|2 71:1:0|1 " \
"by default groups of walks call both insertion alleles, and the SNP on one strand"
vg call nest_off.gbz --read-likelihood --gam nest_off.gam -t 1 -s samp --no-group-candidates \
2>/dev/null > nest_off_walks.vcf
is "$(calls_of nest_off_walks.vcf)" "29:1:0|1 47:1:.|1 71:1:.|1 " \
"with --no-group-candidates the walks alone choose, and call the deletion on one strand"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just barely understandable, since we explain what the write-only Awk code is doing. But in general I don't think parsing VCF with Awk is a great approach. If we can't express what we need with reallu basic cut and tr and sort, then it probably wanted to be a unit test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now cut and tr only, in 298ecfe. Sorting the alleles turned out to misstate the nested haploid records, so it compares the phased GTs as written. The mechanism's unit test is in artifacts/unittest/multipass_caller.cpp.

benedictpaten and others added 10 commits November 16, 2026 14:17
…nchors

A graph sampled by vg haplotypes with its reference paths kept can draw reads
onto a reference path instead of the sampled haplotypes. The guide now says how
to map to a copy of the graph without the reference samples, made with vg gbwt,
and call on the graph that has them, and the mosaic and anchor sections point
to it for such graphs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Q comment

The test of grouped walks parsed the VCF with awk and sorted each GT's alleles.
It now compares each record's POS and phased GT as written, read with cut and
tr; the sort also made the nested haploid records describe an impossible
phasing. Its comment said the reads have no MAPQ, so each may be mismapped.
vg sim leaves them at MAPQ 0, which gives each read the mismapping probability
--mismap-max, and the walks alone miscall the site only when that is at least
0.95, so the test now sets it to 0.95.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…urces

Each A row now ends with a kind column: het, hom, hom-split or haploid for
a site's anchors, and fill-shared, fill-strand or fill-off-call for fill
anchors. The anchor's bool edit becomes a PinSource (snarl, off-panel or
fill), which ranks anchors in the one-anchor-per-read-point rule: a snarl's
anchor keeps every read, an off-panel anchor gives way to snarls and to
earlier off-panel anchors, and a fill anchor gives way to both and to earlier
fill anchors, before the minimum-reads filter. Fill anchors on one node follow
the node's site anchors, ordered by pin. A fill anchor writes '.' for allele,
gqn, explained and reliability. The header gains fill=K in #filters and notes
defining fill anchors and the kind column. Rows of existing anchors are
unchanged apart from the new column.

check_anchors.py reads version 10 and checks the kind column, and
anchors_to_json.py accepts '.' in the allele and explained columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…runs

--anchors-fill N (default 1000 with --anchors-out, 0 for none) places
ceil(L/N)-1 evenly spaced points inside each unitig, a longest run of nodes
joined with no branch, of length L > N. Each unitig is read from its end with
the smaller node ID, so the points depend on the graph alone, and a point on
a join between nodes moves forward into the next node.

After the direct pass, while the read source is open, FillAnchors fetches the
reads on the points' nodes one node-ID window at a time, on several threads,
and places each read at each point with resolve_anchor_pin as an exit pin
along the node's forward strand. The fetch uses the new
SiteReadSource::for_each_read_uncounted, which tallies no read starts and
counts no off-panel candidates, and no pass after the direct pass reads reads,
so the sweep changes no genotype. A read's score is its phred-scaled clamped
mismapping probability.

During the render, each called site marks the points that its strands'
walks cross, among the points it is the innermost site to hold. A point in
no site, or crossed by both strands, is fill-shared in slot 0; one crossed
only by strand s is fill-strand in slot s; one crossed by no called walk is
fill-off-call. PhaseTable::nested_strand tells a nested haploid chain from a
region of ploidy 1. Counts go to stderr when fill anchors are on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Assembly anchors section gains a Fill anchors part: the definition, the
slots and kinds, the rank in the one-anchor-per-read-point rule, and the
caveat that in centromeric arrays and other large repeats a fill anchor
shares the placement errors of the reads it rests on. The row tables gain the
kind column and the '.' fields of fill anchors, and the Options table gains
--anchors-fill.

18_vg_call.t checks that fill anchors leave the VCF unchanged, that
--anchors-fill 20 writes them and 0 writes none, that no read point is in two
anchors, that every A row ends with its kind, that one and four threads write
the same anchor file, and that --anchors-fill needs --anchors-out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fill point's slot and kind now come from its deciding site: the innermost
site holding the point's node that marked a called walk. FillAnchors keeps,
for each point, the sites that hold it, innermost first, and a mark that
records the bits of the deepest site whose walk crossed it. A site that was
not genotyped, as under --no-nested or below a nested site that failed, marks
nothing, so its parent's walks decide its points instead of leaving them
fill-off-call. A genotyped site still decides its own points, whatever its
parent's walks.

A nested site with a one-allele genotype and no known strand now marks
nothing, so its parent's walks put its points on the strand that carries
it. It used to mark them as on every strand, fill-shared. Only a one-allele
genotype at a site that is not nested is taken as a region of ploidy 1.

A read is now placed at a fill point only where it crosses it. A read whose
mapping on the node does not reach the node base before the point is counted
apart from one with no base aligned there, and a read that stops at the point,
or goes on past it only in a soft clip, is refused. Snarl and off-panel pins
are unchanged.

The anchor file's notes and the docs give the deciding site, the frame of a
fill-strand slot, the read placement rule, and the direction a unitig of one
node or a cycle is read in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… further

A windowed read source now ends its site-query fetch statistics when the
first for_each_read_uncounted query starts, so the logged figures describe the
site queries alone, and a separate line reports the fill anchors' fetches.

anchors_to_json.py no longer drops fill anchors under --min-gqn, as vg's
--anchors-min-gqn does not apply to them. The --anchors-out help mentions the
fill anchors.

18_vg_call.t now checks that the anchor file records fill=0 and fill=1000, that
fill anchors from an indexed GAM read over several windows leave the VCF and
the off-panel dump unchanged and do not depend on the thread count, and that
the fill anchors on the nested haploid fixture name the strand that carries
the chain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…an site anchors

A site anchor leaves out reads whose best allele the site was not called
with, which drops some reads misplaced among the copies of a repeat; a fill
anchor keeps every read crossing its point. The fill-anchor docs now say so,
and that reads from close copies elsewhere in the genome can reach anchors
of every kind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The anchors were collected just before a site's line was written, from the called traversals. The
line then numbers those traversals as VCF alleles, and two called traversals that spell one
sequence become one allele, while a traversal that is symbolically the reference becomes allele 0.
So an anchor could say het where the line's GT is 1|1, and its gqn, recomputed for a moved record
over the called traversals and the reference traversal, could differ from the GQN the line wrote,
which is recomputed over the written alleles. On chr20 this was 4 kind and 42 gqn disagreements
with short reads, and 2 and 10 with ONT on the 32-haplotype graph.

The render now writes the line first. The emitter reports which traversal each VCF allele is spelled
from and whether the site got a line, and the anchors replace each called traversal by its
allele's traversal, pool the reads' shares of the two, and compute gqn over the written alleles. A
site with no line, such as a parent written as the reference, keeps its called traversals, and so
do the fill points. Alleles that -L merges as similar are not followed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ss fetches

Finding the fill anchors' reads after the direct pass fetched and parsed every window of reads a
second time. The points are now placed before the direct pass, and the windowed read source
passes each window it fetches whole to FillAnchors::resolve_window_reads, which places the reads
crossing that window's points while the window is in memory. resolve_reads then fetches only the
windows the direct pass never fetched whole. The reads and their order are those of the fetch
they replace, so the anchor file does not change. On chr20, fill anchors now add about 6% to the
run time with ONT reads, against 17%, and about 1% with short reads, against 11%.

SiteReadSource::set_window_observer gives an observer each window a windowed source fetches whole,
without adding its reads to the fetch statistics. FillAnchors::place_points places the points and
groups them by the source's windows; place then takes only the decomposition, and resolve_reads
uses place_points' windows. The fill report says how many windows' reads were placed each way,
and 18_vg_call.t checks that the indexed-GAM run places some as the sites fetch them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

inference Theme/team for genome inference (calling, assembly, etc.)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants