Skip to content

Let giraffe write graph alignments beside its main output (--graph-output) - #5105

Open
scttfrdmn wants to merge 13 commits into
vgteam:masterfrom
scttfrdmn:giraffe-graph-output
Open

scttfrdmn wants to merge 13 commits into
vgteam:masterfrom
scttfrdmn:giraffe-graph-output

Conversation

@scttfrdmn

Copy link
Copy Markdown

What this does

vg giraffe gains two output options:

      --graph-output FILE       also write the graph alignments to FILE, as
                                -o GAM/GAF would, before any surjection
      --graph-output-fmt NAME   format for --graph-output {gam / gaf} [gam]

With -o BAM (or SAM/CRAM), one mapping pass then writes both outputs:

  • the surjected BAM, to stdout as before;
  • the graph alignments the surjecting emitter was given, to FILE, through the same emitter -o gam / -o gaf would build (same graph, same flags, GAF header lines included).

It works with any -o; for example -o gam --graph-output x.gaf --graph-output-fmt gaf writes both GAM and GAF. It is refused with --output-basename (grid search writes one file per combination) and with -n.

Implementation

  • artifacts/tee_alignment_emitter.{hpp,cpp}: a new TeeAlignmentEmitter. It is an AlignmentEmitter that owns two emitters and hands every batch (singles, mapped singles, pairs, mapped pairs, extra messages) to both. The primary emitter gets a copy of each batch, because an emitter may move from what it is given. The copy emitter gets the batch itself.
  • giraffe_main.cpp: wraps the existing emitter in a TeeAlignmentEmitter when --graph-output is set. With no --graph-output, nothing changes.
  • artifacts/: the new files are there, per CLAUDE.md, since they were written with Claude Code; I have read them. I did not run the lint-new-code (SonarLint) step, because there is no Java runtime on this machine.

Use case and measured benefit

Our pipeline uses one mapping pass for two consumers:

  • a linear-reference BAM, for small-variant calling;
  • a GAM, for vg pack → vg call structural-variant genotyping.

Without this, giraffe runs twice per sample (-o bam, then -o gam), or the GAM is surjected afterwards with vg surject. vg surject is not the same surjection as giraffe's internal one: the band padding, the proper-pair limit and the pairing logic all differ. On our data, 0.08% of BAM records differed.

We measured an earlier version of this change. It was a local patch on v1.75.1, the same tee in front of the surjecting emitter, but switched by an environment variable and without the batch copy. The run was a 30× HG002 short-read genome (681,708,020 reads) on the HPRC v1.1 CHM13 graph, on one 64-vCPU Graviton box:

wall CPU
two passes: giraffe → BAM, giraffe → GAM, vg pack 5,188 s 83.9 CPU-h
one pass: giraffe → BAM + GAM, vg pack 2,884 s 46.4 CPU-h
saved −2,304 s (−44%) −37.5 CPU-h (−45%)
  • Cost of the extra output: +2.5–3.0% CPU (+58–71 s) against the same box writing BAM only. That is GAM serialization at BGZF level 1, about 152 GB per genome.
  • Outputs: the one-pass BAM's records were identical to the BAM-only run's. The pack from the one-pass GAM had the same md5 as the pack from the separate -o gam run. The vg call VCF records were identical.

This version adds one copy of each batch for the primary emitter. I have not re-measured it whole-genome. On the 1 kb x test graph, 200,000 simulated pairs at -t 4 (Docker on an arm64 Mac, median of 3 runs), mapping costs almost nothing, so this shows the serialization cost, not the production ratio:

wall user sys
-o BAM 17.1 s 69.9 s 11.5 s
-o BAM --graph-output x.gam 18.1 s 82.8 s 22.7 s
-o GAM alone (the second pass this replaces) 9.0 s 35.2 s 5.3 s

Tests

test/t/50_vg_giraffe.t, paired reads on the x test graph at -t 1, against separate runs:

  • the SAM is byte-identical whether or not GAM or GAF is also written;
  • the GAM written beside SAM holds the same records as a separate -o gam run. time_used is set to null first, since it records each read's mapping time and differs between any two runs. The embedded PARAMS_JSON is also the same;
  • the GAF written beside SAM is byte-identical to a separate -o gaf run;
  • --graph-output with --output-basename is refused.

artifacts/unittest/tee_alignment_emitter.cpp: both emitters receive every alignment and extra message, even when the primary clears what it is given; null emitters are refused.

Local results, master at e0d5918 plus this branch, built in vg's Dockerfile packages stage (Ubuntu 26.04, arm64):

  • prove t/50_vg_giraffe.t: 96/96 pass;
  • vg test "[emitter]": 4 assertions in 2 test cases pass;
  • scripts/lint.py: clean;
  • lint_prose.py --base HEAD: clean.

Add --graph-output FILE and --graph-output-fmt {gam,gaf} to vg giraffe.
The alignments giraffe emits also go, before any surjection, through the
emitter that -o GAM or -o GAF would make, to FILE. With -o BAM, one mapping
pass then gives both a linear-reference BAM and the GAM that vg pack and
vg call need, instead of mapping the reads twice.

A new TeeAlignmentEmitter owns both emitters and gives each every batch;
the primary one gets a copy, since an emitter may move from its batch.

Tests check, against separate runs, that the SAM is unchanged, that the
GAM holds the same records (apart from time_used) and that the GAF is the
same file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@adamnovak adamnovak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this feature is a good idea. We didn't need all the benchmarking; it's pretty clear that writing to GAM and then loading up the graph again and passing over everything again to surject it is slower.

I think we might need to think more about how to explain and structure this though. The PR is thinking of it as Giraffe usually outputting linear alignments, and then in that case you can now additionally output graph alignments, with a file and a format.

But I think we usually present Giraffe as a tool for outputting graph alignments, and then you can optionally change the output to a linear format. If I'm a user expecting graph output, and I see:

      --graph-output FILE       also write the graph alignments to FILE, as
                                -o GAM/GAF would, before any surjection

Then I might wonder if I'm somehow not supposed to expect graph output by default, and if I need either this option or -o to get it. If I want to construct a command line that outputs both GAF and GAM, I can do it with the new flags, but then one of them is the "graph" output and one of them isn't.

I think what we should probably do is move another level up in abstraction, so everything becomes more natural. Instead of output to standard out and also up to exactly one file that can only contain un-surjected alignments, we should have an output file option. We can use the extension on the output file to determine the format, so it can support graph and surjected alignment. And we can have any number of them, so if you want GAF, GAM, and BAM, you can do that. It could replace the output file prefix option; when we need to make multiple files for the old parameter grid search stuff, we can stick the extra name component before the last extension of each file. And if you specify any of these, you won't get standard out output, and -o should be disallowed. To implement all this, the TeeAlignmentEmitter would be extended to any number of downstream emitters.

I'll see if I can get this to look more like that.

Comment thread src/subcommand/giraffe_main.cpp Outdated
Comment on lines +1600 to +1603
if (!graph_output_name.empty() && !output_basename.empty()) {
logger.error() << "A graph output file (--graph-output) cannot be used "
<< "with an output basename (--output-basename)." << std::endl;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why shouldn't this work? It makes sense in the one-setting-combination case.

Comment thread artifacts/tee_alignment_emitter.hpp Outdated
Comment on lines +18 to +30
* A TeeAlignmentEmitter is an AlignmentEmitter that hands every batch of
* alignments to two emitters it owns: a primary one and a copy one.
*
* Both emitters receive the same alignments in the same batches, so each one
* writes what it would have written if it had been the only emitter. This lets
* one mapping run write, for example, surjected SAM/BAM/CRAM through the
* primary emitter and the graph alignments as GAM or GAF through the copy
* emitter, instead of mapping the reads twice.
*
* The primary emitter gets a copy of each batch, and the copy emitter gets the
* batch itself, since an emitter may move from or change what it is given.
* Like any AlignmentEmitter, a TeeAlignmentEmitter may be called from several
* threads at once, and relies on the two emitters it owns to be safe for that.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds a lot like unrefined Anthropic Claude output:

  • It restates invariants of the base class only to tell us that of course they continue to apply.
  • It defines the class in terms of the user-facing feature it is going to enable instead of as an abstraction to be understood in isolation.
  • It doesn't put the cover on the abstraction, so the distinct concepts of "primary emitter" and "copy emitter" leak out to take up slots in the class user's brain when they don't need to.
  • It has a lot of unnecessarily specific qualifiers (we hand "every batch" of alignments to the other emitters, as if a class that decided not to handle some batches was a live possibility).
  • It's 3 paragraphs long when An AlignmentEmitter that emits alignments with two other AlignmentEmitters. would suffice.

Now, this is in artifacts/, where unrefined Anthropic Claude output indeed belongs. But it's such a simple job for the class that I don't know why we wouldn't just fix it up and promote it.

Automated Commit and others added 11 commits November 13, 2026 12:48
Replace --graph-output and --graph-output-fmt with a repeatable
--output-file, which takes the place of standard output. With ranges,
each file name gets the settings slug before its extension, as
--output-basename names do.

Move TeeAlignmentEmitter to src/, taking any number of destinations.

Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Leave -o unset unless given, check output file extensions against the
formats -o takes, and test standard output, one output file and several.

Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Refuse two outputs in the same format, and check -o and output file
extensions together after parsing.

Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Keep each output's format and file in an unordered map, check output
files for writability as they are parsed, and give each output file its
own column in the --report-name TSV.

Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Only SAM, BAM and CRAM get the positional graph, so named coordinates
work beside surjected output or correctness tracking. Check output files
can be written in one place after parsing, sharpen comments, and test
the report, the file cleanup and each refusal's message.

Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Assisted-by: Claude Code:claude-opus-5-5
Reduce tests for error messages and many combinations of output formats.
Rework logging emitter to be simpler.
Rewrite error messages.
Rewrite flag descriptions.
Steal -O flag for new output file option.
Redo comments for TeeAlignmentEmitter.
Rework output file writability detection to not need to make and then
delete files.
@adamnovak

adamnovak commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

OK, I've reworked this to replace --output-basename with the new system, which is -O output.gam -O output.sam etc.

I'm stealing -O from --no-dp, because it makes a lot more sense here, but that's a breaking change in case anyone was using that testing option.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants