Skip to content

vg dedup subcommand for content-aware node chopping - #5115

Open
jltsiren wants to merge 11 commits into
masterfrom
vg-dedup
Open

jltsiren wants to merge 11 commits into
masterfrom
vg-dedup

Conversation

@jltsiren

Copy link
Copy Markdown
Contributor

Changelog Entry

To be copied to the draft changelog by merger:

  • vg dedup subcommand to enable sequence deduplication in GBZ v4.

Description

This PR adds vg dedup subcommand for content-aware node chopping. It takes an unchopped graph as an input and uses contend-defined chunking for finding good positions to chop long nodes. The purpose is to enable sequence deduplication in full graphs.

Instead of storing a concatenation of the sequences over all nodes, GBZ v4 will store an arbitrary superstring of the sequences. In an HPRC v2.1 full graph, the total length of the nodes is 63 Gbp due to unaligned centromeres. Preliminary experiments suggest that the length of the stored superstring should be less than 10 Gbp with the default vg dedup parameters. That should enable working with full graphs on a 32 GB laptop.

@jltsiren jltsiren added the construction Theme/team for pangenome graph construction label Nov 17, 2026
@jeizenga

Copy link
Copy Markdown
Contributor

Hi! I think this strategy will probably conflict with the vg mask implementation, which does a kind of naughty thing and directly modifies the GBWTGraph sequences to hard mask them. I think it would be sufficient to just check the GBZ version there and fail early with a warning if the sequences are compressed in this way, since vg gbwt provides an interface to down-version a given GBZ file.

@jltsiren

Copy link
Copy Markdown
Contributor Author

The in-memory representation will change anyway, regardless of the file format version, so we'll have to change everything that manipulates the sequences directly. But it should be straightforward to linearize the representation and store all sequences independently if requested.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

construction Theme/team for pangenome graph construction

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants