From 4cccbffa91a3e572393da680473e3bdb7191933f Mon Sep 17 00:00:00 2001 From: Glenn Hickey Date: Fri, 13 Nov 2026 14:31:28 -0500 Subject: [PATCH] Add vg lift: lift features to the graph, the gref and every path A tool that scans a cover (the gref paths of a GBZ, or the tigs vg unitigs writes) reports what it finds in the coordinates of the sequence it read, and those coordinates mean nothing to anything else. vg lift translates them back: it takes a BED on a cover sequence or on any path in the GBZ and writes a GAF whose records carry each feature's graph walk, how many non-gref paths visit the whole walk (sp:i:), its gref placement (gr:Z:), its path coordinate (pc:Z:) and its BED name (an:Z:). A feature that several sequences spell at the same graph place is reported once. The path coordinate names a place in the graph without depending on node IDs: PATH:START-END:STRAND on the first path that carries the whole feature, taking the gref path it lies along before a reference or assembly path. Paths keep their coordinates across clipped, filtered and sampled graphs, and no two places in a graph share a coordinate. With a VCF, each ALT allele becomes a feature: the allele's walk without the first and last nodes, which sit on the reference on either side. The walk comes from INFO/AT, or from a PanGenie variant ID for a single-ALT record without one. The output VCF carries each allele's gref placement in INFO/GR and its path coordinate in INFO/PC. Where an allele runs along a gref alt contig and differs from it -- an insertion carrying SNPs, say -- each difference becomes a biallelic record on that contig, written after the calls the way vg deconstruct writes nested records. A haplotype gets 1 where its called allele has the variant, 0 where the allele reads that stretch of the contig without it, and . otherwise, so frequencies are among the carriers of the contig; GQ is the lowest GQ of the calls behind the genotype. This is what lets a caller's output be read in gref coordinates, where the alt contigs hold the sequence the reference lacks. With --depth it writes how many distinct haplotypes read each base of the gref paths as bedGraph, including runs of depth 0, optionally within regions. Whether a VCF belongs to this graph is checked by looking for each ALT allele in the sequence of its walk, allowing for the padding base at either end. A one-base ALT against a longer REF is only that padding base, since the trimming some panels do takes the allele's own sequence out of the columns, and nothing is then left to test. A GT that names an ALT allele its record does not have is read as the reference and counted, since reading it that way is a guess. Features are read as a VCF only when the path names a regular file that htslib reads as one. Probing a FIFO would read its first block into htslib's detection buffer and throw it away on close, leaving whoever opens the path next in the middle of the file, and a VCF has to be a regular file here in any case, since the lift reads it twice. Co-Authored-By: Claude Opus 5 --- autocomp.bash | 2 +- autocomplete.sh | 2 +- doc/vgmanmd.desc.md | 3 +- doc/vgmanmd.py | 2 +- src/feature_lift.cpp | 655 +++++++++++++++++++++ src/feature_lift.hpp | 381 ++++++++++++ src/haplotype_depth.cpp | 353 +++++++++++ src/haplotype_depth.hpp | 160 +++++ src/nested_variants.cpp | 754 ++++++++++++++++++++++++ src/nested_variants.hpp | 330 +++++++++++ src/sorted_vcf_writer.hpp | 2 +- src/subcommand/lift_main.cpp | 1066 ++++++++++++++++++++++++++++++++++ src/vcf_alleles.cpp | 500 ++++++++++++++++ src/vcf_alleles.hpp | 186 ++++++ test/t/59_vg_lift.t | 375 ++++++++++++ 15 files changed, 4766 insertions(+), 5 deletions(-) create mode 100644 src/feature_lift.cpp create mode 100644 src/feature_lift.hpp create mode 100644 src/haplotype_depth.cpp create mode 100644 src/haplotype_depth.hpp create mode 100644 src/nested_variants.cpp create mode 100644 src/nested_variants.hpp create mode 100644 src/subcommand/lift_main.cpp create mode 100644 src/vcf_alleles.cpp create mode 100644 src/vcf_alleles.hpp create mode 100644 test/t/59_vg_lift.t diff --git a/autocomp.bash b/autocomp.bash index d1a78c5a29..dcc4c39371 100755 --- a/autocomp.bash +++ b/autocomp.bash @@ -3,7 +3,7 @@ _vg_complete() local cur prev COMPREPLY=() cur="${COMP_WORDS[COMP_CWORD]}" - opts="add align annotate augment autoindex bench-dist-query benchmark call chains chunk circularize clip cluster combine concat construct convert deconstruct depth describe dotplot filter find gamcompare gampcompare gamsort gbwt genotype giraffe haplotypes help ids index inject kmers map mask mcmc minimizer mod mpmap pack paths primers prune rna sim simplify snarls sort stats surject test trace translate unitigs validate vectorize version view viz zipcode" + opts="add align annotate augment autoindex bench-dist-query benchmark call chains chunk circularize clip cluster combine concat construct convert deconstruct depth describe dotplot filter find gamcompare gampcompare gamsort gbwt genotype giraffe haplotypes help ids index inject kmers lift map mask mcmc minimizer mod mpmap pack paths primers prune rna sim simplify snarls sort stats surject test trace translate unitigs validate vectorize version view viz zipcode" if [ $COMP_CWORD -eq 1 ] then diff --git a/autocomplete.sh b/autocomplete.sh index 7a9ec6c118..bc7dc1e369 100644 --- a/autocomplete.sh +++ b/autocomplete.sh @@ -6,7 +6,7 @@ then local cur prev COMPREPLY=() cur="${COMP_WORDS[COMP_CWORD]}" - opts="add align annotate augment autoindex bench-dist-query benchmark call chains chunk circularize clip cluster combine concat construct convert deconstruct depth describe dotplot filter find gamcompare gampcompare gamsort gbwt genotype giraffe haplotypes help ids index inject kmers map mask mcmc minimizer mod mpmap pack paths primers prune rna sim simplify snarls sort stats surject test trace translate unitigs validate vectorize version view viz zipcode" + opts="add align annotate augment autoindex bench-dist-query benchmark call chains chunk circularize clip cluster combine concat construct convert deconstruct depth describe dotplot filter find gamcompare gampcompare gamsort gbwt genotype giraffe haplotypes help ids index inject kmers lift map mask mcmc minimizer mod mpmap pack paths primers prune rna sim simplify snarls sort stats surject test trace translate unitigs validate vectorize version view viz zipcode" COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) ) return 0 diff --git a/doc/vgmanmd.desc.md b/doc/vgmanmd.desc.md index bad26fe1cd..bf859cfd1c 100644 --- a/doc/vgmanmd.desc.md +++ b/doc/vgmanmd.desc.md @@ -31,7 +31,7 @@ For more in-depth explanations of tools and workflows, see the [general wiki pag - [`vg depth`](#depth): calculate path coverage depth or aggregate alignment coverage Also see [vg pack](#pack) - [`vg pack`](#pack): convert alignments to a compact coverage index. Used with [vg call](#call) - [`vg primers`](#primers): filter primers based on variation in a graph. [wiki page](https://github.com/vgteam/vg/wiki/Primer-Filter) - - [`vg unitigs`](#unitigs): cover the haplotype k-mers of a graph with walks, for scanning with linear tools. + - [`vg unitigs`](#unitigs): cover the haplotype k-mers of a graph with walks, for scanning with linear tools. Use with [vg lift](#lift) - **Working with read alignments** - [`vg augment`](#augment): embed alignments into a graph. - [`vg filter`](#filter): filter alignments by properties. @@ -62,6 +62,7 @@ For more in-depth explanations of tools and workflows, see the [general wiki pag - **Conversion between formats** - [`vg convert`](#convert): convert between handle graph formats and GFA, and between alignment formats. - [`vg inject`](#inject): project alignments on a linear reference onto a graph (bam/sam/cram->gam/gaf). + - [`vg lift`](#lift): lift BED features on a [unitig](#unitigs) cover or on paths to the graph, the gref and every path. - [`vg paths`](#paths): extract a fasta from a graph. [wiki page](https://github.com/vgteam/vg/wiki/Linear-references-and-vg) - [`vg surject`](#surject): project alignments on a graph onto a linear reference (gam/gaf->bam/sam/cram). - [`vg vectorize`](#vectorize): export GAM alignments to other vector formats. diff --git a/doc/vgmanmd.py b/doc/vgmanmd.py index 87cf319b4a..004d368398 100755 --- a/doc/vgmanmd.py +++ b/doc/vgmanmd.py @@ -9,7 +9,7 @@ 'cluster', 'combine', 'construct', 'convert', 'deconstruct', 'depth', 'describe', 'dotplot', 'filter', 'find', 'gamcompare', 'gampcompare', 'gamsort', 'gbwt', 'giraffe', 'haplotypes', 'help', 'ids', - 'index', 'inject', 'map', 'mask', 'minimizer', 'mod', 'mpmap', + 'index', 'inject', 'lift', 'map', 'mask', 'minimizer', 'mod', 'mpmap', 'pack', 'paths', 'primers', 'prune', 'rna', 'sim', 'simplify', 'snarls', 'stats', 'surject', 'test', 'trace', 'unitigs', 'validate', 'vectorize', 'version', 'view', 'zipcode' diff --git a/src/feature_lift.cpp b/src/feature_lift.cpp new file mode 100644 index 0000000000..9e70043498 --- /dev/null +++ b/src/feature_lift.cpp @@ -0,0 +1,655 @@ +/** + * \file feature_lift.cpp + * Lifting features, see feature_lift.hpp. + */ + +#include "feature_lift.hpp" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +namespace vg { + +using handlegraph::handle_t; +using handlegraph::nid_t; +using gbwtgraph::GBWTGraph; + +//------------------------------------------------------------------------------ + +namespace { + +/** + * Parse a BED line into a feature. + * + * @throws std::runtime_error if the line has fewer than 3 columns, bad + * coordinates, no bases, or a strand other than +, - and ".". + */ +BedFeature parse_bed_line(const std::string& line, size_t line_number) { + auto fail = [&](const std::string& problem) { + throw std::runtime_error("BED line " + std::to_string(line_number) + ": " + problem + ": " + line); + }; + std::vector columns; + std::stringstream stream(line); + std::string column; + while (std::getline(stream, column, '\t')) { + columns.push_back(column); + } + if (columns.size() < 3) { + fail("fewer than 3 columns"); + } + BedFeature feature; + feature.sequence = columns[0]; + try { + size_t used_start = 0; + size_t used_end = 0; + feature.start = std::stoull(columns[1], &used_start); + feature.end = std::stoull(columns[2], &used_end); + if (used_start != columns[1].size() || used_end != columns[2].size()) { + fail("bad coordinates"); + } + } catch (const std::invalid_argument&) { + fail("bad coordinates"); + } catch (const std::out_of_range&) { + fail("bad coordinates"); + } + if (feature.start >= feature.end) { + fail("empty feature"); + } + feature.fields.assign(columns.begin() + 3, columns.end()); + if (feature.fields.size() > BED_STRAND_FIELD) { + const std::string& strand = feature.fields[BED_STRAND_FIELD]; + if (strand != "+" && strand != "-" && strand != ".") { + fail("strand is not +, - or ."); + } + } + return feature; +} + +} + +std::vector read_bed_features(std::istream& in) { + std::vector features; + std::string line; + size_t line_number = 0; + while (std::getline(in, line)) { + line_number++; + if (!line.empty() && line.back() == '\r') { + line.pop_back(); + } + if (line.empty() || line[0] == '#' || line.starts_with("track") || line.starts_with("browser")) { + continue; + } + features.push_back(parse_bed_line(line, line_number)); + } + return features; +} + +bool GraphFeature::is_stranded() const { + return fields.size() > BED_STRAND_FIELD && fields[BED_STRAND_FIELD] != "."; +} + +bool PathCoordinate::operator<(const PathCoordinate& other) const { + return std::tie(rank, start, end, is_reverse) < std::tie(other.rank, other.start, other.end, other.is_reverse); +} + +std::string PathCoordinate::text() const { + return path + ':' + std::to_string(start + 1) + '-' + std::to_string(end) + ':' + (is_reverse ? '-' : '+'); +} + +GraphFeature make_graph_feature(const GBWTGraph& graph, const GrefIndex* gref, + WalkInterval interval, std::vector fields, + std::optional before, std::optional after) { + GraphFeature result; + result.interval = std::move(interval); + result.fields = std::move(fields); + // Read the interval the other way, with the nodes on either side. + auto reverse = [&]() { + result.interval = result.interval.reverse(graph); + std::optional old_before = before; + before = after ? std::optional(graph.flip(*after)) : std::nullopt; + after = old_before ? std::optional(graph.flip(*old_before)) : std::nullopt; + }; + bool stranded = result.is_stranded(); + if (stranded && result.fields[BED_STRAND_FIELD] == "-") { + reverse(); + result.fields[BED_STRAND_FIELD] = "+"; + } + + if (gref != nullptr) { + result.placement = gref->place(result.interval, before, after); + } + if (!stranded) { + const GrefPlacement& placement = result.placement; + bool flip = placement.reverse_bases > placement.forward_bases || + (placement.reverse_bases == placement.forward_bases && + result.interval.reverse(graph).compare(result.interval) < 0); + if (flip) { + reverse(); + if (gref != nullptr) { + result.placement = gref->place(result.interval, before, after); + } + } + } + + // Gref paths come first in the priority order, and their ranks are their + // places in the gref. + if (gref != nullptr && result.placement.along_one_path) { + uint32_t path = result.placement.path; + uint64_t start = gref->path_shift(path) + result.placement.coord; + result.coordinate = PathCoordinate { gref->path_name(path), path, start, start + result.interval.length, + result.placement.reverse_bases > 0 }; + } + + // The GBWT holds every path in both orientations, so this counts visits in + // either orientation. Leave out the gref path that the interval lies + // along, if any, because a gref path is a copy of part of another path. + result.support = graph.bd_find(result.interval.walk).size(); + if (result.placement.along_one_path) { + result.support--; + } + return result; +} + +//------------------------------------------------------------------------------ + +Cover::Cover(const GBWTGraph& graph, const GrefIndex* gref) : graph(graph), gref(gref) { +} + +void Cover::add_tigs(std::istream& gaf, const std::function& wanted) { + std::string line; + std::string name; + while (std::getline(gaf, line)) { + // GAF header lines start with @. + if (line.empty() || line[0] == '@' || line[0] == '#') { + continue; + } + name = line.substr(0, line.find('\t')); + if (!wanted(name)) { + continue; + } + WalkInterval interval = parse_gaf_columns(line, graph, name); + if (!tigs.try_emplace(name, std::move(interval)).second) { + throw std::runtime_error("tig " + name + " appears twice in the GAF"); + } + } +} + +bool Cover::is_gref_path(const std::string& name) const { + return gref != nullptr && gref->find_path(name) != GrefIndex::NO_PATH; +} + +std::optional Cover::sequence_length(const std::string& name) const { + if (is_gref_path(name)) { + return gref->path_length(gref->find_path(name)); + } + if (auto found = tigs.find(name); found != tigs.end()) { + return found->second.length; + } + return std::nullopt; +} + +WalkInterval Cover::interval(const std::string& name, size_t start, size_t end) const { + if (is_gref_path(name)) { + return gref->interval(gref->find_path(name), start, end); + } + return tigs.at(name).subinterval(graph, start, end); +} + +//------------------------------------------------------------------------------ + +PathPriority::PathPriority(const gbwtgraph::GBZ& gbz, const std::vector& prefixes, + const std::string& gref_prefix, size_t first_rank) { + const gbwt::GBWT& index = gbz.index; + gbwtgraph::sample_name_set reference_samples = gbwtgraph::parse_reference_samples_tag(index); + // Sort keys: the group (a prefix, reference paths, the rest), then the name. + std::vector> keys; + std::unordered_map groups; + for_each_path_start(index, [&](gbwt::size_type path_id, const std::string& name, uint64_t) { + if (name.compare(0, gref_prefix.size(), gref_prefix) == 0 || groups.contains(name)) { + return; + } + size_t group = prefixes.size() + 1; + for (size_t i = 0; i < prefixes.size(); i++) { + if (name.compare(0, prefixes[i].size(), prefixes[i]) == 0) { + group = i; + break; + } + } + if (group > prefixes.size() && + gbwtgraph::get_path_sense(index, path_id, reference_samples) == handlegraph::PathSense::REFERENCE) { + group = prefixes.size(); + } + groups[name] = group; + keys.emplace_back(group, name); + }); + std::sort(keys.begin(), keys.end()); + for (size_t i = 0; i < keys.size(); i++) { + ranks[keys[i].second] = first_rank + i; + } +} + +size_t PathPriority::rank(const std::string& name) const { + return ranks.at(name); +} + +//------------------------------------------------------------------------------ + +PathDirectory::PathDirectory(const gbwtgraph::GBZ& gbz, const std::function& wanted) { + for_each_path_start(gbz.index, [&](gbwt::size_type path_id, const std::string& name, uint64_t start) { + if (wanted(name)) { + fragments[name].emplace_back(start, PathSteps(gbz.graph, path_id)); + } + }); + for (auto& [name, path_fragments] : fragments) { + std::sort(path_fragments.begin(), path_fragments.end(), [](const Fragment& a, const Fragment& b) { + return a.start < b.start; + }); + } +} + +bool PathDirectory::has_path(const std::string& name) const { + return fragments.contains(name); +} + +std::optional PathDirectory::interval(const std::string& name, uint64_t start, uint64_t end) const { + auto found = fragments.find(name); + if (found == fragments.end()) { + return std::nullopt; + } + for (const Fragment& fragment : found->second) { + if (fragment.start <= start && end <= fragment.start + fragment.steps.length()) { + return fragment.steps.interval(start - fragment.start, end - fragment.start); + } + } + return std::nullopt; +} + +//------------------------------------------------------------------------------ + +void sort_and_merge_features(std::vector& features) { + std::sort(features.begin(), features.end(), [](const GraphFeature& a, const GraphFeature& b) { + if (a.placement.path != b.placement.path) { + return a.placement.path < b.placement.path; + } + if (a.placement.coord != b.placement.coord) { + return a.placement.coord < b.placement.coord; + } + if (int order = a.interval.compare(b.interval); order != 0) { + return order < 0; + } + return a.fields < b.fields; + }); + auto end = std::unique(features.begin(), features.end(), [](const GraphFeature& a, const GraphFeature& b) { + return a.interval.compare(b.interval) == 0 && a.fields == b.fields; + }); + features.erase(end, features.end()); +} + +//------------------------------------------------------------------------------ + +FeatureMatcher::FeatureMatcher(const GBWTGraph& graph, const std::vector& features) : + graph(graph), features(features) { + if (graph.get_node_count() == 0) { + return; + } + min_id = graph.min_node_id(); + starts.assign(graph.max_node_id() - min_id + 2, 0); + // The walk read in reverse starts on its last node. + auto first_node = [&](const GraphFeature& feature, bool is_reverse) { + const std::vector& walk = feature.interval.walk; + return graph.get_id(is_reverse ? walk.back() : walk.front()) - min_id; + }; + for (const GraphFeature& feature : features) { + for (bool is_reverse : { false, true }) { + starts[first_node(feature, is_reverse) + 1]++; + } + } + for (size_t i = 1; i < starts.size(); i++) { + starts[i] += starts[i - 1]; + } + entries.resize(starts.back()); + std::vector filled(starts.begin(), starts.end() - 1); + for (size_t i = 0; i < features.size(); i++) { + for (bool is_reverse : { false, true }) { + entries[filled[first_node(features[i], is_reverse)]++] = (i << 1) | size_t(is_reverse); + } + } +} + +void FeatureMatcher::for_each_match(const gbwt::vector_type& path, + const std::function& visit) const { + if (starts.empty()) { + return; + } + for (size_t step = 0; step < path.size(); step++) { + handle_t handle = GBWTGraph::node_to_handle(path[step]); + size_t node = graph.get_id(handle) - min_id; + for (size_t i = starts[node]; i < starts[node + 1]; i++) { + size_t feature = entries[i] >> 1; + bool is_reverse = entries[i] & 1; + if (visits_walk(path, step, features[feature].interval.walk, is_reverse)) { + visit(feature, step, is_reverse); + } + } + } +} + +bool FeatureMatcher::visits_walk(const gbwt::vector_type& path, size_t step, const std::vector& walk, + bool is_reverse) const { + if (step + walk.size() > path.size()) { + return false; + } + for (size_t j = 0; j < walk.size(); j++) { + handle_t expected = is_reverse ? graph.flip(walk[walk.size() - 1 - j]) : walk[j]; + if (GBWTGraph::node_to_handle(path[step + j]) != expected) { + return false; + } + } + return true; +} + +//------------------------------------------------------------------------------ + +FeatureNodeIndex::FeatureNodeIndex(const GBWTGraph& graph, const std::vector& features) : + graph(graph) { + if (graph.get_node_count() == 0) { + return; + } + min_id = graph.min_node_id(); + // Call visit(node, entry) for each node of the feature's walk that holds feature bases. + auto for_each_entry = [&](size_t feature, const std::function& visit) { + const WalkInterval& interval = features[feature].interval; + size_t node_start = 0; + for (const handle_t& handle : interval.walk) { + size_t node_end = node_start + graph.get_length(handle); + // Walk coordinates of the feature bases on this node. + size_t low = std::max(node_start, interval.offset); + if (size_t high = std::min(node_end, interval.offset + interval.length); low < high) { + visit(graph.get_id(handle) - min_id, { feature, low - interval.offset, high - interval.offset, + low - node_start, high - node_start, graph.get_is_reverse(handle) }); + } + node_start = node_end; + } + }; + starts.assign(graph.max_node_id() - min_id + 2, 0); + for (size_t i = 0; i < features.size(); i++) { + for_each_entry(i, [&](size_t node, const Entry&) { + starts[node + 1]++; + }); + } + for (size_t i = 1; i < starts.size(); i++) { + starts[i] += starts[i - 1]; + } + entries.resize(starts.back()); + std::vector filled(starts.begin(), starts.end() - 1); + for (size_t i = 0; i < features.size(); i++) { + for_each_entry(i, [&](size_t node, const Entry& entry) { + entries[filled[node]++] = entry; + }); + } +} + +void FeatureNodeIndex::for_each_anchor(const gbwt::vector_type& path, const std::vector& coords, + const std::function& visit) const { + if (starts.empty()) { + return; + } + for (size_t step = 0; step < path.size(); step++) { + handle_t handle = GBWTGraph::node_to_handle(path[step]); + size_t node = graph.get_id(handle) - min_id; + size_t node_length = graph.get_length(handle); + for (size_t i = starts[node]; i < starts[node + 1]; i++) { + const Entry& entry = entries[i]; + // A path that reads the node the other way finds the bases at the + // mirrored offsets. + bool same = (graph.get_is_reverse(handle) == entry.is_reverse); + uint64_t start = coords[step] + (same ? entry.node_from : node_length - entry.node_to); + visit({ entry.feature, entry.from, entry.to, start, !same }); + } + } +} + +namespace { + +using Anchor = FeatureNodeIndex::Anchor; + +/// The index of the anchor before the first anchor of a chain. +constexpr size_t NO_ANCHOR = std::numeric_limits::max(); + +/// Return the path coordinate after the last base of the anchor. +uint64_t end_of(const Anchor& anchor) { + return anchor.start + (anchor.to - anchor.from); +} + +/** + * Return true if a chain on a feature of the given length can go from anchor + * before to anchor after. + * + * The path must read both anchors the same way. On the path, after must start + * no more than length bases past the end of before. In the feature, after must + * come later in the order the path reads it, which is decreasing for a path + * that reads the feature in reverse. + */ +bool can_follow(const Anchor& before, const Anchor& after, size_t length) { + return before.is_reverse == after.is_reverse && end_of(before) <= after.start && + after.start - end_of(before) <= length && + (after.is_reverse ? after.to <= before.from : after.from >= before.to); +} + +/// Return true if anchor after starts where anchor before ends, both on the +/// path and in the feature as the path reads it. +bool abuts(const Anchor& before, const Anchor& after) { + return after.start == end_of(before) && (after.is_reverse ? after.to == before.from : after.from == before.to); +} + +/// The best chain that ends at an anchor. +struct ChainEnd { + /// The number of feature bases in the chain. + size_t score = 0; + /// The index of the chain's previous anchor, or NO_ANCHOR. + size_t previous = NO_ANCHOR; +}; + +/** + * Find the chain ending at each anchor that holds the most feature bases. + * + * The anchors are those of one feature, of the given length, sorted by their + * start on the path. + */ +std::vector best_chains(std::span anchors, size_t length) { + std::vector ends(anchors.size()); + for (size_t i = 0; i < anchors.size(); i++) { + const Anchor& anchor = anchors[i]; + size_t bases = anchor.to - anchor.from; + ends[i].score = bases; + for (size_t j = i; j-- > 0;) { + const Anchor& before = anchors[j]; + if (before.start + 2 * length < anchor.start) { + // This anchor and all earlier ones end more than the + // feature's length back. + break; + } + if (can_follow(before, anchor, length) && ends[j].score + bases > ends[i].score) { + ends[i] = { ends[j].score + bases, j }; + } + } + } + return ends; +} + +/** + * Return the place on the path of a chain of anchors. + * + * The chain holds indices into the anchors, in chain order. Anchors that abut + * share a block. + */ +PathOccurrence chain_place(std::span anchors, const std::vector& chain) { + const Anchor& first = anchors[chain.front()]; + PathOccurrence place { first.start, first.start, first.feature, first.is_reverse, 0, {} }; + const Anchor* before = nullptr; + for (size_t i : chain) { + const Anchor& anchor = anchors[i]; + if (before != nullptr && abuts(*before, anchor)) { + place.blocks.back().second = end_of(anchor); + } else { + place.blocks.emplace_back(anchor.start, end_of(anchor)); + } + place.end = end_of(anchor); + place.matched += anchor.to - anchor.from; + before = &anchor; + } + return place; +} + +/** + * Chain the anchors of one feature, of the given length, and add each chain + * that holds at least min_match of the feature's bases to the places. + * + * The anchors are sorted by their start on the path. Chains are taken best first, and each + * anchor goes in one chain, so a later chain stops where it reaches an anchor + * already taken. + */ +void take_chains(std::span anchors, size_t length, double min_match, + std::vector& places) { + std::vector ends = best_chains(anchors, length); + std::vector order(anchors.size()); + std::iota(order.begin(), order.end(), 0); + std::stable_sort(order.begin(), order.end(), [&](size_t a, size_t b) { + return ends[a].score > ends[b].score; + }); + std::vector used(anchors.size(), false); + for (size_t last : order) { + std::vector chain; + for (size_t i = last; i != NO_ANCHOR && !used[i]; i = ends[i].previous) { + chain.push_back(i); + used[i] = true; + } + if (chain.empty()) { + continue; + } + std::reverse(chain.begin(), chain.end()); + PathOccurrence place = chain_place(anchors, chain); + if (static_cast(place.matched) >= min_match * static_cast(length)) { + places.push_back(std::move(place)); + } + } +} + +} + +std::vector chain_anchors(std::vector anchors, + const std::vector& features, double min_match) { + std::sort(anchors.begin(), anchors.end(), [](const Anchor& a, const Anchor& b) { + return std::tie(a.feature, a.start, a.from) < std::tie(b.feature, b.start, b.from); + }); + std::vector result; + auto group = anchors.begin(); + while (group != anchors.end()) { + size_t feature = group->feature; + auto group_end = std::find_if(group, anchors.end(), [&](const Anchor& anchor) { + return anchor.feature != feature; + }); + take_chains(std::span(group, group_end), features[feature].interval.length, min_match, result); + group = group_end; + } + return result; +} + +//------------------------------------------------------------------------------ + +namespace { + +/** + * Search each GBWT path whose name passes the filter with find(steps, coords, + * places), and call visit(name, places) with the places sorted by start. + * + * The coords hold the coordinate of each step on the full path. Paths are + * searched in batches with OpenMP threads. Each batch is reported in path + * order before the next starts. + */ +void sweep_paths(const gbwtgraph::GBZ& gbz, const std::function& include, + const std::function&, + std::vector&)>& find, + const std::function&)>& visit) { + const GBWTGraph& graph = gbz.graph; + struct PathStart { + gbwt::size_type id; + std::string name; + uint64_t start; + }; + std::vector paths; + for_each_path_start(gbz.index, [&](gbwt::size_type path_id, const std::string& name, uint64_t start) { + if (include(name)) { + paths.emplace_back(path_id, name, start); + } + }); + + size_t batch_size = 4 * omp_get_max_threads(); + std::vector> found(batch_size); + for (size_t batch_start = 0; batch_start < paths.size(); batch_start += batch_size) { + size_t batch_end = std::min(batch_start + batch_size, paths.size()); + #pragma omp parallel for schedule(dynamic, 1) + for (size_t i = batch_start; i < batch_end; i++) { + const PathStart& path = paths[i]; + gbwt::vector_type steps = gbz.index.extract(gbwt::Path::encode(path.id, false)); + std::vector coords { path.start }; + for (gbwt::node_type node : steps) { + coords.push_back(coords.back() + graph.get_length(GBWTGraph::node_to_handle(node))); + } + std::vector& places = found[i - batch_start]; + places.clear(); + find(steps, coords, places); + std::sort(places.begin(), places.end(), [](const PathOccurrence& a, const PathOccurrence& b) { + return std::tie(a.start, a.end, a.feature, a.is_reverse) < std::tie(b.start, b.end, b.feature, b.is_reverse); + }); + } + for (size_t i = batch_start; i < batch_end; i++) { + visit(paths[i].name, found[i - batch_start]); + } + } +} + +} + +void for_each_path_occurrences(const gbwtgraph::GBZ& gbz, const std::vector& features, + const std::function& include, + const std::function&)>& visit) { + FeatureMatcher matcher(gbz.graph, features); + sweep_paths(gbz, include, [&](const gbwt::vector_type& steps, const std::vector& coords, + std::vector& places) { + matcher.for_each_match(steps, [&](size_t feature, size_t step, bool is_reverse) { + const WalkInterval& interval = features[feature].interval; + uint64_t walk_start = coords[step]; + uint64_t walk_end = coords[step + interval.walk.size()]; + uint64_t start = is_reverse ? walk_end - interval.offset - interval.length : walk_start + interval.offset; + uint64_t end = start + interval.length; + places.push_back({ start, end, feature, is_reverse, interval.length, { { start, end } } }); + }); + }, visit); +} + +void for_each_path_liftover(const gbwtgraph::GBZ& gbz, const std::vector& features, double min_match, + const std::function& include, + const std::function&)>& visit) { + FeatureNodeIndex index(gbz.graph, features); + sweep_paths(gbz, include, [&](const gbwt::vector_type& steps, const std::vector& coords, + std::vector& places) { + std::vector anchors; + index.for_each_anchor(steps, coords, [&](const FeatureNodeIndex::Anchor& anchor) { + anchors.push_back(anchor); + }); + places = chain_anchors(std::move(anchors), features, min_match); + }, visit); +} + +} diff --git a/src/feature_lift.hpp b/src/feature_lift.hpp new file mode 100644 index 0000000000..de8b39959e --- /dev/null +++ b/src/feature_lift.hpp @@ -0,0 +1,381 @@ +#ifndef VG_FEATURE_LIFT_HPP_INCLUDED +#define VG_FEATURE_LIFT_HPP_INCLUDED + +/** + * \file feature_lift.hpp + * + * Lifting features from linear sequences to the graph, the gref and the paths. + * + * A feature is a BED record on a linear sequence. The sequence is a cover + * sequence, which is a gref path or a tig of vg unitigs (see unitigs.hpp), or + * any path in the GBZ. Each of these sequences spells a walk in the graph, so + * a feature is a WalkInterval. The interval has a gref placement, and it has + * places on the paths that visit it. + * + * A place is exact when a path visits the whole walk. Exact places suit short + * features found by scanning a cover. Every window of at most k bases of a + * haplotype lies within some cover sequence, so a tool that looks at no more + * than k bases at a time finds each feature on every haplotype. Cover + * sequences overlap by fewer than k bases, so a window shorter than k can lie + * within several of them. Its copies lift to the same interval and are kept + * once. + * + * A place is a liftover when a path visits enough of the walk's bases, in + * order. Liftovers suit long features, such as genes, which haplotypes rarely + * share whole. + * + * Each feature also gets a path coordinate (see PathCoordinate), which names it + * exactly as an interval on one path that carries it. + */ + +#include "gref.hpp" +#include "walk_interval.hpp" + +#include +#include +#include +#include +#include +#include + +namespace vg { + +/// Index of the strand in the fields of a feature. The fields start at the 4th +/// BED column, so this is the 6th. +constexpr size_t BED_STRAND_FIELD = 2; + +/** + * A feature in linear coordinates. + * + * It covers bases [start, end) of the named sequence. It reads them in reverse + * if its strand is "-". + */ +struct BedFeature { + std::string sequence; + size_t start = 0; + size_t end = 0; + /// BED columns from the 4th on: name, score, strand and any more. + std::vector fields; +}; + +/** + * Read the features in a BED file, skipping comments and track and browser lines. + * + * @throws std::runtime_error on a malformed line. + */ +std::vector read_bed_features(std::istream& in); + +/** + * A feature's path coordinate: its interval on the first path, in priority + * order, that carries it whole. + * + * Gref paths come first in the order, so a feature that lies along one is + * named there. A feature off the gref is named on the first reference or + * assembly path that carries it (see PathPriority). Paths keep their + * coordinates when a graph is clipped, filtered or sampled, so these names + * hold across such graphs while the naming path is in them. No two graph + * intervals share a path coordinate. + */ +struct PathCoordinate { + /// The path, named as for_each_path_start() names it. + std::string path; + /// Place of the path in the priority order. + size_t rank = 0; + /// Coordinates on the full path. + uint64_t start = 0; + uint64_t end = 0; + /// True if the feature reads the path in reverse. + bool is_reverse = false; + + /// Return true if this coordinate comes first in the priority order. + bool operator<(const PathCoordinate& other) const; + /// Return the coordinate as PATH:START-END:STRAND, 1-based and inclusive. + std::string text() const; +}; + +/** + * A feature in the graph. + * + * It is a walk interval and the BED fields that describe it. A stranded + * feature reads along its interval, and its strand field is "+". A feature + * without a strand has "." there, or no strand field. + */ +struct GraphFeature { + WalkInterval interval; + std::vector fields; + /// Where the interval lies on the gref. Empty text without a gref. + GrefPlacement placement; + /// Number of visits to the interval, in either orientation, by paths other + /// than gref paths. + size_t support = 0; + /// The path coordinate. make_graph_feature() gives one to a feature along + /// a gref path; others get theirs from the paths that carry them. + std::optional coordinate; + + /// Return true if the feature has a strand. + bool is_stranded() const; +}; + +/** + * Make the graph feature for an interval of a linear sequence and the BED + * fields of a feature on it. + * + * A stranded feature is oriented along its strand, which reverses the interval + * if the strand is "-". A feature without a strand takes the orientation with + * the most bases on forward gref strands. On a tie, it takes the orientation + * whose walk sorts first. The gref may be null. A feature along one gref path + * gets its path coordinate there. + * + * The nodes on either side of the interval, on the walk it comes from, help + * place it at the ends of alt contigs, as GrefIndex::place() says. + */ +GraphFeature make_graph_feature(const gbwtgraph::GBWTGraph& graph, const GrefIndex* gref, + WalkInterval interval, std::vector fields, + std::optional before = std::nullopt, + std::optional after = std::nullopt); + +/** + * The priority order of the paths that name features off the gref. + * + * Paths whose names start with the given prefixes come first, in the order of + * the prefixes. Then come reference paths, and then the rest. Each group is in + * name order. All ranks come after those of the gref paths. + */ +class PathPriority { +public: + /// Rank the paths of the GBZ other than gref paths, starting at first_rank. + PathPriority(const gbwtgraph::GBZ& gbz, const std::vector& prefixes, + const std::string& gref_prefix, size_t first_rank); + + /// Return the rank of the path with this name, as for_each_path_start() + /// names it. Requires a ranked path. + size_t rank(const std::string& name) const; + +private: + std::unordered_map ranks; +}; + +/** + * A directory of the sequences of a cover. + * + * It knows the walk that each gref path and each loaded tig spells. + */ +class Cover { +public: + /// Make a cover of the gref paths of the index, which may be null, with no tigs yet. + Cover(const gbwtgraph::GBWTGraph& graph, const GrefIndex* gref); + + /** + * Load the tigs of a GAF from vg unitigs for which wanted(name) is true. + * + * @throws std::runtime_error on a malformed line or a repeated tig name. + */ + void add_tigs(std::istream& gaf, const std::function& wanted); + + /// Return true if the name is that of a gref path. + bool is_gref_path(const std::string& name) const; + /// Return the length of the named sequence, or nothing if the cover lacks it. + std::optional sequence_length(const std::string& name) const; + /// Return bases [start, end) of the named sequence. Requires the sequence + /// to be in the cover, and start < end <= its length. + WalkInterval interval(const std::string& name, size_t start, size_t end) const; + +private: + const gbwtgraph::GBWTGraph& graph; + /// Spells the gref paths. Null without a gref. + const GrefIndex* gref; + /// The walk interval of each loaded tig, by name. + std::unordered_map tigs; +}; + +/** + * A directory of paths in a GBZ, by the names that for_each_path_start() gives. + * + * A path can be split into fragments, such as the pieces of an assembly contig + * in a clipped graph. The directory finds the walk of a stretch of the full + * path in the fragment that holds it. + */ +class PathDirectory { +public: + /// Load the fragments of each path whose name passes the filter. + PathDirectory(const gbwtgraph::GBZ& gbz, const std::function& wanted); + + /// Return true if the directory has fragments of the named path. + bool has_path(const std::string& name) const; + /// Return bases [start, end) of the named full path, or nothing if no + /// fragment holds them all. Requires start < end. + std::optional interval(const std::string& name, uint64_t start, uint64_t end) const; + +private: + struct Fragment { + /// Where the fragment starts on the full path. + uint64_t start; + PathSteps steps; + }; + /// The fragments of each path, by name, sorted by start. + std::unordered_map> fragments; +}; + +/** + * Sort the features along the gref, then by interval and fields, and merge the + * copies of each feature. + * + * Copies have the same interval and the same fields. + */ +void sort_and_merge_features(std::vector& features); + +/// A place where a path visits a feature. +struct PathOccurrence { + /// Coordinates on the full path of the first base of the place and of the + /// base after its last. + uint64_t start; + uint64_t end; + /// Index of the feature. + size_t feature; + /// True if the path reads the feature in reverse. + bool is_reverse; + /// Number of feature bases that the path visits at this place. + size_t matched; + /// The stretches of the place that hold feature bases, as [start, end) + /// coordinates on the full path, in increasing order. + std::vector> blocks; +}; + +/** + * A matcher that finds where paths visit the whole walks of graph features. + * + * It indexes the features by the first node of their walks, read either way. + * At each step of a path, it checks the features indexed at that node against + * the steps that follow. + */ +class FeatureMatcher { +public: + /// Index the features, which must outlive the matcher. + FeatureMatcher(const gbwtgraph::GBWTGraph& graph, const std::vector& features); + + /** + * Call visit(feature, step, is_reverse) for each place where the path + * visits the walk of a feature. + * + * The feature is its index in the features. The step is the path's first + * step on the walk. The flag is_reverse is true if the path reads the walk + * in reverse. + */ + void for_each_match(const gbwt::vector_type& path, + const std::function& visit) const; + +private: + /// Return true if the path, from the step on, visits the walk, read in + /// reverse if is_reverse. + bool visits_walk(const gbwt::vector_type& path, size_t step, const std::vector& walk, + bool is_reverse) const; + + const gbwtgraph::GBWTGraph& graph; + const std::vector& features; + handlegraph::nid_t min_id = 0; + /// Where the entries of each node start, by node ID minus min_id, followed + /// by the number of entries. + std::vector starts; + /// The walks that start on each node, grouped by node. Each is a feature + /// index shifted left by one, with is_reverse in the low bit. A walk read + /// in reverse starts on its last node. + std::vector entries; +}; + +/** + * An index of the nodes on the walks of graph features. + * + * It turns each step of a path onto such a node into an anchor, which is the + * stretch of the feature on that node at its place on the path. + */ +class FeatureNodeIndex { +public: + /// A stretch of a feature at a place on a path. + struct Anchor { + /// Index of the feature. + size_t feature; + /// Bases [from, to) of the feature. + size_t from; + size_t to; + /// Path coordinate of the first base of the stretch on the path. + uint64_t start; + /// True if the path reads the stretch in reverse. + bool is_reverse; + }; + + /// Index the features, which must outlive the index. + FeatureNodeIndex(const gbwtgraph::GBWTGraph& graph, const std::vector& features); + + /// Call visit(anchor) for each step of the path onto a node of a feature + /// walk, in path order. The path coordinate of each step is in coords. + void for_each_anchor(const gbwt::vector_type& path, const std::vector& coords, + const std::function& visit) const; + +private: + /// A stretch of a feature on one node of its walk. + struct Entry { + size_t feature; + /// Bases [from, to) of the feature. + size_t from; + size_t to; + /// Offsets of those bases on the node, as the walk reads it. + size_t node_from; + size_t node_to; + /// True if the walk reads the node in reverse. + bool is_reverse; + }; + + const gbwtgraph::GBWTGraph& graph; + handlegraph::nid_t min_id = 0; + /// Where the entries of each node start, by node ID minus min_id, followed + /// by the number of entries. + std::vector starts; + /// The stretches of the features on each node, grouped by node. + std::vector entries; +}; + +/** + * Chain the anchors that one path gives a set of features into liftovers. + * + * A chain is a run of anchors of one feature that the path reads in the same + * orientation, in the order of the feature's bases. Each anchor ends before the + * next one starts on the path, with a gap of at most the feature's length. + * + * Chains are taken best first. The first chain holds the most feature bases, + * the next one holds the most among the anchors left, and so on. Return each + * chain that holds at least min_match of its feature's bases, as a place on + * the path. + */ +std::vector chain_anchors(std::vector anchors, + const std::vector& features, double min_match); + +/** + * Find where each path visits the whole walks of the features. + * + * For each GBWT path whose name passes the filter, call visit(name, occurrences) + * with the exact places where the path visits the features, sorted by start. + * The name leaves out any subrange or phase block, and the coordinates are on + * the full path. For a haplotype path cut from an assembly, that is the + * contig (see for_each_path_start()). + * + * Paths are searched with OpenMP threads. One thread at a time calls visit(), + * in path order. + */ +void for_each_path_occurrences(const gbwtgraph::GBZ& gbz, const std::vector& features, + const std::function& include, + const std::function&)>& visit); + +/** + * Find where each path visits at least min_match of the bases of each + * feature, in order. + * + * This works like for_each_path_occurrences(), with the places found by + * chain_anchors() in place of exact ones. + */ +void for_each_path_liftover(const gbwtgraph::GBZ& gbz, const std::vector& features, double min_match, + const std::function& include, + const std::function&)>& visit); + +} + +#endif diff --git a/src/haplotype_depth.cpp b/src/haplotype_depth.cpp new file mode 100644 index 0000000000..69b457970b --- /dev/null +++ b/src/haplotype_depth.cpp @@ -0,0 +1,353 @@ +#include "haplotype_depth.hpp" + +#include "feature_lift.hpp" +#include "gref.hpp" + +#include +#include + +#include +#include +#include + +namespace vg { + +using gbwtgraph::GBWTGraph; +using handlegraph::handle_t; +using handlegraph::nid_t; +using handlegraph::PathSense; + +//------------------------------------------------------------------------------ + +PathRegions::PathRegions(std::istream& bed) { + for (BedFeature& feature : read_bed_features(bed)) { + regions[feature.sequence].emplace_back(feature.start, feature.end); + } + for (auto& [name, list] : regions) { + std::sort(list.begin(), list.end()); + size_t kept = 0; + for (const Region& region : list) { + if (kept > 0 && region.first <= list[kept - 1].second) { + list[kept - 1].second = std::max(list[kept - 1].second, region.second); + } else { + list[kept++] = region; + } + } + list.resize(kept); + } +} + +const std::vector* PathRegions::regions_of(const std::string& name) const { + auto found = regions.find(name); + return found == regions.end() ? nullptr : &found->second; +} + +std::vector PathRegions::path_names() const { + std::vector names; + for (const auto& [name, list] : regions) { + names.push_back(name); + } + std::sort(names.begin(), names.end()); + return names; +} + +//------------------------------------------------------------------------------ + +HaplotypeDepth::HaplotypeDepth(const gbwtgraph::GBZ& gbz, const Options& options) { + const GBWTGraph& graph = gbz.graph; + if (graph.get_node_count() == 0) { + return; + } + min_id = graph.min_node_id(); + whole.assign(graph.max_node_id() - min_id + 1, 0); + + // Group the counted paths by haplotype. One haplotype is usually several + // paths: the contigs of an assembly, or their fragments in a clipped graph. + // Counting by haplotype makes a base that two of its paths read count once. + HaplotypePaths by_haplotype = counted_paths(gbz, options.count_references); + haplotypes = by_haplotype.size(); + if (options.regions != nullptr) { + match_regions(by_haplotype, *options.regions); + } + + std::vector*> groups; + for (const auto& [haplotype, paths] : by_haplotype) { + groups.push_back(&paths); + } + // One thread counts each haplotype, so that it can merge the stretches that + // the haplotype's paths read before adding them. + std::vector cut_stretches; + #pragma omp parallel for schedule(dynamic, 1) + for (size_t i = 0; i < groups.size(); i++) { + std::vector stretches; + for (const CountedPath& path : *groups[i]) { + if (options.regions == nullptr) { + add_path_stretches(gbz, path, stretches); + } else if (const std::vector* regions = options.regions->regions_of(path.name)) { + add_region_stretches(gbz, path, *regions, stretches); + } + } + std::vector cut_here; + add_stretches(std::move(stretches), graph, !options.count_visits, cut_here); + #pragma omp critical (haplotype_depth_cuts) + cut_stretches.insert(cut_stretches.end(), cut_here.begin(), cut_here.end()); + } + add_cuts(std::move(cut_stretches)); +} + +HaplotypeDepth::HaplotypePaths HaplotypeDepth::counted_paths(const gbwtgraph::GBZ& gbz, bool count_references) { + HaplotypePaths by_haplotype; + gbwtgraph::sample_name_set reference_samples = gbwtgraph::parse_reference_samples_tag(gbz.index); + for_each_path_start(gbz.index, [&](gbwt::size_type path_id, const std::string& name, uint64_t start) { + if (GrefCover::is_gref_derived(name)) { + return; + } + PathSense sense = gbwtgraph::get_path_sense(gbz.index, path_id, reference_samples); + if (sense == PathSense::GENERIC || (sense == PathSense::REFERENCE && !count_references)) { + return; + } + std::string haplotype = gbwtgraph::get_path_sample_name(gbz.index, path_id, sense) + "#" + + std::to_string(gbwtgraph::get_path_haplotype(gbz.index, path_id, sense)); + by_haplotype[haplotype].emplace_back(path_id, name, start); + }); + return by_haplotype; +} + +void HaplotypeDepth::match_regions(const HaplotypePaths& by_haplotype, const PathRegions& regions) { + std::unordered_set counted_names; + for (const auto& [haplotype, paths] : by_haplotype) { + if (std::none_of(paths.begin(), paths.end(), [&](const CountedPath& path) { + return regions.regions_of(path.name) != nullptr; + })) { + without_regions++; + } + for (const CountedPath& path : paths) { + counted_names.insert(path.name); + } + } + for (const std::string& name : regions.path_names()) { + if (!counted_names.contains(name)) { + unmatched_regions.push_back(name); + } + } +} + +void HaplotypeDepth::add_path_stretches(const gbwtgraph::GBZ& gbz, const CountedPath& path, + std::vector& stretches) { + for (gbwt::node_type node : gbz.index.extract(gbwt::Path::encode(path.id, false))) { + nid_t id = gbwt::Node::id(node); + auto length = static_cast(gbz.graph.get_length(GBWTGraph::node_to_handle(node))); + stretches.push_back({ id, 0, length }); + } +} + +void HaplotypeDepth::add_region_stretches(const gbwtgraph::GBZ& gbz, const CountedPath& path, + const std::vector& regions, + std::vector& stretches) { + uint64_t coord = path.start; + size_t next_region = 0; + for (gbwt::node_type node : gbz.index.extract(gbwt::Path::encode(path.id, false))) { + nid_t id = gbwt::Node::id(node); + auto length = static_cast(gbz.graph.get_length(GBWTGraph::node_to_handle(node))); + // Skip the regions that end before this step. The regions are + // disjoint and sorted, and later steps lie further along the path, so + // no later step needs them. + while (next_region < regions.size() && regions[next_region].second <= coord) { + next_region++; + } + for (size_t r = next_region; r < regions.size() && regions[r].first < coord + length; r++) { + auto from = static_cast(std::max(coord, regions[r].first) - coord); + auto to = static_cast(std::min(coord + length, regions[r].second) - coord); + // A reverse step reads the node from its end. + if (gbwt::Node::is_reverse(node)) { + std::tie(from, to) = std::make_pair(length - to, length - from); + } + stretches.push_back({ id, from, to }); + } + coord += length; + } +} + +void HaplotypeDepth::add_cuts(std::vector cut_stretches) { + std::sort(cut_stretches.begin(), cut_stretches.end(), [](const NodeStretch& a, const NodeStretch& b) { + return a.node < b.node; + }); + for (size_t i = 0; i < cut_stretches.size();) { + std::vector> changes; + size_t j = i; + for (; j < cut_stretches.size() && cut_stretches[j].node == cut_stretches[i].node; j++) { + changes.emplace_back(cut_stretches[j].from, 1); + changes.emplace_back(cut_stretches[j].to, -1); + } + std::sort(changes.begin(), changes.end()); + size_t kept = 0; + for (const auto& change : changes) { + if (kept > 0 && changes[kept - 1].first == change.first) { + changes[kept - 1].second += change.second; + } else { + changes[kept++] = change; + } + } + changes.resize(kept); + cuts.try_emplace(cut_stretches[i].node, std::move(changes)); + i = j; + } +} + +void HaplotypeDepth::add_stretches(std::vector stretches, const GBWTGraph& graph, bool merge, + std::vector& cut_stretches) { + if (merge) { + // A haplotype that reads a base more than once counts once there. + std::sort(stretches.begin(), stretches.end(), [](const NodeStretch& a, const NodeStretch& b) { + return std::tie(a.node, a.from, a.to) < std::tie(b.node, b.from, b.to); + }); + size_t kept = 0; + for (const NodeStretch& stretch : stretches) { + NodeStretch* last = kept > 0 ? &stretches[kept - 1] : nullptr; + if (last != nullptr && last->node == stretch.node && stretch.from <= last->to) { + last->to = std::max(last->to, stretch.to); + } else { + stretches[kept++] = stretch; + } + } + stretches.resize(kept); + } + for (const NodeStretch& stretch : stretches) { + if (stretch.from == 0 && stretch.to == graph.get_length(graph.get_handle(stretch.node))) { + #pragma omp atomic + whole[stretch.node - min_id]++; + } else { + cut_stretches.push_back(stretch); + } + } +} + +uint64_t HaplotypeDepth::bases_read(nid_t id, uint32_t length) const { + if (whole.empty() || id < min_id || size_t(id - min_id) >= whole.size()) { + return 0; + } + uint64_t total = uint64_t(whole[id - min_id]) * length; + if (auto found = cuts.find(id); found != cuts.end()) { + // The count of the stretch after each change holds until the next one. + int64_t depth = 0; + uint32_t at = 0; + for (const auto& [offset, change] : found->second) { + total += uint64_t(depth) * (offset - at); + depth += change; + at = offset; + } + } + return total; +} + +namespace { + +/** + * A builder of the runs of equal depth along a path. + * + * It takes stretches of bases in order along the path. It joins a stretch to + * the run before it if they touch and have the same depth, and reports each + * run once it ends. + */ +class RunBuilder { +public: + /// Report each run with visit(start, end, depth). + explicit RunBuilder(const std::function& visit) : visit(visit) { + } + + /// Add bases [start, end) of the full path, at the given depth. + void extend(uint64_t start, uint64_t end, uint32_t depth) { + if (start == end) { + return; + } + if (open && depth == run_depth && start == run_end) { + run_end = end; + return; + } + finish(); + open = true; + run_start = start; + run_end = end; + run_depth = depth; + } + + /// Report the run in progress, if any. + void finish() { + if (open) { + visit(run_start, run_end, run_depth); + open = false; + } + } + +private: + const std::function& visit; + /// True if a run is in progress. + bool open = false; + uint64_t run_start = 0; + uint64_t run_end = 0; + uint32_t run_depth = 0; +}; + +/// A stretch [from, to) of a node, in its forward orientation, at one depth. +struct DepthPiece { + uint32_t from; + uint32_t to; + uint32_t depth; +}; + +/** + * Return the stretches of a cut node of the given length between the changes + * of its depth, in its forward orientation. + * + * The depth starts at base_depth and changes at each offset in changes. + */ +std::vector depth_pieces(const std::vector>& changes, uint32_t base_depth, + uint32_t length) { + std::vector pieces; + int64_t depth = base_depth; + uint32_t at = 0; + for (const auto& [offset, change] : changes) { + if (offset > at) { + pieces.push_back({ at, offset, uint32_t(depth) }); + } + depth += change; + at = offset; + } + if (at < length) { + pieces.push_back({ at, length, uint32_t(depth) }); + } + return pieces; +} + +} + +void HaplotypeDepth::for_each_run(const PathSteps& path, uint64_t offset, + const std::function& visit) const { + RunBuilder runs(visit); + const std::vector& steps = path.steps(); + for (size_t i = 0; i < steps.size(); i++) { + uint64_t start = offset + steps[i].start; + auto length = static_cast((i + 1 < steps.size() ? steps[i + 1].start : path.length()) - steps[i].start); + gbwt::node_type node = GBWTGraph::handle_to_node(steps[i].handle); + nid_t id = gbwt::Node::id(node); + uint32_t base_depth = whole.empty() ? 0 : whole[id - min_id]; + auto found = cuts.find(id); + if (found == cuts.end()) { + runs.extend(start, start + length, base_depth); + continue; + } + std::vector pieces = depth_pieces(found->second, base_depth, length); + if (gbwt::Node::is_reverse(node)) { + for (auto piece = pieces.rbegin(); piece != pieces.rend(); ++piece) { + runs.extend(start + length - piece->to, start + length - piece->from, piece->depth); + } + } else { + for (const DepthPiece& piece : pieces) { + runs.extend(start + piece.from, start + piece.to, piece.depth); + } + } + } + runs.finish(); +} + +} diff --git a/src/haplotype_depth.hpp b/src/haplotype_depth.hpp new file mode 100644 index 0000000000..3cde61e5ae --- /dev/null +++ b/src/haplotype_depth.hpp @@ -0,0 +1,160 @@ +#ifndef VG_HAPLOTYPE_DEPTH_HPP_INCLUDED +#define VG_HAPLOTYPE_DEPTH_HPP_INCLUDED + +/** + * \file haplotype_depth.hpp + * + * How many haplotypes read each base of a graph, read off along any path. + */ + +#include "walk_interval.hpp" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace vg { + +/** + * A set of regions on the full paths of a graph, by path name. + * + * A path is named as for_each_path_start() names it, so a region on a + * haplotype is in coordinates of its assembly contig. Each path's regions are + * sorted and merged. + */ +class PathRegions { +public: + /// A stretch [first, second) of a full path. + using Region = std::pair; + + /** + * Read the regions in a BED file. Columns past the 3rd are ignored. + * + * @throws std::runtime_error on a malformed line. + */ + explicit PathRegions(std::istream& bed); + + /// Return the regions of the named path, sorted and merged, or null if it has none. + const std::vector* regions_of(const std::string& name) const; + /// Return the names of the paths that have regions. + std::vector path_names() const; + +private: + std::unordered_map> regions; +}; + +/** + * A tally of how many haplotypes read each base of each node of a graph. + * + * It counts distinct haplotypes, by sample and haplotype number, or the visits + * they make, which exceed the haplotypes where one reads a node more than + * once. Haplotype paths count, and reference paths if asked. Gref paths and + * generic paths never count. With regions, a path counts only its bases + * inside its regions, so a path without regions counts nothing. + * + * Most nodes are read whole by every haplotype that reads them and hold one + * count. A node that regions cut holds a count for each stretch of its bases. + */ +class HaplotypeDepth { +public: + struct Options { + /// Count visits instead of distinct haplotypes. + bool count_visits = false; + /// Count reference paths as haplotypes too. + bool count_references = false; + /// Count only the bases of each path inside its regions. Null counts every base. + const PathRegions* regions = nullptr; + }; + + /// Count the haplotypes of the GBZ, with OpenMP threads. + HaplotypeDepth(const gbwtgraph::GBZ& gbz, const Options& options); + + /** + * Call visit(start, end, depth) for each maximal run of bases of equal + * depth along the path, in order, including runs of depth 0. + * + * Coordinates are on the full path, which starts offset bases before the + * path's first base. + */ + void for_each_run(const PathSteps& path, uint64_t offset, + const std::function& visit) const; + + /// Return the bases that the counted haplotypes read on the node: its + /// depth summed over its bases. + uint64_t bases_read(handlegraph::nid_t id, uint32_t length) const; + + /// Return the number of haplotypes counted, with or without regions. + size_t haplotype_count() const { return haplotypes; } + /// Return the number of counted haplotypes that have no regions, and so add nothing. + size_t haplotypes_without_regions() const { return without_regions; } + /// Return the names of paths with regions that are not counted paths in the graph. + const std::vector& uncounted_region_paths() const { return unmatched_regions; } + +private: + /// Bases [from, to) of a node, in its forward orientation. + struct NodeStretch { + handlegraph::nid_t node; + uint32_t from; + uint32_t to; + }; + /// A path that counts toward the depth. + struct CountedPath { + gbwt::size_type id; + /// The name of the full path, as for_each_path_start() gives it. + std::string name; + /// Where the path starts on the full path. + uint64_t start; + }; + /// The counted paths of each haplotype, by sample name and haplotype + /// number, joined with "#". + using HaplotypePaths = std::map>; + + /// Return the paths of the GBZ that count, by haplotype. Haplotype paths + /// count, and reference paths do too if count_references is set. + static HaplotypePaths counted_paths(const gbwtgraph::GBZ& gbz, bool count_references); + /// Count the haplotypes that have no regions, and list the paths with + /// regions that are not counted paths. + void match_regions(const HaplotypePaths& haplotypes, const PathRegions& regions); + /// Append the stretches of nodes that the path reads. + static void add_path_stretches(const gbwtgraph::GBZ& gbz, const CountedPath& path, + std::vector& stretches); + /// Append the stretches of nodes that the path reads inside its regions, + /// which are sorted and disjoint. + static void add_region_stretches(const gbwtgraph::GBZ& gbz, const CountedPath& path, + const std::vector& regions, + std::vector& stretches); + /** + * Add one to the count of each base in the stretches that one haplotype reads. + * + * A stretch of a whole node adds to its count here. A stretch of part of a + * node is appended to cut_stretches. With merge, overlapping stretches of a + * node count once, so the count is of haplotypes and not of visits. + */ + void add_stretches(std::vector stretches, const gbwtgraph::GBWTGraph& graph, bool merge, + std::vector& cut_stretches); + /// Record where the count of each cut node changes along its bases, from + /// the stretches of the cut nodes that all the haplotypes read. + void add_cuts(std::vector cut_stretches); + + handlegraph::nid_t min_id = 0; + /// The count of each node read whole, by node ID minus min_id. + std::vector whole; + /// For each node that regions cut, where its count changes along its + /// forward bases: the offset and the change, sorted by offset. + std::unordered_map>> cuts; + + size_t haplotypes = 0; + size_t without_regions = 0; + std::vector unmatched_regions; +}; + +} + +#endif diff --git a/src/nested_variants.cpp b/src/nested_variants.cpp new file mode 100644 index 0000000000..4034d66dfd --- /dev/null +++ b/src/nested_variants.cpp @@ -0,0 +1,754 @@ +/** + * \file nested_variants.cpp + * Variants on gref alt contigs, see nested_variants.hpp. + */ + +#include "nested_variants.hpp" +#include "sorted_vcf_writer.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +namespace vg { + +using handlegraph::handle_t; +using handlegraph::PathMetadata; +using gbwtgraph::GBWTGraph; + +//------------------------------------------------------------------------------ + +uint64_t NestedVariant::ref_start() const { + return pos - 1; +} + +uint64_t NestedVariant::ref_end() const { + return pos - 1 + ref.size(); +} + +std::string NestedVariant::type() const { + if (ref.size() == 1) { + return alt.size() == 1 ? "SNV" : "INS"; + } + return alt.size() == 1 ? "DEL" : "COMPLEX"; +} + +std::string NestedVariant::id(const std::string& contig_name, const GBWTGraph& graph) const { + // A complex variant is as long as its longer allele. Another is as long as + // what it adds or removes, and a SNV is 1 long. + size_t length = std::max(ref.size() > alt.size() ? ref.size() - alt.size() : alt.size() - ref.size(), 1); + if (ref.size() > 1 && alt.size() > 1) { + length = std::max(ref.size(), alt.size()); + } + return contig_name + "-" + std::to_string(pos) + "-" + type() + "-" + walk_text(alt_walk, graph) + "-" + + std::to_string(length); +} + +//------------------------------------------------------------------------------ + +ContigDecomposer::ContigDecomposer(const GBWTGraph& graph, const GrefIndex& gref) : graph(graph), gref(gref) { +} + +bool ContigDecomposer::is_contig(uint32_t path) const { + return path < gref.path_count() && gref.is_alt_contig(path); +} + +std::vector ContigDecomposer::decompose(const std::vector& walk) const { + // Group the walk's visits to each contig into passes. A visit continues + // the contig's latest pass if it reads the contig the same way, further on. + std::vector> groups; + std::unordered_map latest; + for (size_t step = 0; step < walk.size(); step++) { + const GrefIndex::Position& place = gref.find(graph.get_id(walk[step])); + if (place.path == GrefIndex::NO_PATH) { + continue; + } + if (!is_contig(place.path)) { + // The walk is back on the reference, so it has left every contig. + latest.clear(); + continue; + } + Visit visit { step, place.path, graph.get_is_reverse(walk[step]) != place.is_reverse, place.offset, + place.offset + graph.get_length(walk[step]) }; + if (auto found = latest.find(visit.contig); found != latest.end()) { + const Visit& last = groups[found->second].back(); + if (last.is_reverse == visit.is_reverse && + (visit.is_reverse ? visit.end <= last.start : visit.start >= last.end)) { + groups[found->second].push_back(visit); + continue; + } + } + latest[visit.contig] = groups.size(); + groups.push_back({ visit }); + } + + std::vector passes; + passes.reserve(groups.size()); + for (std::vector& visits : groups) { + passes.push_back(make_pass(walk, std::move(visits))); + } + return passes; +} + +/** + * A walk read in one orientation, with its steps numbered that way. + * + * Read in reverse, the walk's last step comes first, with its handle flipped. + */ +class ContigDecomposer::OrientedWalk { +public: + OrientedWalk(const GBWTGraph& graph, const std::vector& walk, bool is_reverse) : + graph(graph), walk(walk), is_reverse(is_reverse) { + } + + /// Return the handle that this reading visits at step i. + handle_t at(size_t i) const { + return is_reverse ? graph.flip(walk[to_stored(i)]) : walk[i]; + } + /// Return the number in the stored walk of step i of this reading. + size_t to_stored(size_t i) const { + return is_reverse ? walk.size() - 1 - i : i; + } + /// Return the number in this reading of step i of the stored walk. + size_t from_stored(size_t i) const { + return is_reverse ? walk.size() - 1 - i : i; + } + +private: + const GBWTGraph& graph; + const std::vector& walk; + bool is_reverse; +}; + +namespace { + +/// Trim the bases that REF and ALT share off the end, then off the start, +/// keeping one base in each, and move POS past the bases trimmed off the start. +void trim_shared_bases(NestedVariant& variant) { + std::string& ref = variant.ref; + std::string& alt = variant.alt; + while (ref.size() >= 2 && alt.size() >= 2 && ref.back() == alt.back()) { + ref.pop_back(); + alt.pop_back(); + } + size_t shared = 0; + while (ref.size() - shared >= 2 && alt.size() - shared >= 2 && ref[shared] == alt[shared]) { + shared++; + } + ref.erase(0, shared); + alt.erase(0, shared); + variant.pos += shared; +} + +} + +ContigPass ContigDecomposer::make_pass(const std::vector& walk, std::vector visits) const { + ContigPass pass; + pass.contig = visits.front().contig; + pass.first_step = visits.front().step; + pass.last_step = visits.back().step; + uint64_t length = gref.path_length(pass.contig); + + // Read the walk in the contig's orientation, with steps numbered that way. + bool is_reverse = visits.front().is_reverse; + OrientedWalk oriented(graph, walk, is_reverse); + if (is_reverse) { + std::reverse(visits.begin(), visits.end()); + for (Visit& visit : visits) { + visit.step = oriented.from_stored(visit.step); + } + } + // Ask the index where the walk comes past the start of the contig, and + // where it rejoins the graph past the end. The index searches the walk as + // stored, so a pass that reads the contig in reverse asks about the other + // end, and gets back a step to renumber the contig's way. + auto step_before_start = [&](size_t i) { + std::optional found = + is_reverse ? gref.step_past_contig_end(walk, oriented.to_stored(i), pass.contig, true) + : gref.step_before_contig_start(walk, i, pass.contig, false); + return found ? std::optional(oriented.from_stored(*found)) : std::nullopt; + }; + auto step_past_end = [&](size_t i) { + std::optional found = + is_reverse ? gref.step_before_contig_start(walk, oriented.to_stored(i), pass.contig, true) + : gref.step_past_contig_end(walk, i, pass.contig, false); + return found ? std::optional(oriented.from_stored(*found)) : std::nullopt; + }; + + // A pass that joins the contig after its start replaces the contig's head + // if the walk comes past the start. + const Visit& head = visits.front(); + pass.start = head.start; + std::optional before; + if (head.start > 0) { + before = step_before_start(head.step); + } + if (before.has_value()) { + add_variant(pass, oriented, 0, head.start, *before, head.step); + pass.start = 0; + } + // Between two visits, the walk's nodes replace the contig's, unless the + // two nodes follow each other on both. + for (size_t k = 0; k + 1 < visits.size(); k++) { + const Visit& left = visits[k]; + const Visit& right = visits[k + 1]; + if (left.end < right.start || right.step > left.step + 1) { + add_variant(pass, oriented, left.end, right.start, left.step, right.step); + } + } + // A pass that leaves the contig before its end replaces the contig's tail + // if the walk rejoins the graph past the end. + const Visit& tail = visits.back(); + pass.end = tail.end; + std::optional past; + if (tail.end < length) { + past = step_past_end(tail.step); + } + if (past.has_value()) { + add_variant(pass, oriented, tail.end, length, tail.step, *past); + pass.end = length; + } + return pass; +} + +void ContigDecomposer::add_variant(ContigPass& pass, const OrientedWalk& walk, uint64_t ref_start, uint64_t ref_end, + size_t left, size_t right) const { + NestedVariant variant; + variant.contig = pass.contig; + std::string ref_bases; + std::string alt_bases; + variant.ref_walk.push_back(walk.at(left)); + if (ref_start < ref_end) { + WalkInterval replaced = gref.interval(pass.contig, ref_start, ref_end); + ref_bases = replaced.sequence(graph); + variant.ref_walk.insert(variant.ref_walk.end(), replaced.walk.begin(), replaced.walk.end()); + } + variant.ref_walk.push_back(walk.at(right)); + variant.alt_walk.push_back(walk.at(left)); + for (size_t i = left + 1; i < right; i++) { + variant.alt_walk.push_back(walk.at(i)); + alt_bases += graph.get_sequence(walk.at(i)); + } + variant.alt_walk.push_back(walk.at(right)); + if (ref_bases == alt_bases) { + // The walk spells the contig on other nodes. It has no variant + // here, but it still reads this part of the contig. + return; + } + + // Pad with the base before, or after at the start of the contig. + if (ref_start > 0) { + char before = gref.base(pass.contig, ref_start - 1); + variant.ref = before + ref_bases; + variant.alt = before + alt_bases; + variant.pos = ref_start; + } else { + char after = gref.base(pass.contig, ref_end); + variant.ref = ref_bases + after; + variant.alt = alt_bases + after; + variant.pos = 1; + } + trim_shared_bases(variant); + pass.variants.push_back(std::move(variant)); +} + +//------------------------------------------------------------------------------ + +namespace { + +/// Return the level of the contig that a pass of a walk reads. It is one more +/// than the number of the walk's passes along other contigs that enclose the pass. +size_t nesting_level(const std::vector& passes, const ContigPass& pass) { + auto enclosing = std::count_if(passes.begin(), passes.end(), [&](const ContigPass& other) { + return other.contig != pass.contig && other.first_step < pass.first_step && pass.last_step < other.last_step; + }); + return 1 + static_cast(enclosing); +} + +} + +NestedVariantCatalog::NestedVariantCatalog(const VcfCalls& calls, const ContigDecomposer& decomposer, + const GrefIndex& gref, const GBWTGraph& graph) : + allele_readings(calls.alleles.size()) { + // Decompose the walks in parallel, then list each variant once, by its + // contig, position and alleles. + std::vector> passes(calls.alleles.size()); + #pragma omp parallel for schedule(dynamic, 1024) + for (size_t allele = 0; allele < calls.alleles.size(); allele++) { + if (!calls.alleles[allele].walk.empty()) { + passes[allele] = decomposer.decompose(calls.alleles[allele].walk); + } + } + + std::unordered_map contig_of_path; + VariantKeys variant_keys; + for (size_t allele = 0; allele < passes.size(); allele++) { + size_t record = calls.alleles[allele].record; + const VcfSite& site = calls.sites[record]; + for (const ContigPass& pass : passes[allele]) { + size_t level = nesting_level(passes[allele], pass); + auto [found, is_new] = contig_of_path.try_emplace(pass.contig, contig_list.size()); + if (is_new) { + Contig& contig = contig_list.emplace_back(); + contig.path = pass.contig; + contig.name = PathMetadata::parse_locus_name(gref.path_name(pass.contig)); + contig.length = gref.path_length(pass.contig); + contig.level = level; + contig.site_chrom = site.chrom; + contig.site_start = site.pos; + contig.site_end = site.pos + site.ref_length; + contig.last_record = record; + } + size_t contig_index = found->second; + Contig& contig = contig_list[contig_index]; + contig.level = std::min(contig.level, level); + contig.site_start = std::min(contig.site_start, site.pos); + contig.site_end = std::max(contig.site_end, site.pos + site.ref_length); + contig.last_record = std::max(contig.last_record, record); + + // A walk can pass along a contig more than once, as one reader. + Reading& reading = contig.readers[reader_of(allele, contig_index)].passes.emplace_back(); + reading.start = pass.start; + reading.end = pass.end; + for (const NestedVariant& variant : pass.variants) { + reading.variants.push_back(list_variant(variant, contig_index, variant_keys, graph)); + } + } + } + + for (Contig& contig : contig_list) { + std::sort(contig.variants.begin(), contig.variants.end(), [&](size_t a, size_t b) { + const NestedVariant& x = variant_list[a]; + const NestedVariant& y = variant_list[b]; + return std::tie(x.pos, x.ref, x.alt) < std::tie(y.pos, y.ref, y.alt); + }); + } + sort_contigs(); +} + +size_t NestedVariantCatalog::reader_of(size_t allele, size_t contig) { + std::vector>& readings = allele_readings[allele]; + if (auto same = std::find_if(readings.begin(), readings.end(), [&](const std::pair& reading) { + return reading.first == contig; + }); + same != readings.end()) { + return same->second; + } + std::vector& readers = contig_list[contig].readers; + readings.emplace_back(contig, readers.size()); + readers.push_back({ allele, {} }); + return readings.back().second; +} + +size_t NestedVariantCatalog::list_variant(const NestedVariant& variant, size_t contig, VariantKeys& keys, + const GBWTGraph& graph) { + auto [entry, is_new] = keys.try_emplace(std::make_tuple(variant.contig, variant.pos, variant.ref, variant.alt), + variant_list.size()); + if (is_new) { + variant_list.push_back(variant); + ids.push_back(variant.id(contig_list[contig].name, graph)); + contig_list[contig].variants.push_back(entry->second); + } + return entry->second; +} + +void NestedVariantCatalog::sort_contigs() { + std::vector order(contig_list.size()); + std::iota(order.begin(), order.end(), 0); + std::stable_sort(order.begin(), order.end(), [&](size_t a, size_t b) { + return contig_list[a].last_record < contig_list[b].last_record; + }); + std::vector rank(order.size()); + std::vector sorted; + sorted.reserve(order.size()); + for (size_t i = 0; i < order.size(); i++) { + rank[order[i]] = i; + sorted.push_back(std::move(contig_list[order[i]])); + } + contig_list = std::move(sorted); + for (std::vector>& readings : allele_readings) { + for (auto& [contig, reader] : readings) { + contig = rank[contig]; + } + } +} + +//------------------------------------------------------------------------------ + +NestedGenotyper::NestedGenotyper(const NestedVariantCatalog& catalog, const VcfCalls& calls, + std::vector extra_info, + const std::unordered_map& top_level_snarls, + const GBWTGraph& graph, std::ostream& out) : + catalog(catalog), calls(calls), extra_info(std::move(extra_info)), top_level_snarls(top_level_snarls), graph(graph), + out(out), + carried(catalog.contigs().size()), ploidy(calls.samples.size(), 2) { +} + +std::vector NestedGenotyper::header_lines() const { + std::vector lines { + "##INFO=", + "##INFO=", + "##INFO=", + "##INFO=", + "##INFO=", + "##INFO=", + "##FORMAT=", + "##FORMAT=" + }; + std::istringstream nesting(SortedVCFWriter::nesting_info_headers()); + for (std::string line; std::getline(nesting, line);) { + lines.push_back(line); + } + for (const NestedVariantCatalog::Contig& contig : catalog.contigs()) { + if (!contig.variants.empty()) { + lines.push_back("##contig="); + } + } + return lines; +} + +namespace { + +/// Return the number in a GQ field, or -1 if it holds none. +int parse_quality(std::string_view text) { + int parsed = 0; + auto result = std::from_chars(text.data(), text.data() + text.size(), parsed); + return result.ec == std::errc() && result.ptr == text.data() + text.size() ? parsed : -1; +} + +/// Return the number of the ALT allele that an allele of a GT field calls, or +/// 0 if it calls REF or is missing. A number above the record's ALT count is +/// returned as it stands, for the caller to count and read as REF. +size_t alt_number(std::string_view allele) { + size_t number = 0; + auto parsed = std::from_chars(allele.data(), allele.data() + allele.size(), number); + return parsed.ec == std::errc() ? number : 0; +} + +/** + * A table of the allele that each reader of a contig has at each variant on + * the contig. + * + * A reader has 1, for ALT, at a variant that one of its passes has. It has 0, + * for REF, at a variant whose part of the contig a pass reads without the + * variant. It has -1, for no allele, at the rest. + */ +class ReaderAlleleTable { +public: + ReaderAlleleTable(const NestedVariantCatalog::Contig& contig, const std::vector& variants) { + // The index of each variant on the contig, by its index in the catalog. + std::unordered_map local; + for (size_t i = 0; i < contig.variants.size(); i++) { + local[contig.variants[i]] = i; + } + for (const NestedVariantCatalog::Reader& reader : contig.readers) { + rows.push_back(reader_alleles(reader, contig, variants, local)); + } + } + + /// Return the allele that the reader has at the contig's ith variant. + int8_t allele(size_t reader, size_t i) const { + return rows[reader][i]; + } + +private: + /// Return the alleles of the reader, in the order of the contig's variants. + static std::vector reader_alleles(const NestedVariantCatalog::Reader& reader, + const NestedVariantCatalog::Contig& contig, + const std::vector& variants, + const std::unordered_map& local) { + std::vector row(contig.variants.size(), -1); + for (const NestedVariantCatalog::Reading& reading : reader.passes) { + for (size_t i = 0; i < contig.variants.size(); i++) { + const NestedVariant& variant = variants[contig.variants[i]]; + if (reading.start <= variant.ref_start() && variant.ref_end() <= reading.end) { + row[i] = std::max(row[i], 0); + } + } + for (size_t variant : reading.variants) { + row[local.at(variant)] = 1; + } + } + return row; + } + + /// The alleles of each reader, in the order of the contig's variants. + std::vector> rows; +}; + +} + +/** + * A sample's genotype at a variant, from the walks that it carries along the + * variant's contig. + * + * Each walk that has an allele at the variant fills a haplotype with it. The + * walks of a phased sample fill their own haplotypes, and those of an + * unphased one fill the haplotypes in order of allele. More walks than + * haplotypes, or two on one, conflict, and leave every haplotype missing. The + * genotype's GQ is the lowest GQ of the calls of its walks. + */ +class NestedGenotyper::Genotype { +public: + /// Make a sample's genotype at the contig's ith variant. The sample carries + /// the walks along the contig, and has ploidy haplotypes. + Genotype(const std::vector& walks, const ReaderAlleleTable& table, size_t i, size_t ploidy) : + alleles(std::max(ploidy, 1), '.') { + // The allele of each walk that has one here, with its haplotype. + std::vector> values; + bool phased = true; + for (const Carried& walk : walks) { + if (int8_t allele = table.allele(walk.reader, i); allele >= 0) { + values.emplace_back(walk.haplotype, allele); + phased = phased && walk.haplotype >= 0; + lowest_quality = std::min(lowest_quality, walk.quality); + } + } + has_walks = !values.empty(); + conflict = values.size() > alleles.size(); + if (!conflict && phased) { + place_phased(values); + } else if (!conflict) { + place_unphased(values); + } + if (conflict) { + std::fill(alleles.begin(), alleles.end(), '.'); + } + separator = phased && has_walks && !conflict ? '|' : '/'; + } + + /// Return true if the sample's walks conflict. + bool conflicts() const { + return conflict; + } + /// Return the number of haplotypes with an allele. + size_t called() const { + return alleles.size() - std::count(alleles.begin(), alleles.end(), '.'); + } + /// Return the number of haplotypes with the ALT allele. + size_t alt_called() const { + return std::count(alleles.begin(), alleles.end(), '1'); + } + /// Return the GQ, or -1 if it is unknown. + int quality() const { + return !has_walks || conflict || lowest_quality < 0 ? -1 : lowest_quality; + } + /// Append the genotype as GT, such as 0|1. + void append_gt(std::string& text) const { + for (size_t h = 0; h < alleles.size(); h++) { + if (h > 0) { + text += separator; + } + text += alleles[h]; + } + } + +private: + /// Fill the haplotype of each walk with its allele. Conflict if a walk's + /// haplotype is past the last one, or already filled. + void place_phased(const std::vector>& values) { + for (const auto& [haplotype, allele] : values) { + if (size_t(haplotype) >= alleles.size() || alleles[haplotype] != '.') { + conflict = true; + return; + } + alleles[haplotype] = char('0' + allele); + } + } + /// Fill the first haplotypes with the walks' alleles, REF first. + void place_unphased(std::vector>& values) { + std::sort(values.begin(), values.end(), [](const std::pair& a, const std::pair& b) { + return a.second < b.second; + }); + for (size_t h = 0; h < values.size(); h++) { + alleles[h] = char('0' + values[h].second); + } + } + + /// The allele of each haplotype, as '0', '1', or '.' if missing. + std::vector alleles; + /// The separator of the alleles in GT, which is '|' if phased and '/' if not. + char separator = '/'; + /// True if some walk has an allele here. + bool has_walks = false; + bool conflict = false; + /// The lowest GQ of the calls of the walks, which is -1 for a call without one. + int lowest_quality = std::numeric_limits::max(); +}; + +/** + * The sample columns of a record, with the counts for its INFO. + * + * It takes the genotypes of the samples in column order. It keeps the columns + * with GT alone, and with GT and GQ. A record whose calls have no GQ gets the + * first. + */ +class NestedGenotyper::SampleColumns { +public: + /// Add the genotype of the next sample. + void add(const Genotype& genotype, std::string_view sample) { + size_t sample_start = genotypes.size(); + genotypes += '\t'; + genotype.append_gt(genotypes); + allele_count += genotype.called(); + alt_count += genotype.alt_called(); + if (genotype.called() > 0) { + with_data++; + } + if (genotype.conflicts()) { + conflicted.append(conflicted.empty() ? "" : ",").append(sample); + } + genotypes_with_quality.append(genotypes, sample_start).append(":"); + if (genotype.quality() >= 0) { + genotypes_with_quality += std::to_string(genotype.quality()); + has_quality = true; + } else { + genotypes_with_quality += '.'; + } + } + + /// Write INFO/AC, AN, AF and NS, each after a ";". Leave out AF if no + /// haplotype has an allele. + void write_counts(std::ostream& out) const { + out << ";AC=" << alt_count << ";AN=" << allele_count; + if (allele_count > 0) { + out << ";AF=" << static_cast(alt_count) / static_cast(allele_count); + } + out << ";NS=" << with_data; + } + /// Return the names of the samples whose walks conflict, separated by commas. + const std::string& conflicted_samples() const { + return conflicted; + } + /// Write FORMAT and the sample columns, each after a tab. + void write(std::ostream& out) const { + out << (has_quality ? "\tGT:GQ" : "\tGT") << (has_quality ? genotypes_with_quality : genotypes); + } + +private: + /// The sample columns with GT alone, and with GT and GQ. + std::string genotypes; + std::string genotypes_with_quality; + /// True if some genotype has a GQ. + bool has_quality = false; + /// The number of haplotypes with an allele, and with ALT. + size_t allele_count = 0; + size_t alt_count = 0; + /// The number of samples with an allele. + size_t with_data = 0; + std::string conflicted; +}; + +void NestedGenotyper::add_call(size_t record, const VcfCursor& cursor) { + const VcfSite& site = calls.sites[record]; + bool reads_contig = false; + for (size_t i = 0; i < site.alt_count; i++) { + reads_contig |= !catalog.readings(site.first_allele + i).empty(); + } + std::vector genotypes; + std::vector qualities; + if (reads_contig) { + genotypes = cursor.genotypes(); + qualities = cursor.sample_values("GQ"); + } + for (size_t sample = 0; sample < genotypes.size() && sample < calls.samples.size(); sample++) { + int quality = sample < qualities.size() ? parse_quality(qualities[sample]) : -1; + ploidy[sample] = carry_genotype(site, sample, genotypes[sample], quality); + } + + const std::vector& contigs = catalog.contigs(); + while (next_contig < contigs.size() && contigs[next_contig].last_record <= record) { + write_contig(next_contig++); + } +} + +size_t NestedGenotyper::carry_genotype(const VcfSite& site, size_t sample, std::string_view genotype, int quality) { + bool phased = genotype.find('/') == std::string_view::npos; + size_t haplotype = 0; + size_t start = 0; + while (start <= genotype.size()) { + size_t end = std::min(genotype.find_first_of("/|", start), genotype.size()); + // A REF or missing allele adds no walk. + size_t number = alt_number(genotype.substr(start, end - start)); + if (number > site.alt_count) { + // The GT names an allele the record does not have, so there is no + // walk to take. Reading it as REF is a guess, so say how often. + past_end_count++; + number = 0; + } + if (number > 0) { + carry(sample, site.first_allele + number - 1, phased ? int(haplotype) : -1, quality); + } + start = end + 1; + haplotype++; + } + return haplotype; +} + +void NestedGenotyper::carry(size_t sample, size_t allele, int haplotype, int quality) { + for (const auto& [contig, reader] : catalog.readings(allele)) { + if (carried[contig].empty()) { + carried[contig].resize(calls.samples.size()); + } + carried[contig][sample].push_back({ reader, haplotype, quality }); + } +} + +void NestedGenotyper::write_contig(size_t contig_index) { + const NestedVariantCatalog::Contig& contig = catalog.contigs()[contig_index]; + ReaderAlleleTable table(contig, catalog.variants()); + // The walks that each sample carries along the contig. The list is empty + // if no call carried a reader of the contig. + const std::vector>& walks = carried[contig_index]; + const std::vector no_walks; + for (size_t i = 0; i < contig.variants.size(); i++) { + SampleColumns columns; + for (size_t sample = 0; sample < calls.samples.size(); sample++) { + Genotype genotype(walks.empty() ? no_walks : walks[sample], table, i, ploidy[sample]); + if (genotype.conflicts()) { + conflict_count++; + } + columns.add(genotype, calls.samples[sample]); + } + write_record(contig, contig.variants[i], columns); + } + carried[contig_index] = {}; +} + +void NestedGenotyper::write_record(const NestedVariantCatalog::Contig& contig, size_t variant_index, + const SampleColumns& columns) { + const NestedVariant& variant = catalog.variants()[variant_index]; + bool has_samples = !calls.samples.empty(); + out << contig.name << '\t' << variant.pos << '\t' << catalog.id(variant_index) << '\t' << variant.ref << '\t' + << variant.alt << "\t.\t.\tAT=" << walk_text(variant.ref_walk, graph) << ',' + << walk_text(variant.alt_walk, graph); + if (has_samples) { + columns.write_counts(out); + } + out << ";LV=0;CH=" << contig.level; + if (auto snarl = top_level_snarls.find(contig.name); snarl != top_level_snarls.end()) { + out << ";PS=" << snarl->second; + } + out << ";RC=" << contig.site_chrom << ";RS=" << contig.site_start << ";RD=" << contig.site_end; + if (!columns.conflicted_samples().empty()) { + out << ";CONFLICT=" << columns.conflicted_samples(); + } + out << extra_info[variant_index]; + if (has_samples) { + columns.write(out); + } + out << '\n'; +} + +} diff --git a/src/nested_variants.hpp b/src/nested_variants.hpp new file mode 100644 index 0000000000..237b179d4f --- /dev/null +++ b/src/nested_variants.hpp @@ -0,0 +1,330 @@ +#ifndef VG_NESTED_VARIANTS_HPP_INCLUDED +#define VG_NESTED_VARIANTS_HPP_INCLUDED + +/** + * \file nested_variants.hpp + * + * Variants that called alleles carry along gref alt contigs, broken out as + * records on the contigs. + * + * A gref alt contig is one haplotype's copy of sequence that the reference + * lacks, such as an insertion. Another haplotype's ALT allele can run along + * the contig and differ from it, such as an insertion with SNPs in it. + * Comparing the walk of the allele with the contig breaks the SNPs out as + * biallelic variants on the contig, as PanGenie's panel preparation breaks + * variants out of walks by comparing them with the reference. The calls of + * the alleles that read the contig give the variants their genotypes. + */ + +#include "gref.hpp" +#include "vcf_alleles.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace vg { + +/** + * A biallelic variant on a gref alt contig. + * + * Its alleles are in the contig's orientation. As in VCF, an empty allele is + * padded with the contig base before it, or with the base after it at the + * start of the contig. Bases that REF and ALT share at either end are trimmed, + * down to one, as PanGenie's panel preparation trims them. + */ +struct NestedVariant { + /// The gref path of the contig. + uint32_t contig = 0; + /// The 1-based position of the first base of REF on the contig. + uint64_t pos = 0; + std::string ref; + std::string alt; + /// The walks of REF and ALT, from the node before the variant to the node + /// after it. + std::vector ref_walk; + std::vector alt_walk; + + /// Return the 0-based contig coordinate of the first base of REF. + uint64_t ref_start() const; + /// Return the 0-based contig coordinate after the last base of REF. + uint64_t ref_end() const; + /// Return SNV, INS, DEL or COMPLEX, as PanGenie classifies variants. + std::string type() const; + /// Return the ID that PanGenie gives a variant, CONTIG-POS-TYPE-WALK-LENGTH, + /// with the ALT walk. + std::string id(const std::string& contig_name, const gbwtgraph::GBWTGraph& graph) const; +}; + +/// A stretch of a walk that reads a gref alt contig in order, in one orientation. +struct ContigPass { + uint32_t contig = 0; + /// The contig bases [start, end) that the stretch reads. It reads through + /// to an end of the contig if it has a variant there. + uint64_t start = 0; + uint64_t end = 0; + /// The steps of the walk at the first and last nodes that the stretch + /// visits on the contig. + size_t first_step = 0; + size_t last_step = 0; + /// The variants of the stretch relative to the contig, in contig order. + std::vector variants; +}; + +/** + * A decomposer that breaks walks into passes along gref alt contigs, and finds + * the variants of each pass relative to its contig. + * + * Between two consecutive visits of a pass to its contig, the nodes of the + * walk replace those of the contig, as a variant. A pass that leaves the + * contig before its end has a variant there if a later node of the walk + * follows the contig's last node in the graph: the walk's nodes up to that + * node replace the contig's tail. Otherwise the walk really leaves the contig, + * and the pass ends at its last visit. The start of the contig works the same + * way, with a node that precedes the contig's first node. + * + * The reference lies outside every contig, so a walk that visits it has left + * them all. Its next visit to a contig starts a new pass, and a variant at the + * start or end of a contig cannot run across the reference. Only the node that + * closes such a variant can be a reference node, next to where the contig + * branches off. A walk on the reference that shares a node with a contig, such + * as a common SNP allele, therefore reads only that node. + */ +class ContigDecomposer { +public: + /// Make a decomposer for the gref paths of the index that are alt contigs. + ContigDecomposer(const gbwtgraph::GBWTGraph& graph, const GrefIndex& gref); + + /// Return true if the gref path is an alt contig, not a copy of a + /// reference path. + bool is_contig(uint32_t path) const; + /// Return the passes of the walk along alt contigs, in the order of their + /// first visits. + std::vector decompose(const std::vector& walk) const; + +private: + /// A step of a walk onto a node of an alt contig. + struct Visit { + size_t step; + uint32_t contig; + /// True if the walk reads the contig in reverse. + bool is_reverse; + /// The contig bases [start, end) on the node. + uint64_t start; + uint64_t end; + }; + + /// A walk read in one orientation, with its steps numbered that way. + class OrientedWalk; + + /// Return the pass made of the visits, which read one contig in order. + ContigPass make_pass(const std::vector& walk, std::vector visits) const; + /** + * Add to the pass the variant in which the walk's nodes strictly between + * steps left and right replace the contig's bases [ref_start, ref_end). + * + * The walk must read the contig forward. A walk that spells the same + * bases on other nodes has no variant there. + */ + void add_variant(ContigPass& pass, const OrientedWalk& walk, uint64_t ref_start, uint64_t ref_end, size_t left, + size_t right) const; + + const gbwtgraph::GBWTGraph& graph; + /// Finds and spells the contigs, and finds where walks go past their ends. + const GrefIndex& gref; +}; + +/** + * A catalog of the variants that the ALT alleles of a VCF carry along gref alt + * contigs. + * + * It lists each variant once, on its contig, however many alleles carry it. + * The alleles that read a contig are its readers, and the catalog keeps the + * passes of each reader along the contig. + */ +class NestedVariantCatalog { +public: + /// A pass of a reader along a contig. + struct Reading { + /// The contig bases [start, end) that the pass reads. + uint64_t start; + uint64_t end; + /// The variants that the pass has, as indices into variants(). + std::vector variants; + }; + + /// An allele that reads a contig, with its passes along it. + struct Reader { + /// The index of the allele in the calls. + size_t allele; + std::vector passes; + }; + + /// A gref alt contig that alleles read. + struct Contig { + uint32_t path; + /// The name of the contig in VCF, as vg deconstruct names it. + std::string name; + uint64_t length; + /// The variants on the contig, as indices into variants(), in VCF + /// order. + std::vector variants; + std::vector readers; + /// One more than the number of contigs that enclose this one in the + /// walks that read it, as in vg deconstruct's CH tag. + size_t level; + /// The CHROM, first POS and last REF end of the calls that read the + /// contig, as in vg deconstruct's RC, RS and RD tags. + std::string site_chrom; + uint64_t site_start; + uint64_t site_end; + /// The index of the last record with an allele that reads the contig. + size_t last_record; + }; + + /// Decompose the walks of the ALT alleles of the calls. + NestedVariantCatalog(const VcfCalls& calls, const ContigDecomposer& decomposer, const GrefIndex& gref, + const gbwtgraph::GBWTGraph& graph); + + const std::vector& variants() const { return variant_list; } + /// Return the ID of a variant, as PanGenie makes IDs. + const std::string& id(size_t variant) const { return ids[variant]; } + /// Return the contigs, in the order of their last records. + const std::vector& contigs() const { return contig_list; } + /// Return the contigs that an allele reads, as pairs of an index into + /// contigs() and an index into its readers. + const std::vector>& readings(size_t allele) const { return allele_readings[allele]; } + +private: + /// The index of each variant in variants(), by its contig, position and alleles. + using VariantKeys = std::map, size_t>; + + /// Return the index of the allele among the readers of the contig, adding + /// it as a reader if it is not one yet. + size_t reader_of(size_t allele, size_t contig); + /// Return the index of the variant in variants(), adding it to the catalog + /// and to the variants of the contig if it is new. + size_t list_variant(const NestedVariant& variant, size_t contig, VariantKeys& keys, + const gbwtgraph::GBWTGraph& graph); + /// Put the contigs in the order of their last records. + void sort_contigs(); + + std::vector variant_list; + std::vector ids; + std::vector contig_list; + /// For each allele of the calls, the contigs that it reads. + std::vector>> allele_readings; +}; + +/** + * A genotyper that writes the variants of a catalog as VCF records, with + * genotypes from the calls of the alleles that read their contigs. + * + * Each copy of a called ALT allele is a walk that the sample carries. A walk + * that has a variant gives the sample a 1 there, and a walk that reads the + * variant's part of the contig without it gives a 0, even if it has another + * variant there. A sample with no such walk gets "." for the haplotype, as in + * vg deconstruct's records on alt contigs, so allele frequencies are among the + * carriers of the contig. Phased calls give phased genotypes. A sample with + * more walks there than haplotypes gets a missing genotype. + * + * A genotype is only as good as the calls it comes from, so its GQ is the + * lowest GQ of the calls whose walks give it, if they all have one. A record's + * PS names the top-level snarl of its contig, the site that the contig is an + * allele of. + * + * Records go out a contig at a time, as soon as no later call reads the + * contig, so the genotyper holds the walks of a few contigs at most. + */ +class NestedGenotyper { +public: + /** + * Make a genotyper for the calls that writes to out. + * + * @param extra_info Extra INFO for the record of each variant, starting + * with ";" if not empty. + * @param top_level_snarls The top-level snarl of each contig, by its name + * in the VCF, for INFO/PS. Records on contigs that it leaves out get no PS. + */ + NestedGenotyper(const NestedVariantCatalog& catalog, const VcfCalls& calls, std::vector extra_info, + const std::unordered_map& top_level_snarls, + const gbwtgraph::GBWTGraph& graph, std::ostream& out); + + /// Return the header lines that declare the INFO keys, the FORMAT key and + /// the contigs of the records. Contigs without variants are left out. + std::vector header_lines() const; + /// Take the genotypes of a call, then write the records of the contigs + /// that no later call reads. + void add_call(size_t record, const VcfCursor& cursor); + /// Return the number of genotypes set missing because the sample carries + /// more walks there than it has haplotypes. + size_t conflicts() const { return conflict_count; } + /// Return the number of GT alleles that named an ALT allele the record does + /// not have, which are read as REF. + size_t alleles_past_end() const { return past_end_count; } + +private: + /// A walk that a sample carries along a contig. + struct Carried { + /// The index of its allele among the readers of the contig. + size_t reader; + /// The haplotype that carries it, or -1 if the call is not phased. + int haplotype; + /// The GQ of the call, or -1 if it has none. + int quality; + }; + + /// A sample's genotype at a variant. + class Genotype; + /// The sample columns of a record, with the counts for its INFO. + class SampleColumns; + + /** + * Add the walks of the ALT alleles that a sample's genotype calls to the + * contigs that they read. + * + * @param quality The GQ of the call, or -1 if it has none. + * @returns The number of haplotypes in the genotype. + */ + size_t carry_genotype(const VcfSite& site, size_t sample, std::string_view genotype, int quality); + /** + * Add the walk of an ALT allele that a sample carries to each contig that + * the allele reads. + * + * @param haplotype The haplotype that carries the walk, or -1 if the call + * is not phased. + * @param quality The GQ of the call, or -1 if it has none. + */ + void carry(size_t sample, size_t allele, int haplotype, int quality); + /// Write the records of the variants on a contig, and forget its walks. + void write_contig(size_t contig); + /// Write the record of a variant on the contig, with the sample columns. + void write_record(const NestedVariantCatalog::Contig& contig, size_t variant, const SampleColumns& columns); + + const NestedVariantCatalog& catalog; + const VcfCalls& calls; + /// Extra INFO for the record of each variant. + std::vector extra_info; + const std::unordered_map& top_level_snarls; + const gbwtgraph::GBWTGraph& graph; + std::ostream& out; + /// For each contig, the walks that each sample carries along it. Empty + /// until a call reads the contig. + std::vector>> carried; + /// The number of haplotypes of each sample, from its genotypes. + std::vector ploidy; + /// The index of the next contig to write. + size_t next_contig = 0; + size_t conflict_count = 0; + size_t past_end_count = 0; +}; + +} + +#endif diff --git a/src/sorted_vcf_writer.hpp b/src/sorted_vcf_writer.hpp index db9090099f..3baf14ed62 100644 --- a/src/sorted_vcf_writer.hpp +++ b/src/sorted_vcf_writer.hpp @@ -135,7 +135,7 @@ class SortedVCFWriter { /// Run vcflib's vcffixup on a record, filling AC, AN and the like. void vcf_fixup(vcflib::Variant& var) const; - /// The nesting INFO header lines, for both vg call and vg deconstruct. + /// The nesting INFO header lines, for vg call, vg deconstruct and vg lift. static string nesting_info_headers(); /// A record longer than this is refused. diff --git a/src/subcommand/lift_main.cpp b/src/subcommand/lift_main.cpp new file mode 100644 index 0000000000..7ff9d4fd91 --- /dev/null +++ b/src/subcommand/lift_main.cpp @@ -0,0 +1,1066 @@ +/** + * \file lift_main.cpp: lift features on a cover or on paths to the graph, the gref and the paths + */ + +#include +#include + +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "subcommand.hpp" +#include "../feature_lift.hpp" +#include "../gbwtgraph_helper.hpp" +#include "../gref.hpp" +#include "../haplotype_depth.hpp" +#include "../nested_variants.hpp" +#include "../utility.hpp" +#include "../vcf_alleles.hpp" + +using namespace std; +using namespace vg; +using namespace vg::subcommand; + +void help_lift(char** argv) { + cerr << "usage: " << argv[0] << " lift [options] graph.gbz features.bed > features.gaf" << endl + << " " << argv[0] << " lift [options] graph.gbz variants.vcf > variants.lifted.vcf" << endl + << " " << argv[0] << " lift --depth [options] graph.gbz [regions.bed] > depth.bedGraph" << endl + << "Lift features to the graph and the gref, and to each path that visits them." << endl + << "A feature can be on a cover sequence (a gref path of the GBZ or a tig of" << endl + << "vg unitigs) or on any path in the GBZ, named without a subrange or phase" << endl + << "block and in coordinates of the full path (for haplotypes, the assembly" << endl + << "contig). A feature on several sequences at the same graph place is reported" << endl + << "once." << endl + << endl + << "Each feature gets an ID (f1, f2, ...) and a GAF record with that query name," << endl + << "its graph walk, and the tags sp:i: (visits of the whole walk by paths other" << endl + << "than gref paths), gr:Z: (gref placement, see vg unitigs), pc:Z: (path" << endl + << "coordinate) and an:Z: (BED name)." << endl + << endl + << "The path coordinate names a feature exactly, as PATH:START-END:STRAND on the" << endl + << "first path, in priority order, that carries the whole feature: the gref path" << endl + << "it lies along, else a reference or assembly path. Paths keep their" << endl + << "coordinates in clipped, filtered and sampled graphs, and no two places in the" << endl + << "graph share one. Naming a feature off the gref takes a pass over all paths." << endl + << endl + << "A VCF of variants in the graph gives a feature for each ALT allele: its walk" << endl + << "without the first and last nodes, which lie on the reference on either side." << endl + << "The walk comes from INFO/AT (vg deconstruct) or, for a record with one ALT" << endl + << "allele and no AT, from a PanGenie variant ID (CHROM-POS-TYPE-WALK-LENGTH)." << endl + << "The VCF is written out with each allele's gref placement in INFO/GR, its" << endl + << "segments separated by |, and its path coordinate in INFO/PC. A deletion has" << endl + << "no nodes left to place and gets neither. The VCF must be a regular file," << endl + << "plain or compressed, since it is read twice; features on a pipe are read as" << endl + << "BED. Its features are numbered in file order and named by the ID of" << endl + << "their record, with :N added for the Nth ALT allele of a record with several." << endl + << endl + << "Where an allele runs along a gref alt contig and differs from it, such as an" << endl + << "insertion with SNPs in it, each difference becomes a biallelic record on the" << endl + << "contig, written after the calls as vg deconstruct writes nested records. A" << endl + << "haplotype gets 1 if its called allele has the variant, 0 if the allele reads" << endl + << "that part of the contig without it, and . otherwise, so allele frequencies" << endl + << "are among the carriers of the contig. Its GQ is the lowest GQ of the calls" << endl + << "that give the genotype, if they have GQ. With --gref-segs, each such record" << endl + << "gets INFO/PS, the top-level snarl of its contig. A VCF that already has" << endl + << "records on alt contigs gets none added." << endl + << endl + << "With --depth, write how many haplotypes read each base of the gref paths, or" << endl + << "of the paths chosen with -Q, as bedGraph: the path, the 0-based start and end" << endl + << "of a run of bases of equal depth, and the depth, which is the number of" << endl + << "distinct haplotypes (sample and haplotype number) that read them. Runs of" << endl + << "depth 0 are written too. Haplotype paths count, and reference paths with" << endl + << "--references; gref paths and generic paths never do. A regions BED, on paths" << endl + << "named and placed as features are, limits the count to the bases inside it," << endl + << "and a path without regions counts nothing. Given one assembly's QC flags as" << endl + << "regions, the depth is the number of haplotypes flagged there; given the" << endl + << "regions each assembly's QC could assess, it is the number assessed. Unlike" << endl + << "vg depth -P, this counts haplotypes, not path names." << endl + << endl + << "options:" << endl + << " -u, --tigs FILE GAF of the tigs, from vg unitigs -g" << endl + << " -c, --canonical FILE write each feature once to FILE as BED at its path" << endl + << " coordinate, in priority order, with the columns of -a" << endl + << " -P, --priority PREFIX name features off the gref on paths whose name starts" << endl + << " with PREFIX first (may repeat, first given first);" << endl + << " then come reference paths, then the rest by name" << endl + << " -a, --assemblies FILE write each place where a path visits a feature to FILE" << endl + << " as BED on the full path: the feature's name, score" << endl + << " and strand (., 0 and . if it has none), any further" << endl + << " BED columns, and then its ID" << endl + << " -m, --min-match F write a place wherever a path visits at least F of the" << endl + << " feature's bases in order (liftover), and end each row" << endl + << " with that fraction. Without -m, a place must hold the" << endl + << " feature's whole walk. A gap on the path longer than" << endl + << " the feature splits a place." << endl + << " -b, --blocks write places as BED12, with a block for each stretch" << endl + << " that holds feature bases" << endl + << " --gref-segs FILE with a VCF, take the top-level snarl of each alt" << endl + << " contig for INFO/PS from FILE, the table of gref" << endl + << " fragments from vg paths --gref-segs" << endl + << " -Q, --paths-by PREFIX only write places (with --depth, rows) on paths whose" << endl + << " name starts with PREFIX (may repeat). With --depth," << endl + << " the default is the gref paths." << endl + << " --depth write the depth of haplotypes along paths as bedGraph" << endl + << " (see above); takes no -u, -c, -P, -a, -m or -b" << endl + << " --visits with --depth, count the visits of haplotype paths" << endl + << " instead of distinct haplotypes" << endl + << " --references with --depth, count reference paths as haplotypes" << endl + << " -t, --threads N number of threads to use [all available]" << endl + << " -p, --progress show progress" << endl + << " -h, --help print this help message to stderr and exit" << endl; +} + +/// The options and arguments of vg lift. +struct LiftConfig { + string graph_file; + /// The BED or VCF of features, or with --depth the regions BED, if any. + string input_file; + /// True if the features are the ALT alleles of a VCF. + bool vcf_input = false; + string tigs_file; + string canonical_file; + vector priority_prefixes; + string assemblies_file; + vector path_prefixes; + double min_match = 0; + bool blocks = false; + bool progress = false; + bool depth_mode = false; + HaplotypeDepth::Options depth_options; + string gref_segs_file; + + /** + * Read the command line. + * + * Print the help and exit on -h or an unknown option. Report an error on + * options that do not go together, and on missing or extra arguments. + */ + LiftConfig(int argc, char** argv, const Logger& logger); + +private: + /// Read the options, leaving optind at the first argument. + void read_options(int argc, char** argv, const Logger& logger); + /// Return true if an option for lifting features is set. + bool has_lift_options() const; +}; + +LiftConfig::LiftConfig(int argc, char** argv, const Logger& logger) { + read_options(argc, argv, logger); + if (!depth_mode && (depth_options.count_visits || depth_options.count_references)) { + logger.error() << "--visits and --references need --depth" << endl; + } + if (depth_mode) { + if (has_lift_options()) { + logger.error() << "--depth takes no -u, -c, -P, -a, -m, -b or --gref-segs" << endl; + } + if (optind + 1 != argc && optind + 2 != argc) { + logger.error() << "expected a GBZ file and optionally a regions BED" << endl; + } + graph_file = require_exists(logger, argv[optind]); + input_file = optind + 2 == argc ? argv[optind + 1] : ""; + return; + } + if (optind + 2 != argc) { + logger.error() << "expected a GBZ file and a BED or VCF file" << endl; + } + graph_file = require_exists(logger, argv[optind]); + input_file = argv[optind + 1]; + vcf_input = input_file != "-" && is_variant_file(input_file); + if (!vcf_input && !gref_segs_file.empty()) { + logger.error() << "--gref-segs needs a VCF" << endl; + } +} + +void LiftConfig::read_options(int argc, char** argv, const Logger& logger) { + constexpr int OPT_DEPTH = 1001; + constexpr int OPT_VISITS = 1002; + constexpr int OPT_REFERENCES = 1003; + constexpr int OPT_GREF_SEGS = 1004; + + optind = 2; // force optind past command positional argument + while (true) { + static struct option long_options[] = { + {"depth", no_argument, nullptr, OPT_DEPTH}, + {"visits", no_argument, nullptr, OPT_VISITS}, + {"references", no_argument, nullptr, OPT_REFERENCES}, + {"gref-segs", required_argument, nullptr, OPT_GREF_SEGS}, + {"tigs", required_argument, nullptr, 'u'}, + {"canonical", required_argument, nullptr, 'c'}, + {"priority", required_argument, nullptr, 'P'}, + {"assemblies", required_argument, nullptr, 'a'}, + {"min-match", required_argument, nullptr, 'm'}, + {"blocks", no_argument, nullptr, 'b'}, + {"paths-by", required_argument, nullptr, 'Q'}, + {"threads", required_argument, nullptr, 't'}, + {"progress", no_argument, nullptr, 'p'}, + {"help", no_argument, nullptr, 'h'}, + {0, 0, 0, 0} + }; + int option_index = 0; + int c = getopt_long(argc, argv, "u:c:P:a:m:bQ:t:ph?", long_options, &option_index); + if (c == -1) { + break; + } + switch (c) { + case 'u': + tigs_file = require_exists(logger, optarg); + break; + case 'c': + canonical_file = ensure_writable(logger, optarg); + break; + case 'P': + priority_prefixes.emplace_back(optarg); + break; + case 'a': + assemblies_file = ensure_writable(logger, optarg); + break; + case 'm': + min_match = parse(optarg); + if (min_match <= 0 || min_match > 1) { + logger.error() << "-m/--min-match must be in (0, 1]" << endl; + } + break; + case 'b': + blocks = true; + break; + case 'Q': + path_prefixes.emplace_back(optarg); + break; + case 't': + set_thread_count(logger, optarg); + break; + case 'p': + progress = true; + break; + case OPT_DEPTH: + depth_mode = true; + break; + case OPT_VISITS: + depth_options.count_visits = true; + break; + case OPT_REFERENCES: + depth_options.count_references = true; + break; + case OPT_GREF_SEGS: + gref_segs_file = require_exists(logger, optarg); + break; + case 'h': + case '?': + default: + help_lift(argv); + std::exit(EXIT_FAILURE); + } + } +} + +bool LiftConfig::has_lift_options() const { + return !tigs_file.empty() || !canonical_file.empty() || !priority_prefixes.empty() || !assemblies_file.empty() || + min_match > 0 || blocks || !gref_segs_file.empty(); +} + +//------------------------------------------------------------------------------ + +/// Return true if the name starts with one of the prefixes. +static bool starts_with_any(string_view name, const vector& prefixes) { + return std::any_of(prefixes.begin(), prefixes.end(), [&](const string& prefix) { + return name.starts_with(prefix); + }); +} + +/// Return the number of features without a path coordinate. +static size_t count_unnamed(const vector& features) { + return std::count_if(features.begin(), features.end(), [](const GraphFeature& feature) { + return !feature.coordinate.has_value(); + }); +} + +/// Return the ID of the feature with the given index. +static string feature_id(size_t i) { + return "f" + to_string(i + 1); +} + +/** + * Return the contents of a text file, plain or compressed, as a stream. + * + * @throws std::runtime_error if the file cannot be read. + */ +static istringstream read_text_file(const string& path) { + htsFile* file = hts_open(path.c_str(), "r"); + if (file == nullptr) { + throw runtime_error("cannot open " + path); + } + string text; + kstring_t line = KS_INITIALIZE; + int status; + while ((status = hts_getline(file, '\n', &line)) >= 0) { + text.append(line.s, line.l).push_back('\n'); + } + ks_free(&line); + hts_close(file); + if (status < -1) { + throw runtime_error("cannot read " + path); + } + return istringstream(text); +} + +//------------------------------------------------------------------------------ + +/// Read the features of a BED file, and warn that BED12 blocks are not lifted. +static vector read_bed_file(const string& features_file, const Logger& logger, bool progress) { + vector bed_features; + get_input_file(features_file, [&](istream& in) { + try { + bed_features = read_bed_features(in); + } catch (const runtime_error& e) { + logger.error() << e.what() << endl; + } + }); + if (progress) { + cerr << "[vg lift] " << bed_features.size() << " features" << endl; + } + if (std::any_of(bed_features.begin(), bed_features.end(), [](const BedFeature& feature) { + return feature.fields.size() >= 9; + })) { + logger.warn() << "BED12 blocks are not lifted; columns from the 7th on are copied as they are" << endl; + } + return bed_features; +} + +/// Report an error for a feature that ends past the end of its sequence, or +/// that is on a sequence that is not in the cover or the path directory. +static void check_sequences(const vector& bed_features, const Cover& cover, const PathDirectory& paths, + const LiftConfig& config, const Logger& logger) { + for (const BedFeature& feature : bed_features) { + optional length = cover.sequence_length(feature.sequence); + if (length.has_value() && feature.end > *length) { + logger.error() << "feature " << feature.sequence << ":" << feature.start << "-" << feature.end + << " ends past the end of the sequence (" << *length << " bp)" << endl; + } + if (!length.has_value() && !paths.has_path(feature.sequence)) { + logger.error() << "feature on " << feature.sequence << ", which is not a gref path or a path in " + << config.graph_file << ", or a tig" + << (config.tigs_file.empty() ? " (use -u)" : " in " + config.tigs_file) << endl; + } + } +} + +/** + * Lift each feature to the graph, from its sequence in the cover or its path + * in the directory, in order. + * + * A feature on a path lifts only if one fragment of the path holds it. Leave + * out the features that no fragment holds, and count them in skipped. + */ +static vector lift_to_graph(vector bed_features, const Cover& cover, + const PathDirectory& paths, const gbwtgraph::GBWTGraph& graph, + const GrefIndex* gref, size_t& skipped) { + vector features(bed_features.size()); + vector lifted(bed_features.size(), true); + #pragma omp parallel for schedule(dynamic, 1024) + for (size_t i = 0; i < bed_features.size(); i++) { + BedFeature& feature = bed_features[i]; + optional interval; + if (cover.sequence_length(feature.sequence).has_value()) { + interval = cover.interval(feature.sequence, feature.start, feature.end); + } else { + interval = paths.interval(feature.sequence, feature.start, feature.end); + } + if (interval.has_value()) { + features[i] = make_graph_feature(graph, gref, std::move(*interval), std::move(feature.fields)); + } else { + lifted[i] = false; + } + } + skipped = 0; + for (size_t i = 0; i < features.size(); i++) { + if (!lifted[i]) { + skipped++; + } else if (skipped > 0) { + features[i - skipped] = std::move(features[i]); + } + } + features.resize(features.size() - skipped); + return features; +} + +/** + * Read the BED features and lift each to the graph, keeping each feature once. + * + * A feature is on a gref path, on a tig in the tigs file, or on a path in the + * GBZ. Features on a path outside its fragments in the graph are skipped. + */ +static vector lift_bed_features(const LiftConfig& config, const gbwtgraph::GBZ& gbz, + const GrefIndex* gref_pointer, const Logger& logger) { + Cover cover(gbz.graph, gref_pointer); + vector bed_features = read_bed_file(config.input_file, logger, config.progress); + + // Find the sequence of each feature: a gref path, a tig, or a path in the + // GBZ. Load only the tigs and paths that features are on. + unordered_set wanted; + for (const BedFeature& feature : bed_features) { + if (!cover.is_gref_path(feature.sequence)) { + wanted.insert(feature.sequence); + } + } + if (!wanted.empty() && !config.tigs_file.empty()) { + get_input_file(config.tigs_file, [&](istream& in) { + try { + cover.add_tigs(in, [&](const string& name) { return wanted.contains(name); }); + } catch (const runtime_error& e) { + logger.error() << e.what() << endl; + } + }); + } + PathDirectory paths(gbz, [&](const string& name) { + return wanted.contains(name) && !cover.sequence_length(name).has_value(); + }); + check_sequences(bed_features, cover, paths, config, logger); + + size_t skipped = 0; + vector features = lift_to_graph(std::move(bed_features), cover, paths, gbz.graph, gref_pointer, + skipped); + if (skipped > 0) { + logger.warn() << skipped << " features on paths are not within one fragment of the path in " + << config.graph_file << " and were skipped" << endl; + } + sort_and_merge_features(features); + if (config.progress) { + cerr << "[vg lift] " << features.size() << " distinct features in the graph" << endl; + } + return features; +} + +//------------------------------------------------------------------------------ + +/// The feature index of an ALT allele without a feature. The VCF gives no walk +/// for it, or its walk has no nodes besides the first and last. +constexpr size_t NO_FEATURE = numeric_limits::max(); + +/// Return the INFO/GR and INFO/PC values of a feature, "." where it has none. +static pair lifted_values(const GraphFeature& feature) { + pair values(".", "."); + if (!feature.placement.text.empty()) { + // Commas separate the values of the alleles. + values.first = feature.placement.text; + std::replace(values.first.begin(), values.first.end(), ',', '|'); + } + if (feature.coordinate) { + values.second = feature.coordinate->text(); + } + return values; +} + +/** + * A VCF of graph variants whose ALT alleles are features to lift. + * + * It adds a feature for each ALT allele with bases to place. It also adds one + * for each variant that the alleles carry along gref alt contigs, unless the + * VCF already has records on the contigs. It writes the VCF back with the gref + * placement and path coordinate of each allele, followed by the records of + * those nested variants. + */ +class LiftedVcf { +public: + /// Read the VCF, and add the features of its ALT alleles and of their + /// nested variants to the features. + LiftedVcf(const string& vcf_file, const gbwtgraph::GBWTGraph& graph, const GrefIndex& gref, + vector& features, const Logger& logger, bool progress); + + /** + * Write the VCF to standard output with the tags of its ALT alleles, + * followed by the records of the nested variants. + * + * @param features The features, named and placed. + * @param top_level_snarls The top-level snarl of each alt contig, by its + * name in the VCF, for the PS of the nested records. + */ + void write(const vector& features, const unordered_map& top_level_snarls, + const Logger& logger) const; + +private: + /** + * Add a feature for each ALT allele whose walk has nodes besides the + * first and last, in file order. + * + * Each feature is stranded and reads along its allele, so that its + * placement and path coordinate say which way the allele reads its paths. + * + * @returns For each ALT allele, the index of its feature, or NO_FEATURE. + */ + vector add_allele_features(vector& features, const GrefIndex* gref, const Logger& logger, + bool progress); + /// Return true if a record is on a gref alt contig. Such a VCF already + /// has its nested variants, as vg deconstruct or vg lift wrote them. + bool has_contig_records(const ContigDecomposer& decomposer, const GrefIndex& gref) const; + /** + * Add a feature for the ALT allele of each nested variant, its walk + * without the first and last nodes, if that has any nodes. + * + * @returns For each variant, the index of its feature, or NO_FEATURE. + */ + vector add_nested_features(vector& features, const GrefIndex* gref) const; + /// Return INFO/GR, the gref placement, and INFO/PC, the path coordinate, + /// for each ALT allele. + vector lifted_tags(const vector& features) const; + /// Return the INFO/GR and INFO/PC entries of the record of each nested + /// variant, each starting with ";", or empty if the variant has neither. + vector nested_tags(const vector& features) const; + + string vcf_file; + const gbwtgraph::GBWTGraph& graph; + VcfCalls calls; + /// The variants that the ALT alleles carry along gref alt contigs. + optional catalog; + /// The index of the feature of each ALT allele, or NO_FEATURE. + vector allele_features; + /// The index of the feature of each nested variant, or NO_FEATURE. + vector nested_features; +}; + +LiftedVcf::LiftedVcf(const string& vcf_file, const gbwtgraph::GBWTGraph& graph, const GrefIndex& gref, + vector& features, const Logger& logger, bool progress) : + vcf_file(vcf_file), graph(graph) { + try { + calls = read_vcf_calls(vcf_file, graph, vg::get_thread_count()); + } catch (const runtime_error& e) { + logger.error() << e.what() << endl; + } + const GrefIndex* gref_pointer = gref.path_count() > 0 ? &gref : nullptr; + allele_features = add_allele_features(features, gref_pointer, logger, progress); + ContigDecomposer decomposer(graph, gref); + if (has_contig_records(decomposer, gref)) { + logger.warn() << vcf_file << " already has records on gref alt contigs, so no nested variants " + << "are added" << endl; + return; + } + if (gref_pointer == nullptr) { + return; + } + catalog.emplace(calls, decomposer, gref, graph); + nested_features = add_nested_features(features, gref_pointer); + if (progress) { + const auto& contigs = catalog->contigs(); + size_t contigs_with_variants = count_if(contigs.begin(), contigs.end(), + [](const NestedVariantCatalog::Contig& contig) { + return !contig.variants.empty(); + }); + cerr << "[vg lift] " << catalog->variants().size() << " nested variants on " << contigs_with_variants + << " gref alt contigs, of " << contigs.size() << " that ALT alleles read" << endl; + } +} + +vector LiftedVcf::add_allele_features(vector& features, const GrefIndex* gref, + const Logger& logger, bool progress) { + vector& alleles = calls.alleles; + vector result(alleles.size(), NO_FEATURE); + size_t first_feature = features.size(); + size_t without_walk = 0; + size_t foreign = 0; + string foreign_example; + for (size_t i = 0; i < alleles.size(); i++) { + if (alleles[i].walk.empty()) { + without_walk++; + } + if (!alleles[i].holds_alt) { + if (foreign == 0) { + foreign_example = alleles[i].name; + } + foreign++; + } + if (alleles[i].interval) { + result[i] = features.size(); + features.emplace_back(); + } + } + #pragma omp parallel for schedule(dynamic, 1024) + for (size_t i = 0; i < alleles.size(); i++) { + if (result[i] != NO_FEATURE) { + features[result[i]] = make_graph_feature(graph, gref, std::move(*alleles[i].interval), + { alleles[i].name, "0", "+" }, + alleles[i].walk.front(), alleles[i].walk.back()); + } + } + if (progress) { + cerr << "[vg lift] " << alleles.size() << " ALT alleles, " << (features.size() - first_feature) + << " with bases to place" << endl; + } + if (without_walk > 0) { + logger.warn() << without_walk << " ALT alleles have no walk in INFO/AT or in a PanGenie variant ID" << endl; + } + if (foreign > 0) { + logger.warn() << foreign << " ALT alleles, such as " << foreign_example + << ", are not in the sequences of their walks; is the VCF from this graph?" << endl; + } + return result; +} + +bool LiftedVcf::has_contig_records(const ContigDecomposer& decomposer, const GrefIndex& gref) const { + unordered_set contig_names; + for (uint32_t path = 0; path < gref.path_count(); path++) { + if (decomposer.is_contig(path)) { + contig_names.insert(PathMetadata::parse_locus_name(gref.path_name(path))); + } + } + return std::any_of(calls.sites.begin(), calls.sites.end(), [&](const VcfSite& site) { + return contig_names.contains(site.chrom); + }); +} + +vector LiftedVcf::add_nested_features(vector& features, const GrefIndex* gref) const { + const vector& variants = catalog->variants(); + vector result(variants.size(), NO_FEATURE); + for (size_t i = 0; i < variants.size(); i++) { + if (variants[i].alt_walk.size() > 2) { + result[i] = features.size(); + features.emplace_back(); + } + } + #pragma omp parallel for schedule(dynamic, 1024) + for (size_t i = 0; i < variants.size(); i++) { + if (result[i] != NO_FEATURE) { + const vector& walk = variants[i].alt_walk; + WalkInterval interval; + interval.walk.assign(walk.begin() + 1, walk.end() - 1); + interval.length = interval.walk_length(graph); + features[result[i]] = make_graph_feature(graph, gref, std::move(interval), { catalog->id(i), "0", "+" }, + walk.front(), walk.back()); + } + } + return result; +} + +vector LiftedVcf::lifted_tags(const vector& features) const { + VcfAlleleInfo placements { "GR", + "##INFO=", + vector(allele_features.size(), ".") }; + VcfAlleleInfo coordinates { "PC", + "##INFO=", + vector(allele_features.size(), ".") }; + for (size_t i = 0; i < allele_features.size(); i++) { + if (allele_features[i] != NO_FEATURE) { + tie(placements.values[i], coordinates.values[i]) = lifted_values(features[allele_features[i]]); + } + } + return { placements, coordinates }; +} + +vector LiftedVcf::nested_tags(const vector& features) const { + vector tags(catalog->variants().size()); + for (size_t i = 0; i < tags.size(); i++) { + if (nested_features[i] != NO_FEATURE) { + auto [placement, coordinate] = lifted_values(features[nested_features[i]]); + tags[i] = (placement == "." ? "" : ";GR=" + placement) + (coordinate == "." ? "" : ";PC=" + coordinate); + } + } + return tags; +} + +void LiftedVcf::write(const vector& features, const unordered_map& top_level_snarls, + const Logger& logger) const { + // The genotyper writes the nested records to a temporary file as it goes, + // and they follow the calls. + string nested_file = temp_file::create("vg-lift-nested"); + ofstream nested_out(nested_file); + optional genotyper; + vector nested_header; + if (catalog.has_value() && !catalog->variants().empty()) { + genotyper.emplace(*catalog, calls, nested_tags(features), top_level_snarls, graph, nested_out); + nested_header = genotyper->header_lines(); + } + try { + write_vcf_with_allele_info(vcf_file, cout, lifted_tags(features), nested_header, + [&](size_t record, const VcfCursor& cursor) { + if (genotyper) { + genotyper->add_call(record, cursor); + } + }, vg::get_thread_count()); + } catch (const runtime_error& e) { + logger.error() << e.what() << endl; + } + nested_out.close(); + ifstream nested_in(nested_file); + if (nested_in.peek() != ifstream::traits_type::eof()) { + cout << nested_in.rdbuf(); + } + nested_in.close(); + temp_file::remove(nested_file); + if (genotyper && genotyper->alleles_past_end() > 0) { + logger.warn() << genotyper->alleles_past_end() << " GT alleles name an ALT allele their record does " + << "not have, and are read as the reference in the nested records" << endl; + } + if (genotyper && genotyper->conflicts() > 0) { + logger.warn() << genotyper->conflicts() << " nested genotypes are missing because the sample carries more " + << "walks there than it has haplotypes" << endl; + } +} + +/** + * Return the top-level snarl of each alt contig, by its name in the VCF, from + * a table of gref fragments, or none without a table. + */ +static unordered_map read_contig_snarls(const string& gref_segs_file, const Logger& logger) { + unordered_map top_level_snarls; + if (gref_segs_file.empty()) { + return top_level_snarls; + } + try { + istringstream segs = read_text_file(gref_segs_file); + for (auto& [contig, snarl] : GrefCover::read_top_level_snarls(segs)) { + top_level_snarls.try_emplace(PathMetadata::parse_locus_name(contig), std::move(snarl)); + } + } catch (const runtime_error& e) { + logger.error() << gref_segs_file << ": " << e.what() << endl; + } + return top_level_snarls; +} + +//------------------------------------------------------------------------------ + +/** + * Write BED columns 4 to 6 (name, score and strand) for a place of the + * feature, each after a tab. The place reads the feature in reverse if + * is_reverse is set. + */ +static void write_name_score_strand(ostream& out, const GraphFeature& feature, bool is_reverse) { + const vector& fields = feature.fields; + out << '\t' << (!fields.empty() ? fields[0] : ".") << '\t' << (fields.size() > 1 ? fields[1] : "0") << '\t'; + if (feature.is_stranded()) { + out << (is_reverse ? '-' : '+'); + } else { + out << '.'; + } +} + +/// Write the BED columns of the feature from the 7th on, then the ID of the +/// feature, which has index i, each after a tab. +static void write_rest(ostream& out, const GraphFeature& feature, size_t i) { + const vector& fields = feature.fields; + for (size_t j = BED_STRAND_FIELD + 1; j < fields.size(); j++) { + out << '\t' << fields[j]; + } + out << '\t' << feature_id(i); +} + +/// Write BED columns 7 to 12 for a place, each after a tab. The thick part is +/// the whole place, the color is 0, and there is a block for each stretch that +/// holds feature bases. +static void write_blocks(ostream& out, const PathOccurrence& place) { + out << '\t' << place.start << '\t' << place.end << "\t0\t" << place.blocks.size() << '\t'; + for (const auto& [block_start, block_end] : place.blocks) { + out << (block_end - block_start) << ','; + } + out << '\t'; + for (const auto& [block_start, block_end] : place.blocks) { + out << (block_start - place.start) << ','; + } +} + +/** + * Write the BED row of a place on the named path, as -a writes it. + * + * With blocks, the row is BED12. With a min_match, it ends with the fraction + * of the feature's bases that the place holds. + */ +static void write_place(ostream& out, const string& path, const PathOccurrence& place, const GraphFeature& feature, + bool blocks, double min_match) { + out << path << '\t' << place.start << '\t' << place.end; + write_name_score_strand(out, feature, place.is_reverse); + if (blocks) { + write_blocks(out, place); + } + write_rest(out, feature, place.feature); + if (min_match > 0) { + out << '\t' << static_cast(place.matched) / static_cast(feature.interval.length); + } + out << '\n'; +} + +/// Name the feature on the path at a place, if the place holds the whole +/// feature in one block and the path comes before the feature's naming path, +/// if any, in the priority order. +static void name_by_place(GraphFeature& feature, const string& path, size_t rank, const PathOccurrence& place) { + // A place that holds the whole feature in one block is a path that carries it. + if (size_t length = feature.interval.length; + place.matched != length || place.blocks.size() != 1 || place.end - place.start != length) { + return; + } + PathCoordinate here { path, rank, place.start, place.end, place.is_reverse }; + if (!feature.coordinate || here < *feature.coordinate) { + feature.coordinate = std::move(here); + } +} + +/** + * Name the features off the gref by the paths that carry them, and write the + * places of the features on the paths chosen with -Q for -a. + * + * One pass over the paths other than gref paths does both. Naming uses every + * path, whatever -Q says. + */ +static void name_and_place_features(const gbwtgraph::GBZ& gbz, vector& features, + const LiftConfig& config, size_t gref_path_count, const Logger& logger) { + PathPriority priority(gbz, config.priority_prefixes, GrefCover::gref_prefix, gref_path_count); + ofstream out; + if (!config.assemblies_file.empty()) { + out.open(config.assemblies_file); + if (!out) { + logger.error() << "could not open " << config.assemblies_file << " for writing" << endl; + } + } + size_t written = 0; + auto visit = [&](const string& name, const vector& places) { + bool write = out.is_open() && (config.path_prefixes.empty() || starts_with_any(name, config.path_prefixes)); + size_t rank = priority.rank(name); + for (const PathOccurrence& place : places) { + name_by_place(features[place.feature], name, rank, place); + if (write) { + write_place(out, name, place, features[place.feature], config.blocks, config.min_match); + written++; + } + } + }; + auto not_gref = [&](string_view name) { + return !name.starts_with(GrefCover::gref_prefix); + }; + if (config.min_match > 0) { + for_each_path_liftover(gbz, features, config.min_match, not_gref, visit); + } else { + for_each_path_occurrences(gbz, features, not_gref, visit); + } + if (config.progress && out.is_open()) { + cerr << "[vg lift] " << written << " places on paths" << endl; + } +} + +/// Write a GAF record for each feature to standard output, with its tags. +static void write_feature_gaf(const vector& features, const gbwtgraph::GBWTGraph& graph) { + for (size_t i = 0; i < features.size(); i++) { + const GraphFeature& feature = features[i]; + write_gaf_columns(cout, graph, feature_id(i), feature.interval); + cout << "\tsp:i:" << feature.support; + if (!feature.placement.text.empty()) { + cout << "\tgr:Z:" << feature.placement.text; + } + if (feature.coordinate) { + cout << "\tpc:Z:" << feature.coordinate->text(); + } + if (!feature.fields.empty()) { + cout << "\tan:Z:" << feature.fields.front(); + } + cout << '\n'; + } +} + +/// Write each feature with a path coordinate once, as a BED row at that +/// coordinate, in priority order. +static void write_canonical(const vector& features, const string& canonical_file, + const Logger& logger) { + ofstream out(canonical_file); + if (!out) { + logger.error() << "could not open " << canonical_file << " for writing" << endl; + } + vector order; + for (size_t i = 0; i < features.size(); i++) { + if (features[i].coordinate) { + order.push_back(i); + } + } + std::stable_sort(order.begin(), order.end(), [&](size_t a, size_t b) { + return *features[a].coordinate < *features[b].coordinate; + }); + for (size_t i : order) { + const PathCoordinate& place = *features[i].coordinate; + out << place.path << '\t' << place.start << '\t' << place.end; + write_name_score_strand(out, features[i], place.is_reverse); + write_rest(out, features[i], i); + out << '\n'; + } +} + +//------------------------------------------------------------------------------ + +/// Load the regions in a BED file, or none if the file name is empty. +static unique_ptr read_regions(const string& regions_file, const Logger& logger) { + unique_ptr regions; + if (regions_file.empty()) { + return regions; + } + get_input_file(regions_file, [&](istream& in) { + try { + regions = make_unique(in); + } catch (const runtime_error& e) { + logger.error() << e.what() << endl; + } + }); + return regions; +} + +/// A path to write depth along. +struct OutputPath { + /// The name of the full path, as for_each_path_start() gives it. + string name; + /// Where the path starts on the full path. + uint64_t start; + gbwt::size_type id; +}; + +/** + * Return the paths to write depth along, sorted by name and then along each + * full path. They are the gref paths, or those whose names start with one of + * the prefixes. + */ +static vector output_paths(const gbwtgraph::GBZ& gbz, const vector& path_prefixes) { + vector outputs; + for_each_path_start(gbz.index, [&](gbwt::size_type path_id, const string& name, uint64_t start) { + if (path_prefixes.empty() ? GrefCover::is_gref_derived(name) : starts_with_any(name, path_prefixes)) { + outputs.emplace_back(name, start, path_id); + } + }); + std::sort(outputs.begin(), outputs.end(), [](const OutputPath& a, const OutputPath& b) { + return std::tie(a.name, a.start) < std::tie(b.name, b.start); + }); + return outputs; +} + +/** + * Write the depth of haplotypes along paths as bedGraph on standard output. + * + * The paths are the gref paths, or those whose names start with one of the + * prefixes, in name order and then in order along each full path. + * + * @param regions_file A BED of the regions to count in, or empty to count + * every base. + */ +static void write_depth(const gbwtgraph::GBZ& gbz, const string& graph_file, const string& regions_file, + HaplotypeDepth::Options options, const vector& path_prefixes, const Logger& logger, + bool progress) { + unique_ptr regions = read_regions(regions_file, logger); + options.regions = regions.get(); + HaplotypeDepth depth(gbz, options); + if (progress) { + cerr << "[vg lift] counted " << depth.haplotype_count() << " haplotypes"; + if (regions) { + cerr << ", " << depth.haplotypes_without_regions() << " of them without regions"; + } + cerr << endl; + } + if (const vector& unmatched = depth.uncounted_region_paths(); !unmatched.empty()) { + logger.warn() << unmatched.size() << " paths in " << regions_file << ", such as " << unmatched.front() + << ", are not counted paths in " << graph_file << ", so their regions are ignored" << endl; + } + + vector outputs = output_paths(gbz, path_prefixes); + if (outputs.empty()) { + logger.warn() << "no " << (path_prefixes.empty() ? "gref paths" : "paths chosen with -Q") << " in " + << graph_file << ", so no depth is written" << endl; + } + size_t rows = 0; + const gbwtgraph::GBWTGraph& graph = gbz.graph; + // The nodes that the output paths visit, by node ID minus the smallest. + nid_t min_id = graph.get_node_count() > 0 ? graph.min_node_id() : 0; + vector on_output(graph.get_node_count() > 0 ? graph.max_node_id() - min_id + 1 : 0, false); + for (const OutputPath& path : outputs) { + PathSteps steps(graph, path.id); + for (const PathSteps::Step& step : steps.steps()) { + on_output[graph.get_id(step.handle) - min_id] = true; + } + depth.for_each_run(steps, path.start, [&](uint64_t start, uint64_t end, uint32_t value) { + cout << path.name << '\t' << start << '\t' << end << '\t' << value << '\n'; + rows++; + }); + } + cout.flush(); + if (progress) { + cerr << "[vg lift] " << rows << " depth rows on " << outputs.size() << " paths" << endl; + } + + // Say how much of what the haplotypes read the rows leave out, such as + // haplotype sequence in gref fragments too short to keep. + uint64_t read = 0; + uint64_t left_out = 0; + graph.for_each_handle([&](const handle_t& handle) { + nid_t id = graph.get_id(handle); + uint64_t bases = depth.bases_read(id, static_cast(graph.get_length(handle))); + read += bases; + if (!on_output[id - min_id]) { + left_out += bases; + } + }); + if (left_out > 0) { + logger.info() << left_out << " of the " << read << " " << (options.count_visits ? "visited" : "haplotype") + << " bases counted are on nodes that no output path visits, so the rows leave them out" + << endl; + } +} + +//------------------------------------------------------------------------------ + +int main_lift(int argc, char** argv) { + Logger logger("vg lift"); + if (argc == 2) { + help_lift(argv); + return 1; + } + LiftConfig config(argc, argv, logger); + if (config.depth_mode) { + gbwtgraph::GBZ gbz; + load_gbz(gbz, config.graph_file, config.progress); + write_depth(gbz, config.graph_file, config.input_file, config.depth_options, config.path_prefixes, logger, + config.progress); + return 0; + } + unordered_map top_level_snarls = read_contig_snarls(config.gref_segs_file, logger); + + gbwtgraph::GBZ gbz; + load_gbz(gbz, config.graph_file, config.progress); + const gbwtgraph::GBWTGraph& graph = gbz.graph; + if (!config.tigs_file.empty()) { + // Tigs from another graph would name the wrong walks. + require_compatible_reference(config.tigs_file, nullptr, &gbz, false); + } + GrefIndex gref(gbz, GrefCover::gref_prefix); + const GrefIndex* gref_pointer = gref.path_count() > 0 ? &gref : nullptr; + if (config.progress) { + cerr << "[vg lift] gref: " << gref.path_count() << " paths, " << gref.total_length() << " bp" << endl; + } + + vector features; + optional lifted_vcf; + if (config.vcf_input) { + lifted_vcf.emplace(config.input_file, graph, gref, features, logger, config.progress); + } else { + features = lift_bed_features(config, gbz, gref_pointer, logger); + } + if (count_unnamed(features) > 0 || !config.assemblies_file.empty()) { + name_and_place_features(gbz, features, config, gref.path_count(), logger); + } + if (lifted_vcf.has_value()) { + lifted_vcf->write(features, top_level_snarls, logger); + } else { + write_feature_gaf(features, graph); + } + cout.flush(); + + if (!config.canonical_file.empty()) { + write_canonical(features, config.canonical_file, logger); + } + if (size_t nameless = count_unnamed(features); nameless > 0) { + logger.warn() << nameless << " features are not carried whole by any path, so they have no path coordinate" + << endl; + } + return 0; +} + +// Register subcommand +static Subcommand vg_lift("lift", "lift features on a cover or on paths to the graph, gref and assemblies", WIDGET, main_lift); diff --git a/src/vcf_alleles.cpp b/src/vcf_alleles.cpp new file mode 100644 index 0000000000..a86d4721da --- /dev/null +++ b/src/vcf_alleles.cpp @@ -0,0 +1,500 @@ +/** + * \file vcf_alleles.cpp + * VCF alleles as graph walks, see vcf_alleles.hpp. + */ + +#include "vcf_alleles.hpp" + +#include +#include +#include +#include +#include + +#include + +namespace vg { + +using handlegraph::handle_t; +using gbwtgraph::GBWTGraph; + +namespace { + +/// Split the text at each separator. +std::vector split(std::string_view text, char separator) { + std::vector parts; + size_t start = 0; + size_t end = text.find(separator); + while (end != std::string_view::npos) { + parts.push_back(text.substr(start, end - start)); + start = end + 1; + end = text.find(separator, start); + } + parts.push_back(text.substr(start)); + return parts; +} + +/** + * Return the ALT walk in a PanGenie variant ID, or nothing if the ID is not one. + * + * The ID is CHROM-POS-TYPE-WALK-LENGTH, where TYPE is SNV, INS, DEL or COMPLEX. + */ +std::optional pangenie_walk(std::string_view id) { + // The walk and the length hold no dashes, so we find the fields from the + // end. The chromosome name may hold dashes. + size_t length_dash = id.rfind('-'); + if (length_dash == std::string_view::npos || length_dash == 0) { + return std::nullopt; + } + size_t walk_dash = id.rfind('-', length_dash - 1); + if (walk_dash == std::string_view::npos || walk_dash == 0) { + return std::nullopt; + } + size_t type_dash = id.rfind('-', walk_dash - 1); + if (type_dash == std::string_view::npos) { + return std::nullopt; + } + std::string_view type = id.substr(type_dash + 1, walk_dash - type_dash - 1); + std::string_view walk = id.substr(walk_dash + 1, length_dash - walk_dash - 1); + std::string_view length = id.substr(length_dash + 1); + bool is_type = (type == "SNV" || type == "INS" || type == "DEL" || type == "COMPLEX"); + bool is_walk = (!walk.empty() && (walk[0] == '>' || walk[0] == '<') && + walk.find_first_not_of("<>0123456789") == std::string_view::npos); + bool is_length = (!length.empty() && length.find_first_not_of("0123456789") == std::string_view::npos); + return is_type && is_walk && is_length ? std::optional(walk) : std::nullopt; +} + +/// Return true if the sequence contains the bases, ignoring case. +bool contains_bases(std::string_view sequence, std::string_view bases) { + auto same = [](char x, char y) { + return std::toupper(static_cast(x)) == std::toupper(static_cast(y)); + }; + return bases.empty() || std::search(sequence.begin(), sequence.end(), bases.begin(), bases.end(), same) != sequence.end(); +} + +/** + * Return true if the sequence of an allele's walk holds its ALT column. + * + * VCF puts a padding base before an indel, or after one at the start of a + * contig, and the walk lacks it. Tools such as PanGenie's panel preparation + * also trim the bases that ALT shares with REF from the columns, but keep the + * whole nodes in the walk. + */ +bool holds_allele(std::string_view sequence, std::string_view ref, std::string_view alt) { + if (contains_bases(sequence, alt)) { + return true; + } + if (alt.size() <= 1) { + // A one-base ALT against a REF of the same length is a substitution, which + // the search above has tested against the walk. Against a longer REF it is + // taken as untestable rather than tested: it is usually VCF's padding base, + // left in the column by trimming that took the allele's own sequence out, + // and where the ALT is a placeholder such as * there is no sequence to test + // at all. + return ref.size() > alt.size(); + } + // One end of the ALT is the padding base, which lies outside the walk. + return contains_bases(sequence, alt.substr(1)) || + contains_bases(sequence, alt.substr(0, alt.size() - 1)); +} + +/// Return the value of the given field, counting from 0, in a sample column +/// whose fields are separated by colons, or nothing if it has fewer fields. +std::string_view sample_field(std::string_view column, size_t field) { + std::string_view value = column; + for (size_t skipped = 0; skipped < field; skipped++) { + size_t colon = value.find(':'); + if (colon == std::string_view::npos) { + return {}; + } + value = value.substr(colon + 1); + } + return value.substr(0, value.find(':')); +} + +} + +//------------------------------------------------------------------------------ + +bool is_variant_file(const std::string& path) { + // Probe a regular file only. Opening a FIFO or a process substitution reads + // its first block into htslib's format-detection buffer, and closing the + // file throws that block away, so whoever opens the path next starts in the + // middle. A VCF has to be a regular file here in any case, since the lift + // reads it twice. + struct stat info; + if (stat(path.c_str(), &info) != 0 || !S_ISREG(info.st_mode)) { + return false; + } + htsFile* file = hts_open(path.c_str(), "r"); + if (file == nullptr) { + return false; + } + bool result = (hts_get_format(file)->category == variant_data); + hts_close(file); + return result; +} + +VcfCursor::VcfCursor(const std::string& path, int threads) { + file = hts_open(path.c_str(), "r"); + if (file == nullptr) { + throw std::runtime_error("could not open " + path); + } + const htsFormat* format = hts_get_format(file); + if (format->format == bcf) { + hts_close(file); + throw std::runtime_error(path + " is BCF, which vg reads only as VCF"); + } + if (format->compression == bgzf && threads > 1) { + hts_set_threads(file, threads); + } +} + +VcfCursor::~VcfCursor() { + ks_free(&buffer); + hts_close(file); +} + +bool VcfCursor::next() { + do { + int status = hts_getline(file, '\n', &buffer); + if (status == -1) { + return false; + } + if (status < -1) { + throw std::runtime_error("could not read the VCF"); + } + if (buffer.l > 0 && buffer.s[buffer.l - 1] == '\r') { + buffer.s[--buffer.l] = '\0'; + } + } while (buffer.l == 0); + if (at_header()) { + return true; + } + std::string_view text = line(); + size_t start = 0; + for (size_t i = 0; i < FIXED_COLUMNS; i++) { + size_t end = text.find('\t', start); + if (end == std::string_view::npos) { + if (i + 1 < FIXED_COLUMNS) { + throw std::runtime_error("VCF record with fewer than 8 columns: " + std::string(text.substr(0, 100))); + } + end = text.size(); + } + columns[i] = { start, end }; + start = end + 1; + } + return true; +} + +bool VcfCursor::at_header() const { + return buffer.l > 0 && buffer.s[0] == '#'; +} + +std::string_view VcfCursor::line() const { + return std::string_view(buffer.s, buffer.l); +} + +std::string_view VcfCursor::column(Column column) const { + return line().substr(columns[column].first, columns[column].second - columns[column].first); +} + +std::string_view VcfCursor::after_info() const { + return line().substr(columns[INFO].second); +} + +std::vector VcfCursor::alt_alleles() const { + std::string_view alt = column(ALT); + if (alt == ".") { + return {}; + } + return split(alt, ','); +} + +std::optional VcfCursor::info_value(std::string_view key) const { + std::string_view info = column(INFO); + if (info == ".") { + return std::nullopt; + } + for (std::string_view entry : split(info, ';')) { + size_t equals = entry.find('='); + if (entry.substr(0, equals) == key) { + return equals == std::string_view::npos ? std::string_view() : entry.substr(equals + 1); + } + } + return std::nullopt; +} + +std::vector VcfCursor::genotypes() const { + std::string_view rest = after_info(); + // The FORMAT column, and then the sample columns, each after a tab. + std::vector format_and_samples = split(rest.substr(std::min(rest.size(), 1)), '\t'); + if (std::string_view format = format_and_samples.front(); rest.empty() || format.substr(0, format.find(':')) != "GT") { + return {}; + } + std::vector result; + result.reserve(format_and_samples.size() - 1); + for (size_t i = 1; i < format_and_samples.size(); i++) { + result.push_back(format_and_samples[i].substr(0, format_and_samples[i].find(':'))); + } + return result; +} + +std::vector VcfCursor::sample_values(std::string_view key) const { + std::string_view rest = after_info(); + if (rest.empty()) { + return {}; + } + std::vector format_and_samples = split(rest.substr(1), '\t'); + std::vector keys = split(format_and_samples.front(), ':'); + size_t field = std::find(keys.begin(), keys.end(), key) - keys.begin(); + if (field == keys.size()) { + return {}; + } + std::vector result; + result.reserve(format_and_samples.size() - 1); + for (size_t i = 1; i < format_and_samples.size(); i++) { + result.push_back(sample_field(format_and_samples[i], field)); + } + return result; +} + +std::string VcfCursor::locus() const { + return std::string(column(CHROM)) + ":" + std::string(column(POS)); +} + +//------------------------------------------------------------------------------ + +namespace { + +/// Throw an error about the record at the cursor. +[[noreturn]] void fail_record(const VcfCursor& cursor, const std::string& problem) { + throw std::runtime_error("VCF record at " + cursor.locus() + ": " + problem); +} + +/** + * Return the walk of each ALT allele of the record at the cursor, as text. + * + * The walks come from INFO/AT, which starts with the REF walk, or else from + * the ID of a record with one ALT allele, if that is a PanGenie variant ID. A + * record with neither has none. + * + * @throws std::runtime_error if INFO/AT does not have a walk for each allele. + */ +std::vector alt_walks(const VcfCursor& cursor, size_t alt_count) { + std::vector walks; + if (std::optional at = cursor.info_value("AT")) { + walks = split(*at, ','); + if (walks.size() != alt_count + 1) { + fail_record(cursor, "INFO/AT has " + std::to_string(walks.size()) + " walks for " + + std::to_string(alt_count + 1) + " alleles"); + } + walks.erase(walks.begin()); + } else if (alt_count == 1) { + if (std::optional walk = pangenie_walk(cursor.column(VcfCursor::ID))) { + walks.push_back(*walk); + } + } + return walks; +} + +/** + * Give an ALT allele the walk in the text, and the interval of its bases if + * the walk has nodes between its first and last. + * + * @param alt The ALT column of the allele. + * @throws std::runtime_error if the walk is not in the graph, or has fewer + * than two nodes. + */ +void add_walk(VcfAllele& allele, std::string_view walk, std::string_view alt, const VcfCursor& cursor, + const GBWTGraph& graph) { + try { + allele.walk = parse_walk(walk, graph); + } catch (const std::runtime_error& e) { + fail_record(cursor, e.what()); + } + if (allele.walk.size() < 2) { + fail_record(cursor, "allele walk " + std::string(walk) + " has fewer than two nodes"); + } + // The allele is the walk without its first and last nodes, which lie on + // the reference on either side of the variant. + if (allele.walk.size() > 2) { + WalkInterval interval; + interval.walk.assign(allele.walk.begin() + 1, allele.walk.end() - 1); + interval.length = interval.walk_length(graph); + allele.holds_alt = holds_allele(interval.sequence(graph), cursor.column(VcfCursor::REF), alt); + allele.interval = std::move(interval); + } +} + +} + +VcfCalls read_vcf_calls(const std::string& path, const GBWTGraph& graph, int threads) { + VcfCalls calls; + std::vector& alleles = calls.alleles; + VcfCursor cursor(path, threads); + while (cursor.next()) { + if (cursor.at_header()) { + // The column names line ends with the sample names, after FORMAT. + if (std::vector names = split(cursor.line(), '\t'); + cursor.line().starts_with("#CHROM") && names.size() > 9) { + calls.samples.assign(names.begin() + 9, names.end()); + } + continue; + } + std::vector alts = cursor.alt_alleles(); + VcfSite& site = calls.sites.emplace_back(); + site.chrom = cursor.column(VcfCursor::CHROM); + try { + site.pos = std::stoull(std::string(cursor.column(VcfCursor::POS))); + } catch (const std::exception&) { + fail_record(cursor, "bad POS"); + } + site.ref_length = cursor.column(VcfCursor::REF).size(); + site.first_allele = alleles.size(); + site.alt_count = alts.size(); + + std::vector walks = alt_walks(cursor, alts.size()); + for (size_t i = 0; i < alts.size(); i++) { + VcfAllele& allele = alleles.emplace_back(); + allele.name = cursor.column(VcfCursor::ID); + if (alts.size() > 1) { + allele.name += ":" + std::to_string(i + 1); + } + allele.record = calls.sites.size() - 1; + if (i < walks.size() && walks[i] != ".") { + add_walk(allele, walks[i], alts[i], cursor, graph); + } + } + } + return calls; +} + +namespace { + +/// Return the start of the declaration on a header line, up to the end of its +/// ID, or nothing if the line declares nothing with an ID. +std::string_view declaration_of(std::string_view line) { + size_t id = line.find("=", id)) + : std::string_view(); +} + +/** + * An editor that adds INFO keys with a value for each ALT allele to the lines + * of a VCF. + * + * In the header, it declares the added keys just before the column names, in + * place of any old declarations, along with the added header lines that the + * VCF lacks. In each record, it replaces any old values of the keys with the + * values of the record's ALT alleles. + */ +class InfoEditor { +public: + /// Make an editor that adds the INFO keys and the header lines. + InfoEditor(const std::vector& info, const std::vector& header) : + info(info), header(header) { + } + + /// Write a header line. Write the added declarations before the column + /// names line. + void write_header_line(std::string_view line, std::ostream& out) { + std::string_view declaration = declaration_of(line); + if (declaration.starts_with("##INFO= allele_count) { + throw std::runtime_error("VCF record at " + cursor.locus() + " has more ALT alleles than when it was read"); + } + editor.edit_info(cursor.column(VcfCursor::INFO), first, count, fields); + size_t info_start = cursor.column(VcfCursor::INFO).data() - line.data(); + out << line.substr(0, info_start) << fields << cursor.after_info() << '\n'; + first += count; + if (listener) { + listener(record, cursor); + } + record++; + } + if (first != allele_count) { + throw std::runtime_error("the VCF has fewer ALT alleles than when it was read"); + } +} + +} diff --git a/src/vcf_alleles.hpp b/src/vcf_alleles.hpp new file mode 100644 index 0000000000..37874e2e19 --- /dev/null +++ b/src/vcf_alleles.hpp @@ -0,0 +1,186 @@ +#ifndef VG_VCF_ALLELES_HPP_INCLUDED +#define VG_VCF_ALLELES_HPP_INCLUDED + +/** + * \file vcf_alleles.hpp + * + * The ALT alleles of a VCF of graph variants, as walks in the graph. + * + * Tools that describe variants in a graph record the walk of each allele. vg + * deconstruct writes the walk of every allele of a record in INFO/AT. + * PanGenie's panel preparation splits sites into biallelic variants named + * CHROM-POS-TYPE-WALK-LENGTH, with the ALT walk in the name, and PanGenie keeps + * that name as the ID of each call. + * + * The first and last nodes of each walk lie on the reference, on either side + * of the variant. For vg deconstruct they are the boundary nodes of the + * variant's snarl. The allele is the walk without them. It spells the ALT + * sequence without its padding base, and with any bases that a tool trimmed + * from both REF and ALT. + */ + +#include "walk_interval.hpp" + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace vg { + +/// Return true if the path names a regular file that htslib reads as VCF or +/// BCF. Anything else, a FIFO among them, reads as false. +bool is_variant_file(const std::string& path); + +/** + * A cursor over the lines of a VCF file, plain or compressed. + * + * On a record line, it splits off the eight fixed columns, from CHROM to INFO. + * It leaves the sample columns as they are, so a VCF with many samples reads + * quickly and can be copied exactly. + */ +class VcfCursor { +public: + /// The fixed columns of a record line, in order. + enum Column { CHROM, POS, ID, REF, ALT, QUAL, FILTER, INFO, FIXED_COLUMNS }; + + /** + * Open the VCF and decompress it with the given number of threads. + * + * @throws std::runtime_error if the file cannot be opened or is BCF. + */ + VcfCursor(const std::string& path, int threads); + ~VcfCursor(); + VcfCursor(const VcfCursor&) = delete; + VcfCursor& operator=(const VcfCursor&) = delete; + + /** + * Move to the next line, skipping empty ones. + * + * @returns false at the end of the file. + * @throws std::runtime_error on a read error, or on a record line with + * fewer than eight columns. + */ + bool next(); + + /// Return true if the line is a header line, which starts with #. + bool at_header() const; + /// Return the line, without its line break. + std::string_view line() const; + /// Return a fixed column of a record line. + std::string_view column(Column column) const; + /// Return the rest of a record line after its INFO column, from the tab + /// before FORMAT on. It is empty in a VCF without samples. + std::string_view after_info() const; + /// Return the ALT alleles of a record line. ALT "." has none. + std::vector alt_alleles() const; + /// Return the value of an INFO key on a record line, or nothing if the + /// record lacks the key. A flag has an empty value. + std::optional info_value(std::string_view key) const; + /// Return the GT of each sample on a record line, in sample order. Return + /// none if the record has no samples, or if FORMAT does not start with GT. + std::vector genotypes() const; + /// Return the value of a FORMAT key for each sample on a record line, in + /// sample order, empty where a sample leaves it out. Return none if the + /// record has no samples or no such key. + std::vector sample_values(std::string_view key) const; + /// Return the place of the record, as CHROM:POS. + std::string locus() const; + +private: + htsFile* file = nullptr; + /// The line, read by htslib. + kstring_t buffer = KS_INITIALIZE; + /// The [start, end) offsets of each fixed column in a record line. + std::array, FIXED_COLUMNS> columns; +}; + +/// An ALT allele of a VCF record, as a walk in the graph. +struct VcfAllele { + /// The ID of the record, followed by :N for the Nth ALT allele of a record + /// with several. + std::string name; + /// The index of the allele's record among the records of the VCF. + size_t record = 0; + /// The walk of the allele, from the node before the variant to the node + /// after it. Empty if the VCF does not give it. + std::vector walk; + /// The bases of the walk without its first and last nodes, if any nodes + /// are left. A deletion's walk has none. + std::optional interval; + /// True if the sequence of the interval holds the ALT column, apart from a + /// padding base. A walk from another graph usually does not. + bool holds_alt = true; +}; + +/// The place of a VCF record on its reference. +struct VcfSite { + std::string chrom; + /// The 1-based position of the first base of REF. + uint64_t pos = 0; + uint64_t ref_length = 0; + /// The index of the record's first ALT allele among the ALT alleles of the + /// VCF. The record's other ALT alleles follow it. + size_t first_allele = 0; + size_t alt_count = 0; +}; + +/// The records of a VCF of graph variants, with their ALT alleles as walks. +struct VcfCalls { + /// The sample names, in column order. + std::vector samples; + /// The place of each record, in file order. + std::vector sites; + /// The ALT alleles of all the records, in file order. + std::vector alleles; +}; + +/** + * Read the records of a VCF of graph variants, and the walks of their ALT + * alleles. + * + * Each allele's walk comes from INFO/AT. A record without AT and with one ALT + * allele takes the walk from its ID if that is a PanGenie variant ID. + * + * @throws std::runtime_error on a malformed VCF, on a walk that is not in the + * graph, or on a walk with fewer than two nodes. + */ +VcfCalls read_vcf_calls(const std::string& path, const gbwtgraph::GBWTGraph& graph, int threads); + +/// An INFO key with one value for each ALT allele of a VCF (Number=A). +struct VcfAlleleInfo { + std::string key; + /// The ##INFO line that declares the key. + std::string header; + /// The value for each ALT allele of the VCF, in file order. A "." is a + /// missing value. + std::vector values; +}; + +/** + * Copy a VCF to out with INFO keys added. + * + * The keys replace any values and declarations that the VCF has for them. A + * record gets no value for a key if all its alleles have "." there. + * + * @param header Header lines to add, each unless the VCF already declares its + * INFO key, FORMAT key or contig. + * @param listener Called with the index of each record and a cursor on it, + * after the record is written. + * @throws std::runtime_error if the VCF does not have one ALT allele for each + * value. + */ +void write_vcf_with_allele_info(const std::string& path, std::ostream& out, const std::vector& info, + const std::vector& header, + const std::function& listener, int threads); + +} + +#endif diff --git a/test/t/59_vg_lift.t b/test/t/59_vg_lift.t new file mode 100644 index 0000000000..51e567cb56 --- /dev/null +++ b/test/t/59_vg_lift.t @@ -0,0 +1,375 @@ +#!/usr/bin/env bash + +BASH_TAP_ROOT=../deps/bash-tap +. ../deps/bash-tap/bash-tap-bootstrap + +PATH=../bin:$PATH # for vg + +export LC_ALL="C" # force a consistent sort order + +plan tests 68 + +# A triallelic SNP with the haplotypes stored as W lines at offset 1000 of contig c, +# as Minigraph-Cactus stores haplotype fragments +awk -v OFS='\t' '$1 == "P" && $2 ~ /^hap/ { + n = split($3, steps, ",") + walk = "" + for (i = 1; i <= n; i++) { + walk = walk (substr(steps[i], length(steps[i])) == "+" ? ">" : "<") substr(steps[i], 1, length(steps[i]) - 1) + } + print "W", "s", ($2 == "hapA" ? 1 : 2), "c", 1000, 1201, walk + next +} { print }' nesting/core_snp_in_flanks.gfa > lift.gfa +vg paths -x lift.gfa -Q x --compute-gref > lift.gref.vg +vg convert -f lift.gref.vg > lift.gref.gfa +vg gbwt -G lift.gref.gfa --gbz-format -g lift.gbz +vg unitigs -r -k 11 -g lift.tigs.gaf lift.gbz > lift.tigs.fa + +# The same 5 bases on the gref and on both junction tigs, a stranded feature across the +# SNP, a feature with no BED name, and a duplicate line +printf 'gref_x\t92\t97\tg5\t0\t+\nu1\t2\t7\tg5\t0\t+\nu2\t2\t7\tg5\t0\t+\n' > lift.bed +printf 'u1\t8\t13\tsnpA\t5\t-\nu2\t8\t13\tsnpB\nu2\t8\t13\tsnpB\ngref_x\t0\t5\n' >> lift.bed +vg lift -u lift.tigs.gaf -a lift.asm.bed lift.gbz lift.bed > lift.gaf +is $? 0 "vg lift runs on a cover of gref paths and junction tigs" + +is "$(awk 'END { print NR }' lift.gaf)" "4" "copies of a feature on several cover sequences are kept once" + +is "$(grep -c $'\tsp:i:3\tgr:Z:gref_x:93-97:+:5=\tpc:Z:gref_x:93-97:+\tan:Z:g5$' lift.gaf)" "1" "a feature on the gref is placed at its gref coordinates, which are also its path coordinate" + +is "$(grep -c $'<6<4<2\t.*\tgr:Z:gref_x:99-103:-:2=1X2=\tpc:Z:s#1#c:1099-1103:-\tan:Z:snpA$' lift.gaf)" "1" "a feature on the - strand of a tig is lifted to the reverse walk, with the SNP as a substitution" + +is "$(awk '$4 == "g5"' lift.asm.bed | cut -f 1-3,6 | sort | tr '\t\n' ' ;')" "s#1#c 1092 1097 +;s#2#c 1092 1097 +;x 92 97 +;" "a haplotype place is on its contig, past the fragment's offset" + +is "$(awk '$4 == "snpA"' lift.asm.bed | cut -f 1-6 | tr '\t' ' ')" "s#1#c 1098 1103 snpA 5 -" "a stranded feature keeps its fields and gets the strand of the path" + +vg paths -x lift.gbz -F -Q 's#1#c' | awk 'NR > 1' | tr -d '\n' | cut -c 99-103 > lift.place.txt +awk '$1 == ">u1" { getline; print substr($0, 9, 5) }' lift.tigs.fa > lift.feature.txt +is "$(cat lift.place.txt) $(awk '{ print length($0) }' lift.place.txt)" "$(cat lift.feature.txt) 5" "the bases at a place on a haplotype are those of the feature on the tig" + +is "$(awk '$4 == "." && $5 == 0 && $6 == "."' lift.asm.bed | awk 'END { print NR }')" "3" "a feature without a name, score or strand gets ., 0 and ." + +join -t $'\t' <(cut -f 1,13 lift.gaf | sed 's/sp:i://' | sort) <(awk -v OFS='\t' '{ print $NF }' lift.asm.bed | sort | uniq -c | awk -v OFS='\t' '{ print $2, $1 }') > lift.support.txt +is "$(awk -F '\t' '$2 == $3 { n++ } END { print n + 0 }' lift.support.txt) $(cut -f 1 lift.gaf | sort | uniq -d | awk 'END { print NR }')" "4 0" "each feature has a unique ID and its support counts its places on paths" + +vg lift -u lift.tigs.gaf -a lift.s1.bed -Q 's#1' lift.gbz lift.bed > /dev/null +is "$(cut -f 1 lift.s1.bed | sort -u)" "s#1#c" "places can be limited to paths by name prefix" + +vg lift -u lift.tigs.gaf -t 1 -a lift.t1.bed lift.gbz lift.bed > lift.t1.gaf +vg lift -u lift.tigs.gaf -t 4 -a lift.t4.bed lift.gbz lift.bed > lift.t4.gaf +cmp -s lift.t1.gaf lift.t4.gaf && cmp -s lift.t1.bed lift.t4.bed +is $? 0 "the output does not depend on the thread count" + +# Path coordinates: a feature on the gref and on tigs, features off the gref on tigs, and +# features on a path, along the gref and off it +cp lift.bed lift.names.input.bed +printf 'u2\t9\t11\tcg\nx\t95\t105\talonggref\ns#1#c\t1098\t1103\toffgref\n' >> lift.names.input.bed +vg lift -u lift.tigs.gaf -c lift.names.bed lift.gbz lift.names.input.bed > lift.names.gaf 2> lift.names.err +is "$(grep -c $'\tpc:Z:s#2#c:1100-1101:+\tan:Z:cg$' lift.names.gaf)" "1" "a feature off the gref is named on the path that carries it, in contig coordinates" + +is "$(grep -c $'\tpc:Z:gref_x:96-105:+\tan:Z:alonggref$' lift.names.gaf)" "1" "a feature read from a path along the gref is named on the gref" + +is "$(grep -c $'\tpc:Z:s#1#c:1099-1103:+\tan:Z:offgref$' lift.names.gaf) $(grep -c . lift.names.err)" "1 0" "a feature read from a path off the gref is named on a path that carries it" + +is "$(cut -f 1-4 lift.names.bed | tr '\t\n' ' ;')" "gref_x 0 5 .;gref_x 92 97 g5;gref_x 95 105 alonggref;s#1#c 1098 1103 offgref;s#1#c 1098 1103 snpA;s#2#c 1098 1103 snpB;s#2#c 1099 1101 cg;" "the canonical BED has each feature once, at its path coordinate, in priority order" + +grep -v '^@' lift.tigs.gaf > lift.tigs.noheader.gaf +vg lift -u lift.tigs.noheader.gaf lift.gbz lift.names.input.bed 2> /dev/null | cmp -s - lift.names.gaf +is $? 0 "a tig GAF without a header gives the same output" + +printf 'u99\t0\t5\n' > lift.unknown.bed +vg lift -u lift.tigs.gaf lift.gbz lift.unknown.bed > /dev/null 2>&1 +is $? 1 "a feature on an unknown sequence is an error" + +printf 'u1\t10\t30\n' > lift.long.bed +vg lift -u lift.tigs.gaf lift.gbz lift.long.bed 2>&1 > /dev/null | grep -q "past the end" +is $? 0 "a feature past the end of its sequence is an error" + +vg lift lift.gbz lift.bed 2>&1 > /dev/null | grep -q "use -u" +is $? 0 "a feature on a tig needs the tig GAF" + +# A plain cover, without a gref +vg gbwt -G lift.gfa --gbz-format -g lift.plain.gbz +vg unitigs -k 11 -g lift.plain.gaf lift.plain.gbz > lift.plain.fa +printf 'u1\t0\t5\tp\nu3\t9\t14\tq\n' > lift.plain.bed +vg lift -u lift.plain.gaf -a lift.plain.asm.bed lift.plain.gbz lift.plain.bed > lift.plain.out.gaf +is "$(grep -c 'gr:Z:' lift.plain.out.gaf)" "0" "features on a plain cover have no gref placement" + +is "$(awk 'END { print NR }' lift.plain.asm.bed)" "4" "features on a plain cover are lifted to every path that visits them" + +# Path to path: features on paths, in full-path coordinates +printf 'x\t80\t120\tspan\t0\t+\ns#1#c\t1080\t1120\tfromA\t0\t-\n' > lift.paths.bed +vg lift -a lift.paths.exact.bed lift.gbz lift.paths.bed > /dev/null +is "$(awk '$4 == "span"' lift.paths.exact.bed | cut -f 1-3)" "$(printf 'x\t80\t120')" "a feature across a SNP has only its own exact place" + +vg lift -m 0.9 -a lift.paths.lift.bed lift.gbz lift.paths.bed > /dev/null +is "$(awk '$4 == "span" { print $1, $2, $3, $6, $NF }' lift.paths.lift.bed | tr '\n' ';')" "x 80 120 + 1;s#1#c 1080 1120 + 0.975;s#2#c 1080 1120 + 0.975;" "a liftover reaches the haplotypes with a SNP, at their contig coordinates" + +is "$(awk '$4 == "fromA" && $1 == "x" { print $2, $3, $6 }' lift.paths.lift.bed)" "80 120 -" "a feature on a haplotype contig lifts to the reference on its strand" + +vg lift -m 0.9 -b -a lift.paths.blocks.bed lift.gbz lift.paths.bed > /dev/null +is "$(awk '$4 == "span" && $1 == "s#1#c" { print $7, $8, $10, $11, $12 }' lift.paths.blocks.bed)" "1080 1120 2 20,19, 0,21," "blocks leave out the SNP base" + +printf 's#1#c\t10\t20\tbefore\n' > lift.outside.bed +vg lift lift.gbz lift.outside.bed 2>&1 > /dev/null | grep -q "1 features on paths are not within one fragment" +is $? 0 "a feature outside the fragments of its path is skipped with a warning" + +# A 50 bp insertion on the haplotypes, after base 70 of x +vg gbwt -G nesting/core_ins50.gfa --gbz-format -g lift.ins.gbz +printf 'x\t50\t110\twide\n' > lift.ins.bed +vg lift -m 0.9 -b -a lift.ins.blocks.bed lift.ins.gbz lift.ins.bed > /dev/null +is "$(awk '$1 == "hapA" { print $2, $3, $10, $11, $12, $NF }' lift.ins.blocks.bed)" "50 160 2 20,40, 0,70, 1" "a liftover spans an insertion shorter than the feature" + +printf 'x\t60\t100\tnarrow\n' > lift.ins.bed +vg lift -m 0.7 -a lift.ins.lift.bed lift.ins.gbz lift.ins.bed > /dev/null +is "$(awk '$1 == "hapA" { print $2, $3, $NF }' lift.ins.lift.bed)" "120 150 0.75" "an insertion longer than the feature splits it" + +printf 'x\t0\t10\tshared\n' > lift.shared.bed +is "$(vg lift lift.ins.gbz lift.shared.bed | grep -o 'pc:Z:[^[:space:]]*') $(vg lift -P x lift.ins.gbz lift.shared.bed | grep -o 'pc:Z:[^[:space:]]*')" "pc:Z:hapA:1-10:+ pc:Z:x:1-10:+" "a feature carried by several paths is named on the first in priority order" + +vg unitigs -k 11 -g lift.ins.tigs.gaf lift.ins.gbz > /dev/null +vg lift -u lift.ins.tigs.gaf lift.gbz lift.bed 2>&1 > /dev/null | grep -q "not compatible" +is $? 0 "tigs of another graph are rejected" + +# VCF input: the 50 bp insertion as a gref alt path, with the haplotypes as W lines at +# offset 1000 of contig c. Haplotype 2 has a SNP at the last base of the insertion. +awk -v OFS='\t' '$1 == "P" && $2 ~ /^hap/ { + n = split($3, steps, ",") + walk = "" + for (i = 1; i <= n; i++) { + walk = walk (substr(steps[i], length(steps[i])) == "+" ? ">" : "<") substr(steps[i], 1, length(steps[i]) - 1) + } + print "W", "s", ($2 == "hapA" ? 1 : 2), "c", 1000, 1190, walk + next +} { print }' nesting/core_ins50.gfa > lift.vcf.gfa +vg paths -x lift.vcf.gfa -Q x --compute-gref > lift.vcf.gref.vg +vg convert -f lift.vcf.gref.vg > lift.vcf.gref.gfa +vg gbwt -G lift.vcf.gref.gfa --gbz-format -g lift.vcf.gbz +vg deconstruct -P x -a lift.vcf.gbz > lift.decon.vcf 2> /dev/null +vg lift -c lift.decon.bed lift.vcf.gbz lift.decon.vcf > lift.decon.lifted.vcf +is $? 0 "vg lift reads the allele walks of a VCF from vg deconstruct" + +is "$(grep '^x' lift.decon.lifted.vcf | cut -f 8 | tr ';' '\n' | grep -E '^(GR|PC)=' | tr '\n' ' ')" "GR=gref_x_1_alt:1-50:+:50=,gref_x_1_alt:1-50:+:49=1X PC=gref_x_1_alt:1-50:+,s#2#c:1071-1120:+ " "each ALT allele, its walk without the first and last nodes, is placed and named on the first path that carries it, and an allele that leaves the alt contig early and rejoins the reference past its end replaces the end of the contig" + +is "$(grep -c -E '^##INFO=3>5>6-1 A G 0|1 AT=>3>4>6,>3>5>6 AN=2 CH=1 RC=x RS=70 " "a SNP at the end of an insertion, where the walk leaves the alt contig, is broken out as a phased record on the contig" + +is "$(grep -c '^##contig=' lift.decon.lifted.vcf)" "1" "the lifted VCF declares the alt contigs of its nested records" + +is "$(cut -f 1-6 lift.decon.bed | tr '\t\n' ' ;')" "gref_x_1_alt 0 50 >2>6:1 0 +;s#2#c 1070 1120 >2>6:2 0 +;s#2#c 1119 1120 gref_x_1_alt-50-SNV->3>5>6-1 0 +;" "the canonical BED names each allele by its record ID and number, and each nested variant by its ID" + +# PanGenie calls: the ALT walk is in the ID, and there is no AT +INS1=$(grep -v '^#' lift.decon.vcf | cut -f 5 | cut -d , -f 1) +INS2=$(grep -v '^#' lift.decon.vcf | cut -f 5 | cut -d , -f 2) +printf '##fileformat=VCFv4.2\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\n' > lift.pg.vcf +printf 'x\t70\tx-70-INS->2>3>4>6-50\tT\t%s\t.\tPASS\tAC=1\n' "$INS1" >> lift.pg.vcf +printf 'x\t70\tx-70-INS->2>3>5>6-50\tT\t%s\t.\tPASS\t.\n' "$INS2" >> lift.pg.vcf +printf 'x\t70\tx-70-DEL->2>6-1\tTA\tT\t.\tPASS\t.\nx\t70\tnowalk\tT\tTA\t.\tPASS\t.\n' >> lift.pg.vcf +vg lift lift.vcf.gbz lift.pg.vcf > lift.pg.lifted.vcf 2> lift.pg.err +is "$(grep '^x' lift.pg.lifted.vcf | cut -f 3,8 | tr '\t\n' ' ;')" "x-70-INS->2>3>4>6-50 AC=1;GR=gref_x_1_alt:1-50:+:50=;PC=gref_x_1_alt:1-50:+;x-70-INS->2>3>5>6-50 GR=gref_x_1_alt:1-50:+:49=1X;PC=s#2#c:1071-1120:+;x-70-DEL->2>6-1 .;nowalk .;" "PanGenie IDs give the walks of calls without AT, and a deletion has no bases to place" + +is "$(grep '^gref' lift.pg.lifted.vcf | awk -F '\t' '{ print $1, $2, $4, $5, NF }')" "gref_x_1_alt 50 A G 8" "calls without samples give nested records without genotypes" + +# Nested genotypes take the lowest GQ of the calls that give them, and PS from the gref segments. +vg paths -x lift.vcf.gfa -Q x --compute-gref --gref-segs lift.vcf.segs.tsv > /dev/null +printf '##fileformat=VCFv4.2\n##FORMAT=\n' > lift.gq.vcf +printf '##FORMAT=\n' >> lift.gq.vcf +printf '#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tA\tB\tC\n' >> lift.gq.vcf +printf 'x\t70\tx-70-INS->2>3>4>6-50\tT\t%s\t.\tPASS\t.\tGT:GQ\t0/1:40\t1/1:12\t0/0:50\n' "$INS1" >> lift.gq.vcf +printf 'x\t70\tx-70-INS->2>3>5>6-50\tT\t%s\t.\tPASS\t.\tGT:GQ\t0/1:20\t0/0:9\t0/1\n' "$INS2" >> lift.gq.vcf +vg lift --gref-segs lift.vcf.segs.tsv lift.vcf.gbz lift.gq.vcf > lift.gq.lifted.vcf +is "$(grep '^gref' lift.gq.lifted.vcf | cut -f 8 | tr ';' '\n' | grep '^PS=') $(grep '^gref' lift.gq.lifted.vcf | cut -f 9- | tr '\t' ' ')" "PS=>1>7 GT:GQ 0/1:20 0/0:12 1/.:." "nested genotypes take the lowest GQ of their calls, or none if a call has none, and PS names the top-level snarl of the contig" + +vg lift --gref-segs lift.vcf.segs.tsv lift.vcf.gbz lift.bed 2>&1 | grep -q "needs a VCF" +is $? 0 "--gref-segs needs a VCF" + +# The haplotypes differ at the first base of the insertion instead of the last. +printf 'H\tVN:Z:1.0\nS\t1\tGGATCACAGT\nS\t2\tAGTCCGAGGA\nS\t3\tA\nS\t4\tG\n' > lift.head.gfa +printf 'S\t5\tTCTCAGCAGCGACAGCGCATGATCGTAGCAGCACTATGTGACTGCACTC\nS\t6\tCGTCAATACG\n' >> lift.head.gfa +for edge in 1,2 2,6 2,3 2,4 3,5 4,5 5,6; do + printf 'L\t%s\t+\t%s\t+\t0M\n' ${edge%,*} ${edge#*,} >> lift.head.gfa +done +printf 'P\tx\t1+,2+,6+\t*\nW\ts\t1\tc\t1000\t1080\t>1>2>3>5>6\nW\ts\t2\tc\t1000\t1080\t>1>2>4>5>6\n' >> lift.head.gfa +vg paths -x lift.head.gfa -Q x --compute-gref > lift.head.gref.vg +vg convert -f lift.head.gref.vg > lift.head.gref.gfa +vg gbwt -G lift.head.gref.gfa --gbz-format -g lift.head.gbz +vg deconstruct -P x -a lift.head.gbz > lift.head.vcf 2> /dev/null +vg lift lift.head.gbz lift.head.vcf > lift.head.lifted.vcf +is "$(grep -o 'GR=[^;]*' lift.head.lifted.vcf | head -1) $(grep '^gref' lift.head.lifted.vcf | cut -f 1,2,4,5 | tr '\t' ' ')" "GR=gref_x_1_alt:1-50:+:50=,gref_x_1_alt:1-50:+:1X49= gref_x_1_alt 1 A G" "an allele that joins the alt contig late from the reference replaces the start of the contig" + +printf 's#2#c\t1040\t1150\trev\t0\t-\n' > lift.rev.bed +is "$(vg lift lift.vcf.gbz lift.rev.bed | grep -o 'gr:Z:[^[:space:]]*')" "gr:Z:gref_x:71-100:-:30=,gref_x_1_alt:1-50:-:49=1X,gref_x:41-70:-:30=" "a walk that reads an alt contig in reverse replaces its start the same way" + +is "$(grep -c '1 ALT alleles have no walk' lift.pg.err)" "1" "alleles without walks are counted" + +bgzip -c lift.pg.vcf > lift.pg.vcf.gz +vg lift lift.vcf.gbz lift.pg.vcf.gz 2> /dev/null | cmp -s - lift.pg.lifted.vcf +is $? 0 "a compressed VCF gives the same output" + +vg lift lift.vcf.gbz lift.pg.lifted.vcf 2> /dev/null | cmp -s - lift.pg.lifted.vcf +is $? 0 "lifting a lifted VCF replaces its tags" + +printf '##fileformat=VCFv4.2\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\nx\t70\tx-70-INS->2>3>4>6-50\tT\tTAAAA\t.\tPASS\t.\n' > lift.foreign.vcf +vg lift lift.vcf.gbz lift.foreign.vcf 2>&1 > /dev/null | grep -q "is the VCF from this graph" +is $? 0 "an allele that is not in the sequence of its walk gets a warning" + +printf '##fileformat=VCFv4.2\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\nx\t70\tx-70-INS->2>99>6-1\tT\tTA\t.\tPASS\t.\n' > lift.foreign.vcf +vg lift lift.vcf.gbz lift.foreign.vcf 2>&1 > /dev/null | grep -q "node 99 is not in the graph" +is $? 0 "a walk through a node that is not in the graph is an error" + +# Nested variants: a 63 bp insertion with three SNP sites, as PanGenie calls of two +# samples. Haplotype s#1 has the insertion that becomes the gref alt contig, s#2 has all +# three SNPs, t#2 has the middle one, and t#1 lacks the insertion. +printf 'H\tVN:Z:1.0\nS\t1\tACGTACGTACGTACGTACGT\nS\t2\tTTGACCAGTAGGCAT\nS\t3\tA\nS\t13\tG\nS\t4\tCCGATTAGCATGACG\n' > lift.snp3.gfa +printf 'S\t5\tC\nS\t15\tT\nS\t6\tGGTACCTAGGATCCA\nS\t7\tG\nS\t17\tA\nS\t8\tTAGCTAGGCTTACGA\nS\t9\tCATGCATGCATGCATGCATG\n' >> lift.snp3.gfa +for edge in 1,2 1,9 2,3 2,13 3,4 13,4 4,5 4,15 5,6 15,6 6,7 6,17 7,8 17,8 8,9; do + printf 'L\t%s\t+\t%s\t+\t0M\n' ${edge%,*} ${edge#*,} >> lift.snp3.gfa +done +printf 'P\tx\t1+,9+\t*\nW\ts\t1\tc\t1000\t1103\t>1>2>3>4>5>6>7>8>9\nW\ts\t2\tc\t1000\t1103\t>1>2>13>4>15>6>17>8>9\n' >> lift.snp3.gfa +printf 'W\tt\t1\tc\t1000\t1040\t>1>9\nW\tt\t2\tc\t1000\t1103\t>1>2>3>4>15>6>7>8>9\n' >> lift.snp3.gfa +vg paths -x lift.snp3.gfa -Q x --compute-gref > lift.snp3.gref.vg +vg convert -f lift.snp3.gref.vg > lift.snp3.gref.gfa +vg gbwt -G lift.snp3.gref.gfa --gbz-format -g lift.snp3.gbz +# The ALT sequence of a walk, after the padding base T +ins() { + for node in "$@"; do + awk -v node=$node '$1 == "S" && $2 == node { printf "%s", $3 }' lift.snp3.gfa + done +} +call() { + printf 'x\t20\tx-20-INS-%s-63\tT\tT%s\t.\tPASS\t.\tGT\t%s\t%s\n' "$1" "$(ins $(echo $1 | tr '<>' ' ' | cut -d ' ' -f 3-9))" "$2" "$3" +} +printf '##fileformat=VCFv4.2\n##FORMAT=\n' > lift.snp3.vcf +printf '#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\ts\tt\n' >> lift.snp3.vcf +call '>1>2>3>4>5>6>7>8>9' 0/1 0/0 >> lift.snp3.vcf +call '>1>2>13>4>15>6>17>8>9' 0/1 0/0 >> lift.snp3.vcf +call '>1>2>3>4>15>6>7>8>9' 0/0 0/1 >> lift.snp3.vcf +vg lift lift.snp3.gbz lift.snp3.vcf > lift.snp3.lifted.vcf 2> lift.snp3.err +is "$(grep -c 'not in the sequences of their walks' lift.snp3.err)" "0" "the ALT alleles are the sequences of their walks, so the VCF reads as this graph's" +is "$(grep '^gref' lift.snp3.lifted.vcf | cut -f 1,2,4,5,10,11 | tr '\t\n' ' ;')" "gref_x_1_alt 16 A G 0/1 0/.;gref_x_1_alt 32 C T 0/1 1/.;gref_x_1_alt 48 G A 0/1 0/.;" "SNPs in a called insertion are broken out as records on its alt contig, with . for haplotypes without the contig" + +is "$(grep '^gref' lift.snp3.lifted.vcf | cut -f 8 | tr ';' '\n' | grep -E '^(AC|AN)=' | tr '\n' ' ')" "AC=1 AN=3 AC=2 AN=3 AC=1 AN=3 " "nested allele counts are among the haplotypes that carry the contig" + +sed 's/x-20-INS->1>2>13>4>15>6>17>8>9-63/x-20-INS-<9<8<17<6<15<4<13<2<1-63/' lift.snp3.vcf > lift.snp3.rev.vcf +vg lift lift.snp3.gbz lift.snp3.rev.vcf 2> /dev/null | grep '^gref' | cmp -s - <(grep '^gref' lift.snp3.lifted.vcf) +is $? 0 "a walk that reads the alt contig backwards gives the same nested records" + +awk -F '\t' -v OFS='\t' '$3 == "x-20-INS->1>2>3>4>5>6>7>8>9-63" { $10 = "1/1" } { print }' lift.snp3.vcf > lift.snp3.conflict.vcf +vg lift lift.snp3.gbz lift.snp3.conflict.vcf > lift.snp3.conflict.lifted.vcf 2> lift.snp3.conflict.err +is "$(grep '^gref' lift.snp3.conflict.lifted.vcf | cut -f 10 | sort -u) $(grep -c 'CONFLICT=s;' lift.snp3.conflict.lifted.vcf) $(grep -c '3 nested genotypes are missing' lift.snp3.conflict.err)" "./. 3 1" "a sample with more walks along a contig than haplotypes gets missing genotypes" + +# Contig ends: t#2's insertion is the alt contig, s#1 lacks its first base, and s#2 leaves +# it after that base +printf 'H\tVN:Z:1.0\nS\t1\tACGTACGTACGTACGTACGT\nS\t21\tT\nS\t22\tCAGATTTTCATATTATGCAGAAAATCTACTTCGCCTGATACGAGTCGGTTATCTTCGGAT\n' > lift.ends.gfa +printf 'S\t9\tCATGCATGCATGCATGCATG\n' >> lift.ends.gfa +for edge in 1,21 1,22 1,9 21,22 21,9 22,9; do + printf 'L\t%s\t+\t%s\t+\t0M\n' ${edge%,*} ${edge#*,} >> lift.ends.gfa +done +printf 'P\tx\t1+,9+\t*\nW\ts\t1\tc\t1000\t1100\t>1>22>9\nW\ts\t2\tc\t1000\t1041\t>1>21>9\n' >> lift.ends.gfa +printf 'W\tt\t1\tc\t1000\t1040\t>1>9\nW\tt\t2\tc\t1000\t1101\t>1>21>22>9\n' >> lift.ends.gfa +vg paths -x lift.ends.gfa -Q x --compute-gref > lift.ends.gref.vg +vg convert -f lift.ends.gref.vg > lift.ends.gref.gfa +vg gbwt -G lift.ends.gref.gfa --gbz-format -g lift.ends.gbz +vg deconstruct -P x -a lift.ends.gbz > lift.ends.vcf 2> /dev/null +vg lift lift.ends.gbz lift.ends.vcf 2> /dev/null | grep '^gref' | awk -F '\t' '{ print $2, $4, length($4), $5, $10, $11 }' > lift.ends.txt +is "$(cut -d ' ' -f 1,3- lift.ends.txt | tr '\n' ';') $(head -1 lift.ends.txt | cut -d ' ' -f 2)" "1 2 C 1|0 .|0;1 61 T 0|1 .|0; TC" "variants at the start and the end of an alt contig are broken out, padded after at the start" + +# A walk on the reference that shares a contig's last node doesn't read the contig: node 4 is a +# SNP allele that s#1 carries and that ends t#2's insertion, the alt contig +printf 'H\tVN:Z:1.0\nS\t1\tACGTACGTACGTACGTACGT\nS\t2\tGATTACAGATTACAGATTAC\nS\t3\tA\nS\t4\tG\nS\t9\tCATGCATGCATGCATGCATG\n' > lift.touch.gfa +printf 'S\t5\tTTTCCTCATGCAATTCAAAACCATGTCCGT\nS\t6\tA\nS\t7\tCC\nS\t8\tAATGTAGGCGAAATAGTAAACCATTTTAC\n' >> lift.touch.gfa +for edge in 1,2 2,3 3,9 2,4 4,9 1,5 5,6 5,7 6,8 7,8 8,4; do + printf 'L\t%s\t+\t%s\t+\t0M\n' ${edge%,*} ${edge#*,} >> lift.touch.gfa +done +printf 'P\tx\t1+,2+,3+,9+\t*\nW\ts\t1\tc\t1000\t1061\t>1>2>4>9\nW\ts\t2\tc\t1000\t1061\t>1>2>3>9\n' >> lift.touch.gfa +printf 'W\tt\t1\tc\t1000\t1101\t>1>5>6>8>4>9\nW\tt\t2\tc\t1000\t1102\t>1>5>7>8>4>9\n' >> lift.touch.gfa +vg paths -x lift.touch.gfa -Q x --compute-gref > lift.touch.gref.vg +vg convert -f lift.touch.gref.vg > lift.touch.gref.gfa +vg gbwt -G lift.touch.gref.gfa --gbz-format -g lift.touch.gbz +vg deconstruct -P x -a lift.touch.gbz > lift.touch.vcf 2> /dev/null +is "$(vg lift lift.touch.gbz lift.touch.vcf 2> /dev/null | grep '^gref' | cut -f 2,4,5,10,11 | tr '\t' ' ' | tr '\n' ';')" "31 CC A ./. 1|0;" "a walk on the reference that shares a contig's last node does not read the contig" + +# A contig inside a contig: s#2 and t#1 detour from gref_x_1_alt onto gref_x_2_alt, which +# has a SNP between them +printf 'H\tVN:Z:1.0\nS\t1\tACGTACGTACGTACGTACGT\nS\t9\tCATGCATGCATGCATGCATG\n' > lift.nest.gfa +printf 'S\t2\tTTTCCTCATGCAATTCAAAACCATGTCCGTAATGTAGGCGAAATAGTAAACCATTTTACG\n' >> lift.nest.gfa +printf 'S\t3\tTGGCTAGTGTCACTGCGCACAGTAAACATTATCGCACATTTTTAACGGGTGAGCGGGCAT\n' >> lift.nest.gfa +printf 'S\t5\tGGCCCCCCACGATCAGCAGTTCGGCTTGTGAGGTCTTCGCCGGGTGGTCTCCCGCATTTA\n' >> lift.nest.gfa +printf 'S\t4\tACGGGATGTTTAGCGGGGCC\nS\t6\tA\nS\t7\tC\nS\t8\tCAACCAACGCAGTGGTGGCCGGCGTCTTT\n' >> lift.nest.gfa +for edge in 1,2 1,9 2,3 2,4 4,6 4,7 6,8 7,8 8,3 3,5 3,9 5,9; do + printf 'L\t%s\t+\t%s\t+\t0M\n' ${edge%,*} ${edge#*,} >> lift.nest.gfa +done +printf 'P\tx\t1+,9+\t*\nW\ts\t1\tc\t1000\t1220\t>1>2>3>5>9\nW\ts\t2\tc\t1000\t1210\t>1>2>4>6>8>3>9\n' >> lift.nest.gfa +printf 'W\tt\t1\tc\t1000\t1210\t>1>2>4>7>8>3>9\nW\tt\t2\tc\t1000\t1040\t>1>9\n' >> lift.nest.gfa +vg paths -x lift.nest.gfa -Q x --compute-gref > lift.nest.gref.vg +vg convert -f lift.nest.gref.vg > lift.nest.gref.gfa +vg gbwt -G lift.nest.gref.gfa --gbz-format -g lift.nest.gbz +vg deconstruct -P x -a lift.nest.gbz > lift.nest.vcf 2> /dev/null +vg lift lift.nest.gbz lift.nest.vcf 2> /dev/null | grep '^gref' > lift.nest.lifted.txt +is "$(awk -F '\t' '$1 == "gref_x_2_alt" { print $2, $4, $5, $10, $11 }' lift.nest.lifted.txt) $(grep '^gref_x_2_alt' lift.nest.lifted.txt | grep -o 'CH=[0-9]*')" "21 A C .|0 1|. CH=2" "a SNP on a contig inside a contig is broken out on the inner contig" + +is "$(awk -F '\t' '$1 == "gref_x_1_alt" { print $2, length($5) }' lift.nest.lifted.txt | tr '\n' ';')" "60 51;60 51;120 1;" "the detour onto the inner contig is an insertion on the outer one" + +# A haplotype that goes through the insertion twice counts once +vg paths -x nesting/nested_snp_in_ins_cycle.gfa -Q x --compute-gref -l 1 > lift.cycle.gref.vg 2> /dev/null +vg convert -f lift.cycle.gref.vg > lift.cycle.gref.gfa +vg gbwt -G lift.cycle.gref.gfa --gbz-format -g lift.cycle.gbz 2> /dev/null +vg deconstruct -P x -a lift.cycle.gbz > lift.cycle.vcf 2> /dev/null +is "$(vg lift lift.cycle.gbz lift.cycle.vcf 2> /dev/null | grep '^gref' | cut -f 1,2,4,5,10,11 | tr '\t' ' ')" "gref_x_1_alt 2 A T 0|1 0|1" "a walk that passes along a contig twice counts once" + +# Depth of haplotypes along the gref. In lift.gbz, both haplotypes skip the reference's SNP base. +is "$(vg lift --depth lift.gbz | tr '\t\n' ' ;')" "gref_x 0 100 2;gref_x 100 101 0;gref_x 101 201 2;" "the depth counts the haplotypes that read each gref base, with runs of depth 0" + +printf 's#1#c\t1000\t1050\n' > lift.depth.bed +is "$(vg lift --depth lift.gbz lift.depth.bed | tr '\t\n' ' ;')" "gref_x 0 50 1;gref_x 50 201 0;" "regions limit the depth to their bases, and a haplotype without regions counts nothing" + +is "$(vg lift --depth -Q 's#2' lift.gbz lift.depth.bed | tr '\t\n' ' ;')" "s#2#c 1000 1050 1;s#2#c 1050 1201 0;" "-Q writes the depth along other paths, in contig coordinates" + +printf 'zz#1#c\t0\t10\n' >> lift.depth.bed +vg lift --depth lift.gbz lift.depth.bed 2>&1 > /dev/null | grep -q "1 paths in lift.depth.bed, such as zz#1#c, are not counted paths" +is $? 0 "regions on paths that are not counted get a warning" + +vg lift --depth -a lift.depth.asm.bed lift.gbz 2>&1 | grep -q "takes no -u, -c, -P, -a, -m, -b or --gref-segs" +is $? 0 "--depth rejects the options that lift features" + +vg lift --depth lift.gbz 2>&1 > /dev/null | grep -q "2 of the 402 haplotype bases counted are on nodes that no output path visits" +is $? 0 "the depth says how many haplotype bases the rows leave out, such as SNP alleles too short for the gref" + +# A haplotype that reads node 2 twice, a haplotype that reads the graph in reverse, and a +# reference path +printf 'H\tVN:Z:1.1\tRS:Z:ref\nS\t1\tAAAAAAAAAA\nS\t2\tCCCC\nS\t3\tGGGGGGGGGG\nL\t1\t+\t2\t+\t0M\nL\t2\t+\t2\t+\t0M\nL\t2\t+\t3\t+\t0M\n' > lift.loop.gfa +printf 'W\tref\t0\tx\t0\t24\t>1>2>3\nW\ts\t1\tc\t0\t28\t>1>2>2>3\nW\ts\t2\tc\t0\t24\t<3<2<1\n' >> lift.loop.gfa +vg paths -x lift.loop.gfa -Q ref --compute-gref > lift.loop.gref.vg +vg convert -f lift.loop.gref.vg > lift.loop.gref.gfa +vg gbwt -G lift.loop.gref.gfa --gbz-format -g lift.loop.gbz +is "$(vg lift --depth lift.loop.gbz | tr '\t\n' ' ;') $(vg lift --depth --visits lift.loop.gbz | cut -f 4 | tr '\n' ' ')" "gref_ref#0#x 0 24 2; 2 3 2 " "a haplotype that reads a node twice counts once, and twice as visits" + +is "$(vg lift --depth --references lift.loop.gbz | cut -f 4)" "3" "--references counts reference paths too" + +printf 's#2#c\t0\t5\n' > lift.loop.bed +is "$(vg lift --depth lift.loop.gbz lift.loop.bed | tr '\t\n' ' ;')" "gref_ref#0#x 0 19 0;gref_ref#0#x 19 24 1;" "regions on a haplotype that reads the gref in reverse count the gref bases it reads" + +printf 's#1#c\t11\t13\ns#1#c\t15\t17\n' > lift.loop.bed +is "$(vg lift --depth lift.loop.gbz lift.loop.bed | cut -f 2-4 | tr '\t\n' ' ;') $(vg lift --depth --visits lift.loop.gbz lift.loop.bed | cut -f 4 | tr '\n' ' ')" "0 11 0;11 13 1;13 24 0; 0 2 0 " "regions that cut a node count each of its bases once per haplotype" + +rm -f lift.depth.bed lift.depth.asm.bed lift.loop.gfa lift.loop.gref.vg lift.loop.gref.gfa lift.loop.gbz lift.loop.bed + +rm -f lift.vcf.gfa lift.vcf.gref.vg lift.vcf.gref.gfa lift.vcf.gbz lift.decon.vcf lift.decon.bed lift.decon.lifted.vcf +rm -f lift.pg.vcf lift.pg.vcf.gz lift.pg.lifted.vcf lift.pg.err lift.foreign.vcf lift.vcf.segs.tsv lift.gq.vcf lift.gq.lifted.vcf +rm -f lift.head.gfa lift.head.gref.vg lift.head.gref.gfa lift.head.gbz lift.head.vcf lift.head.lifted.vcf lift.rev.bed +rm -f lift.snp3.err lift.snp3.gfa lift.snp3.gref.vg lift.snp3.gref.gfa lift.snp3.gbz lift.snp3.vcf lift.snp3.lifted.vcf lift.snp3.rev.vcf +rm -f lift.snp3.conflict.vcf lift.snp3.conflict.lifted.vcf lift.snp3.conflict.err +rm -f lift.ends.gfa lift.ends.gref.vg lift.ends.gref.gfa lift.ends.gbz lift.ends.vcf lift.ends.txt +rm -f lift.touch.gfa lift.touch.gref.vg lift.touch.gref.gfa lift.touch.gbz lift.touch.vcf +rm -f lift.nest.gfa lift.nest.gref.vg lift.nest.gref.gfa lift.nest.gbz lift.nest.vcf lift.nest.lifted.txt +rm -f lift.cycle.gref.vg lift.cycle.gref.gfa lift.cycle.gbz lift.cycle.vcf + +rm -f lift.paths.bed lift.paths.exact.bed lift.paths.lift.bed lift.paths.blocks.bed lift.outside.bed +rm -f lift.ins.gbz lift.ins.bed lift.ins.blocks.bed lift.ins.lift.bed lift.ins.tigs.gaf +rm -f lift.names.input.bed lift.names.bed lift.names.gaf lift.names.err lift.tigs.noheader.gaf lift.shared.bed + +rm -f lift.gfa lift.gref.vg lift.gref.gfa lift.gbz lift.tigs.gaf lift.tigs.fa lift.bed lift.gaf lift.asm.bed +rm -f lift.place.txt lift.feature.txt lift.support.txt lift.s1.bed lift.t1.bed lift.t4.bed lift.t1.gaf lift.t4.gaf +rm -f lift.unknown.bed lift.long.bed lift.plain.gbz lift.plain.gaf lift.plain.fa lift.plain.bed lift.plain.out.gaf lift.plain.asm.bed