Skip to content

Worked example: nf-core/sarek on HG002 chromosome 20

Everything on this page comes from one real run on 23 September 2026. The output files are in the repository under tests/golden/m0/. A test recomputes each hash and metric shown here.

What was run
Pipeline nf-core/sarek 3.10.0, commit 8ccac7ad37b05dd792447763bf9671b719824587, profile docker
Engine Nextflow 26.04.6
Caller GATK4 HaplotypeCaller (tools: haplotypecaller, skip_tools: baserecalibrator)
Reference material HG002 (NA24385), a public GIAB reference sample
Reads GIAB HG002 Illumina HiSeq 60x GRCh38 BAM, chromosome 20, half the reads kept (samtools view -s 42.5, seed 42)
Reference genome GRCh38 no-alt analysis set, chromosome 20 only
Truth set GIAB HG002 v4.2.1 GRCh38 small variants and confident regions, restricted to chr20
Comparator hap.py v0.3.12, vcfeval engine, jmcdani20/hap.py@sha256:0812d37e…b489
Machine Laptop, WSL2 Ubuntu 24.04, 8 threads, 12 GB for Docker; about an hour of wall time

1. Resolve

aver resolve found 212 dependencies and 0 unpinned:

Kind Count
Parameters 122
Containers (each resolved to a digest) 54
Tools (versions read from the container tags) 34
Engine 1
Reference genome (by SHA-256) 1

Snapshot hash:

9648e1c847fa46727dbbd472b8b63cdaace73a62da5904a6bcbb242f849cdad0

Two resolutions of the same pipeline a day apart, with separate Nextflow caches, produced this same hash.

The fasta parameter was a local path when Aver resolved the pipeline. The snapshot stores it as {"sha256": "f0167a33…2efc"} instead, so the hash depends on the file's contents, not on where the file sits on this laptop.

2. Reference data

Aver checks every input against a SHA-256 recorded in its catalogue before it uses the file. Each derived file carries a manifest (aver-derivation.json) that records its inputs, the exact commands that made it, and the container digest those commands ran in:

  • Truth set: bcftools view -r chr20 on the published v4.2.1 VCF (adb4d4a5…). The confident-regions BED (fba9a57c…) is filtered to chr20.
  • Reference: chr20 extracted from the analysis set (fb4243eb…) and indexed with samtools faidx.
  • Reads: streamed from the public GIAB bucket, subsampled, name-sorted and written as paired FASTQ.

3. Benchmark

germline-chr20-sarek  ·  HG002  ·  giab-hg002-v4.2.1-grch38  ·  region chr20
snapshot 9648e1c847fa46727dbbd472b8b63cdaace73a62da5904a6bcbb242f849cdad0
comparator jmcdani20/hap.py:v0.3.12@sha256:0812d37e7210011e407914deb3da2094ec8258077b21d3211e694e5ca303b489
          Recall  Precision        F1        TP      FN      FP
SNV     99.4294%   98.9941%  99.2113%     70926     407     721
INDEL   97.8323%   98.7786%  98.3031%     11012     244     141
cache key 6150a64c4e2089ca3389bc908a10cf1ae145595dfd34e5e5f3bf041826423c2a

The cache key covers the snapshot hash, the truth set, the comparator version, the SHA-256 of the reference genome and the region. If you run the benchmark again with the same inputs, Aver reuses the recorded result instead of recomputing it.

What the numbers do and do not say

These figures describe this configuration on chromosome 20 at roughly 30x coverage. They are not a performance claim for sarek or for GATK, and not an acceptance criterion. The laboratory sets its own criteria. Aver records the measurement and flags when it changes.

3b. Where the misses are: stratified metrics

Running the same VCF against the core GIAB v3.6 GRCh38 stratifications (aver compare ... --stratification core) splits the numbers by genomic context. The whole-region figures stay exactly the same:

Stratified recall (PASS), GIAB stratifications:
                                                     SNV     INDEL  SNV truth  INDEL truth
  AllTandemRepeatsandHomopolymers_slop5         98.3375%  97.0893%       3970         7421
  AllHomopolymers_ge7bp_imperfectge11bp_slop5   99.6574%  97.2727%       2043         4950
  segdups                                       94.2384%  94.3662%       1510          142
  gclt25orgt65_slop50                           98.9269%  97.8616%       4007          795
  lowmappabilityall                             88.3584%  79.7468%       2723          158
  alldifficultregions                           96.5810%  96.9370%      10939         7868
  notinalldifficultregions                      99.9454%  99.9115%      60394         3388

Outside the difficult regions, recall is above 99.9% for both classes. About 374 of the 407 missed SNVs are in difficult regions, roughly 317 of them in low-mappability sequence. For INDELs, about 216 of the 244 misses sit in tandem repeats and homopolymers. These counts come from truth totals × (1 − recall). The difficult and not-difficult rows add up exactly to the whole-region truth totals.

Each stratification BED is fetched from NCBI and must match the md5 NCBI publishes before Aver uses it. Aver then records its SHA-256. The AWS mirror carries releases only up to v3.3, so for v3.6 NCBI's published md5 is the independent check.

4. Two things the run surfaced

The sample-name guard stopped the first comparison. Aver compares only VCFs whose sample name is a registered reference material. sarek names the sample <patient>_<sample>, so the VCF said HG002_HG002, and the guard refused it. Aver now accepts a composed name only when every part is a registered name for the same material. HG002_HG002 passes; HG002_anything-else is still refused, and the error never repeats the name it rejected.

No variant filtering was applied. With HaplotypeCaller, sarek filters variants only when --dbsnp or --known_indels is set. It logged a warning, and the PASS and ALL rows in summary.csv came out identical. That is a property of this pipeline configuration, and Aver records it the same way it records any other parameter. Adding dbSNP would change the configuration, produce a new snapshot hash, and show up as drift once a baseline is promoted.

Reproduce it

aver init --preset nf-core/sarek --name germline-chr20-sarek --clinical
aver resolve -o snapshot.json      # expect 9648e1c8…cdad0 for the same revision and parameters
aver benchmark