Repository navigation
Conversation
METRICS was read and documented but never passed on, so METRICS=1 produced no metrics files on Illumina runs. Each denoising step (dada, dada-pooled, dada-pseudo; fwd and rev) now writes $OUT/<step>.metrics.json, matching run_pacbio.sh. The pseudo-mode per-sample control run is left out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sums the shuffle, bud and p_update phase timers from --metrics-json files. Single pass keeps every comparison and drops those three phases, so their share is the most it can save. Phase 0 of the #323 plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
…r change (#323) Phase 1 of #323. Adds Partition {divisive, single-pass} and BirthRule {pvalue, unoise} to DadaParams, and the flags to dada and dada-pooled under Experimental. divisive is R DADA2 and the default; a non-default value is recorded in the output params, so default output is byte-identical. single-pass is rejected as not implemented yet, --birth-rule without single-pass is rejected, and dada-pseudo does not take the flags. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 2 of #323. --partition single-pass visits the uniques once, most abundant first; each becomes a new centre or joins the existing centre with the most expected reads. Choices settled on #323: expected reads are lambda x the centre's own reads (not the cluster total), the best centre is the one with the most expected reads, and the MIN_FOLD/MIN_HAMMING/ MIN_ABUNDANCE/singleton gates and the Bonferroni-corrected OMEGA_A test apply as in the divisive bud. Nothing is reassigned after the pass. Each birth gets the same b_compare_parallel a divisive split does, so screen, alignment and lambda are unchanged. Decision time is booked as the bud phase. --birth-rule unoise is still rejected (phase 4). Default output is byte-identical to main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Apply b_p_update's greedy lock rule to each newly born centre: lock the centre, and every raw whose best centre is now the new one and which that centre alone is expected to produce more reads of than were observed. Locked raws are skipped by later comparisons, as in the divisive loop. On the MiSeq SOP pooled (fwd), the ASV set is unchanged (534, 52 reads move between clusters), and alignments fall from 12.1M to 9.65M against 9.36M for divisive with 3% fewer clusters. A unit test pins that single pass aligns the same pairs as divisive when only locks from a new centre separate them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 3 of #323. Under --partition single-pass, --reassign runs b_shuffle2 once after the pass: each non-centre raw moves to the cluster with the most expected reads, priced against cluster reads as in the divisive loop. It moves reads between ASVs but cannot add or remove one. Recorded in params only when set; rejected without single-pass. On the MiSeq SOP (fwd) it moves 41 raws pooled and leaves the ASV set unchanged; abundance moved against divisive falls from 549 to 487 reads pooled and from 866 to 796 per-sample. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 4 of #323. --partition single-pass --birth-rule unoise absorbs a unique into a centre when its skew is at most beta(d) = 1/2^(2d+1) (Edgar 2016, doi:10.1101/081257; alpha = 2 as in vsearch). d is the Levenshtein distance from the alignment: substitutions plus internal gap columns, end gaps excluded (vsearch --iddef 2). Among qualifying centres the unique joins the one with the smallest d, ties to the more abundant. A unique no centre absorbs becomes a centre at --min-abund reads or more, otherwise it is left unassigned and its reads uncorrected (B::unassigned). Each centre gets its own alignment pass against uniques with at most half its reads, because the lambda-based store filter and greedy locks could drop a qualifying centre. DADA2's p-value, gates and locks do not apply; --reassign is rejected with this rule. Reproduces vsearch 2.32 --cluster_unoise exactly on the pooled MiSeq SOP fwd reads: 449/449 centres at minsize 8 and 642/642 at minsize 4; alpha 3 as a negative control differs by 61. Default and p-value single-pass output are byte-identical to before. Documented on the dada command page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run_arms.sh learns one error model (with indel rates) per dataset, then runs divisive, single-pass, single-pass + reassign (each with and without the indel λ) and the UNOISE rule through merge, chimera removal and optional reference-eval. It refuses an arm whose output JSON does not record the settings it was asked for. summarize_arms.py reports ASV set identity and Bray-Curtis against a baseline arm, dada time and truth-set scores. phase_share.py moves into the same folder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l arm (#323) --birth-rule unoise no longer applies OMEGA_C after the pass: every member a centre absorbed counts toward its ASV, as in UNOISE3. Only unassigned uniques (below --min-abund, no qualifying centre) are left out of map. New unit test, mutation-checked against the old rule. run_arms.sh gains a `vsearch` arm: the UNOISE3 workflow end to end (merge, --fastq_maxee 1, derep, --cluster_unoise, --uchime3_denovo, every read mapped to the ZOTUs at 97%), converted by vsearch_to_seqtab.py into the files the other arms produce. On the pooled MiSeq SOP forward reads its centroids equal the unoise arm's (449/449) with the maxEE filter off. A failed step now names itself instead of exiting silently. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Passes flags to remove-bimera-denovo in every DADA2-side arm, e.g. --min-fold-parent-over-abundance 3.5 for PacBio; recorded in config.txt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…323) timed() falls back to bash's time keyword (wall and CPU seconds) where no GNU/BSD time exists, as on the cluster, and takes TIME_BIN. summarize_arms.py adds wall_s and peak_mb: the dada step's from --metrics-json, the vsearch arm's --cluster_unoise from time_fwd.txt (GNU, BSD or bash format). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Working branch for #323 (single-pass partitioning experiment). It starts with phase 0, measuring the most single pass could save; later phases will land here too.
Changes so far
dev/concordance/run_illumina.sh:METRICS=1was read and documented but never passed on, so Illumina runs wrote no metrics. Each denoising step (dada,dada-pooled,dada-pseudo; fwd and rev) now writes$OUT/<step>.metrics.json, matchingrun_pacbio.sh. The pseudo-mode per-sample control run is left out.dev/phase_share.py: sums theshuffle,budandp_updatephase timers from--metrics-jsonfiles. Single pass keeps every comparison and drops those three phases, so their share of the summed phases is its maximum possible saving. A share only holds at the thread count it was measured at.Verified
run_illumina.shon the two-sample fixture, withPOOL=trueandPOOL=false: both exit 0 and write fwd and rev metrics files.phase_share.pyon a local MiSeq SOP run (20 samples, fwd): removable share is 0.3% at 1 thread and 1.9% at 10 threads when pooled, and 1.1% per-sample.Still to do
Cluster runs, 362-sample MiSeq and PacBio 95 pooled, at 24 and 96 threads.
🤖 Generated with Claude Code