Protocol
Reproduce a Result under Controlled Budgets
Reproduce a manageable published result. A carefully explained failure to reproduce passes when the experiment is sound; a matching number without provenance fails.
70–120 active hours60–90 min sessionsAdvanced
Mastery contract
Core protocol · Standard 1.0 · Reproduce a manageable published result. A carefully explained failure to reproduce passes when the experiment is sound; a matching number without provenance fails.
Loading your learning records…
Reasoning — not yet passed
The target declaration separates exact reproduction from scale-reduced replication and gives a justified comparison tolerance.
Reliable implementation — not yet passed
The repository rebuilds the method and baseline from a clean environment with exact configurations and seeds.
Reproducible experiment — not yet passed
All seeds and ablations are reported with compute/data budgets and uncertainty; no best-run-only result is accepted.
Defense and handoff — not yet passed
Write a 6–10 page replication report that attributes discrepancies to tested causes or labels them unresolved; obtain one independent review.
All four criteria must pass. Activity completion does not satisfy them.
An independent reviewer reruns your work and varies something you did not rehearse: a fresh input, a different working directory, or a declared edge case.
Pilot decisions are provisional, not external credentials. For an appeal or sensitive artifact, contact your designated pilot operator with the submission ID. Export your assessment history.
What you will do
6 total
Choose one paper result and derive its core objective or architectural mechanism; write a target table with metric, tolerance, scale and disconfirming outcome.
Implement the method from its description, then audit against official code; record every discrepancy and produce a fixed environment/configuration.
Run the target and strongest feasible baseline over at least 3 seeds and 2 mechanism ablations; retain checkpoints and per-run metrics.
Debug one deliberate train/eval mismatch and investigate the largest gap from the paper without changing the target after seeing results.
Write a 500–900 word technical report linking the reasoning, code, measurements, and limitations; include the project decision below.
Assess all 4 mastery criteria against saved artifacts, request an independent review, and repeat each failed criterion on a fresh example.
Essentials
A reading session alone never satisfies a mastery criterion.
Store source references, assumptions, and artifact paths beside every result.
Keep an untouched check case that differs from the worked example.
Record all attempts, including failures and results that contradict your prediction.
Use the stated comparison conditions; document every deviation before drawing a conclusion.
Protocol guardrails
Do
- +State the expected result before running the comparison.
- +Keep one minimal reproducible failing case when debugging.
- +Record environment versions and the exact command used.
- +Compare explanations with saved intermediate values.
- +Ask a reviewer to challenge the weakest assumption.
Don't
- ×Do not copy a worked solution and present it as an independent implementation.
- ×Do not tune against held-out evaluation outcomes.
- ×Do not report only the best seed or discard inconvenient runs.
- ×Do not equate elapsed hours or a completed run with a passed assessment.
- ×Do not conceal reduced-scale experiments behind claims about the original full-scale result.
Protocol authorship
Written by NuthinButta as instructional design. The teaching sources are the official references linked in each milestone; the exercises, workload and pass thresholds are ours, not their authors'.