← ValiChord

Phase 0: Validation Workload Discovery

How long does it take to check whether published computational research actually works? Nobody knows. This study finds out.

The Gap

Funders mandate data sharing. Journals require code availability. Registered Reports pre-register hypotheses. All of these initiatives share one untested assumption: that someone can sit down with a published study — the code, the data, the methods — and check whether the results are correct, at a cost someone is willing to pay.

No one has ever measured whether that’s true. Without workload data, you can’t design infrastructure. You can’t set compensation rates. You can’t tell a funder what validation will cost per study.

The Study

We recruit 25–30 UK-based computational researchers, pay them at professional rates (£500 per task) to validate published studies across a difficulty spectrum, and measure what happens.

• 20–25 studies across a difficulty spectrum (easy to extreme)
• 3 independent validators per study — enables inter-rater reliability analysis
• Structured time tracking: setup, data, execution, troubleshooting
• Exploratory regression on which study features track with validation time
• Validator environment fingerprint captured at recruitment for Phase 1 matching
• Exit survey on sustainable participation and incentive adequacy

Primary Research Questions

• What is the distribution of validation workload across computational studies?
• What task characteristics are associated with validation time?
• What proportion of studies are validatable within reasonable timeframes?
• Does validator competence explain more variance than study characteristics?
• What support do validators need, and what are common barriers?

What Phase 0 Produces

• Time ranges by difficulty tier with inter-rater reliability metrics (ICC)
• Candidate predictors for a future difficulty-assessment system
• Exclusion criteria: study types uneconomical to validate
• Preliminary compensation estimates grounded in observed time
• Academic publication — first systematic dataset on validation workload
• Policy briefing for UKRI and the UK Metascience Unit

Context: The Infrastructure Already Exists

Phase 0 is not a proof of concept for whether ValiChord can be built — it has been built. The four-DNA Holochain infrastructure is implemented and integration-tested (87 tests pass against live conductors). Phase 0 provides the empirical evidence needed to design the operational layer: what to accept, what to exclude, what validators need, and what the economics look like.