ValiChord Complete — Open Design Questions

Precedents, Likely Approaches, and Resolution Phases

Purpose

The following thirteen questions do not have complete answers yet. They are documented because they are the questions that funders, ethics boards, journal editors, and institutional partners will ask first — and because honest acknowledgment of open problems is more credible than silence.

Each question includes: the problem, precedents from existing reproducibility initiatives that inform the design space, ValiChord's likely approach, and which phase resolves it.

These questions are listed in the Vision & Architecture companion document. This document provides the full treatment.

The Questions

1. Do original authors need to consent to validation? ValiChord's likely model is dual-path: author-initiated submission as the primary route, with third-party submission permitted for published work where code and data are already public, subject to notification and right of reply. This mirrors the approach used by the Reproducibility Projects (Psychology and Cancer Biology), which contacted original authors for materials and clarifications but did not require permission to proceed, and cascad, which verifies papers without requiring author consent when materials are public. The exact balance between openness and procedural fairness is a Phase 1 governance question informed by the PI and ethics review.

2. Who pays for compute? The validator fee covers intellectual labour, but computational reproduction can require significant infrastructure — cloud computing, GPU hours, HPC access. ValiChord's likely model separates validator compensation (for time) from compute provision (as infrastructure): lightweight studies use the validator's own resources; high-compute studies require the submitting institution to provide access or a central bursary. Precedents include the Reproducibility Project: Cancer Biology (centrally funded, averaging ~$52K per replication) and AEA/cascad economics replications (journal-funded). Phase 0 captures compute requirements as a data dimension to inform Phase 1 costing. Some studies may prove unvalidatable at current funding levels — that is an honest finding, not a system failure.

3. What happens after a negative Harmony Record? The likely process mirrors established precedents (Reproducibility Projects, cascad, Registered Reports): original author notified before publication, given a defined response window, their response embedded in the Harmony Record as part of the permanent record. No automatic downstream action — no forced retraction, no funder notification. The Harmony Record is information, not a verdict. Journals and funders query the integration layer and decide their own responses. The exact notification timeline and escalation process are Phase 1 governance questions.

4. Original author's right of reply. If a study doesn't reproduce, the original author may have a legitimate explanation — a missing dependency, an undocumented configuration, a known hardware sensitivity. The Harmony Record format already preserves multiple perspectives; extending this to include the author's response as a permanent, visible part of the record is architecturally natural. Readers see the full picture: what the validators found, what the author says, and the unresolved questions.

5. How are Phase 0 studies selected? Study selection should be governed by the PI with documented rationale, transparent inclusion and exclusion criteria, and deliberate recruitment across the difficulty spectrum — including studies expected not to reproduce cleanly. Precedents (the Reproducibility Projects used steering committees with external advisors) show that documented selection processes are standard. The selection rationale should be published alongside results.

6. Restricted and sensitive research. ValiChord's likely approach is tiered: open-data studies validated normally; restricted-data studies validated through secure access arrangements (NDA, institutional agreements, secure enclaves) where feasible; fully embargoed studies excluded or limited to open pipeline components. Precedents include cascad (temporary access via secure enclaves), epidemiology initiatives (federated analysis, simulated data), and 3ie development economics (de-identified data, restricted-access protocols). Phase 0 focuses on openly available studies; the restricted/excluded boundary is a Phase 2+ question.

7. Holochain platform dependency. If Holochain stalls or fails to reach production maturity, ValiChord's core concepts — content-addressed storage, cryptographic commitments, distributed validation, tamper-evident audit trails — are established distributed systems patterns implementable on other platforms. What Holochain provides is an unusually clean fit (agent-centric GDPR compliance, no global consensus requirement, native behavioural analysis). The mitigation is architectural awareness: maintaining clear separation between the conceptual model and the implementation platform so that governance, evidence, and institutional relationships transfer even if the engineering needs rebuilding.

8. Validator training and calibration. Even computationally skilled researchers need structured onboarding to ValiChord's specific process — environment documentation, reproduction standards, time recording, commit-reveal mechanics. Precedents range from cascad (supervised practice validations) to Cochrane (structured reviewer training modules) to the Reproducibility Project: Psychology (detailed protocols with coordinator check-ins). ValiChord's likely model is a practice validation on a known study, process documentation, and coordinator support. Phase 0 itself will reveal what validators found unclear, directly informing training design.

9. Correcting a flawed Harmony Record. If a validator commits fraud or a systematic error is discovered in the validation process, the append-only architecture means the record cannot be deleted — but it can be annotated and superseded. This mirrors established practice: journals issue corrections and expressions of concern, Retraction Watch maintains public records, and the principle throughout is that flawed work is marked rather than erased. The original Harmony Record stays; a correction notice is appended with rationale. The annotation format and governance trigger are Phase 1 questions.

10. Long-term record preservation. Harmony Records live on Holochain's DHT, which requires active nodes. If ValiChord ceases to exist, records become inaccessible. Precedents for academic preservation are mature: LOCKSS/CLOCKSS, Crossref DOI persistence, Portico, Internet Archive. ValiChord's likely approach is dual — active use on the DHT plus periodic export in a standard archival format (structured JSON or XML with published schema) to institutional repositories or preservation services. The archival format and partnerships are Phase 2 questions.

11. Validator identity verification. At Phase 0 scale, identity is trivial — known individuals recruited through the host institution. At Phase 3 with thousands of global validators, fake identities and sockpuppet accounts become real risks. Precedents centre on institutional identity linking (ORCID, institutional email verification, clinical trial registry sign-off). ValiChord's likely model ties registration to ORCID or institutional email, with the social distance mapping already in the architecture (co-authorship graphs, institutional caps) detecting coordinated fakes. Exact verification processes are Phase 1 questions.

12. Submission-side cherry-picking. Voluntary submission means researchers disproportionately submit work they're confident will reproduce, biasing ValiChord's record toward success — the publication bias problem applied to validation. The precedent is instructive: Registered Reports solved publication bias by committing to publish regardless of outcome; ValiChord's parallel is funder mandates or journal integration (Phase 2 goals). During voluntary adoption, the limitation should be stated clearly in aggregate reporting: "these results reflect voluntarily submitted studies and may not be representative of the broader literature."

13. Cross-border data jurisdiction. Phase 0 is UK-based; Phase 3 is global. Cross-border validation involving sensitive data creates jurisdictional complexity — GDPR adequacy decisions, data transfer agreements, ethics reciprocity. Precedents include international clinical trials, CERN's global collaborations, and the GDPR's own Standard Contractual Clauses. ValiChord's architecture provides a partial technical solution: patient data stays local, only cryptographic proofs are distributed. For restricted data, institutional data transfer agreements would be required — standard practice in international research. The specific legal frameworks are Phase 2+ questions requiring legal expertise.

Companion Documents: - ValiChord Vision & Architecture — The source document from which this is drawn - ValiChord Technical Reference — Architecture sketches for engineering discussion - ValiChord Governance Framework — How the system resists corruption and capture - ValiChord Phase 0 Proposal — Workload Discovery Pilot (£69K, 6 months)

Contact: Ceri John — topeuph@gmail.com