Living Evidence Observatory

The problem worth solving

What could make it stand out

System architecture

Data and reproducibility

Start with a synthetic benchmark

• SciFact: Fact or Fiction research paper
• Crossref REST API documentation
• Crossref corrections and retractions guidance
• W3C PROV-O

Baselines you must beat

1. Snapshot RAG that indexes all available text without explicit publication or observation times.
2. Latest-document retrieval that selects the newest source and generates a single summary, revealing the cost of treating recency as authority.
3. A provenance graph without scope matching, isolating false contradiction detection caused by different experimental conditions.

Measure the claim

Targets below are proposed success criteria. No results have been achieved on your behalf.

Experiments and ablations

1. Replay a sequence where early supporting evidence is followed by a conflicting study and then a correction. Compare what each system claims at the time of observation, not with hindsight from the complete event log.
2. Add five mirrors or derivative articles for one source and verify that the evidence assessment does not count them as five independent studies. Remove lineage deduplication to quantify the resulting distortion.
3. Present apparently contradictory results from different budgets or datasets. Test scope matching with and without structured claim fields and manually inspect all false conflict alerts.

Your execution plan

0 of 12 deliverables checked. Tick a deliverable after recording evidence in your project repository.

Working seed

Run this small, deterministic core first. It demonstrates the central mechanism. The complete system, experiments, and deployment are your capstone work.

"""Living Evidence: event replay with origin deduplication and withdrawal."""
from dataclasses import dataclass
import json


@dataclass(frozen=True)
class Event:
    observed_at: int
    kind: str
    origin: str
    claim: str
    stance: str = ""
    published_at: int = 0
    record_id: str = ""


def evidence_state(events, claim, as_of):
    # Stances are authored fixture labels, not automated scientific judgements.
    origins = {}
    withdrawals = set()
    lineage = {}
    for event in sorted(events, key=lambda e: (e.observed_at, e.record_id)):
        if event.observed_at > as_of or event.claim != claim:
            continue
        if event.published_at > as_of:
            continue
        lineage.setdefault(event.origin, set()).add(event.record_id)
        if event.kind == "withdraw":
            withdrawals.add(event.origin)
        elif event.kind == "evidence":
            if event.stance not in {"supports", "refutes"}:
                raise ValueError("Unknown stance")
            previous = origins.get(event.origin)
            if previous and previous != event.stance:
                raise ValueError("Conflicting same-origin labels need adjudication")
            origins[event.origin] = event.stance
    active = {origin: stance for origin, stance in origins.items() if origin not in withdrawals}
    labels = set(active.values())
    status = "insufficient_evidence"
    if labels == {"supports"}:
        status = "supporting_evidence_only"
    elif labels == {"refutes"}:
        status = "refuting_evidence_only"
    elif len(labels) == 2:
        status = "contested_evidence"
    return {"as_of": as_of, "status": status, "active_origins": sorted(active),
            "withdrawn_origins": sorted(withdrawals),
            "lineage": {k: sorted(v) for k, v in sorted(lineage.items())}}


def main():
    events = [
        Event(1, "evidence", "study_a", "cache_reduces_latency", "supports", 1, "paper_a"),
        Event(2, "evidence", "study_a", "cache_reduces_latency", "supports", 1, "mirror_a"),
        Event(3, "evidence", "study_b", "cache_reduces_latency", "refutes", 2, "paper_b"),
        Event(4, "withdraw", "study_b", "cache_reduces_latency", "", 4, "notice_b"),
    ]
    states = [evidence_state(events, "cache_reduces_latency", t) for t in (1, 2, 3, 4)]
    assert states[0]["status"] == "supporting_evidence_only"
    assert len(states[1]["active_origins"]) == 1
    assert states[2]["status"] == "contested_evidence"
    assert states[3]["withdrawn_origins"] == ["study_b"]
    assert evidence_state(list(reversed(events)), "cache_reduces_latency", 3) == states[2]
    print(json.dumps(states, sort_keys=True, indent=2))


if __name__ == "__main__":
    main()

Failure modes to investigate

• Similar wording can conceal different study conditions. Compare metric, dataset, intervention, and budget before assigning a contradiction relation, and send unresolved cases to review.
• Metadata services may lack an update or expose it after a delay. Preserve both publication and observed timestamps and describe source coverage without claiming completeness.
• A graph can make weak evidence look authoritative. Keep evidence spans inspectable, separate study independence from citation count, and avoid translating an evidence status directly into a truth guarantee.

Your demo, moment by moment

1. Ask a tightly scoped research question and open the evidence spans supporting the current assessment.
2. Replay a conflicting paper with the same experimental scope. Show the summary becoming contested and compare the two studies' conditions.
3. Add several mirrors of one paper, then a clearly labeled simulated correction. Show source deduplication and the specific summaries invalidated by the update.
4. Move the as-of date backward and recover the assessment justified by evidence available then, followed by the held-out evaluation and a false-conflict example.

Write the resume bullet after the experiment

Replace every placeholder with your actual measurements. Keep the dataset size, baseline, and evaluation conditions available for interview questions.

A research extension

Related work to challenge your idea