Forgetful Memory Lab

The problem worth solving

What could make it stand out

System architecture

Data and reproducibility

Start with a synthetic benchmark

• LongMemEval research paper
• LoCoMo research paper
• W3C PROV-O provenance ontology

Baselines you must beat

1. Recency-only lexical retrieval over append-only history, with the same reader and context size.
2. Temporal filtering without purpose scope or derivative lineage, exposing whether time handling alone addresses the failure.
3. Source-record deletion with no summary or cache invalidation, plus a full-clear control that reveals the cost of overbroad forgetting.

Measure the claim

Targets below are proposed success criteria. No results have been achieved on your behalf.

Experiments and ablations

1. Replay the same questions before an update, after an update, and after a late correction. Compare valid-time filtering with simple newest-record selection and inspect ambiguous overlapping intervals.
2. Delete one source whose facts appear in a direct extraction, a summary, a summary-of-summary, and a cached answer. Probe each surface separately and test attempted reingestion of the deleted source.
3. Vary history length and summary compression while fixing retrieval budget. Report temporal accuracy, deletion work, latency, and retained utility rather than selecting whichever memory setting maximizes one metric.

Your execution plan

0 of 12 deliverables checked. Tick a deliverable after recording evidence in your project repository.

Working seed

Run this small, deterministic core first. It demonstrates the central mechanism. The complete system, experiments, and deployment are your capstone work.

"""Forgetful Memory Lab: temporal reads and transitive in-memory erasure."""
from dataclasses import dataclass
import json


@dataclass(frozen=True)
class Fact:
    id: str
    key: str
    value: str
    valid_from: int
    valid_to: int
    purpose: str
    parents: tuple = ()


class Memory:
    def __init__(self):
        self.facts = {}
        self.cache = {}
        self.tombstones = set()

    def add(self, fact):
        if fact.id in self.tombstones or any(p in self.tombstones for p in fact.parents):
            raise ValueError("Cannot resurrect erased provenance")
        self.facts[fact.id] = fact
        self.cache.clear()

    def retrieve(self, key, day, purpose):
        cache_key = (key, day, purpose)
        if cache_key not in self.cache:
            choices = [f for f in self.facts.values() if f.key == key
                       and f.valid_from <= day < f.valid_to and f.purpose == purpose]
            chosen = max(choices, key=lambda f: (f.valid_from, f.id), default=None)
            self.cache[cache_key] = chosen.value if chosen else None
        return self.cache[cache_key]

    def erase(self, source_id):
        doomed = {source_id}
        while True:
            expanded = doomed | {f.id for f in self.facts.values()
                                 if any(p in doomed for p in f.parents)}
            if expanded == doomed:
                break
            doomed = expanded
        for fact_id in doomed:
            self.facts.pop(fact_id, None)
        self.tombstones.update(doomed)
        self.cache.clear()
        return sorted(doomed)


def main():
    memory = Memory()
    memory.add(Fact("old", "city", "Pune", 1, 10, "travel"))
    memory.add(Fact("new", "city", "Bengaluru", 10, 99, "travel"))
    memory.add(Fact("summary", "profile", "Bengaluru traveller", 10, 99, "travel", ("new",)))
    memory.add(Fact("derived", "greeting", "Welcome from Bengaluru", 10, 99, "travel", ("summary",)))
    assert memory.retrieve("city", 5, "travel") == "Pune"
    assert memory.retrieve("city", 12, "travel") == "Bengaluru"
    assert memory.retrieve("city", 12, "marketing") is None
    removed = memory.erase("new")
    assert removed == ["derived", "new", "summary"]
    assert memory.retrieve("city", 12, "travel") is None
    assert memory.retrieve("profile", 12, "travel") is None
    assert memory.retrieve("city", 5, "travel") == "Pune"
    print(json.dumps({"deleted_ids": removed, "remaining_ids": sorted(memory.facts),
                      "current_city": memory.retrieve("city", 12, "travel"),
                      "historical_city": memory.retrieve("city", 5, "travel")}, sort_keys=True, indent=2))


if __name__ == "__main__":
    main()

Failure modes to investigate

• Deleting raw text alone leaves summaries and embeddings behind. Make the storage inventory part of the experiment and fail the receipt if any declared surface cannot be checked.
• An exact string scan misses paraphrased information. Add fixture-derived semantic probes and manual adjudication, and report their coverage separately from deterministic canary checks.
• A system can achieve perfect forgetting by erasing everything. Pair deletion probes with unrelated retained-memory controls and measure the resulting utility loss.

Your demo, moment by moment

1. Ask the synthetic assistant where a fictional person lived on an earlier date and where they live now; inspect the two different source intervals.
2. Change the permitted purpose and show that a fact available for travel planning is unavailable to a marketing request.
3. Delete a recent location source and display removal of its extracted fact, nested summary, and cached answer, followed by a post-deletion abstention.
4. Ask an unrelated retained question to prove useful memory remains, then show the before-and-after benchmark and the exact deletion boundary.

Write the resume bullet after the experiment

Replace every placeholder with your actual measurements. Keep the dataset size, baseline, and evaluation conditions available for interview questions.

A research extension

Related work to challenge your idea