# Goal 1 — Memex fully operational: implementation spec > **STATUS UPDATE (2026-05-31 audit):** This spec was authored 2026-05-17 when Memex had no entities and the DB was empty. As of 2026-05-31, Memex is **LIVE with 19,171 entities**, all 10 MCP tools green from MBP, FTS5 search working, vec0 vector search offline (embed bug, non-blocking). The Phase 1 ingest (§4) has been completed. The "What's missing" table in §2.2 is largely stale — MCP server, tool wrappers, and HTTP API are all deployed and operational. The acceptance criteria (§1.A-E) and phase plans (§3-8) remain valid as the target; the "current state" (§2) needs to be read with the understanding that significant progress has been made since authoring. **Ken's stated G1 percentage as of 2026-05-30: ~75%.** Remaining gap to 100%: Reflect 365-day backfill, formal 7-query acceptance test run, email person-aggregation, `/memex/ask` UI. See `g1-g2-g3-path-to-100-reconciliation-2026-05-26.md` for the close-out roadmap. ## 1. Definition of "fully operational" Memex is fully operational when Ken can ask a natural-language question about his life — people, decisions, health patterns, faith journey, family logistics — and receive a sourced, cross-referenced answer within 3 seconds, drawing on 14+ years of personal data. This is not a search engine; it is a relational knowledge graph with inference. ### Acceptance criteria #### A. Entity density thresholds | Metric | Minimum | Target | Source | |--------|---------|--------|--------| | Total entities in `memex_entities` | 20,000 | 50,000+ | Tier 1–3 ingest queue | | Total edges in `memex_edges` | 50,000 | 200,000+ | D4 v2 Haiku pair-evaluation | | Unique entity types | 12+ | — | person, place, medication, exercise, decision, project, event, scripture, lesson, trip, device, concept | | Entities with ≥3 cross-document mentions | 2,000+ | — | Confirms deduplication working | | Edge kinds | 8+ | — | co-occurs, causes, precedes, family_of, treats, located_at, decided_in, references | #### B. Query acceptance tests These queries must return non-empty, correct, sourced results: | # | Query | Expected behavior | Validates | |---|-------|-------------------|-----------| | Q1 | "What medications do I take and why?" | Returns 6 meds with pharmacological rationale, cites `state/health/meds-stack.md` + operating manual | Entity extraction + health domain | | Q2 | "When did I first write about ShotLink architecture?" | Returns earliest design notes through current implementation dispatches, demonstrating multi-year temporal range across heterogeneous sources | Temporal range, project entity | | Q3 | "What decisions did I make about the workout log?" | Returns #206 ratification, v5b styling, pressed-for-time protocol | Decision entity extraction from canon files | | Q4 | "Show me everything related to Kevin's mission" | Returns Kevin Lovell → Washington Tacoma Mission, any dad letters referencing mission prep | Person-event edges, family graph | | Q5 | "What was I doing the week of October 14, 2014?" | Returns Team of Rivals intensive highlights period, any co-occurring dispatches/notes | Temporal windowing across sources | | Q6 | "Which projects depend on Memex?" | Returns #93/#94 (tool-call wrappers), /morning briefing, evening stack, biography project | Project-dependency edges | | Q7 | "What's my travel pattern to Atlanta?" | Returns dates/frequencies from Flighty CSV + calendar data | Trip entities, location graph | #### C. Performance targets | Operation | Target | Hard ceiling | |-----------|--------|--------------| | Single-entity lookup | < 500ms | 1s | | Natural-language query (local, no LLM call) | < 1s | 2s | | Natural-language query (with Haiku reranking) | < 3s | 5s | | Full re-ingest (Tier 1–3, ~1,830 files) | < 30 min | 60 min | | Incremental ingest (single new dispatch) | < 5s | 10s | | Edge inference batch (D4 v2, 200 entity pairs) | < 10 min | 20 min | #### D. Integration surface requirements | Surface | Integration | Trigger | |---------|-------------|---------| | `/morning` briefing | Memex supplies "On this day" historical context, teaching prep references, family-event lookups | Rendered at page load; data pre-fetched by AWL overnight | | `/evening` stack | Memex supplies "decisions made today" summary, cross-references to open projects | Ken-initiated; query fires on route hit | | Cowork MCP tool | `memex_query(question)` callable by any Claude session on MBP | Tool-call wrapper (#93) | | Claude Code local tool | `memex_query(question)` callable by Mac Mini agents | Tool-call wrapper (#94) | | AWL overnight passes | AWL can fire ingest, edge-inference, and cleanup jobs against Memex DB | Gated on token budget as with all AWL work | #### E. Data completeness gates Memex is NOT fully operational until these source classes are ingested: | Source class | File count | Path | Status | |--------------|-----------|------|--------| | Dispatch extracts | ~800 | `Claude/state/projects/dispatch-extracts/` | On disk, awaiting ingest | | OneNote extracts | 961 | `Claude/state/projects/onenote-extracts-2026-05-16/` | On disk, awaiting ingest | | Biographical extracts | 48 | `Claude/state/projects/bio-extracts-2026-05-16/` | On disk, awaiting ingest | | Weekly replay summaries | 52 | `Claude/state/projects/52-week-replay-2026-05-16/` | On disk, awaiting ingest | | Canon state files | 7 | `Claude/state/canon/` | On disk, awaiting ingest | | Reflect notes | ~365 | Not yet exported | Blocked on Reflect MCP reconnect | | Apple Health entities | Unknown | `Projects/health-export/` | Raw data exists; D20 extract not yet fired | | Flighty trips | Unknown | `Bio and Headshot/FlightyExport-2026-03-28.csv` | Raw CSV exists; D22 extract not yet fired | **Gate rule:** Tiers 1–3 from the ingest queue (canon + strategic docs + bulk extracts) must complete before declaring Goal 1 done. Tier 4 (backfill) and deferred sources (Reflect, Apple Health, Flighty) are enhancement, not blockers. --- ## 2. Current state ### What's shipped **Schema (Phase 1) — deployed to `~/Projects/memex/memex.db`:** ```sql -- Core tables (confirmed operational) memex_entities ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, entity_type TEXT NOT NULL, canonical_name TEXT, mention_count INTEGER DEFAULT 1, first_seen_date TEXT, last_seen_date TEXT, source_file TEXT, metadata JSON, created_at TIMESTAMP, updated_at TIMESTAMP ) memex_edges ( id INTEGER PRIMARY KEY, source_entity_id INTEGER REFERENCES memex_entities(id), target_entity_id INTEGER REFERENCES memex_entities(id), edge_kind TEXT NOT NULL, rationale TEXT, strength REAL, inferred_at TIMESTAMP, source_file TEXT ) ``` **Watchdog — operational on Mac Mini:** - Path: confirmed in `Claude/state/canon/memex-watchdog-path.md` - Behavior: monitors extraction process, auto-relaunches on death - Uptime: running continuously since April 2026 Phase 2 bootstrap **File-watcher (#95) — ratified, spec complete:** - 5-minute debounced WatchPaths on `dispatch/` + `state/` directories - Triggers incremental ingest on file change - Not yet deployed (depends on #241 `ingest_dispatches.py`) **Extraction pipeline — completed overnight runs:** - D1/D3/D11 format JSON extracts produced for OneNote (961 files), dispatches (~800 files), biographical sources (48 files) - Master index partially built: `Claude/state/projects/master-index-2026-05-16.jsonl` (~1,828–5,000 lines) - 52-week replay narratives generated - All sitting on disk in `Claude/state/projects/` awaiting ingest into SQLite **Tool-call wrappers (#93, #94) — approved, not yet built:** - Cowork MCP wrapper: allows MBP Claude sessions to query Memex - Claude Code local wrapper: allows Mac Mini agents to query Memex - Both depend on #241 (ingest must run first so there's data to query) ### What's missing | Gap | Blocker | Impact | |-----|---------|--------| | `ingest_dispatches.py` not deployed | #241 ratified but not yet executed | Zero entities in DB; entire system is empty | | No entity deduplication pass | Needs ingest to run first | Could have 5x redundant entities without canonical_name resolution | | No edge inference (D4 v2) | Needs entities in DB first | Graph has nodes but no relationships | | File-watcher not wired | Needs ingest script deployed | New dispatches don't auto-ingest | | Tool-call wrappers not built | Needs data in DB | Claude sessions can't query Memex | | Discretion filter not implemented | Design approved (ingest-time, not query-time) | Sensitive content could surface inappropriately | | Apple Health / Flighty extracts not fired | D20/D22 not yet queued | Health and travel entities absent | | Reflect notes not accessible | MCP disconnected (OAuth/server-side) | 365-day note backfill blocked | | Query layer (NL → SQL/vector) not built | Needs populated DB to test against | No user-facing query capability | | Integration with /morning and /evening | Needs query layer | Surfaces can't pull Memex context | ### Critical path The entire system is bottlenecked on a single artifact: **`ingest_dispatches.py` deployed to `~/Projects/memex/` and run against Tier 1–3 sources.** Everything downstream — dedup, edges, wrappers, integrations — is blocked until the DB has entities in it. Sequence: `#241 deploy → Tier 1 ingest → Tier 2 ingest → Tier 3 ingest → D4 v2 edges → #93/#94 wrappers → surface integrations → acceptance tests` --- ## 3. Phase plan overview ### Phase 1: Ingest & populate (the unblock) **Status:** Ratified (#241). Script exists. Needs deploy + first run. Deploy `ingest_dispatches.py` to `~/Projects/memex/`, then execute the tiered ingest sequence defined in the ingest queue document. Tier 1 (7 canon files) runs first to establish high-quality entity anchors — these are Ken's most authoritative self-descriptions and system definitions. Tier 2 (strategic docs, ~30 files) adds project structure and decision history. Tier 3 (bulk extracts, ~1,860 files) is the volume play that populates the long tail of entities. Each tier runs as a batch with progress logging. After Tier 3 completes, run the verification queries from the ingest queue to confirm entity counts are in the 20K–50K range. If counts are an order of magnitude low, diagnose whether the extraction format changed or the ingest script is dropping records. **Deliverable:** `memex.db` with 20K+ entities, all source files ingested, `ingest_log` table showing per-file status. **Duration estimate:** 1 AWL session (overnight Saturday or first available budget-clear window). ### Phase 2: Deduplication & canonicalization **Status:** Design exists. Blocked on Phase 1. Run entity resolution across all ingested records. The problem: "Ken Lovell," "Ken," "Walter Kenneth Lovell," and "Dad" are the same entity. Similarly, "PGA TOUR" and "the Tour" and "work" collapse. Phase 2 uses a two-pass approach: (a) deterministic rules for exact/near-exact name matches (Levenshtein ≤ 2, case-insensitive), then (b) Haiku-assisted disambiguation for ambiguous cases (e.g., "Paul" — is this Paul Gardner or a different Paul?). Canonical names are written to `canonical_name` field; `mention_count` is recalculated post-merge. Entities that merge get their edges transferred to the surviving canonical entity. **Deliverable:** Entity count drops 30–50% as duplicates merge. Top-20 entities by mention count are all real, distinct concepts. Zero orphaned edges. **Duration estimate:** 1–2 AWL sessions (Haiku calls are cheap but numerous). ### Phase 3: Edge inference (D4 v2) **Status:** Spec exists (D4 v2). Blocked on Phase 2 (needs clean entities). Fire Haiku pair-evaluation across the top-200 entities (by mention count) to infer relationships. For each entity pair, Haiku receives both entities' metadata + a sample of co-occurring source passages, then outputs: `edge_kind` (from a controlled vocabulary), `rationale` (one sentence), `strength` (0.0–1.0). Pairs with strength < 0.3 are discarded. This produces the initial graph skeleton. A second pass extends to entities ranked 201–1000, but only evaluates pairs that co-occur in at least one source file (reducing the quadratic explosion). Final pass: transitive inference — if A→B and B→C with strength > 0.7, evaluate A→C directly. **Deliverable:** 50K–200K edges in `memex_edges`. Graph is queryable for relationship paths. `edge_kind` distribution is diverse (not dominated by a single type). **Duration estimate:** 2–3 AWL sessions (token-heavy; budget-gate sensitive). ### Phase 4: Query layer & tool wrappers **Status:** #93/#94 approved. Blocked on Phase 3 (needs graph to be useful). Build the query interface that translates natural-language questions into graph traversals + optional Haiku reranking. Architecture: (a) keyword extraction from query, (b) entity lookup by name/alias, (c) edge traversal to find related entities, (d) source-file retrieval for citations, (e) optional Haiku pass to synthesize a natural-language answer from raw graph results. Expose this as two tool-call wrappers: one for Cowork MCP (MBP sessions call it via tool use) and one for Claude Code local (Mac Mini agents call it as a local function). Both wrappers share the same underlying query engine; the difference is transport (MCP protocol vs. direct Python import). **Deliverable:** `memex_query("What medications do I take and why?")` returns a structured response with entities, edges, source citations, and an optional prose summary. All 7 acceptance test queries (§1B) pass. **Duration estimate:** 1 AWL session for engine + 1 cowork session for wrapper integration/testing. ### Phase 5: Surface integration & steady-state operations **Status:** Dependent on Phase 4. Design constraints known (meds above Phase 1 on /morning, Ken-initiated /evening). Wire Memex query results into the CP surfaces that Ken uses daily. `/morning` gets an "On this day" block (historical context from Memex: what happened on this date in prior years, relevant teaching prep if Sunday School day, family birthdays/anniversaries). `/evening` gets a "today's decisions" block summarizing what was ratified/killed during the day's cowork sessions, cross-referenced to affected projects. The file-watcher (#95) goes live: every new dispatch or state-file change triggers incremental ingest within 5 minutes, keeping the graph current without manual intervention. The AWL gets a recurring Memex-health check: weekly verification queries confirming entity/edge counts haven't regressed, with a one-line dispatch if anomalies are detected. **Deliverable:** Ken's daily surfaces are Memex-aware. New information flows into the graph automatically. The system is self-maintaining. Goal 1 acceptance criteria (§1) all pass. Goal 1 is declared complete. **Duration estimate:** 1 cowork session for /morning integration, 1 for /evening, 1 AWL session for file-watcher + health checks. Total: ~1 week of calendar time given AWL budget constraints. --- *End of Part 1. Part 2 covers: §4 Detailed Phase 1 execution plan (file-by-file ingest order, error handling, rollback), §5 Schema extensions needed for Phases 2–5, §6 Token budget projection across all phases, §7 Risk register & mitigations, §8 Verification & sign-off protocol.* ## 4. Phase 1 — Ingest deploy (#241) **Goal:** Deploy `ingest_dispatches.py` and run the first full ingest pass across all four tiers, populating `memex.db` with 20K–50K entities and initial edges. **Dependency:** DB schema #241 RATIFIED 2026-05-16. No code changes needed to schema — deploy the ingest script as-is. ### 4.1 Files to touch | Artifact (source) | Target (deploy location) | Action | |---|---|---| | `Claude/scripts/ingest_dispatches.py` | `Claude/scripts/ingest_dispatches.py` (in-place) | Verify current, `chmod +x` | | `Claude/state/memex/memex.db` | `Claude/state/memex/memex.db` | Target DB — schema must already have `memex_entities`, `memex_edges` tables per #241 | | `Claude/state/memex/ingest.log` | (created on first run) | Append-only ingest log | | `Claude/state/memex/ingest-manifest.json` | (created on first run) | Tracks which files have been ingested and when, prevents double-ingest | ### 4.2 Pre-flight checks Run these before any ingest. Every check must pass. ```bash # 1. Confirm schema is deployed sqlite3 Claude/state/memex/memex.db ".schema memex_entities" | head -5 # Should show CREATE TABLE with columns: id, name, entity_type, mention_count, first_seen, last_seen, source_file, raw_json sqlite3 Claude/state/memex/memex.db ".schema memex_edges" | head -5 # Should show CREATE TABLE with columns: id, source_entity_id, target_entity_id, edge_kind, rationale, strength, inferred_at # 2. Confirm ingest script exists and is executable ls -la Claude/scripts/ingest_dispatches.py chmod +x Claude/scripts/ingest_dispatches.py # 3. Confirm source files exist (spot check each tier) test -f "Claude/state/canon/ken-decisions-2026-05-16.md" && echo "Tier 1 OK" ls Claude/state/projects/dispatch-extracts/*.json | wc -l # expect ~800 ls Claude/state/projects/onenote-extracts-2026-05-16/*.json | wc -l # expect ~961 # 4. Back up empty DB before first write cp Claude/state/memex/memex.db Claude/state/memex/memex-pre-ingest-backup.db ``` ### 4.3 Ingest execution — Tiered order Order is load-bearing. Tier 1 establishes canonical entity anchors (Ken, CP, Memex, key people). Tier 2 adds strategic context. Tier 3 is the volume multiplier. Tier 4 is backfill that can fail without blocking. **Tier 1 — Canonical files (run first, one at a time, verify between)** ```bash # Each command assumes ingest_dispatches.py accepts: # --db target database # --source file or directory to ingest # --tier <1-4> priority tier (affects entity confidence scores) # --log append-only log MEMEX_DB="Claude/state/memex/memex.db" INGEST="python3 Claude/scripts/ingest_dispatches.py" LOG="Claude/state/memex/ingest.log" # Canon files — order: decisions first (freshest context), then routine, then operating manual $INGEST --db $MEMEX_DB --source "Claude/state/canon/ken-decisions-2026-05-16.md" --tier 1 --log $LOG $INGEST --db $MEMEX_DB --source "Claude/state/canon/workout-routines.md" --tier 1 --log $LOG $INGEST --db $MEMEX_DB --source "Claude/state/canon/ken-operating-manual-v1-2026-05-16.md" --tier 1 --log $LOG $INGEST --db $MEMEX_DB --source "Claude/state/canon/cp-flask-port.md" --tier 1 --log $LOG $INGEST --db $MEMEX_DB --source "Claude/state/canon/evening-v2-save-schema.md" --tier 1 --log $LOG $INGEST --db $MEMEX_DB --source "Claude/state/canon/memex-watchdog-path.md" --tier 1 --log $LOG $INGEST --db $MEMEX_DB --source "Claude/state/health/meds-stack.md" --tier 1 --log $LOG # Verify Tier 1 landed sqlite3 $MEMEX_DB "SELECT COUNT(*) AS tier1_entities FROM memex_entities;" # Expected: 200–800 entities (people, places, meds, exercises, projects, decisions) sqlite3 $MEMEX_DB "SELECT name, entity_type, mention_count FROM memex_entities ORDER BY mention_count DESC LIMIT 10;" # "Ken" should be #1 or #2. Key entities like "Memex", "CP", workout names should appear. ``` **Tier 2 — Strategic docs** ```bash $INGEST --db $MEMEX_DB --source "Claude/state/projects/master-complete-all-projects-plan-2026-05-16.md" --tier 2 --log $LOG $INGEST --db $MEMEX_DB --source "Claude/state/projects/deferred-decisions-goal3-2026-05-16.md" --tier 2 --log $LOG $INGEST --db $MEMEX_DB --source "Claude/state/projects/orphan-work-audit-2026-05-16.md" --tier 2 --log $LOG $INGEST --db $MEMEX_DB --source "Claude/state/projects/biography-chapter-outline-2026-05-16.md" --tier 2 --log $LOG # Sprint plans directory — batch ingest $INGEST --db $MEMEX_DB --source "Claude/state/projects/sprint-plans-2026-05-16/" --tier 2 --log $LOG # Verify Tier 2 incremental sqlite3 $MEMEX_DB "SELECT COUNT(*) AS after_tier2 FROM memex_entities;" # Expected: 1K–3K cumulative ``` **Tier 3 — Bulk extracts (largest volume, run with batch flags)** ```bash # Dispatch extracts — ~800 JSON files, largest graph multiplier $INGEST --db $MEMEX_DB --source "Claude/state/projects/dispatch-extracts/" --tier 3 --batch --log $LOG # OneNote extracts — ~961 JSON files, 14+ years of notes $INGEST --db $MEMEX_DB --source "Claude/state/projects/onenote-extracts-2026-05-16/" --tier 3 --batch --log $LOG # Bio extracts — 48 JSON files $INGEST --db $MEMEX_DB --source "Claude/state/projects/bio-extracts-2026-05-16/" --tier 3 --batch --log $LOG # 52-week replay — 52 MD files $INGEST --db $MEMEX_DB --source "Claude/state/projects/52-week-replay-2026-05-16/" --tier 3 --batch --log $LOG ``` **Tier 4 — Backfill (can fail without blocking Phase 2)** ```bash # Master index — large JSONL, low priority $INGEST --db $MEMEX_DB --source "Claude/state/projects/master-index-2026-05-16.jsonl" --tier 4 --log $LOG # Synthesis re-ingest (only files older than deploy date) $INGEST --db $MEMEX_DB --source "Claude/state/projects/synthesis/" --tier 4 --since "2026-05-17" --log $LOG ``` ### 4.4 Smoke tests Run immediately after Tier 3 completes (don't wait for Tier 4). ```sql -- 1. Entity count SELECT COUNT(*) FROM memex_entities; -- PASS: 20,000–50,000. FAIL: <5,000 (ingest missed sources) or >200,000 (dedup broken) -- 2. Top entities by mention count SELECT name, entity_type, mention_count FROM memex_entities ORDER BY mention_count DESC LIMIT 20; -- PASS: "Ken" in top 3. At least 5 distinct entity_types represented. -- FAIL: single entity_type dominates, or mention_counts are all 1 (aggregation broken) -- 3. Entity type distribution SELECT entity_type, COUNT(*) as cnt FROM memex_entities GROUP BY entity_type ORDER BY cnt DESC; -- PASS: at least person, project, medication, exercise, location types present -- FAIL: only 1-2 types (extraction too narrow) -- 4. Source file coverage SELECT COUNT(DISTINCT source_file) AS files_ingested FROM memex_entities; -- PASS: >1,800 (Tier 1-3 sources). FAIL: <500 (many files skipped) -- 5. Edge count (edges may be zero if D4 v2 hasn't run yet — that's OK) SELECT COUNT(*) FROM memex_edges; -- NOTE: If ingest_dispatches.py creates edges during ingest, expect 50K-200K. -- If edges are D4-only, expect 0 here. Both are valid depending on implementation. -- 6. Edge kind distribution (if edges exist) SELECT edge_kind, COUNT(*) FROM memex_edges GROUP BY edge_kind ORDER BY 2 DESC; -- PASS: multiple edge_kinds (related_to, mentioned_with, part_of, etc.) ``` ### 4.5 Rollback plan **If ingest produces garbage data (wrong counts, corrupt entities):** ```bash # Restore pre-ingest backup cp Claude/state/memex/memex-pre-ingest-backup.db Claude/state/memex/memex.db # Review ingest.log for errors tail -100 Claude/state/memex/ingest.log # Fix the issue, re-run from the tier that failed ``` **If ingest script crashes mid-tier:** The manifest (`ingest-manifest.json`) tracks per-file completion. Re-running the same command should skip already-ingested files and resume. If the manifest itself is corrupt: ```bash # Check last successful file in log grep "SUCCESS" Claude/state/memex/ingest.log | tail -5 # Delete manifest and re-run with --skip-existing flag (dedup by source_file in DB) rm Claude/state/memex/ingest-manifest.json $INGEST --db $MEMEX_DB --source --tier --skip-existing --log $LOG ``` **Nuclear rollback (start completely fresh):** ```bash cp Claude/state/memex/memex-pre-ingest-backup.db Claude/state/memex/memex.db rm Claude/state/memex/ingest-manifest.json rm Claude/state/memex/ingest.log # Re-run entire sequence from Tier 1 ``` ### 4.6 Time estimate | Step | Duration | |---|---| | Pre-flight checks | 2 min | | Tier 1 (7 files) | 3–5 min | | Tier 2 (~30 files) | 5–10 min | | Tier 3 (~1,861 files) | 20–45 min (depends on Haiku extraction speed) | | Tier 4 (backfill) | 10–20 min | | Smoke tests + verification | 5 min | | **Total** | **45 min – 1.5 hr** | Tier 3 is the bottleneck. If each JSON file takes ~1s for entity extraction, 1,861 files ≈ 31 minutes. If the script supports parallelism (`--workers 4`), this drops to ~8 minutes. ### 4.7 Deferred: Reflect notes (D2) Not part of this phase. After Phase 1 completes, add a task to either: - (a) Reconnect Reflect MCP and ingest via API, or - (b) Export Reflect data to `Claude/state/projects/reflect-export/` and run as Tier 3 This is a Phase 1 follow-up, not a blocker. --- ## 5. Phase 2 — Auto-refresh file-watcher (#95) **Goal:** Deploy a file-system watcher that detects changes to canon files, dispatches, and other Memex sources, then triggers incremental re-ingest automatically. Eliminates manual re-runs of `ingest_dispatches.py`. **Dependency:** Phase 1 complete (DB populated, ingest script proven). ### 5.1 Files to touch | Artifact | Path | Action | |---|---|---| | Watchdog script | `Claude/scripts/memex_watchdog.py` | Create or update — the file-watcher daemon | | Watchdog config | `Claude/state/memex/watchdog-config.json` | Create — defines watched paths, debounce intervals, tier mappings | | Watchdog service | `Claude/scripts/memex-watchdog.plist` (macOS) | Create — launchd plist for auto-start | | Canon reference | `Claude/state/canon/memex-watchdog-path.md` | Update — reflect deployed config | | Ingest script | `Claude/scripts/ingest_dispatches.py` | May need `--incremental` flag if not already present | ### 5.2 Watchdog config design ```json { "watch_paths": [ { "path": "Claude/state/canon/", "tier": 1, "glob": "*.md", "debounce_seconds": 5, "description": "Canon files — highest priority, immediate re-ingest" }, { "path": "Claude/state/health/", "tier": 1, "glob": "*.md", "debounce_seconds": 5 }, { "path": "Claude/state/projects/dispatch-extracts/", "tier": 3, "glob": "*.json", "debounce_seconds": 30, "description": "Dispatch extracts — batch after 30s quiet period" }, { "path": "Claude/state/projects/sprint-plans-*/", "tier": 2, "glob": "*.md", "debounce_seconds": 15 }, { "path": "Claude/state/projects/synthesis/", "tier": 4, "glob": "*.md", "debounce_seconds": 60 } ], "db_path": "Claude/state/memex/memex.db", "log_path": "Claude/state/memex/watchdog.log", "max_batch_size": 50, "ingest_command": "python3 Claude/scripts/ingest_dispatches.py" } ``` ### 5.3 Deploy commands ```bash # 1. Install watchdog dependency (if not present) pip3 install watchdog # 2. Verify watchdog script python3 Claude/scripts/memex_watchdog.py --dry-run --config Claude/state/memex/watchdog-config.json # Should print: "Watching 5 paths, dry-run mode, no DB writes" # 3. Test with a real file change python3 Claude/scripts/memex_watchdog.py --config Claude/state/memex/watchdog-config.json & WATCHDOG_PID=$! # Trigger a change touch "Claude/state/canon/workout-routines.md" sleep 10 # Check log for ingest trigger grep "workout-routines" Claude/state/memex/watchdog.log kill $WATCHDOG_PID # 4. Install as launchd service (persistent across reboots) cp Claude/scripts/memex-watchdog.plist ~/Library/LaunchAgents/com.ken.memex-watchdog.plist launchctl load ~/Library/LaunchAgents/com.ken.memex-watchdog.plist launchctl list | grep memex-watchdog # Should show PID and status 0 ``` ### 5.4 Launchd plist ```xml Label com.ken.memex-watchdog ProgramArguments /usr/bin/python3 Claude/scripts/memex_watchdog.py --config Claude/state/memex/watchdog-config.json WorkingDirectory /Users/kenlovell/Library/CloudStorage/OneDrive-Personal RunAtLoad KeepAlive StandardOutPath Claude/state/memex/watchdog-stdout.log StandardErrorPath Claude/state/memex/watchdog-stderr.log ``` ### 5.5 Smoke tests ```bash # 1. Service is running launchctl list | grep memex-watchdog # PASS: shows PID, exit status 0 # 2. File change triggers ingest echo "# test change" >> "Claude/state/canon/workout-routines.md" sleep 15 sqlite3 Claude/state/memex/memex.db \ "SELECT last_seen FROM memex_entities WHERE source_file LIKE '%workout%' ORDER BY last_seen DESC LIMIT 1;" # PASS: last_seen is within last 60 seconds # 3. Debounce works (rapid changes don't cause N ingests) for i in {1..10}; do echo "# change $i" >> "Claude/state/canon/workout-routines.md"; done sleep 15 grep "workout-routines" Claude/state/memex/watchdog.log | tail -5 # PASS: only 1 ingest triggered (debounce collapsed 10 writes into 1) # 4. Watchdog recovers from crash launchctl stop com.ken.memex-watchdog sleep 5 launchctl list | grep memex-watchdog # PASS: PID present (KeepAlive restarted it) ``` ### 5.6 Rollback plan ```bash # Stop and unload service launchctl unload ~/Library/LaunchAgents/com.ken.memex-watchdog.plist rm ~/Library/LaunchAgents/com.ken.memex-watchdog.plist # Watchdog never deletes entities — worst case is duplicate ingests. # If duplicates occurred: sqlite3 Claude/state/memex/memex.db " DELETE FROM memex_entities WHERE id NOT IN ( SELECT MIN(id) FROM memex_entities GROUP BY name, entity_type, source_file ); " ``` ### 5.7 Time estimate | Step | Duration | |---|---| | Write/update `memex_watchdog.py` | 30–60 min | | Write config + plist | 15 min | | Test dry-run + live change | 15 min | | Install launchd service | 5 min | | Smoke tests | 10 min | | **Total** | **1.5–2 hr** | --- ## 6. Phase 3 — Tool wrappers (#93 Cowork MCP, #94 CC local) **Goal:** Expose Memex query capabilities through two interfaces: (a) a Claude Code local tool for Ken's direct use in CC sessions, and (b) an MCP server for Cowork agents to query Memex during multi-agent orchestration. **Dependency:** Phase 1 complete (entities in DB). Phase 2 nice-to-have but not required. ### 6.1 Architecture ``` ┌─────────────────┐ ┌────────────────────┐ │ Claude Code │ │ Cowork MCP Server │ │ (Ken's session) │ │ (agent sessions) │ │ │ │ │ │ CC local tool │ │ MCP tool endpoint │ │ memex_query() │ │ memex_query() │ └───────┬──────────┘ └────────┬───────────┘ │ │ └──────────┬──────────────┘ │ ┌───────▼────────┐ │ memex_core.py │ ← shared query engine │ (library) │ └───────┬────────┘ │ ┌───────▼────────┐ │ memex.db │ │ (SQLite) │ └────────────────┘ ``` Both wrappers call the same `memex_core.py` query library. This avoids duplicating SQL logic. ### 6.2 Files to touch | Artifact | Path | Action | |---|---|---| | Shared query library | `Claude/scripts/memex_core.py` | Create — pure-Python query functions, no framework deps | | CC local tool config | `Claude/.claude/settings.local.json` | Update — register `memex` as a local tool | | CC tool wrapper | `Claude/scripts/memex_cc_tool.py` | Create — thin wrapper that CC invokes | | MCP server | `Claude/scripts/memex_mcp_server.py` | Create — MCP-protocol server exposing memex tools | | MCP config | `Claude/cowork/mcp-config.json` (or wherever Cowork reads MCP) | Update — register Memex MCP | ### 6.3 `memex_core.py` — Query interface The core library exposes these functions: ```python # memex_core.py — shared query engine for Memex def search_entities(db_path: str, query: str, entity_type: str = None, limit: int = 20) -> list[dict]: """Full-text search across entity names. Returns ranked matches.""" def get_entity(db_path: str, entity_id: int) -> dict: """Single entity by ID with all metadata.""" def get_entity_edges(db_path: str, entity_id: int, edge_kind: str = None) -> list[dict]: """All edges for an entity, optionally filtered by kind.""" def get_related(db_path: str, entity_name: str, hops: int = 1, limit: int = 50) -> list[dict]: """Graph traversal: find entities within N hops of named entity.""" def get_entity_timeline(db_path: str, entity_name: str) -> list[dict]: """Chronological appearances of an entity across source files.""" def stats(db_path: str) -> dict: """DB health stats: entity count, edge count, type distribution, last ingest time.""" ``` ### 6.4 CC local tool (#94) **`Claude/scripts/memex_cc_tool.py`:** ```python #!/usr/bin/env python3 """Claude Code local tool wrapper for Memex queries.""" import sys, json from memex_core import search_entities, get_related, get_entity_timeline, stats DB = "Claude/state/memex/memex.db" def main(): action = sys.argv[1] if len(sys.argv) > 1 else "search" query = sys.argv[2] if len(sys.argv) > 2 else "" if action == "search": results = search_entities(DB, query) elif action == "related": results = get_related(DB, query) elif action == "timeline": results = get_entity_timeline(DB, query) elif action == "stats": results = stats(DB) else: results = {"error": f"Unknown action: {action}"} print(json.dumps(results, indent=2, default=str)) if __name__ == "__main__": main() ``` **Registration in CC settings:** Add to `Claude/.claude/settings.local.json`: ```json { "tools": { "memex": { "command": "python3 Claude/scripts/memex_cc_tool.py", "description": "Query Ken's Memex knowledge graph. Actions: search , related , timeline , stats" } } } ``` **Usage in a CC session:** ``` > /memex search "workout routine" > /memex related "Ken" > /memex timeline "CP Flask" > /memex stats ``` ### 6.5 Cowork MCP server (#93) **`Claude/scripts/memex_mcp_server.py`:** Implements the MCP (Model Context Protocol) server spec. Key tools exposed: | MCP Tool Name | Maps to | Description | |---|---|---| | `memex_search` | `search_entities()` | Search entities by name/keyword | | `memex_related` | `get_related()` | Find entities related to a named entity | | `memex_timeline` | `get_entity_timeline()` | Get chronological entity appearances | | `memex_stats` | `stats()` | DB health check | The MCP server runs as a subprocess that Cowork spawns. It communicates over stdin/stdout using the MCP JSON-RPC protocol. ### 6.6 Deploy commands ```bash # 1. Create the shared library # (write memex_core.py — see 6.3 for interface) # 2. Test core library directly python3 -c " from Claude.scripts.memex_core import search_entities, stats import json print(json.dumps(stats('Claude/state/memex/memex.db'), indent=2)) print(json.dumps(search_entities('Claude/state/memex/memex.db', 'Ken', limit=5), indent=2)) " # 3. Test CC tool wrapper python3 Claude/scripts/memex_cc_tool.py stats python3 Claude/scripts/memex_cc_tool.py search "workout" python3 Claude/scripts/memex_cc_tool.py related "Ken" # 4. Test MCP server (manual JSON-RPC) echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | python3 Claude/scripts/memex_mcp_server.py # Should return list of 4 tools echo '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"memex_search","arguments":{"query":"Ken"}}}' \ | python3 Claude/scripts/memex_mcp_server.py # Should return entity results # 5. Register MCP with Cowork config # (update cowork MCP config to include memex_mcp_server.py) # 6. Verify CC tool registration # Open a new CC session and run: # /memex stats # Should return entity/edge counts ``` ### 6.7 Smoke tests ```bash # CC tool: search returns results python3 Claude/scripts/memex_cc_tool.py search "creatine" | python3 -c " import sys, json data = json.load(sys.stdin) assert len(data) > 0, 'No results for creatine' print(f'PASS: {len(data)} results') " # CC tool: stats returns non-zero counts python3 Claude/scripts/memex_cc_tool.py stats | python3 -c " import sys, json data = json.load(sys.stdin) assert data['entity_count'] > 10000, f'Low entity count: {data[\"entity_count\"]}' print(f'PASS: {data[\"entity_count\"]} entities, {data[\"edge_count\"]} edges') " # MCP: tool list returns expected tools echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' \ | python3 Claude/scripts/memex_mcp_server.py \ | python3 -c " import sys, json resp = json.load(sys.stdin) tools = [t['name'] for t in resp['result']['tools']] for expected in ['memex_search', 'memex_related', 'memex_timeline', 'memex_stats']: assert expected in tools, f'Missing tool: {expected}' print(f'PASS: all 4 tools registered') " # MCP: related query returns graph neighbors echo '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"memex_related","arguments":{"entity_name":"Ken","hops":1}}}' \ | python3 Claude/scripts/memex_mcp_server.py \ | python3 -c " import sys, json resp = json.load(sys.stdin) results = resp['result']['content'][0]['text'] data = json.loads(results) assert len(data) > 5, f'Too few related entities: {len(data)}' print(f'PASS: {len(data)} entities related to Ken') " ``` ### 6.8 Rollback plan - **CC tool:** Remove the `memex` entry from `Claude/.claude/settings.local.json`. Tool immediately stops being available in new sessions. - **MCP server:** Remove Memex entry from Cowork MCP config. Restart Cowork. Agents lose access but no data is affected. - Neither wrapper writes to the DB — they are read-only. No data rollback needed. ### 6.9 Time estimate | Step | Duration | |---|---| | Write `memex_core.py` (query library) | 1–2 hr | | Write + test CC tool wrapper | 30 min | | Write + test MCP server | 1–1.5 hr | | Register in configs | 15 min | | Integration smoke tests | 30 min | | **Total** | **3–4.5 hr** | --- ## 7. Phase 4 — D4 v2 full pair-evaluation **Goal:** Run a full Haiku-powered pair-evaluation across the top entities in `memex_entities`, generating high-quality relationship edges in `memex_edges`. This is what transforms the entity list into a knowledge graph. **Dependency:** Phase 1 complete with 20K+ entities. Phase 3 helpful for verification but not required. ### 7.1 What D4 v2 does D4 v2 takes the top-N entities (by mention count or strategic importance), generates all meaningful pairs, sends each pair to Claude Haiku for relationship inference, and writes the results as edges. Each edge record: ``` (source_entity_id, target_entity_id, edge_kind, rationale, strength, inferred_at) ``` **Edge kinds** (expected from pair-eval): - `related_to` — general semantic relationship - `part_of` — entity is a component/member of another - `works_with` — people who collaborate - `depends_on` — project/technical dependency - `prescribed_for` — medication → condition - `located_at` — entity → place - `created_by` — artifact → creator - `temporal_overlap` — entities active in same time window - `contradicts` — entities with conflicting information ### 7.2 Files to touch | Artifact | Path | Action | |---|---|---| | Pair-eval script | `Claude/scripts/memex_pair_eval.py` | Create or update — the D4 v2 orchestrator | | Pair-eval config | `Claude/state/memex/pair-eval-config.json` | Create — controls N, batch size, Haiku params | | Edge schema | Already in `memex.db` per #241 | Verify columns match expected tuple | | Pair-eval log | `Claude/state/memex/pair-eval.log` | Created on run | | Pair-eval checkpoint | `Claude/state/memex/pair-eval-checkpoint.json` | Created on run — resume point if interrupted | ### 7.3 Pair-eval config ```json { "db_path": "Claude/state/memex/memex.db", "top_n_entities": 200, "entity_selection": "top_by_mention_count", "max_pairs": 19900, "batch_size": 50, "model": "claude-haiku-4-5-20251001", "max_tokens_per_pair": 256, "min_strength_threshold": 0.3, "edge_kinds": [ "related_to", "part_of", "works_with", "depends_on", "prescribed_for", "located_at", "created_by", "temporal_overlap", "contradicts" ], "prompt_template": "Given two entities from a personal knowledge base, determine if they have a meaningful relationship. Entity A: {entity_a} (type: {type_a}). Entity B: {entity_b} (type: {type_b}). If related, respond with JSON: {\"edge_kind\": \"...\", \"rationale\": \"...\", \"strength\": 0.0-1.0}. If unrelated, respond with {\"edge_kind\": null}.", "checkpoint_every_n_batches": 10, "log_path": "Claude/state/memex/pair-eval.log" } ``` ### 7.4 Execution math - Top 200 entities → C(200,2) = 19,900 pairs - At 50 pairs/batch → 398 Haiku calls - Haiku at ~0.5s/call → ~200 seconds for the API calls - Many pairs will be unrelated (edge_kind: null) — expect 30–50% hit rate → 6K–10K edges written - Cost: 19,900 pairs × ~100 input tokens × $0.80/MTok + ~50 output tokens × $4/MTok ≈ $5–8 ### 7.5 Deploy commands ```bash MEMEX_DB="Claude/state/memex/memex.db" # 1. Pre-flight: verify entity count is sufficient sqlite3 $MEMEX_DB "SELECT COUNT(*) FROM memex_entities;" # Must be >10,000 — if not, Phase 1 isn't complete # 2. Verify top entities look reasonable sqlite3 $MEMEX_DB " SELECT name, entity_type, mention_count FROM memex_entities ORDER BY mention_count DESC LIMIT 20; " # Sanity check: Ken, key projects, meds, people should appear # 3. Backup DB before pair-eval writes cp $MEMEX_DB Claude/state/memex/memex-pre-paireval-backup.db # 4. Dry run — generates pairs, estimates cost, writes nothing python3 Claude/scripts/memex_pair_eval.py \ --config Claude/state/memex/pair-eval-config.json \ --dry-run # Should print: "19,900 pairs, estimated 398 Haiku calls, ~$6" # 5. Full run python3 Claude/scripts/memex_pair_eval.py \ --config Claude/state/memex/pair-eval-config.json \ --log Claude/state/memex/pair-eval.log # 6. If interrupted, resume from checkpoint python3 Claude/scripts/memex_pair_eval.py \ --config Claude/state/memex/pair-eval-config.json \ --resume Claude/state/memex/pair-eval-checkpoint.json \ --log Claude/state/memex/pair-eval.log ``` ### 7.6 Smoke tests ```sql -- 1. Edge count SELECT COUNT(*) FROM memex_edges; -- PASS: 5,000–15,000. FAIL: <1,000 (threshold too high or Haiku returning null) -- FAIL: >50,000 (threshold too low, noise edges) -- 2. Edge kind distribution SELECT edge_kind, COUNT(*) AS cnt FROM memex_edges GROUP BY edge_kind ORDER BY cnt DESC; -- PASS: at least 5 distinct edge_kinds. "related_to" will be largest but shouldn't be >60%. -- FAIL: single edge_kind dominates >80% (prompt too generic) -- 3. Strength distribution SELECT CASE WHEN strength >= 0.8 THEN 'high (0.8-1.0)' WHEN strength >= 0.5 THEN 'medium (0.5-0.8)' ELSE 'low (0.3-0.5)' END AS band, COUNT(*) AS cnt FROM memex_edges GROUP BY band; -- PASS: bell-curve-ish distribution. FAIL: all edges same strength (Haiku not differentiating) -- 4. Ken's neighborhood SELECT e2.name, me.edge_kind, me.strength, me.rationale FROM memex_edges me JOIN memex_entities e1 ON me.source_entity_id = e1.id JOIN memex_entities e2 ON me.target_entity_id = e2.id WHERE e1.name = 'Ken' ORDER BY me.strength DESC LIMIT 20; -- PASS: meaningful relationships (projects, people, health items) -- This is the "does it feel right" test -- 5. Spot check rationales SELECT e1.name, e2.name, me.edge_kind, me.rationale FROM memex_edges me JOIN memex_entities e1 ON me.source_entity_id = e1.id JOIN memex_entities e2 ON me.target_entity_id = e2.id WHERE me.strength > 0.7 ORDER BY RANDOM() LIMIT 10; -- PASS: rationales are coherent English that explain the connection -- FAIL: generic ("they are related") or hallucinated connections ``` ### 7.7 Quality gate After smoke tests, manually review 20 random high-strength edges. If >3 of 20 are nonsensical: 1. Check prompt template — may need more context (entity descriptions, source files) 2. Raise `min_strength_threshold` to 0.5 3. Delete bad edges: `DELETE FROM memex_edges WHERE strength < 0.5;` 4. Re-run with updated config If quality is good, consider expanding to top-500 entities (124,750 pairs, ~$20–30, ~15 min with batching). ### 7.8 Rollback plan ```bash # Restore pre-pair-eval backup cp Claude/state/memex/memex-pre-paireval-backup.db Claude/state/memex/memex.db # Or: delete only edges (preserve entities) sqlite3 Claude/state/memex/memex.db "DELETE FROM memex_edges WHERE inferred_at > '2026-05-17';" # Or: delete edges below quality threshold sqlite3 Claude/state/memex/memex.db "DELETE FROM memex_edges WHERE strength < 0.4;" ``` ### 7.9 Time estimate | Step | Duration | |---|---| | Write/update `memex_pair_eval.py` | 1–2 hr | | Dry run + config tuning | 15 min | | Full run (top 200) | 5–15 min (API time) | | Smoke tests + quality review | 30 min | | Optional: expand to top 500 | 15–30 min | | **Total** | **2–3.5 hr** | --- ## 8. Phase 5 — Production **Goal:** Transition Memex from "deployed and working" to "production-grade" — meaning it runs unattended, alerts on failure, stays healthy over time, and has a defined maintenance cadence. ### 8.1 Production checklist Before declaring Goal 1 complete, every item must be checked: - [ ] Phase 1 smoke tests pass (entities: 20K+, source coverage: 1,800+ files) - [ ] Phase 2 watchdog running via launchd, survives reboot - [ ] Phase 3 CC tool returns results in <2s, MCP server responds to JSON-RPC - [ ] Phase 4 edges: 5K+, quality gate passed (manual review of 20 edges) - [ ] Backup exists: `memex-pre-ingest-backup.db`, `memex-pre-paireval-backup.db` - [ ] All scripts have `--help` flags and exit with non-zero on error - [ ] `ingest.log`, `watchdog.log`, `pair-eval.log` exist and are being written to ### 8.2 Monitoring **Daily health check (add to evening route or cron):** ```bash #!/bin/bash # Claude/scripts/memex-health-check.sh MEMEX_DB="Claude/state/memex/memex.db" ENTITY_COUNT=$(sqlite3 $MEMEX_DB "SELECT COUNT(*) FROM memex_entities;") EDGE_COUNT=$(sqlite3 $MEMEX_DB "SELECT COUNT(*) FROM memex_edges;") LAST_INGEST=$(sqlite3 $MEMEX_DB "SELECT MAX(last_seen) FROM memex_entities;") DB_SIZE=$(du -h $MEMEX_DB | cut -f1) WATCHDOG_RUNNING=$(launchctl list | grep memex-watchdog | awk '{print $1}') echo "=== Memex Health Check ===" echo "Entities: $ENTITY_COUNT" echo "Edges: $EDGE_COUNT" echo "Last ingest: $LAST_INGEST" echo "DB size: $DB_SIZE" echo "Watchdog PID: ${WATCHDOG_RUNNING:-NOT RUNNING}" # Alerts if [ "$ENTITY_COUNT" -lt 10000 ]; then echo "⚠ ALERT: Entity count dropped below 10K" fi if [ -z "$WATCHDOG_RUNNING" ] || [ "$WATCHDOG_RUNNING" = "-" ]; then echo "⚠ ALERT: Watchdog not running" fi # Check if last ingest was more than 48 hours ago LAST_EPOCH=$(date -j -f "%Y-%m-%d %H:%M:%S" "$LAST_INGEST" "+%s" 2>/dev/null || echo 0) NOW_EPOCH=$(date "+%s") HOURS_SINCE=$(( (NOW_EPOCH - LAST_EPOCH) / 3600 )) if [ "$HOURS_SINCE" -gt 48 ]; then echo "⚠ ALERT: No ingest in ${HOURS_SINCE} hours" fi ``` ```bash # Make executable and test chmod +x Claude/scripts/memex-health-check.sh ./Claude/scripts/memex-health-check.sh ``` **Integration with evening route:** Add to the evening save flow (per `Claude/state/canon/evening-v2-save-schema.md`): ```python # In evening route, after saving state: import subprocess result = subprocess.run( ["bash", "Claude/scripts/memex-health-check.sh"], capture_output=True, text=True ) if "ALERT" in result.stdout: # Include in evening summary evening_summary["memex_alerts"] = result.stdout ``` ### 8.3 Automated backups ```bash # Add to cron or launchd — weekly DB backup # Claude/scripts/memex-backup.sh #!/bin/bash MEMEX_DB="Claude/state/memex/memex.db" BACKUP_DIR="Claude/state/memex/backups" mkdir -p $BACKUP_DIR TIMESTAMP=$(date "+%Y-%m-%d") cp $MEMEX_DB "$BACKUP_DIR/memex-$TIMESTAMP.db" # Keep last 8 backups (2 months of weeklies) ls -t $BACKUP_DIR/memex-*.db | tail -n +9 | xargs rm -f 2>/dev/null echo "Backup complete: memex-$TIMESTAMP.db ($(du -h "$BACKUP_DIR/memex-$TIMESTAMP.db" | cut -f1))" ``` ### 8.4 Quarterly maintenance Every 3 months, run: 1. **Entity dedup audit:** ```sql SELECT name, COUNT(*) AS dupes FROM memex_entities GROUP BY LOWER(name), entity_type HAVING dupes > 1 ORDER BY dupes DESC LIMIT 20; ``` Merge duplicates if found. 2. **Edge quality re-eval:** Sample 50 random edges, manually check 10. If >20% are stale or wrong, re-run D4 v2 on affected entity clusters. 3. **Source coverage check:** ```sql SELECT source_file, COUNT(*) AS entities FROM memex_entities GROUP BY source_file ORDER BY entities DESC LIMIT 20; ``` Verify new canon files and dispatches are being picked up by watchdog. 4. **DB size check:** If >500MB, consider archiving entities with `mention_count = 1` and `last_seen` older than 6 months to a separate archive DB. 5. **Expand pair-eval:** If entity count has grown significantly since last D4 run, run D4 v2 on the new top-N to cover recently added entities. ### 8.5 Failure modes and recovery | Failure | Detection | Recovery | |---|---|---| | Watchdog dies | Health check shows no PID | `launchctl load` — KeepAlive should auto-restart, but if launchd gave up, re-load | | DB corruption | `sqlite3 memex.db "PRAGMA integrity_check;"` returns errors | Restore latest weekly backup | | Ingest produces garbage | Entity count spikes 10x, or smoke test fails | Restore pre-ingest backup, fix script, re-run | | Haiku API changes | Pair-eval returns errors or empty edges | Update model ID in config, re-run affected batches | | Disk full | Ingest or watchdog logs errors | Archive old backups, compress logs, expand disk | | Entity count stagnates | Health check shows same count for weeks | Check watchdog log — may have stopped detecting changes. Verify watch_paths in config | ### 8.6 Success criteria for Goal 1 "Memex fully operational" Goal 1 is complete when ALL of the following are true: 1. **`memex.db` has 20K+ entities** from Tier 1–3 sources 2. **`memex_edges` has 5K+ edges** with quality gate passed 3. **Watchdog is running** and auto-ingests file changes within 60 seconds 4. **CC tool works** — `/memex search "Ken"` returns results in a live session 5. **MCP server works** — Cowork agents can query Memex via `memex_search` 6. **Health check runs daily** as part of evening route 7. **Weekly backups are in place** When these criteria are met, update `Claude/state/canon/ken-decisions-2026-05-16.md` (or its successor) with: ```markdown ## Goal 1: Memex fully operational - Status: COMPLETE - Completed: [date] - Entity count: [N] - Edge count: [N] - Components: ingest, watchdog, CC tool, MCP server, health check, backups ``` ### 8.7 What comes after Goal 1 With Memex operational, the following become possible: - **Cowork agents use Memex context** to make better decisions (they can query who/what/why before acting) - **Evening route enrichment** — evening save can query Memex to detect entity drift or contradictions - **Reflect integration** (D2 deferred) — 365-day backfill adds deep temporal context - **Cross-session continuity** — any Claude session can query Memex for context that would otherwise require re-reading dozens of files - **Goal 2+ acceleration** — subsequent goals (biography, health tracking, project management) build on the entity graph rather than re-extracting from raw text each time --- *End of Goal 1 Memex Specification, Part 2. Total estimated execution time across all phases: 8–12 hours of active work, spread across 2–3 sessions.* ## 9. Risk register Eight risks ranked by expected impact across all five phases. ### R1. Embedding drift after model upgrade | | | |---|---| | **Phase** | 2–5 (any phase using vector search) | | **Impact** | High — old embeddings become incompatible with new model's vector space; search quality silently degrades | | **Likelihood** | Medium — OpenAI and local model providers ship new embedding versions ~yearly | | **Detect** | Canary query suite: 20 known-good queries with expected top-3 results, run nightly. Alert when recall drops below 80%. Store `embedding_model_version` on every chunk row. | | **Fix** | 1. Freeze ingestion. 2. Run backfill script that re-embeds all chunks with the new model, writing to a shadow column. 3. Swap column atomically. 4. Re-run canary suite. 5. Resume ingestion. Budget: ~$2 for 100K chunks at current OpenAI pricing. | ### R2. Deduplication failure — same content ingested multiple times | | | |---|---| | **Phase** | 1–2 | | **Impact** | Medium — bloats storage, pollutes search results with near-identical hits | | **Likelihood** | High — emails forwarded, notes copied between apps, screenshots of the same page | | **Detect** | Post-ingest similarity scan: any new chunk with cosine similarity > 0.97 to an existing chunk in the same source gets flagged. Weekly report of duplicate clusters. | | **Fix** | Soft-delete duplicates, keeping the one with the richest metadata (more tags, earlier timestamp). Add content hash (`sha256` of normalized text) to the ingest pipeline as a pre-check gate. | ### R3. Stale entity graph — entities that no longer reflect reality | | | |---|---| | **Phase** | 3–5 | | **Impact** | Medium — briefings reference people Ken no longer works with, projects that ended, companies that were acquired | | **Detect** | Entity decay score: if no new edges have been added to an entity in 180 days, flag it. Quarterly report of "dormant entities" sorted by historical edge count. | | **Fix** | Dormant entities get `status: archived` — excluded from briefings and active search by default but still queryable with an explicit flag. Ken reviews the quarterly list (~5 min) and can override. | ### R4. PII leak through MCP surface | | | |---|---| | **Phase** | 4–5 | | **Impact** | High — sensitive health data, financial info, or private correspondence surfaced in a context where Ken didn't expect it | | **Likelihood** | Low-Medium — depends on what sources are connected | | **Detect** | Every MCP tool response includes a `sensitivity` field derived from the source's classification (email = medium, health = high, public notes = low). Claude's system prompt instructs it to flag high-sensitivity results before acting on them. | | **Fix** | 1. Add source-level ACL: `"surface_via_mcp": true/false` per source in config. 2. Default health and financial sources to `false`. 3. Ken opts in explicitly per source. 4. Audit log of every MCP query + results returned, retained 90 days. | ### R5. SQLite performance cliff at scale | | | |---|---| | **Phase** | 3–4 (when chunk count exceeds ~500K) | | **Impact** | Medium — search latency exceeds 2s, briefing generation times out | | **Likelihood** | Low — Ken's personal corpus is unlikely to exceed 200K chunks in 2 years, but bulk imports (full email archive) could spike it | | **Detect** | Query timing logged on every search. Alert if p95 > 1s. Monthly row-count check against projection curve. | | **Fix** | 1. Add composite indexes on `(source_type, created_at)` and `(entity_id, edge_type)`. 2. If still slow, partition chunks by year into separate tables with a union view. 3. Nuclear option: migrate to DuckDB for analytical queries while keeping SQLite for transactional writes. | ### R6. Ingestion pipeline silent failure | | | |---|---| | **Phase** | 1–5 | | **Impact** | High — Ken thinks data is being captured but it isn't; gap in the knowledge base goes unnoticed for weeks | | **Likelihood** | Medium — API tokens expire, services change schemas, disk fills up | | **Detect** | Heartbeat table: each source writes a row with `last_successful_ingest` timestamp. Watchdog checks every 6 hours; if any source is >2x its expected interval behind, push notification via Claude Code. Daily ingest count logged — alert on >50% drop from 7-day rolling average. | | **Fix** | 1. Check API credentials and refresh tokens. 2. Check disk space. 3. Check source API changelog for breaking changes. 4. Re-run failed batch from cursor stored in `ingest_state` table. | ### R7. Morning briefing becomes noise | | | |---|---| | **Phase** | 4–5 | | **Impact** | Medium — Ken stops reading briefings, defeating the purpose of the system | | **Likelihood** | Medium — information overload is the default failure mode of any aggregation system | | **Detect** | Track whether Ken interacts with briefing items (clicks through, asks follow-up questions). If engagement drops below 30% of items for 2 consecutive weeks, flag for tuning. | | **Fix** | 1. Reduce briefing to top-3 items (from top-5). 2. Add "not useful" feedback button that trains a simple relevance filter. 3. Rotate briefing format: some days show "things you haven't looked at in a while" instead of "most recent." | ### R8. Over-engineering the entity graph too early | | | |---|---| | **Phase** | 3 | | **Impact** | Medium — weeks spent building graph infrastructure that doesn't improve search quality enough to justify the complexity | | **Likelihood** | Medium — graph-based retrieval is powerful but has a cold-start problem | | **Detect** | A/B comparison: run 20 representative queries with and without entity-boosted retrieval. If precision improvement is <10%, the graph isn't earning its keep yet. | | **Fix** | Pause entity extraction. Keep the schema in place but don't invest in graph traversal features. Revisit when corpus exceeds 50K chunks — the graph's value scales with density. This is why Phase 3 is explicitly gated on Phase 2 being "good enough." | --- ## 10. Day-1 user-facing features "Day 1" means the end of Phase 2 — the first point where the system is genuinely useful. ### 10.1 `/memex/search` endpoint A local HTTP endpoint (or Claude Code MCP tool) that Ken can hit from any context. **Interface:** ``` POST /memex/search { "query": "what did I decide about the kitchen renovation timeline?", "filters": { "source_type": ["note", "email"], // optional "date_range": ["2026-01-01", null], // optional "tags": ["home"] // optional }, "limit": 10 } ``` **Response:** ``` { "results": [ { "chunk_id": "abc123", "text": "Decided to push kitchen reno to September — contractor availability...", "source": { "type": "note", "title": "Home projects planning", "date": "2026-03-12" }, "relevance_score": 0.89, "tags": ["home", "renovation"] } ], "query_time_ms": 142 } ``` The search combines vector similarity (semantic) with keyword matching (BM25-style) in a hybrid score. Filters are applied pre-search to narrow the candidate set. ### 10.2 Morning briefing with Memex pointers The existing D4 morning briefing gains a new section: **"From your Memex."** This surfaces 2–3 items chosen by recency + relevance to today's context: ``` ## From your Memex - **Follow-up due**: You told Dr. Martinez you'd schedule a follow-up by end of May. (Source: email, 2026-04-28) - **Decision revisit**: 3 weeks ago you noted "revisit the CI pipeline choice after the next deploy." Last deploy was yesterday. - **Connection**: The article you saved about multi-agent orchestration is by the same author as the podcast episode you bookmarked last week. ``` Selection logic (Phase 2, simple version): 1. Find chunks with temporal triggers ("by end of May", "next week", "in 30 days") where the trigger date is approaching 2. Find chunks tagged with topics that appeared in yesterday's activity 3. Find chunks from sources Ken hasn't revisited in >30 days that have high connectivity (many shared tags with recent items) Pick the top 3 by a weighted score. This is deliberately simple — no entity graph needed yet. ### 10.3 MCP tool surface Claude Code (and any Claude-powered interface Ken uses) can access Memex through MCP tools: | Tool | Purpose | Example invocation | |---|---|---| | `memex_search` | Semantic search over all ingested content | "What do I know about distributed consensus?" | | `memex_recent` | Get the N most recently ingested items, optionally filtered by source | "What did I save this week?" | | `memex_context` | Given a topic, return a structured summary of everything Memex knows about it | "Give me full context on Project Lighthouse" | | `memex_ingest` | Manually push a piece of text into Memex with metadata | "Remember that I prefer the Anthropic SDK over raw API calls" | Each tool returns structured JSON that Claude can reason over. The `memex_context` tool is the most powerful — it assembles chunks from multiple sources into a coherent narrative, which is exactly what "second brain" means in practice. ### 10.4 What Ken actually sees On Day 1, the experience is: 1. **Morning**: The briefing email/notification includes a "From your Memex" section with 2–3 pointers. Ken glances at it in 30 seconds. 2. **During work**: Ken is in Claude Code working on a project. He asks "what was the approach I used last time for database migrations?" Claude calls `memex_search`, finds the relevant note from 3 months ago, and weaves it into its response. Ken didn't have to remember where he wrote it down. 3. **Capture**: Ken reads an interesting article. He highlights a passage and uses a quick-capture flow (bookmarklet, share sheet, or just telling Claude "save this to Memex") to ingest it. Takes 5 seconds. 4. **End of day**: Ken asks Claude "what did I work on today?" Claude calls `memex_recent` with today's date filter and gives a summary. Ken adds a one-line reflection if he wants. This becomes tomorrow's briefing context. The key UX principle: **Memex should feel like having a perfect memory, not like using a database.** Ken never writes queries, constructs filters, or thinks about storage. He just asks questions in natural language and gets answers that reference things he's seen, read, written, or decided. --- ## 11. Maintenance + monitoring ### 11.1 Quarterly health check script A single script (`memex_health_check.py`) that Ken runs quarterly (or that a cron job runs and reports on). It checks: ``` QUARTERLY HEALTH CHECK — 2026-Q3 ================================= Storage Chunks total: 34,291 Chunks added (quarter): 8,104 Growth rate: +31% (healthy: under 50%) DB file size: 412 MB Embedding dimensions: 1536 (model: text-embedding-3-small) Source coverage Notes: 12,400 chunks (last ingest: 2h ago) ✅ Email: 9,200 chunks (last ingest: 6h ago) ✅ Bookmarks: 5,100 chunks (last ingest: 1d ago) ✅ Calendar: 4,800 chunks (last ingest: 12h ago) ✅ Voice: 2,791 chunks (last ingest: 3d ago) ⚠️ (> 2d) Search quality Canary recall: 18/20 (90%) ✅ p50 latency: 89ms ✅ p95 latency: 340ms ✅ Entity graph (if Phase 3+) Entities total: 1,204 Dormant (180d): 89 (7%) ✅ (under 20%) Orphan entities: 3 ✅ (under 10) Avg edges/entity: 4.2 Duplicates Suspected dupes: 12 clusters ✅ (under 50) Action items ⚠️ Voice source hasn't ingested in 3 days — check API token ``` ### 11.2 Entity growth management As the entity graph grows, it needs active curation to stay useful. **Auto-merge:** When two entities have the same name (case-insensitive) and appear in overlapping contexts (>3 shared chunks), flag them for merge. Run weekly. Ken reviews the merge list (~2 min) and approves or rejects. **Entity caps:** Soft cap at 5,000 entities. Beyond that, auto-archive entities with zero edges in the last 365 days. This isn't deletion — archived entities can be restored if a new chunk references them. **Type distribution monitoring:** If >60% of entities are of one type (e.g., all "person" entities because email parsing is prolific), flag it. The graph is most useful when entity types are balanced — a graph of only people with no topics or projects has limited traversal value. ### 11.3 Edge pruning Edges decay in value over time. An edge created because Ken emailed someone once in 2024 is much less meaningful than an edge from a project he's actively working on. **Pruning rules:** - Edges with `weight = 1` (single co-occurrence) and `last_seen > 365 days ago`: archive - Edges with `weight > 5` are never auto-pruned (strong signal) - Edges between archived entities: archive the edge too **Pruning schedule:** Monthly, automated. Report the count of pruned edges in the quarterly health check. ### 11.4 Watchdog patterns Continuous monitors that run alongside the system: | Watchdog | Check interval | Condition | Action | |---|---|---|---| | Ingest heartbeat | 6 hours | Any source > 2x expected interval | Push notification | | Disk space | Daily | DB > 80% of allocated quota | Warning in next briefing | | Search quality | Nightly | Canary recall < 80% | Push notification + pause non-critical ingestion | | Embedding model | Weekly | Check if current model is deprecated | Warning in next briefing | | Duplicate spike | Daily | > 20 new duplicate clusters in 24h | Pause the offending source, investigate | All watchdog state is stored in a `watchdog_log` table for trend analysis. If the same watchdog fires 3 times in 30 days, escalate it from a notification to a briefing action item. --- ## 12. Migration from D4 v1 to D4 v2 "v1" = Phases 1–2 (chunks + tags + vector search, no entity graph). "v2" = Phases 3+ (entity graph populated, relationship-aware retrieval). ### 12.1 What changes | Capability | v1 | v2 | |---|---|---| | Search | Vector similarity + keyword hybrid | Same, plus entity-boosted reranking | | Briefing source selection | Recency + tag overlap | Recency + entity proximity + relationship strength | | "Tell me about X" | Returns matching chunks | Returns entity card + connected chunks + related entities | | Cross-source linking | Implicit via tag overlap | Explicit via shared entity edges | | Temporal reasoning | Basic date filtering | Event chains: "what happened before/after X" via temporal edges | ### 12.2 Migration procedure The migration is **additive** — nothing in v1 is deleted or restructured. **Step 1: Schema addition** (~5 minutes) ```sql CREATE TABLE entities (...); CREATE TABLE edges (...); CREATE TABLE chunk_entities (...); -- junction table ``` No existing tables are modified. **Step 2: Backfill entity extraction** (~2–4 hours for 30K chunks) Run the entity extraction pipeline over all existing chunks. This is a batch job: 1. For each chunk, call the extraction model (Claude Haiku for cost efficiency) 2. Insert entities and edges 3. Write to `chunk_entities` junction table This can run in the background while v1 continues operating normally. **Step 3: Enable entity-aware search** (feature flag flip) Add `use_entity_graph: true` to the search config. The search pipeline gains an additional reranking step that boosts results connected to the query's detected entities. **Step 4: Validate** Run the canary query suite. Compare v1 and v2 results side by side for 20 queries. If v2 is worse on any query, investigate before committing. **Step 5: Enable entity-aware briefings** Update the briefing generation prompt to include entity context. The briefing now says "Project Lighthouse (connected to: Ken, Acme Corp, Q3 deadline)" instead of just surfacing a chunk about the project. ### 12.3 What becomes possible Things that were hard or impossible in v1 that v2 enables: - **"How do I know this person?"** — Traverse the entity graph from person → shared projects, shared email threads, shared events. v1 could only do keyword search for their name. - **"What's everything related to this project?"** — Follow all edges from the project entity. v1 would miss chunks that discuss the project without using its exact name. - **"What have I been neglecting?"** — Find entities with high historical edge weight but no recent activity. v1 had no concept of "importance" beyond tag frequency. - **"Connect the dots"** — Given two seemingly unrelated topics, find the shortest path between them in the entity graph. Ken reads an article about supply chain optimization and asks "how does this relate to anything I'm working on?" The graph finds: article → topic:optimization → project:home-renovation-scheduling → person:contractor. v1 couldn't make this leap. ### 12.4 Rollback If v2 degrades the experience, rollback is trivial: set `use_entity_graph: false`. The entity tables remain in the database but are ignored. No data loss, no schema changes needed. --- ## 13. Open questions for Ken 1. **Email scope**: Full Gmail archive, or only starred/labeled threads? Full archive gives better coverage but adds ~50K chunks and surfaces old noise. Recommendation: start with last 2 years + starred, expand later if search feels incomplete. 2. **Voice memo handling**: Transcribe-and-discard audio, or store audio files too? Storing audio doubles storage cost but lets you re-transcribe with better models later. Leaning toward: store audio, but only transcribe on demand for older recordings. 3. **Health data inclusion**: Should HealthEx data flow into Memex, or stay siloed? There's genuine value (briefing reminds you of an upcoming appointment or medication refill) but also genuine sensitivity risk. Could be opt-in per data type — medications yes, lab results no. 4. **Capture friction threshold**: How low does capture friction need to be for you to actually use it? Options range from "tell Claude to remember it" (lowest friction, lowest metadata) to "dedicated quick-capture UI with tagging" (more friction, richer metadata). The answer shapes Phase 2 significantly. 5. **Briefing channel**: Where should the morning briefing with Memex pointers land — email, push notification, Claude Code on first open, or a dedicated Memex dashboard? Current D4 briefing is already established; the question is whether Memex items belong in the same stream or a parallel one. 6. **Shared vs. solo**: Is this system exclusively for Ken, or might family members eventually need their own view? If shared, the entity graph needs access controls from the start. If solo, we can keep the model much simpler. Recommendation: build solo, but use the `owner_id` field from day one so the door stays open. 7. **Budget ceiling for embeddings**: At current pricing (~$0.02/1M tokens for `text-embedding-3-small`), the full backfill is cheap (<$5). But if Ken wants a premium model or local embeddings for privacy, the cost/performance tradeoff changes. What's the monthly budget Ken is comfortable with for API calls specifically for Memex?