# Goal 2 — PM/AWL fully operational: implementation spec > **STATUS UPDATE (2026-05-31 audit):** Authored 2026-05-17. The "current state" (§2) describes an AWL stall ~15 days old as of authoring — now ~28 days. The CP `/pm/budgets` and `/pm` routes are operational (verified HTTP 200 in tonight's CP restart). Token endpoint calibration drift (§8) status unknown — needs mini verification. **Ken's stated G2 percentage as of 2026-05-30: ~85%.** See `g1-g2-g3-path-to-100-reconciliation-2026-05-26.md` for close-out roadmap. The open questions (§12) may have been partially resolved — mini should verify which OQs are still open. **Status:** DRAFT — pending Ken ratification **Author:** Opus, MBP session, 2026-05-17 **Scope:** Project management automation + Always-Working Loop v2 promotion + three-axis budget enforcement **Canonical location:** this document supersedes scattered references across `always-working-loop-canon.md`, `awl-budgeting-three-priorities-2026-05-16.md`, and `command-post-sprint.md` --- ## 1. Definition of "fully operational" Goal 2 is **done** when all five of the following are true simultaneously for 48 continuous hours: | # | Criterion | Verification | |---|-----------|-------------| | F1 | AWL v2 selector fires autonomously, picks the highest-priority unblocked task, executes it, and continues to the next — zero silent gaps | `session_log` query: no rows with `outcome IS NULL` or inter-session gaps > 35 min without a logged reason | | F2 | Three-axis budget (tokens, Mini compute, Ken time) is enforced automatically — AWL hard-blocks when any axis saturates | `budget_signals.py` returns `blocked=True` for the saturated axis; `session_log` records `outcome='budget_gate'` with axis name | | F3 | `/pm/budgets` renders a live three-axis glance panel; `/pm/sprint` shows current sprint tasks with status, owner, and next-action | HTTP 200 on both routes; glance panel values match `token_calibration`, `session_log` aggregate, and pending NEEDS-KEN count within ±5% | | F4 | CP token endpoint (`/api/tokens/anthropic-summary`) reads within ±2% of Anthropic native billing | Spot-check: `weekly_pct` value vs. Anthropic console usage page on the same Tuesday-to-Tuesday window | | F5 | Quarterly spin-row archive runs unattended on first Saturday of quarter, posts a one-line dispatch | `session_log_archive_pre_spin_fix` table exists; dispatch file appears in `Claude/dispatch/` with rows-moved count | Ken declares Goal 2 **shipped** when all five criteria pass the 48h window and he has visually confirmed `/pm/budgets` on his phone or MBP browser. --- ## 2. Current state ### 2.1 What exists today | Component | Location | Status | |-----------|----------|--------| | AWL v1 loop | `~/Library/LaunchAgents/com.kenlovell.awl-continue.plist` (WatchPaths on `triggers/always-work-continue`) | **LIVE** — fires sessions, but stalled ~15 days on `class_g_human_review` gate | | AWL v2 selector | `~/Scripts/always-working-loop/select_next_task.py` (v2 branch) | **WIRED** — passes dry-fire; blocked on live cron flip | | `AWL_SELECTOR` env var | Plist `EnvironmentVariables` dict | **STAGED** — value `v2` not yet exported; plist currently runs v1 path | | `budget_signals.py` | `~/Scripts/always-working-loop/budget_signals.py` | **SHIPPED** — scores tasks across three axes; column-name bug fixed | | `queue_selector.py` v2 | `~/Scripts/always-working-loop/queue_selector.py` | **SHIPPED** — 9-class taxonomy; calls `budget_signals.py` | | AWL safety net | `~/Library/LaunchAgents/com.kenlovell.awl-safety-net.plist` | **LIVE** — circuit breaker via `triggers/always-work-stop` | | AWL dispatch watcher | `~/Library/LaunchAgents/com.kenlovell.awl-new-dispatch.plist` | **LIVE** — re-wakes loop on `mbp-to-mini_*` / `ken-to-mini_*` | | `AWL_OUTCOME_GUARD=1` | Plist env var (April 26) | **LIVE** — downgrades spurious "shipped" when duration < 5s and no `token_usage` row | | CP `/pm/budgets` | Route in `~/Projects/command-post/routes/` | **PATCHED** — nav audit T64 confirmed HTTP 200 (2026-05-17 06:07) | | CP `/api/tokens/anthropic-summary` | Route in `~/Projects/command-post/routes/` | **LIVE but MISCALIBRATED** — reads 13% weekly, actual Anthropic native is 6% | | `token_calibration` table | `~/Projects/command-post/data/command_post.db` | **LIVE** — `calibrated_at: 2026-04-24T19:52:48`, `weekly_pct: 13`, `color_tier: green` | | `session_log` table | same DB | **LIVE** — 5,357+ rows including ~5,189 pre-fix spin rows | | `project_tasks` table | same DB | **LIVE** — 138+ rows, AWL queue source | | `pm_rundown.py` | `~/Projects/command-post/scripts/pm_rundown.py` | **LIVE** — daily morning PM dashboard | | `sync_project_tasks_from_dispatch.py` | `~/Projects/command-post/scripts/sync_project_tasks_from_dispatch.py` | **LIVE** — daily 5:15 AM via `com.kenlovell.sync-project-tasks-from-dispatch` launchd | | Spin-row cleanup (#120) | Ratified 2026-05-16 | **NOT EXECUTED** — dispatch marked "shipped" but no confirmation of actual SQL execution | | AWL v1.3 observability (#118) | Ratified 2026-05-16 | **NOT BUILT** — P90, awaiting Ken batch review clearance | | AWL retry/backoff (#119) | Ratified 2026-05-16 | **NOT BUILT** — P80, awaiting Ken batch review clearance | ### 2.2 What's broken 1. **AWL execution stall (~15 days since ~2026-05-02)** — all open items gated on `class_g_human_review`. No tasks shipping. Bottleneck is Ken decision bandwidth, not engineering. 2. **"Primary executor not found"** — T6 cowork noted this when attempting v2 live flip. Root cause: plist doesn't export `AWL_SELECTOR=v2` and/or executor script path is wrong. 3. **Smoke test SystemExit 1** — `cc_spawn:smoke_test_unknown_chunk_2026-04-28-1830` exited 1 on April 28, no RCA documented. Open 19 days. 4. **Token endpoint calibration drift** — CP reads 13%, Anthropic native reads 6%. The token Monte Carlo runs at midnight and 5 AM are ingesting or computing incorrectly. 5. **Classifier returning `unknown`** — auto-recovery classifier produced `unknown` on at least one AWL session; may be L19 silent ImportError in a try/except block. --- ## 3. Token budget layer ### 3.1 Architecture ``` ┌─────────────────────────────────────────────────────────────┐ │ ANTHROPIC API │ │ (source of truth: billing page, usage API) │ └──────────────────────┬──────────────────────────────────────┘ │ polled by ┌──────────────────────▼──────────────────────────────────────┐ │ token_calibration_job.py (launchd: midnight + 5 AM) │ │ ─ reads Anthropic usage API or scrapes billing summary │ │ ─ writes to token_calibration table │ │ ─ runs Monte Carlo projection for end-of-week estimate │ └──────────────────────┬──────────────────────────────────────┘ │ writes ┌──────────────────────▼──────────────────────────────────────┐ │ command_post.db :: token_calibration │ │ columns: calibrated_at, cap_reference, weekly_pct, │ │ projection_pct, color_tier, extra_usage_pct, │ │ extra_usage_spent_usd, week_start, week_end │ └──────────────────────┬──────────────────────────────────────┘ │ read by ┌───────────┴───────────┐ ▼ ▼ /api/tokens/ budget_signals.py anthropic-summary (AWL gate input) (CP dashboard) ``` ### 3.2 Budget window - **Cycle:** Tuesday 13:00 ET → next Tuesday 12:59 ET (Anthropic billing cadence) - **Only Anthropic-billed tokens count.** Mini's Ollama/local tokens (Mistral Small, Qwen, BGE-M3) are free and unlimited. - **Current week (2026-05-12 → 2026-05-19):** `weekly_pct: 13` per CP (actual: ~6% native + Sunday T65 burn wave adding ~10-15%) ### 3.3 Auto-enforcement `budget_signals.py` returns a budget score for each candidate task. The enforcement tiers: | `weekly_pct` | Color tier | AWL behavior | |-------------|-----------|--------------| | 0–50% | `green` | Normal execution; no restrictions | | 51–70% | `yellow` | Prefer narrow-context tasks (fewer state-file loads); defer heavy rebuilds | | 71–85% | `orange` | Only tasks with `cost_anthropic_dollars < $1.00` or `actor: mini-local` | | 86–100% | `red` | **Hard block** — AWL logs `outcome='budget_gate'`, does NOT touch continuation sentinel, writes dispatch to Ken | | >100% (overshoot) | `red` | Same as 86–100%; additionally surfaces on `/morning` as critical alert | **Refill curve:** Linear within the Tuesday-to-Tuesday window. At any point, `remaining_budget = (1 - weekly_pct) * weekly_cap`. No carry-forward across weeks. ### 3.4 Recalibration plan (CP token endpoint) **Problem:** `/api/tokens/anthropic-summary` reports `weekly_pct: 13` but Anthropic native billing shows ~6%. **Root cause investigation steps:** 1. **Read the token MC script** (`token_calibration_job.py` or equivalent scheduled task) — identify what data source it reads. Is it scraping the Anthropic dashboard, calling an API, or computing from local `token_usage` table rows? 2. **Compare `token_usage` table sum vs. Anthropic native** — `SELECT SUM(total_cost_usd) FROM token_usage WHERE ts >= '2026-05-12'` vs. Anthropic console for the same window. If they match, the bug is in the percentage computation (wrong `cap_reference`). If they don't match, the bug is in token recording (over-counting or double-counting sessions). 3. **Check `cap_reference` value** — currently `100`. This is supposed to represent the weekly Anthropic budget in dollars. If it's wrong, every percentage is wrong. 4. **Check for double-counting** — do both MBP and Mini record to `token_usage`? If so, MBP sessions that don't go through the Anthropic billing (e.g., Cowork sessions billed differently) inflate the number. 5. **Fix:** update the MC script to use the correct data source. If Anthropic has a usage API, use it directly rather than computing from local records. **Acceptance:** After fix, `/api/tokens/anthropic-summary` `weekly_pct` matches Anthropic native within ±2% when both are checked within the same hour. **DB task:** New — create as `#TBD` in `project_tasks` with `title: 'CP token endpoint recalibration'`, `priority: high`, `project_id: command-post`, `estimated_hours: 2`. --- ## 4. Mini compute layer ### 4.1 What "Mini compute" means The Mac Mini M4 Pro runs multiple concurrent workloads: | Workload | Resource | Scheduling | |----------|----------|-----------| | AWL cc_spawn sessions | CPU + memory + API | One-at-a-time via continuation sentinel | | Memex extraction watchdog | CPU + disk I/O | 5-min debounced WatchPaths on `dispatch/` + `state/` | | Memex HTTP server | Port 8766 | Always-on | | Command Post Flask | Port 5001 (Caddy → 80) | Always-on | | Ollama inference | GPU (Metal) | On-demand; ~14 GB for Mistral Small 24B | | Trigger-watcher | CPU (light) | WatchPaths on `triggers/`, ThrottleInterval=15 | | Dispatch sync | CPU (light) | Daily 5:15 AM | | Token MC | CPU (light) | Midnight + 5 AM | ### 4.2 Capacity signal `budget_signals.py` computes Mini capacity from: - **Heartbeat recency:** Mini's `mini_progress_heartbeat.py` writes every 30 min. If last heartbeat > 60 min old, Mini is likely stuck or down. - **Active cc_spawn count:** currently capped at 1 (sequential model). If future work raises this, the cap becomes the denominator. - **Outcome distribution (last hour):** ratio of `shipped` vs. `spin`/`no_task_available`/`failed` outcomes in `session_log`. High failure ratio = throttle new spawns. ### 4.3 Ollama routing Tasks with `actor: mini-local` bypass the Anthropic budget entirely: - **Eligible workloads:** Memex ingestion, podcast summaries, Readwise tagging, bio OCR cleanup, document classification, health data trend analysis (under privacy boundary) - **Model:** Mistral Small 3.2 24B Q4_K_M (14 GB, ~24 tok/s on M4 Pro Metal) - **Embeddings:** BGE-M3 via Ollama - **Savings:** ~$125/mo vs. Claude API for these batch workloads - **Budget integration:** Ollama tasks contribute zero to `weekly_pct`; they contribute to Mini compute capacity only ### 4.4 Per-project caps Not yet implemented. Future enhancement: `project_tasks` could carry a `max_weekly_anthropic_dollars` column per `project_id`. The selector would sum actual spend per project from `token_usage` and skip projects at their cap. **Banked for post-Goal-2 — not in scope for "fully operational."** --- ## 5. Ken time layer ### 5.1 The scarcest resource Per Ken's explicit ranking (2026-05-16): **Ken time > Token use > Mini compute cycles.** Ken's hours are not renewable. Tokens can be purchased. Compute capacity recovers when tasks complete. But every NEEDS-KEN dispatch, every spec interview prompt, every decision that requires Ken's attention consumes the single non-renewable budget in the system. ### 5.2 Signal sources | Signal | Source | Cadence | |--------|--------|---------| | Pending NEEDS-KEN dispatches | `ls Claude/dispatch/ \| grep needs-ken` | Real-time (file count) | | Outstanding Ken decisions in `project_tasks` | `SELECT COUNT(*) FROM project_tasks WHERE awaiting LIKE '%ken%' AND status != 'done'` | Queried by selector at task-selection time | | Ken-flagged priorities | `project_tasks.priority` where `assignee = 'ken'` | Static until Ken updates | | KEN-INPUT-QUEUE depth | `state/projects/KEN-INPUT-QUEUE.md` item count | Updated by evening prep (RULE 18) | ### 5.3 Enforcement The Ken-time budget has a **soft target**: pending NEEDS-KEN count should not exceed 5 at any time during work hours. | Pending NEEDS-KEN | AWL behavior | |-------------------|--------------| | 0–3 | Normal: tasks may generate NEEDS-KEN dispatches freely | | 4–5 | Caution: selector prefers tasks that do NOT produce Ken-input requests | | 6+ | **Soft block on Ken-generating tasks:** AWL only picks tasks with `actor: mini-local` or `actor: mini-Claude` that do NOT produce a NEEDS-KEN dispatch. Existing NEEDS-KEN queue surfaces as amber alert on `/morning` | ### 5.4 Calendar-aware gating When Ken is unavailable (travel, family events, weekends), the system should NOT accumulate NEEDS-KEN dispatches that will pile up: - **Weekends (Sat/Sun):** NEEDS-KEN generation reduced to max 1 per day (batched into Sunday evening or Monday morning prep) - **Travel days:** detected via `state/calendar/locations.json` + work calendar ICS. During travel, Ken-time budget tightens to max 2 pending NEEDS-KEN total. - **Family-time-aware:** when tomorrow-location resolves to a family member's city (J11b kid-priority infrastructure), the system assumes Ken's attention is family-focused and further restricts NEEDS-KEN generation to max 1 pending. ### 5.5 Implementation path Calendar-awareness requires the J11b kid-priority calendar wiring (#217, Sprint 6) to ship first. Until then, the Ken-time layer operates on the simpler signal: pending NEEDS-KEN count from the dispatch folder + `project_tasks` query. --- ## 6. AWL v2 rollout ### 6.1 Pre-flip checklist Before setting `AWL_SELECTOR=v2` in the live plist: | # | Check | Method | Status | |---|-------|--------|--------| | 1 | RCA "primary executor not found" (Sprint Task 1.1) | Audit plist for missing PATH/HOME/env exports | **NOT DONE** | | 2 | Executor script path resolves from launchd context | `launchctl print` on the plist, verify all paths | **NOT DONE** | | 3 | v2 dry-fire produces correct task selection | Run selector with `--dry-run` flag; compare output to expected | **PASSED** (T6 cowork) | | 4 | `AWL_OUTCOME_GUARD=1` is set | Plist inspection | **DONE** (April 26) | | 5 | Smoke test SystemExit 1 resolved (Sprint Task 1.3) | Re-run smoke test; exit code 0 | **NOT DONE** | | 6 | Rollback handle documented | `unset AWL_SELECTOR` in plist reverts to v1 | **DONE** | ### 6.2 Flip procedure ```bash # 1. Edit plist to add AWL_SELECTOR=v2 /usr/libexec/PlistBuddy -c "Add :EnvironmentVariables:AWL_SELECTOR string v2" \ ~/Library/LaunchAgents/com.kenlovell.awl-continue.plist # 2. Unload and reload the agent launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/com.kenlovell.awl-continue.plist launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.kenlovell.awl-continue.plist # 3. Touch continuation sentinel to fire first v2 session touch ~/Library/CloudStorage/OneDrive-Personal/Claude/triggers/always-work-continue # 4. Watch session_log for first v2 entry sqlite3 ~/Projects/command-post/data/command_post.db \ "SELECT * FROM session_log ORDER BY started_at DESC LIMIT 3;" ``` ### 6.3 A/B verification The T64 cowork on Sunday 2026-05-17 06:07 reportedly flipped `AWL_SELECTOR=v2` in the plist for an A/B trial. Verification requires: 1. **Confirm the flip stuck:** read the plist after the launchd reload. If `AWL_SELECTOR=v2` is present in `EnvironmentVariables`, v2 is active. 2. **First v2 session result:** query `session_log` for the first row after the flip time. Check `exit_code = 0` and `outcome != 'spin'`. 3. **Compare v1 vs. v2 task selection:** run the selector in both modes against the current `project_tasks` state. v2 should pick a higher-priority unblocked task than v1 would have. 4. **48h stability criterion (Sprint Task 3.2):** after all Week 2 features ship, run v2 for 48 continuous hours. Zero silent gaps. Zero unhandled exceptions. Retry/backoff fires at least once (or is confirmed unnecessary). Summary report to Ken. ### 6.4 Rollback If any v2 session exits non-zero: ```bash /usr/libexec/PlistBuddy -c "Delete :EnvironmentVariables:AWL_SELECTOR" \ ~/Library/LaunchAgents/com.kenlovell.awl-continue.plist launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/com.kenlovell.awl-continue.plist launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.kenlovell.awl-continue.plist ``` This immediately reverts to v1 behavior on the next session. No code changes required. --- ## 7. Budget breach UX ### 7.1 Surfaces When any of the three budget axes breaches its threshold, the system surfaces alerts on three channels: | Channel | When | What | |---------|------|------| | `/morning/v5b` glance panel | Every morning | Three-axis status: `Tokens: 73% remaining · Mini: 14/30 (47%) · Pending Ken: 2` | | `/pm/budgets` | On-demand | Full budget detail with color tiers, refill projection, and per-project breakdown | | AWL dispatch | On breach | Auto-generated dispatch: `mini-to-ken_alert_budget-breach--.md` | ### 7.2 Glance panel spec (for `/morning/v5b` and `/pm/budgets`) ``` ┌─────────────────────────────────────────────────────────────┐ │ TODAY'S BUDGET │ │ │ │ Tokens ████████░░ 73% remaining [green] │ │ Mini ██████░░░░ 14/30 in-flight [yellow] │ │ Pending Ken ██░░░░░░ 2 NEEDS-KEN [green] │ │ │ │ Budget week: May 12 – May 19 (day 6 of 7) │ │ Projection: 89% of cap by Tuesday 13:00 │ └─────────────────────────────────────────────────────────────┘ ``` Color tiers per axis: | Axis | Green | Yellow | Orange | Red | |------|-------|--------|--------|-----| | Tokens | 0–50% used | 51–70% | 71–85% | 86%+ | | Mini | 0–50% capacity | 51–75% | 76–90% | 91%+ or heartbeat stale | | Pending Ken | 0–3 | 4–5 | — | 6+ | ### 7.3 Breach dispatch format ```yaml --- from: mini-awl to: ken type: alert date: 2026-05-20T14:30:00-04:00 status: new priority: urgent --- # Budget breach: TOKEN axis at 87% **Weekly window:** May 19 – May 26 (day 2 of 7) **Current spend:** $87.12 of $100.00 cap **Projection:** 134% by Tuesday 13:00 at current burn rate **Cause:** T65 Sunday burn wave (7 Sonnet extractions) consumed ~$42 in one batch. **AWL action taken:** Hard-blocked. No new Anthropic-billed sessions until weekly_pct drops below 85% or budget window resets. **Recommendation:** (1) Wait for Tuesday reset, or (2) Ken manually override by touching `triggers/budget-override`. ``` ### 7.4 Override mechanism A file sentinel `triggers/budget-override` allows Ken to manually override a budget gate for the remainder of the current budget window. AWL checks for this file at session start; if present, budget gate is bypassed. The file is automatically deleted at the next Tuesday 13:00 reset. --- ## 8. CP token endpoint recalibration plan This is the highest-priority Goal 2 maintenance task. The system cannot be "fully operational" while its primary budget display is off by 2x. ### 8.1 Investigation protocol ```bash # Step 1: What does the CP endpoint actually return? curl -s http://127.0.0.1:5001/api/tokens/anthropic-summary | python3 -m json.tool # Step 2: What's in the calibration table? sqlite3 ~/Projects/command-post/data/command_post.db \ "SELECT * FROM token_calibration ORDER BY calibrated_at DESC LIMIT 5;" # Step 3: What's the raw token_usage sum for this week? sqlite3 ~/Projects/command-post/data/command_post.db \ "SELECT SUM(total_cost_usd), COUNT(*) FROM token_usage WHERE ts >= '2026-05-12';" # Step 4: Compare to Anthropic native (manual — Ken or agent checks console) # Anthropic usage page for wklovell@gmail.com, May 12-19 window # Step 5: Find the MC script find ~/Projects/command-post ~/Scripts -name '*token*' -o -name '*calibrat*' -o -name '*monte*' 2>/dev/null ``` ### 8.2 Likely root causes (ranked by probability) 1. **Double-counting:** Both MBP Cowork sessions (billed differently via Anthropic's Cowork tier) and Mini Claude Code sessions write to `token_usage`, but only Mini sessions count against the Anthropic API budget. Fix: filter `token_usage` to only `source = 'mini-cc'` or equivalent. 2. **Wrong `cap_reference`:** If the weekly cap is $100 but the system thinks it's $50, all percentages are doubled. Fix: update `cap_reference` to match actual Anthropic plan. 3. **Stale calibration:** `calibrated_at: 2026-04-24` — the calibration is 23 days old. If the MC script hasn't run since then, the percentage reflects April data extrapolated forward. Fix: ensure MC launchd jobs are running. 4. **Extra usage counting:** The `extra_usage_pct: 74` / `extra_usage_spent_usd: $551.62` fields suggest the system is tracking total historical spend, not just weekly. If these bleed into the weekly percentage calculation, the number is inflated. ### 8.3 Fix validation After any fix, run this validation: ```bash # Trigger a fresh calibration python3 ~/Projects/command-post/scripts/token_calibration_job.py --force # Read the result curl -s http://127.0.0.1:5001/api/tokens/anthropic-summary | python3 -m json.tool # Compare weekly_pct to Anthropic console # Acceptance: within ±2% ``` **DB task:** Create `project_tasks` row: `title: 'Token endpoint recalibration'`, `project_id: command-post`, `priority: high`, `estimated_hours: 2`, `assignee: mini-Claude`. --- ## 9. #118 / #119 AWL v1.3 wire-in ### 9.1 #118 — Observability (P90) **What:** Capture per-session telemetry: input tokens, output tokens, total cost USD, wall-clock duration, exit code, and task ID. **Schema addition to `session_log`:** ```sql ALTER TABLE session_log ADD COLUMN input_tokens INTEGER; ALTER TABLE session_log ADD COLUMN output_tokens INTEGER; ALTER TABLE session_log ADD COLUMN total_cost_usd REAL; ALTER TABLE session_log ADD COLUMN duration_seconds INTEGER; ALTER TABLE session_log ADD COLUMN task_id INTEGER REFERENCES project_tasks(id); ``` **Wire-in point:** `session_wrapper.sh` captures these values from `cc_spawn` output (Claude Code prints token usage at session end). The wrapper parses the output, writes the row to `session_log`, then touches the continuation sentinel. **Data flow:** `session_log` → `pm_rundown.py` → `/morning/v5b` AWL Health card → `/pm/budgets` glance panel. **Acceptance:** - At least one live session populates all new columns - `pm_rundown.py` renders the telemetry in its daily output - `/pm/budgets` shows per-session cost history for the last 24h **Dependency:** Ken must clear the P90 human-review gate (Sprint Task 1.4). Already ratified 2026-05-16 — the gate is administrative, not technical. ### 9.2 #119 — Retry/backoff (P80) **What:** When `cc_spawn` fails with a transient error (HTTP 429 rate-limit, network timeout, resource contention), retry with exponential backoff. Do NOT retry permanent failures (bad task config, missing executor, budget gate). **Implementation in `session_wrapper.sh`:** ```bash MAX_RETRIES=3 RETRY_DELAY=30 # seconds, doubles each retry for attempt in $(seq 1 $MAX_RETRIES); do cc_spawn "$TASK_ARGS" 2>&1 EXIT_CODE=$? if [ $EXIT_CODE -eq 0 ]; then break # success fi # Check if transient (rate-limit, timeout) vs permanent (config error) if is_transient_error $EXIT_CODE; then sleep $((RETRY_DELAY * (2 ** (attempt - 1)))) else break # permanent failure, don't retry fi done ``` **Transient vs. permanent classification:** | Exit code / signal | Classification | Action | |--------------------|---------------|--------| | 429 (rate limit) | Transient | Retry with backoff | | SIGTERM / timeout | Transient | Retry once, then surface | | Connection refused | Transient | Retry with backoff | | Exit 1 (config error) | Permanent | Log and skip | | Exit 2 (missing file) | Permanent | Log and skip | | Budget gate | Permanent | Log `outcome='budget_gate'` and halt | **Acceptance:** - Unit test: simulate a 429 → retry → success sequence - Unit test: simulate a permanent error → no retry, immediate log - No regression in existing smoke tests - Retry events logged in `session_log` with `outcome='retry_attempt'` and attempt number **Dependency:** Ken must clear the P80 human-review gate (Sprint Task 1.4). Already ratified 2026-05-16. --- ## 10. #120 Quarterly archive — plist spec ### 10.1 Ratified cadence | Parameter | Value | |-----------|-------| | Frequency | First Saturday of each quarter | | Time | 04:00 ET | | Threshold | `COUNT(*) > 1000` where `outcome IN ('spin','no_task_available') AND date < date('now','-90 days')` | | Target table | `session_log_archive_pre_spin_fix` | | Dispatch | One-line: `mini-to-ken_status_spin-archive-.md` with rows-moved count | ### 10.2 SQL ```sql -- Create archive table (idempotent) CREATE TABLE IF NOT EXISTS session_log_archive_pre_spin_fix AS SELECT * FROM session_log WHERE 0; -- Archive qualifying rows INSERT INTO session_log_archive_pre_spin_fix SELECT * FROM session_log WHERE outcome IN ('spin', 'no_task_available') AND started_at < date('now', '-90 days'); -- Capture count for dispatch -- (script reads this before DELETE) -- Remove archived rows from live table DELETE FROM session_log WHERE outcome IN ('spin', 'no_task_available') AND started_at < date('now', '-90 days'); ``` ### 10.3 Plist ```xml Label com.kenlovell.awl-quarterly-archive ProgramArguments /usr/bin/python3 /Users/kenlovell/Scripts/always-working-loop/quarterly_spin_archive.py StartCalendarInterval Month1 Weekday6 Hour4 Minute0 Month4 Weekday6 Hour4 Minute0 Month7 Weekday6 Hour4 Minute0 Month10 Weekday6 Hour4 Minute0 EnvironmentVariables HOME /Users/kenlovell PATH /opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin StandardOutPath /Users/kenlovell/Library/Logs/awl-quarterly-archive/stdout.log StandardErrorPath /Users/kenlovell/Library/Logs/awl-quarterly-archive/stderr.log ``` **Note:** launchd's `StartCalendarInterval` with both `Month` and `Weekday` fires on every Saturday in that month, not just the first. The Python script must self-gate: `if datetime.now().day > 7: sys.exit(0)`. ### 10.4 Initial sweep The ratified initial sweep of 5,189 rows should run immediately (not wait for quarterly cadence). This is Sprint Task 3.1 — dry-run first, Ken approves SQL output, then live execution. ### 10.5 Plist installation ```bash # Create log directory mkdir -p ~/Library/Logs/awl-quarterly-archive # Copy plist cp com.kenlovell.awl-quarterly-archive.plist ~/Library/LaunchAgents/ # Load launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.kenlovell.awl-quarterly-archive.plist # Verify launchctl print gui/$(id -u)/com.kenlovell.awl-quarterly-archive ``` --- ## 11. Risk register | # | Risk | Likelihood | Impact | Mitigation | |---|------|-----------|--------|------------| | R1 | **Ken decision bottleneck recurs** — the 15-day stall repeats after this spec is ratified | **HIGH** | **CRITICAL** | Time-box: if Ken batch review (Task 1.4) not complete by 2026-05-21, proceed with P80+P90 only; kill P70 tasks. RULE 18 evening prep surfaces pending decisions daily. | | R2 | **"Primary executor not found" is deeper than plist edit** — launchd environment issues (FDA, credential chain, PATH) require system-level changes | **MEDIUM** | **HIGH** | Cap RCA at 2h. If unresolved, escalate to Ken with four candidate fixes: (a) hardcode `CLAUDE_BIN` path, (b) grant Full Disk Access, (c) fix PATH export, (d) fix HOME export. Ken picks one. | | R3 | **v2 live session exits non-zero** — rollback required, v2 promotion delayed | **MEDIUM** | **MEDIUM** | Rollback handle is pre-wired. Define rollback criteria before flip: any exit ≠ 0 on first 3 sessions → revert. | | R4 | **Token endpoint recalibration reveals deeper accounting problem** — MBP and Mini counting differently, or Anthropic API not exposing the data we need | **MEDIUM** | **HIGH** | Fall back to manual weekly spot-check until automated reconciliation ships. AWL gates on the more conservative of two readings (current behavior). | | R5 | **Classifier tuning degrades routing quality** — tasks that used to route correctly now get misclassified | **LOW** | **HIGH** | A/B compare on historical sessions before deploying. Keep v1 classifier as fallback. Gate on Ken sign-off. | | R6 | **Spin-row cleanup SQL deletes active rows** — WHERE clause too broad | **LOW** | **HIGH** | Dry-run mandatory. Transaction with explicit ROLLBACK point. Ken reviews row count before COMMIT. Archive table preserves all deleted rows for 90-day recovery. | | R7 | **Sunday T65 burn wave consumes significant budget** — seven Sonnet extractions in one batch could push the week near its cap | **MEDIUM** | **MEDIUM** | T61 watchdog caps at `session > 60` or `weekly > 70%`. If T65 pushes past 40%, AWL auto-throttles to narrow-context-only tasks for the remainder of the week. | | R8 | **Quarterly archive plist fires on wrong Saturday** — launchd `Month`+`Weekday` combination fires every Saturday in the month, not just the first | **HIGH** | **LOW** | Python script self-gates: `if day > 7: exit(0)`. Extra fires are harmless (exit immediately). | | R9 | **Calendar-aware Ken-time gating misidentifies travel days** — bad data in `locations.json` or stale ICS feed causes false travel detection | **LOW** | **MEDIUM** | Travel detection is additive (tightens budget), not subtractive. False positive = slightly over-cautious, not harmful. False negative = no change from current behavior. | --- ## 12. Open questions | # | Question | Owner | Blocking | Notes | |---|----------|-------|----------|-------| | OQ-1 | **Did Fix B (launchd credential investigation) run ~2026-04-28?** Which of the four fixes applied? | Ken | YES — blocks AWL v2 live flip (Sprint 1.1/1.2) | If Ken doesn't recall, the RCA re-runs from scratch (~0.5h). Not a hard block, just redundant work. | | OQ-2 | **Ken batch review of 5 class_g_human_review tasks** — approve/defer each of P70-P90 items | Ken | YES — blocks all of Sprint Week 2 | Already ratified 2026-05-16. Administrative gate only. Deadline: 2026-05-21 or scope reduces. | | OQ-3 | **Is the AWL classifier returning `unknown` due to L19 silent ImportError?** | Agent | No — investigation, not decision | Audit classifier source for bare `except:` or `try/except ImportError: pass` blocks. | | OQ-4 | **Sprint 6 sequencing: should Sabbath-aware + Constitutional value-aware classification ship before or after AWL v2 live flip?** | Ken | No — sequencing preference | Recommendation: AFTER. V2 should prove stable on the basic taxonomy first. | | OQ-5 | **Smoke test RCA** — `cc_spawn:smoke_test_unknown_chunk_2026-04-28-1830` SystemExit 1 | Agent | YES — must resolve before 48h stability run | Open 19 days. Sprint Task 1.3. | | OQ-6 | **Token endpoint: is `cap_reference: 100` correct?** What is the actual weekly Anthropic budget in dollars? | Ken | YES — blocks recalibration | If the cap is $200 and the system thinks it's $100, all percentages are 2x. | | OQ-7 | **Per-project token caps: desired?** Should different projects have different weekly Anthropic budgets? | Ken | No — post-Goal-2 enhancement | Banked for future. Current model: one global weekly cap. | | OQ-8 | **Budget override sentinel: approved?** Should `triggers/budget-override` bypass the budget gate until next Tuesday? | Ken | No — UX question | Proposed in §7.4. Alternative: Ken directly touches the continuation sentinel, which bypasses nothing but fires the next session (which would re-block). | | OQ-9 | **T64 AWL_SELECTOR=v2 flip status** — the Sunday 2026-05-17 06:07 session reportedly flipped v2 in the plist. Did it stick? Did a session fire? | Agent | YES — determines whether Sprint 1.2 is already done | Check: read plist, query session_log for post-flip entries. | | OQ-10 | **Evening stack mockup** (#130, #158) — awaiting Ken visual ratification since 2026-05-16 13:50 at `Claude/state/mockups/evening-stack-mockup-2026-05-16.html` | Ken | No — Goal 3, not Goal 2 | Included for completeness. Does not block PM/AWL. | --- ## Appendix A — DB task cross-reference | DB task # | Title | Goal 2 section | Sprint task | Status | |-----------|-------|----------------|-------------|--------| | #118 | AWL v1.3 observability | §9.1 | 2.1 | Ratified, not built | | #119 | AWL retry/backoff | §9.2 | 2.2 | Ratified, not built | | #120 | AWL session_log spin-row cleanup | §10 | 3.1 | Ratified + RECURRING, initial sweep not executed | | #107–#115 | Auto-correction sweep RULE-1 through RULE-8 | — | — | Ratified, skeleton + sweep batch, flag-only v1 | | #217 | J11b kid-priority calendar wiring | §5.5 | CP-W2-01 | Ratified, Sprint 6 | | #241 | Memex ingest_dispatches.py deploy | — | — | Ratified, Goal 1 | | #93, #94 | Memex tool-call wrappers | — | — | Approved, depends on #241 | | NEW | CP token endpoint recalibration | §8 | — | Proposed in this spec | | NEW | AWL quarterly archive plist | §10.3 | — | Proposed in this spec | | NEW | Budget override sentinel | §7.4 | — | Proposed, pending OQ-8 | ## Appendix B — File path index | Path | Purpose | |------|---------| | `~/Library/LaunchAgents/com.kenlovell.awl-continue.plist` | Main AWL loop | | `~/Library/LaunchAgents/com.kenlovell.awl-new-dispatch.plist` | Dispatch watcher | | `~/Library/LaunchAgents/com.kenlovell.awl-safety-net.plist` | Kill switch / circuit breaker | | `~/Scripts/always-working-loop/select_next_task.py` | v1/v2 task selector | | `~/Scripts/always-working-loop/session_wrapper.sh` | Session boundary manager | | `~/Scripts/always-working-loop/budget_signals.py` | Three-axis budget scoring | | `~/Scripts/always-working-loop/queue_selector.py` | 9-class task taxonomy + v2 logic | | `~/Projects/command-post/data/command_post.db` | Unified CP database | | `~/Projects/command-post/scripts/pm_rundown.py` | Daily PM dashboard | | `~/Projects/command-post/scripts/sync_project_tasks_from_dispatch.py` | Dispatch → DB sync | | `~/Projects/command-post/scripts/com.kenlovell.sync-project-tasks-from-dispatch.plist` | Daily 5:15 AM sync | | `Claude/state/setups/awl-budgeting-three-priorities-2026-05-16.md` | Three-priority budgeting model | | `Claude/state/canon/projects/always-working-loop-canon.md` | AWL canonical state | | `Claude/state/projects/command-post.md` | CP architecture + decisions | | `Claude/triggers/always-work-continue` | Continuation sentinel | | `Claude/triggers/always-work-stop` | Kill switch sentinel | | `Claude/dispatch/` | 2,614+ dispatch files | --- *End of Goal 2 implementation spec. Next action: Ken ratifies, then Sprint Week 1 begins 2026-05-19.*