--- project: yahoo-mail-download-concrete-instructions-2026-05-25 classification: step-by-step instructions for executing the Yahoo Mail download (the prior options doc was strategic; this is operational) status: READY TO EXECUTE — awaiting Ken kick-off (likely Sat 5/30 post-Mayo per prior sequencing) authored_by: mbp-cowork honors_principles: [P-2, P-7, P-10] related: - state/projects/yahoo-mail-ingest-options-2026-05-24.md (the options doc that landed on Option D revised) - help.yahoo.com/kb/mail/SLN28681.html (Yahoo's own IMAP-export guidance) --- # Yahoo Mail download — concrete step-by-step ## Where we landed (recap) After Ken supplied Yahoo's SLN28681 docs, the conclusion was: - Yahoo's "Download My Data" doesn't export mail (only settings) - The official path is IMAP via Yahoo's purpose-built archive server: `export.imap.mail.yahoo.com:993` - mbsync (isync) on mini is the right tool — runs as launchd, incremental forever - Phase 1 = initial baseline sync (multi-day, runs in background) - Phase 2 = launchd daemon for incremental delta sync going forward - Phase 3 = Apple Mail spot-check (sanity verification only) - Privacy P-10: mail bodies stay local on mini, Memex indexes metadata only initially, monthly gpg-encrypted .tgz backup to OneDrive ## Ken-side concrete instructions (do these first) ### Step 1 — Generate Yahoo app password (one-time, 2 min) Yahoo doesn't allow OAuth for IMAP. You must generate an app-specific password. 1. Open browser → https://login.yahoo.com/account/security 2. Sign in if not already 3. Scroll to "Other ways to sign in" or "App password" 4. Click **"Generate app password"** (or "Generate and manage app passwords") 5. Yahoo asks "What app is requesting access?" → type: `mini-mbsync-2026-05-30` 6. Yahoo generates a 16-character password (formatted like `abcd efgh ijkl mnop` with spaces) → **copy it to clipboard** 7. DON'T paste this anywhere except where Step 4 below directs **Save this password ONLY into Apple Keychain. NOT into chat. NOT into config files plaintext. NOT into a note.** ### Step 2 — Pre-flight check for large folders (5 min) Yahoo's documentation: *"Folders containing upwards of 100 thousand or more emails may have issues downloading."* 1. Open Yahoo Mail web at mail.yahoo.com 2. In the left sidebar, expand all folders 3. For each folder, hover over the count or look at the folder header — note any folder ≥100,000 messages 4. Likely candidates: **Inbox**, **Sent**, **Archive**, possibly **Trash** 5. For any folder ≥100K: - Create a sub-folder like `Inbox-pre-2020` via right-click → New Folder - Move old emails into the sub-folder (date-based bulk select) - Goal: get each folder below 100K 6. Verify: re-count each folder; all should be <100K before initial sync If your largest folder is well under 100K (say, 30K-50K), skip this — Yahoo's warning is for unusual cases. ### Step 3 — SSH to mini, open Keychain, save the app password (3 min) From MBP, SSH to mini: ```bash ssh kenlovell@192.168.1.171 ``` Then open Keychain Access on mini (you'll need to be at the mini physically or via screen sharing for this part — Keychain Access has no CLI for adding passwords with proper attributes): 1. Cmd+Space → "Keychain Access" → open 2. File → New Password Item 3. Fill in: - Keychain Item Name: `yahoo-imap-export` - Account Name: your Yahoo email address (e.g., `walter.lovell@yahoo.com`) - Password: paste the 16-char app password from Step 1 4. Save The password is now in Keychain. Mini's mbsync config will reference it via the macOS `security` CLI without needing to read plaintext. ### Step 4 — Tell mini you're ready (1 min, to me via Cowork) After Steps 1-3 are done, send me a chat message: > "Yahoo app password generated and stored in mini Keychain as `yahoo-imap-export`. Pre-flight folder check done — largest folder is N messages. Ready for mini to start mbsync." I'll dispatch mini to begin Phase 1 (initial baseline sync) at the time we agreed (Sat 5/30 post-Mayo). ## Mini-side concrete instructions (mini executes these) ### Step 5 — Install isync on mini (one-time, 2 min) Via SSH or directly on mini: ```bash brew install isync which mbsync # should return /opt/homebrew/bin/mbsync mbsync --version # confirm 1.4.x or later ``` ### Step 6 — Write mbsyncrc config (one-time, 5 min) Create `~/.mbsyncrc` on mini with this content (mini writes; the password reference uses macOS `security` CLI so no plaintext on disk): ``` # Yahoo Mail archive sync via Yahoo's export IMAP server # Per https://help.yahoo.com/kb/mail/SLN28681.html IMAPAccount yahoo-archive Host export.imap.mail.yahoo.com Port 993 User wklovell@yahoo.com PassCmd "security find-internet-password -a wklovell@yahoo.com -s export.imap.mail.yahoo.com -w 2>/dev/null || security find-generic-password -s yahoo-imap-export -w" SSLType IMAPS PipelineDepth 1 AuthMechs LOGIN IMAPStore yahoo-archive-remote Account yahoo-archive MaildirStore yahoo-archive-local Path ~/Mail/yahoo/ Inbox ~/Mail/yahoo/INBOX/ SubFolders Verbatim Channel yahoo-archive-channel Far :yahoo-archive-remote: Near :yahoo-archive-local: Patterns * Create Near Expunge None SyncState * ``` Replace `wklovell@yahoo.com` with Ken's actual Yahoo address. ### Step 7 — Create the Maildir target (one-time, 30 sec) ```bash mkdir -p ~/Mail/yahoo ``` ### Step 8 — Kick off Phase 1 baseline sync (Sat 5/30, run in screen) Sat 5/30 morning, in a `screen` session (so disconnects don't kill it): ```bash screen -S yahoo-sync mbsync -a 2>&1 | tee ~/Library/Logs/yahoo-mbsync-baseline-$(date +%Y%m%d).log # Ctrl-A D to detach ``` Yahoo's own warning: *"For accounts with a lot of content, it can take several days or longer."* Estimate: 2-5 days for ~20-year archive. To check progress without re-attaching: ```bash tail -50 ~/Library/Logs/yahoo-mbsync-baseline-*.log ls ~/Mail/yahoo/ -la du -sh ~/Mail/yahoo/ ``` Mini posts progress dispatches every 6h: total size, folder counts, errors if any. ### Step 9 — Phase 2 incremental daemon (after Phase 1 completes) Once Phase 1 finishes (mbsync exits cleanly), mini writes a launchd plist `com.kenlovell.yahoo-mail-sync.plist`: ```xml Labelcom.kenlovell.yahoo-mail-sync ProgramArguments /opt/homebrew/bin/mbsync-a StartInterval1800 RunAtLoad StandardOutPath/Users/kenlovell/Library/Logs/yahoo-mbsync.log StandardErrorPath/Users/kenlovell/Library/Logs/yahoo-mbsync-err.log EnvironmentVariables PATH/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin ``` Save to `~/Library/LaunchAgents/com.kenlovell.yahoo-mail-sync.plist` then load: ```bash launchctl load ~/Library/LaunchAgents/com.kenlovell.yahoo-mail-sync.plist ``` New mail flows in every 30 min indefinitely. ### Step 10 — Ingest pipeline (after Phase 1 baseline lands) Mini writes `~/Projects/memex/ingest_yahoo_mail.py` (Python, `email` module + custom extractor): - Walk `~/Mail/yahoo/` Maildir tree - For each .eml or message file: parse headers (From, To, CC, Subject, Date, Message-ID, In-Reply-To) - Insert into a new SQLite table `yahoo_mail_messages` at `~/Projects/memex/yahoo_mail.db` - Insert FTS5 index on Subject + To + From + body-first-2KB for searchability - Extract entities (person names from headers + signatures) → Memex graph nodes - Skip body content for now; selective embedding deferred to Phase 2 Ingest runs after each mbsync completes (mini launchd post-hook). For Phase 1 baseline, single bulk ingest after the multi-day mbsync finishes. ### Step 11 — Phase 3 Apple Mail spot-check (parallel, optional) If Yahoo is already in Apple Mail on MBP or mini: ```bash find ~/Library/Mail/V*/IMAP-*@imap.mail.yahoo.com -name '*.emlx' | wc -l ``` Compare to mbsync's count. If counts diverge >5%, investigate (could indicate Apple Mail didn't have full sync). ## Privacy + storage discipline - All mail bodies stay on mini local at `~/Mail/yahoo/` and `~/Projects/memex/yahoo_mail.db` - Memex indexes metadata only (sender / recipient / subject / date / message-id) — NOT body text - Monthly: encrypted backup ```bash cd ~/Mail && tar czf - yahoo | gpg --symmetric --cipher-algo AES256 > "$HOME/OneDrive/backups/yahoo-mail-$(date +%Y%m).tgz.gpg" ``` Ken sets the passphrase once via Keychain; the script reads from there. - Selective body embedding: opt-in per filter (date range, sender, subject keyword). NOT blanket. Body content does NOT enter Memex's main graph DB. - FTS5 search endpoint at `~/Projects/memex/yahoo_search.py` exposes a CLI: `python yahoo_search.py --from sarka@gmail.com --since 2018` → returns matching messages with snippet. ## Estimated costs + timing | Phase | Wall time | Ken time | Mini time | |---|---|---|---| | Steps 1-4 (Ken prep) | 10 min | 10 min | 0 | | Step 5-7 (mini config) | 10 min | 0 | ~10 min | | Step 8 (baseline sync) | 2-7 days | 0 (background) | ~0 (CPU light, network heavy) | | Step 10 (ingest after baseline) | 1-4h | 0 | 1-4h | | Step 9 (Phase 2 daemon) | ongoing | 0 | ongoing 30-min cycle | **Disk:** estimate 5-15 GB for the Maildir + 200-500 MB for the SQLite + FTS index. **Memex impact:** +tens of thousands of entity nodes (every unique sender/recipient over 20 years becomes a person node). Memex graph density increases significantly. Should still fit; mini reports after ingest. ## When to kick off Per prior plan: **Saturday 2026-05-30** post-Mayo decompression. Runs through following week. Searchable archive available ~June 6. ## Answered questions (Ken 2026-05-25 00:40 ET) 1. **Yahoo email**: `wklovell@yahoo.com` ✓ (locked into mbsyncrc above) 2. **Apple Mail on Yahoo**: present on **iPhone + iPad** (not on MBP or mini). → **Implication**: Phase 3 Apple Mail spot-check from Mac is NOT viable (iOS Apple Mail caches are sandboxed and not accessible from mini). Drop Phase 3 from plan as primary verification. If desired, Ken can add Yahoo to Apple Mail on MBP later as redundancy, but mbsync alone is sufficient. 3. **Encryption passphrase**: defer until Phase 1 lands ✓ — Ken picks then. Backup script stays unconfigured until Phase 1 mbsync completes. ## Revised Phase plan (Phase 3 dropped) - **Phase 1** — initial mbsync baseline (multi-day, Sat 5/30 kick-off) - **Phase 2** — launchd 30-min incremental delta sync (ongoing) - ~~Phase 3 — Apple Mail spot-check~~ — DROPPED. Yahoo not in Apple Mail on Mac; iOS caches inaccessible. - **Phase 4 (post-Phase-1)** — Ken picks encryption passphrase; mini configures monthly gpg-encrypted backup to OneDrive — mbp-cowork, 2026-05-25 00:10 ET / answers locked 00:40 ET — ready for Sat 5/30 kick-off pending Ken's Step 4 ready-signal