Find Your Neighbor

Pop-up museum kiosk for MIT CCC's rt@Boston project. Visitors select a conversation guide question, record a spoken response, confirm thematic tags, and are matched to a real community speaker ("neighbor") from the rt@Boston corpus via PCA-based tag similarity. They can then email the neighbor's story home.

Built for touchscreen + external podcast mic, runs fully offline (email queues and flushes on reconnect).

---

Architecture

┌─────────────────────────────────────────────────────┐
│  Electron (app/)                                    │
│  • Spawns FastAPI sidecar on port 8765              │
│  • Loads renderer/index.html in kiosk window        │
│  • Handles macOS mic TCC permissions                │
└────────────────┬────────────────────────────────────┘
                 │ IPC / HTTP
┌────────────────▼────────────────────────────────────┐
│  React renderer (renderer/)                         │
│  • Bundled by esbuild (build.js → renderer/dist/)   │
│  • Enum router (router.js) — no react-router        │
│  • D3 visualization, all 6 screens                  │
└────────────────┬────────────────────────────────────┘
                 │ HTTP :8765
┌────────────────▼────────────────────────────────────┐
│  FastAPI sidecar (backend/)                         │
│  • faster-whisper transcription                     │
│  • RAG tag suggestion (nomic-embed-text via Ollama) │
│  • PCA neighbor matching (NumPy)                    │
│  • Session logging (sessions.db)                    │
│  • Email queue (queue.db, stdlib smtplib)           │
└─────────────────────────────────────────────────────┘

Layer	Choice
App shell	Electron 30, offline-first, kiosk mode
Frontend	React 18 + D3, bundled by esbuild
Styling	CSS variables only — no Tailwind, no CSS-in-JS
Backend	Python FastAPI, sidecar spawned by Electron
Transcription	faster-whisper (local, base.en model)
Tag suggestion	RAG with nomic-embed-text via Ollama; falls back to LLM (llama3.2), then keyword
Neighbor matching	NumPy cosine similarity on PCA tag vectors
Email	stdlib smtplib + SQLite queue (offline-safe)
Session logging	SQLite (sessions.db)

---

Prerequisites

• Node 18+ and npm
• Python 3.10+
• Ollama — for tag suggestion embeddings and LLM fallback:

  ollama pull nomic-embed-text   # ~270 MB — RAG embeddings (primary)
  ollama pull llama3.2:3b        # LLM fallback (optional)

• ffmpeg — only needed if regenerating silent audio stubs

---

Setup

1. Node dependencies

npm install

2. Python dependencies

pip install -r backend/requirements.txt

3. Generate PCA projection parameters

Run once after any change to data/highlights.json. Writes (x, y) positions back into the file and saves data/projection_params.json:

python3 scripts/precompute_positions.py

4. Precompute tag embeddings

Run once (and again if the codebook or Ollama model changes). Requires Ollama running with nomic-embed-text pulled:

python3 scripts/precompute_tag_embeddings.py
# writes data/tag_embeddings.json

If this file is missing, /suggest_tags gracefully falls back to the Ollama LLM ranker, then keyword matching.

5. Configure environment (optional)

Copy .env.example to .env and fill in SMTP credentials for email delivery. Without FYN_SMTP_HOST the backend runs in dry-run mode (logs emails, marks them sent).

cp .env.example .env

---

Running

npm start          # builds renderer bundle, then launches Electron

Electron spawns the FastAPI sidecar and polls /health before showing the window. In development (!app.isPackaged) the window opens in a normal frame with DevTools available; the install build runs kiosk: true.

Use a different Python interpreter:

FYN_PYTHON=/path/to/python3 npm start

Other scripts

Command	Does
npm run build	Build renderer bundle once
npm run watch	Rebuild on file change (run electron . separately)
npm run dev	Build once + launch Electron

Run the backend standalone

python3 -m uvicorn backend.main:app --port 8765
curl localhost:8765/codebook     # parent→child tag codebook
curl localhost:8765/health       # {"ok": true, "highlights": 1332}

---

Screen flow

Attract (idle)
  └─ tap anywhere
       └─ Conversation Guide
            └─ select question + "Add your voice →"
                 └─ Add Your Voice (record)
                      └─ stop → /upload_audio → /transcribe →
                           └─ Sensemaker (transcript + tag chips)
                                └─ "Find my neighbor →" → /match
                                     └─ Find My Neighbor (D3 viz + bio panel)
                                          ├─ "Next Neighbor →" (cycles in place)
                                          └─ "Email home" → /queue_email
                                               └─ Email Home modal
                                                    └─ SEND / dismiss → Attract

Idle timeout: 90 seconds of no interaction returns any screen to Attract.

---

Data

Key files

File	Description
data/highlights.json	1,332 corpus highlights with (x, y), Q-code, codes, speaker, excerpt
data/projection_params.json	PCA parameters for visitor placement — generated by precompute_positions.py
data/tag_embeddings.json	Codebook tag vectors for RAG — generated by precompute_tag_embeddings.py
data/codebook.json	Curated parent→child tag taxonomy (falls back to data-derived grouping if absent)
data/conversation_guide.json	Questions shown on the Conversation Guide screen
data/speaker_bios.json	Speaker name, org, and bio for the bio panel

One-off data scripts (root level)

Script	Purpose
scripts/precompute_positions.py	PCA-project highlights → (x, y); save params for visitor placement
scripts/precompute_tag_embeddings.py	Embed codebook tags via Ollama for RAG tag suggestion
scripts/backfill_conversation_ids.py	Backfill conversation_id fields in highlights
scripts/fetch_catalog_entries.py	Pull catalog entries from the source data
scripts/generate_speaker_bios.py	Generate stub speaker_bios.json from corpus data
scripts/download_audio.py	Download corpus audio clips (~458 MB)

---

Backend API

Method	Endpoint	Description
GET	/health	Liveness check; returns highlight count
GET	/codebook	Parent→child tag taxonomy
GET	/highlights	Full corpus highlight list
POST	/upload_audio	Save a recorded clip; returns audio_path
POST	/transcribe	Transcribe via faster-whisper; deletes clip by default
POST	/suggest_tags	RAG/LLM/keyword tag suggestions for a transcript
POST	/match	PCA placement + cosine neighbor matching for confirmed tags
POST	/log_session	Write session row to sessions.db
POST	/queue_email	Queue an email (sends async); record contact opt-in

Tag suggestion pipeline

/suggest_tags runs these in order, stopping at the first success:

1. RAG — cosine similarity between transcript embedding and precomputed tag vectors (nomic-embed-text via Ollama). Requires data/tag_embeddings.json.
2. LLM — Ollama (llama3.2:3b) prompted to return ranked tag IDs as JSON.
3. Keyword — bag-of-words overlap between transcript and tag label strings.
4. Empty — all tags shown unranked; visitor selects manually.

Email queue

All emails write to queue.db immediately. A background thread drains the queue every FYN_FLUSH_INTERVAL seconds (default 60). If SMTP is unavailable, rows stay queued and retry on the next tick — so emails sent on flaky venue wifi still arrive.

Contact opt-ins (the "MIT CCC can contact me" checkbox) are stored in a separate contacts table in sessions.db. A row exists only when the visitor consented.

---

Configuration

All config is via environment variables (.env at project root, loaded by the backend on startup):

Variable	Default	Description
FYN_PYTHON	python3	Python interpreter Electron uses to spawn the backend
FYN_SMTP_HOST	(none — dry-run)	SMTP server hostname
FYN_SMTP_PORT	587	SMTP port
FYN_SMTP_USER	—	SMTP username
FYN_SMTP_PASS	—	SMTP password
FYN_FROM_EMAIL	—	Sender address
FYN_FROM_NAME	Find Your Neighbor	Sender display name
FYN_FLUSH_INTERVAL	60	Email queue drain interval (seconds)
FYN_EMBED_MODEL	nomic-embed-text	Ollama model for RAG embeddings
FYN_EMBED_TIMEOUT	5	Embedding request timeout (seconds)
FYN_OLLAMA_TIMEOUT	8	LLM tag suggestion timeout (seconds)
FYN_RTBOSTON_URL	https://realtalk.boston	Base URL for email links
FYN_HIGHLIGHT_URL	https://realtalk.boston/highlights	Per-highlight listen URL prefix

---

Project status

Phases 1–4 complete.

• Phase 1 (Foundation): Electron shell, FastAPI sidecar, screen router, data loading, /codebook, placeholder assets, fully offline.
• Phase 2 (Input Flow): Attract → Conversation Guide → Add Your Voice → Sensemaker, fully interactive. Real MediaRecorder capture, faster-whisper transcription, tag suggestion, PCA matching.
• Phase 3 (Visualization & Matching): D3 dot map with all 1,332 corpus highlights, PCA visitor placement, cosine neighbor matching, bio panel, audio playback, "Next Neighbor" cycling.
• Phase 4 (Backend Services): Whisper transcription, RAG + LLM tag suggestion, session logging, SMTP email queue (offline-safe), contact opt-in.

Phase 5 (Polish) — animations, transitions, kiosk hardening, asset swap-in — pending install hardware.

---

Key constraints

• Visitor recordings are never retained — the clip is deleted immediately after transcription (unless the visitor explicitly opts to contribute it).
• Nodes in the visualization are highlights (1,332), not speakers (~250). Multiple nodes can map to the same speaker.
• The bio panel is built into the visualization layout (~40% right column), not a slide-over.
• The visitor's PCA placement is a matrix multiply (<1 ms) — no model inference at matching time.
• All interactive elements must work with both mouse and touch (no hover-only states).

---

Spec documents

• PRD_FindYourNeighbor.md — full product requirements
• DESIGN_SPEC.md — design tokens, component specs, layout
• CLAUDE.md — Claude Code project context (tech stack, data contracts, build phases)