Skip to content

GBRAIN_VERIFY.md

Latest commit

 

History

History
410 lines (295 loc) · 13.3 KB

GBRAIN_VERIFY.md

File metadata and controls

410 lines (295 loc) · 13.3 KB
Raw

GBrain Installation Verification Runbook

One-command equivalent: gbrain bootstrap verify runs the whole install contract (round-trip, graph floor, and more) automatically and exits non-zero on failure — it is the first thing to run after any install. See docs/guides/bootstrap.md. This runbook is the manual, deep-verification companion: use it when bootstrap verify fails and you need to isolate which layer broke, or when you want to understand what "healthy" looks like check by check.

Something feels off, or a gbrain call failed? Start with the agent operator protocol: the error's code and fix.next say what happened, who acts and how to verify; gbrain errors <code> explains a code offline, and the troubleshooting symptom table covers symptoms that are not a single error. Then use the checks below.

Run these checks after install to confirm every part of GBrain is working. Each check includes the command, expected output, and what to do if it fails. When gbrain doctor reports timeline_history, derived_visibility or unsealed pages, use gbrain repair; when a write is refused with a named reason, see write refusal reasons.

The most important check is #4 (live sync). "Sync ran" is not the same as "sync worked." A sync that silently skips pages because of a pooler bug is worse than no sync at all, because you think it's working.

1. Schema Verification

Command:

gbrain doctor --json

Expected: All checks return "ok":

  • connection: connected, N pages
  • pgvector: extension installed
  • rls: enabled on all tables (Postgres/Supabase brains only — PGLite brains skip this check; the embedded engine has no remote surface)
  • schema_version: current
  • embeddings: coverage percentage

A check that carries skipped_segmented: N (message ending in skipped (segmented): N) did not read N pages stored in segments, so its ok covers only the pages it read; see segmented page storage.

If it fails: The doctor output includes specific fix instructions for each check. See skills/setup/SKILL.md Error Recovery table.

1a. Migration Plan and DB Probe

Command:

gbrain apply-migrations --list      # per-version status: applied / partial / wedged / pending / future
gbrain apply-migrations --dry-run   # what a real run would apply or resume

Both surfaces are read-only — they never run orchestrators or schema migrations, even when combined with --yes. Each prints a Database: probe line above the plan, so an unreachable database is distinguishable from a clean one:

  • Database: connected, schema vN (latest M) — the pre-flight probe connected. N behind M means schema migrations are pending; a plain run with --yes applies them.
  • Database: UNREACHABLE (<reason>) — the pre-flight connect failed. The reason is scrubbed through both credential redactors (URL userinfo + connection-info), so it is safe to paste into issues and CI logs.
  • Database: not probed (<reason>) — no probe was attempted. Normal on PGLite (pglite manages schema in-process): the orchestrators handle the schema lifecycle internally there, and the probe would briefly hold the single-writer lock.

Expected: Database: connected, schema vN (latest N) on Postgres, or Database: not probed (pglite manages schema in-process) on PGLite, followed by All migrations up to date.

If it fails: an unreachable database does not stop a default run — orchestrator migrations still run their filesystem-only phases. Scripts and CI that need a hard signal add --require-db: with it, --list and --dry-run exit 1 when the probe failed, and a real run prints the probe line and aborts with exit 1 before any orchestrator runs. The observational sibling is gbrain doctor --no-migrate, which connects probe-only so a clean-or-behind schema is reported on as-is instead of being auto-migrated before the health checks run.

2. Skillpack Loaded

Check: Ask the agent: "What is the brain-agent loop?"

Expected: The agent describes the read-write cycle documented in docs/guides/brain-agent-loop.md: detect entities, read brain, respond with context, write brain, sync.

If it fails: The agent hasn't loaded the skillpack. Have it read docs/GBRAIN_SKILLPACK.md (the index) and follow the Core Patterns links.

3. Auto-Update Configured

Command:

gbrain check-update --json

Expected: Returns JSON with current_version, latest_version, update_available (boolean). The cron gbrain-update-check is registered.

If it fails: See docs/guides/upgrades-auto-update.md for how to register the update-check cron.

4. Live Sync Actually Works

This is the most important check. Three parts.

4a. Coverage Check

Compare page count in the DB against syncable file count in the repo:

gbrain stats

Then count syncable files:

find /data/brain -name '*.md' \
  -not -path '*/.*' \
  -not -path '*/.raw/*' \
  -not -path '*/ops/*' \
  -not -name 'README.md' \
  -not -name 'index.md' \
  -not -name 'schema.md' \
  -not -name 'log.md' \
  | wc -l

Expected: Page count in gbrain stats should be close to the file count. Some difference is normal (files added since last sync), but if page count is less than half the file count, sync is silently skipping pages.

If page count is way too low (Supabase/Postgres brains): The #1 cause is an unreachable direct connection on an IPv4-only host. (PGLite brains have no network layer — for them, check that the sync cron/watch is actually running.) GBrain uses the Transaction pooler (port 6543) for reads, but routes migrations, DDL, and sync transactions to a derived direct connection (db.<ref>.supabase.co:5432), which is IPv6-only.

  • On an IPv4-only host, reads work but sync transactions fail and silently skip pages.
  • Fix: set GBRAIN_DIRECT_DATABASE_URL to the Session pooler string (port 5432 on the pooler.supabase.com host, IPv4), or enable Supabase's IPv4 add-on. Then run gbrain sync --full to reimport everything.

4b. Embed Check

gbrain stats

Expected: Embedded chunk count should be close to total chunk count.

If embedded is much lower than total:

gbrain embed --stale

Without a key for the configured embedding provider (VOYAGE_API_KEY for the default voyage:voyage-4, OPENAI_API_KEY for OpenAI models), embeddings can't be generated. Keyword search still works without embeddings, but hybrid/semantic search won't.

4c. End-to-End Test

This is the real test. Edit a brain page, push, wait, search.

  1. Edit a page in the brain repo (e.g., correct a fact on a person's page):
# Example: fix a line in alice-example's page
cd /data/brain
# Make a small edit to one .md file, then commit and push only that file
gbrain git commit-push -m "test: verify live sync" -- people/alice-example.md

It must exit 0 and print pushed <sha> to origin/<branch>. On any other result, stop the test and follow outcomes (retry once on commit_lock_busy; ask the user on integrate_conflict).

  1. Wait for the next sync cycle (cron interval or --watch poll).

  2. Search for the corrected text:

gbrain search "<text from the correction>"

Expected: The search returns the corrected text, not the old version.

If it returns old text: Sync failed silently. Check:

  • Is the sync cron registered and running?
  • Is gbrain sync --watch still alive (if using watch mode)?
  • Run gbrain config get sync.last_run to see when sync last ran.
  • Run gbrain sync --repo /data/brain manually and check for errors.
  • If sync errors mention an unreachable host or connection timeout, the direct connection isn't reachable on IPv4 (see 4a above).

5. Embedding Coverage

Command:

gbrain stats

Expected: Embedded chunk count matches (or is close to) total chunk count.

If zero or very low: the embedding provider key (VOYAGE_API_KEY for the default model, OPENAI_API_KEY for OpenAI models) may be missing or invalid. Check:

echo $VOYAGE_API_KEY | head -c 10   # or $OPENAI_API_KEY

If blank, set the key. Then:

gbrain embed --stale

5a. Fact and take vectors

Chunk coverage says nothing about facts and takes, which are searched by vector too (fact dedup and consolidation, think, takes search --semantic).

Command:

gbrain doctor --json

Expected: the fact_take_vectors check is ok. On a keyless brain (embeddings disabled) it reports "not applicable" and stays ok. details.facts.partially_embedded counts facts longer than the 8000-character embedding input: their vector covers only the start of the fact. They count as current, so nothing re-embeds them or spends on them again (#6975).

If it warns (code stale_vectors): its details count, per source, facts and takes with no vector for their current text under the brain's embedding model, and details.fix lists the exact commands. Takes are repaired by the ordinary stale pass:

gbrain embed --stale

Facts are repaired per source, after a preview (paid provider calls need explicit authorization):

gbrain embed --facts --stale --source <source-id> --dry-run --json
gbrain embed --facts --stale --source <source-id> --yes --max-cost-usd 1

If the check says the vectors were not verified (code vectors_not_verified), run gbrain migrate embeddings --status to see which model and width the brain records.

6. Brain-First Lookup Protocol

Check: Ask the agent about a person or concept that exists in the brain.

Expected: The agent uses gbrain search or gbrain query FIRST, not grep or external APIs. The response includes brain-sourced context with source attribution.

If it fails: The brain-first lookup protocol isn't injected into the agent's system context. See skills/setup/SKILL.md Phase D.

7. Knowledge Graph Wired

The graph layer needs to be populated for brains with pre-existing content. New writes are auto-linked, but pages imported before the graph existed need a one-time backfill.

Command:

gbrain stats | grep -E 'links|timeline'

Expected: Both links and timeline_entries are non-zero (assuming the brain has content with entity references and dated markdown).

If it's zero on a brain with imported content: Run the backfill.

gbrain extract links --source db --dry-run | head -5    # preview
gbrain extract links --source db                         # commit
gbrain extract timeline --source db
gbrain stats                                             # confirm > 0

Bonus check — graph traversal works:

# Pick any well-connected slug from your brain
gbrain graph-query people/<some-person-slug> --depth 2

Expected: Indented tree of typed edges (--attended-->, --works_at-->, etc.). If the slug has no inbound or outbound links, try a different one or run extract again.

If extract finds nothing: Your pages may not use entity-reference syntax. The extractor matches [Name](people/slug), [Name](../people/slug.md), and bare people/slug references. If your brain uses a different format, the auto-link heuristics won't find them — file an issue with a sample page.

8. JSONB Frontmatter Integrity

A Postgres-backed brain can carry double-encoded JSONB columns (frontmatter->>'key' returns NULL, GIN indexes are inert). gbrain upgrade runs gbrain repair-jsonb automatically via the v0_12_2 orchestrator. Verify the repair succeeded.

Command:

gbrain repair-jsonb --dry-run --json

Expected: totalRepaired: 0 across all 5 columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). A zero count means every row is properly-typed JSON objects, not string-encoded JSON.

If the count is > 0: The repair didn't run or was interrupted. Re-run without --dry-run:

gbrain repair-jsonb

Idempotent. PGLite brains always report 0 (the embedded engine never double-encodes).

Bonus check — the doctor's dedicated JSONB scan agrees:

gbrain doctor --json | grep -o '"name":"jsonb_integrity"[^}]*'

Expected: the fragment contains "status":"ok" ("All JSONB columns store objects/arrays"). If it reports double-encoded rows, run gbrain repair-jsonb.

Quick Verification (all checks in one pass)

# 0. The one-command contract check (exits non-zero on failure)
gbrain bootstrap verify

# 1. Schema
gbrain doctor --json

# 2. Sync recency
gbrain config get sync.last_run

# 3. Page count + embed coverage
gbrain stats

# 4. Search works
gbrain search "test query from your brain content"

# 5. Catch any unembedded chunks
gbrain embed --stale

# 6. Auto-update
gbrain check-update --json

# 7. Knowledge graph populated (links + timeline > 0)
gbrain stats | grep -E 'links|timeline'

# 8. JSONB integrity (Postgres only, PGLite always 0)
gbrain repair-jsonb --dry-run --json

If all eight return successfully, the installation is healthy. For the full end-to-end sync test (4c), push a real change and verify it appears in search.