The Rust engine every native client embeds: a local SQLite working copy of a slice of one Marfa server, kept current from its event log, read locally, and written to through a queue that holds each write until the server answers it. conformance/spec/device.md, queue-and-verdicts.md and folders.md say what it does; the fixtures under conformance/src/suites/device/ test it through the marfa binary and the Node binding.
| Crate | What it is |
|---|---|
marfa-core |
The engine. src/contract.rs is written from openapi.json by pnpm generate |
marfa-cli |
The marfa binary |
bindings/swift |
The UniFFI crate and build.sh, which packages it for withmarfa/marfa-swift; a cargo workspace of its own |
bindings/node |
The napi-rs module @withmarfa/core |
marfa --help lists every command, the exit codes and the --json error shape; marfa operations maps each published operation to its command. marfa device --db PATH is a working copy, and marfa folders a folder on disk that carries its own store.
- Requests: every command sends through
marfa_core::http::Http, the stack the working copy uses: it follows no redirect, reads the contract header on every answer, and renews a refused credential once. A command waits up to 90 seconds for an answer to begin and as long again to read it, which an operation that works before it answers needs; an upload waits on the server's work, and a streamed answer fails after 45 seconds of silence. The binary writes each path, query name, body key and enum value by hand.commands/driven.rsruns each command in this process against a local server that refuses every request but the root, and holds each request a command sends, up to that refusal, toopenapi.json: the method and path, each query name, each key of a JSON body, and each enum value a flag names. A flag that takes any text leaves its value to the server. A rename in the document fails that test. - Docs:
marfa docsreads the public docs site, which is not a Marfa server, so it is the one command that does not send throughHttp. It has a small client of its own incommands/docs.rsthat sends no credential, checks no contract, follows up to five redirects, and waits 10 seconds to connect and 30 seconds each for an answer to begin and to be read. It refuses--urland--key. It readshttps://docs.marfa.so, or the address inMARFA_DOCS_URL. - Server:
--url, thenMARFA_API_URL, then the server a kept credential made current. A command that sends a key from--keyorMARFA_API_KEYhas no such fallback, so the key is never sent to the server the keychain made current: with no server named it is refused withno_server. That coversfolders pull,folders restoreanddevice blobs get, which otherwise run offline, andfolders watchwith no directory, since one key cannot serve every folder's server.login,logoutandkeys forgetsend no key, so an exported key does not change where they look. - Credential:
--key, thenMARFA_API_KEY, then the operating system's keychain, wheremarfa keys keepandmarfa loginput one. Never a plain file. With both variables set, a command reads and writes no keychain, which is how an agent runs it.--no-keychainorMARFA_NO_KEYCHAIN=1(alsotrue,yesoron) turns the keychain off for a run: no command reads or writes it, a bare command asks for--urlorMARFA_API_URL, andkeys keep,keys forget,loginandlogoutrefuse with a usage error that says how to pass a key instead.marfa login --print-tokennever touches the keychain either, so its server comes from--urlorMARFA_API_URL. A secrets manager supplies the variables:op run -- marfa items listorinfisical run -- marfa items list. Prefer the variable, the keychain, or standard input (marfa keys keepreads the key from it) to--key: a process argument is readable by other users of the machine and stays in shell history. - Plain
http: before a command sends a credential, a device code or a refresh token to anhttpaddress whose host is notlocalhost, in127.0.0.0/8or::1, it prints one warning to standard error naming the address, and goes on.--allow-http, orMARFA_ALLOW_HTTPset to1,true,yesoron, states that it is intended, for a private network that is already encrypted, and silences it. Each address is warned of once per run. A refusal in--jsonfollows the warning line on standard error. - Synced and network storage: when
folders addmakes a folder, or a watch starts on one, in a directory belowLibrary/Mobile Documents(iCloud Drive) orLibrary/CloudStorage(the File Provider services, such as Dropbox, OneDrive, Google Drive and Box) of the home directory, in a directory macOS reports as ubiquitous (isUbiquitousItem, as it does forDesktopandDocumentswith iCloud's "Desktop & Documents" on, and for every File Provider item, so the warning names iCloud Drive or another cloud service), or on a file systemstatfsreports assmbfs,nfs,afpfsorwebdav(macOS) or as NFS, CIFS or SMB2 (Linux), it prints one warning to standard error naming the cause, and goes on. The home directory is the one the operating system records for the user and the oneHOMEnames. Each folder is warned of once per run, including when a watch of every folder starts it again. A refusal in--jsonfollows the warning line on standard error. A provider that keeps its files elsewhere is not detected. The core makes the finding (Folder::storage_cause), and the Swift binding gives it to an app onFolders::add's answer and on the watch'sWatchingevent; the command line prints the warning. - Unattended runs:
MARFA_KEYCHAINnames a keychain file to use instead of the person's, on macOS, and refuses every prompt. Tests and CI use it; CI fails a run that changed the login keychain'smarfaentries.MARFA_AUTOSTART_ROOTnames a directory that stands in for the home directory forfolders autostart: the agent is written below it, and the command runs thelaunchctlorsystemctlin itsbin, never the system's. The device fixtures use it with a fake service manager, so no test installs a login agent.
Refresh, login, logout, keys keep and keys forget coordinate through one private lock per stored server for each credential store, independent of HOME, TMPDIR and XDG_RUNTIME_DIR. The lock files contain no credentials. For the person's own keychain, they stay under .marfa-credential-locks in the home directory recorded by the operating system. For a keychain file that MARFA_KEYCHAIN names, they stay in a folder beside the file, named for it with .locks added, so a run that removes its keychain's folder leaves no locks behind; a keychain file in ~/Library/Keychains uses the home directory's locks, because the person's own keychain is one of them. Leave lock files in place while commands may be running. An unsafe lock path is refused before reading or changing stored credentials. keys forget works offline without refreshing a token. Logout removes the local sign-in and reports whether server revocation succeeded; without a saved revocation endpoint, it reports revoked: false.
| Exit | Meaning |
|---|---|
| 0 | Completed, including a completed drain containing refused writes |
| 1 | Request refused or invalid, including another contract |
| 2 | Invalid command line, missing store, or missing server |
| 3 | Environmental failure, including network failure, rate limits, and full local storage |
| 4 | Working-copy or queue rule, other store fault, or unavailable credential storage |
| 5 | Missing or refused credential, or ended sign-in |
device drain preserves its complete stdout report when it exits 3 for an unavailable pass or undelivered writes, or 5 for a credential-stopped pass. It prints no second error envelope for these report outcomes. Credential stop takes precedence if both occur. A completed pass with refused writes exits 0 and its plain output separately counts refused verdicts; JSON retains each typed verdict. Held writes alone do not make an environmental failure. A direct renewal error prints its typed error on stderr with empty stdout; earlier answers remain committed and the current queued request, idempotency key, and refusal count stay unchanged. Local renewal failures carry no invented server status. An environmental renewal failure while sending a queued request returns the unavailable report; a renewal failure during a follow-up read can end the call with its typed error.
A drain that finds 50 or more creates of items, or of edges, that it has never sent, can send now and that no other kind of unanswered write is queued ahead of, sends them in pages of up to 1,000 through POST /items/bulk and POST /edges/bulk, items first, and reads what each page created back through POST /items/bulk-get, 100 rows to a request. A first sync of a folder therefore takes about 30 requests for each thousand files rather than four for each file, and stays inside the server's request limit. Each entry names the id the create was queued with, so a page that names a row the server already holds skips it rather than writing it again. A write a page does not create under its own id, or that the read-back does not find, goes on its own door under its idempotency key in the same drain and gets the verdict it would have got there. A create larger than 1 MiB, the server's default cap on a single request, goes on its own door, so the server refuses it there as it would any other. A page the server limits with 429, or refuses with 401 where the credential cannot be renewed, leaves every write in it unsent and uncounted, for the next drain to send in a page again; a 401 also blocks the queue credential_refused and stops the drain, as it does for a write on its own door. A page whose answer was lost leaves each write sent, for the next drain to send on its own door. Where the read-back fails for now, each write the page created is recorded accepted and read back before the next drain sends anything, never sent again; a read-back the server refuses stays owed, and the drain goes on. An edit queued behind a page's create goes on version 1, the version the create made, unless the read-back finds the row holding what the edit read.
A copy saves before it has reached a server. device types declare takes the types an app saves, as one JSON object or an array, and device types declared lists them; until it hydrates the copy checks each write against them and the types Marfa ships, and a hydration registers the ones the instance lacks where the key may, and says which it could not. A drain needs a completed hydration: until the first one completes, it is refused no_cursor and sends nothing. A follow on a copy never hydrated, or whose hydration was interrupted, instead waits for a hydration to complete, asking the server nothing, and one on an expired copy is refused copy_expired; device follow holds the store while it waits, so no other process can hydrate it meanwhile. A follow that has its server again after losing it drains the queue once where a drain has work, on a thread of its own so the stream is still read, and tells queue.drained with the drain's report or queue.not_drained with the error it ended with; a folder's watch drains in its own pass instead. That drain waits for one under way to end and does only the work left. The follow ending, by a stop or an error, stops that drain as a drain's stop does. device types served and device edge-types served read the server's catalog without hydrating, so a slice can be chosen first. hydrate, catch-up, drain and the two served reads stop on Ctrl-C and exit 3 with the code canceled, leaving the store consistent: a stopped hydration refuses reads, a stopped catch-up keeps its cursor, and a stopped drain leaves unsent writes queued. A stop does not wait on an upload in flight, which can take as long as the server takes, or on a write whose operation takes an Idempotency-Key in openapi.json (an item, edge, tag, metadata or extension write): that write stays queued as sent, under its key, an answer to it that arrives later is dropped, and the next drain sends it again. The server answers a keyed write from its record of the key, and takes an upload by its bytes. Nor does it wait on a page of creates: each write in the page stays queued as sent, and the next drain sends each on its own door under its key, where the server answers a create the page made as a repeat. Each binding takes a Stop; the caller raises it when its language's cancellation is requested.
A watch is the one source that an app's listeners share. Core::watch (watch in the Swift and Node bindings) adds a listener to the one watch a store has, and starts it if it was idle: a writer with a server holds the event stream and tells what a follow tells, and a store opened to read tells store.saved each time another process saves. However many listeners there are, one source runs, and only while one is attached. A hydration or catch-up takes the stream from the watch for its duration and gives it back, so an app need not stop its listeners around either, and either one called from inside a listener is refused invalid. A listener hears what changes after it joins, in order, and is first told the outage or failure that holds then. It is told server.reachable when any request reaches the server, whether the stream's, a hydration's or catch-up's that succeeded, or a drain's that answered a write. A stream that fails ends no listener: each is told watch.failed once, and no stream is held until the next hydration or catch-up. The first stream the watch has after the server was lost, or after a drain left a write undelivered, drains the queue once; the store keeps that debt (drain_owed in its meta table), so a restart does not lose it. changed must return promptly: one thread tells every listener. device watch runs one listener and ends with the error when the watch fails.
Reading commands on an absent store, including --reader and device changes, report no_store and exit 2. Queue JSON retains dependency history; plain “waiting on” names only unresolved dependencies.
A move of an edge's source changes the server's read view, so a drain that sends one the server takes ends copy_expired, but only after it has sent the writes queued behind it. Each is judged as it would have been, the reads their answers owe are left for the hydration, and any other change to the read view, or one met by a read an earlier drain left owed, still ends the drain at once. One hydration then covers every move the drain sent. A drain that also meets a wait, an unavailable server, a credential stop or a write it cannot deliver returns its report with the copy expired, and the next call is refused copy_expired; an error of its own, such as a stop or a redirect, is the one it ends with. A folders push, with or without --send-only, any of whose drains, the one after the pull included, ends copy_expired because the read view changed, as it does once the folder's key is replaced, hydrates the copy again and drains once more, so the writes still waiting are sent in the same run, each once; its report names the hydration as rebuilt, and tells the verdicts of both drains, a refusal from before the change included, and a verdict whose own read met the change. Where the drain after the pull rebuilt the copy, the push pulls once more, so the directory shows what the new view holds, and drains again where that pull placed a file; a file whose row the new key cannot read stays, and the next scan reports it lost. It rebuilds once a run: a view that changes again ends the run copy_expired, and a hydration that fails ends it with that failure. Either way the writes still waiting stay queued. A folder's watch does the same and keeps watching.
Read remote folder health with marfa folders health FOLDER_ID to inspect the last reports from each sign-in and local copy.
A program on the machine can see whether folder syncing runs, and how each folder fares, in the record of folder watches that marfa folders watch keeps beside the folder registry.
A folder remains bound to the directory it opened. If that directory or its state is replaced during a pass, the pass reports root_gone and a watch waits for the original binding to return. The registry records the identity of the directory each folder opened and keeps the folder listed while another directory, empty or not, stands at its path; it drops an entry for lost state only where the directory it opened holds none. Admission resolves symbolic links in the directories above the folder, such as /var and /tmp on macOS, and records the resolved path. It does not follow a symbolic link at the folder's own directory, at any component below it, or at any component that a later .. climbs back out of, and refuses such a path with invalid, naming the first such link. A path in which a .. climbs back out of a directory that does not exist is refused with invalid before anything is created. Every command but an add takes a path that the folder registry lists as listed, without resolving it again, and every command but an add and a removal refuses it with invalid where a symbolic link now stands in it. Any other spelling reaches a listed folder only where that folder's listed path, opened without following links, is the directory the spelling reaches; where a listed path now reaches that directory only through a symbolic link, the spelling could mean either folder, and every command but an add and a removal refuses it with invalid.
A marfa folders add that is refused or fails removes what it made: each directory it created, such as the folder's own directory and its .marfa, and each file it put in .marfa. It keeps every directory and file that was there before, and removes nothing from a .marfa that already holds a store. An add refused after it writes the folder's settings, as one that would sit inside another folder is, has already replaced any .marfa/folder.yaml that was there. It removes a directory only while that directory is still the one it made and is empty, and it follows no link.
marfa folders remove removes the folder's database, cache, and settings while preserving its item files. It leaves empty administrative directories, including .marfa, in place. It never follows a symbolic link. It takes a path through one only where that path is exactly the path the registry lists, and refuses any other path through a link with invalid, naming the listed path of the folder that path reaches. It deletes a listed folder's state only where the store at the folder's listed path is the one the registry lists for it; otherwise, as when the directory is gone or now reached through a symbolic link, it only takes the folder off the list, and reports removed as false with the reason. A removal that deletes a folder's state also takes every registry entry for that folder's store off the list, so a folder moved since it was listed leaves no entry behind. An empty .marfa does not identify a nested folder or prevent adding the directory again. Folder storage is supported on Apple platforms and Linux on filesystems that report nonzero file and directory creation times. A root without usable birth identity, or an unsupported platform, is refused before administrative state is created. Entries without usable birth identity are held. An incomplete inventory, including an unreadable file or child directory, suppresses missing-file journaling and deletion throughout the folder until a complete pass can rule out a move.
marfa folders watch with no directory watches every folder the registry lists, each with a watch of its own on a thread of its own, in one process. Each folder uses the server its copy was hydrated from: with the command's own credential where that is the server the command names, and otherwise with the credential the keychain keeps for that server, so a key named for one server is never sent to another. A folder that has never hydrated uses the server the command names. The registry is read again every 2 seconds: a folder listed since joins the watch, and the watch of one no longer listed ends. A registry that cannot be read leaves the folders watched as they are. A folder that cannot be opened, because its first sync waits, another process works it, or there is no credential for its server, is reported once and tried again every 5 seconds. The folders whose directories cannot be found, as a deleted directory or a volume that is away, are reported together in one line once that set has stayed the same for 2 seconds, and again only when another folder joins it; each is looked for again 5 seconds later, then at a wait that doubles up to 5 minutes, on its own thread, so a volume that does not answer holds up no other folder. The record of folder watches gives each the reason missing. They stay listed, since a volume that is away cannot be told from a deleted directory and the entry lets another folder hold a delete for a file that may have moved into it; folders remove takes one off the list. A folder whose watch ends, as when the server refuses its credential or its changes stop reaching it, is reported once and tried again after 60 seconds; one whose credential was refused is opened again only with another credential. A reason is reported again only when it changes, or after the folder was watched in between. Every line about one folder names its directory, and with --json each such line is one JSON object with a dir field. A folder whose directory is gone can be removed, and leaves the watch.
One process works a folder at a time. A command that works a folder, such as folders add over an existing folder, folders confirm, folders restore, folders remove, folders scan, folders pull, folders push, folders hydrate or folders unmatched --remove, and finds it held asks the process holding it to let it go, and waits up to 10 seconds. A watch, of one folder or of every folder, sees the request within a fraction of a second, finishes the step of its pass under way (sending the settings file's edit, the scan, the drain or the pull, so no write is cut off), lets the folder go and says so, and goes on with the other folders it watches. It tries the folder again every second, with the server and credential it began with, and takes it back, saying so, once the command ends. A wait the server named to the last drain of that folder that left a write undelivered, the watch's or the command's, still holds when the watch takes the folder back, or a watch of it starts, until it is over or a later drain has writes taken with no wait. A folder the command removed or took off the registry leaves the watch, and a watch of that folder alone ends. A folder held by another process, with no command asking for it, for more than 10 seconds is said once to be held by another process, and watched again once that process lets it go. A process that does not let go in time, such as a suspended watch, one whose step waits on a slow server, or another command still working the folder, leaves the command refused with reading_handle. The request is a lock the asking command holds in the folder's .marfa directory while it waits and works, which the system lets go however the command ends, so a command that crashes never leaves the folder blocked. A request that cannot be made, as in a read-only .marfa, leaves the command waiting all the same and then refused with the reason. A watch never asks, so a second watch of a folder is refused at once. The Swift binding does the same for an app: its add, sync, confirm_first_sync, confirm, restore and remove ask and wait up to 10 seconds, and are refused with ReadingHandle, naming the folder, past that. A watch an app runs through the binding lets the folder go to a command or another call, and takes it back, as the command line's watch does, and tells its listener with the events HandedOver, HeldElsewhere, TakenBack, Removed and Unlisted; a folder removed or taken off the registry ends that watch. status asks no one.
marfa folders autostart on makes every folder sync from login, and starts the sync now. On macOS it writes a launchd agent, ~/Library/LaunchAgents/so.marfa.folder-watch.plist, and loads it into the gui/<uid> domain with launchctl enable and launchctl bootstrap. On Linux it writes a systemd user unit, marfa-folder-watch.service in $XDG_CONFIG_HOME/systemd/user or ~/.config/systemd/user, and runs systemctl --user daemon-reload, enable and restart. On any other system the command is refused with invalid. It writes nothing outside your home directory and needs no administrator.
The agent runs marfa folders watch --log FILE with no directory, by the absolute path on was run by, found through PATH for a bare name, where that path leads to the binary running; a symbolic link, such as a package manager's, is kept as a link, so an upgrade that replaces what it leads to does not break the agent. Otherwise it is the path the operating system gives for the running binary. It holds no credential and no server: the only variable it sets is MARFA_FOLDER_REGISTRY, to the absolute path of the registry the command resolved, so the watch lists what folders list lists in your shell whatever environment the service manager gives it. Each folder uses the server its copy came from and the key the keychain keeps for that server, as any watch of every folder does. on refuses --url and --key, and says so when MARFA_API_KEY or MARFA_API_URL is set, since the watch reads neither. Telemetry variables are not carried, so the watch at login sends no telemetry. Running on again changes nothing unless the agent would differ, as after the binary moved; then it replaces the agent and starts the watch again. marfa folders autostart off stops the watch and removes the agent, and running it again changes nothing. Where the service manager cannot be asked, as with no user bus, or cannot stop the watch, off removes the agent's file anyway, so the watch does not start at the next login, and exits 3 saying what it could not do. An on that failed before the service manager took the new agent is taken up by the next on: launchd's old agent is unloaded before the file changes, and systemd's NeedDaemonReload shows a unit whose file changed since it was read.
The service manager starts the watch again only after it ends with a failure, and no sooner than 30 seconds after it last started: launchd KeepAlive with SuccessfulExit set to false and ThrottleInterval 30, and systemd Restart=on-failure with RestartSec=30. SIGTERM, which off, a logout and systemctl --user stop send, stops the watch as Ctrl-C does, and it exits 0, so it is not started again. Every thread the binary starts beside a command, such as telemetry's, blocks SIGTERM, so it reaches the thread that stops the watch.
The watch writes its output to folder-watch.log beside the folder registry, readable by you alone. A log of 1 MiB or more is moved to folder-watch.log.1, replacing the one there, when the watch starts and within 10 seconds while it runs, so the two files hold about 2 MiB at most. Each start writes a line with the time and the process ID.
marfa folders autostart status asks no server and needs no credential. It says whether the agent is installed, whether launchd or systemd runs the watch and as which process, the exit status the watch last ended with, the binary the agent runs and whether that binary still exists and is the one running status, where the watch logs, and, from the record of folder watches, whether the watch's entry is current and which folders are not watched and why. It says the folders whose directories cannot be found in one line, with their count. It names each folder the watch has no credential for: one not watched for no_credential, no_keychain, unauthorized or signed_out. On Linux the watch reads its keys from the Secret Service, so the keychain must be unlocked at login, as PAM unlocks GNOME Keyring or KWallet; otherwise status names every folder this way. A menu-bar app or another program reads the same record, and turns syncing from login on or off with this command, rather than running a second watch.
Run the CLI as the server's operating-system account, using the absolute path in MARFA_CONTROL_SOCKET:
marfa --socket "$MARFA_CONTROL_SOCKET" setup open
marfa --socket "$MARFA_CONTROL_SOCKET" setup claim --email owner@example.com
marfa --socket "$MARFA_CONTROL_SOCKET" owner recoversetup open issues a five-minute, single-use handoff and opens the browser. Use --no-browser to print the link. setup claim and owner recover ask for a hidden password. For automation, add --stdin and send a JSON object containing email and password for claim, or password for recovery. For a remote terminal claim, use --url https://marfa.example setup claim; its JSON input also needs code. Secrets are never command-line options. Recovery ends browser sessions and keeps ordinary keys, apps and passkeys.
--socket cannot be combined with --url, --key, MARFA_API_URL or MARFA_API_KEY. It uses no keychain and never falls back to HTTP. A command that streams its answer, such as blobs download, reaches the server through the socket too. Local authority carries no key, so a command whose operation needs one gets 401 from the server, and the command says it needs a key, which --socket does not carry. Item properties, edges, tags, search, counts, events and exports need a key, so items list, items tag, items bulk-action tags, events and export are refused. items summaries, items trash, the blobs commands, and items bulk-action job and items bulk-action cancel work without one. marfa --socket "$MARFA_CONTROL_SOCKET" keys create mints a key. Linux requires getfacl from the acl package. The socket directory must be owned by the server account with mode 0700, without access ACLs or symbolic links. The socket is 0600. A live collision is refused. After a crash, the server replaces a stale socket only when a connection is refused. A live listener or a non-socket file at that path is left intact and prevents startup. The listener holds a reservation on a private <socket-path>.lock file until its socket cleanup finishes. The file stays on disk, and a process crash releases the reservation automatically. Do not delete that file while a server is running or starting. Distinct processes need distinct socket paths.
Set MARFA_CONTROL_ONLY=true to run only the private listener when public HTTP cannot start. In the service container, marfa-entrypoint control-only starts that mode against the configured database; stop an existing listener or choose a distinct private socket path first.
cargo fmt --all --check
cargo clippy --locked --workspace --all-targets --no-deps -- -D warnings
cargo nextest run --locked --workspace
cargo build -p marfa-cli # the device fixtures refuse a binary older than its sourceThen the same three in bindings/swift. Every test binary runs under scripts/test-limits.sh, which caps the file size and processor time it can take.
marfa-cli/COMMANDS.md is the reference for every command, written as Markdown from the clap command tree: the global options and exit codes once, then each command's description, usage, and tables of its arguments and options. The docs site includes it as its CLI reference. A test fails when the committed file is out of date; after changing a command or its help text, regenerate it with MARFA_WRITE_COMMANDS=1 cargo test -p marfa-cli --bin marfa reference and commit it. The site renders the file as MDX, so another test fails on a <, >, {, or } outside a code span or fenced code block. In help text, put a placeholder or any other of these characters in backticks.
scripts/server-up.sh claims a fresh instance through its private socket and mints an ordinary working key with explicit permissions. It exports MARFA_TEST_KEY, MARFA_TEST_SOCKET, and the throwaway owner credentials from scripts/test-owner.example.env as MARFA_TEST_OWNER_EMAIL and MARFA_TEST_OWNER_PASSWORD. With MARFA_SERVER_KEEP, the key and owner survive restart. Incomplete kept credentials are refused. scripts/server-keys.test.sh exercises both boots and the owner's sign-in. Linux requires the acl package so the server can check socket ACLs.
Three jobs check the core. scripts/ci-required.ts decides from the changed paths which of them a pull request runs. A draft pull request runs none of them; marking it ready for review, and every push after, runs the ones its changes name.
Core checks (Linux)(ci.yml, Ubuntu) runs the three commands above for the core workspace. It is the only job that lints the credential store used when the operating system is not macOS, and the only one that tests the in-memory test keychain the tests use in its place. A pull request runs it when it changes the core,openapi.jsonorci.yml; a change to the Swift crate alone does not.Core checks(ci.yml, macOS) runs the same three commands, with the tests under a check that the login keychain is unchanged, then the three inbindings/swift. It keeps the macOS keychain and the Apple builds. A pull request runs it when it changes the core,openapi.jsonorci.yml. Its last step runsbindings/swift/build.sh, aswithmarfa/marfa-swiftdoes, when the change reaches the Swift crate, its lockfile or the script, so a dependency bump that stops the Swift glue from generating fails here.Core / Rust, bindings, live(core-live.yml, macOS) runs the tests that need a live server (scripts/server-up.shboots one;--run-ignored onlyruns them), builds the Swift crate in release mode for the Apple targets, builds and tests the Node module, and proves the Node binding against a booted server.ci.ymlcalls it for a pull request that changes the core,openapi.json,core-live.ymlorci.yml, soFull CIwaits for it. The Node module's JavaScript, test and script files run only here.
A push to main runs ci.yml in full unless it reaches only the format check and ci.yml passed on the commit before it. It runs core.yml, which calls the live job, when it changes the core, openapi.json, core.yml, core-live.yml, or the server and the packages the live tests boot it from. The nightly run and a manual dispatch run every job.
A pull request that changes the server and leaves openapi.json alone runs none of the three. A server change that breaks the live tests or the Node binding's proof is therefore found on the push to main, not before the merge. The Ubuntu jobs in ci.yml that run the conformance suite and the CLI scenarios build the marfa binary and run no Rust tests.
The core can report its own work, and sends it nowhere itself. It records through the tracing crate, and the host decides where the records go by installing a subscriber, such as an OpenTelemetry layer. Telemetry is off until a host turns it on in two steps:
- Enable the
telemetryCargo feature ofmarfa-core. Without it no tracing code or dependency is built, and the Swift and Node bindings leave it off. - Install a
tracingsubscriber in the host process. Until one is installed,tracingdrops every record.
marfa-core = { path = "marfa-core", features = ["telemetry"] }Cargo builds a package with a feature when any package in the same build asks for it. A command such as cargo build --workspace therefore builds the Node module's copy of marfa-core with the feature too, as soon as the marfa binary enables it. Build the Node module from its own directory, as pnpm run build does.
| Span | Covers | Fields |
|---|---|---|
drain |
One drain pass, called or started by a follow | outcome, answered, held, undelivered, unsent, unmade |
catch_up |
One Core::catch_up |
outcome, applied, skipped |
write |
One queued write sent to the server, inside a drain |
outcome, kind, status |
write_page |
One page of queued creates sent to the server, inside a drain |
outcome, kind, entries, status |
search |
One Core::search or Core::search_shown |
outcome, hits |
outcome says how the operation ended. A catch_up ends ok, error or canceled, and a search ends ok or error. A drain can also end unavailable, where the server could not be reached or asked for time, or credential_refused. A write or a write_page ends answered, unavailable, error, or canceled where a stop left it in flight. kind is the queued write's kind, such as create_item, and entries the number of writes in the page.
Two events carry code, the same word CoreError::code gives everywhere else: an ERROR event when an operation returns an error, and a WARN event when a request the drain makes gets no answer or cannot get through, and the drain leaves the work for a later pass. A failure is recorded once: a write error that ends the drain is only the drain's ERROR. A write the server answers, even with a 5xx or a 429, records no event; its write span ends unavailable with the status. A stop the caller raises ends the span canceled and records no event.
The core records only an operation's name, a count, an outcome, a write's kind, an HTTP status and an error code. It never records an item's content, title, tag, id, key or natural key, a credential, an address, a header, a path, a query or an error message. marfa-core/src/telemetry.rs is the only source file outside the tests that names tracing, and marfa-core/src/telemetry_tests.rs runs each operation with recognizable secrets and fails if any reaches a span or an event.
The binary enables the feature and installs a subscriber that exports over OTLP, but only when the environment names a collector. With none named it installs nothing, starts no thread and sends nothing, and its standard output, standard error and exit codes are what they would be without telemetry. The settings are the standard OpenTelemetry variables, and two for PostHog:
| Variable | Effect |
|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT |
The collector's base address, http or https. The trace goes to v1/traces under it. Setting it turns telemetry on. |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT |
The address the trace goes to, as written. It also turns telemetry on and takes precedence. |
OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_TRACES_HEADERS |
Headers as key=value pairs separated by commas, values percent-encoded. The traces variable wins for a header both name. |
OTEL_EXPORTER_OTLP_TIMEOUT, OTEL_EXPORTER_OTLP_TRACES_TIMEOUT |
Milliseconds one send may take. The default is 2000. The binary waits no longer than this at exit. |
OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG |
always_on (the default), always_off or traceidratio with a ratio from 0 to 1. A parentbased_ form acts as the sampler it wraps. |
OTEL_SERVICE_NAME |
The service name on the trace. The default is marfa-cli. It takes precedence over a service.name in OTEL_RESOURCE_ATTRIBUTES. |
OTEL_RESOURCE_ATTRIBUTES |
Attributes to add to the trace and the exception, as comma-separated key=value pairs with percent-encoded values, such as deployment.environment=staging. One named here replaces the command's own of the same name. A list with a pair that cannot be read is ignored whole. |
OTEL_SDK_DISABLED |
true turns telemetry off. |
MARFA_POSTHOG_HOST |
The PostHog host that receives a failed run's exception, https or, to this machine, http. It needs the next variable and an endpoint. |
MARFA_POSTHOG_PROJECT_TOKEN |
The PostHog project token for those exceptions. It needs the previous variable and an endpoint. |
A malformed OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT turns telemetry off for that run. Without --json it prints one line on standard error that names the variable and never its value; with --json it prints nothing, since standard error is then one JSON object. A query in the base address stays a query: http://host:4318/?token=abc posts to /v1/traces?token=abc. A header, timeout, sampler or ratio that cannot be used is left out, and the default applies.
Each run is one trace. Its root span is named for the command path, such as marfa items create, and carries exit_code. The core's spans are beneath it, including those made on other threads. A command that fails adds an error event to the root with code, the CLI's error code, and exit_code; a command that exits 3 or 5 after printing a drain report has an event with exit_code only. A command that ends because Ctrl-C stopped it (including a canceled hydration, catch-up, drain or served read, which exits 3), or because its output was closed, is not a failure: it keeps its exit_code and adds no event and no error status. A command line that clap refuses is not a run and exports nothing.
The binary exports only the command path, exit_code, dropped_spans, the core's span names, counts, outcomes, kinds, statuses and error codes, durations, and error and warning events with code. marfa-cli/src/telemetry/layer.rs holds the list of field names and drops every other field, and any value that is not a short plain word. An error message, an argument, an address, a key, a path and an item's content are therefore never sent, even though a message can echo them. Adding a field means adding it to that list.
The binary posts OTLP/HTTP JSON with the HTTP client it already uses, so the build adds no second HTTP stack and no async runtime. It does not implement protobuf, gzip, gRPC, metrics, logs or a retry, and it ignores OTEL_EXPORTER_OTLP_PROTOCOL. A collector that accepts only protobuf receives nothing useful. A redirect is not followed, so a header reaches only the address named. A proxy named by HTTPS_PROXY, HTTP_PROXY or ALL_PROXY carries the post to a collector on another host and is not used for one on this machine, which a proxy cannot reach.
Spans are sent from a thread of their own, in batches of at most 512 every 5 seconds, so that a long-lived command such as folders watch reports while it runs. At most 2048 spans wait for a send; past that a span is dropped and the root's dropped_spans counts it. At the end of a run the binary sends what is left and waits up to the timeout, whatever the collector does. A collector that is unreachable, slow or refusing never changes the output or the exit code, and the binary reports none of those failures.
folders watch, events and device changes run until the person stops them, and the device commands that stop on Ctrl-C (hydrate, follow, catch-up, drain and the served reads) end when stopped. Ctrl-C ends each of these through its normal exit, so the trace is sent first. folders watch, events and device changes then exit with status 130, which marfa --help lists with the exit codes; the others keep the exit they give a stopped command. A folders watch pass, in any folder it watches, that has not finished 2 seconds after Ctrl-C, because it waits on a server that does not answer, is left behind and the watch ends anyway; what it had not finished stays queued or on disk. folders watch of one folder, events and device changes take Ctrl-C once they are open, so before then Ctrl-C ends the process as the signal does and sends nothing; a watch of every folder takes it from the start. A second Ctrl-C ends the process at once with status 130 and sends nothing. SIGTERM stops folders watch as the first Ctrl-C does, within the same 2 seconds, and it then exits 0; a second SIGTERM changes nothing. A command with no stop of its own, such as a request in flight, is ended by Ctrl-C and sends nothing, and so is a process that is killed.
A failed command can also reach PostHog's error tracking, which groups $exception events that a trace does not produce. The binary sends one only when an OTLP endpoint, MARFA_POSTHOG_HOST and MARFA_POSTHOG_PROJECT_TOKEN are all set and OTEL_SDK_DISABLED is not true, the three settings the server asks for. With any of them missing the binary loads and sends nothing, and its exit codes are unchanged. Its output is unchanged too, except that with an endpoint and exactly one of the two PostHog variables set, it prints one line on standard error that names the variable and never its value, and prints nothing under --json; the trace is exported as usual. The host must be an https address, or an http address on this machine (localhost, 127.0.0.0/8 or ::1), because the project token travels in the request body, and --allow-http does not widen that. For any other host, and for an address that carries a user name or password, the binary prints the same one line and sends nothing. A query or fragment in the host is dropped.
A run fails by the rule of the trace's error event: not a run that Ctrl-C stopped, and not a canceled one. A command that panics also fails: the binary ends the trace as failed with status 101, sends one event of type panic, and lets the panic go on, so its message on standard error and its exit status are what they are without telemetry. The panic's message and location are never sent, since either can carry content. The binary posts one event to /i/v0/e/ under the host, with the HTTP client it already uses, once, with no retry, no redirect and no reading of the answer past its status. It posts beside the trace's last send and waits within the same timeout (OTEL_EXPORTER_OTLP_TIMEOUT, 2000 milliseconds by default), so a PostHog that is unreachable, slow or refusing neither changes the output or the exit code nor holds the exit past that timeout. A proxy is used or not by the trace's rule above. OTEL_TRACES_SAMPLER does not apply: the exception is sent whether or not the trace is sampled.
The event has no person. Its distinct id is random and new for each process, $process_person_profile is false and $geoip_disable is true, and the binary stores no id. Its properties are:
- The exception:
$exception_listwith one entry, whosetypeis the CLI's error code and whosevalueis a fixed sentence for that kind of error, and$exception_levelset toerror. It has no stack trace, because a CLI error has none. A run that fails with an exit status and no error, as a drain does when its report exits 3 or 5, has the typeexitand the meaning of that exit code. - The run:
command, the command's names without any argument (items create),error_codewhere there is an error,exit_code,service.nameas the trace's,service.version,$os,arch,$liband$lib_version.
The value is never the error's message, which can echo a field name, an id, a path or an address. marfa-cli/src/telemetry/describe.rs holds one static sentence for each kind of error, and a new kind does not compile until it has one. The project token is in the request body's api_key field and nowhere else. An argument, an address, a key, a path and an item's content are never sent.
Swift: bindings/swift/build.sh builds the XCFramework and its glue for macOS, iOS and the simulator. withmarfa/marfa-swift runs it at the commit it pins.
Both bindings hand properties, a queued write's body and its answer across as values, not as JSON text: the Node module as plain objects, the Swift crate as JsonValue, JsonObject and JsonMember, which the generated Swift holds as types of its own. A JsonObject keeps its keys in order. Sent to the core, a number that is not finite and a key that repeats are refused Invalid. An integer that i64 holds crosses exact; any other number crosses as an f64. A queued write's answer crosses as the JSON it held, or as its text where it was not JSON.
An app signed in with OAuth holds a refresh token the core never sees, so it renews its own credential. The Swift binding's Core::open and Folders::new take an optional CredentialRenewer. When the server refuses the token sent, the core calls its renew with that token, on the thread that was refused, and sends the request again, once, with the token it answers. Within one Core or one Folders, renew is called once at a time: calls refused together wait for one renewal and send its token, a folder still sending the token renewed away gets the new one without a call, and a folder the Folders open after a renewal starts with the renewed token. Separate Core and Folders objects call renew on their own and may call it at the same time, so an app that opens more than one with the same sign-in must serialize renewals across them. A renewer must:
- Keep the new access token and refresh token before it returns, because a refresh token is spent once the server answers.
- Return the access token it holds, without renewing, when
refuseddiffers from it. - Throw
Ended, and delete the refresh token, on a token endpoint'sinvalid_grantorinvalid_client. - Give the token request a time limit, and throw
Unavailablewhen it times out or there is no network, because stopping a watch waits for a renewal under way. - Never wait synchronously on the main thread inside
renew. - Never log the refused token.
A renewer that cannot renew throws a RenewalError, and the queued writes stay queued whichever it throws: Ended, for a token endpoint's invalid_grant or invalid_client, ends the call SignedOut, and every later refusal ends SignedOut without asking the renewer again; Unavailable ends it Network, which a drain takes as environmental and reports as unavailable; Failed, or any other error the renewer throws, ends it Invalid. The Node module takes no renewer.
The core refuses, when it queues a write, a body that nests deeper than MAX_JSON_DEPTH (64) levels, counting each array or object inside another and the body's own levels, because the server refuses a request body nested more deeply (items/json-depth). It is refused Invalid, for every binding, and nothing is queued. An item's or an edge's properties sit one level down in the body, so a property's value may nest 62 levels; an extension's body is the object itself. The Swift binding refuses a value deeper than the limit before it converts it.
A row whose stored properties are not one JSON object is damaged. The core's reads refuse a damaged row, and so does every write that reads an item's or an edge's stored properties to write from them (an edit, a delete, a read-back, a hold beneath a waiting write), so nothing writes from empty properties; a read that needs only an edge's ends and type, such as a hydration looking for the edge it is about to create, still finds it, and a server copy of the row replaces it. The Swift binding's list, get, search, edges_from, edges_to, edges_of_type, list_in_folder, search_in_folder and conflicted_copies_of keep a damaged row instead: it comes with empty properties and unreadable_properties naming why, so one damaged row fails no list or search. A damaged row matches no property condition (properties.<field> in a filter), so a filtered list or search leaves it out, and a body link's lookup of placements ignores a damaged placement edge. A write waiting on a damaged row stays queued: a hydration or catch-up that has not fetched the row again leaves it as it is, and the server's copy takes the write when it arrives. These are the core's *_kept reads. The command line and the Node module still refuse the read.
cargo-swift embeds one UniFFI bindgen, and the crate's uniffi has to write the metadata that bindgen reads. build.sh pins cargo-swift; Cargo.toml and .github/dependabot.yml keep uniffi to the version that cargo-swift embeds. To move to a newer uniffi, wait for a cargo-swift release that embeds the matching bindgen, then change the cargo-swift pin, uniffi and the Dependabot ignore in one pull request.
Node: it sits outside the pnpm workspace, so every pnpm call carries --ignore-workspace:
cd bindings/node
pnpm install --ignore-workspace
pnpm --ignore-workspace run build
pnpm --ignore-workspace run check
pnpm --ignore-workspace run testThe Node binding's withdraw(id) and discard(id) return Promise<boolean>: await them to read the result or catch a refusal. withdraw reads rows from the server, and discard can wait for a running drain. Both run off the JavaScript thread.
Every surface names a core error by CoreError::code: the code in the binary's --json envelope, and code on an error the Node module throws or rejects with. The Node error is an Error whose message starts code: , with the properties the Swift error carries as fields: serverCode, status, retryAfterSeconds, origin, location, path, reason, unsent, expected, got, served, writeSent and hash, each where the error has it. onEnd of a follow receives that same error, or one coded listener_threw where an onChange threw. The Swift error answers code() with the same word, held by a test that crosses every CoreError. A catch-up keeps the catalog it took whatever ends it, so a failed one says whether it changed: catalogChanged on the Node error, CatchUpError.Failed(error:catalogChanged:) in Swift, and catalog_changed in the binary's --json envelope.
scripts/binding-proof.sh drives the built Node binding through an offline write and its verdict against a real server.