Skip to content

Latest commit

 

History

History
151 lines (133 loc) · 8.75 KB

testing.md

File metadata and controls

151 lines (133 loc) · 8.75 KB

Regression and integration tests

Default suite

Run task studio:install once, then task test. The default suite includes the root Go module, both nested ACP modules, generated ACP binding checks, and the Studio Svelte/TypeScript check. Browser tests are opt-in and are not silently counted as passing when Chromium is missing.

The native shell is an isolated module. Run task desktop:check for its tests and vet; see desktop verification for checks against the assembled engine. The root suite covers the shared HTTP runtime. The Q Studio desktop release workflow builds installation artifacts on Windows, macOS and Linux for both amd64 and arm64. macOS ARM64 cross-builds with Apple's SDK on Intel macOS and verifies architecture plus cross-target vet; the other five targets test their assembled engines on matching hardware. Numeric v<version> tag pushes publish the complete artifact set to a GitHub Release only after all builds succeed; manual runs can build without publishing. A successful package build does not replace a platform UI/login test.

UI-state regression tests cover file persistence across server restarts and ports, concurrent partial updates and legacy migration without resurrecting cleared drafts. Browser checks in ui-state.spec.mjs use two real HTTP origins and also exercise blocked localStorage and save retries. The shared browser fixture resets only its isolated UI-state file between tests.

Focused integration and regression checks:

go test ./studio ./gitwork ./tools/builtin -count=1 -timeout=3m

New Studio runtime integration tests use a real SessionHost, managed Gateway child, Loom, filesystem tools, and session/event storage. Only the model is a local OpenAI-compatible HTTP fixture. They cover:

  • Guidance sent through the HTTP command API, interruption, a successor run, redirect events, latest-run lookup, actual file creation, and persisted guidance/tool results.
  • Question answers, stale call IDs, pause/resume, and rejection after completion.
  • Recovery from an active run with a stale snapshot and a partial final log record; a second restart must still replay all durable events exactly once.
  • Complete JSON records missing a final newline, corrupt complete records, and cursor gaps. Only incomplete final JSON can be discarded during recovery.
  • Run logs beyond 128 MiB, cursor paging after repeated restarts, and records at the writable event-size boundary. Recovery streams the existing NDJSON format; only record offsets remain in memory, and pages read their event bodies from disk. Missing indexed records return an error instead of an empty success.
  • Gateway and System One provider creation/deletion, duplicate rejection, handler reopen, last-provider protection, and preservation of saved settings when Gateway application fails.

Git regressions use real temporary repositories and worktrees: an unchanged task closes and removes its lease; parent changes, branch switches, and dirty parents reject a merge without changing either checkout.

Command regressions run the platform shell: a wait timeout keeps the command alive, a later wait observes completion, runtime close terminates a running command, and output cursors report eviction and drain without duplicate bytes.

Studio browser suite

task studio:install
npm --prefix ./studio/frontend run test:e2e:install
task studio:test

npm --prefix ./studio/frontend run test:e2e is equivalent. Playwright is pinned in the frontend lockfile; the install command installs its matching Chromium. There are no retries. A missing browser fails with an installation diagnostic.

The suite builds the production frontend, then runs a Go test fixture on a random loopback port. That fixture starts an isolated real Studio handler and SessionHost with temporary settings, projects, workspaces, and local model responses. It does not contact an external model or use your user settings or provider credentials. The fixture shutdown endpoint allows Go to close its services and remove temporary data before Playwright stops the process. Gateway providers and the native System One catalog are configured with local HTTP fixtures, so settings cases also work when run individually.

Browser assertions cover project/workspace selection and reset, a guided turn without page refresh, replay after reload, repeated tool IDs in later turns, Markdown headings and highlighted code, automatic checkbox persistence, and project creation/edit cancellation/update/deletion. The folder browser starts from home, closes before its owning dialog on Escape, and passes the selected workspace to real session registration. Reopening a dialog resets its inputs. Switching sessions aborts a held event request and rejects its late content; a delayed session detail response cannot overwrite a newer selection. Input is disabled until that selection finishes loading. Integration browser tests exercise MCP transport, environment grants, role grants, invalid JSON recovery, and deletion; queued LSP writes stay in their original repository while the selected path changes. Ignore edits keep their document's revision and flush on panel/route changes. A disposable portable skill renders read-only and reindexes through the real local services. Subagent tests cover profile role/prompt/tool/delegation grants and deletion, queued revision chains and scope moves during a repository switch, and ACP connection secret masking, enabled options, role bindings, and deletion. They do not launch an external ACP process. The fixture also seeds a nested delegation tree through workspace storage APIs. Browser assertions select a child from another root, switch transcripts, and delete a grandchild through Studio while preserving its parent and root. This checks persisted tree navigation and deletion, rather than model-driven delegation execution. Changes tests use real temporary Git repositories: late diffs cannot replace a new repository selection, queued commit messages retain their order, and execution commits only the reviewed repository. Folder browsing preserves paths typed before its initial home lookup finishes. Operations tests cancel an old usage period, refresh on the timer, and stop polling after navigation. Global settings tests cover role/group creation and deletion, queued group edits across sections, Gateway/System One provider renames followed by another save, decision role routing, listener persistence, and both servers' API key creation and revocation. Provider discovery uses local HTTP catalogs, rather than live provider interoperability. Files tests exercise directory expansion, raw syntax highlighting and copying, line selection retained in the URL, Raw/Diff switching and reload, and clean, deleted, binary and bounded large-file previews. Delayed reads cannot replace a new file or mode. A real temporary Git worktree supplies content distinct from its parent for session file-link tests, including project workspace switching and reopening the same link. Files API tests also cover subdirectory Git scope, directory pagination and symlink escapes, with read-only index verification. Full-file Diff assertions retain unchanged first/last lines, check real old/new line numbers and gutter markers around additions and removals, preserve multiline syntax highlighting, and switch between HEAD/worktree and index comparisons. Help tests cover command reference completeness, section navigation, wheel and Home/End scrolling, terminal shortcut disclosure, and Studio navigation shortcuts at 1440px, 900px, and 390px widths. The CLI commit regression executes the real command entry point in an isolated non-repository and verifies that it enters the Git-backed terminal workflow without opening Studio; commit-agent tests cover proposal review, approval, index validation, and execution in temporary repositories. The suite also verifies settings scrolling at 1440px and 900px widths. Settings keep their selected section, cached snapshot, and pending save queue across page navigation and browser history. Tool call and result cards must share both edges at 900px, 1440px, 2560px, and 3440px widths, collapsed and expanded, without horizontal transcript overflow. Unhandled JavaScript errors and unexpected HTTP errors fail the suite. A new session's expected latest-run 404 is allowed.

Failed tests retain screenshots, an error context, and Playwright traces under studio/frontend/test-results/ (ignored by Git). For example, from the frontend directory:

npx playwright show-trace test-results/<case>/trace.zip

The suite establishes deterministic runtime and UI behavior. It does not measure live model quality, provider interoperability, or all visual layouts. Windows execution checks the Windows shell and file behavior; run the same suite on POSIX to establish its platform-specific behavior.