Reliability operations

Product boundary and capability preflight

OpenRealtime's Python voice runtime is the reusable framework. The Next.js workbench is a reference application for voice, text and digital-human workflows. Provider configuration does not imply that an external service is healthy. The current homepage wires Anam for hosted sessions; HeyGen/custom selections remain unavailable there even if the separate session API is configured.

GET /api/avatar-capabilities returns configuration/admission states for asset management, Anam, HeyGen and local rendering. It is uncached, contains no keys or internal service URLs, and uses the same access policy as the respective APIs. The homepage explains missing access/configuration before avatar startup, offers administrator login when applicable, and allows retry when preflight fails. Text input remains available inside an active session; the homepage has no separate text-chat entry. Creation and library pages show the asset-access notice. Session APIs still enforce access on every request; the capability response never authorizes an operation.

Anam and HeyGen require an authenticated admin session or the explicitly enabled, same-origin local development policy. A client-supplied header does not authorize a paid session. Local rendering additionally requires the preview policy.

Avatar catalog concurrency and recovery

Production Compose uses OPENREALTIME_CATALOG_LOCK_BACKEND=postgres with PostgreSQL advisory locks. Lock identity is derived from the shared absolute directory path. Nested work in different directories reuses one lock connection; re-entering the same directory fails instead of permitting concurrent mutation. A separate pool keeps auth queries from being starved by lock holders. Killing a writer closes its connection and releases its locks. This still requires a shared durable filesystem. Stop every writer before changing lock backends or upgrading the history/Memory directory format; old and new locking mechanisms must not run against the same files concurrently.

The following recovery instructions apply to the default local filesystem mode.

The file repository uses a .catalog-lock directory acquired through exclusive mkdir for the complete read/modify/write operation. All cooperating processes using the same filesystem path therefore serialize changes. Reads that also purge expired records and reads of replaceable image bytes participate in the same lock. Catalog writes retain atomic temporary-file replacement. Invalid catalog records stop reads and writes rather than being silently filtered.

Set AVATAR_CATALOG_DIR to one persistent shared directory when running multiple processes. The filesystem must support atomic directory creation and rename. Separate replica-local filesystems do not share data; this backend is unsuitable for deployments that cannot meet these conditions. The Python PostgreSQL state setting does not migrate Avatar metadata. A transactional database adapter is a future deployment option, not part of this file-backend change.

Acquisition waits up to five seconds, then reports a retriable error. A crashed writer may leave a lock behind. We deliberately do not remove a lock based on age, since a paused but living process may still own it. To recover:

1. Stop every application process using the directory.
2. Back up the directory and inspect catalog.json for valid JSON and schema.
3. Remove the empty .catalog-lock directory with rmdir.
4. Restart the application and verify the catalog can be listed.

Do not remove the lock while any writer remains active. Restore a damaged catalog from backup before restarting; do not replace it with an empty array.

The local recovery regression starts an independent Node writer, waits for its lock-acquired acknowledgement, sends SIGKILL, and waits for the actual exit. A second write must time out without changing the synthetic catalog. Only after that confirmed stop does the test remove the empty lock and verify the existing record remains readable. Run pnpm exec vitest run apps/web/lib/avatar-catalog-lock.test.ts. This proves the documented offline recovery path, not automatic stale-lock detection or safety when an unobserved writer remains alive.

Explicit confirmations

The browser renders a native modal dialog using the server-owned summary. Focus starts on Reject, background controls are inert, Tab remains within the dialog, and focus returns to the initiating control when it closes. Escape rejects. Strong-auth actions cannot be approved by this interface. Decisions carry the confirmation ID and action digest and are validated by the server. Pending confirmations are re-emitted during recovery; a UI click alone does not prove that a protected action completed.

Verification profiles

• pnpm test: deterministic unit/contract checks. No external services or untracked local media are required. Three installed-media/evidence checks are explicitly skipped here.
• pnpm test:media:local: checks those actual optional media and research evidence files, failing when they have not been installed. It does not synthesize or substitute evidence for the original evaluation runs.
• pnpm build: production compilation and framework checks.
• pnpm exec playwright test tests-e2e/text-fallback.spec.ts tests-e2e/confirmation-access.spec.ts tests-e2e/avatar-capabilities.spec.ts: deterministic browser regressions, including text retry and keyboard access.
• uv run python scripts/check_mypy.py: blocking gate against config/mypy-baseline.json. Diagnostic identity includes file, message and error code, with occurrence counts. Source line movement does not create a regression; additional errors or checker execution failures fail the gate. The baseline is reviewed technical debt, not a suppression of all future errors. Shrink it when fixing old errors; do not expand it merely to pass CI.
• pnpm test:voice:live: real-provider smoke, requiring explicit credentials. Passing deterministic tests does not verify current network, audio quality, vendor quota, or GPU renderer health.

CI runs Python tests and the type-regression gate, web lint/typecheck/unit tests, production build, and the browser regression subset. Browser failures retain screenshots/traces as CI artifacts. Node 22 and frozen dependency lockfiles keep these checks reproducible.

Frontend state boundaries

apps/web/voice/session/transcript.ts owns pure semantic transcript/caption projection: turn identity, ASR revisions, assistant settlement and spoken/display selection. It has no React or browser audio dependency. use-voice-session keeps compatible re-exports while coordinating transport and device input/output runtimes. This is an incremental separation; extracting the remaining runtime coordination is a future refactor, not a claim that the complete hook has been rewritten.

Internal history reconciliation read

The control plane exposes POST /conversation-history/read for a trusted local Web service. The caller supplies tenant_id, user_id, thread_id, and optional after_sequence (default 0) and limit (default 200, maximum 500). Derive identity from the authenticated principal and the saved conversation association; never forward arbitrary browser identity fields. The control plane remains behind the deployment network boundary and must not be publicly exposed.

The endpoint rejects external Host values, browser Origin, Authorization headers, query parameters and transfer encoding. It returns only USER-visible user/assistant text with canonical item ID, Journal sequence, Session ID, turn ID, generation and status. Internal/tool data and provider metadata are omitted. next_sequence is a cursor to pass as the next after_sequence; it advances over hidden events too. A full page can have a cursor even when the next page is empty. Erased threads return 410, and a missing reader returns 503. Responses disable caching.

The Web service exposes authenticated GET /api/conversations/{id}/runtime with after_sequence and limit. It resolves the saved conversation for the authenticated owner before calling the internal reader; browser identity/thread query fields cannot change that scope. The bridge rejects malformed or oversized responses (2 MiB), strips undeclared fields and disables caching. This read does not mutate saved history or pending drafts. History details now provide a manual server-record check with 50-event pages, visible-text comparison after the last page, and a separate owner-scoped unconfirmed draft view. Owner digests from both Web reads must match before reading IndexedDB. Already acknowledged snapshots are not classified as pending drafts. After all server pages and local draft checks succeed, matching role/text prefixes can be combined in a text preview. Branching content remains separate. The export preserves original Web/server entries and the local draft, marks whether server paging and draft reads are complete, and includes the preview without changing stored data. This does not prove canonical message identity from text equality. For prefix-compatible saved Web records with no local pending draft or draft read failure, history details offer explicit restoration of the missing server suffix. POST /api/conversations/{id}/recover requires authenticated write access, the expected owner digest and Web revision. It rereads all server pages outside the file lock, rejects generating entries/repeated canonical IDs, then acquires the normal writer lock and rechecks owner, revision, expiry and tombstones before appending. Divergent or shorter server projections are rejected, original Web text and custom titles are preserved. Matching partial entries may gain the server-confirmed final state; newly finalized user messages are also eligible for role-controlled Memory extraction. The revision increases only for an actual append or final-state repair. The last server Session becomes the continuation association. Added final user messages run the existing role-controlled Memory extraction under the same writer lock; extraction failure is reported separately from the successful history commit. Recovery never changes IndexedDB drafts. Other tabs with stale drafts get the existing revision conflict and retain their draft.

Final-state repair remains available when message counts match. Eligibility uses the same pure rule in UI and persistence: exact role/text match and confirmed final source state; generating/cancelled or explicitly non-final sources cannot settle a partial message. A shorter server projection does not offer repair. The completed comparison can be restarted from sequence zero, rereading the current owner-scoped draft without modifying it. Cancelled server messages are displayed as failed/non-final rather than completed.

Server paging has a 30-second soft deadline (an in-flight request can add up to 10 seconds), a 10,000-page guard and the existing 100,000-entry Web capacity. Failures before commit preserve the original record. An uncertain success after client timeout requires refresh and rechecking the current revision; no automatic blind overwrite is permitted. Branch resolution and the full real browser, PostgreSQL and production recovery flow still require verification. Do not replace a user's unconfirmed draft with this projection or claim browser history has been reconciled merely because the endpoint responds.

Avatar identity and saved catalog precedence

Treat avatar_id as identity; display names are editable labels. Two avatars with the same name remain distinct, and the latest updated_at wins only when records have the same ID. A newer deletion or a replacement with fewer actions must not be discarded in favor of older, richer material.

The current avatar catalog comes from the server aggregate. Bundled presets must not be merged into that response, including an empty catalog, and a failed request must not be converted into a successful preset catalog. Presets are explicit templates, not runtime recovery data. Saved images and actions remain associated by avatar and action IDs, including when a built-in ID is renamed.

The home stage waits for the current catalog and checks that the selected avatar matches the role. An ID in browser preferences alone does not validate its cached image or action. On catalog failure, hide the local stage, clear its background image, block starting the avatar session, and display an error. Realtime provider previews remain independent of local catalog availability.

Regression checks: pnpm exec vitest run apps/web/lib/avatar-library-catalog-client.test.ts apps/web/components/call/home-preview-workspace.test.tsx apps/web/components/workspace/home-backdrop.test.tsx.