Loresis — Technical Data-Processing Specification

1. Scope

This document describes the data mechanics of Loresis, a multi-tenant SaaS in which each customer organization ("org") gets one private chat that answers questions from that org's own connected sources. The boundary covered is the system as implemented in this repository and its deploy scripts: Next.js/TypeScript application, Neon Postgres, the on-host ingester, the per-tenant agent runtime, and the external services the code calls. It states what the code does, not what any vendor's terms promise; vendor commitments are flagged separately in Section 10.

2. Data we take

Every connector below executes only tool calls on a fixed, code-level read-only allowlist (ALLOWED_TOOL_SLUGS); any tool slug not on the list is rejected before it reaches Composio, regardless of what the underlying OAuth grant permits (src/lib/composio.ts:698-742).

Not fully traced in this pass: Google Calendar (listEvents) field-by-field inventory; the exact MIME-type exclusion list for Google Drive ingest beyond folders/trashed items.

3. Data the user gives us directly

• Account identity (WorkOS AuthKit SSO): user.email, user.id, user.firstName, user.lastName captured on session/login (src/lib/sessionUser.ts:74-82). Org membership type (employee/external), role, and status are stored per person in org_people (scripts/init-db.sql:1738-1749).
• Chat messages: full text of every user message and assistant reply, stored verbatim in Postgres chats (title, sdk_session_id, org_id, user_id) and messages (chat_id, role, content, tool_activity jsonb, verification jsonb); no redaction at write time (scripts/init-db.sql:201-239). Chat history is personal — scoped per user, not shared org-wide.
• Feedback: per-message thumbs up/down with the submitting user's id/email (message_feedback), and a free-standing feedback widget (feedback_submissions: kind, score, message, page_path, context jsonb) — both resolved only from the verified server-side session, never from request-body fields (scripts/init-db.sql:787-825; src/app/api/track/route.ts:11-24).
• Product analytics: POST /api/track via navigator.sendBeacon carries a durable localStorage anonymousId, event name, pathname, referrer, UTM params, and a size-capped JSON props blob; the server resolves userId/email/orgId only from the verified session and writes to product_events (src/app/api/track/route.ts:11-30; src/lib/trackBeacon.ts:1-34). No third-party analytics SDK is present in package.json (package.json:16-35).
• Voice dictation: recorded audio clip (≤25MB) forwarded to OpenAI's /v1/audio/transcriptions (model gpt-4o-transcribe) to produce composer text (src/app/api/voice/transcribe/route.ts:1-22,125-138).
• Voice realtime chat: the browser opens a direct WebRTC connection to OpenAI using an ephemeral client secret; raw microphone audio streams to OpenAI without transiting the app server, and when the model invokes the wiki-question tool, the wiki-derived answer text is sent back to OpenAI over the data channel (src/app/api/voice/session/route.ts:86-102; src/lib/voiceClient.ts:91-92,141,225-236).

4. Where it is stored

Postgres (Neon) is a single shared database — one organizations row per tenant, with every other customer-data table hanging off it via org_id FK, mostly ON DELETE CASCADE (scripts/init-db.sql:11-17). Two connection lanes exist: DATABASE_URL (owner role, bypasses RLS, used for nearly all reads/writes) and DATABASE_URL_RLS (app_rls role, used only for the indexing_nonces claim path) (scripts/init-db.sql:379-386; src/lib/db.ts:61-88).

Representative tables and what they hold in plaintext:

On-disk wiki: $ORG_WIKI_BASE/<slug> (env-configured, documented default /srv/liveceo/wikis), one directory per org, restructured as raw/<source>/<channel>/<date>.md and curated pages (src/lib/env.ts:568-573; src/lib/wikiBuild.ts:302-350,917-919). The on-host ingester writes as an unprivileged user with filesystem access restricted to that path (deploy/ingester/liveceo-ingester.service:8-53). Every wiki rebuild snapshots the prior directory to a same-disk .bak sibling before overwrite (deploy/ingester/refresh-slack.mjs:320-329). In production ("container" runtime), each org's agent container gets only that org's wiki directory bind-mounted read-only; the shared app container gets no wiki mount at all (deploy/lib/rollout.sh:253-260).

Backups: every 15 minutes, a whole-database pg_dump (custom format, no table filters) plus one wiki-<slug>.tar.zst per org wiki directory are rsynced over SSH to a separate Hetzner "vault" host in a different datacenter (deploy/backup/backup-run.sh:1-20,68-95,159-161; deploy/backup/provision-vault-here.sh:35,40-45). Promoted backups are protected by filesystem permissions (0440, root:vault) on the vault, not by encryption (deploy/backup/vault/promote-and-prune.py:74-98; no gpg/LUKS/cipher call found in deploy/backup/).

Ephemeral agent-session copy: the claude-agent-sdk writes a session transcript (prompts, tool results — i.e. wiki content read that turn — and replies) as JSONL under CLAUDE_CONFIG_DIR. In production this is /tmp/claude on the tenant container's writable tmpfs, persisting for the life of that long-lived container to support resume across turns (deploy/agent-runner/runner-server.mts:69-71,650). A separate dev-only in-process path mints a fresh /tmp/claude-cfg-<uuid> per turn on the real app-host filesystem with no found deletion call (src/lib/agent.ts:128-130).

5. How it is protected

In transit: Neon connections use sslmode=require (src/lib/db.ts:64). Public HTTPS is terminated by Caddy with Let's Encrypt, including on-demand TLS gated by an internal /api/internal/tls-check endpoint that only authorizes hosts marked "active" in org_custom_domains (deploy/Caddyfile.template:27-33,63-64; src/app/api/internal/tls-check/route.ts:11-30,55-67). Traffic between the app and each per-tenant agent container (the chat prompt and streamed tool events) travels as plain HTTP, not TLS, over an internal-only Docker network, authenticated by a shared-secret header instead of transport encryption (src/lib/agentContainerClient.ts:58,174,269-276). Backups travel over SSH with a pinned host key (deploy/backup/backup-run.sh:159-161).

At rest: secretBox (AES-256-GCM, per-secret random IV, format iv:tag:ct) is the sole at-rest sealing primitive, used under seven dedicated per-purpose *_KEY_ENC_KEY passphrases — Composio project keys, Slack/Telegram/custom-connector/Telegram-personal/iMessage/Recall-calendar credentials (src/lib/secretBox.ts:14-54; src/lib/env.ts requireEnv sites). Customer content is never sealed this way. A full scan of every _enc column confirms all are credential columns except auth_handoff.payload (a ≤60s single-use SSO ticket). Chat messages, wiki pages/markdown, meeting transcripts, imported AI-chat exports, and tabular dataset content are stored plaintext in Postgres columns and in plaintext markdown files on disk — protected only by transport TLS and whatever the underlying disk/volume provides (not established from this repo; see Section 10). Off-server backup artifacts (database.dump, wiki-*.tar.zst) carry no encryption step (gpg/LUKS) anywhere in deploy/backup/; protection there is 0440/root:vault filesystem permissions only. The iMessage channel stores the sender's phone/Apple-ID address only as an HMAC-SHA256 keyed digest, never in the clear.

Tenant isolation: a single shared Postgres database/schema serves every org. Real Postgres Row-Level Security is enabled on exactly one table, indexing_nonces (scripts/init-db.sql:353-368); every other tenant-scoped table is explicitly REVOKE ALL'd from the app_rls role (72 such statements), so isolation is enforced by application-level WHERE org_id = $1 scoping over the owner-privileged connection pool, not by the database engine. chats and messages get neither RLS nor an explicit REVOKE — isolation there rests on Postgres's default-deny (no GRANT was ever issued to app_rls). Per-org dataset content sits in physically separate tables named by a hash of org+connector+endpoint, not a shared table filtered by column (src/lib/datasetStore.ts:77-80).

Sandboxing: production chat turns run in a long-lived, per-tenant Docker container under gVisor (runsc), non-root uid/gid 70000:70000, --cap-drop ALL, read-only root filesystem, only that org's wiki bind-mounted read-only, a writable 512MB tmpfs at /tmp, and no direct internet route — egress is limited to a CONNECT-only proxy allowlisted to api.anthropic.com:443 (deploy/agent-lifecycle/start-agent.sh:114-142; deploy/egress-proxy/proxy.mjs:1-67). A JS-level path-containment PreToolUse hook additionally blocks any Read/Grep/Glob call resolving (lexically or via symlink) outside the tenant's wiki root — documented as defense-in-depth behind the bind-mount as the primary boundary (src/lib/agentCore.ts:1573-1607,1580-1581; src/lib/wikiContainment.ts:170). A separate, non-sandboxed "inprocess" runtime mode also exists, selected by the required env var AGENT_RUNTIME with no default (src/lib/agentDispatch.ts:1-24,77-84); deploy tooling's own comments label inprocess "current" and container "Phase 2" (deploy/redeploy-custom-domains.sh:38) — which mode runs in the current production deployment is not established from this repo.

Per-person filtering: for orgs that opt into access management, each caller's effective policy resolves to a policy_hash, and the turn is jailed under a materialized _filtered/<policy_hash> subtree with the same lexical + realpath containment check, throwing (never falling back to the full tree) on any malformed or spoofed value (src/lib/wikiContainment.ts:194-258).

Access control: WorkOS AuthKit SSO plus a signed, per-request "mb-org" cookie pinned to the session's user id (src/proxy.ts:268-279); org membership from org_domains/ org_email_allowlist (src/lib/db.ts:546-568; scripts/init-db.sql:138-152); admin roles via org_people for access-management-enabled orgs (src/lib/orgAdminGate.ts:38-58); workspace deletion gated separately by organizations.owner_email (src/lib/sessionUser.ts:150-175); platform-operator consoles gated by an OWNER_EMAILS env allowlist, 404'ing (not 403'ing) for non-owners (src/lib/ownerGate.ts:35-49,67-74; src/lib/env.ts:696-709).

6. How it is processed

Answer path: every chat turn invokes the Anthropic claude-agent-sdk query() with model: "sonnet", granting only Read, Grep, Glob over the org's markdown wiki plus a small set of purpose-built MCP tools (alerts, scheduled jobs, dataset query, tasks, gated web search) (src/lib/agentCore.ts:1525-1531,1539-1557). There is no vector database or embeddings anywhere in the codebase — retrieval is the model navigating an index.md-rooted file tree (repo-wide grep confirmed no pgvector/pinecone/weaviate/qdrant/embedding library code). Bash, Write, Edit, NotebookEdit, WebFetch, WebSearch, Task, TodoWrite are explicitly disallowed (src/lib/agentCore.ts:1566). Turns are capped at 40 tool-use turns and load no host Claude settings (maxTurns: 40, settingSources: [], src/lib/agentCore.ts:1626-1627). Prior conversation context is carried by resuming the SDK's own session id, not by re-sending prior messages (src/app/api/chats/[chatId]/messages/route.ts:456,562). The system prompt instructs the model to treat everything read from the wiki as third-party data to report on, never as instructions (src/lib/agentCore.ts:643-656). Alert evaluation and scheduled-job runs execute through the identical dispatcher/sandbox path as live chat (src/lib/alertEval.ts:145; src/lib/scheduledJobRun.ts:102,204).

Sandbox boundary: as described in Section 5 — gVisor container, read-only per-org wiki mount, egress fenced to api.anthropic.com:443 only.

Analytical lane (mcp__datasets__query): model-submitted SQL is validated by a pure AST gate — single SELECT only, forbidden statement types rejected, only allowlisted functions permitted (src/lib/datasetQuery.ts:9-25,41-88,153-166). Execution runs inside BEGIN TRANSACTION READ ONLY with an 8-second statement timeout, a 200-row cap, and 300-character per-cell truncation (src/lib/datasetQuery.ts:27-29,421-430,477-482). In container mode the agent reaches this only via an HMAC-authenticated callback to the app using a per-org derived token; the signing key itself is never mounted into any tenant container (src/app/api/internal/dataset-query/route.ts:18-30,126-131,146-173; deploy/agent-lifecycle/start-agent.sh:104-112).

Curation pipeline: the wiki curation/build pass runs unsandboxed inside the app's own Next.js process, triggered by an internal, secret-gated endpoint POST /api/internal/build-wiki (src/app/api/internal/build-wiki/route.ts:213-216,922). It sends the org's raw ingested source dumps (up to ~640,000 characters / ~160k tokens per call) as the user turn of a direct Anthropic Messages API call, model claude-opus-4-8 by default (WIKI_CURATION_MODEL override) (src/lib/wikiCurate.ts:754-757,854-863,878-880). A structural grounding linter rejects any curated page whose citations don't resolve to a real raw dump; rejected pages are dropped and recorded in a visible sentinel, and total curation failure throws rather than shipping a raw-only wiki (src/lib/wikiCurate.ts:9-14,617).

Web search: mcp__web__search sends a model-composed query (≤400 characters, distinct from the user's verbatim question) to Perplexity's sonar model via POST https://api.perplexity.ai/chat/completions, refused until the wiki has been searched at least once and capped at 3 calls/turn; in container mode this is proxied through an app-side callback since the container's own egress is fenced to Anthropic only (src/lib/webSearchProvider.ts:12-19,50-59; src/lib/webSearch.ts:74,82,84; src/lib/agentCore.ts:1253,1449-1499).

Media description: eligible native Drive images (≤4MB) and Figma rendered thumbnails are sent to an Anthropic vision call (claude-haiku-4-5) to produce a plain-text description stored in the wiki, gated behind WIKI_MEDIA_DESCRIPTION_ENABLED (src/lib/googledriveIngest.ts:105-119,198-206,244-250; src/lib/mediaDescribe.ts:1-6,29-36).

7. Where it goes

8. Who can access it

• End users: their own chat history (chats/messages are scoped per user, not shared org-wide), and, for access-management-enabled orgs, only the wiki subset their resolved policy grants (materialized under _filtered/<policy_hash>).
• Org admins (access-management-enabled orgs only): org_people rows with role='admin' and status='active' can create/edit access_rules; every such mutation is written to an append-only access_rule_audit table (scripts/init-db.sql:1817-1826). Orgs that have not enabled access management have no admin/member distinction beyond plain membership.
• Operator (platform owner): gated by the OWNER_EMAILS env allowlist. The /owner and /ops consoles expose cross-tenant plan/usage/funnel metadata (org list, plan, indexing status, connected-source counts, daily chat/reindex counts) and a failures view carrying pe.email and arbitrary pe.props JSON (src/app/api/owner/orgs/route.ts:1-45; src/app/api/owner/analytics/route.ts:264-274; src/app/api/ops/summary/route.ts:29-150) — none of the routes inspected read wiki_pages, chats, or messages content directly. Twelve deploy scripts SSH to the app server as root, which has direct access to DATABASE_URL (the RLS-bypassing owner credential) and the plaintext on-disk wiki files. No logging of ordinary reads of tenant content by anyone — including the operator — was found; only owner.denied and owner.plan_changed events are logged.
• Anthropic, as the model provider answering chat turns, receives the full text of whichever wiki pages the agent reads that turn, plus the user's question, as an intrinsic mechanic of producing an answer (src/lib/agentCore.ts:23,1525,1539-1542).

9. Retention and deletion

Deletion is two-phase. An immediate, single Postgres transaction discovers every org-keyed table via a catalog scan (so new tables are automatically covered), explicitly deletes chats (cascades handle messages/artifacts), and drops each org's physical ds_<hash> dataset tables by name; before commit, the transaction re-queries everything and rolls back if any residual row is found (src/lib/orgDeletion.ts:137-230,459-467). Two tables are exempt and survive an org's deletion indefinitely: org_deletions (the deletion receipt) and product_events — the latter carries no FK to organizations, no TTL, and stores plaintext email/user_id (src/lib/orgDeletion.ts:104-111; scripts/init-db.sql:2431-2446).

A deferred host purge, drained by a root-privileged reaper roughly every 20 seconds, removes the org's agent container, its wiki directory and .bak sibling, its dataset-query secret file, and cached TLS certificates (with a full Caddy restart, since a reload does not flush its certificate cache) (deploy/agent-lifecycle/reap-deleted-orgs.sh:93-202). The receipt is marked complete only on a purge tick that finds nothing left to remove.

Third-party revocation (Composio connected accounts, Telegram bot/iMessage webhooks, Telegram personal MTProto logout, Recall calendars) is attempted before the database transaction; a failure blocks the whole deletion by default unless the caller passes force=true (src/lib/orgRevoke.ts:93-341; src/lib/orgDeletionCore.ts:72-123). Some grants are structurally unrevokable by Loresis: the Composio project shell, the customer's own Slack/Telegram app registration, Google Meet Domain-Wide Delegation (requires the customer's own Workspace admin), and credentials embedded in custom-connector manifests (src/lib/orgRevoke.ts:343-397).

Off-server backups are deliberately unreachable by the deletion path — the app server's push credential can only write to the vault's incoming area, never read or modify the promoted store (docs/backup-and-restore.md:108-115). A manual --forget-org command can purge a promoted per-org wiki archive, but nothing in the product calls it automatically (deploy/backup/vault/promote-and-prune.py:219-249,267-275). Whole-database dumps cannot have one org's rows surgically removed; a deleted org's rows persist inside already-taken dumps until that dump ages out under a hard 28-day retention ceiling that overrides every keep-rule (deploy/backup/vault/liveceo-vault-promote.service:9-23; deploy/backup/vault/promote-and-prune.py:150-248).

Disconnecting a single connected source removes only the credential; content already ingested into the wiki is preserved (src/app/api/meeting-recording/disconnect/route.ts:1-4; src/app/api/slack/disconnect/route.ts:1-18; src/app/api/integrations/disconnect/route.ts:8-37). chats/messages carry no expires_at/TTL column, and no scheduled deletion job against them was found — content accumulates for the life of the org (scripts/init-db.sql:201-239). No bulk data-export or portability endpoint exists anywhere in the codebase; the only retrieval path while an org is active is page-by-page wiki reading through the authenticated in-app UI (src/app/api/wiki-raw/route.ts:11-38; src/app/api/wiki/[...path]/route.ts:32-60).

Two incidental copies have independent lifetimes outside this deletion path: the local Chief log-spool file (used only when the external log endpoint is unreachable, capped at 10MB, not org-scoped, not touched by org deletion) (src/lib/chiefLog.ts:29-30,67), and each agent container's tmpfs session transcript, which is wiped when that container is recreated — including by the deletion reaper's docker rm -f (deploy/agent-runner/runner-server.mts:69-71,650; deploy/agent-lifecycle/reap-deleted-orgs.sh:99).

10. Open points for the owner

Not yet verified in code

• Which AGENT_RUNTIME mode (inprocess vs. container) is actually live on the current production deployment — this determines whether the gVisor sandbox/egress-proxy protections described in Sections 5–6 are the operative boundary today.
• Whether the egress-proxy allowlist and gVisor/cap-drop/read-only container flags are actually applied on the running production fleet, versus present only in deploy scripts/docs.
• Exhaustive audit of every chiefWarn/chiefError call site's msg/stack arguments — these fields bypass all redaction (only ctx is key-name filtered), so whether any call site has ever passed a fragment of customer content through them is unconfirmed beyond the samples checked.
• Whether Hetzner (app server and vault) and Neon apply disk-level encryption at rest — no such configuration is visible from this repo; this is a hosting-provider platform property.
• Whether the host OS reclaims the dev-only /tmp/claude-cfg-<uuid> session directories; no application-level cleanup was found.
• Whether any live production database instance still holds a pre-rename plaintext Telegram personal phone number (a documented migration gap).
• Full field-by-field inventory of Google Calendar ingest (listEvents) and the exact MIME-type exclusions in Google Drive ingest.
• Whether Composio's S3-staging pattern (confirmed for Google Drive downloads) also applies to other connectors' attachment/content downloads.
• Whether restore-drill-chat.mjs (which runs the production agent core against real restored data) is ever executed against genuine production data, and on what class of machine.
• Whether the GitHub connector is live for any current customer org.
• Whether the CI check covering the agent-runner image's file closure is a required, merge-blocking branch-protection check.
• Whether org admins can see individual employees' product_events rows.

Depends on a third-party contract

• Anthropic's training/retention policy for API traffic from the answer-path agent, the wiki curation pass, artifact generation, and media description.
• OpenAI's training/retention policy for dictation transcription and realtime voice audio.
• Perplexity's training/retention policy for web-search queries.
• Composio's own data-handling, retention, and training policy for content it proxies on every connector call.
• Recall.ai's data-handling policy for meeting audio/video, and the implications of it holding a standing Google Calendar OAuth refresh token as "operational custodian."
• The data-handling policy of whichever of Gladia/Deepgram/AssemblyAI/Recall's own ASR is configured as MEETINGS_ASR_PROVIDER for a given deployment.
• Whether Telegram's own terms of service permit the MTProto "Telegram Personal" business-ingest connector.
• Google's Domain-Wide Delegation policy implications for org-wide Meet-transcript access without per-user consent.
• Neon's and WorkOS's own security posture, encryption-at-rest guarantees, and sub-processor status.
• Whether Resend, WorkOS, Hetzner, and Composio are formally listed as sub-processors in existing vendor agreements.
• The authoritative status of the "30-day org-erasure" promise the backup-retention ceiling is tied to (currently sourced to an internal roadmap document, not a published customer commitment).
• The Chief operator log-ingest endpoint's own retention and encryption policy for events it receives, given that its msg/stack fields are unfiltered by content.