Gamertan Observatory

Gamertan Observatory is a self-hosted, organization-aware observability platform in active private development. It is being built for logs, metrics, traces, incidents, and deployment evidence on modest Linux servers.

This repository currently contains the first security and durability vertical slice. It is not a public preview yet.

Current development surface

• strict, versioned server configuration;
• organization/project/environment/service-scoped ingestion sources;
• hashed, rotatable source credentials;
• bounded log, metric, trace, and deployment batches;
• immutable checksummed zstd raw segments committed before acknowledgement, with an atomic cold forensic tier instead of destructive hot-window expiry;
• SQLite control state and independently migratable per-organization projections;
• sequence deduplication, replay rejection, and crash recovery;
• enforced organization-aware hot/cold retention, crash-recoverable archival, indefinite forensic preservation by default, explicitly enabled final retirement, projection compaction, budgeted cold queries, and queryable five-minute metric rollups;
• an exact five-minute HTTP-status/normalized-route log projection for the common error-count view, with one bounded raw fragment at an unaligned time boundary rather than a whole-window rescan;
• an offline, explicitly approved per-organization projection rebuild that validates checksummed raw truth, rebuilds activated descriptors beside the live database, and atomically replaces only the disposable projection;
• pinned Web Foundations v0.1.0-preview.3 users, secure sessions, personal organizations, teams, invitations, and scoped access grants;
• local single-use operator bootstrap with an application-generated one-time credential, mandatory password replacement, and transactional session revocation; platform operation remains deliberately separated from telemetry access;
• a typed, bounded query AST and parser;
• one validated AST shared by text and visual-builder inputs, including safe regular expressions, aggregates, grouping, windows, sorting, and limits;
• explain planning that injects resource scope independently, resolves field descriptors, reports indexes and estimated scan, enforces budgets, and requires a separate permission for sensitive or unknown fields;
• bounded typed execution over per-organization projections, including filters, safe regular expressions, lookback windows, aggregation buckets, grouping, exact raw-sample percentiles, explicitly marked approximate rollup percentiles, sorting, and stable tabular results;
• same-origin session, health, readiness, native ingestion, query-parse, query-explain, and authenticated query-execution HTTP endpoints;
• strict agent configuration, private outage spooling, and HTTPS delivery;
• continuous bounded file tailing with durable cursors, truncation/rename handling, independent stream ordering, and crash recovery from spool checkpoints;
• short-lived, single-use, scope-bound agent enrollment and self-revocation if a newly issued credential cannot be persisted;
• whitelist-only adapters for Caddy access JSON, Web Foundations requestlog JSONL, and Tend deployment events; minimized defaults omit queries, addresses, referrers, user agents, cookies, credentials, and arbitrary fields, while explicitly configured privacy-policy-backed fields remain bounded and individually selectable.
• authenticated OTLP/HTTP protobuf endpoints for logs, metrics, and traces, with bounded identity/gzip decoding, source-owned scope, secret-key deny rules, typed signal conversion, and automatic per-stream sequencing.
• unprivileged Linux host metrics from bounded /proc, cgroup v2, named filesystems, and explicitly selected PID files; no shell, root agent, Docker socket, environment read, or command-line capture is involved.
• idempotent per-organization descriptor proposals aggregated once per raw segment; unknown fields remain sensitive, high-cardinality, and unindexed until an organization owner reviews them;
• explicit descriptor activation from a strict private review file, with each custom index built beside the active per-organization version and switched atomically; prior index versions remain intact and later ingestion follows the active descriptor registry;
• versioned saved queries and dashboard definitions with organization-scoped panel references, optimistic revisions, server-generated identifiers, and a source-control export that omits tenant and operator runtime metadata;
• authorized CLI export and explicitly approved atomic import of those strict dashboard bundles, with all destination identities and resource scopes independently revalidated;
• a server-rendered Sandwich Hime interface for public orientation, local sign-in, organization selection, recent log/metric/trace/deployment tables, typed-text and assisted saved-query creation, one-panel dashboard creation, bounded dashboard execution, accessible visual summaries with full table alternatives, and strict source-control-safe JSON export;
• a bounded organization-authorized SSE invalidation channel that carries only generic refresh events; the interface, tables, navigation, and sign-out flow remain functional without JavaScript;
• durable organization-scoped alert rules that execute bounded saved-query ASTs, open incidents after configurable consecutive matches, and preserve pending, firing, acknowledged, silenced, and resolved lifecycle events;
• an accessible server-rendered incident inbox with ordinary CSRF-protected response forms and no telemetry or incident detail in SSE payloads;
• an installable PWA shell, application badge state, and an explicit opt-in read-only offline incident snapshot that excludes response capabilities and high-risk private fields;
• optional privacy-preserving Web Push with an explicit user gesture, organization-scoped subscriptions, delivery-time authorization, a bounded non-blocking queue, and one fixed generic encrypted message;
• content-addressed static assets and a strict CSP with no inline-style or inline-script exception; the production Sandwich Hime dependency boundary contains only the pinned Apache-2.0 Sando runtime, not the Hime-san compiler.
• a Tend schema-2 singleton deployment contract with a stateless, loopback-only candidate that cannot open or mutate Observatory data, followed by routed public-origin validation and automatic last-good restoration.

The server injects resource scope from an authenticated source. Payloads cannot select an organization or resource. Unknown fields remain stored in raw truth, but are not indexed by default.

Deliberate current limits

Self-service invitation delivery is deliberately excluded from the first preview, and the visual query builder remains deliberately smaller than the complete typed query language. Tend activation, rollback, and identical- artifact redeployment have been exercised against the live Observatory service. The separate medium-fleet agent soak and browser-vendor Web Push delivery evidence are not complete. No release tag should be created until the published preview gates in docs/ROADMAP.md are satisfied.

Development

./scripts/preview-gate.sh

Linux is the supported deployment platform. Development requires Go 1.26.6 with GOTOOLCHAIN=local.

The local first-operator procedure is documented in docs/BOOTSTRAP.md. It is a development surface, not yet a public-preview installation promise.

The typed text/builder query contract and authorized local workflow are documented in docs/QUERY.md. Descriptor discovery and its current review boundary are documented in docs/SCHEMA.md. Retention, approved per-organization overrides, quota enforcement, and metric rollup behavior are documented in docs/RETENTION.md. The bounded preview gate includes the complete verifier, short fuzz smoke, and a development capacity proof. Extended security, release-scale capacity, fleet, and soak campaigns remain explicit milestone evidence. See docs/CAPACITY.md. The retained optimization ledger and ordered lower-level storage case studies are documented in docs/PERFORMANCE.md. The proposed raw-first, lazy/eager time-partitioned read model is documented as an unimplemented design in docs/ADAPTIVE_PROJECTIONS.md. The persisted dashboard model and export boundary are documented in docs/DASHBOARDS.md. The current Sandwich Hime interface and progressive-enhancement boundary are documented in docs/INTERFACE.md. The bounded rule evaluator and durable response lifecycle are documented in docs/INCIDENTS.md. The installable shell and opt-in offline privacy boundary are documented in docs/PWA.md. The strict, non-authoritative Tend deployment-evidence boundary is documented in docs/TEND.md. The application deployment and isolated candidate boundary are documented in docs/DEPLOYMENT.md. Offline corruption recovery and projection rebuild are documented in docs/RECOVERY.md.

Licensing

Application, agent, query, storage, migration, and release code is AGPL-3.0-only. Copyable files under examples/ are 0BSD. See LICENSES.md.