Skip to content

Operate QuickLookup

Production requires PostgreSQL, ClickHouse, the API/admin process, and the publication worker. On-demand services also require the CTX PostgreSQL/FDW path and their upstream providers.

Readiness and monitoring

  • /healthz confirms the process is running.
  • /readyz confirms the required PostgreSQL connection can answer.
  • /metrics exposes request, publication, source, cache, and ingestion metrics for the monitoring system.
  • The admin dashboard shows publication backlog and source health for operators.

Readiness does not prove that ClickHouse, every FDW relation, or every upstream provider is healthy. Monitor those dependencies and run a bounded representative query for each critical service family.

Publication worker

Run at least one continuously supervised worker:

uv run python manage.py run_publication_worker

Alert on old pending records, repeated failures, and a growing backlog. A failed delivery does not remove the last delivered revision. Correct the cause and retry the accepted publication; do not create a replacement revision solely to restart delivery.

Event delivery also needs its configured semantic-event transport. Without it, event publication records fail with event_delivery_unconfigured; stored events can remain in ClickHouse, but downstream semantic delivery is not healthy.

Back up and restore

PostgreSQL is the catalogue authority: back up definitions, grants, revisions, publications, audit data, and account mappings. ClickHouse holds native rows, event rows, and analytical projections. Back up both systems to recover a usable service.

Restore in this order:

  1. PostgreSQL and its schema migrations.
  2. ClickHouse databases and tables.
  3. API/admin and worker configuration, including secrets and provider endpoints.
  4. Publication worker processing.
  5. Health checks, catalogue counts, grants, publication backlog, and bounded service queries using representative tokens.

Do not assume an intact ClickHouse volume is sufficient: rows are discoverable only when their PostgreSQL definitions, delivered revisions, and grants also exist.

Source and schema changes

Source definitions and Kubernetes configuration belong in Git and follow the reviewed GitOps path. After a CTX deployment, use Sync from CTX to refresh adapter metadata, then verify live columns, grants, source health, a preview, and one unified service query.

Never repair production by editing FDW tables, catalogue rows, or Kubernetes resources by hand. Make the source or manifest change through its owned workflow so the next deployment does not reverse the repair.