Skip to content

Sources and FDW

A source adapter describes what QuickLookup is allowed to execute. A source binding applies that adapter—or native storage—to a service with narrower limits. Service definitions refer to bindings, not directly to database relations.

Resolving a place before you query

Several CTX relations exist to answer where is this rather than what happened. They are lookups, so they never back a series binding, and the API surface that uses them is POST /v1/resolve rather than a service query.

Relation Answers Reach
geocode free text → a ranked list of places, each with its OSM reference, extent and precision the OpenStreetMap extract we host
reverse_geocode any of eleven location inputs → a point and the address on it as above, plus the global resolvers below
country_by_name a country's name, ISO code or capital → the country and its capital point every country, in many languages
ip_location, phone_resolve, email_resolve, url_resolve, whois_resolve an identifier → a country worldwide
coord_parse, geohash_encode_tbl, geohash_decode_tbl a coordinate form → a point worldwide, computed

Two things follow from that table and are worth stating plainly:

  • Only the OSM-backed relations are limited. Everything that resolves to a country answers for the whole world. country_by_name exists so that free text naming a country we hold no map for still resolves, at country precision, instead of returning nothing.
  • Precision is part of the answer, not an assumption. geocode and reverse_geocode return precision derived from the gazetteer's own addresstype, and the country-level resolvers return country. A consumer that ignores it will pin a phone number on a street.

An outline is available from geocode and reverse_geocode through the with_geometry input. It is opt-in: a bounded query selects every column an adapter maps, so inferring the request from column selection would make every lookup fetch a municipality's polygon.

Choose a source mode

Requirement Mode Data path
Store observations in QuickLookup stored ClickHouse cst.data_points
Store events in QuickLookup stored ClickHouse ql.calendar_events
Query an approved CTX source on_demand Allowlisted PostgreSQL FDW relation
Query another approved provider on_demand Registered HTTP, library, or ClickHouse adapter
Introduce a source that is not registered Source definition review GitOps change, deployment, sync, then binding

Configure native storage

Create a source binding with mode=stored and:

  • no adapter;
  • fixed parameter mode;
  • a positive maximum interval;
  • a maximum row count no greater than 100,000;
  • no cache requirement or credential value.

Stored bindings are valid only for time series and event feeds. Lookups are on-demand services.

Use a CTX FDW adapter

  1. Open Sources → Source adapters.
  2. Select Sync from CTX if the inventory is stale. Sync reads live foreign tables, columns, grants, and ClickHouse bridge state; it does not change CTX relations.
  3. Filter by result grain, provider, availability, or health.
  4. Open the adapter and inspect Contract, Live columns, Limits, and Sync notes.
  5. Run Query this relation with bounded inputs. This diagnostic is read-only.
  6. Select Create source binding. Keep mode=on_demand and choose fixed or caller-supplied parameters as permitted by the adapter.
  7. Narrow the adapter's interval, row, or cache limits if the service needs stricter bounds. A binding cannot widen them.

The published service contract decides which inputs reach the adapter and which fields can be returned. Validation runs before the FDW query.

Fixed and caller-supplied inputs

Use fixed parameters when a service represents one configured slice of a source. Use caller-supplied parameters only when the published contract should expose those inputs. Required alternative groups—such as coordinates or place name—must come from the adapter's contract.

Never put a password, token, connection string, or other secret value in a source definition, binding, or contract. A binding may contain only a runtime secret reference.

Register a new source

Only an internal operator can create a source definition. The workflow is:

  1. Describe the upstream type and bounded query contract.
  2. Save to run schema and secret-material validation.
  3. Submit the generated change for GitOps review.
  4. Deploy the approved CTX/adapter configuration.
  5. Run Sync from CTX.
  6. Test the registered adapter before using it in a binding.

Use a declarative definition for supported ClickHouse, HTTP, and library sources. Add code only when the resolver cannot be represented by the declarative contract.

Failure and cache behavior

Fetched rows have one home, ClickHouse; the catalog master database keeps no result data. Every fetched interval is recorded next to the data (ql.source_coverage) with its fetch time and row count, and every delivery read goes through CTX PostgreSQL.

  • Contract errors return 422 without executing the source.
  • A window is served from storage (source=clickhouse, reused=true) while its coverage is valid: fetched within the source's freshness period, or fetched when the data was already older than the source's settle horizon (settled history is never fetched twice), or produced by a source that declares no freshness period at all. An interval the source answered with no rows never settles: it is re-asked once its period lapses.
  • Only the gaps of a window are fetched (source=fdw), each gap on its own and chunked by the source's interval cap; every completed chunk is stored and covered before the next starts. A chunk failure after partial progress returns 503 naming the covered and the missing sub-intervals; the next request fetches only what is still missing.
  • Identical concurrent requests make one source call (a transaction-scoped lock, no stored record); the others wait and read the stored result.
  • An unavailable or recently failed source is not called for its gaps: the valid stored intervals are returned with source=clickhouse_stale and the missing sub-intervals named in coverage.missing; 503 only when nothing is stored. The source is retried once per health grace period.
  • A refusal about the request itself — a window past where a feed stops, an input the source does not hold — is not a failure of the source: it is audited, it leaves the source's health alone, and its 503 names no retryAt, because asking again later changes nothing.
  • If the ClickHouse write fails after a successful fetch, the caller still receives the rows and the failure is counted; nothing is covered, so the window is fetched again next time.
  • Lookups are cached once, at the source layer inside CTX (ctx.resolver_cache), for the freshness period each relation declares; source=cache marks a fresh cached answer, source=cache_stale one served past its period because the source failed. The cache fails open: when it is unreachable the source is asked directly.
  • Stored services do not fall back to an FDW source.
  • Source health and coverage are visible on the service detail page and in query responses.