Data sources and database

The tools never call an upstream system directly. They read marts through the app's data service, which picks the source, applies the source policy, syncs upstream snapshots into the app's own database and falls back to synthetic data when a source is excluded, unconfigured or still syncing.

SourcesCopy Anchor Link

Every figure carries provenance.source and provenance.basis. The wording follows the basis: asking-price figures read Asking prices from public listings, as of YYYY-MM-DD, never "sales", "transactions" or "DLD". Tools that need history (area_price_trend) or off-plan counts refuse a snapshot source rather than guess.

Source policyCopy Anchor Link

SOURCE_POLICY is one switch for every surface (chat, REST, MCP):

• allow-all (default)
• exclude-listings: never serve the listings source; fall back to the default data
• exclude:<id>,<id>: exclude sources by id

GET /api/status shows which data is being served, whether a sync is running, and which sources the policy excludes.

A superuser can also set the source, the source policy and every listings setting below in Admin › Data; a value set there overrides its environment variable until "Use environment default" is chosen. The listings token can be stored there only when SETTINGS_ENCRYPTION_KEY is set, and LISTINGS_API_TOKEN from the environment is used only while LISTINGS_API_URL also comes from the environment: pointing the URL elsewhere in the panel requires entering the token for that address. Changes to this section need a login from the last 15 minutes.

Listings APICopy Anchor Link

Set, in next/.env.local:

DATA_SOURCE=listings-api
LISTINGS_API_URL=...    # base URL ending in /v1
LISTINGS_API_TOKEN=...  # Bearer token; never commit it

On the first request the app starts a background sync (about three requests per area, paced 250 ms apart by default) and serves the default data until it finishes; afterwards it serves the stored snapshot and refreshes it every LISTINGS_SYNC_TTL_HOURS (default 24). The contract (endpoints, parameters, response fields) and the mapping into the marts are documented in docs/upstream-api.md in the repository; any service that implements it can be configured. The configured provider's terms apply, and this repository ships no listings data, provider URL or credentials.

A local mock with invented values runs with pnpm --filter @alghafat-market/cli mock-listings-api and serves http://127.0.0.1:3071/v1 (token dev-token).

The app's databaseCopy Anchor Link

The app has its own database:

• Locally: an embedded Postgres (PGlite) under .data/pglite. Nothing to install; created and migrated on first use.
• Hosted: any Postgres via DATABASE_URL=postgres://... (for example Neon). Same schema and migrations (Drizzle ORM).

It stores:

Watches, shortlists and briefs belong either to an account or, for guests, to a random id kept in the browser; when a guest logs in, that browser's data moves into the account. Guest chats are never stored. "Delete my data" removes what belongs to the account (or the browser), and deleting an account removes everything stored with it. See Accounts and admin for retention and the hourly clean-up.

The local database is single-process: while the web app runs, use the app itself (it syncs and imports on its own); stop it before running CLI commands that open the database:

pnpm --filter @alghafat-market/cli listings-sync
pnpm --filter @alghafat-market/cli db-status
pnpm --filter @alghafat-market/cli superuser --email you@example.com

With DATABASE_URL set, the CLI uses that database and can run while the app is up. The superuser command reads next/.env.local and next/.env (without overriding variables already set), so it targets the same database as the web app, and prints the target before it writes.