Monitoring Stack (Prometheus & Grafana)

What this covers

How to deploy, configure, and extend the optional Prometheus + Grafana monitoring stack that provides real-time observability into every Tessallite service. This page explains the architecture, what data is collected, how the dashboard is organised, and how to add your own metrics and panels. It also explains the separate host collector for raw runtime logs.

Overview

Tessallite ships with an optional monitoring stack that runs as a completely separate Docker Compose project. It is not required to run the platform and can be started or stopped independently without affecting any Tessallite service.

The stack consists of three containers:

All three containers live in the monitoring/ directory at the workspace root, separate from the main tessallite/infra/ Docker Compose stack.

How it connects to Tessallite

Both stacks share a Docker network called tessallite_net. This allows Prometheus (running in the monitoring stack) to reach every Tessallite service by container name, even though they are managed by different Docker Compose projects.

The main stack creates the network automatically on docker compose up. The monitoring stack's deploy script also creates it if it does not exist, so either stack can be started first.

Deploying the monitoring stack

Prerequisites

• Docker and Docker Compose v2 installed
• The main Tessallite stack running (or at least one docker compose up to create the shared network)

Steps

1. Navigate to the monitoring directory: cd monitoring/
2. Create the environment file: cp .env.example .env
3. Set GRAFANA_ADMIN_PASSWORD in .env
4. Deploy: bash deploy.sh (Linux/macOS/Git Bash) or deploy.bat (Windows)
5. Open the dashboards:
   • Prometheus: http://127.0.0.1:9090
   • Grafana: http://127.0.0.1:3001 (username: admin, password: your .env value)

Teardown

bash teardown.sh              # stop and remove data
bash teardown.sh --keep-data  # stop but preserve volumes

What data is collected

Scraped services

Prometheus scrapes metrics from all seven Tessallite services:

Platform metrics

Model-level usage metrics

Aggregate refresh metrics

Raw runtime logs

The optional host collector reads Docker output for the selected Compose project and sends bounded, redacted batches through the existing system-admin API. It covers stdout and stderr for the eight configured services:

model-service, query-router, gateway, scheduler, optimizer, agent-service, frontend (nginx), and postgres.

This is a Docker Compose host process. It includes container startup and subprocess output, does not collect browser developer-console messages, and does not claim a Kubernetes collector. It uses the existing system-admin login and does not add an application Docker-socket mount or a new credential scheme. See the raw runtime log collector runbook for protected environment, Linux systemd and Windows Scheduled Task setup.

The standard Helm chart keeps systemLogs.enabled=false because it does not provide a Kubernetes collector; Docker Compose keeps the host collector enabled.

Raw records are stored in tess_system.system_logs and are visible only to system administrators. The UI provides server-side message/logger/instance search, service/severity/time filters, stable Load older paging, live five-second polling with pause, row details and a confirmed expired-record purge.

The raw-log metrics have bounded labels and expose accepted events by service and level, write failures, truncated reads, collector heartbeat age, enablement, latest error timestamp and purged rows. The scheduler independently measures relation bytes and estimated rows every 60 seconds and after each hourly purge; the storage panel and alert use that scheduler target. A stale or missing measurement is shown as no current value. No message, user identity or event UUID is a metric label.

Dashboard sections

The Grafana dashboard is organised into three collapsible sections with four filter variables at the top: Service, Tenant, Project, and Model. Three raw runtime-log panels sit alongside those sections.

Service Health (7 panels)

Live service status tiles, uptime over time, scrape duration, per-service request rate, error rate (5xx), and latency percentiles (p95 and p50).

Query Routing and Aggregates (4 panels)

Query routing distribution (source/aggregate/pocket), HTTP error rate by service, refresh run duration, and refresh completion rate.

Model Health and Usage (7 panels)

Per-model query throughput, latency p95, protocol distribution (SQL vs DAX), route distribution, query errors by type, bytes processed, and rows returned.

Raw runtime logs

Adding custom metrics

All Tessallite metrics are defined in tessallite/shared/metrics.py using the Python prometheus-client library.

1. Define the metric in shared/metrics.py using Counter, Histogram, or Gauge.
2. Import and instrument in the relevant service code (e.g., from shared.metrics import MY_COUNTER).
3. Rebuild the service: docker compose build <service> && docker compose up -d <service>

The new metric appears automatically on /metrics. Prometheus begins scraping it on the next 15-second cycle. No Prometheus configuration changes are needed.

Adding a Grafana panel

1. Open Grafana and navigate to the Tessallite Platform Overview dashboard.
2. Click Edit, then Add panel.
3. Write a PromQL query referencing your metric.
4. Save the dashboard.

To make the panel permanent, export the dashboard JSON and save it to monitoring/grafana/tessallite-dashboard.json.

Adding a new scrape target

1. Add PrometheusMiddleware and a /metrics endpoint to the new service.
2. Add a scrape job to monitoring/prometheus.yml.
3. Restart Prometheus: cd monitoring/ && docker compose restart prometheus

Data retention

Prometheus retains time-series data for 15 days by default. To change this, edit the --storage.tsdb.retention.time argument in monitoring/docker-compose.yml and restart.

Raw runtime log retention

SYSTEM_LOG_RETENTION_DAYS is the application .env setting. It defaults to 30 days; 0 disables expiry. The scheduler runs the shared purge hourly in 5,000-row batches, up to 100,000 rows per run. System Admin → System logs → Purge expired uses the same cutoff, removes only rows older than it, shows the cutoff and asks for confirmation. Deleting rows lets PostgreSQL reuse space; it does not promise an immediate reduction in database file size.

Only tess_system.system_logs is covered. tess_system.system_audit_events, tenant audit_events and tenant query_logs have separate meanings and are not removed by this operation.

The raw-log alert rules cover a recent error, storage above 512 MiB for ten minutes, a stale or missing scheduler storage measurement (even when collection is disabled), a missing collector heartbeat and truncated or failed collection. They are Prometheus rules only; this repository has no Alertmanager route or contact point, so a firing rule does not prove external notification delivery.

Frequently asked questions

Do I need the monitoring stack to run Tessallite?
No. It is entirely optional. Tessallite operates normally without it.

Will stopping the monitoring stack affect Tessallite?
No. The monitoring containers are independent. Stopping them has zero impact on the platform.

Where is monitoring data stored?
In Docker volumes: monitoring_prometheus_data and monitoring_grafana_data. Use teardown.sh --keep-data to preserve them.

Can I use this in production?
Yes, for small to medium deployments. For high availability, consider a managed Prometheus service and point it at the same /metrics endpoints.

Related

• Architecture Overview
• Service Reference
• Deploy Locally
• System Configuration