Metrics and alerts

Lolly Work exports Prometheus metrics on GET /metrics. This page lists every metric, the alerts the Helm chart can install and the Grafana dashboard that comes with them. The alerts and the dashboard are for a platform team that already runs Prometheus and Grafana, and both are off by default. Without them, Work still raises its own console alerts for blob offload and credential expiry (keep Lolly Work running).

Scraping

/metrics needs a bearer token, LW_METRICS_TOKEN, from anywhere except an evaluation instance's own loopback address:

• with the token set, every request must present it (Authorization: Bearer ...), and a wrong token gets 401;
• without the token, an evaluation instance answers on loopback only, and a production instance, one behind reverse-proxy sign-in or one with rateLimit.trustedProxyHops above 0 answers 404 to everyone.

With the Helm chart, set secrets.metricsToken (or an LW_METRICS_TOKEN key in your existingSecret) and turn on the ServiceMonitor. It reads the token from the same Secret:

metrics:
  serviceMonitor:
    enabled: true
    interval: 30s
    labels:
      release: kube-prometheus-stack   # what your Prometheus's serviceMonitorSelector matches

Your own scrape job works as well. Give it the bearer token and see selecting the series for the alerts.

Metric reference

Counters count from process start, so a restart sets them back to zero; rate() and increase() allow for that. A counter, and each of its label combinations, appears with its first event, so Prometheus can measure an increase only from the second event on. Gauges are read when Prometheus scrapes.

Requests and the process

Audit and SIEM

Files

Catalog sources

One series per source and metric.

Notifications, flags and OpenTelemetry

The render worker exports no metrics. Its health shows in its readiness probe and in the status of server renders, the /api/v1/render and /render/:spec routes of lw_http_requests_total.

Suggested alerts

The chart can install twelve alerts as a Prometheus Operator PrometheusRule, next to the ServiceMonitor:

metrics:
  serviceMonitor:
    enabled: true
  prometheusRule:
    enabled: true
    labels:
      release: kube-prometheus-stack   # what your Prometheus's ruleSelector matches

Each alert carries a severity label, a summary and a description that says what to check first. Some limits to know:

• LollyWorkRenderFailures cannot tell a failing render worker (502), a busy one (503 RENDER_BUSY) and a hooked tool on an instance without a worker (501) apart, because the metric records only the status class. The error.code of the response and the worker's log can.
• The two offload alerts fire at 90% of the thresholds at which Work raises its own console alert. The read alert uses the last day's rate rather than a 30-day increase, because Prometheus often keeps less than 30 days, so it needs a day of samples before it can fire.
• LollyWorkRateLimitBucketsHigh reads rateLimit.maxBuckets from the chart's config (inline or --set-file), or uses Work's default of 50,000.
• No alert ships for lw_ofrep_evaluation_errors_total or lw_otlp_dropped_total. If you use your own flag service or OpenTelemetry export, alert on their rate() staying above zero.

Alert options

Selecting the series

Every expression selects this release's series with job and namespace. By default they are the labels the chart's ServiceMonitor produces: job is the release's Service name (lolly-work for a release called lolly-work) and namespace is the release namespace. If your own scrape job collects Work's metrics, set metrics.prometheusRule.seriesSelector to matchers for its series, for example job="lolly-work-prod".

Checking the rules

The chart's tests render the rules and run promtool check rules on them, along with the unit tests in deploy/helm/files/prometheus-rules.test.yaml, which cover every alert. To check a rendered rule yourself, take spec.groups from the rendered PrometheusRule into a file of its own and run promtool check rules on that file.

Dashboard

The chart can also install a Grafana dashboard as a ConfigMap:

metrics:
  grafanaDashboard:
    enabled: true

The ConfigMap carries the label grafana_dashboard: "1", which the Grafana sidecar of kube-prometheus-stack watches by default. Change metrics.grafanaDashboard.labels for another sidecar, set metrics.grafanaDashboard.namespace if the sidecar watches one namespace only, and use metrics.grafanaDashboard.annotations for a folder (grafana_folder: Lolly). To import by hand, use deploy/helm/dashboards/lolly-work.json.

The dashboard has three variables, the Prometheus data source, the namespace and the job, and five rows:

• Health: the audit chain, the share of readiness checks passed, uptime, memory, the SIEM backlog and the rate limiter's addresses;
• Requests: requests by status class, server errors by route, the busiest routes and rate-limited requests by surface;
• Renders: server renders by status class and the share that failed;
• Files and delivery: files stored and read by blob driver, response bytes by delivery class and configuration polls from Lolly apps;
• Integrations: a table of catalog sources and notification outcomes.

Qualification

On 2026-10-27, Work 1.1.2 ran on K3s 1.35 with kube-prometheus-stack 92.14.0 and Grafana 13.2.4. The ServiceMonitor scrape was up with the bearer token, Prometheus loaded all twelve rules with no errors, every metric the rules read had a live series (the failure-only series aside, which have their paired success series), and the sidecar loaded the dashboard with data in each row.