Designing a Scalable Database Observability Platform

Database observability becomes difficult long before the first dashboard looks crowded. The real pressure appears when one platform must discover thousands of changing database resources, ingest telemetry without becoming a noisy neighbor, preserve tenant boundaries, and still answer operational questions during an incident.

This article develops a reference design for that problem. PostgreSQL is the working example, but the boundaries apply to most managed database fleets. The goal is not to prescribe one vendor stack. It is to show how requirements should drive collection, transport, storage, querying, alerting, and recovery decisions.

Start with outcomes, not telemetry

A platform should begin with the decisions it needs to support. A database operator might ask:

• Which databases are unavailable or approaching resource limits?
• Did a topology change precede a latency regression?
• Is a tenant exceeding its agreed telemetry budget?
• Can an incident responder reconstruct the state of a cluster at a specific time?
• Are gaps caused by the monitored database, the collector, transport, or storage layer?

These questions lead to functional requirements: topology discovery, metrics collection, event correlation, tenant-aware querying, alert evaluation, and auditability. They also expose non-functional requirements that often dominate the architecture:

Concern	Design question
Freshness	How old may a signal be before it is operationally misleading?
Completeness	How are missing intervals detected and explained?
Isolation	Can one tenant exhaust shared ingestion, query, or storage capacity?
Durability	Which telemetry can be replayed, and for how long?
Cost	Which dimensions, resolutions, and retention periods are economically justified?
Security	Where are credentials, tenant identifiers, and sensitive query data allowed to exist?

An architecture review should attach an owner and a measurable indicator to each critical requirement. “Highly scalable” is not a requirement. “Sustain the forecast peak sample rate while keeping ingestion lag within an agreed freshness window” is testable, even when the numerical target is still awaiting capacity measurements.

Separate the control plane from the data plane

The control plane knows what should be observed. It integrates with cloud APIs, Kubernetes, service registries, or a platform inventory to discover database instances and their topology. It owns configuration, credential references, tenant policy, collector assignment, and desired scrape state.

The data plane performs the repetitive work: collect, validate, enrich, buffer, transport, store, query, and evaluate telemetry. Keeping these responsibilities separate limits the blast radius of discovery failures and allows the high-throughput path to scale independently.

This is a design proposal, not a mandatory component list. A smaller estate may combine several boxes into one deployable unit. The boundaries remain useful because they identify ownership and failure propagation.

Make topology a first-class signal

Static target lists fail in dynamic fleets. A managed PostgreSQL service may promote replicas, replace failed nodes, rotate endpoints, or move workloads between regions. Metrics without topology context can produce technically correct but operationally false conclusions.

Discovery should maintain a versioned resource graph: organization, tenant, service, cluster, instance, role, region, and lifecycle state. Each collector assignment should have a revision and lease. A collector reports both collected samples and its current inventory revision, allowing the platform to distinguish “database stopped emitting” from “target was intentionally removed.”

Topology events deserve their own durable stream. Promotion, failover, resize, maintenance, and deletion events can then be correlated with metric changes without encoding rapidly changing state into every metric label.

Design multi-tenancy at every boundary

Adding tenant_id to a table does not create isolation. The tenant context must survive authentication, assignment, collection, transport, storage, caching, queries, alerts, and audit logs.

A practical policy has four layers:

1. Identity: derive tenant context from authenticated control-plane data, never from an untrusted sample label.
2. Admission: enforce per-tenant limits for targets, series cardinality, sample rate, payload size, and query concurrency.
3. Storage: include the tenant in partitioning and authorization boundaries, with no cross-tenant query path available to normal callers.
4. Fairness: schedule ingestion and queries so one burst cannot starve other tenants.

Per-tenant quotas should produce explicit rejection or throttling telemetry. Silent dropping converts a capacity problem into a misleading dashboard.

Build an ingestion contract

Collectors should emit a stable envelope around samples: tenant identity, resource identity, collection timestamp, observed timestamp, schema version, collector identity, and topology revision. The normalization stage validates that envelope, applies controlled labels, rejects malformed or excessive dimensions, and records why data was rejected.

OpenTelemetry database conventions provide a useful interoperability vocabulary for database telemetry, but a platform still needs its own resource and tenancy contract 1. Treat semantic-convention versions as schema dependencies and roll them out deliberately.

At-least-once transport is usually the practical choice. Consumers must therefore be idempotent. A deterministic key derived from tenant, resource, series, and timestamp can help detect duplicates, but the exact strategy depends on the storage engine and whether late corrections are allowed.

Use Kafka as a pressure boundary, not a cure

A durable log such as Kafka can decouple collection from storage and absorb finite bursts. Partitioning by a stable tenant-resource key preserves ordering for a series while spreading work across consumers. It also creates replay and recovery options.

Kafka does not remove overload. If sustained input exceeds sustainable processing, lag grows until retention, disk, or recovery-time limits are reached. The platform needs:

• Bounded producer and collector buffers
• Per-tenant admission limits before the shared log
• Lag, age-of-oldest-record, rejection, and retry indicators
• Exponential backoff with jitter for transient failures
• A dead-letter policy for structurally invalid records, not temporary outages
• Load shedding rules that protect critical signals before optional high-cardinality data

Prometheus remote write provides a concrete example of queue sharding, backoff, and the memory tradeoffs of larger buffers 2. Its published protocol also defines ordering, retry behavior, and the expectation that senders may persistently buffer during receiver outages 3. Those are established protocol behaviors; copying its exact queue settings into another architecture would still require workload testing.

Prometheus and ClickHouse solve different shapes of work

Prometheus is strong when operators need a familiar metric model, PromQL, local alert evaluation, and a focused operational window. Its local TSDB is intentionally single-node; remote interfaces connect it to longer-term systems 4. This makes it a good hot operational layer when the cardinality and retention envelope is controlled.

ClickHouse is a column-oriented analytical database. It can be attractive when the workload needs large scans, high-cardinality event analysis, longer retention, or correlation across metrics, events, and traces. It also asks the platform team to own table design, partitioning, materialized views, deduplication behavior, query governance, and tenant enforcement.

A defensible hybrid is:

• Prometheus-compatible storage and query semantics for recent operational metrics and alerting
• An analytical store for longer-horizon investigation, topology events, aggregates, and high-cardinality exploration

The cost is duplicated pipelines and consistency questions. If operators cannot explain which store answers which question, the hybrid has added complexity without creating a useful boundary. Start with workload classes and query patterns, then choose the smallest storage design that satisfies them.

Scale horizontally around stable ownership

Collectors can scale horizontally when assignments are explicit and leased. Ingestion consumers can scale by transport partition. Query services can remain stateless when authorization, limits, and caches are externalized. Alert evaluators require stable rule ownership to avoid duplicate notifications during rebalancing.

Useful shard keys distribute load and preserve the ordering that downstream logic actually needs. Tenant-only partitioning can create hot partitions; random partitioning can destroy per-series order. A composite hash of tenant and resource is often a better starting proposal, but skew must be measured using real inventory and sample-rate distributions.

Autoscaling should consider lag and work age, not CPU alone. A consumer at low CPU may still be blocked on storage; a collector may be CPU-idle while throttled by database connection limits.

Treat retention as a product and cost policy

Retention is not one number. Raw high-resolution samples, downsampled aggregates, topology events, alert history, and audit records have different operational value and compliance implications.

A tiered proposal might keep recent raw data for incident response, retain coarser aggregates for trend analysis, and preserve topology and audit events according to governance needs. Each tier should define:

• Resolution and time range
• Expected query patterns
• Deletion semantics and verification
• Storage and retrieval cost
• Tenant-specific overrides, if allowed

Capacity planning starts with series count, samples per series, bytes per sample after realistic compression, replication, indexes, compaction overhead, and safety margin. Measure these values in a representative load test; do not rely on a universal bytes-per-sample constant.

Alert on user-relevant failure and telemetry integrity

Database alerts should connect to an actionable condition: availability, saturation, replication health, backup validity, or an agreed service objective. Google’s SRE guidance recommends a small number of user-centered indicators and iterative SLO refinement 5. The same discipline helps prevent an observability platform from becoming an alert generator rather than a decision system.

The platform must also monitor itself. Collection coverage, ingestion lag, rejected samples, stale topology revisions, query errors, rule-evaluation delay, and notification delivery are part of the product. A green database dashboard is not trustworthy when its collector has been disconnected for an hour.

Security boundaries belong in the architecture

Collectors should use least-privilege database roles and retrieve short-lived credentials or references from an approved secret system. Avoid collecting raw query text by default; it may contain personal or confidential data. If normalized query fingerprints are needed, define the normalization and redaction contract explicitly.

Encrypt traffic, authenticate every service hop, validate tenant context at the query boundary, and log administrative configuration changes. Separate control-plane authority from data-plane write access. Threat-model label injection, cardinality attacks, replay, cross-tenant cache keys, malicious dashboard queries, and compromised collectors.

Plan recovery before adding scale

Failure recovery should answer what happens when each dependency is unavailable:

Failure	Desired behavior
Database unreachable	Record scrape failure without retry storms or connection exhaustion
Collector lost	Lease expires; another collector takes ownership after a bounded delay
Transport unavailable	Buffer within a hard limit, then shed by declared priority
Storage slow	Apply backpressure, preserve critical signals, expose growing lag
Normalizer bug	Pause affected consumers and replay from a known offset after correction
Region lost	Continue regional isolation or fail over according to an explicit recovery objective

Replay tests, restore drills, and overload exercises are more valuable than a diagram that merely contains replicated components. Recovery time is constrained by backlog size and sustainable drain rate, so normal operation must reserve enough headroom to catch up after an outage.

A capacity review checklist

Before implementation, measure or estimate ranges for:

• Tenants, clusters, instances, and churn per day
• Active series and label-cardinality distributions
• Average and peak samples per second
• Burst duration and regional correlation
• Ingestion and query service times
• Retention by resolution and replication factor
• Alert rule count, evaluation interval, and fan-out
• Acceptable telemetry loss, freshness, and recovery windows

Then test the failure cases, not only the happy-path peak. Disconnect storage, slow a partition, create a cardinality spike, rotate topology during a scrape, and exhaust one tenant’s quota. The platform is scalable only when its overload behavior is understandable.

Closing perspective

A scalable database observability platform is a controlled feedback system. Discovery tells it what should exist; collectors report what they can observe; transport creates a finite recovery window; storage supports distinct operational and analytical questions; and policy keeps shared resources fair and secure.

The central tradeoff is not Prometheus versus ClickHouse or Kafka versus direct writes. It is how much complexity to accept in exchange for a measured requirement. Start with explicit decisions and failure budgets, keep boundaries observable, and add distributed machinery only when evidence justifies it.