Building a Self-Hosted Time-Series Database and Observability Stack with Prometheus, Thanos, and Grafana

Quick Summary (TL;DR)

• Prometheus handles scraping and short-term storage (15–30 days); Thanos extends it with unlimited retention in object storage, global querying, and downsampling.
• Deploy a Thanos sidecar next to every Prometheus instance to upload blocks to S3-compatible storage (e.g., MinIO).
• Run Store Gateway, Query, Query Frontend, and a single Compactor per bucket for a production-grade setup.
• Use Prometheus federation for hierarchical scaling across clusters, and point Grafana at the Thanos Query endpoint as a Prometheus data source.
• Watch out for cardinality explosions — they will break Prometheus long before storage becomes the bottleneck.

Why Self-Host a Time-Series Database and Observability Stack?

Managed observability platforms charge by ingested series, retained samples, and query volume. For teams running Kubernetes clusters with tens of thousands of active series, those bills scale faster than the infrastructure itself. Self-hosting Prometheus, Thanos, and Grafana gives you fixed costs, full data ownership, and control over retention — critical when compliance requires metrics to stay on-premises.

The trade-off is operational responsibility: you must design storage, manage compaction, and scale the query layer. This guide walks through a production-ready architecture that handles hundreds of millions of samples per day without heroics.

Architecture Overview: Prometheus, Thanos, and Grafana

The stack has three logical layers: collection, long-term storage/query, and visualization. Each Prometheus instance writes local blocks and ships them via a sidecar to object storage. Thanos components then read those blocks and expose a unified PromQL endpoint.

flowchart TD
    subgraph Collection
        P1[Prometheus A + Sidecar]
        P2[Prometheus B + Sidecar]
    end
    subgraph Object Storage
        S3[(S3 / MinIO Bucket)]
    end
    subgraph Thanos Layer
        SG[Store Gateway]
        Q[Query]
        QF[Query Frontend]
        C[Compactor]
        R[Ruler]
    end
    G[Grafana]
    P1 -->|upload blocks| S3
    P2 -->|upload blocks| S3
    S3 --> SG
    S3 --> C
    SG --> Q
    QF --> Q
    Q --> G
    R --> S3
    P1 -.->|StoreAPI| Q
    P2 -.->|StoreAPI| Q

Prometheus: The Metrics Collection Engine

Prometheus scrapes targets over HTTP, stores samples in a local TSDB organized into 2-hour blocks, and evaluates alerting/recording rules. It is not designed for long retention — beyond ~30 days, query performance degrades and disk usage balloons.

Thanos: Long-Term Storage and Global Querying

Thanos adds four capabilities Prometheus lacks:

Capability	Component	Benefit
Block upload to object storage	Sidecar	Infinite retention at object-storage cost
Global query across clusters	Query	Single PromQL endpoint for all Prometheus instances
Historical query acceleration	Compactor (downsampling)	5m/1h resolution for long ranges
HA deduplication	Query	Merges duplicate series from replica pairs

Grafana: Visualization and Dashboarding

Grafana treats Thanos Query as a Prometheus data source. All standard PromQL, alerting, and template variables work unchanged — you simply get access to years of data instead of weeks.

Designing Your Time-Series Database for Scale

Estimating Storage Requirements

Use this formula as a starting point:

Daily bytes ≈ active_series × samples_per_day × bytes_per_sample × replication_factor

For 1M active series at 15s scrape interval (5,760 samples/day) and ~1.5 bytes/sample compressed:

1,000,000 × 5,760 × 1.5 ≈ 8.6 GB/day  →  ~3.1 TB/year

Add 20–30% headroom for index overhead and WAL. Thanos downsampling reduces query cost but increases storage by ~5–10% (it stores additional 5m and 1h blocks).

Choosing an Object Storage Backend

Backend	Best For	Notes
AWS S3	Cloud-native, high durability	Native Thanos support, best performance
MinIO	Self-hosted, on-prem	S3-compatible, easy to operate, erasure coding
Ceph RGW	Existing Ceph clusters	S3-compatible, mature
GCS / Azure Blob	Multi-cloud	Supported via objstore config

For fully self-hosted deployments, MinIO is the pragmatic default. Run at least 4 nodes with erasure coding for durability.

Deploying Prometheus with Thanos Sidecar

Prometheus Configuration for Thanos Integration

The sidecar requires Prometheus to expose the StoreAPI and enable the remote-read receiver only if you’re using --receive. The critical flags:

# prometheus.yml
global:
  scrape_interval: 15s
  external_labels:
    cluster: prod-eu
    replica: A   # set B on the second replica

storage:
  tsdb:
    retention.time: 24h   # keep local short; Thanos holds the rest

Launch flags:

prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/prometheus \
  --storage.tsdb.min-block-duration=2h \
  --storage.tsdb.max-block-duration=2h \
  --web.enable-lifecycle

Pinning min and max block duration to 2h is mandatory — it prevents Prometheus from compacting blocks beyond what the sidecar can upload.

Thanos Sidecar Setup

thanos sidecar \
  --tsdb.path=/prometheus \
  --prometheus.url=http://localhost:9090 \
  --objstore.config-file=/etc/thanos/bucket.yml \
  --grpc-address=0.0.0.0:10901 \
  --http-address=0.0.0.0:10902

bucket.yml:

type: S3
config:
  bucket: thanos-metrics
  endpoint: minio.internal:9000
  access_key: thanos
  secret_key: <secret>
  insecure: true

The sidecar uploads completed 2h blocks within minutes. It also serves the StoreAPI, letting Thanos Query read recent data directly from Prometheus without waiting for uploads.

Configuring Thanos Components: Store, Query, Compactor, and More

Thanos Store Gateway

Store Gateway exposes object-storage blocks via the StoreAPI. Its performance depends on caching the index locally:

thanos store \
  --data-dir=/var/thanos/store \
  --objstore.config-file=/etc/thanos/bucket.yml \
  --index-cache-size=2GB \
  --grpc-address=0.0.0.0:10901

Run at least two replicas behind a headless service so Query can load-balance.

Thanos Query and Query Frontend

Query fans out PromQL to all StoreAPI endpoints and deduplicates HA pairs. Query Frontend sits in front for query splitting, caching, and retries:

thanos query \
  --http-address=0.0.0.0:9090 \
  --store=dnssrv+_grpc._tcp.thanos-store.monitoring.svc \
  --store=dnssrv+_grpc._tcp.prometheus.monitoring.svc \
  --query.replica-label=replica \
  --query.replica-label=prometheus_replica

The --query.replica-label flag is essential: without it, HA replicas produce duplicate series and inflated counts.

Thanos Compactor

The Compactor is stateful and must run as a single instance per bucket — running two will corrupt data. It performs three jobs:

1. Compact 2h blocks into larger blocks.
2. Create 5m and 1h downsampled versions.
3. Delete blocks beyond retention.

thanos compact \
  --data-dir=/var/thanos/compact \
  --objstore.config-file=/etc/thanos/bucket.yml \
  --retention.resolution-raw=180d \
  --retention.resolution-5m=1y \
  --retention.resolution-1h=3y \
  --wait

Use a Kubernetes StatefulSet with replicas: 1 and a PodDisruptionBudget to prevent split-brain during upgrades.

Thanos Ruler (Optional)

Ruler evaluates recording and alerting rules against the global view — useful when rules must aggregate across clusters:

thanos rule \
  --data-dir=/var/thanos/rule \
  --rule-file=/etc/thanos/rules/*.yaml \
  --query=dnssrv+_http._tcp.thanos-query.monitoring.svc \
  --alertmanagers.url=http://alertmanager:9093

Prometheus Federation for Hierarchical Scaling

Federation lets a central Prometheus scrape aggregated metrics from leaf Prometheus instances. Use it when:

• Individual clusters exceed 5–10M active series.
• You want a “global view” with lower resolution and lower cost.
• Cross-cluster SLOs need pre-aggregated data.

scrape_configs:
  - job_name: 'federate'
    scrape_interval: 60s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{__name__=~"job:.*"}'   # only recording rules
    static_configs:
      - targets: ['prometheus-eu:9090', 'prometheus-us:9090']

Federation is not a substitute for Thanos — it only transfers pre-aggregated series. Combine both: Thanos for raw long-term data, federation for cross-cluster rollups.

Grafana Dashboarding for High-Cardinality Workloads

Connecting Grafana to Thanos Query

Add a Prometheus data source pointing to http://thanos-query:9090. Enable “HTTP Method: POST” — Thanos accepts long queries that would exceed URL limits with GET. Set scrape interval to match your Prometheus config (typically 15s).

Best Practices for High-Cardinality Dashboards

Practice	Why
Use recording rules for expensive queries	Pre-compute at ingestion, not at dashboard load
Avoid {__name__=~".+"} wildcards	Forces full index scan
Add sum by (...) early in PromQL	Reduces data transferred from Store Gateway
Set max_source_resolution=auto in Grafana	Leverages downsampled blocks
Use $__rate_interval instead of fixed windows	Scales with scrape interval
Limit template variable queries with label_values matchers	Prevents cardinality floods

A common anti-pattern is dashboarding raw high-cardinality counters like http_request_duration_seconds_bucket. Aggregate with recording rules:

groups:
  - name: http_aggregations
    interval: 30s
    rules:
      - record: job:http_request_duration_seconds:p95
        expr: histogram_quantile(0.95, sum by (le, job) (rate(http_request_duration_seconds_bucket[5m])))

Operational Best Practices and Troubleshooting

• Monitor Thanos itself. Scrape sidecar, store, query, and compactor metrics. Alert on thanos_objstore_bucket_operation_failures_total and thanos_compact_group_compactions_failures_total.
• Cap sidecar upload concurrency. Set --objstore.config-file with max_idle_conns tuned to your S3 provider; too many parallel uploads cause throttling.
• Test retention deletes in staging. --retention.resolution-raw deletes are irreversible.
• Watch for block overlap. If two Prometheus HA replicas share external_labels incorrectly, Compactor will refuse to compact overlapping blocks.
• Sizing Query. A single Query pod handles ~50–100 QPS of medium PromQL. Scale horizontally behind Query Frontend with --query-frontend.downstream-url.
• Use --query.timeout and --query.max-concurrent to prevent a rogue dashboard from exhausting Query workers.

Conclusion

A self-hosted Prometheus + Thanos + Grafana stack delivers long-term retention, global querying, and full data ownership at a predictable cost. The architecture is well-trodden: sidecars upload blocks, Store Gateway serves them, Compactor downsamples and prunes, and Query provides a unified PromQL endpoint. Start with a single Prometheus and MinIO bucket, validate the pipeline end-to-end, then scale out with federation and additional Store Gateway replicas as your cardinality grows. The biggest risk is not storage — it’s cardinality. Instrument your scrapes, enforce relabeling discipline, and your stack will comfortably handle years of metrics.

FAQ

What is the difference between Prometheus and Thanos? Prometheus is a metrics collection and short-term storage system, typically retaining data for 15–30 days. Thanos extends Prometheus by providing long-term storage in object storage, global querying across multiple Prometheus instances, and downsampling for efficient historical queries. Thanos is not a replacement for Prometheus but a complement.

Can I use Thanos without Prometheus? No, Thanos relies on Prometheus to generate metrics blocks. The Thanos sidecar uploads these blocks to object storage. However, you can use Thanos with other Prometheus-compatible systems like VictoriaMetrics or Cortex, but Prometheus is the most common and well-supported.

How much storage do I need for long-term metrics? Storage depends on ingestion rate, retention period, and cardinality. A rough estimate: each active time series generates about 1–2 bytes per sample after compression. For 1M active series with a 15s scrape interval, expect ~8–20 GB per day. Use the formula in the guide to calculate precisely.

What object storage backends does Thanos support? Thanos supports major object storage providers including AWS S3, Google Cloud Storage, Azure Blob Storage, OpenStack Swift, and any S3-compatible storage like MinIO or Ceph. For self-hosted, MinIO is a popular choice.

How does Thanos handle high cardinality? Thanos itself doesn’t reduce cardinality; it stores and queries what Prometheus ingests. To manage high cardinality, optimize Prometheus scrape configurations, use relabeling to drop unnecessary labels, and consider recording rules to pre-aggregate data. Thanos Query can handle high-cardinality queries but may require more resources.

What is the role of the Thanos Compactor? The Compactor applies Prometheus’ 2-hour compaction to blocks uploaded by sidecars, creates downsampled versions (5m and 1h resolution) for faster long-range queries, and enforces retention policies by deleting old blocks from object storage. It should run as a single instance per bucket.

Can I use Grafana with Thanos? Yes, Grafana connects to Thanos Query as a Prometheus data source. Simply point Grafana to the Thanos Query endpoint (e.g., http://thanos-query:9090) and use PromQL queries as usual. Thanos Query supports the Prometheus HTTP API.

How do I scale Prometheus for high availability? Run multiple Prometheus instances with identical configurations, each scraping the same targets. Use Thanos sidecars on each and a Thanos Query layer to deduplicate results. This provides HA without complex sharding. For scaling beyond a single Prometheus, use federation or sharding by target.