Mini metrics server setup VictoriaMetrics (Datadog-like)

This is a very minimal setup how to achieve a minimal metrics server for a small project or an embedded setup. Install VictoriaMetrics. https://docs.victoriametrics.com/quick-start/#starting-vm-single-from-a-binary

Install

Essentially:

VM_VERSION="v1.112.0"
wget -O /tmp/victoria-metrics.tar.gz "https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/${VM_VERSION}/victoria-metrics-linux-arm64-${VM_VERSION}.tar.gz"
sudo tar -xvf /tmp/victoria-metrics.tar.gz -C /usr/local/bin
sudo useradd -s /usr/sbin/nologin victoriametrics || true
sudo mkdir -p /var/lib/victoria-metrics && sudo chown -R victoriametrics:victoriametrics /var/lib/victoria-metrics

Create systemd unit file. Note, unlike in the documentation, we specify extra paths to scraping configs and for metrics -promscrape.config=/etc/promscrape.yaml --vmui.customDashboardsPath=/etc/vmdashboards

sudo bash -c 'cat <<END >/etc/systemd/system/victoriametrics.service
[Unit]
Description=VictoriaMetrics service
After=network.target

[Service]
Type=simple
User=victoriametrics
Group=victoriametrics
ExecStart=/usr/local/bin/victoria-metrics-prod -storageDataPath=/var/lib/victoria-metrics -retentionPeriod=90d -selfScrapeInterval=10s -promscrape.config=/etc/promscrape.yaml --vmui.customDashboardsPath=/etc/vmdashboards

SyslogIdentifier=victoriametrics
Restart=always

PrivateTmp=yes
ProtectHome=yes
NoNewPrivileges=yes

ProtectSystem=full

[Install]
WantedBy=multi-user.target
END'

Setup

Drop promscrape.yaml in /etc/.

global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'my_stuff'
    static_configs:
      - targets: ['localhost:8001']
(

Drop dashboard config in /etc/vmdashboards/. inspired by https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmselect/vmui/dashboards/perJobUsage.json

{
  "title": "Example",
  "rows": [
    {
      "title": "Per-job resource usage",
      "panels": [
        {
          "title": "Per-job CPU usage",
          "width": 6,
          "expr": [
            "sum(rate(process_cpu_seconds_total)) by (job)"
          ]
        },
        {
          "title": "Per-job RSS usage",
          "width": 6,
          "expr": [
            "sum(process_resident_memory_bytes) by (job)"
          ]
        },
        {
          "title": "Per-job disk read",
          "width": 6,
          "expr": [
            "sum(rate(process_io_storage_read_bytes_total)) by (job)"
          ]
        },
        {
          "title": "Per-job disk write",
          "width": 6,
          "expr": [
            "sum(rate(process_io_storage_written_bytes_total)) by (job)"
          ]
        }
      ]
    },
    {
      "title": "Free/used disk space",
      "panels": [
        {
          "unit": "MB",
          "expr": [
            "sum(vm_data_size_bytes{type!=\"indexdb\"}) / 1024 / 1024",
            "vm_free_disk_space_bytes / 1024 / 1024"
          ],
          "alias": [
            "usage space",
            "free space"
          ]
        }
      ]
    }
  ]
}

Tip: Dashboards appear in sorted order, so name them with leading numbers eg. 1-mydashboard.json

Reload config and start service sudo systemctl daemon-reload && sudo systemctl enable --now victoriametrics.service

Use

Install prometheus client. For python pip install prometheus-client

Use client according to docs: https://prometheus.github.io/client_python/getting-started/three-step-demo/

from prometheus_client import start_http_server, Summary

metr_proc = Summary('process_func_seconds', 'Time spent processing function')

# Decorate function with metric.
@metr_proc.time()
def process_func():
    # my stuff
    return 
    
if __name__ == '__main__':
    start_http_server(8001)

    for _ in range(1000):
        process_func()

Go to the web dashboard 'http://localhost:8428/vmui/' or just http://localhost:8428/ for overview of available APIs.

Logs

VictoriaMetrics logs to syslog /var/log/syslog by default.

martin-kokos/mini_metrics.md

Install

Setup

Use

Logs