Skip to content

Instantly share code, notes, and snippets.

@rma92

rma92/Readme.md

Last active November 29, 2024 16:45
Show Gist options
  • Save rma92/20426d37d2526574efc2736dd7c05353 to your computer and use it in GitHub Desktop.
Save rma92/20426d37d2526574efc2736dd7c05353 to your computer and use it in GitHub Desktop.
Download ZIP
Monitoring Infra: Set up victoria metrics!
Raw
Readme.md

I have an OpenBSD 7.0 monitoring server that's been sitting around for years to hold monitoring infra. Let's get it going with VictoriaMetrics. Seems good things come to those who wait, since now VictoriaMetrics is a better option than what was available when the infrastructure was set up in early 2022.

Firewall Config and Install URL (setup)

Set up config for /etc/pf.conf, we will probably need to unblock more for monitoring later.

set skip on lo
block all
#Allow SSH / anti-lockout:
pass in quick proto tcp from any to any port = 22

#Allow prometheus scrape monitoring outbound
pass out quick proto { tcp, udp } from any to any port { 9100 }

#Allow any DNS:
#pass out quick proto { tcp, udp } from any to any port domain

#Allow only known DNS:
pass out quick proto { tcp, udp } from any to { 8.8.8.8, 1.1.1.1, 8.8.4.4, 4.2.2.2, 108.61.10.10 } port domain

#Servers for packages: (ftp.lysator.liu.se)
pass out quick proto { tcp, udp } from any to { 130.236.254.251, 130.236.254.253} port {http, https, ftp}

#block return   # block stateless traffic
#pass           # establish keep-state

# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010

# Port build user does not need network
block return out log proto {tcp udp} user _pbuild

pfctl -nf /etc/pf.conf
pfctl -f /etc/pf.conf

For old openbsd packages, set the mirror to https://ftp.lysator.liu.se/pub/OpenBSD:

echo https://ftp.lysator.liu.se/pub/OpenBSD > /etc/installurl

Set up VictoriaMetrics

  • get victoria-metrics-openbsd-amd64-v1.106.1.tar.gz and vmutils-openbsd-amd64-v1.106.1.tar.gz
  • tar -xf victoria-metrics-openbsd-amd64-v1.106.1.tar.gz
  • tar -xf vmutils-openbsd-amd64-v1.106.1.tar.gz Run as root:
mkdir /etc/vm
mv victoria-metrics-prod /etc/vm
useradd -d /var/empty -s /sbin/nologin -c "VictoriaMetrics user" -L daemon vmuser
vi /etc/rc.d/victoriametrics

Put in the file:

#!/bin/ksh
# VictoriaMetrics service script for OpenBSD

daemon="/etc/vm/victoria-metrics-prod"
user="vmuser"
config="/etc/vm/victoria-metrics.conf"
flags=""

. /etc/rc.d/rc.subr

rc_bg=YES
rc_cmd $1

Finish setting up the service:

chmod +x /etc/rc.d/victoriametrics
rcctl enable victoriametrics

Make sure it runs, but stop it so we can configure it:

rcctl start victoriametrics
rcctl stop victoriametrics

Configure victoriametrics

As root:

vi /etc/vm/victoria-metrics.conf

Put in the file:

# Config for VictoriaMetrics
-storageDataPath=/var/lib/victoria-metrics
-retentionPeriod=30
-httpListenAddr=:8428

Set permissions (as root)

chown vmuser:vmuser /etc/vm/victoria-metrics.conf
chmod 600 /etc/vm/victoria-metrics.conf

Allow access to the write interface in pf (Only if you're in a reasonably secure network, otherwise skip)

As root:

vi /etc/pf.conf

Add a line for this to the file:

# Allow Prometheus write interface
pass in on egress proto tcp from any to any port 8428
# Allow established connections (response traffic)
pass out proto tcp from any to any flags S/SA keep state

Test, then load the pf rules, and confirm they worked.

pfctl -nf /etc/pf.conf
pfctl -f /etc/pf.conf
pfctl -sr | grep 8428

Set up Vmauth

Vmauth is a proxy VictoriaMetrics wrote that can simply put basic auth in front of VictoriaMetrics. It listens on port 8427 by default. `vi /etc/vm/vmauth.yaml

users:
- username: foo
  password: bar
  url_prefix: "http://localhost:8428/"

Now add an rc.d config. vi /etc/rc.d/vmauth_prod

#!/bin/sh

daemon="/etc/vm/vmauth-prod"
daemon_flags="-auth.config=/etc/vm/vmauth.yaml"
daemon_user="vmuser"
rc_bg="YES"

. /etc/rc.d/rc.subr

rc_cmd $1

And fix the permissions, enable and start the service:

sudo chmod +x /etc/rc.d/vmauth_prod
rcctl enable vmauth_prod
rcctl start vmauth_prod

Set up relayd to add HTTPS:

set up relayd as a proxy:

Create SSL key:

 openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout /etc/ssl/private/self-signed.key -out /etc/ssl/self-signed.crt

set up relayd.conf, and set up relayd service

#create /etc/relayd.conf
rcctl enable relayd
rcctl check relayd

relayd.conf contents (change or remove the host header)

log state changes

http protocol "http" {
    pass request quick header "Host" value "*rm.vg"
    pass request quick header "User-Agent" value "Mozilla*"
    pass request quick header "User-Agent" value "Links*"
    pass request quick header "User-Agent" value "Lynx*"
    pass request quick header "User-Agent" value "W3m*"
    pass request quick header "User-Agent" value "Opera*"
    pass request quick header "User-Agent" value "ELinks*"
    match response header set "Cache-Control" value "max-age=86400"
    block
}

relay "web" {
    listen on 0.0.0.0 port 34080
    protocol "http"
    forward to 127.0.0.1 port 8427
}

protocol "https" {
  tls ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:..."
  tls keypair "self-signed"

  match request header append "X-Forwarded-For" value "$REMOTE_ADDR"
  match request header append "X-Forwarded-Port" value "$REMOTE_PORT"

  match response header set "Referrer-Policy" value "same-origin"
  match response header set "X-Frame-Options" value "deny"
  match response header set "X-XSS-Protection" value "1; mode=block"
  match response header set "X-Content-Type-Options" value "nosniff"
  match response header set "Strict-Transport-Security" value "max-age=31536000; includeSubDomains; preload"
  match response header set "Content-Security-Policy" value "default-src 'none'; ..."
  match response header set "Permissions-Policy" value "accelerometer=(), .."
  match response header set "Cache-Control" value "max-age=86400"
  return error
  pass
}

relay wwwtls {
  listen on 0.0.0.0 port 34443 tls
  protocol https
  forward to 127.0.0.1 port 8427
}

You can change tls port to 443 and just put this behind a normal cloudflare proxy... use firewall on the VM's host to allow https only from cloudflare.

Setup Cloudflared tunnel

curl -L -o /etc/vm/cloudflared https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-openbsd-amd64
chmod +x /etc/vm/cloudflared
mv /etc/vm/cloudflared /usr/local/bin

Allow Cloudflared through the firewall by editing pf.conf (vi /etc/pf.conf) and reloading it (rcctl -f /etc/pf.conf)

# Allow outbound traffic to Cloudflare's network
pass out proto tcp to any port 7844
pass out proto udp to any port 7844
pass out proto tcp to any port 443

TODO: use their IPs from https://www.cloudflare.com/ips-v4/# to further restrict this. Log into Cloudflare and create a tunnel

cloudflared tunnel login
cloudflared tunnel create my-tunnel

Edit the config file: vi /usr/local/etc/cloudflared/config.yml

tunnel: <UUID>  # Replace <UUID> with your tunnel's UUID
credentials-file: /usr/local/etc/cloudflared/<UUID>.json

ingress:
  - hostname: xmon.i.rm.vg
    service: https://localhost:34443
  - service: http_status:404

fix permsisions, run the tunnel

chmod 600 /usr/local/etc/cloudflared/config.yml
cloudflared tunnel run my-tunnel

Add it as a service:

sudo vi /etc/rc.d/cloudflared

#!/bin/sh

daemon="/usr/local/bin/cloudflared"
daemon_flags="tunnel run my-tunnel"
daemon_user="cloudflare"  # Replace with a dedicated user
rc_bg="YES"

. /etc/rc.d/rc.subr

rc_cmd $1

Fix perms and enable / start the service:

sudo chmod +x /etc/rc.d/cloudflared
sudo rcctl enable cloudflared
sudo rcctl start cloudflared
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment