Stephan Fitzpatrick knowsuchagency

Citadel OOM Incident — 2026-02-09

Symptoms

SSH to citadel (5.78.40.219) hung — TCP connected but no SSH banner returned

Hetzner VNC console showed continuous OOM killer messages:

Memory cgroup out of memory: Killed process XXXX (windmill)
Memory cgroup out of memory: Killed process XXXX (uv)

CPU pinned at ~1000-1500% (OOM kill/respawn loop)

Datalake Explorer GPT - Instructions

You help users explore and analyze data from Augment Risk's CRM datalake. Use the queryDucklake action to run SQL queries (DuckDB dialect).

Behavior Guidelines

Be proactive, not inquisitive. Infer user intent from context and act on it. Do not bombard users with clarifying questions. Make reasonable assumptions and execute. If you're wrong, the user will guide or correct you. It's better to attempt something useful than to ask for permission.

Always check schema before querying. Before running any analytics query, first run DESCRIBE main.table_name for every table you plan to query or join. Column names and types vary - never assume. This prevents errors and ensures accurate results.

Use Your ChatGPT Subscription for AI Coding Workloads

The Problem

You're paying for ChatGPT Plus/Pro ($20-200/month) AND buying OpenAI API tokens separately for AI-assisted coding. That's double-dipping.

The Solution

opencode-openai-codex-auth lets you authenticate OpenCode with your existing ChatGPT subscription, giving you access to GPT-5.x models for coding tasks without purchasing additional API credits.

Build Unlimited AI Agents for $200/month

Bottom line: The Claude Max plan delivers $800-1,600+ worth of API tokens - and you can use it programmatically via the Agent SDK.

Plan	Monthly Cost	API Token Value	ROI
Pro	$20	~$40-80	2-4x
Max 5x	$100	~$200-400	2-4x
Max 20x	$200	~$800-1,600+	4-8x

MCP-DSL Token Efficiency Experiment Results

Date: 2025-12-31 Budget Used: ~$5-10 (estimated from 2x ~20min A10G training runs + eval)

Summary

MCP-DSL demonstrates significant advantages over JSON-RPC for small model fine-tuning:

Resilient SSH Port Forwarding on macOS with launchd

A simple, robust way to maintain persistent SSH tunnels that automatically reconnect after network interruptions, sleep/wake cycles, or server restarts.

Overview

Instead of manually running SSH tunnels or using third-party tools like autossh, we use macOS's built-in launchd service manager. It handles:

Automatic startup on login
Automatic restart when the connection drops

Docker Container Hardening Plan for Dokploy

This guide hardens Docker container isolation on a system running Dokploy with untrusted public templates.

Overview

Step	Action	Risk Level
1	Backup current configuration	None
2	Apply Docker daemon security defaults	Low

Mount Remote Server (dokploy) via SSHFS on macOS

This guide documents how to mount a remote server's directory as a network drive on macOS using SSHFS.

Prerequisites

macOS (tested on macOS Sequoia)
SSH access to remote server configured in ~/.ssh/config
Homebrew installed

Testing Dokploy Webhook Endpoints

Problem

When attempting to manually test a Dokploy webhook endpoint for auto-deployment, the API returns a "Branch Not Match" error:

{
    "message": "Branch Not Match"
}

	[tasks.scrape]
	description = "scrape and extract readable content from a URL using readability-cli"
	usage = '''
	arg "<url>" help="URL to scrape and extract readable content from"
	'''
	run = "curl -s \"$usage_url\" \| npx -y mozilla-readability-cli \"$usage_url\" \| uvx html2text --ignore-links"