Andrew Gerhold zeushammer

High availability monitoring with Sensu

sensu | May 8, 2013

Redundancy. Availability. Scalability. Should sound familiar to you if you work in the web. Every system I build has to fit those 3 main criterias. I would also throw in manageability. If I can’t use Chef with it, I’m probably trying to use the wrong tool for the job.

There’s always been one exception though: my monitoring tool. Nagios. Zabbix. Zenoss. Shinken. Used them all. Each of them have shortcomings when it comes to the four criterias listed above. Different ones for each.

So, that said, a few months back, I was searching for something fresh. Something I could easily manage with Chef. Because I tend to forget things, and I wanted to automate as much as possible our monitoring solution. Don’t get me wrong, there’s nothing as good as developers to monitor your stuff, but I tend to like to know there’s something wrong before they show up at my desk. Even if it’s only 1-2 minutes ;)

Latency numbers every programmer should know

L1 cache reference ......................... 0.5 ns
Branch mispredict ............................ 5 ns                     on recent CPU
L2 cache reference ........................... 7 ns                     14x L1 cache
Mutex lock/unlock ........................... 25 ns
Main memory reference ...................... 100 ns                     20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy ............. 3,000 ns  =   3 µs
Send 2K bytes over 1 Gbps network ....... 20,000 ns  =  20 µs
SSD random read ........................ 150,000 ns  = 150 µs

Read 1 MB sequentially from memory ..... 250,000 ns = 250 µs 4X memory

	#!/usr/bin/env python

	import os
	import sys
	import pwd
	import random
	import struct
	import paramiko
	import tempfile
	import SocketServer

	#!/bin/bash

	GOEDITOR="/Applications/Atom.app"
	# practice does the following
	# creates a new directory named after the project
	# creates a file under that directory named $project.go with a small template
	# NOTE: specifically not using main.go so I can distinguish the files in my editor tabs
	# opens the file in my editor
	# changes to the new project directory

	#
	# CORS header support
	#
	# One way to use this is by placing it into a file called "cors_support"
	# under your Nginx configuration directory and placing the following
	# statement inside your location block(s):
	#
	# include cors_support;
	#
	# A limitation to this method is that Nginx doesn't currently send headers

	## Configure eth0
	#
	# vi /etc/sysconfig/network-scripts/ifcfg-eth0

	DEVICE="eth0"
	NM_CONTROLLED="yes"
	ONBOOT=yes
	HWADDR=A4:BA:DB:37:F1:04
	TYPE=Ethernet
	BOOTPROTO=static

	On a mac, `brew install awscli gnu-sed` before.

	streamname=staging;aws kinesis describe-stream --stream-name $streamname --output text \| grep SHARDS \| awk '{print $2}' \| while read shard; do aws kinesis get-shard-iterator --stream-name $streamname --shard-id $shard --shard-iterator-type LATEST --output text \| while read iterator; do while output=`aws kinesis get-records --shard-iterator $iterator --output text`; do iterator=`echo "$output" \| head -n1`; echo "$output" \| gsed 1d \| grep RECORDS \| while read record; do echo $record \| awk '{print $2}' \| base64 -D; done; done; done; done