Austin Parker austinlparker

Parsing PDS Logs With OpenTelemetry

This configuration should parse your ATProto/Bluesky PDS logs into nicely formatted and structured OpenTelemetry Logs.

You'll need to use OpenTelemetry Collector Contrib, and you can either install it on your host or run it as a container (if running as a container, you'll need to map the Docker log path in as a read-only volume).

This also assumes that your Docker daemon.json has "tag": "{{.Name}}|{{.ImageName}}|{{.ID}}" set under log-opts (this is how we parse the container metadata).

As an exercise to the reader, you can use the count connector to parse the log stream into Prometheus or OTLP metrics. You can also add in hostmetrics receiver to get memory, CPU, disk, etc. utilization.

	// replace the <USER DISPLAY NAME> with the display name of the user you wish to hide reposts from in your following feed
	function processElements() {
	document.querySelectorAll('[aria-label="Reposted by <USER DISPLAY NAME>"]').forEach(element => {
	let parent = element.parentElement;
	while (parent) {
	if (parent.tagName === 'DIV' && !parent.className) {
	parent.className = 'hide';
	break;
	}
	parent = parent.parentElement;

	#!/usr/bin/env fish
	# env-helper
	# you'll need the 1Password CLI installed to use this
	# change the vault name to whichever vault you want to use

	function confirm
	while true
	read -p 'echo "Confirm (y/n):"' -l confirm

	switch $confirm

	docker compose build emailservice --no-cache
	WARN[0000] The "TRACETEST_DEV" variable is not set. Defaulting to a blank string.
	[+] Building 32.7s (14/14) FINISHED
	=> [emailservice internal] load .dockerignore 0.0s
	=> => transferring context: 50B 0.0s
	=> [emailservice internal] load build definition from Dockerfile 0.0s
	=> => transferring dockerfile: 566B 0.0s
	=> [emailservice internal] load metadata for docker.io/library/ruby:3.1.4-alpine 1.6s
	=> [emailservice] importing cache manifest from ghcr.io/open-telemetry/demo:1.4.0-emailservice 0.0s
	=> [emailservice base 1/1] FROM docker.io/library/ruby:3.1.4-alpine@sha256:1a316159d7fa08cd32d5f1fa50577dc2ed8e2e7 0.0s

	2023-07-20T15:00:58.8818125Z Requested labels: ubuntu-latest
	2023-07-20T15:00:58.8818433Z Job defined at: open-telemetry/opentelemetry-demo/.github/workflows/build-images.yml@refs/pull/989/merge
	2023-07-20T15:00:58.8818616Z Reusable workflow chain:
	2023-07-20T15:00:58.8818713Z open-telemetry/opentelemetry-demo/.github/workflows/checks.yml@refs/pull/989/merge (cc4a29c78704979d2824d4d004c7e38aa2e19425)
	2023-07-20T15:00:58.8818812Z -> open-telemetry/opentelemetry-demo/.github/workflows/build-images.yml@refs/pull/989/merge (cc4a29c78704979d2824d4d004c7e38aa2e19425)
	2023-07-20T15:00:58.8818921Z Waiting for a runner to pick up this job...
	2023-07-20T15:01:01.9391698Z Job is waiting for a hosted runner to come online.
	2023-07-20T15:01:02.8626369Z Job is about to start running on the hosted runner: GitHub Actions 292 (hosted)
	2023-07-20T15:01:15.3668113Z Current runner version: '2.306.0'
	2023-07-20T15:01:15.3703164Z ##[group]Operating System

	2022/11/03 13:02:14 proto: duplicate proto type registered: jaeger.api_v2.PostSpansRequest
	2022/11/03 13:02:14 proto: duplicate proto type registered: jaeger.api_v2.PostSpansResponse
	2022-11-03T13:02:14.928Z info service/telemetry.go:110 Setting up own telemetry...
	2022-11-03T13:02:14.928Z info service/telemetry.go:140 Serving Prometheus metrics {"address": ":8888", "level": "basic"}
	2022-11-03T13:02:14.929Z info components/components.go:30 In development component. May change in the future. {"kind": "exporter", "data_type": "metrics", "name": "logging/debug", "stability": "in development"}
	2022-11-03T13:02:14.929Z warn [email protected]/factory.go:110 'loglevel' option is deprecated in favor of 'verbosity'. Set 'verbosity' to equivalent value to preserve behavior. {"kind": "exporter", "data_type": "metrics", "name": "logging/debug", "loglevel": "debug", "equivalent verbosity level": "detailed"}
	2022-11-03T13:02:14.929Z info components/components.go:30 In development component. May change in the future.

	# bundle install
	# LS_ACCESS_TOKEN=<token> bundle exec ruby main.rb
	main.rb
	---
	Bundler.require
	OpenTelemetry::SDK.configure do \|c\|
	c.service_name = 'ruby-test'
	c.add_span_processor(
	OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
	exporter: OpenTelemetry::Exporter::OTLP::Exporter.new(

	FROM ubuntu:18.04

	RUN set -x \
	# Install package dependencies
	&& apt-get update \
	&& apt-get install --no-install-recommends --no-install-suggests -y \
	wget \
	gzip \
	gnupg2 \
	software-properties-common \

	(samcli27) aparker@irvine  ~/go/src/github.com/apprenda/rebind   master ●  samdev local start-api --docker-network lambda-local -t templates/results-template.yml -n etc/devenv/results/docker-dynamo-environment.json --skip-pull-image --debug -d 5985
	2018-06-19 13:58:20 local start-api command is called
	2018-06-19 13:58:20 3 resources found in the template
	2018-06-19 13:58:20 Found Serverless function with name='RebindIngestService' and CodeUri='../bin/cmd/ingest'
	2018-06-19 13:58:20 Found Serverless function with name='RebindQueryService' and CodeUri='../bin/cmd/query'
	2018-06-19 13:58:20 Trying paths: ['/Users/aparker/.docker/config.json', '/Users/aparker/.dockercfg']
	2018-06-19 13:58:20 Found file at path: /Users/aparker/.docker/config.json
	2018-06-19 13:58:20 Found 'auths' section
	2018-06-19 13:58:20 Auth data for https://index.docker.io/v1/ is absent. Client might be using a credentials store instead.
	2018-06-19 13:58:20 Found 'credsStore' section

	List of Organizations with positive Labour Market Opinion (LMO) between June 1st, 2010

	and June 13th, 2012, by organization province/territory location

	Prepared by:
	Integrity Division
	Temporary Foreign worker Program

	0756504 BC LTD HUMPTYS 31
	0786 INVESTMENTS LTD JUGO JUICE