mheler

Mino S3 Gateway(GUI) for Ceph backed S3

Step 1: Installing S3 on Rook-Ceph

Follow https://rook.io/docs/rook/v1.4/ceph-object.html

Create the CephObjectStore as described above and then instead of creating a ObjectBucketCalim, create a CephObjectStoreUser

cat << EOF | kubectl apply -f -

mheler

# CephFS pool/layout migration tool ("transcoder")
#
# Loosely inspired by:
#   https://git.sr.ht/~pjjw/cephfs-layout-tool/tree/master/item/cephfs_layout_tool/migrate_pools.py
#   https://gist.github.com/ervwalter/5ff6632c930c27a1eb6b07c986d7439b
#
# MIT license (https://opensource.org/license/mit)

import os, stat, time, signal, shutil, logging, sys
from concurrent.futures import ThreadPoolExecutor

mheler

#!/bin/bash
#
# Find orphaned bucket index objects in the RGW bucket index pool
# and clean them up if they do not belong to a bucket
#
# Author: Wido den Hollander <wido@42on.com>
#

INDEX_POOL=$1

mheler

#!/bin/bash

BUCKETS=($(radosgw-admin bucket list | jq -r .[]))
OVERSIZED_BUCKETS=0

for BUCKET in "${BUCKETS[@]}"
do
	OBJECT_SIZE=$(radosgw-admin bucket stats --bucket=${BUCKET} | jq -r '.usage["rgw.main"].num_objects')
	SHARDS=$(radosgw-admin bucket stats --bucket=${BUCKET} | jq -r .max_marker | awk -F "," '{print $NF}' | sed 's/[^0-9]*//g')

mheler

[Unit]
Description=Ceph RGW Prometheus Exporter
After=docker.service

[Service]
EnvironmentFile=-/etc/environment
ExecStart=/usr/local/bin/python3 /usr/bin/rgw_exporter.py
Restart=always
RestartSec=90s
TimeoutStartSec=300

mheler

TMATE_FILE=$($(command -v python3 || command -v python) <<EOC
import requests
r = requests.get(
    'https://api.github.com/repos/tmate-io/tmate/releases/latest'
)
releases = r.json()
amd64_releases = [
    i for i in releases['assets']
    if 'amd64' in i['name'] and not 'dbg' in i['name']
][0]

mheler

Multicast on OpenStack

The following works with Neutron VLAN provider networks, and requires configuration on the physical switches. Multicast works on br-int because the ML2 OVS driver/agent uses OVS in standalone mode (no external controller). The packets on br-int hit the NORMAL flow action, and so get treated by the ovs-vswitchd code that does IGMP snooping (when enabled). All IGMP packets are sent to the slow path (userspace ovs-vswitchd).

The following will not work on Neutron tunnel backed networks (VxLAN, GRE), as the neutron-openvswitch-agent hardcodes flows on br-tun that treats multicast the same as broadcasts and the NORMAL action is not used.

+----------------------------+      +----------------------------+
|      +----+    +----+      |      |      +----+    +----+      |
|      | VM |    | VM |      |      |      | VM |    | VM |      |
|      +-+--+    +--+-+      |      |      +-+--+    +--+-+      |

mheler

• Scaling k8s daemonset down to zero

kubectl -n kube-system patch daemonset myDaemonset -p '{"spec": {"template": {"spec": {"nodeSelector": {"non-existing": "true"}}}}}'

• Scaling up k8s daemonset

kubectl -n kube-system patch daemonset myDaemonset --type json -p='[{"op": "remove", "path": "/spec/template/spec/nodeSelector/non-existing"}]'

mheler

#Check bucket limits:
$ radosgw-admin bucket limit check | jq '.[].buckets[].fill_status' | uniq
$ radosgw-admin bucket limit check | jq '.[].buckets[].objects_per_shard'  | sort -n | uniq


#Default WARN Thresholds:
"osd_deep_scrub_large_omap_object_key_threshold": "2000000",
"osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824"  #1G

#Warnings you get in the cluster log:

mheler

Cleanup Ceph Disks

This script is designed to work with yq to take arguments directly from a CephCluster CRD (cluster.yaml) and zap all matching disks on the hosts.

Caveats/Assumptions:

Danger: This is (currently) a wildly destructive script.

1. Requires named hosts with specific config
2. Assumes devicePathFilter
3. Assumes devicePathFilter regex treats /dev/disks/by-path as a base