Richard Mathie Richard-Mathie

Real Time Sorts and Joins

oh dir is there anything important in here?

Flask Cache Redis Cluster Backend

This impliments a redis cluster backend for Flask-Cache as the standard redis cache cannot handle a redis cluster.

Usage

class Config(object):

Cassandra Concurrent Writer

A class which constructs a concurrent writer object so that you can write async to cassandra, but limit the number of concurrent writes to some number, and block when we hit that limit. This can result in improved write throuput withough having to construct your query with genorators, as the default cassandra concurrent method requires.

I would sugest adding callbacks to handle write time outs, as you may want to retry the write.

Usage

Redis Cluster Setup with Docker Swarm

Setup

./redis.sh

Test

test the redis cluster

First ssh into the node you want to add lable to

docker-machine ssh aws-node

Depending on wheather the machine is using upstart or systemd you will need to modify a config file:

systemd:

sudo vim /etc/systemd/system/docker.service

edit the line starting ExecStart to add label

	import itertools

	def partition(items, predicate=int, n=2):
	""" Partition an iterable of `items` into `n` streams by `predicate`
	"""
	def filterer(i, tee):
	return (item for pred, item in tee if pred == i)
	pred_items = ((predicate(item), item) for item in items)
	teed_items = enumerate(itertools.tee(pred_items, n))
	return (filterer(x, t) for x, t in teed_items)

	from threading import Thread, Event
	from queue import Queue, Empty

	from kafka import KafkaConsumer, KafkaProducer
	from kafka.structs import TopicPartition, OffsetAndMetadata
	from kafka.consumer.fetcher import ConsumerRecord
	from kafka.errors import CommitFailedError

	class KafkaWorker(object):
	def __init__(self, topics, hosts, poll_timeout=1):

	version: 2.1

	default_working_dir: &working_container
	working_directory: /app
	docker:
	- image: docker:18.06.1-ce-git

	default_steps:
	steps:
	- setup_remote_docker

	credentials.py
	venv/