jeffjirsa

From 014e09019b14f404898cd9c16384ee94ac293dab Mon Sep 17 00:00:00 2001
From: Jeff Jirsa <[email protected]>
Date: Sun, 8 Jan 2017 23:21:03 -0800
Subject: [PATCH] POC: Faster StreamingHistogram

---
 .../org/apache/cassandra/io/sstable/SSTable.java   |   3 +-
 .../io/sstable/metadata/MetadataCollector.java     |   2 +-
 .../apache/cassandra/utils/StreamingHistogram.java | 127 ++++++---
 .../test/microbench/StreamingHistogramBench.java   | 307 +++++++++++++++++++++

jeffjirsa

If you want to put some numbers on that, and taking a January start for 4.0
(I know it won't happen but bear with me), you get:
- We release 4.0 on January 2017 (it's both a "feature" and "testing"
release). We can call 3.0 the "stable" branch for the transition, which
thus enter it's critical bug only phase.
- From Februray to June, we release 4.1 -> 4.5 from trunk ("feature"), and
4.0.1 -> 4.0.5 from 4.0 with only bug fixes ("testing"). If a critical
fix on 3.0 comes, we cut a release.
- In July, we rotate, so we release 5.0, both a "feature" and "testing"
release from trunk, as well as 4.0.6 with bug fixes (but from that point

jeffjirsa

Current cluster is now: test
[node1 ERROR] objc[10536]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/bin/java and /Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/libinstrument.dylib. One of the two will be used. Which one is undefined.
[node2 ERROR] objc[10537]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/bin/java and /Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/libinstrument.dylib. One of the two will be used. Which one is undefined.

 random
---------------------
 2352308556279529416

(1 rows)

jeffjirsa

Trivial repro on 3.7 with scripts below: 

{code}
#!/bin/sh

ccm remove test

ccm create test -v 3.7 -n 1
sed -i'' -e 's/row_cache_size_in_mb: 0/row_cache_size_in_mb: 100/g' .ccm/test/node1/conf/cassandra.yaml
ccm start

jeffjirsa

    [junit] Test org.apache.cassandra.cache.AutoSavingCacheTest FAILED
    [junit] Test org.apache.cassandra.cql3.KeyCacheCqlTest FAILED (timeout)
    [junit] Test org.apache.cassandra.cql3.LargeCompactValueTest FAILED
    [junit] Test org.apache.cassandra.db.BatchlogManagerTest FAILED
    [junit] Test org.apache.cassandra.db.CleanupTest FAILED
    [junit] Test org.apache.cassandra.db.ColumnFamilyStoreTest FAILED
    [junit] Test org.apache.cassandra.db.KeyCacheTest FAILED
    [junit] Test org.apache.cassandra.db.NonIndexedRowEntryTest FAILED
    [junit] Test org.apache.cassandra.db.RecoveryManagerTruncateTest FAILED
    [junit] Test org.apache.cassandra.db.ScrubTest FAILED

jeffjirsa

## TimeWindowCompactionStrategy

TimeWindowCompactionStrategy is designed specifically for workloads where it's beneficial to have data on disk grouped by the timestamp of the data, a common goal when the workload is time-series in nature or when all data is written with a TTL. In an expiring/TTL workload, the contents of an entire SSTable likely expire at approximately the same time, allowing them to be dropped completely, and space reclaimed much more reliably than when using SizeTieredCompactionStrategy or LeveledCompactionStrategy. The basic concept is that TimeWindowCompactionStrategy will create 1 sstable per file for a given window, where a window is simply calculated as the combination of two primary options:

* `compaction_window_unit`: A Java TimeUnit (MINUTES, HOURS, or DAYS). The default value is DAYS
* `compaction_window_size`: The number of units that make up a window. The default value is 1

Taken together, the operator can specify windows of virtually any size, and TimeWindowCompactionStrategy

jeffjirsa

cqlsh> create keyspace test with replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
cqlsh> use test;
cqlsh:test> drop table if exists test0;
cqlsh:test> CREATE TABLE test0 (
        ...     pk int,
        ...     a int,
        ...     b text,
        ...     s text static,
        ...     PRIMARY KEY (pk, a)
        ... );

jeffjirsa

echo 'include_datacenters=datacenter1,datacenter2,datacenter3' > ~/.ccm/test/node3/conf/cassandra-dctopology.properties 
echo 'include_datacenters=datacenter1,datacenter2,datacenter3' > ~/.ccm/test/node2/conf/cassandra-dctopology.properties

jeffjirsa

diff --git a/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java b/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java
index 15287bd..9d0b086 100644
--- a/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java
+++ b/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java
@@ -123,7 +123,7 @@ public class DateTieredCompactionStrategy extends AbstractCompactionStrategy

     private List<SSTableReader> getCompactionCandidates(Iterable<SSTableReader> candidateSSTables, long now, int base)
     {
-        Iterable<SSTableReader> candidates = filterOldSSTables(Lists.newArrayList(candidateSSTables), options.maxSSTableAge, now);
+        Iterable<SSTableReader> candidates = filterOldSSTables(Lists.newArrayList(candidateSSTables), options.maxSSTableAge, now, options.useMinTimestamp);

jeffjirsa

CREATE TYPE form_option (
    left int,
    operator text,
    right int,
);

CREATE TABLE forms (
    page_id int,
    form_id int,
    form_option_id timeuuid,