bekerov

val builder = new KStreamBuilder()
val twitterStream: KStream[String, String] = builder.stream[String, String]("twitter")

import be.icteam.demo.{KafkaStreamWrapper => KSW}

val pipeline =
  KSW.filter { (k: String, v: String) => v.contains("apache")} _ andThen
  KSW.filter((k, v) => true) andThen
  KSW.foreach((k, v) => println(v))

bekerov

REF

https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/ https://archive.ics.uci.edu/ml/datasets/Beijing+PM2.5+Data

bekerov

Optimized determination of Geotiff bounds

This is functional but is merely a workaround until we get vsis3 - you can try out preliminary vsis3 support with GDAL 2.1 and rasterio 0.32a1

Goal: Based on the GeoTiff and TIFF specs, manually parse out tags to allow for the most IO-efficient reading of georeferencing information.

It works almost the same as rio info --bounds but gives a json array, and it's fast

$ time rio info --bounds R4C1.tif

153.984375 24.2578125 154.072265625 24.345703125

bekerov

# run with a custom --n
# python run_luigi.py SquaredNumbers --local-scheduler --n 20

import luigi

class PrintNumbers(luigi.Task):
    n = luigi.IntParameter(default=10)

    def requires(self):
        return []

bekerov

bekerov

Interesting Machine Learning / Deep Learning Scenarios

This gist aims to explore interesting scenarios that may be encountered while training machine learning models.

Increasing validation accuracy and loss

Let's imagine a scenario where the validation accuracy and loss both begin to increase. Intuitively, it seems like this scenario should not happen, since loss and accuracy seem like they would have an inverse relationship. Let's explore this a bit in the context of a binary classification problem in which a model parameterizes a Bernoulli distribution (i.e., it outputs the "probability" of the true class) and is trained with the associated negative log likelihood as the loss function (i.e., the "logistic loss" == "log loss" == "binary cross entropy").

Imagine that when the model is predicting a probability of 0.99 for a "true" class, the model is both correct (assuming a decision threshold of 0.5) and has a low loss since it can't do much better for that example. Now, imagine that the model

bekerov

# X_train contains word indices (single int between 0 and max_words)
# Y_train0 contains class indices (single int between 0 and nb_classes)

X_train = sequence.pad_sequences(X_train, maxlen=maxlen, padding='post')
X_test = sequence.pad_sequences(X_test, maxlen=maxlen, padding='post')

Y_train = np.zeros((batchSize,globvars.nb_classes))#,dtype=np.float32)
for t in range(batchSize):
    Y_train[t][Y_train0[t]]=1
Y_test = np.zeros((len(Y_test0),globvars.nb_classes))#,dtype=np.float32)

bekerov

# Assumptions:
#
# Assumes boto is installed which comes with the s3put command
#
# Assumes your S3 keys are in ~/.boto in the form:
#
# [Credentials]
# aws_access_key_id = XXX
# aws_secret_access_key = XXX
#

bekerov

"""
Copyright (C) 2012 S. Girvin

Permission is hereby granted, free of charge, to any person obtaining a copy of this software
and associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial

bekerov

"""Print most frequent N-grams in given file.

Usage: python ngrams.py filename

Problem description: Build a tool which receives a corpus of text,
analyses it and reports the top 10 most frequent bigrams, trigrams,
four-grams (i.e. most frequently occurring two, three and four word
consecutive combinations).

NOTES