Intro to Apache Spark: code example for RDD animation

sample.scala

// load error messages from a log into memory
// then interactively search for various patterns

// base RDD
lines = spark.textFile("hdfs://…")

// transformed RDDs
errors = lines.filter(lambda s: s.startswith("ERROR"))
messages = errors.map(lambda s: s.split("\t")[2])
messages.cache()

// actions
messages.filter(lambda s: "mysql" in s).count()
messages.filter(lambda s: "php" in s).count()