aseigneurin

Spark - Parquet files

Basic file formats - such as CSV, JSON or other text formats - can be useful when exchanging data between applications. When it comes to storing intermediate data between steps of an application, Parquet can provide more advanced capabilities:

• Support for complex types, as opposed to string-based types (CSV) or a limited type system (JSON only supports strings, basic numbers, booleans).
• Columnar storage - more efficient when not all the columns are used or when filtering the data.
• Partitioning - files are partitioned out of the box
• Compression - pages can be compressed with Snappy or Gzip (this preserves the partitioning)

The tests here are performed with Spark 2.0.1 on a cluster with 3 workers (c4.4xlarge, 16 vCPU and 30 GB each).

pachanka

#!/bin/bash
#
# Usage search-with-gpg path/to/encrypted/files/*
# 

if [ -z "$1" ]; then
    echo "Usage: $0 'search string' [path/to/encrypted/files/*]";
    exit 1;
else
    SEARCH=$1;

brennanMKE

import * as mongoose from 'mongoose';

export let Schema = mongoose.Schema;
export let ObjectId = mongoose.Schema.Types.ObjectId;
export let Mixed = mongoose.Schema.Types.Mixed;

export interface IHeroModel extends mongoose.Document {
  name: string;
  power: string;

stephenturner

Installing gcc 4.8 and Linuxbrew on CentOS 6

The GCC distributed with CentOS 6 is 4.4.7, which is pretty outdated. I'd like to use gcc 4.8+. Also, when trying to install Linuxbrew you run into a dependency loop where Homebrew's gcc depends on zlib, which depends on gcc. Here's how I solved the problem.

Note: Requires sudo privileges.

Resources:

• http://superuser.com/a/676337/88393: Forum response on using CERN's open Scientific Linux distribution of RHEL's developer toolset.
• http://linux.web.cern.ch/linux/devtoolset/: CERN's developer toolset installation instructions.

oza

YARN

1. General resource management layer on HDFS
2. A part of Hadoop

Spark

1. In memory processing framework

Spark on YARN

digital-wonderland

What

Unit files to deploy an ElasticSearch cluster on CoreOS via Fleet.

Service discovery & registration is done via etcd.

[email protected] provides a dumb discovery service by registering an elasticsearch host if it should be up. [email protected] registers the service only if it is running.

A service & timer unit for elasticsearch curator is provided which does some housekeeping.

gigorok

php -a -d auto_prepend_file=./vendor/autoload.php

chaitanyagupta

Reader Macros in Common Lisp

This post also appears on lisper.in.

Reader macros are perhaps not as famous as ordinary macros. While macros are a great way to create your own DSL, reader macros provide even greater flexibility by allowing you to create entirely new syntax on top of Lisp.

Paul Graham explains them very well in [On Lisp][] (Chapter 17, Read-Macros):

The three big moments in a Lisp expression's life are read-time, compile-time, and runtime. Functions are in control at runtime. Macros give us a chance to perform transformations on programs at compile-time. ...read-macros... do their work at read-time.

Stanback

#
# CORS header support
#
# One way to use this is by placing it into a file called "cors_support"
# under your Nginx configuration directory and placing the following
# statement inside your **location** block(s):
#
#   include cors_support;
#
# As of Nginx 1.7.5, add_header supports an "always" parameter which

rossant

"""
MIT License

Copyright (c) 2017 Cyrille Rossant

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is