Stuart Horsman StuHorsman-zz

StuHorsman-zz / gist:efc633442e9b3a242b11

Created August 19, 2015 07:30

Replicate R packages

	installedLibs <- installed.packages()
	writeLibs <- paste0("install.packages(\"", installedLibs[, 1], "\")")
	write.table(writeLibs, file = "~/install_libs.R", row.names = F, col.names = F, quote = F)

StuHorsman-zz / gist:0cbc4d49ff6f6ab22306

Created May 20, 2015 01:17

Extract column names from Hive table

hive -S -e 'SET hive.cli.print.header=true; SELECT * FROM default.some_table LIMIT 0;' | sed -e 's/\t/,/g' > headers.txt

StuHorsman-zz / gist:a19653be9b2425a6aa29

Created October 15, 2014 03:02

Using fpm to build RHEL packages

	fpm -s dir -t rpm -n "cabal" -v "1.20.0.3" -p "cabal-1.20.0.3.rpm" /opt/cabal/1.20.0.3
	fpm -s dir -t rpm -n "ghc" -v "7.8.3" -p "ghc-7.8.3.rpm" /opt/ghc/7.8.3

StuHorsman-zz / gist:16eac4096273288014e5

Created October 15, 2014 00:21

die() function

die () { printf "\nError during cabal-install bootstrap:\n$1\n" >&2 && exit 2 ;}

StuHorsman-zz / gist:0ed851f01bf926377bef

Created September 16, 2014 11:33

Mysql on Ubuntu setup (CM)

	# install mysql
	cd ~/cdh-preinstall
	apt-get install mysql-server mysql-client libmysql-java -y

	# stop and copy in new my.cnf
	sudo service mysql stop
	sudo cp files/my.cnf.ubuntu /etc/mysql/my.cnf

	# restart mysql and secure
	sudo mysql start

StuHorsman-zz / gist:b5be0adac5bfb40ad9c0

Created August 31, 2014 23:16

Formatted sorted md5sum on Mac

find . -name *.jar -exec md5 {} \; | egrep -v 'tests|original' | cut -d'/' -f4 | sed 's/)//g' | awk -F'=' '{print $2 " " $1'} | sort -k2

StuHorsman-zz / gist:f8631c730b55372e6a13

Created July 24, 2014 00:37

Install (specific) CDH version of Hadoop with brew

	$ cd $(brew --prefix)
	$ brew update
	$ brew uninstall hadoop
	Uninstalling /usr/local/Cellar/hadoop/2.4.1...
	$ brew versions hadoop
	Warning: brew-versions is unsupported and will be removed soon.
	You should use the homebrew-versions tap instead:
	https://github.com/Homebrew/homebrew-versions

	2.4.1 git checkout a20feb2 /usr/local/Library/Formula/hadoop.rb

StuHorsman-zz / gist:7fe1b5165eaaa74c8297

Last active August 29, 2015 14:04

Hadoop Pi Example

	# for tarball download extracted into /usr/local
	hadoop jar /usr/local/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.3.0-cdh5.1.0.jar pi 10 1000000
	# for brew install using latest 2.4.1 release
	hadoop jar /usr/local/Cellar/hadoop/2.4.1/libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar pi 10 1000000
	# for CDH 4.6
	hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples-2.0.0-mr1-cdh4.6.0.jar pi 10 1000000

StuHorsman-zz / gist:bc37b232078367873307

Created June 12, 2014 04:52

R: Sample a dataframe

df[sample(nrow(df), 100, replace=TRUE), ]

StuHorsman-zz / gist:9925431

Last active September 12, 2019 00:22

Set debug on namenode (example block placement policy)

sudo -u hdfs hadoop daemonlog -setlevel <namenode host>:50070 org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy DEBUG