pick-columns.sh shell utility to pick columns. Uses awk

pick-columns.sh

#!/bin/bash
# This selects columns (indexing is 1-based, 0 means the entire file)
# from whatever's passed as stdin
# To select columns from the end, pass a negative number
# To select the last field, pass NF

declare -a my_cols
_a="$1"
while [ ! -z "$_a" ]; do
	my_cols+=("$_a")
	shift 1
	_a="$1"
done

_code='{'
for el in "${my_cols[@]}"; do
	if [ "$el" = "NF" ]; then
	  _fragment='printf $NF;'
	elif [ $el -lt 0 ]; then
		_fragment="i = NF + $el; printf \"%s \", \$i;"
	else
		_fragment="printf \"%s \", \$$el;"
	fi
	_code=$(printf "%s%s" "$_code" "$_fragment")
done
_code=$(printf "%s%s" "$_code" 'print ""}')

echo "Code is $_code"

awk "$_code"

zz-Example_use.md

pick-columns.sh example

This looks for data in a directory with many files, selects 1000 of them randomly, gets their sizes, and then uses mymaths.py to compute the mean, variance and stdev and other properties

du -k $(find /directory/with/many/files -type f) | \
  shuf -n 1000 | \
  pick-columns.sh 1 | \
  mymaths.py mean variance stdev multimode min max

Output:

mean: 21062.96
variance: 5526661943.796364
stdev: 74341.52233978239
multimode: 4.0
min: 4.0
max: 583948.0