blacktty blacktty

原文：运维帮

随着Splunk越来越被大家熟知和认可，现在市面上也不断涌各种同类产品，作为大数据搜索界的翘楚Splunk和ElasticSearch，绝对值得我们去学习，探索和使用，因此为了造福Splunk的铁粉和新粉们，小编特邀了Splunk的资深架构师，江湖人称“陶指导”的陶刚为大家就架构，功能，产品线，概念等方面将Splunk和ElasticSearch做了一下全方位的对比，希望能够给大家在制定大数据搜索方案的时候有所帮助。

陶刚在Splunk上海担任资深架构师，负责数据采集和云平台产品的技术架构。拥有丰富的企业级产品的开发经验，对数据科学，数据可视化和机器学习等领域有着浓厚的兴趣。同时是足球和炉石传说的狂热爱好者，也是大圣庞卡足球队的当家球霸和炉石传说俱乐部最受追捧的明星会长。

本文就架构，功能，产品线，概念等方面就ElasticSearch和Splunk做了一下全方位的对比，希望能够大家在制定大数据搜索方案的时候有所帮助。

Introduction

When applications are running in production, they become black boxes that need to be traced and monitored. One of the simplest, yet main, ways to do so is logging. Logging allows us - at the time we develop our software - to instruct the program to emit information while the system is running that will be useful for us and our sysadmins.

最终实现效果:

无版本概念,任何本地文件均可增量升级到最新.服务器不用管理多版本
内存小,100M文件升级时只占用500KB内存.

使用流程:

制作新版本,上传HTTP File Server.
Client自动计算差异,下载差异,合并差异.
done!

#0.起源

	package main

	import (
	"context"
	"log"
	"net/http"
	"time"

	"github.com/gin-gonic/gin"
	)

	input {
	stdin {
	codec => multiline {
	pattern => "^\[%{MONTHDAY}-%{MONTH}-%{YEAR} %{TIME} %{TZ}\]"
	negate => true
	what => "previous"
	auto_flush_interval => 10
	}
	type => "php-error"
	}

	all: lib run

	lib:
	g++ -shared -fPIC -o libhello.so libhello.cpp hello.cpp
	run:
	luajit main.lua
	clean:
	rm *.so

	#yum install haproxy
	#configure haproxy-cloudera.cfg
	#haproxy -f /etc/haproxy/haproxy-cloudera.cfg
	#http://seawolf-3.vpc.wonderland.com:1936/
	#https://cbonte.github.io/haproxy-dconv/

	global
	daemon
	nbproc 1
	maxconn 100000

	from impala.dbapi import connect as impala_connect
	import csv
	import sys
	import pdb
	import MySQLdb as mysql

	global connection
	global cursor

	def connect():

	#!/usr/bin/python
	# impala-queries.py
	# List, query and terminate in-flight Impala queries through the CM API
	# -----------------------------------------------------------------------
	# Copyright (C) 2015 Cloudera and Ben White

	import urllib2, json, sys

	# Cloudera Manager credentials

	echo "Flipping tables! (╯°□°）╯︵ ┻━┻"

	num_rules=3

	real=3 # exposed to the ELB as port 443
	test=4 # used to install test certs for domain verification
	health=5 # used by the ELB healthcheck

	blue_prefix=855
	green_prefix=866