1. 云栖社区>
  2. PHP教程>
  3. 正文

elasticsearch+kibana+logstash搭建日志监控系统

作者:用户 来源:互联网 时间:2017-12-01 17:10:00

日志elasticsearchlogstashkibana搭建监控系统

elasticsearch+kibana+logstash搭建日志监控系统 - 摘要: 本文讲的是elasticsearch+kibana+logstash搭建日志监控系统, 1. 下载 https://www.elastic.co/downloads/kibana https://www.elastic.co/downloads/elasticsearch https://www.elastic.co/dow

1. 下载


https://www.elastic.co/downloads/kibana


https://www.elastic.co/downloads/elasticsearch


https://www.elastic.co/downloads/logstash


2. 解压到制定的文件夹下面并重命名


mv elasticsearch-2.3.3elasticsearch


mvkibana-4.5.1kibana


mv logstash-2.3.2logstash


3. 安装插件

$cd elasticsearch
head
$bin/plugin install mobz/elasticsearch-head
watcher
$bin/plugin install license
$bin/plugin install watcher
shield
$bin/plugin install shield

4. 配置shield插件

#创建管理员用户
$bin/shield/enusers useradd es_admin -r admin
创建Logstash用户
$bin/shield/enusers useradd logstashserver -r logstash
创建kibana用户
$bin/shield/enusers useradd kibanaserver -r kibana4_server

5. 配置kibana

$cd cabana
$ vi config/kibana.yml
设置server.host #例如"127.0.0.1"
设置elasticsearch.username# 例如:kibanaserver
设置elasticsearch.password
保存退出

6. 启动

$cd elasticsearch
$bin/elasticsearch
$cd kibana
$bin/kibana

7. 测试


es: http://127.0.0.1:9200/_plugin/head/


kibana:http://127.0.0.1:5601


输入管理员账号和密码 es_admin/password


如果一切正常。说明elasticsearch 和 kibana搭建成功


8. 配置logstash


cd logstash


(1) hello world


vi logstash-simple.conf


input {


stdin { }


}


output {


elasticsearch {


hosts => ["localhost:9200"]


user => "logstashserver"


password => "newpass"


}


stdout {


codec => rubydebug }


}


bin/logstash -flogstash-simple.conf


接着输入Hello World


去http://127.0.0.1:9200/_plugin/head/可以看到以logstash开头的indices


(2) 复杂点的logstash配置


vilogstash-filter.conf


input { stdin { } }


filter {grok { match => { "message" => "%{COMBINEDAPACHELOG}" }}


date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]} }


output {elasticsearch {


hosts => ["localhost:9200"]


user => "logstashserver"


password => "newpass"


}


stdout { codec => rubydebug }


}


bin/logstash -f logstash-filter.conf


输入以下


127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"


(3) syslog


vi logstash-syslog.conf


input {


tcp {


port => 5000


type => syslog

}


udp {


port => 5000


type => syslog


} }


filter {


if [type] == "syslog”


{grok {


match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:/[%{POSINT:syslog_pid}/])?: %{GREEDYDATA:syslog_message}" }


add_field => [ "received_at", "%{@timestamp}" ]


add_field => [ "received_from", "%{host}" ] }


date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]


}} }


output {elasticsearch {


hosts => ["localhost:9200”]


user => "logstashserver"


password => "newpass"

}


stdout { codec => rubydebug } }


再另外打开一个终端


telnet localhost 5000


输入hello logstash


(4) File Input


logstash-tutorial-dataset 文件下载链接


https://download.elastic.co/demos/logstash/gettingstarted/logstash-tutorial.log.gz


vilogstash-first.conf


input {


file {


path => "/Users/fdrong/LogProject/logstash-tutorial-dataset"


start_position => beginning


}


}


filter {


grok {


match => { "message" => "%{COMBINEDAPACHELOG}"}


}


geoip {


source => "clientip"


}


}


output {


elasticsearch {


hosts => ["localhost:9200"]


user => "logstashserver"


password => "newpass"


}


stdout {


codec => rubydebug }


}


测试以下语法是否有错


bin/logstash -f logstash-first.conf —configtest


如果没有错误


bin/logstash -f logstash-first.conf


如果看到控制台打印解析后的日志(JSON格式)说明配置成功


9. 配置Watcher


(1). add a condition that simply checks to see if the search input returned any hits


curl -u es_admin —XPUT 'http://localhost:9200/_watcher/watch/log_error_watch' -d '{


"trigger" : { "schedule" : { "interval" : "10s" } },


"input" : {


"search" : {


"request" : {


"indices" : [ "logs" ],


"body" : {


"query" : {


"match" : { "message": "error" }}}}}},


"condition" : {


"compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}


}


}'


(2) 观察结果


curl —u es_admin -XGET 'http://localhost:9200/.watch_history*/_search?pretty' -d '{


"query" : {


"bool" : {


"must" : [


{ "match" : { "result.condition.met" : true }},


{ "range" : { "result.execution_time" : { "from" : "now-10s"}}}


]


}


}


}'


(3)添加错误日志


curl -u es_admin —XPOST 'http://localhost:9200/logs/event' -d '{


"timestamp" : "2015-05-17T18:12:07.613Z",


"request" : "GET index.html",


"status_code" : 404,


"message" : "Error: File not found"


}’


然后再用第二步的命令查看出现很多hints说明执行成功


(4)删除


curl -XDELETE 'http://localhost:9200/_watcher/watch/log_error_watch'


10. 相关链接


logstash document


https://www.elastic.co/guide/index.html


elastic download


https://www.elastic.co/downloads


wacher


https://www.elastic.co/guide/en/watcher/current/watch-log-data.html#log-add-condition


shield


https://eligao.com/shield-on-elasticsearch/

以上是elasticsearch+kibana+logstash搭建日志监控系统的全部内容,在云栖社区的博客、问答、公众号、人物、课程等栏目也有其他相关内容,欢迎继续使用右上角搜索按钮进行搜索日志 , elasticsearch , logstash , kibana , 搭建 监控系统 ,以便于您获取更多的相关知识。

弹性可伸缩的计算服务,助您降低 IT 成本,提升运维效率

40+云计算产品,6个月免费体验

现在注册,免费体验40+云产品,及域名优惠!

云服务器9.9元/月,大学必备
Apache Spark Meetup China 第1期 最全资料下载