ELK日志平台是一个完整的日志分析系统，有三个开源工具构建组成，分别是：Elasticsearch、Logstash和Kibana。Elasticsearch用于数据分析和深度搜索；Logstash作用是从其他服务器上传输和转发日志，对其集中管理，进行分析；Kibana则是提供了强大的UI展示，将数据可视化。
安装ELK日志平台
ELK基础环境需要java环境，官网要求5.x版本要大于java8。而且安装方式多样化，支持zip、tar.gz、rpm包、deb包、window环境还有docker环境。根据自己喜好选择吧。
我选择的是yum安装，简单方便，系统要求的话没有辣么严格，官网说yum安装方式不再支持centos5.x系列了，非要用centos5.x就去使用tar.gz包吧，官网有具体方法，不再复述。yum安装方式centos6.x和centos7.x都可以，但是我推荐用centos7.x安装，不知道为啥，感觉centos7.x支持更好，centos6.x装完经常会出问题。
还有一点需要说下就是，ELK各个组件版本要一致，官网要求的！
在一个就是安装顺序，为的是确保每个组件相互调用时都能正常运行：
1、Elasticsearch
-
X-Pack for Elasticsearch
Kibana
-
X-Pack for Kibana
LogstashBeatsElasticsearch Hadoop
安装Elasticsearch
1、导入Elasticsearch安装包PGP Key
|
1
2
|
rpm --
import
https:
//artifacts
.elastic.co
/GPG-KEY-elasticsearch
|
2、创建yum源
|
1
2
3
4
5
6
7
8
9
10
|
[root@localhost ~]
# cat >> /etc/yum.repos.d/elasticsearch.repo <<EOF
> [elasticsearch-5.x]
> name=Elasticsearch repository
for
5.x packages
> baseurl=https:
//artifacts
.elastic.co
/packages/5
.x
/yum
> gpgcheck=1
> gpgkey=https:
//artifacts
.elastic.co
/GPG-KEY-elasticsearch
> enabled=1
> autorefresh=1
>
type
=rpm-md
> EOF
|
3、安装、启动Elasticsearch进程并开机启动
|
1
2
3
4
|
[wangpeng@localhost ~]$
sudo
yum
install
elasticsearch
[wangpeng@localhost ~]$
sudo
/bin/systemctl
daemon-reload
[wangpeng@localhost ~]$
sudo
/bin/systemctl
enable
elasticsearch.service
[wangpeng@localhost ~]$
sudo
systemctl start elasticsearch.service
|
4、检查Elasticsearch是否已经启动
查看9200、9300是否已经启动
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
[wangpeng@localhost ~]$ curl http:
//localhost
:9200
{
"name"
:
"F5Mw8Pp"
,
"cluster_name"
:
"elasticsearch"
,
"cluster_uuid"
:
"zVEeXtPNTaeH-TKah7Buzw"
,
"version"
: {
"number"
:
"5.4.0"
,
"build_hash"
:
"780f8c4"
,
"build_date"
:
"2017-04-28T17:43:27.229Z"
,
"build_snapshot"
:
false
,
"lucene_version"
:
"6.5.0"
},
"tagline"
:
"You Know, for Search"
}
|
5、配置Elasticsearch
rpm包配置文件在/etc/elasticsearch下面的elasticsearch.yml
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
vim
/etc/elasticsearch/elasticsearch
.yml
cluster.name: elasticsearch-
test
node.name: node-1
path.data:
/var/lib/elasticsearch
path.logs:
/var/log/elasticsearch
network.host: 0.0.0.0
[wangpeng@localhost ~]$
sudo
systemctl restart elasticsearch.service
[wangpeng@localhost ~]$ curl http:
//localhost
:9200
{
"name"
:
"node-1"
,
"cluster_name"
:
"elasticsearch-test"
,
"cluster_uuid"
:
"zVEeXtPNTaeH-TKah7Buzw"
,
"version"
: {
"number"
:
"5.4.0"
,
"build_hash"
:
"780f8c4"
,
"build_date"
:
"2017-04-28T17:43:27.229Z"
,
"build_snapshot"
:
false
,
"lucene_version"
:
"6.5.0"
},
"tagline"
:
"You Know, for Search"
}
|
6、将/etc/elasticsearch/配置拷贝到/usr/share/elasticsearch/config下面
|
1
2
3
4
|
[wangpeng@localhost ~]$
sudo
mkdir
/usr/share/elasticsearch/config
[wangpeng@localhost ~]$
sudo
ln
-sf
/etc/elasticsearch/
*
/usr/share/elasticsearch/config/
[wangpeng@localhost ~]$
sudo
chown
-R elasticsearch:elasticsearch
/usr/share/elasticsearch
[wangpeng@localhost ~]$
sudo
systemctl restart elasticsearch.service
|
注意：这一点好多人不会注意，因为你不修复也不会启动失败，但是就是写不进数据进去，这个坑好久才发现，看下日志会报错，但是却能启动，我也是服了！~
7、装个head插件
这个插件5.X官网不再支持了，插件命令没有了，因为它有自己x-pack插件了，但是我装了x-pack发现着实让人吐血，有安全认证方面的问题，导致elk各种问题出现，目前还没研究明白，时间不充裕。
这个head插件我是直接抄的网上大神制作，略有改动。
7.1、下载并配置nodejs
由于head插件本质上还是一个nodejs的工程，因此需要安装node，使用npm来安装依赖的包。（npm可以理解为maven）
去官网下载nodejs，https://nodejs.org/en/download/
|
1
2
3
4
5
6
|
wget https:
//nodejs
.org
/dist/v8
.1.1
/node-v8
.1.1-linux-x64.
tar
.xz
tar
xf node-v8.1.1-linux-x64.
tar
.xz
mv
node-v8.1.1-linux-x64
/usr/local/node
chown
-R elasticsearch:elasticsearch
/usr/local/node
ln
-sf
/usr/local/node/bin/node
/usr/bin/node
ln
-sf
/usr/local/node/bin/npm
/usr/bin/npm
|
7.2、安装grunt
|
1
2
|
npm
install
-g grunt-cli
ln
-sf
/usr/local/node/bin/grunt
/usr/bin/grunt
cd
/var/lib/elasticsearch
|
7.3、下载、安装并配置head
|
1
2
3
4
5
6
|
yum -y
install
git
cd
/var/lib/elasticsearch
git clone git:
//github
.com
/mobz/elasticsearch-head
.git
chown
-R elasticsearch:elasticsearch elasticsearch-
head
/
cd
elasticsearch-
head
/
npm
install
|
7.4、配置head文件
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
[root@elk-node1 ~]
# cd /var/lib/elasticsearch/elasticsearch-head/
vim Gruntfile.js
connect: {
server: {
options: {
port: 9100,
hostname
:
"0.0.0.0"
,
base:
'.'
,
keepalive:
true
}
}
}
[root@elk-node1 elasticsearch-
head
]
# cd _site/
[root@elk-node1 _site]
# vim app.js
|
把localhost修改成你es的服务器地址:
|
1
|
this.base_uri = this.config.base_uri || this.prefs.get(
"app-base_uri"
) ||
"http://10.10.10.10:9200"
;
|
7.5、启动head插件
|
1
|
grunt server &
|
安装Kibana
1、yum安装Kibana
|
1
|
[wangpeng@localhost ~]$
sudo
yum
install
kibana
|
rpm包配置文件在/etc/kibana下面的kibana.yml
/etc/kibana/kibana.yml
2、配置Kibana文件
|
1
2
3
4
|
[wangpeng@localhost ~]$ vim
/etc/kibana/kibana
.yml
server.port: 5601
server.host:
"0.0.0.0"
elasticsearch.url:
"http://localhost:9200"
|
3、启动并设置开机启动
|
1
2
3
|
[wangpeng@localhost ~]$
sudo
systemctl
enable
kibana.service
Created
symlink
from
/etc/systemd/system/multi-user
.target.wants
/kibana
.service to
/etc/systemd/system/kibana
.service
[wangpeng@localhost ~]$
sudo
systemctl start kibana.service
|
安装Logstash
1、yum安装Logstash
|
1
2
3
4
5
|
[wangpeng@localhost ~]$
sudo
yum -y
install
logstash
[wangpeng@localhost ~]$
sudo
systemctl start logstash.service
[wangpeng@localhost ~]$
sudo
ln
-s
/usr/share/logstash/bin/logstash
/usr/bin/logstash
[wangpeng@localhost ~]$
sudo
-u logstash sh -c
'mkdir -pv /usr/share/logstash/config'
[wangpeng@localhost ~]$
sudo
-u logstash sh -c
'ln -s /etc/logstash/* /usr/share/logstash/config/'
|
2、测试Logstash是否能正常运行
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
[wangpeng@localhost ~]$
sudo
logstash -e
'input {stdin{}}output { stdout{}}'
hello world
2017-06-02T07:14:13.130Z localhost hello world
[wangpeng@localhost ~]$
sudo
logstash -e 'input {stdin{}}output {
stdout{codec=>rubydebug}}'
hello world
The stdin plugin is now waiting
for
input:
{
"@timestamp"
=> 2017-06-02T07:17:44.053Z,
"@version"
=>
"1"
,
"host"
=>
"localhost"
,
"message"
=>
"hello world"
}
|
3、写个测试文件，测试一下es是否能够接受数据
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
[wangpeng@localhost ~]$ vim
/etc/logstash/conf
.d
/test
.conf
input{
stdin{}
}
output{
elasticsearch{
hosts =>
"127.0.0.1:9200"
index =>
"test-messages-%{+YYYY.MM.dd}"
}
}
[wangpeng@localhost ~]$ logstash -f
/etc/logstash/conf
.d
/test
.conf -t
Sending Logstash's logs to
/var/log/logstash
which
is now configured via log4j2.properties
Configuration OK
[wangpeng@localhost ~]$ logstash -f
/etc/logstash/conf
.d
/test
.conf
Sending Logstash's logs to
/var/log/logstash
which
is now configured via log4j2.properties
The stdin plugin is now waiting
for
input:
hello world
this is
test
message
study logstash
|
4、Kibana里添加该索引（测试），只要es里面能产生索引，Kibana就能加在上去
安装调试先到这里。
