UDN-企业互联网技术人气社区

板块导航

浏览  : 2834
回复  : 0

[运维] 使用ELK(Elasticsearch + Logstash + Kibana) 搭建日志集中分析平台...

[复制链接]
哥屋恩的头像 楼主
发表于 2015-10-28 21:14:14 | 显示全部楼层 |阅读模式

  设置FQDN

  创建SSL证书的时候需要配置FQDN

  1. #修改hostname
  2. cat /etc/hostname
  3. elk

  4. #修改hosts
  5. cat /etc/hosts

  6. 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
  7. ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

  8. 127.0.0.1 10-10-87-19
  9. 10.10.87.19 elk.ooxx.com elk

  10. #刷新环境
  11. hostname -F /etc/hostname

  12. #复查结果
  13. hostname -f
  14. elk.ooxx.com

  15. hostname
  16. elk
复制代码


  服务端

  Java

  1. cat /etc/redhat-release
  2. CentOS release 6.5 (Final)

  3. yum install java-1.7.0-openjdk
  4. java -version

  5. java version "1.7.0_85"
  6. OpenJDK Runtime Environment (rhel-2.6.1.3.el6_6-x86_64 u85-b01)
  7. OpenJDK 64-Bit Server VM (build 24.85-b03, mixed mode)
复制代码


  Elasticsearch

  1. #下载安装
  2. wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.noarch.rpm
  3. yum localinstall elasticsearch-1.7.1.noarch.rpm

  4. #启动相关服务
  5. service elasticsearch start
  6. service elasticsearch status

  7. #查看Elasticsearch的配置文件
  8. rpm -qc elasticsearch

  9. /etc/elasticsearch/elasticsearch.yml
  10. /etc/elasticsearch/logging.yml
  11. /etc/init.d/elasticsearch
  12. /etc/sysconfig/elasticsearch
  13. /usr/lib/sysctl.d/elasticsearch.conf
  14. /usr/lib/systemd/system/elasticsearch.service
  15. /usr/lib/tmpfiles.d/elasticsearch.conf

  16. #查看端口使用情况
  17. netstat -nltp

  18. Active Internet connections (only servers)
  19. Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
  20. tcp        0      0 0.0.0.0:9200                0.0.0.0:*                   LISTEN      1765/java           
  21. tcp        0      0 0.0.0.0:9300                0.0.0.0:*                   LISTEN      1765/java           
  22. tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1509/sshd           
  23. tcp        0      0 :::22                       :::*                        LISTEN      1509/sshd  

  24. #测试访问
  25. curl -X GET http://localhost:9200/
复制代码


  Kibana

  1. #下载tar包
  2. wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz
  3. #解压
  4. tar zxf kibana-4.1.1-linux-x64.tar.gz -C /usr/local/
  5. cd /usr/local/
  6. mv kibana-4.1.1-linux-x64 kibana

  7. #创建kibana服务
  8. vi /etc/rc.d/init.d/kibana

  9. #!/bin/bash
  10. ### BEGIN INIT INFO
  11. # Provides:          kibana
  12. # Default-Start:     2 3 4 5
  13. # Default-Stop:      0 1 6
  14. # Short-Description: Runs kibana daemon
  15. # Description: Runs the kibana daemon as a non-root user
  16. ### END INIT INFO

  17. # Process name
  18. NAME=kibana
  19. DESC="Kibana4"
  20. PROG="/etc/init.d/kibana"

  21. # Configure location of Kibana bin
  22. KIBANA_BIN=/usr/local/kibana/bin

  23. # PID Info
  24. PID_FOLDER=/var/run/kibana/
  25. PID_FILE=/var/run/kibana/$NAME.pid
  26. LOCK_FILE=/var/lock/subsys/$NAME
  27. PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN
  28. DAEMON=$KIBANA_BIN/$NAME

  29. # Configure User to run daemon process
  30. DAEMON_USER=root
  31. # Configure logging location
  32. KIBANA_LOG=/var/log/kibana.log

  33. # Begin Script
  34. RETVAL=0

  35. if [ `id -u` -ne 0 ]; then
  36.         echo "You need root privileges to run this script"
  37.         exit 1
  38. fi

  39. # Function library
  40. . /etc/init.d/functions

  41. start() {
  42.         echo -n "Starting $DESC : "

  43. pid=`pidofproc -p $PID_FILE kibana`
  44.         if [ -n "$pid" ] ; then
  45.                 echo "Already running."
  46.                 exit 0
  47.         else
  48.         # Start Daemon
  49. if [ ! -d "$PID_FOLDER" ] ; then
  50.                         mkdir $PID_FOLDER
  51.                 fi
  52. daemon --user=$DAEMON_USER --pidfile=$PID_FILE $DAEMON 1>"$KIBANA_LOG" 2>&1 &
  53.                 sleep 2
  54.                 pidofproc node > $PID_FILE
  55.                 RETVAL=$?
  56.                 [[ $? -eq 0 ]] && success || failure
  57. echo
  58.                 [ $RETVAL = 0 ] && touch $LOCK_FILE
  59.                 return $RETVAL
  60.         fi
  61. }

  62. reload()
  63. {
  64.     echo "Reload command is not implemented for this service."
  65.     return $RETVAL
  66. }

  67. stop() {
  68.         echo -n "Stopping $DESC : "
  69.         killproc -p $PID_FILE $DAEMON
  70.         RETVAL=$?
  71. echo
  72.         [ $RETVAL = 0 ] && rm -f $PID_FILE $LOCK_FILE
  73. }

  74. case "$1" in
  75.   start)
  76.         start
  77. ;;
  78.   stop)
  79.         stop
  80.         ;;
  81.   status)
  82.         status -p $PID_FILE $DAEMON
  83.         RETVAL=$?
  84.         ;;
  85.   restart)
  86.         stop
  87.         start
  88.         ;;
  89.   reload)
  90. reload
  91. ;;
  92.   *)
  93. # Invalid Arguments, print the following message.
  94.         echo "Usage: $0 {start|stop|status|restart}" >&2
  95. exit 2
  96.         ;;
  97. esac

  98. #修改启动权限
  99. chmod +x /etc/rc.d/init.d/kibana

  100. #启动kibana服务
  101. service kibana start
  102. service kibana status

  103. #查看端口
  104. netstat -nltp

  105. Active Internet connections (only servers)
  106. Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
  107. tcp        0      0 0.0.0.0:9200                0.0.0.0:*                   LISTEN      1765/java           
  108. tcp        0      0 0.0.0.0:9300                0.0.0.0:*                   LISTEN      1765/java           
  109. tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1509/sshd           
  110. tcp        0      0 0.0.0.0:5601                0.0.0.0:*                   LISTEN      1876/node           
  111. tcp        0      0 :::22                       :::*                        LISTEN      1509/sshd
复制代码


  Logstash

  1. #下载rpm包
  2. wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm
  3. #安装
  4. yum localinstall logstash-1.5.4-1.noarch.rpm

  5. #设置ssl,之前设置的FQDN是elk.ooxx.com
  6. cd /etc/pki/tls
  7. #openssl req -x509  -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.crt -subj /CN=logstash.example.com
  8. openssl req -subj '/CN=elk.ooxx.com/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt


  9. #创建一个01-logstash-initial.conf文件
  10. cat > /etc/logstash/conf.d/01-logstash-initial.conf << EOF
  11. input {
  12.   lumberjack {
  13.     port => 5000
  14.     type => "logs"
  15.     ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
  16.     ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  17.   }
  18. }


  19. filter {
  20.   if [type] == "syslog" {
  21.     grok {
  22.       match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
  23.       add_field => [ "received_at", "%{@timestamp}" ]
  24.       add_field => [ "received_from", "%{host}" ]
  25.     }
  26.     syslog_pri { }
  27.     date {
  28.       match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
  29.     }
  30.   }
  31. }

  32. output {
  33.   elasticsearch { host => localhost }
  34.   stdout { codec => rubydebug }
  35. }
  36. EOF

  37. #启动logstash服务
  38. service logstash start
  39. service logstash status

  40. #查看5000端口
  41. netstat -nltp

  42. Active Internet connections (only servers)
  43. Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
  44. tcp        0      0 0.0.0.0:9200                0.0.0.0:*                   LISTEN      1765/java           
  45. tcp        0      0 0.0.0.0:9300                0.0.0.0:*                   LISTEN      1765/java           
  46. tcp        0      0 0.0.0.0:9301                0.0.0.0:*                   LISTEN      2309/java           
  47. tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1509/sshd           
  48. tcp        0      0 0.0.0.0:5601                0.0.0.0:*                   LISTEN      1876/node           
  49. tcp        0      0 0.0.0.0:5000                0.0.0.0:*                   LISTEN      2309/java           
  50. tcp        0      0 :::22                       :::*                        LISTEN      1509/sshd  


  51. #启动服务
  52. service logstash-forwarder start
  53. service logstash-forwarder status

  54. #访问Kibana,Time-field name 选择 @timestamp


  55. #增加节点和客户端配置一样,注意同步证书

  56. /etc/pki/tls/certs/logstash-forwarder.crt
复制代码


  客户端

  Logstash Forwarder

  1. #登陆到客户端,安装Logstash Forwarder
  2. wget https://download.elastic.co/logstash-forwarder/binaries/logstash-forwarder-0.4.0-1.x86_64.rpm
  3. yum localinstall logstash-forwarder-0.4.0-1.x86_64.rpm

  4. #查看logstash-forwarder的配置文件位置
  5. rpm -qc logstash-forwarder
  6. /etc/logstash-forwarder.conf

  7. #备份配置文件
  8. cp /etc/logstash-forwarder.conf /etc/logstash-forwarder.conf.save

  9. #编辑 /etc/logstash-forwarder.conf,需要根据实际情况进行修改

  10. cat > /etc/logstash-forwarder.conf << EOF
  11. {
  12.   "network": {
  13.     "servers": [ "elk.ooxx.com:5000" ],

  14.     "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",

  15.     "timeout": 15
  16.   },

  17.   "files": [
  18.     {
  19.       "paths": [
  20.         "/var/log/messages",
  21.         "/var/log/secure"
  22.       ],

  23.       "fields": { "type": "syslog" }
  24.     }
  25.   ]
  26. }
复制代码


  配置Nginx日志策略

  1. #修改客户端配置
  2. vi /etc/logstash-forwarder.conf

  3. {
  4.   "network": {
  5.     "servers": [ "elk.ooxx.com:5000" ],

  6.     "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",

  7.     "timeout": 15
  8.   },

  9.   "files": [
  10.     {
  11.       "paths": [
  12.         "/var/log/messages",
  13.         "/var/log/secure"
  14.       ],
  15.       "fields": { "type": "syslog" }
  16.     }, {
  17.       "paths": [
  18.         "/app/local/nginx/logs/access.log"
  19.       ],
  20.       "fields": { "type": "nginx" }
  21.     }
  22.   ]
  23. }

  24. #服务端增加patterns
  25. mkdir /opt/logstash/patterns
  26. vi /opt/logstash/patterns/nginx

  27. NGUSERNAME [a-zA-Z\.\@\-\+_%]+
  28. NGUSER %{NGUSERNAME}
  29. NGINXACCESS %{IPORHOST:remote_addr} - - \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATH:path}(?:%{URIPARAM:param})? HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}

  30. #官网pattern的debug在线工具
  31. https://grokdebug.herokuapp.com/

  32. #修改logstash权限
  33. chown -R logstash:logstash /opt/logstash/patterns

  34. #修改服务端配置
  35. vi /etc/logstash/conf.d/01-logstash-initial.conf

  36. input {
  37.   lumberjack {
  38.     port => 5000
  39.     type => "logs"
  40.     ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
  41.     ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  42.   }
  43. }


  44. filter {
  45.   if [type] == "syslog" {
  46.     grok {
  47.       match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
  48.       add_field => [ "received_at", "%{@timestamp}" ]
  49.       add_field => [ "received_from", "%{host}" ]
  50.     }
  51.     syslog_pri { }
  52.     date {
  53.       match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
  54.     }
  55.   }
  56.   if [type] == "nginx" {
  57.     grok {
  58.        match => { "message" => "%{NGINXACCESS}" }
  59.     }
  60.   }
  61. }

  62. output {
  63.   elasticsearch { host => localhost }
  64.   stdout { codec => rubydebug }
  65. }
复制代码


  其它注意事项

  修改kibana端口

  1. #编辑kibana.yaml
  2. vi /usr/local/kibana/config/kibana.yml


  3. # Kibana is served by a back end server. This controls which port to use.
  4. #port: 5601
  5. port: 80

  6. # The host to bind the server to.
  7. host: "0.0.0.0"

  8. # The Elasticsearch instance to use for all your queries.
  9. elasticsearch_url: "http://localhost:9200"

  10. # preserve_elasticsearch_host true will send the hostname specified in `elasticsearch`. If you set it to false,
  11. # then the host you use to connect to *this* Kibana instance will be sent.
  12. elasticsearch_preserve_host: true

  13. # Kibana uses an index in Elasticsearch to store saved searches, visualizations
  14. # and dashboards. It will create a new index if it doesn't already exist.
  15. kibana_index: ".kibana"

  16. # If your Elasticsearch is protected with basic auth, this is the user credentials
  17. # used by the Kibana server to perform maintence on the kibana_index at statup. Your Kibana
  18. # users will still need to authenticate with Elasticsearch (which is proxied thorugh
  19. # the Kibana server)
  20. # kibana_elasticsearch_username: user
  21. # kibana_elasticsearch_password: pass

  22. # If your Elasticsearch requires client certificate and key
  23. # kibana_elasticsearch_client_crt: /path/to/your/client.crt
  24. # kibana_elasticsearch_client_key: /path/to/your/client.key

  25. # If you need to provide a CA certificate for your Elasticsarech instance, put
  26. # the path of the pem file here.
  27. # ca: /path/to/your/CA.pem

  28. # The default application to load.
  29. default_app_id: "discover"

  30. # Time in milliseconds to wait for elasticsearch to respond to pings, defaults to
  31. # request_timeout setting
  32. # ping_timeout: 1500

  33. # Time in milliseconds to wait for responses from the back end or elasticsearch.
  34. # This must be > 0
  35. request_timeout: 300000

  36. # Time in milliseconds for Elasticsearch to wait for responses from shards.
  37. # Set to 0 to disable.
  38. shard_timeout: 0

  39. # Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
  40. # startup_timeout: 5000

  41. # Set to false to have a complete disregard for the validity of the SSL
  42. # certificate.
  43. verify_ssl: true

  44. # SSL for outgoing requests from the Kibana Server (PEM formatted)
  45. # ssl_key_file: /path/to/your/server.key
  46. # ssl_cert_file: /path/to/your/server.crt

  47. # Set the path to where you would like the process id file to be created.
  48. # pid_file: /var/run/kibana.pid

  49. # If you would like to send the log output to a file you can set the path below.
  50. # This will also turn off the STDOUT log output.
  51. # log_file: ./kibana.log

  52. # Plugins that are included in the build, and no longer found in the plugins/ folder
  53. bundled_plugin_ids:
  54. - plugins/dashboard/index
  55. - plugins/discover/index
  56. - plugins/doc/index
  57. - plugins/kibana/index
  58. - plugins/markdown_vis/index
  59. - plugins/metric_vis/index
  60. - plugins/settings/index
  61. - plugins/table_vis/index
  62. - plugins/vis_types/index
  63. - plugins/visualize/index
复制代码


  JVM调优

  1. #修改elasticsearch.in.sh
  2. vi /usr/share/elasticsearch/bin/elasticsearch.in.sh

  3. if [ "x$ES_MIN_MEM" = "x" ]; then
  4.     ES_MIN_MEM=1g
  5. fi
  6. if [ "x$ES_MAX_MEM" = "x" ]; then
  7.     ES_MAX_MEM=1g
复制代码
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关于我们
联系我们
  • 电话:010-86393388
  • 邮件:udn@yonyou.com
  • 地址:北京市海淀区北清路68号
移动客户端下载
关注我们
  • 微信公众号:yonyouudn
  • 扫描右侧二维码关注我们
  • 专注企业互联网的技术社区
版权所有:用友网络科技股份有限公司82041 京ICP备05007539号-11 京公网网备安1101080209224 Powered by Discuz!
快速回复 返回列表 返回顶部