filebeat+redis+logstash+es收集nginx日志

nginx日志输出部分省略了，只写主要的部分

1、filebeat配置

[[email protected] filebeat]# cat filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /data/openresty/nginx/logs/*.log
  tags: ["access"]
fields:
  service: "saas-activity"
fields_under_root: true
output.redis:
  hosts: ["172.2.2.1:6379"]   #输出到redis的机器
  password: "[email protected]"  #redis密码
  key: "nginx-log"  #redis中日志数据的key值ֵ
  db: 1   #redis库
  timeout: 5
[[email protected] filebeat]#

2、logstash配置

logstash配置分2个文件配置，分别是input和output配置文件

[[email protected] conf.d]$ cat input.conf
input {
      redis {
              host => "172.2.2.1"
              port => "6379"
              password => "[email protected]"
              db => "1"
              data_type => "list"
              key => "nginx-log"
      }
}
[[email protected] conf.d]$

[[email protected] conf.d]$ cat output.conf
filter {
    json {
        #取哪个字段的值
        source => "message"
        #想要删除的字段
        #remove_field => ["host.name"]
    }
}
output {
      elasticsearch {
              hosts => ["172.2.2.1:9200"]
              index => "nginx-log-%{+YYYY-MM-dd}"
              user => "elastic"
              password => "[email protected]"
      }
}
[[email protected] conf.d]$ 

其他中间件按照正常配置就可以了，配置完成后logstash会去redis取日志，被消费的日志会自动从redis删除