Logstash is a tool for reading logs and send it to the elastic search.Lets say we have object is printed in the log file and we can't write a grok pattern to entire log file. We need to extract certain information from that object. I have log file as below
Above log i gave simple example of object but many case object would be very big and its not possible to write the grok pattern for entire log. So in order to avoid this scenario we need to use macro called GREEDYDATA
I have wrote the grok till http method type post "171.741.544.442 - kurt [28/Jun/2015:01:48:10 -0700] "POST" rest of the part going to greedydata. Which means entire object goes to greedy data. Once greedy data is generated we are applying filter on top of it to get the state.Above example solve the problem of reading the logs in between in logstash and reading objects printed in the log file using logstash.
171.741.544.442 - kurt [28/Jun/2015:01:48:10 -0700] "POST User[name:raja:age:25 Location[city:blr;state:ka;country:india]]]
Above log i gave simple example of object but many case object would be very big and its not possible to write the grok pattern for entire log. So in order to avoid this scenario we need to use macro called GREEDYDATA
%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{GREEDYDATA}state:%{WORD:state}
I have wrote the grok till http method type post "171.741.544.442 - kurt [28/Jun/2015:01:48:10 -0700] "POST" rest of the part going to greedydata. Which means entire object goes to greedy data. Once greedy data is generated we are applying filter on top of it to get the state.Above example solve the problem of reading the logs in between in logstash and reading objects printed in the log file using logstash.