Solutions for the log management, internship 2014.
This is to work on Windows.
- Logstash (1.4.2)
- Elasticsearch (1.2.1)
- Kibana (3.1.0)
- Java (1.7.0)
- Ruby (1.7.0)
- A centralized server that collects log files.
- Log Parser 2.2 (Microsoft)
This will present you some tips to improve your log collection.
The curator is by untergeek.
The windows scripts were written by me thanks to help found online.
See the wiki for more information on what will be added next.
Auto delete of indices in ElasticSearch
You can find it here: https://github.com/elasticsearch/curator
- Python for Windows
- IDLE (to edit .py files)
- curator_sauv.py : the script to execute daily to cleanse indices. (indice name format: DD-MM-YYYY)
- curator.py : the script to execute daily to cleanse indices. (indice name format: YYYY-MM-DD)
- curator_base.py : the script to execute daily to cleanse indices. (indice name format: logstash-YYYY-MM-DD)
To delete indices older than N days: from the curator directory ~ curator delete --older-than N
Note : the -p "" is not necessary for the curator.py script because it has been modified to forget any prefix to suit our needs.
At any time, curator -h or curator COMMAND -h Support: https://github.com/elasticsearch/curator
(c) @untergeek
I don't own elasticsearch-curator, I simply modified a bit the python code to adapt it to my needs.
This allows compression with 7za (7-zip command line executable) http://downloads.sourceforge.net/sevenzip/7za920.zip to zip in 7z your folders if they are 90 days old. (you can change this in the batch script)
This asks the user running the script for parameters, and will extract the correct day selected, and unzip it into a directory chosent by the user.
This script uses Log Parser 2.2 to convert .evt in .tsv to be taken into account by Logstash. You just have to change the paths and you're good to go :) This is useful isf you already have all your logs at the same spot, and it will prevent you from having to get them at the source (meaning on each server, which is redundant...).
This script is to be run every day to check for new files to convert (with a scheduled task.) This allows the automatization of the conversion of evt windows log events to humand-readable and more importantly, logstash-readable files.
A newly created template to help create predefined mappings for indices, that will be created automatically. Check this link for more information on template.