ELK (ElasticSearch - LogStash - Kibana) is my favourite stack for managing and analysing server logs for years. Recently, when deploying a log management module for a small company on Amazon Web Service environment, I found that AWS Kinesis can help me achieve the same target with much less effort of server management and configuration. … Continue reading Log analysis with Kinesis – Lambda – ElasticSearch – Kibana pipeline
Tag: data pipeline
Reparsing old log files with Logstash 5
In my current research, a set of log files need to be processed by Logstash before sent to ElasticSearch. After a the first run, I realized that the log format was changed once, which made my Logstash configuration fail in processing half of the log lines. I updated the filter and ran Logstash again, hoping … Continue reading Reparsing old log files with Logstash 5