Problems

I was facing two problems with my Logstash setup

• Logstash service constantly required manual restarts (no longer indexing, hanging process)
• Local log files were filling up my root partition

Symptom

Logstash failed to:

• Index events in the queue
• Trim its own logs =)

Cause

Indexing

I’m not sure why it was failing to index. I didn’t have time to learn why and opted for the lazy approach to fixing a non-native service i didn’t know much about

“who cares let’s restart the service every night and see if my problem disappears”

Log filling up the disk

The log failed to write anything else because the disk was full

Resolving the problem

Rotate the Logstash log files daily (and restart the indexing service at the same time)

Implementation Specific Information

My configuration was tailored to the way I set up logstash. You may need to change a few things (see below)

What you need to provide

In order to configure the logrotate.d script for your environment you will need to know the following information

1. Path to logstash log file

2. Command to:

   • Start the logstash indexer service
   • Stop the logstash indexer service

Values I used

Below are the values I used for the environment specific info

1. Path to logstash log file

   /var/log/logstash/*.log
   

2. Command to:

   • Start the logstash indexer service

     start logstash-central
     

   • Stop the logstash indexer service

     stop logstash-central
     

Script that I used

# create the logrotate file for the logstash indexer

cat <<'EOF' > /etc/logrotate.d/logstash-indexer
# filename: /etc/logrotate.d/logstash-indexer
# Invoke Manually: /usr/sbin/logrotate /etc/logrotate.conf
/var/log/logstash/*.log{
    daily
    size=50M
    missingok
    rotate 2
    compress
    notifempty
    copytruncate
     prerotate
        stop logstash-central
     endscript

     postrotate
        start logstash-central
     endscript
}
EOF

# See how big the log is before you rotate
ls -lh /var/log/logstash/central.log

# Manually invoke logrotate
/usr/sbin/logrotate /etc/logrotate.conf

# See how big the log is after you rotate
ls -lh /var/log/logstash/central.log