Things I learned in Linux whilst syslog was exploding

calendarSep 27, 2023 · 2 min read  ·
Share on: linkedincopy

Exploring Linux Log Management

Today, my Linux adventure began with a bang as the syslog on my nginx server went haywire, causing disk space issues. While Linux can be intimidating, I embrace such challenges as opportunities to learn something new.

Finding the Space Hogs

First things first, I needed to identify which files were hogging the most space on my Linux box:

1# List the top 10 largest files/folders in the root directory
2sudo du -aBm / 2>/dev/null | sort -nr | head -n 10

It turns out, the syslog files were throwing a wild party! Assessing Available Disk Space

To gauge how much disk space I had left, I ran:

1df

How much space does each syslog file is consuming?

1du -h syslog*

Clearing Syslog Manually

To regain some space, I decided to clear the syslog file. But be cautious, clearing logs can have consequences:

1# Clear syslog manually
2sudo cat /dev/null > /var/log/syslog

or

1# Another way to clear syslog
2sudo truncate -so /var/log/syslog

The Art of Log Rotation

While cleaning up the syslog, I stumbled upon Linux's built-in log rotation system called logrotate. It automates log management and prevents log files from growing indefinitely.

1# Check logrotate status
2sudo cat /var/lib/logrotate/status

Here's what you need to know about log rotation:

  • Logrotate follows a predefined schedule or file size limit for rotation.
  • You can define the maximum number of log files to keep.
  • General logrotate configuration is in /etc/logrotate.conf, and application-specific settings reside in /etc/logrotate.d/appName.

In my particular case, the primary culprit was the limited disk capacity of our production machine, which was equipped with a mere 30GB of storage space. In a moment of urgency, I opted to clear the syslog file as a temporary measure to regain control. To ensure that we won´t be facing the same issue soon, three steps were taken:

  1. Expanded disk capacity
  2. Changed Logrotate policy to set a max file size of 3GB and max 4 files
  3. Add azure monitor alerts when disk space is lower than 30%

In summary, managing Linux logs can be a bit of an adventure, but with tools like logrotate, you can keep your system running smoothly and your disk space under control.