Okay, let’s talk about getting NATS logs under control.
The Log Headache
So, I was running NATS for a bit, handling messaging between different parts of our system. It worked great, really solid. But after a while, I noticed the server disk space was slowly disappearing. Took a peek, and wouldn’t you know it, the NATS log file was getting huge. Like, gigabytes huge. Just one massive file, making it a pain to even open, let alone find anything specific if something went wrong.
I knew I needed to do something about it. You can’t just let logs grow forever, right? That’s asking for trouble. Either the disk fills up completely, or searching logs becomes impossible.
Figuring Out Rotation
My first thought was, surely NATS has something built-in for this. So I went digging into the configuration file. Found options for where to put the log file (`log_file`) and how much detail to log (`debug`, `trace`), but couldn’t immediately spot obvious settings for automatic rotation based on size or time in the version I was using back then.
There was mention of sending signals to the server, like `nats-server –signal reopen`, which sounded like it could be part of a solution, maybe telling NATS to close and reopen its log file handle. That got me thinking.
If NATS itself wasn’t going to handle the whole rotation schedule, moving files, compressing them, etc., then the next logical step was to use a system tool. On Linux, the go-to tool for this stuff is almost always `logrotate`.
Getting `logrotate` Involved
So, the plan became: let `logrotate` handle the actual rotation (moving the current log file, compressing old ones, deleting the really old ones), and then just make sure NATS knows to start writing to a fresh file afterwards.
Here’s roughly what I did:
- Created a `logrotate` config file: Went into `/etc/logrotate.d/` and made a new file, let’s call it `nats`.
- Configured the basics: Inside that file, I put the path to my NATS log file. Told `logrotate` how I wanted it handled. Something like this:
- Rotate daily.
- Keep maybe 7 or 14 old logs (`rotate 7`).
- Compress the old logs (`compress`).
- Don’t rotate if the log file is empty (`missingok`).
- Don’t freak out if the log file isn’t there (`notifempty`).
- The important bit – telling NATS: This is where that signal comes in. `logrotate` has a `postrotate` script section, which runs after it has done the rotating. In there, I needed to tell the NATS server to reopen its log file. The command `nats-server –signal reopen` seemed designed for exactly this. So I added that command to the `postrotate` section. You might need to figure out the exact command or signal (like `pkill -HUP nats-server` if you prefer) depending on your setup and how you run NATS. The key is, NATS needs to let go of the old file handle and grab the new one.
Making Sure It Worked
To test it without waiting a whole day, you can force `logrotate` to run your config. Something like `logrotate –force /etc/logrotate.d/nats`. I ran that, then checked the log directory. Sure enough, the old `*` was now `*.*` (or similar, depending on exact config), and a new, empty `*` file had appeared. Watched it for a few seconds, and saw NATS writing new log entries into the new file. Success!
Final Thoughts
It just works now. Logs get rotated automatically every day, compressed, and old ones get cleaned up. No more giant log files eating all the disk space. It wasn’t super complicated, mostly just standard `logrotate` stuff with that one extra step of signaling the NATS server.
Sometimes you expect a fancy built-in feature, but falling back on solid system tools like `logrotate` is often easier and just as effective. It’s a good reminder that you don’t always need the application itself to solve every single operational problem; the environment it runs in often has tools ready to help.
