Logging Best Practices: Proven Techniques for Services

Learn logging best practices for services and optimize your logging strategy with proven techniques.

Photo by Dulcey Lima on Unsplash

Logging might appear to be a small, straightforward task, but don’t let that fool you, it holds significant weight in the context of software development.

Trust me, when done correctly, logging can provide valuable insights into the performance and usage of the application, which can help improve its overall reliability and user experience.

Let’s dive in.

---

1. Logging level

Log levels help sort log messages by their importance or severity, and this sorting makes it easier for developers to spot and act on key events and issues.

Generally, you’ll find 5 core log levels:

1. “debug”: Useful for deep dives into issues, providing detailed info for troubleshooting.

2. “info”: Gives you the big picture about what’s happening in the app, like user activities and system events.

3. “warning”: Flags possible concerns that might need a look, this could be stuff like system slowdowns or events that aren’t breaking things but still need watching.

4. “error”: Here you get details about real problems in the app, complete with info on exceptions and other clues to help you find out what went wrong.

5. “fatal”: This one’s for major critical error that make the app stop working entirely.

“Hey, I’ve also seen a ‘trace’ log level. What’s that about?”

Yeah, the ‘trace’ log level is a thing in some languages, and it’s for really detailed tracking of the code’s flow. It’s even more in-depth than ‘debug’.

You might see entries like:

“Line 10 — /app/package/modules/user/user.go, found user Aiden,
line 16: variable ‘result’ set to ‘result’ + ‘c’, Line 18: function ‘calculateSum’ returned ‘result’ = 10”

It’s a good practice to change the priority of logging level in production, so that we don’t get overwhelmed by debug logs on production, and keep things running smoothly for our team.

2. Structured Logs

For sizable projects, structured logging formats like JSON or CSV are often the go-to, rather than sticking with plain text.

“So what’s this structured logging format all about?”

I’ve touched on the Why and How of Structured Logging in past discussions. In a nutshell, it’s a method to log events and messages in your system in an organized manner. This format helps developers sift through logs more effectively to find what matters most.

What Are Structured Logs and How Do They Improve Performance?

Indexing

Indexing is indeed a helpful tool for sifting through log data, but moderation is key.

Keep in mind that it comes with some downsides like taking up more storage, using more memory, and potentially slowing down the rate at which new log entries are saved. So, use it wisely.

3. What to log?

Just Log What Matters

Logging too many details can make your log files a mess and it’s tough to find what you’re looking for, uses up too much storage. Not to mention, analyzing the data becomes a slow process.

Trust me, I’ve had to scroll through endless JSON logs just to find one piece of info hidden among a lot of useless stuff and it’s annoying.

Log with context

Logging isn’t just about recording data; it’s about knowing the background of that data. That way, you can better understand what’s happening.

So when you log, think about adding:

• Timestamp: knowing when an event took place is crucial, especially for solving time-sensitive issues.

• Service information: mention the service or component that’s sending the log message. For instance, if it’s coming from ‘order-service-0’ service or ‘RetrieveOrderBusiness’ component, you’ll have an easier time sorting things out.

• User information: it’s good to know who is involved and what their situation is, are they banned or are they a high-value customer?

• Request information: include the Request ID and details about the device or browser used, as well as the client version, etc

So, it’s key to add context to your logs while keeping them focused and lean.

Security Comes First

Logging needs a security-first approach so you’ve got to keep sensitive data safe and ensure that no one’s messing around with your logs.

1. Sensitive info: before you even think about logging sensitive data, you better mask or redact it, you don’t want that stuff falling into the wrong hands.

2. Limit access to log data: only people who really need it should be allowed to see your log data or you can set this up with role-based access controls (RBAC) and other security steps.

3. Keep logs for a reasonable time: keep your logs around for a useful, reasonable amount of time. This is not just to meet legal rules, but also to make sure you can look into stuff later if you need to.

When it comes to keeping logs for a reasonable period of time, we also have a technique called log rotation.

4. Log rotation

Log rotation is how you keep your logs manageable and ensure you’re not losing anything important.

“What is log rotation?”

Basically, it’s about moving or storing away old logs so that they don’t get too big or eat up your storage and you aim to keep things manageable while avoiding unnecessary waste of space.

Let’s break down some methods:

1. Compress: if you don’t need the log files right now but might later, just compress them to save room.

2. Archive: for logs you want to keep long-term, move them somewhere else, like another disk or even a remote server.

3. Delete: you remove the entire file from the system, freeing up the disk space it was occupying.

4. Truncate: means that you’re emptying its contents, taking its size back to zero while keeping the file there for new logs, so you don’t have to make a new one.

“When should you do these?”

Typically, log rotation is done based on the size of log files and the amount of disk space available.

Usually, you’d rotate logs based on how big they’re getting or how much storage you’ve got left. But my team?

We do it differently and we don’t sweat storage size.

We simply hold on to logs for one month for the less important services, and for three months for our main operations.