What Are Structured Logs and How Do They Improve Performance?

Logging information in a structured format for better analysis and processing of log data.

Photo by Kier... in Sight on Unsplash

Structured logging is crucial for understanding your software state and resolving issues efficiently. It tidies up all the event details inside your system, so finding issues and hunting down bugs becomes a smooth process.

What is a Structured Logging Format?

Structured logging organizes system messages and events in a clean, easy-to-read manner, simplifying the process of understanding and troubleshooting your software.

Unlike traditional logs that are basically a jumble of plain text, structured logs employ a well-defined layout like JSON or XML to neatly arrange the information.

Text-based log: “err: user Aiden take an empty order at 2023–01–01T00:00:00Z”

In a JSON log, it’d look like this:

{
  "level": "error" 
  "timestamp": "2023–01–01T00:00:00Z",
  "user": "Aiden",
  "error": "no document found",
  "message": "took an empty order"
}

Thanks to this more organized setup, searching through log data and managing logs through automation become much more straightforward tasks.

This allows you to quickly grasp what’s going on in your system, easily spot issues, and rectify them.

“So structured log is just JSON, right?”

No, structured logs aren’t limited to JSON, even plain text can be structured. Consider these examples:

• “user=Aiden level=error msg=’took an empty order’”

• “Aiden error ‘took an empty order’

Both qualify as structured logs, the main thing is to follow the formatting rules that your team has established and that your logging system can support.

While JSON or field-based formatting is often preferred, it’s not the only way to go.

Searchable and Queryable

In my view, the standout benefit of using a structured logging format is its searchability and query capabilities. With structured logs, you can sift through data by focusing on particular fields or attributes, like the timestamp, log level, or the message itself.

This makes it much simpler to locate specific log entries and troubleshoot issues.

Here’s a quick example to illustrate:

{ app="order-service" } | json | logLevel = "error" | userName="Aiden" 

In this query, I’m hunting for errors tied to user Aiden and logs that match these specific “logLevel” and “userName” fields will show up, making it super simple to zero in on what you’re looking for.

This way, pinpointing specific log entries and sorting out issues becomes almost effortless, even when you’re dealing with more complex scenarios.

Super-High Performance

One of the biggest perks of structured logging is that it’s a lot more efficient for log management systems and other tools to work with.

“Does that mean it’s quicker to parse?”

Well, yes, but there’s more to it and the secret sauce here is “indexing”.

By indexing specific fields, it makes searching and querying log data much faster, especially when dealing with large amounts of log data.

Let me break it down a bit. Imagine you have a log entry like this:

{
  "timestamp": "2022-01-01T12:00:00Z",
  "level": "error",
  "message": "An error occurred while processing a request",
  "user_id": "123456"
}

If you want to scan all logs based on the ‘user_id,’ going through each log to compare that field could take quite some time. Now, if you’ve indexed the ‘user_id’ field, the system sets up a faster way to access that data, drastically speeding up your searches.

And we’re talking about making your search and query operations more than 100 times faster compared to not indexing.

“Can I index all the fields?”

You could, but be cautious.

Indexing isn’t free since it uses up more storage and memory and can slow down how quickly new data is written. So you’ll want to be choosy about which fields to index to keep things running smoothly.

Monitoring

Structured logging has its advantages, and one of them is its utility in creating rich yet straightforward dashboards and reports:

Take, for example, if you want to know something like “how many errors popped up for a specific user like Aiden in the past 10 hours?” or “the total orders Aiden placed”. With structured logs, tracking these specifics is a piece of cake.

Could you do this with unstructured logs? Nope, you’d miss out on this level of detail and convenience.

Wide-range Support

Structured logging is widely supported by many programming languages and frameworks.

For languages like Python, Java, and Go, you’ll find either built-in functionalities or third-party libraries tailored for structured logging. In Go, you’ve got options like logrus, zerolog, and zap, while Python users can turn to structlog.

And let’s not overlook centralized logging systems, platforms like Elasticsearch, Logstash, Kibana, Splunk, as well as Grafana with Loki through Promtail, all support structured logging formats.

This compatibility allows you to effortlessly funnel your structured logs into these systems for subsequent storage, analysis, and monitoring.

---

And, if you want to know more about using logs effectively in a technical way, I’ve put together an article covering the best practices for logging.