When you're troubleshooting an issue in your app or host, you need to have complete log data that you can analyze. An application log provides granular information about events that occur in your application, including errors, user activity, and HTTP requests. Each line in a log includes a timestamp and information about the event, providing vital data that you can use to analyze how your application is performing—and troubleshoot errors when they arise.
Application logs are structured records of events and activities generated by software applications. They capture a wide range of information, including error messages, user interactions, system events, and application-specific data.
These logs are valuable for several reasons. Firstly, they serve as a crucial tool for troubleshooting and diagnosing issues. When something goes wrong, logs provide a historical record of what happened, enabling developers and system administrators to identify the root cause and resolve problems swiftly.
Additionally, logs are essential for auditing and compliance, helping organizations track user activities and system changes. They also play a significant role in performance monitoring, enabling the measurement of key metrics and the identification of areas for improvement.
Overall, application logs are a valuable source of information that supports debugging, security, compliance, and the optimization of software systems.
However, logs are only useful if you’re managing them correctly in the context of the application, and that means handling each stage in the log management lifecycle, including:
In this article, you’ll learn the basics of handling each step in the log management process. To learn more about why logs and log management are important, see What is log management?
Let’s take a look at each stage in the process.
The first step is to generate logs for each service in your application. So how do you do that? There are several options:
Let’s briefly go over each option.
First, many services and technologies automatically generate logs. For instance, if you use an MVC platform like Rails, Django, or .NET, logs are automatically outputted to the terminal in development. Cloud services like AWS and Azure also provide logs by default. You’ll need to take a look at the documentation for each service to determine whether logs are provided by default, whether they need to be turned on, and whether you need to configure them further.
If you’re creating custom frontends and backends, you’ll need to take a closer look at the documentation for the frameworks you’re using and the programming languages themselves. For example, both Ruby and Python have a `Logger` class that you can implement to add custom log messages. As a coding best practice, you should implement error handling within your applications—and you can add custom log messages whether or not the application successfully handles those errors.
Finally, if you’re using a log management solution like New Relic, you can automatically instrument your applications by installing agents in your code that send telemetry data to the platform you’re using. You can also use an open source solution like Prometheus to monitor services and generate log data.
If you’re starting from scratch, don’t reinvent the wheel. Use a solution that can instrument your services for you, and use the logs that your services already provide.
After your services are creating logs—either by default or because you’ve configured them to do so—you should collect them in one place where they can be analyzed, queried, and stored. It is extremely important to centralize your logs. Otherwise, you’ll have a difficult time troubleshooting issues and understanding how your services are interacting with each other. For example, a problem in an upstream service may cause an error in a downstream service—but if you are sending the logs for those services to different places, you won’t be able to correlate and analyze the data to find the root cause of the problem.
If you’re using a platform that provides instrumentation, agents automatically collect and send log data to the platform for you. For services that you don’t instrument, you need to set up log forwarding. This can be a real headache, especially if you need to forward logs from a lot of services. New Relic provides logs in context, which supports automatically forwarding logs without the need to install or maintain third-party software. Otherwise, you’ll need to go through the process of reviewing documentation for each service, setting up log forwarding, and ensuring that logs are properly forwarded to the centralized location you’ve specified.
There’s one other very important thing you need to consider: do any of your services produce events that absolutely shouldn’t be logged? An example is protected health information, which needs to be HIPAA-compliant. You may need additional configuration to ensure that sensitive data isn’t logged.