Logging in Python

Python logging is a useful tool that can be used to log and trace events that take place during the execution of your program. As an alternative to using print statements to debug code, the Python logging module enables you to capture detailed information, such as errors, warnings, or custom messages. The logs give the developer a precise account of the execution events to help them understand what went wrong or right during development, which makes debugging and maintaining code easier. In this blog, we shall discuss the functionality of Python logging, its levels, handlers, and examples of its practical use in detail.

Table of Contents:

• What is Logging in Python?
• Why Use Logging Instead of Print Statements?
• How Python Logging Works?
• Python Logging Levels
• Python Logging Classes and Functions
• Configuring Logging in Python
• Common Logging Handlers
• Logging Exceptions in Python
• Python Logging Examples
• Formatting and Filtering Logs in Python
• Custom Logger and Filters
• Best Practices for Python Logging
• Common Mistakes To Avoid In Python Logging
• Real-World Use Cases
• Conclusion

What is Logging in Python?

In Python, logging is a mechanism for recording events, messages, and errors that occur during program execution. Rather than using print statements, developers use the internal logging module to monitor the flow and behaviour of the applications. It assists in debugging, monitoring, and performance analysis of the system. Python logging is able to log to files, to the console, or to other external systems to enhance issue tracking and maintenance.

Why Use Logging Instead of Print Statements?

Print() can be useful for a quick check or debugging, but it is less useful in large projects. The Python logging module offers a structured, flexible, and scalable way to write messages, while allowing you to specify a logging level (INFO, DEBUG, ERROR, etc.). The logging framework is more efficient compared to using print statements because logs can be easily written to files, rotated, and organized by importance. Using logging in Python will give you enhanced traceability, cleaner code, and improved tracking of issues throughout modules or production environments.

Unlock Python: Power Up Your Programming Skills

Dive into real-world coding projects and become confident in writing clean, efficient Python code.

Explore Program

How Python Logging Works?

The logging module of Python provides a standardized framework to log messages produced by your application. Loggers, handlers, formatters, and filters are the four primary components.

• Logger: This is the entry point, and a logger is created in order to log messages. 
• Handler: This describes where the logs are sent to (console, file, etc.).
• Formatter: This controls how the logs are rendered (what format messages are presented, what the timestamp looks like, etc.)
• Filter: The filter determines which logs are allowed to pass through.

These components allow log messages to be properly structured and organized, making it easier to debug, monitor, and maintain applications effectively.

Python Logging Levels

In Python logging, levels are used to define the severity of messages. Each level helps filter logs, which are based on importance. Below is a list of standard logging levels in Python (from lowest to highest):

1. DEBUG: It is used for detailed diagnostic information during development.
2. INFO: This level is used to indicate that things are progressing as expected
3. WARNING:  This message is used to show that something unexpected happened, but not necessarily the failure of the program
4. ERROR: The message indicates a serious problem that has affected the functionality
5. CRITICAL: The message describes a very serious error or one that has caused the program or system to stop running.

Using correct and appropriate Python logging levels helps to maintain cleaner logs and improves debugging efficiency.

Python Logging Classes and Functions

The Python logging module is used to provide a flexible framework for managing log messages through various classes and functions. These components are used to define how, where, and what to log.

1. Logger Class

The logger class is the main component of the Python logging module. It allows developers to create log messages at different severity levels, such as DEBUG, INFO, WARNING, ERROR, and CRITICAL. By using methods like logger.info() or logger.error(), you can record runtime information, and each logger can be configured independently, which makes it easy to separate logs for different parts of your application.

2. Handler Class

The Handler class determines the destination of your log messages. For example, you may use a StreamHandler to show logs in the console or a FileHandler to write logs to a file. Handlers are effective when you want to log to two destinations simultaneously. For example, you can show critical errors in the terminal while saving detailed logs to a text file.

3. Formatter Class

The Formatter class is responsible for the format of the log messages. It specifies what the format should be for each log entry, which could include details such as timestamps, levels, file names, and messages. You can define the output format, using the string pattern to make your log messages easier to read and consistent across your application.

4. Filter Class

The Filter class gives you precise control over which log messages are handled by the logging package. If you have a particular module or want to limit messages to a specific log level, you can use filters to include or exclude those messages from being processed. This allows you to have cleaner logs and only record information that is relevant.

5. Logging Functions

The logging module also has a number of functions to facilitate configuration and message handling. The basicConfig() function allows for a basic configuration of the logging output, while getLogger() returns a logger instance for the application. The exception() function is a useful method for capturing and logging exceptions that include a traceback for ease of debugging.

Configuring Logging in Python

You can configure the logging in three main ways:

• Using basic configuration
• Logging from Multiple Modules
• Logging in Multi-Threaded Programs

Let us explore each of these one by one:

1. Basic Configuration

The easiest option for configuring logging is to simply use the basicConfig() method. It allows you to configure the log level, log format, and output file in one line of code. For example, you can specify that messages that are at level INFO and above should be logged along with their timestamp and message format. This method is ideal for small projects or scripts where advanced configuration is not required.

2. Logging from Multiple Modules

In larger code bases that span multiple files or modules, you might want to use different loggers for each module. By calling getLogger(__name__) inside each module, independent loggers can be created that share a common configuration but generate different output. This retains clarity and traceability in the logs, particularly for debugging a system with multiple components.

3. Logging in Multi-Threaded Programs

Using logging can be tricky in multi-threaded applications because several threads may try to write to the same log file at once. The Python logging module is thread-safe, and it is careful to keep log messages consistent and in the correct order. You can use more specialized handlers, such as QueueHandler and QueueListener, to improve logging performance.

Common Logging Handlers

In Python, logging handlers are used to decide where the log messages will go. The Python logging module is used to provide several built-in handlers to manage different logging outputs like console, files, or system logs. Below are some of the most commonly used handlers.

1. StreamHandler

The StreamHandler is responsible for sending log messages to output streams (e.g., console or terminal). This handler is the default handler when printing logs to the screen. This handler is useful for development and debugging, as you can see the message in real time.

2. FileHandler

The FileHandler will write log messages directly to a file. You can specify the file path and determine if you want to overwrite or append the logs while the program is running each time. This handler is well-suited for applications that require you to historically record activities or error messages over the duration of a program.

3. RotatingFileHandler

The RotatingFileHandler manages the size of log files. When a log file reaches a designated limit, it will create a new file. This handler keeps the log file sizes controlled and limits the disk space needed for logging. This handler is typically used in long-running applications and/or server-based environments.

4. SysLogHandler

The SysLogHandler sends application log messages to the system log service. It will be used in enterprise environments for instances where you collect logs from various applications and monitor them in a centralized logging service.

Logging Exceptions in Python

Once a program has an error, it should be captured and recorded to be accessed during debugging and maintenance. This can be easily done using the Python logging module, which enables you to record the exceptions along with their traceback. Developers can use logging.exception() or logger.exception() instead of making use of the print statements to display errors. This automatically records details like the error message, the exception type, and the line where the exception happened to take place.

Python Logging Examples

Python logging is a module that is user-friendly, allowing the capture of messages, error handling, and applications. To illustrate how to use the module, some useful examples of Python logging can be provided below.

Example 1: Basic Logging Setup

This example demonstrates how to enable the Python logging function, which allows one to log messages on the console.

Output:

Explanation: Here, each message is printed with its log level, which shows the importance of the messages.

Example 2: Logging to a File

This example is used to store the logs in a file instead of printing them on the console using the Python logging module.

Output:

Explanation: Here, all the messages are saved with timestamps for review in the future and debugging.

Example 3: Logging Exceptions

Here, the logging.exception() method is used to log the errors along with the traceback details.

Output:

Explanation: Here, it logs the exact error and shows where the program crashed for easier debugging.

Example 4: Logging with Multiple Levels

This shows how the different Python logging levels can be used to control the message effectively.

Output:

Explanation: Here, the messages appear according to their severity level( minor issues to major issues).

Formatting and Filtering Logs in Python

When you are working with Python’s logging module, raw logs can be messy and hard to read. Formatting and filtering help control how logs look and which logs appear. The Formatter class is used to structure the log messages with details like timestamp, level, and message. The Filter class is used to decide which log records should be processed or ignored.

Example: Custom Log Format and Filter

Output:

Explanation: Here, only the INFO details about the messages are shown because the filter allowed only that level.

Custom Logger and Filters

A default logger in Python suits well with small-scale projects, but in large applications, many loggers are required, one per module or purpose. Custom loggers provide more control, allowing you to specify unique levels of logs, handlers, as well as filters per component of your program.

Output:

Explanation: Here, we have created a custom logger(appLogger) that is used to write the logs to both the console and a file with having consistent format.

Adding Custom Filters

If you want to manage what each logger is used to record, you can also define filters. For example, if you want to log only the warning in one file while keeping debug logs elsewhere.

Explanation: Here, a custom filter is created to allow only warning and higher-level logs (like error or critical) to be recorded in the log file.

Best Practices for Python Logging

1. Set the Right Level: DEBUG, INFO, WARNING, ERROR, and CRITICAL are set levels that have individual purposes and usage. DEBUG provides lots of information, INFO provides general information, and higher levels are used in case of serious problems.
2. Avoid Print: The logging module should be used instead of all the print() calls, which logging offers a time stamp and improved message control.
3. Configuration of Logging: Ensure you have configured your logging setup, which is useful in ensuring you keep your log output clean and steady. 
4. Add Useful Details: Add useful details like time, module name, because it assists in determining the location and time of creation of the module logs.
5. No Sensitive Data: It is recommended that you do not log your sensitive and personal information, such as passwords, API keys, and so on, as this may lead to security issues.

Common Mistakes to Avoid in Python Logging

1. Using Print(): Some developers still use print() for debugging. This results in unclear debugging information and doesn’t support log levels or timestamps. You should always use the logging module instead. 
2. Not Setting Log Level: If you leave your log level set to the default, you will miss important messages. You should define a log level, likely INFO or DEBUG, at the beginning of the code. 
3. Ignoring Exception Logging: Errors or exceptions in your code are easily fixed if logs at the time are included. Use logging.exception() inside any except block to include both the exception message and any traceback.
4. Overwriting Log Files: Several developers will write logging information to the same file. This can overwrite old data. Use rotating handlers to keep older logs safe.
5. Logging Too Much Information: Recording every detail can make the log complex and hard to read. Log only the information that is important and helps to find and fix the problem faster.

Real-World Use Cases

1. Web Applications: When developing web applications within frameworks like Django or Flask, logging is useful in tracking user interactions, errors, and slow requests. Developers use logging to quickly debug and enhance performance. 
2. Data Processing Pipelines: Logging is common in data pipelines to document every phase of the pipeline, including steps like data extraction, transformation, and loading. 
3. API Monitoring: With APIs, logging tracks every request and response. Logging can help to monitor the usage, discover when the task has failed, and identify hard bugs in the request pipeline.
4. Background Tasks: Many applications that utilize schedulers or background workers depend on logging tasks to document the results of the task. Logging will allow you to check if all of the tasks succeed or not.
5. Security Auditing: Logging helps keep systems secure. It is useful to document failed unauthorized access attempts and to catch other anomalies in usage.

Start Coding in Python for Free: No Experience Needed!

Begin writing real Python code through interactive, beginner-friendly modules completely for free.

Explore Program

Conclusion

Python Logging is a utility of Python that effectively allows tracing the behavior of an application and debugging issues. It assists in the documentation of key events, mistakes, and activities of a system without print statements. It has functionality such as level of logging, handlers, and formatting to enable developers to monitor the performance and troubleshoot easily in real time. Understanding the concepts, functions, and best practices will help to implement the Python Logging function effectively.

Take your skills to the next level by enrolling in the Python Course today and gaining hands-on experience. Also, prepare for job interviews with Python Interview Questions prepared by industry experts.

Logging in Python – FAQs