Skip to content

Logging

The Python logging module is a powerful, highly configurable logger for applications. The downside to logging is that it can be a little difficult to wrap your head around at first, and there are many, many, many ways to configure it.

I have settled on using logging.config.dictConfig to configure my logging, and that is how my notes/code are written.

Todo

  • Document formatters
    • Document format strings
    • Document date format strings
  • Document handlers
    • Console/Stream
    • File
      • RotatingFile
      • TimedRotatingFile
    • Socket
  • Document loggers
    • Document adding handlers & formatters
    • Document "pre-defining" loggers for loggers that don't exist yet/haven't been called yet.
    • Document overriding 3rd-party loggers

Configuring logging module

The logging module can be configured a number of ways, most notably with basicConfig(), but my personal favorite is with dictConfig(), which lets you pre-define your entire applications' logging (all current and potential future loggers) in a Python dict. No need to import handlers, formatters, and other miscellaneous classes from the logging module; with dictConfig(), everything is a string or integer!

Shameless-plug

I wrote a module, red-logging to aid with logging boilerplate. This module is specific to my own needs and uses, and may not be suitable for you or your project(s), but can serve as a helpful reference for some patterns.

red-logging does not import any dependencies, it simply organizes some of logging's functionality into classes, context managers, and functions.

Configure with dictConfig()

You can use a custom dict to configure an entire app's logging all in one place.

Warning

When using logging.config.dictConfig(logging_config_dict), be careful to only call dictConfig() once per execution. Calling this method multiple times can lead to instability in your logging as the logging config dicts overwrite each other.

dictConfig() should be called very early in your program's execution. You can put it in your root __init__.py file, or very early in the main.py file (or whatever entrypoint you run). If your project has multiple entrypoints, you can put dictConfig() below your if __name__ == "__main__", but imports may fire log messages that do not get caught before the logger is configured, or they may be missed entirely.

Finally, you can use a setup_logging() function, which you can store in a module and import into whatever entrypoint script you target. This is a configurable and flexible way to manage logging.

dictConfig keys

The following options are available as dict keys for your logging config dict.

Key Description
version an integer indicating the schema version that is being used. If the logging configuration schema changes in the future, the version key will be used to indicate which version of the schema the dictConfig is using. This allows the dictConfig function to handle both current and future schema versions.
formatters a dictionary with each key being a formatter id and its value describing how to configure the corresponding Formatter instance.
filters a dictionary with each key being a filter id and its value describing how to configure the corresponding Filter instance.
handlers a dictionary with each key being a handler id and its value describing how to configure the corresponding Handler instance. All other keys are passed through as keyword arguments to the handler’s constructor.
loggers a dictionary with each key being a logger name and its value describing how to configure the corresponding Logger instance.
root the configuration for the root logger. It’s processed like any other logger, except that the propagate setting is not applicable.
incremental a boolean indicating whether the configuration specified in the dictionary should be merged with any existing configuration, or should replace entirely. Its default value is False, which means that the specified configuration replaces any existing configuration.
disable_existing_loggers a boolean indicating whether any non-root loggers that currently exist should be disabled. If absent, this parameter defaults to True. Its value is ignored when incremental is True.

Logging config dict template

This logging_config_template dict can serve as a base/root for your project. Copy/paste the code and modify it to your needs, adding formatters, handlers, etc.

logging_config dict template
1
2
3
4
5
6
7
8
9
logging_config_template: dict = {
    "version": 1,
    "disable_existing_loggers": False,
    "propagate": True,
    "root": {},
    "formatters": {},
    "handlers": {},
    "loggers": {},
}

Example logging config dict

This is an example of a fully configured logging config dict. It includes a couple of console handlers, one which filters messages to only show CRITICAL and above, a custom format, and a number of loggers.

Note

You only need to configure the root ("") logger once; the configuration below configures te root logger in the "root" dictionary key, as well as in "loggers": {"": {}}. In a real logging configuration dict, you would only do one of these. I personally prefer putting my configuration in the top-level "root" key.

Example logging config dict
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
## Root logger config. Use with .dictConfig(log_config)
log_config: dict = {
    "version":1,
    ## When True, clears all logging configuration.
    #  Best used only on the root logger
    "disable_existing_loggers": True,
    ## Configure the root logger
    "root":{
        ## Set handlers
        "handlers" : ["console"],
        ## Set log level string
        "level": "DEBUG"
    },
    ## Configure handlers
    "handlers":{
        ## Default console logger
        "console":{
            ## Use stdout stream
            "stream": "ext://sys.stdout",
            ## Name of formatter
            "formatter": "std_out",
            ## The logging module to subclass
            "class": "logging.StreamHandler",
            ## Log level for this logger
            "level": "DEBUG"
        },
        ## Only log CRITICAL messages with with `.getLogger("cli")`
        "cli": {
            "console": {
                "level": "CRITICAL",
                "class": "logging.StreamHandler",
                "formatter": "consoleFormatter",
            },
        }
    },
    ## Configure formatters for log messages
    "formatters": {
        ## Set formatting for std_out formatter messages & dates
        "std_out": {
            "format": "%(asctime)s : %(levelname)s : %(module)s : %(funcName)s : %(lineno)d : (Process Details : (%(process)d, %(processName)s), Thread Details : (%(thread)d, %(threadName)s))\nLog : %(message)s",
            "datefmt":"%d-%m-%Y %I:%M:%S"
        }
    },
    ## (Optional) pre-configure logger namespaces.
    #  Example: you have a module named `requests.py`; define
    #  a "requests" logger below, and when `requests.py` uses
    #  `logging.getLogger(__name__)`, it will use the "requests"
    #  logger config.
    #  This can also be used to override module loggers, like `uvicorn`
    "loggers": {
        "": {
            "handlers": ["console"],
            ## Hide all but warning messages on root logger
            "level": "WARNING",
            "propagate": False
        },
        ## if __name__ == '__main__'
        '__main__': {
            'handlers': ['console'],
            'level': 'DEBUG',
            'propagate': False
        },
        ## Suppress all log messages.
        #  Usage: logging.getLogger("silent")
        "silent": {
            "level:" "NOTSET"
        },
        ## Disable logs from the requests module by setting to WARNING
        "requests": {
            ## Set to warning to disable logging of 3rd party modules
            "level": "WARNING"
        },
        ## Set logging for a module named my_module in advance
        "my_module": {
            "level": "INFO"
        }
    },
}

Log Levels

Value Int Level Description
NOTSET 0 It is the lowest level in the logging hierarchy and is used to indicate that no specific logging level has been set for a logger or a handler (more on handler later on in the article). It is essentially a placeholder level that is used when the logging level is not explicitly defined.
DEBUG 10 It is used for low-level debugging messages that provide detailed information about the code’s behavior. These messages are typically used during development and are not required in production.
INFO 20 It is used to log informational messages about the program’s behavior. These messages can be used to track the program’s progress or to provide context about the user.
WARNING 30 It is used to log messages that indicate potential issues or unexpected behavior in the program. These messages do not necessarily indicate an error but are useful for diagnosing problems.
ERROR 40 It is used to log messages that indicate an error has occurred in the program. These messages can be used to identify and diagnose problems in the code.
CRITICAL 50 It is used to log messages that indicate a critical error has occurred that prevents the program from functioning correctly. These messages should be used sparingly and only when necessary.

Creating a custom log level

Example: VERBOSE

Example VERBOSE custom log level
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import logging

# Define the custom log level
VERBOSE = 15
logging.VERBOSE = VERBOSE
logging.addLevelName(logging.VERBOSE, 'VERBOSE')

# Set up basic logging configuration for the root logger
logging.basicConfig(level=logging.DEBUG)


# Define a custom logging method for the new level
def verbose(self, message, *args, **kwargs):
    if self.isEnabledFor(logging.VERBOSE):
        self._log(logging.VERBOSE, message, args, **kwargs)


# Add the custom logging method to the logger class
logging.Logger.verbose = verbose

# Create a logger instance
logger = logging.getLogger()

# Log a message using the custom level and method
logger.verbose("This is a verbose message")
Title URL
Python docs: logging https://docs.python.org/3/library/logging.html
Python docs: logging.config https://docs.python.org/3/library/logging.config.html
Python docs: logging.config.dictConfig https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig
Python docs: LogRecord attributes (formatting) https://docs.python.org/3/library/logging.html#logrecord-attributes
Python docs: time.strftime (timestamp formatting) https://docs.python.org/3/library/time.html#time.strftime
Python docs: logging Handler objects https://docs.python.org/3/library/logging.html#handler-objects