Skip to content

python-stdlib/logging: add formatter, handlers #482

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

mkomon
Copy link

@mkomon mkomon commented Feb 15, 2022

The current version 0.3 of logging.py is very lightweight. Too lightweight, in fact, as all it does out of the box is print into sys.stderr.

I'm suggesting to add features focused on embedded devices where MicroPython is mostly used. What I mean by that are specifically CircularFileHandler and SysLogHandler.

CircularFileHandler is a handler that writes logs into a file and starts at the beginning of the file once it reaches a defined limit. This prevents the exhaustion of free space that is a risk of naive file handlers. This handler does not overwrite existing log file which preserves logs across device reboots.

SysLogHandler extends SocketHandler and mostly implements RFC 5424 - The Syslog Protocol. I say "mostly" because it does not implement TLS transport due to memory constraints. It does implement the protocol message format, and supports both TCP and UDP transport.

To implement SyslogHandler custom log message formatting is required. I borrowed Formatter from pycopy-lib/logging/logging/init.py with minimal changes. I added support for defaults argument which allows to pass system IP or hostname into the formatter used for SysLogHandler without adding undesired dependency of logging on network. Both %-style and {-style string formats are supported.

Finally, I extended basicConfig to support filename (uses CircularFileHandler), format and datefmt arguments which simplifies the usage of most of the functionality of the module.

add formatting support and embedded device oriented handlers (circular log file, socket and syslog)
@andrewleech
Copy link
Contributor

Thanks for this, I agree the existing logging module is barely more than something to satify an import in existing libraries. In other projects I've also extended this library to suit needs, and I'm sure many others have too - wasteful duplication of effort. So thanks for submitting this to share!

I certainly support pulling in updates from pycopy when they make sense. As that fork has a common history with this one I prefer to cherry-pick / rebase commits from there to preserve the author attribution - when good work is done I try to acknowledge it directly. For example I've recently pulled in pycopy changed to unittest in #487 which I then build on top of in #488

If you'd like I'd be happy to help pull in pycopy commits to logging in a separete PR that you could rebase on top of?

Separately though, I like your rationalle for the two handlers! However being different to any cpython logging handlers I think they'll need to be documented in some way (maybe just adding a readme to the folder) with a note that these are non-compliant with cpython and some basic details on their usage (separate to the example provided).

Similarly, these handlers probably wont be suitable for use on all platforms / ports - some don't have socket for instance. As such I think they should go in separate python files that users can choose to install / deploy to device or not as needed. This would allow easier exclusion of the code size/functionality in applciations where it won't be needed.
This would probably require refactoring into a module folder logging where logging.py gets renamed to __init__.py and the handlers are in separate files.
This refactor could complicate updates for existing users of the logging module though, I'm not sure how the micropython-lib maintainers feel about this?

@andrewleech
Copy link
Contributor

With the CircularFileHandler, was there a particular reason you chose to start rewriting over the same file in a circular fashion rather than rotating to a second file instead like in https://docs.python.org/3/library/logging.handlers.html#rotatingfilehandler ? In terms of size runaway protection, having a max of 1 backup on a rotation style handler would be similar.

In my mind it feels like the circular overwriting of a single file would be harder to read as you have to scan through the file to find the oldest messages then read on from there? I might be missing something here though :-)

@andrewleech
Copy link
Contributor

I've just had a chat with others and there's a guidance that everything in python-stdlib is expected to match cpython versions of the library, or at least be a subset of matching functionality. (I'm planning on documenting this more clearly somewhere... soon)

Anything that's a micropython specific change should go in a separate module in micropython dir, so the handlers could be in separate micropython/logging.SyslogHandler and micropython/logging.CircularFileHandler packages that can be separetly installed when needed.

@@ -7,6 +10,14 @@
DEBUG = 10
NOTSET = 0

SYSLOG_FORMAT = "<%(pri)s>1 %(asctime)s %(ip)s - - - - %(message)s"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these go in the respective classes using it instead of globally?

@@ -15,7 +26,13 @@
DEBUG: "DEBUG",
}

_stream = sys.stderr
_syslog_severity = {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question

@@ -108,21 +272,34 @@ def getLogger(name="root"):
_loggers[name] = l
return l

def critical(msg, *args):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can log and exception also be added while at it?

@ThinkTransit
Copy link
Contributor

Below is an implementation of the CPython RotatingFileHandler for consideration.

class RotatingFileHandler(Handler):
    def __init__(self, filename, maxsize=128_000, backupCount=5):
        if maxsize < 256:
            raise ValueError("maxsize must be at least 256 B")
        self.filename = filename
        self.maxsize = maxsize
        self.backupCount = backupCount
        self.formatter = Formatter()

        try:
            # checks if file exists and prevents overwriting it
            self.pos = os.stat(self.filename)[6]
            self.file = open(filename, "r+")
        except OSError:
            self.pos = 0
            self.file = open(filename, "w+")

        self.file.seek(self.pos)

    def shouldRollover(self, record):
        """
        Determine if rollover should occur.
        Basically, see if the supplied record would cause the file to exceed
        the size limit we have.
        """
        message = self.format(record)
        if len(message) + self.pos > self.maxsize:
            return True
        else:
            return False


    def doRollover(self):
        """
        Do a rollover.
        """
        # Close current file first
        self.file.close()

        if self.backupCount > 0:
            for i in range(self.backupCount - 1, 0, -1):
                sfn = "%s.%d" % (self.filename, i)
                dfn = "%s.%d" % (self.filename, i + 1)
                try:
                    os.rename(sfn, dfn)
                except:
                    print(f"Cant move file {sfn} to {dfn}")

            dfn = self.filename + ".1"
            try:
                os.rename(self.filename, dfn)
            except:
                print(f"Couldnt move file {self.filename} to {dfn}")

        print("Rollover complete")
        self.pos = 0
        self.file = open(self.filename, "w+")
        self.file.seek(self.pos)

@jimmo
Copy link
Member

jimmo commented Jul 21, 2023

Closing due to inactivity.

Significant improvements were made to logging in #507, however a SocketHandler (and SysLogHandler) and CircularFileHandler might be worthwhile adding as extension modules. (i.e. I don't think they should go in the default package, but can be installed optionally. See e.g. collections-defaultdict for an example of an extension module)

@jimmo jimmo closed this Jul 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants