Skip to content

gh-127567: Avoid forceful shutting down of handlers during reconfiguration #127690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

Agent-Hellboy
Copy link
Contributor

@Agent-Hellboy Agent-Hellboy commented Dec 6, 2024

@@ -3554,6 +3554,32 @@ def test_config15_ok(self):
handler = logging.root.handlers[0]
self.addCleanup(closeFileHandler, handler, fn)

def test_clear_existing_handlers_preserves_active_handlers(self):
Copy link

@aparshin aparshin Dec 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Agent-Hellboy I don't understand how this test covers the problem described in the original issue... The problem in the issue is with handlers of loggers that are not supposed to be modified by new configuration, not with handlers defined in the configuration...

I would propose the following test:

def test_disable_existing_loggers(self):
        fn = make_temp_file(".log", "test_logging-disable-existing-loggers-")
        file_handler = logging.FileHandler(fn, mode="w")
        file_handler.setFormatter(logging.Formatter("%(message)s"))
        root_logger = logging.getLogger()
        root_logger.addHandler(file_handler)

        config = {"version": 1, "disable_existing_loggers": False}

        # we have disable_existing_loggers=False,
        # so, all handlers should continue working
        self.apply_config(config)

        msg = "test message"
        logging.warning(msg)
        file_handler.close()
        with open(fn, encoding='utf-8') as f:
            data = f.read().strip()
        os.remove(fn)
        self.assertEqual(data, msg)

Copy link
Contributor Author

@Agent-Hellboy Agent-Hellboy Dec 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, I was testing assert fileHandler in root_logger.handlers only, I have used yours test now

@Agent-Hellboy Agent-Hellboy changed the title gh-127567: Avoid shuting down handlers during reconfiguration gh-127567: Avoid shutting down handlers during reconfiguration Dec 6, 2024
@@ -0,0 +1 @@
Avoid shutting down handlers during reconfiguration
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should clarify that this is related to the logging module.

@ZeroIntensity ZeroIntensity added needs backport to 3.12 only security fixes needs backport to 3.13 bugs and security fixes labels Dec 6, 2024
"""Clear and close handlers that are no longer in use."""
active_handlers = {
handler
for logger in logging.Logger.manager.loggerDict.values()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason why now the test is passing is that the root logger is not included in manager.loggerDict(). If you change the following line in the test

root_logger = logging.getLogger()

to

root_logger = logging.getLogger("any-logger")

the test will fail.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, I simply copied your test without giving a thought

@@ -285,9 +285,20 @@ def _install_loggers(cp, handlers, disable_existing):


def _clearExistingHandlers():
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we call this _clearExistingHandlers() after we applied all the modifications from config, not before as it's happening now?

Copy link
Contributor Author

@Agent-Hellboy Agent-Hellboy Dec 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added a check

#Remove any existing handlers

here as well to counter it , I will do that for fileconfig as well

@@ -285,9 +285,20 @@ def _install_loggers(cp, handlers, disable_existing):


def _clearExistingHandlers():
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we rename the method to better reflect its behaviour? To something like _clearNotUsedHandlers.

logging._handlers.clear()
logging.shutdown(logging._handlerList[:])
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the logging.shutdown() method calls handler.acquire(), handler.flush() and handler.release(). Also, it iterate all these handlers in a specific order (not sure why).

Don't we need all these actions here?

Copy link
Contributor Author

@Agent-Hellboy Agent-Hellboy Dec 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it iterate all these handlers in a specific order (not sure why)

didn't get your question, it is to protect the handler in a concurrent environment. A mutex lock , acquire operation release

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My question is why your implementation in this PR doesn't used them?

@Agent-Hellboy Agent-Hellboy changed the title gh-127567: Avoid shutting down handlers during reconfiguration gh-127567: Avoid forceful shutting down of handlers during reconfiguration Dec 7, 2024
@aparshin
Copy link

aparshin commented Dec 9, 2024

@Agent-Hellboy I see the following issues in this PR:

  1. I think that the original issue can be solved without introduction of new config's property remove_handlers. I described my vision of how it can be done in the issue.
  2. I don't understand how it's related to the disable_existing_loggers config's property. Either we want to disable exiting handlers or not, the behaviour related to closing handlers should be exactly the same (particularly, we shouldn't close handlers even if we decide to disable them, etc.).
  3. This PR is missing the regression test case. If it fixes a problem, there should be a meaningful test case that failed before the PR, and is working fine with this PR.

@tomasr8 tomasr8 removed the needs backport to 3.12 only security fixes label Apr 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants