ElementTree.iterparse "leaks" file descriptor when not exhausted #101438

colesbury · 2023-01-30T20:47:39Z

The PR #31696 attempts to fix the "leak" of file descriptors when the iterator is not exhausted. That PR fixes the warning, but not the underlying issue that the files aren't closed until the next tracing garbage collection cycle.

Note that there isn't truly a leak of file descriptors. The file descriptors are eventually closed when the file object is finalized (at cyclic garbage collection). The point of the ResourceWarning (in my understanding) is that waiting until the next garbage collection cycle means that you may temporarily have a lot of unwanted open file descriptors, which could exhaust the global limit or prevent successful writes to those files on Windows.

# run with ulimit -Sn 1000
import xml.etree.ElementTree as ET
import tempfile

import gc
gc.disable()

def run():
    with tempfile.NamedTemporaryFile("w") as f:
        f.write("<document />junk")

        for i in range(10000):
            it = ET.iterparse(f.name)
            del it

run()

On my system, after lowering the file descriptor limit to 1000 (via ulimit -Sn 1000) I get:

OSError: [Errno 24] Too many open files: '/tmp/tmpwwmd9gp6'

Linked PRs

The text was updated successfully, but these errors were encountered:

ericvsmith · 2023-01-30T22:37:24Z

I think this would require something like PEP 533 “Deterministic cleanup for iterators“ to solve.

colesbury · 2023-01-30T22:52:43Z

In this case, I think you can fix it just by avoiding reference cycles that involve the generator. Below is an example refactoring of iterparse.

def iterparse(source, events=None, parser=None):
    # Use the internal, undocumented _parser argument for now; When the
    # parser argument of iterparse is removed, this can be killed.
    pullparser = XMLPullParser(events=events, _parser=parser)
    _root = None

    def iterator(source):
        nonlocal _root
        close_source = False
        try:
            if not hasattr(source, "read"):
                source = open(source, "rb")
                close_source = True
            yield None
            while True:
                yield from pullparser.read_events()
                # load event buffer
                data = source.read(16 * 1024)
                if not data:
                    break
                pullparser.feed(data)
            root = pullparser._close_and_return_root()
            yield from pullparser.read_events()
            _root = root
        finally:
            if close_source:
                source.close()

    class IterParseIterator(collections.abc.Iterator):
        def __init__(self, source):
            self.it = iterator(source)
        def __next__(self):
            return next(self.it)
        @property
        def root(self):
            nonlocal _root
            return _root
    it = IterParseIterator(source)
    del iterator, IterParseIterator

    next(it)
    return it

This avoids two reference cycles that captured the generator created by iterator(source):

Classes naturally form reference cycles, so move iterator(source) to the instance instead of a class attribute
Move _root up so that iterator(source) doesn't refer to the instance it.

I'm not sure about a few things including:

How do you test this?
Is it.root important? It's not documented. Is it only for testing?

EDIT: the problem is not capturing of the file descriptor, but the generator iterator(source).

rhettinger · 2023-01-30T23:05:20Z

@colesbury Doe the issue persist if you add gc.collect() after the del it?

colesbury · 2023-01-30T23:33:17Z

@rhettinger everything is cleaned up properly once the garbage collector runs. So adding a gc.collect() after the del it would collect the generator and close the file descriptor.

Refactor IterParseIterator to avoid a reference cycle between the iterator() function and the IterParseIterator() instance. This leads to more prompt clean-up of the "source" file if the returned iterator is not exhausted and not otherwise part of a reference cycle. This also avoids a test failure in the GC implementation for the free-threaded build: if the "source" file is finalized before the "iterator()" generator, a ResourceWarning is issued leading to a failure in test_iterparse(). In theory, this warning can occur in the default build as well, but is much less likely because it would require an unlucky scheduling of the GC between creation of the generator and the file object in order to change the order of finalization.

The iterator returned by ElementTree.iterparse() may hold on to a file descriptor. The reference cycle prevented prompt clean-up of the file descriptor if the returned iterator was not exhausted.

…honGH-114269) The iterator returned by ElementTree.iterparse() may hold on to a file descriptor. The reference cycle prevented prompt clean-up of the file descriptor if the returned iterator was not exhausted. (cherry picked from commit ce01ab5) Co-authored-by: Sam Gross <colesbury@gmail.com>

…-114269) (GH-114499) The iterator returned by ElementTree.iterparse() may hold on to a file descriptor. The reference cycle prevented prompt clean-up of the file descriptor if the returned iterator was not exhausted. (cherry picked from commit ce01ab5) Co-authored-by: Sam Gross <colesbury@gmail.com>

…-114269) (GH-114500) The iterator returned by ElementTree.iterparse() may hold on to a file descriptor. The reference cycle prevented prompt clean-up of the file descriptor if the returned iterator was not exhausted. (cherry picked from commit ce01ab5) Co-authored-by: Sam Gross <colesbury@gmail.com>

…honGH-114269) The iterator returned by ElementTree.iterparse() may hold on to a file descriptor. The reference cycle prevented prompt clean-up of the file descriptor if the returned iterator was not exhausted.

colesbury added the type-bug An unexpected behavior, bug, or error label Jan 30, 2023

markshannon pushed a commit to faster-cpython/cpython that referenced this issue Apr 19, 2023

pythongh-101438: possible fix for ElementTree.iterparse leak

e3bd9af

iritkatriel added the stdlib Python modules in the Lib dir label Nov 26, 2023

bedevere-app bot mentioned this issue Jan 18, 2024

gh-101438: Avoid reference cycle in ElementTree.iterparse. #114269

Merged

This was referenced Jan 23, 2024

[3.12] gh-101438: Avoid reference cycle in ElementTree.iterparse. (GH-114269) #114499

Merged

[3.11] gh-101438: Avoid reference cycle in ElementTree.iterparse. (GH-114269) #114500

Merged

serhiy-storchaka closed this as completed Jan 23, 2024

serhiy-storchaka added 3.11 only security fixes 3.12 only security fixes 3.13 bugs and security fixes labels Jan 23, 2024

serhiy-storchaka mentioned this issue Jan 24, 2024

Add the close method for ElementTree.iterparse() object #69893

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ElementTree.iterparse "leaks" file descriptor when not exhausted #101438

ElementTree.iterparse "leaks" file descriptor when not exhausted #101438

colesbury commented Jan 30, 2023 •

edited by bedevere-app bot

Loading

ericvsmith commented Jan 30, 2023

colesbury commented Jan 30, 2023 •

edited

Loading

rhettinger commented Jan 30, 2023

colesbury commented Jan 30, 2023

ElementTree.iterparse "leaks" file descriptor when not exhausted #101438

ElementTree.iterparse "leaks" file descriptor when not exhausted #101438

Comments

colesbury commented Jan 30, 2023 • edited by bedevere-app bot Loading

Linked PRs

ericvsmith commented Jan 30, 2023

colesbury commented Jan 30, 2023 • edited Loading

rhettinger commented Jan 30, 2023

colesbury commented Jan 30, 2023

colesbury commented Jan 30, 2023 •

edited by bedevere-app bot

Loading

colesbury commented Jan 30, 2023 •

edited

Loading