Skip to content

SSLContext.set_default_verify_paths holds GIL for > 1s #94637

Closed
@efroemling

Description

@efroemling

Hi; I'm the author of a game engine which makes heavy use of Python. The engine sends various network requests in background threads for various purposes. Recently I switched these to use https instead of http, and soon after noticed my Android builds in particular started hitching badly.

I tracked this down to my main logic thread occasionally spending upwards of 1 second waiting for a GIL lock, and further tracked that down to SSLContext.set_default_verify_paths(). That gets called by the urllib requests I'm firing off in my bg thread and winds up starving all other threads as it holds on to the GIL for the entirety of the SSL_CTX_set_default_verify_paths() call, which for some reason is taking quite a while on some Android devices.

I can separately look into why the underlying call is taking so long in the Android case, but regardless I'm able to completely eliminate the hitches by releasing the GIL for that call (enclosing the SSL_CTX_set_default_verify_paths() call with PySSL_BEGIN_ALLOW_THREADS/PySSL_END_ALLOW_THREADS).

Is that safe and reasonable to do for that call? I'd be happy to make a PR if so.

This was all tested on Python 3.10.5.

Thanks
-Eric

cpython/Modules/_ssl.c

Lines 4301 to 4310 in 760b8cf

static PyObject *
_ssl__SSLContext_set_default_verify_paths_impl(PySSLContext *self)
/*[clinic end generated code: output=0bee74e6e09deaaa input=35f3408021463d74]*/
{
if (!SSL_CTX_set_default_verify_paths(self->ctx)) {
_setSSLError(get_state_ctx(self), NULL, 0, __FILE__, __LINE__);
return NULL;
}
Py_RETURN_NONE;
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.10only security fixes3.11only security fixes3.12only security fixesperformancePerformance or resource usagetopic-SSL

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions