Skip to content

Multiprocessing integration #291

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
untitaker opened this issue Mar 13, 2019 · 8 comments
Open

Multiprocessing integration #291

untitaker opened this issue Mar 13, 2019 · 8 comments
Labels
Feature New Integration Integrating with a new framework or library Python SDK Triaged Has been looked at recently during old issue triage

Comments

@untitaker
Copy link
Member

We do support fork() in general, but the multiprocessing module has some quirks.

  • atexit does not get called, so events get lost
  • Unhandled exceptions are not captured

An integration should fix both. We should also consider Billard, a fork of mulitprocessing here. If we manage to properly hook into both versions, we can catch some startup errors in Celery as well.

If you're interested in this, please give this post a 👍 so it's easier to track.

@untitaker
Copy link
Member Author

untitaker commented Mar 20, 2019

@rouge8 @crepererum can you clarify whether you want this because of celery or something else? After creating this issue I discovered that Celery issues need to be tackled separately.

@crepererum
Copy link
Contributor

Dask is able to use local processes as a local backend, and it would be really helpful to at least get the queues flushed properly (atexit). But maybe a direct dask (and even distributed) integration would be more helpful.

@untitaker
Copy link
Member Author

@crepererum could you open a new issue wrt Dask support? The more I think about this (multiprocessing) support the more I thikn it's an A/B problem.

@rouge8
Copy link

rouge8 commented Mar 21, 2019

can you clarify whether you want this because of celery or something else?

I was reviewing the open issues while migrating away from the old raven library. :) We have some code that uses multiprocessing, but I haven't compared behavior with raven and sentry_sdk or anything yet.

@ashb
Copy link

ashb commented Jan 15, 2020

A change in Apache Airflow (to use os.fork, and thus the recommended os._exit() has broken that integration.

It would be nice to have a way to explicitly call the Sentry exit handler. https://issues.apache.org/jira/browse/AIRFLOW-6569 /cc @tiopi

@github-actions
Copy link

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!


"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

@davegravy
Copy link

davegravy commented Mar 1, 2024

are there any workarounds for the lack of unhandled exception capture?

All I can think of is a decorator, ie

from functools import wraps

def capture_exceptions(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        try:
            return func(*args, **kwargs)
        except Exception as e:
            sentry_sdk.capture_exception(e)
            raise
    return wrapper

@capture_exception
def my_worker_function:
  sentry_sdk.init(dsn=f"https://{sentry_key}@{sentry_host}.ingest.sentry.io/{sentry_project_number}")
  a = 1/0

Downside is that all exceptions are reported as handled.

@getsantry getsantry bot moved this to Waiting for: Product Owner in GitHub Issues with 👀 2 Mar 1, 2024
@antonpirker
Copy link
Member

Hey @davegravy

This looks solid. For the problem with "handled" you could something like we do in our integrations to have a custom _capture_exception method that sets the handled flag to false. See here: https://github.com/getsentry/sentry-python/blob/sentry-sdk-2.0/sentry_sdk/integrations/asyncio.py#L79-L95

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature New Integration Integrating with a new framework or library Python SDK Triaged Has been looked at recently during old issue triage
Projects
Status: No status
Development

No branches or pull requests