Skip to content

feat: Implement Synchronous Metrics Reader #4542

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

Jayclifford345
Copy link
Contributor

@Jayclifford345 Jayclifford345 commented Apr 11, 2025

Description

Addresses #4549

This PR introduces a new SynchronousExportingMetricReader that implements batched metric export without background threads. Unlike the existing PeriodicExportingMetricReader which uses a scheduler, this reader collects and exports metrics on-demand when explicitly requested through collect() calls, making it ideal for short-lived environments like serverless functions where predictable, explicit metric collection is preferred.

The reader maintains metrics in a queue and exports them when:

  1. The queue size reaches a configurable batch threshold
  2. force_flush() is called
  3. shutdown() is called

I decided to implement this feature with serverless environments like AWS Lambda in mind, where you need to explicitly collect and export metrics before the function terminates, without relying on background threads that may not have time to execute.

Type of change

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

How Has This Been Tested?

  • Unit test suite for SynchronousExportingMetricReader
  • Integration test with sample applications (with and without flush)
  • Verified behavior with different batch sizes and export conditions

Does This PR Require a Contrib Repo Change?

  • Yes.
  • No.

Checklist:

  • Followed the style guidelines of this project
  • Changelogs have been updated
  • Unit tests have been added
  • Documentation has been updated

@Jayclifford345 Jayclifford345 requested a review from a team as a code owner April 11, 2025 15:14
Copy link

linux-foundation-easycla bot commented Apr 11, 2025

CLA Signed

The committers listed above are authorized under a signed CLA.

@Jayclifford345 Jayclifford345 changed the title Work in progress feat: Implement SynchronousExportingMetricReader Apr 11, 2025
@Jayclifford345 Jayclifford345 changed the title feat: Implement SynchronousExportingMetricReader feat: Implement Synchronous Metrics Reader Apr 11, 2025
@tammy-baylis-swi
Copy link
Contributor

Thanks for submitting!

What other benefits come with this new Reader, instead of using PeriodicExportingMetricReader with export_interval_millis=math.inf

@Jayclifford345
Copy link
Contributor Author

Hi @tammy-baylis-swi! Thanks for looking into this. You raise a fair point on using export_interval_millis=math.inf. My design choice was to emulate a similar structure to the batch exporter for logs, where you have direct control over batch size and queue. This works well when you don't have an Otel collector available but still plan to use an Otel-compatible endpoint. I assume the periodic workaround method would be:

  1. Set export interval to inf (which skips thread creation)
  2. Invoke collection which will immediately export

I would be happy to look into merging the queue system into periodic reader if that makes more sense. If not I would be happy to contribute an example of this to the repo to make the feature more explicit.

@tammy-baylis-swi
Copy link
Contributor

I assume the periodic workaround method would be:

  1. Set export interval to inf (which skips thread creation)
  2. Invoke collection which will immediately export

Yes correct.

you have direct control over batch size and queue

That's great actually!!

merging the queue system into periodic reader if that makes more sense

Imo it'd be nice to keep this separate as an additional concrete Reader class like you have it, like with exporters/processors for logs and traces.

Please could you have a check of the failing tests. There's a few failures like

        # Verify success and metrics exported
        self.assertTrue(success)
>       self.assertEqual(len(exporter.metrics), 1)
E       AssertionError: 4 != 1

@Jayclifford345
Copy link
Contributor Author

Hi @tammy-baylis-swi, yes I shall look into the test. I think I had some weird logic which in the test which was passing locally but not here. Will get that fixed Monday. Documentation wise is there anything I need to do? I noticed that the documentation was being built but I couldn't find a specific page for the SDK. I assume part of it is being built from the inline class description?

@Jayclifford345
Copy link
Contributor Author

Hi @tammy-baylis-swi,
That should be the fixes to the tests and some of the batching logic. I realised there was a fundamental issue with how I was "batching" (should have stuck by the not shipping code on fridays) metric samples which should now be resolved. Let me know what you think :)

@tammy-baylis-swi
Copy link
Contributor

Hi @Jayclifford345 , yes the docstrings are used to generate some of the docs with sphinx -- style guide links in the contributing.md. For the new example (much appreciated), the new class is part of the SDK so I don't think doc-requirements.txt nor docs/examples/metrics/reader/requirements.txt need to be updated.

I think you need to do a local tox -e docs and push the updates if you haven't already.

Thanks for the test updates! Those util-http I don't think are to do with your changes.

@Jayclifford345
Copy link
Contributor Author

Hi @tammy-baylis-swi, can confirm the new reader is showing up in the docs as expected after building.

The failed test seems to be a mismatch in the workflow, not being able to find a specific commit hash. Not quite sure how to resolve that one.

Shall we try to run the test again and see if it clears up on its own?

@tammy-baylis-swi
Copy link
Contributor

@Jayclifford345 Please merge main into your branch, then I'll re-run

Also: Please could you create an accompanying Issue (Feature Request) to link to this PR! https://github.com/open-telemetry/opentelemetry-python/issues

@Jayclifford345
Copy link
Contributor Author

Hi @tammy-baylis-swi,
That is main merged and here is the issue: #4549. Ready for the next run. Thanks again for being persistent with this!

@Jayclifford345
Copy link
Contributor Author

Jayclifford345 commented Apr 16, 2025

Hmm @tammy-baylis-swi, I checked the error

Check that anchor links are lowercase
I did an initial check and couldn't see any uppercase

- Add `setuptools` to `install_requires`
  ([#2334](https://github.com/open-telemetry/opentelemetry-python/pull/2334))
- Add otlp entrypoint for log exporter
  ([#2322](https://github.com/open-telemetry/opentelemetry-python/pull/2322))
- Support insecure configuration for OTLP gRPC exporter
  ([#2350](https://github.com/open-telemetry/opentelemetry-python/pull/2350))

Copy link
Contributor

@tammy-baylis-swi tammy-baylis-swi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Jayclifford345 for creating that issue! I've also updated OP with a quick link to it.

Lgtm. I can't merge so I'm passing this on to @open-telemetry/python-maintainers to continue review. Feel free to also join the weekly Python SIG zoom meeting to request more eyes.

Yes I'm not sure what that markdown-link-check error is about 🤔 What you introduced to CHANGELOG looks ok to me.

@tammy-baylis-swi tammy-baylis-swi requested a review from a team April 16, 2025 17:36
@Jayclifford345
Copy link
Contributor Author

Thanks a lot @tammy-baylis-swi! I added a note to the agenda look forward to meeting you there :)

@Jayclifford345
Copy link
Contributor Author

I checked the otel specfication and sadly it looks like this would be out of current scope since we already implemented all of the required metrics readers: https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricreader

Happy to pull the plug if this is more trouble than it`s worth. I can just open a new PR to give an example of the alternative method if that helps instead? Let me know what you think though after review

@Jayclifford345 Jayclifford345 requested a review from emdneto April 17, 2025 18:17
@lzchen
Copy link
Contributor

lzchen commented Apr 24, 2025

@Jayclifford345

Apologies for the late review. It seems to me the only difference between this functionality and the current workaround in the periodic metric reader (with infinity) is the batching capability. While I do agree that our current components does not satisfy your specific situation, I have reservations in adding new components to the already stable api/sdk package, especially ones that are not in the spec. We will also have to support this metric reader forever (even past your use case) in case other people take a dependency on it. I'd be much more likely to approve this if we perhaps make this an experimental component (with an underscore) with guidance and documentation indicating to users to try at their own risk. wdyt?

@Jayclifford345
Copy link
Contributor Author

Hi @lzchen, I absolutely agree with your comments above. Sorry, I couldn't attend the Python sig meeting yesterday. I am happy to turn it into an experimental feature and see if it lands for the otel community. I want to make sure that I am not putting you all in a bad spot though where we are moving away from the specification outlined. I am happy to support this feature in the meantime. I ended up creating an example with the workaround here as an official method: #4559. I will work on the changes to get it moved into an experimental mode. Let me know though if there are any processes I need to follow in the meantime with regards to the specification :). Thanks again for the review.

@Jayclifford345
Copy link
Contributor Author

I think, based on the discussion in the issue, this is out of scope for now and with the given workaround now documented in #4559 I think this works as a good alternative. If this arises again in the future I am happy for this PR to be reopened and the code used in another implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants