Binary risk control - implementation batch 1 #735
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
f883ca9 to
13462fc
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR implements binary classification risk control in its simplest form, focusing on mono-risk and unidimensional lambda using thresholding on predict_proba. The implementation includes cleanup of existing risk control code and introduction of new binary classification risk control components.
Key changes:
- Remove unnecessary components from existing LTT procedure (lambda=None check, p_values output)
- Add support for array-based n_obs in p-value calculations for binary classification scenarios
- Implement BinaryClassificationRisk class with predefined instances for precision, recall, and accuracy
- Introduce BinaryClassificationController for threshold-based binary classification risk control
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| mapie/control_risk/ltt.py | Remove p_values return, lambda=None check, and add array support for n_obs |
| mapie/control_risk/p_values.py | Add array support for n_obs parameter in Hoeffding-Bentkus p-value computation |
| mapie/risk_control.py | Add BinaryClassificationRisk class and predefined risk instances |
| mapie/risk_control_draft.py | Implement BinaryClassificationController with threshold-based risk control |
| mapie/tests/test_control_risk.py | Update tests for LTT changes and add n_obs array testing |
| mapie/tests/test_risk_control.py | Add comprehensive tests for BinaryClassificationRisk instances |
| mapie/init.py | Remove risk_control_draft from exports |
Comments suppressed due to low confidence (2)
mapie/tests/test_risk_control.py:848
- The test should verify the specific condition that leads to None result (effective_sample_size == 0) rather than using a general elif clause. Consider adding an explicit assertion that effective_sample_func returns 0 when result is None.
elif result is None:
mapie/risk_control_draft.py:18
- The entire BinaryClassificationController class is marked with pragma: no cover, indicating missing test coverage. This class implements core functionality and should have comprehensive unit tests.
class BinaryClassificationController: # pragma: no cover
- Use BinaryClassificationRisk to compute risk - Use warning instead of error when risk is not controled. Throw error when predicting - Remove useless check on lambda=None in ltt_procedure - Remove useless p_values from ltt_procedure outputs - Add possibility to pass an array of n_obs to ltt_procedure and subsequent p-values calculations (needed for binary classification)
- Fix bentkus_p_value calculation - Fix and move higher_is_better logic in the same place - Implement unit test for BinaryClassificationRiskControl - Fix parametrizing of existing test
8f2a127 to
ed18964
Compare
37c79dc to
4404ad1
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
Implementation of binary classification risk control in its simplest form:
Content
This PR includes (lists incrementally updated):
Existing risk-control code:
New risk-control code: