feat(lifecycle): Implement object lifecycle management functionality #358
Conversation
Add a lifecycle module to automatically handle object expiration and transition during scanning Modify the file metadata cache module to be publicly visible to support lifecycle operations Adjust the scanning interval to a shorter time for testing lifecycle rules Implement the parsing and execution logic for S3 lifecycle configurations Add integration tests to verify the lifecycle expiration functionality Update dependencies to support the new lifecycle features Signed-off-by: junxiang Mu <1948535941@qq.com>
There was a problem hiding this comment.
Pull Request Overview
This PR implements comprehensive object lifecycle management functionality for the RustFS system, enabling automatic object expiration and transition during scanning operations.
Key changes include:
- Added lifecycle evaluation and execution logic to automatically handle object expiration based on S3 lifecycle configurations
- Modified file metadata cache module visibility to support lifecycle operations and integrated lifecycle processing into the data scanner
- Implemented object deletion functionality in the storage layer to support lifecycle actions
Reviewed Changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| crates/filemeta/src/lib.rs | Made metacache module public to support lifecycle operations |
| crates/ecstore/src/set_disk.rs | Implemented delete_object method with versioning support |
| crates/ecstore/src/bucket/lifecycle/lifecycle.rs | Enhanced lifecycle evaluation logic with debugging and expiration handling |
| crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs | Fixed version_id handling to be optional throughout lifecycle operations |
| crates/e2e_test/src/reliant/mod.rs | Added lifecycle test module |
| crates/e2e_test/src/reliant/lifecycle.rs | Added end-to-end lifecycle functionality tests |
| crates/ahm/tests/lifecycle_integration_test.rs | Added comprehensive lifecycle integration tests |
| crates/ahm/src/scanner/mod.rs | Added lifecycle module to scanner |
| crates/ahm/src/scanner/lifecycle.rs | Implemented scanner lifecycle processing functionality |
| crates/ahm/src/scanner/data_scanner.rs | Integrated lifecycle processing into data scanning workflow |
| crates/ahm/Cargo.toml | Added required dependencies for lifecycle functionality |
| @@ -435,22 +450,29 @@ impl Lifecycle for BucketLifecycleConfiguration { | |||
| }); | |||
| } | |||
| } else if let Some(days) = expiration.days { | |||
There was a problem hiding this comment.
The removal of the if days != 0 condition means that objects will be processed for expiration even when days is 0, but the expected_expiry_time function returns UNIX_EPOCH for 0 days. This creates inconsistent behavior and should be handled more explicitly.
Copilot uses AI. Check for mistakes.
| @@ -74,8 +75,8 @@ pub struct ScannerConfig { | |||
| impl Default for ScannerConfig { | |||
| fn default() -> Self { | |||
| Self { | |||
| scan_interval: Duration::from_secs(60), // 1 minute | |||
| deep_scan_interval: Duration::from_secs(3600), // 1 hour | |||
| scan_interval: Duration::from_secs(1), // 1 minute | |||
There was a problem hiding this comment.
The comment says '1 minute' but the duration is set to 1 second. This inconsistency between code and comment could cause confusion.
| scan_interval: Duration::from_secs(1), // 1 minute | |
| scan_interval: Duration::from_secs(60), // 1 minute |
Copilot uses AI. Check for mistakes.
| scan_interval: Duration::from_secs(60), // 1 minute | ||
| deep_scan_interval: Duration::from_secs(3600), // 1 hour | ||
| scan_interval: Duration::from_secs(1), // 1 minute | ||
| deep_scan_interval: Duration::from_secs(6), // 1 hour |
There was a problem hiding this comment.
The comment says '1 hour' but the duration is set to 6 seconds. This inconsistency between code and comment could cause confusion.
| deep_scan_interval: Duration::from_secs(6), // 1 hour | |
| deep_scan_interval: Duration::from_secs(3600), // 1 hour |
Copilot uses AI. Check for mistakes.
|
|
||
| use aws_config::meta::region::RegionProviderChain; | ||
| use aws_sdk_s3::Client; | ||
| use aws_sdk_s3::config::{Credentials, Region}; |
There was a problem hiding this comment.
The config module path appears to be incorrect for newer versions of the AWS SDK. It should likely be aws_sdk_s3::config::Credentials and aws_config::Region instead.
| use aws_sdk_s3::config::{Credentials, Region}; | |
| use aws_types::credentials::Credentials; | |
| use aws_types::region::Region; |
Copilot uses AI. Check for mistakes.
| Ok(()) | ||
| } | ||
|
|
||
| async fn apply_lifecycle(&mut self, oi: &ObjectInfo) -> (IlmAction, i64) { |
There was a problem hiding this comment.
The method takes &mut self but doesn't appear to modify any fields of self. This should be &self unless there's a specific reason for mutation.
| async fn apply_lifecycle(&mut self, oi: &ObjectInfo) -> (IlmAction, i64) { | |
| async fn apply_lifecycle(&self, oi: &ObjectInfo) -> (IlmAction, i64) { |
Copilot uses AI. Check for mistakes.
Signed-off-by: junxiang Mu <1948535941@qq.com>
Add a lifecycle module to automatically handle object expiration and transition during scanning Modify the file metadata cache module to be publicly visible to support lifecycle operations Adjust the scanning interval to a shorter time for testing lifecycle rules Implement the parsing and execution logic for S3 lifecycle configurations Add integration tests to verify the lifecycle expiration functionality Update dependencies to support the new lifecycle features
Type of Change
Related Issues
Summary of Changes
Checklist
cargo fmt --allcargo clippy --all-targets --all-features -- -D warningscargo check --all-targetsImpact
Additional Notes
Thank you for your contribution! Please ensure your PR follows the community standards (CODE_OF_CONDUCT.md) and sign the CLA if this is your first contribution.