Skip to content

fix: Suggest solving the forefront handling of reclaimed requests #537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

Pijukatel
Copy link
Contributor

Drop forefront info from local chache as it is unreliable and complicates the flow. Only the platform knows the real forefront, do not try to keep imperfect copy of it locally and rather design the system to work without being aware of the cached forefront.

Drop forefront info from local chache as it is unreliable and complicates the flow.
Only the platform knows the real forefront, do not try to keep imperfect copy of it locally and rather design the system to work without being aware of the cached forefront.
@janbuchar
Copy link
Contributor

Only the platform knows the real forefront

Actually, the platform doesn't store that information either (it just uses it to put the request at the right end of the queue). And adding it to userData won't help, because list(andLock)Head doesn't return it (because it may be very large).

do not try to keep imperfect copy of it locally and rather design the system to work without being aware of the cached forefront.

I'd actually prefer to just avoid solving the problem if we're using locking.

@Pijukatel
Copy link
Contributor Author

Actually, the platform doesn't store that information either (it just uses it to put the request at the right end of the queue). And adding it to userData won't help, because list(andLock)Head doesn't return it (because it may be very large).

What is locally tracked as "forefront" is on platform the "tip" of the queue. So that is what I meant by the "Only the platform knows the real forefront"

Copy link
Contributor

@vdusek vdusek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks 👍 .

@Pijukatel Pijukatel changed the title Suggest solving the forefront handling of reclaimed requests fix: Suggest solving the forefront handling of reclaimed requests Aug 12, 2025
@Pijukatel Pijukatel marked this pull request as ready for review August 12, 2025 06:35
@Pijukatel Pijukatel merged commit 8b7fc77 into new-apify-storage-clients Aug 12, 2025
41 of 45 checks passed
@Pijukatel Pijukatel deleted the suggested-forefront-fix-new-storages branch August 12, 2025 06:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants