Skip to content

Using Faraday in_parallel without immediately raising Faraday::TimeoutError #54

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
d11wtq opened this issue May 9, 2025 · 1 comment

Comments

@d11wtq
Copy link

d11wtq commented May 9, 2025

We're not currently using Async in our project, though we are looking to switch some code that was previously using Faraday with the :typhoeus adapter to instead use the :async_http adapter.

One feature we were relying on was the ability to run a batch of requests in parallel without raising exceptions if one of the requests hits a timeout. Say 20 requests are run in parallel and one of them times out, we still want the 19 successful responses (and we can retry the one timeout separately).

When I simulate this situation with the :async_http adapter, what I'm seeing is that the Async::TimeoutError is handled, then re-raised as Faraday::TimeoutError, even when running in parallel mode. This means that the whole in_parallel block raises an exception and therefore the other responses cannot be reached.

What typhoeus does instead is it puts two values into the response env:

  • env[:typhoeus_connection_failed] (can be understood as Faraday::ConnectionFailed with the details found in env[:typhoeus_return_message])
  • env[:typhoeus_timed_out] (can be understood as Faraday::TimeoutError)

Is there any way to keep the same capability of firing off 20 requests and letting some of them timeout or hit connection issues without failing all the others?

@ioquatix
Copy link
Member

ioquatix commented May 9, 2025

The simplest way would be to use Async to fan out the requests, but still use Faraday's interface for individual requests. Since your level of complexity is higher than normal users, I'd actually suggest you just use Async::HTTP::Internet with Async rather than trying to shoe-horn this usage into Faraday.

Beyond that, perhaps an official extension to Faraday would make more sense.

cc @iMacTia

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants