Skip to content

[HttpClient] Let curl handle transfer encoding #54517

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 8, 2024
Merged

[HttpClient] Let curl handle transfer encoding #54517

merged 1 commit into from
Apr 8, 2024

Conversation

michaelhue
Copy link
Contributor

Q A
Branch? 5.4
Bug fix? yes
New feature? no
Deprecations? no
Issues Fix #54516, Fix #54491
License MIT

Removes the Transfer-Encoding: chunked header that is added for chunked request bodies. This will allow curl to handle the transfer encoding and set appropriate headers based on the HTTP protocol version it negotiated with the server, since HTTP/2 does not support chunked transfer encoding.

@carsonbot
Copy link

Hey!

I see that this is your first PR. That is great! Welcome!

Symfony has a contribution guide which I suggest you to read.

In short:

  • Always add tests
  • Keep backward compatibility (see https://symfony.com/bc).
  • Bug fixes must be submitted against the lowest maintained branch where they apply (see https://symfony.com/releases)
  • Features and deprecations must be submitted against the 7.1 branch.

Review the GitHub status checks of your pull request and try to solve the reported issues. If some tests are failing, try to see if they are failing because of this change.

When two Symfony core team members approve this change, it will be merged and you will become an official Symfony contributor!
If this PR is merged in a lower version branch, it will be merged up to all maintained branches within a few days.

I am going to sit back now and wait for the reviews.

Cheers!

Carsonbot

@nicolas-grekas
Copy link
Member

nicolas-grekas commented Apr 8, 2024

Thanks for investigating and for the reproducer.
When I run the code you shared, I have everything green:
✅ HTTP/1 default behavior
✅ HTTP/2 default behavior
✅ HTTP/1 suppressed header
✅ HTTP/2 suppressed header

Could this be related to some curl bug? Which exact version of curl and php do you have ? (php --ri curl && php -v)

Mine:

  • curl 7.81.0
  • php 8.3.4

@michaelhue
Copy link
Contributor Author

michaelhue commented Apr 8, 2024

Hey Nicolas, that's interesting!

  • curl: 8.7.1
  • PHP: 8.3.4

I checked the curl issues and found some posts that may be related:

While I'm not sure this is the bug I'm experiencing, there is a relevant quote:

Applications are encouraged to not force "chunked", but rather set length information for a POST. By setting -1, curl will auto-select chunked on HTTP/1.1 and work properly on other HTTP versions.

That is the behavior this pull request would address.


I just checked the servers where I first experienced #54491 and they are also on curl 8.7.1.

@nicolas-grekas
Copy link
Member

Thank you @michaelhue.

@nicolas-grekas nicolas-grekas merged commit de1409e into symfony:5.4 Apr 8, 2024
8 of 12 checks passed
@michaelhue michaelhue deleted the fix_54516 branch April 8, 2024 13:33
This was referenced Apr 29, 2024
@joelwurtz
Copy link
Contributor

Since this patch, our HTTP client does not add transfer-encoding: chunked header when specifying the body as a callable

We have to force the header to make it work, is this a problem between curl version and this patch ?

FYI :

php version : 8.3.6
curl: 7.64.0

@ddegasperi
Copy link

I've to apply also the "Transfer-encoding: chunked" header to make my request (see code snipped) work again after this patch:

$formData = new FormDataPart([
    'degree' => '90',
    'file' => new DataPart(
        $fileResource, // a resource not a string
        'image.jpg',
        'image/jpeg'
    )
]);

$response = $client->request('POST', '/api/image/rotate/', [
    'headers' => [
        ...$formData->getPreparedHeaders()->toArray(),
        'Transfer-encoding: chunked' // with this header the patch works, without it does not send the data
    ],
    'body' => $formData->bodyToIterable(),
]);

PHP Version: 7.4.33
Curl: 7.61.1

Do we know from which Curl version we should apply the following behavior?

Applications are encouraged to not force "chunked", but rather set length information for a POST. By setting -1, curl will auto-select chunked on HTTP/1.1 and work properly on other HTTP versions.

Since Symfony 5.4 is supported also for PHP >=7.2.5, which often runs on systems where these dependencies are not necessarily updatable, I wonder if it was right to backport this patch to this version?

@nicolas-grekas
Copy link
Member

Closed issues cannot be tracked so please open one to describe the bug

@nicolas-grekas
Copy link
Member

@ddegasperi this solution might break once we fix the bug

@devon-seitz
Copy link

FYI updating to curl 8.8 fixed this issue for me

@nicolas-grekas
Copy link
Member

FYI I think we should revert this, see #57569

nicolas-grekas added a commit that referenced this pull request Jun 28, 2024
…ding", use HTTP/1.1 for Mailgun (nicolas-grekas)

This PR was squashed before being merged into the 5.4 branch.

Discussion
----------

[HttpClient][Mailer] Revert "Let curl handle transfer encoding", use HTTP/1.1 for Mailgun

| Q             | A
| ------------- | ---
| Branch?       | 5.4
| Bug fix?      | yes
| New feature?  | no
| Deprecations? | no
| Issues        | Fix #57544
| License       | MIT

This PR reverts #54517 and fixes #54491 by forcing HTTP/1.1 when talking to Mailgun

The issue that #54517 tries to fix shouldn't happen since curl 7.51, which dates back from 2016 (curl/curl@d4c5a91). If you experience it, try upgrading.

The previous code worked fine for years but the new one generated reports a few weeks after its release so I'd feel more confident reverting.

Commits
-------

efc93cd [HttpClient][Mailer] Revert "Let curl handle transfer encoding", use HTTP/1.1 for Mailgun
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants