Skip to content

[Console] - Special characters encoding issues #37278

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
isaac-gros opened this issue Jun 14, 2020 · 4 comments · Fixed by #41113
Closed

[Console] - Special characters encoding issues #37278

isaac-gros opened this issue Jun 14, 2020 · 4 comments · Fixed by #41113

Comments

@isaac-gros
Copy link

isaac-gros commented Jun 14, 2020

Symfony version
Symfony Console 5.0 (only this bundle).

Description
I'm only using Symfony Console to generate PHP code and I have to ask informations to the user. When I run my script, I must use special characters to display a question, and it is displayed nicely. But when I submit informations with special characters, all of them are replaced by blank spaces.

Example

Please write the value for the field "Libellé" : # The character rendering is fine there
> My favourite libellé # User input
The following libellé has been defined : My favourite libell  # Only the answer has been "sanitized"

Possible solutions
I tried to add setTrimmable(false) to the Question object, but it didn't help. I also tried to create a validate function, to see if I could get the input in it raw form, but the input is still sanitized. I also tried to save the input in a variable and convert it to UTF-8 but it couldn't change something about it. I tried the PHP functions iconv and mb_convert_encoding and none of them could solve the issue.

Do you guys have an idea about the issue origin and if there is a way to fix it ?

@isaac-gros
Copy link
Author

Any idea ? 🤔

@YaFou
Copy link
Contributor

YaFou commented Jun 22, 2020

Some issues are related to this problem: #36324 and #35842. This problem is caused by the function fgets which does not take into account accents and special characters. I've not tested but this answer seems good. Are you on Windows @isaac-gros?

@isaac-gros
Copy link
Author

Some issues are related to this problem: #36324 and #35842. This problem is caused by the function fgets which does not take into account accents and special characters. I've not tested but this answer seems good. Are you on Windows @isaac-gros?

Yes I am on Windows. I added sapi_windows_cp_set(437); in the main file that runs the console and it fixed the problem. If it can help anyone, my main file now looks like this :

#!/usr/bin/env php
<?php

require __DIR__.'/vendor/autoload.php';

use Symfony\Component\Console\Application;

sapi_windows_cp_set(437);

$application = new Application();
$application->run();

Thanks for the help @YaFou !

@YaFou
Copy link
Contributor

YaFou commented Jun 22, 2020

Yes I am on Windows. I added sapi_windows_cp_set(437); in the main file that runs the console and it fixed the problem. If it can help anyone, my main file now looks like this :

#!/usr/bin/env php
<?php

require __DIR__.'/vendor/autoload.php';

use Symfony\Component\Console\Application;

sapi_windows_cp_set(437);

$application = new Application();
$application->run();

Thanks for the help @YaFou !

I've opened a pull request to fix this problem: #37385.

However be careful about calling the function sapi_windows_cp_set because only Windows supports this function. Use the following condition instead:

if(function_exists('sapi_windows_cp_set')) {
    sapi_windows_cp_set(437);
}

chalasr added a commit that referenced this issue Jul 6, 2020
This PR was merged into the 3.4 branch.

Discussion
----------

[Console] Fixes question input encoding on Windows

| Q             | A
| ------------- | ---
| Branch?       | 3.4 <!-- see below -->
| Bug fix?      | yes
| New feature?  | no <!-- please update src/**/CHANGELOG.md files -->
| Deprecations? | no <!-- please update UPGRADE-*.md and src/**/CHANGELOG.md files -->
| Tickets       | Fix #35842, Fix #36324, Fix #37495 and Fix #37278 <!-- prefix each issue number with "Fix #", if any -->
| License       | MIT
| Doc PR        | no <!-- required for new features -->
<!--
Replace this notice by a short README for your feature/bugfix. This will help people
understand your PR and can be used as a start for the documentation.

Additionally (see https://symfony.com/releases):
 - Always add tests and ensure they pass.
 - Never break backward compatibility (see https://symfony.com/bc).
 - Bug fixes must be submitted against the lowest maintained branch where they apply
   (lowest branches are regularly merged to upper ones so they get the fixes too.)
 - Features and deprecations must be submitted against branch master.
-->

To ask a question to a user, the [QuestionHelper](https://github.com/symfony/symfony/blob/master/src/Symfony/Component/Console/Helper/QuestionHelper.php) use [`fgets`](https://www.php.net/manual/en/function.fgets.php). However, special characters are not supported on Windows with this function (like accents: `é`, `à`, `ö`). The solution is to set a special encoding with [`sapi_windows_cp_get`](https://www.php.net/manual/en/function.sapi-windows-cp-get).

> Before
```php
$stream = fopen('php://stdin', 'r');
$input = fgets($stream);
echo $input;

// input: "Bonjour à tous"
// output: 'Bonjour \ tous" or "Bonjour   tous"
```

> After
```php
// Check if the function exists because it only exists on Windows
if(function_exists('sapi_windows_cp_set')) {
    sapi_windows_cp_get(437);
}

$stream = fopen('php://stdin', 'r');
$input = fgets($stream);
echo $input;

// input: "Bonjour à tous"
// output: 'Bonjour à tous"
```

*Thanks to @bnjmnfnk for the solution 😉*

Commits
-------

4288df4 [Console] Fixes question input encoding on Windows
nicolas-grekas added a commit that referenced this issue May 7, 2021
This PR was submitted for the 5.x branch but it was squashed and merged into the 5.2 branch instead.

Discussion
----------

[Console] Fix Windows code page support

| Q             | A
| ------------- | ---
| Branch?       | 5.2
| Bug fix?      | yes
| New feature?  | no
| Deprecations? | no
| Tickets       | Fix #37385, Fix #35842, Fix #36324, Fix #37495, Fix #37278
| License       | MIT

Corrects previous fixes that dealt with the mojibake problem on Windows where an OEM code page was applied to an input string and then messed with PHP.internal_encoding setting used by the script. This caused strings with different encodings to be displayed on the console output.

Commits
-------

be68682 [Console] Fix Windows code page support
nicolas-grekas added a commit that referenced this issue May 13, 2021
This PR was merged into the 4.4 branch.

Discussion
----------

[Console] Fix Windows code page support

| Q             | A
| ------------- | ---
| Branch?       | 4.4
| Bug fix?      | yes
| New feature?  | no
| Deprecations? | no
| Tickets       | Fix #37385, Fix #35842, Fix #36324, Fix #37495, Fix #37278
| License       | MIT

Corrects mojibake problem on Windows where an OEM code page was applied to an input string and then messed with PHP.internal_encoding setting used by the script. This caused strings with different encodings to be displayed on the console output.

Commits
-------

4145278 [Console] Fix Windows code page support
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants