Skip to content

Parse environment variable into array #40906

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Nyholm opened this issue Apr 22, 2021 · 37 comments
Closed

Parse environment variable into array #40906

Nyholm opened this issue Apr 22, 2021 · 37 comments

Comments

@Nyholm
Copy link
Member

Nyholm commented Apr 22, 2021

Symfony version(s) affected: 5.2.6

Description
I cannot parse an environment variable into an array.

How to reproduce

# env
TRUSTED_HOSTS="[\"10.0.0.1\", \"10.0.0.2\"]"
# config/packages/framework.yaml
framework:
    trusted_hosts: '%env(json:TRUSTED_HOSTS)%'

I get:

Invalid type for path “framework.trusted_hosts.0”. Expected one of “bool”, “int”, “float”, “string”, but got “array”.

See reproducer repo: https://github.com/Nyholm/sf-issue-40906

@stof
Copy link
Member

stof commented Apr 22, 2021

I think this is caused by the special logic in the Configuration class that wraps a string into an array with 1 element when getting a string in trusted_hosts. This logic does not currently check whether the string it receives is an env placeholder.

@Nyholm
Copy link
Member Author

Nyholm commented Apr 22, 2021

Thank you.

Here is the code you reference

So this issue is specifically for trusted_hosts.

@stof
Copy link
Member

stof commented Apr 22, 2021

Well, we could have similar issues in other places doing some logic in beforeNormalization.
Custom normalization logic is not automatically aware of env placeholders (precisely because it is custom logic)

@ro0NL
Copy link
Contributor

ro0NL commented Apr 22, 2021

see #27683, #37836, #40654 :)

i think #40654 (comment) is the ideal solution

@nicolas-grekas
Copy link
Member

nicolas-grekas commented Apr 22, 2021

To my understanding, this fails because "json" declares returning "string|int|bool|...|array".
But the logic in Config considers that such a return-type is not compatible with a config option that accepts only "array".
I think this logic is too restrictive and that it should be fixed.

@ro0NL
Copy link
Contributor

ro0NL commented Apr 22, 2021

json declares type array|null.

The issue is ifString then array($str) during config normalization. The env placeholder technically matches as such, but then from the env type POV we see array(array|null) for path trusted_hosts.0

@Nyholm
Copy link
Member Author

Nyholm commented Apr 22, 2021

The way I see it, we are using too many normalisations. The ifString then array($str) should be removed.

@nicolas-grekas
Copy link
Member

I agree we're trying to be too smart with these shortcuts.
At least here we can check if the value is an env placeholder, and not wrap it when yes.

@Nyholm
Copy link
Member Author

Nyholm commented Apr 22, 2021

If we dont convert a string to an array, we will get the following exception:

A dynamic value is not compatible with a "Symfony\Component\Config\Definition\PrototypedArrayNode" node type at path "framework.trusted_hosts".

@ro0NL
Copy link
Contributor

ro0NL commented Apr 22, 2021

it inherits

protected function allowPlaceholders(): bool
{
return false;
}

for prototyped arrays we can allow array-typed placeholders 👍 In this regard i agree with @nicolas-grekas more or less that we can loosen it a bit. Stricly spoken the env doesnt tell us the expected child prototype (eg. string[]) nor can we be compatible with minNumberOfElements. I really do think it should be restricted to array envs only though.

At least here we can check if the value is an env placeholder, and not wrap it when yes

By making ifString() opt-out from envs? eg. #40654 (comment) (maybe via a ifString(true) migration path?)

The ifString then array($str) should be removed.

What's the proposal ... case by case? Deprecate beforeNorm() as a whole? The above?

@ro0NL
Copy link
Contributor

ro0NL commented Apr 22, 2021

another issue is $config['prototyped_node'] will start giving strings at compile-time ...

@stof
Copy link
Member

stof commented Apr 22, 2021

I don't think we should deprecated beforeNormalization. This hook is useful when wanting to provide BC layer. And it also allows doing custom normalizations when needed.
However, we should potentially deprecated some places doing such custom normalizations when removing them would allow supporting env placeholders in the node.

@ro0NL
Copy link
Contributor

ro0NL commented Apr 22, 2021

potentially deprecated some places

the "some places" is exactly what started bugging me in #29270 (comment) 😅 but perhaps exposing $isEnv works after all.

@carsonbot
Copy link

Hey, thanks for your report!
There has not been a lot of activity here for a while. Is this bug still relevant? Have you managed to find a workaround?

@Guuzen
Copy link

Guuzen commented Oct 23, 2021

Relevant for me. But my case is about allowPlaceholders(): false
Found that It restricts specification of emails in mailer.envelope.recipients via env variable

@carsonbot carsonbot removed the Stalled label Oct 23, 2021
@tchapi
Copy link

tchapi commented Dec 13, 2021

Relevant to me too; And the documentation at https://symfony.com/doc/current/configuration/env_var_processors.html#built-in-environment-variable-processors is still wrong, as far as I understand ...

@carsonbot
Copy link

Hey, thanks for your report!
There has not been a lot of activity here for a while. Is this bug still relevant? Have you managed to find a workaround?

@carsonbot
Copy link

Friendly reminder that this issue exists. If I don't hear anything I'll close this.

@fattybenji
Copy link

I haven't seen any new developments on this, I would still like to see it.

@carsonbot carsonbot removed the Stalled label Jun 28, 2022
@Guuzen
Copy link

Guuzen commented Jun 28, 2022

still need this

@aymanbaba
Copy link

same here, i need a solution to this. Thanks

@garrigou
Copy link

same too me

@elements-team
Copy link

same too

@carsonbot
Copy link

Hey, thanks for your report!
There has not been a lot of activity here for a while. Is this bug still relevant? Have you managed to find a workaround?

@speller
Copy link

speller commented Feb 1, 2023

I would like to see a solution to convert environment variables into arrays without custom code, just writing configurations.

@VincentFarcy
Copy link

this works form me (symfony 4.4) :

# env
PRIVATE_PROJECTS="'test.localhost', 'test1.localhost'"
# security.yaml :
- { path: ^/.*, allow_if: "is_authenticated() || request.getHost() not in [%env(string:PRIVATE_PROJECTS)%]" }

hope this will help ;)

@MoZg0
Copy link

MoZg0 commented Aug 10, 2023

Same error with symfony/mailer recipients field

A dynamic value is not compatible with a "Symfony\Component\Config\Definition\PrototypedArrayNode" node type at path "framework.mailer.envelope.recipients".

@daffoxdev
Copy link
Contributor

daffoxdev commented Sep 28, 2023

Same with framework.lock. I'm trying to pass an array of memcached servers, but it not work.
MEMCACHE_SERVERS='memcached://m1.docker:11211,memcached://m2.docker:11211'
or
MEMCACHE_SERVERS="['memcached://m1.docker:11211','memcached://m2.docker:11211']"
not working

The lock memcache store looks following if I'm passing csv:

-store: Symfony\Component\Lock\Store\MemcachedStore {#22171 ▼
    -memcached: Memcached {#23004 ▼
      servers: array:1 [▼
        0 => array:3 [▼
          "host" => "m1.docker:11211,memcached"
          "port" => 11211
          "type" => "TCP"
        ]
      ]
      options: {▶}
    }
    -initialTtl: 300
    -useExtendedReturn: null
  }

and exception Invalid type for path "framework.lock.resources.default.0". Expected one of "bool", "int", "float", "string", but got "array" if I'm passing array in .ENV and trying to env processor as "json" or "array". As I understand it is due env-vars processors loads after framework configuration parsing.

Custom solution 1 via Extension:

Solved it via custom extension. Created parameter and then reused it in the yaml file. Correct me if it's bad practice, but it's only way I have found how to solve it for me.

    public function prepend(ContainerBuilder $container): void
    {
        $envName = 'MEMCACHE_SERVERS';

        $serversAsCsv = getenv($envName);
        
        if (empty($serversAsCsv)) {
            throw new LogicException(sprintf('Check that %s env is set and not empty', $envName));
        }

        $hostsList = str_getcsv($serversAsCsv, ',', '"', PHP_VERSION_ID >= 70400 ? '' : '\\');

        $container->setParameter('cache.memcached.servers', $hostsList);
    }

Custom solution 2 via framework.php:

<?php

use Symfony\Config\FrameworkConfig;

return static function (FrameworkConfig $framework): void {
    $framework->lock()
        ->resource('default', explode(',', getenv('MEMCACHE_SERVERS_2')))
    ;
};

@ivoba
Copy link
Contributor

ivoba commented Oct 11, 2023

I tried with a Extension and the prepend method, but it seems that at this point the .env file is not loaded yet.

So for the mailer config i just did not use the framework.mailer.envelope.recipients config but wrote a dedicated Listener for the onMessage event that gets the ENV var injected and will transform it to an array.

...
class MailerCatchallListener implements EventSubscriberInterface
{
    /**
     * @var null|Address[]
     */
    private ?array $recipients = null;
    public function __construct(?string $recipients)
    {
        if($recipients) {
            $recipientsArray = explode(',', $recipients);
            $this->recipients = Address::createArray($recipientsArray);
        }
    }

    public function onMessage(MessageEvent $event): void
    {
        if ($this->recipients) {
            $event->getEnvelope()->setRecipients($this->recipients);
        }
    }

    public static function getSubscribedEvents(): array
    {
        return [
            // framework.mailer.catchall_recipients will override
            MessageEvent::class => ['onMessage', -254],
        ];
    }
}
App\Listener\MailerCatchallListener:
  arguments:
    $recipients: '%env(string:MAILER_CATCHALL_RECIPIENTS)%'

Would still be handy though to be able to use the FrameworkBundle config option.

@daffoxdev
Copy link
Contributor

daffoxdev commented Oct 11, 2023

@ivoba, I suppose, the mine suggestions not worked well cause I forgot to mention that we force to use putenv in bootstrap (bad practice, but as it is). It was long ago when started to use that, but as I remember this is key moment why my code works. Correct me someone, please, if I'm wrong.

$dontEnv = (new Dotenv());
// Nicolas Grekas: "Injecting variables via dependency injection, that's the recommended way. i.e. code shouldn't care where the value comes from."
// https://github.com/symfony/symfony/issues/35388#issuecomment-576027130
$dontEnv->usePutenv(true); // <---- THIS ONE. By default it is turn off
$dontEnv->bootEnv(dirname(__DIR__).'/.env');

@b3n3d1k7
Copy link

b3n3d1k7 commented Apr 3, 2024

Why can't trusted_hosts just be handled like trusted_headers? They are also just a list of strings, right?
image
Edit: ok, this causes other problems because of the dynamic value check and ArrayNode not allowing placeholders.

It would be really helpful, if trusted_hosts could be set to an array using an env variable though.

@pacproduct
Copy link

pacproduct commented Jul 26, 2024

I'm facing this issue too on a Symfony 6.4 app. Is there still no way to pass an array parsed from en environment variable to properties framework.trusted_proxies and framework.trusted_hosts?

Using the JSON trick (e.g. trusted_hosts: '%env(json:TRUSTED_HOSTS)%') still triggers the following error:
Invalid type for path "framework.trusted_hosts.0". Expected one of "bool", "int", "float", "string", but got "array".


Instead, I altered my index.php file to add the following to it, but it feels hacky:
(Variables are made available in $_SERVER earlier thanks to Dotenv)

if ($trustedProxies = $_SERVER['TRUSTED_PROXIES'] ?? false) {
    Request::setTrustedProxies(explode(',', $trustedProxies), Request::HEADER_X_FORWARDED_FOR | Request::HEADER_X_FORWARDED_PORT | Request::HEADER_X_FORWARDED_PROTO);
}

if ($trustedHosts = $_SERVER['TRUSTED_HOSTS'] ?? false) {
    Request::setTrustedHosts(explode(',', $trustedHosts));
}

@lpotherat
Copy link

Up, still not able to use env var to populate trusted_proxy configuration.

This behaviour is not documented anywhere, and the other "trusted_" configuration works fine with env vars.

I'm not expert in the core of the symfony framework, but the fact that the custom logic to validate the input is done before the environment resolution breaks the ability to use env for this specific configuration, and break the consistency in configurability.

So, this bug is still relevant :)

@nicolas-grekas
Copy link
Member

Fixed by #58161 actually.

@lpotherat
Copy link

Oh fine, that closes a 3.5years old bug :), thank you @nicolas-grekas 👍

@c33s
Copy link

c33s commented Sep 26, 2024

@nicolas-grekas is there another ticket for a generic solution or is it a wont-fix?

i was hoping to be able to parse all kind of variables into arrays. i tried to configure the admin_recipients of the notifier with an environment variable which brought me to this ticket.

also other projects like a2lix/TranslationFormBundle#349 would depend on a more generic solution.

related symfony issues/PRs:

@nicolas-grekas
Copy link
Member

No generic solution is possible unfortunately: the way the config defines or checks the passed values requires special handling for env vars.
It's always possible to contribute the require changes to make env var supported where that'd make sense.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests