Skip to content

Move dotenv from require-dev to require #25643

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
antonmedv opened this issue Dec 31, 2017 · 39 comments
Closed

Move dotenv from require-dev to require #25643

antonmedv opened this issue Dec 31, 2017 · 39 comments
Labels

Comments

@antonmedv
Copy link

Q A
Bug report? no
Feature request? yes
BC Break report? no
RFC? no
Symfony version 4.0.2

Hi,

Consider moving dotenv dependency from require-dev to require by default.

Yes, I read docs about .env for development and setting environmental parameters on system.

But I don't think what there is some profit in it at all. I don't use Apache mod-php and docker. I want to be able to configure multiple application on one host easily.

@jakzal
Copy link
Contributor

jakzal commented Dec 31, 2017

What's your goal? Even if you don't use mod-php or docker there's usually a way to set environment variables.

If you don't want to follow 12lve factor app and use environment variables to configure them, simply use configuration parameters. Just like before we supported .env.

@antonmedv
Copy link
Author

antonmedv commented Dec 31, 2017

What's your goal?

Run two different application on same machine. Or two instances of same app but with different environmental params.

Even if you don't use mod-php or docker there's usually a way to set environment variables.

And it is an ugly solution. Take a look at setting env inside php-fpm env[APP] = something pool config.

If you don't want to follow 12lve factor app and use environment variables to configure them,

12lve factor app isn't a silver bullet, don't pray on it. Consider developer experience first.

simply use configuration parameters. Just like before we supported .env.

I will. But I don't understand why complicate thinks for newcomers? Why not make it default? (or at least mention it as not "a bad practice").

PS Laravel uses .env files by default.

PPS What is your concerns against .env files?

@jakzal jakzal added the Dotenv label Dec 31, 2017
@Reflex14
Copy link

Reflex14 commented Jan 2, 2018

This really bothers me too, for another reason. I am a system administrator for mostly Symfony websites.

You can not assume that every Symfony user knows how to properly configure a safe web server.

On badly configured shared hosting servers, a local attacker could try to read the virtual host directory (Apache or NginX) and database credentials would be exposed.

Some users might leave a phpinfo.php in the public directory (we've all done that), again exposing database credentials and the secret. A remote attack.

I think this is a bad practice and I want this to be noted in the documentation.

It should be the other way around: If you know how to hide the environment variables, remove the dotenv package. Best practice is not applicable here.

@javiereguiluz
Copy link
Member

@antonmedv you tell @jakzal to not consider 12factor a silver bullet ... but he told you precisely that you don't need to follow 12factor if you don't want. Besides, you say that using parameters "complicates things" and we should think of developer experience. However, using parameters simplify everything. In Symfony 4 I use regular Symfony params (like in Symfony 2 and 3) and I don't use env vars for that reason; they complicate everything, specially the deployment on servers, where you need a way to manage those env vars.

@antonmedv
Copy link
Author

@javiereguiluz okay, lets update docs and say what is okay to use .env or paramelers.yaml on production. Not only environmental parameters.

@antonmedv
Copy link
Author

@jakzal see, @Reflex14 and @javiereguiluz both using files for storing credentials 😃

@Reflex14
Copy link

Reflex14 commented Jan 3, 2018

@antonmedv well no. But that's not the point. I really do think you should use environment configurations. It makes it so much faster to scale or deploy a new staging server in my case.

But how easy is it for an experienced developer to disable the dotenv package, and how hard is it for newcomers to figure out how to properly use environment configurations?

@bitgandtter
Copy link
Contributor

I will add a personal use case, im a docker user and i have clusters of swarm and k8s both supports secrets which are more secure than env variables, those secrets are created at runtime on each container so i use them as .env and move dotenv to my required section so that way i dont leak env variables on configurations and the flow between development and production is seamless. IMHO will bring more benefits using dotenv that dont.

@nicolas-grekas
Copy link
Member

@bitgandtter good news, your use case is already covered, see https://symfony.com/blog/new-in-symfony-3-4-advanced-environment-variables

@bitgandtter
Copy link
Contributor

Yes, and always its a balance between what you gave and what you put on developers hands.

1* Update and configure prod/config.yaml to extract the env from the gaven on the file
2* Include dotenv on required and change nothing

for us was better option 2, i understand this does not have to be everybody way but less configurations leave to less issues between dev env and prod env

@antonmedv
Copy link
Author

I agree with @bitdagger
Moving dotenv to require from require-dev is much better solution. We don't need to do anything.
ping @fabpot

@mvrhov
Copy link

mvrhov commented Jan 4, 2018

For those arguing in moving from -dev to require.
Why am I being forced install a dependency I do not need, because you are too lazy to write composer req "symfony/dotenv"

@antonmedv
Copy link
Author

@mvrhov it's okay, I would like to metion it in docs what it is okay to do that.

It's not an antipattern.

@boite
Copy link
Contributor

boite commented Jan 7, 2018

As alluded to by @mvrhov, if your project depends on dotenv then you really should make that dependency explicit: composer require symfony/dotenv.

@ThomasTr
Copy link
Contributor

ThomasTr commented Jan 7, 2018

I am now in the process to deploy my App. What‘s the best practise to use the env-vars in booth, the web app and the console? For the web app i place them in the nginx config. For my console commands i can put them to etc/environment. So all variables are stored twice. I think for dB credentials etc/environment is not the best place.

@MvanRietschoten
Copy link

Just keep this in mind when making decisions.

Keep development, staging, and production as similar as possible
https://12factor.net/dev-prod-parity

For me it is use dotenv all the way or don't use dotenv at all.
However, with the awesome symfony flex defaults that add these variables for me... from a developers perspective it is indeed easier to use dotenv all the way.

@antonmedv
Copy link
Author

Also I think this is bad to be default:

if (!isset($_SERVER['APP_ENV'])) {
    if (!class_exists(Dotenv::class)) {
        throw new \RuntimeException('APP_ENV environment variable is not defined. You need to define environment variables for configuration or add "symfony/dotenv" as a Composer dependency to load variables from a .env file.');
    }
    (new Dotenv())->load(__DIR__.'/../.env');
}

We should avoid this and only check existence of dotenv package.

if (class_exists(Dotenv::class)) {
    (new Dotenv())->load(__DIR__.'/../.env');
}

@ByScripts
Copy link

Personally, I don't mind running composer req symfony/dotenv manually if needed.

But first, is it really such a bad practice to use DotEnv in production? And if it really is, maybe the doc should be clear about why it is such a bad practice.

Recently I deployed a Sf4 app in production, and it was painful because of environment vars (but maybe I'm not very good at this?)

I defined my env vars in my Apache VirtualHost configuration... but when running a CLI command, obviously these env vars are inexistant, so I had to duplicate them in /etc/environment.

And if there is a better way, it should be explained in the doc.

@prakash
Copy link

prakash commented Jan 30, 2018

One of the benefits of environment variable is that it is not cached within Symfony which means we can modify it without clearing/warming cache but it is very hard and not possible to set on several conditions.

  • we need to specify same configurations into two places (for web and cli)
  • environment variables does not work while running php as CGI mode as it will be renamed with REDIRECT_ prefix
  • it is not possible to set environment variables on shared hosting as user do not have access for virtualhost modification

I recently have similar issues while trying to deploy application into cPanel server (PHP running as CGI), and was trying several days to properly set environment variables. Without any success, I was left with two possible solution; either go with old style parameter file or move Dotenv to require. While using old style parameter file, I need to clear and warmup cache every-time I need to change something and it also require further modification to the codebase. So, I simply moved Dotenv into require and everything worked as normal without any modification (except I have to use export APP_ENV=prod once while installing dependencies during deployment).

PS: most of the deployment tools listed https://symfony.com/doc/current/deployment.html does not work (e.g.: EasyDeployBundle, Deployer, Capistrano with Symfony plugin etc...). I guess, this issue is somehow related.

@antonmedv
Copy link
Author

Yes, I'm working now on bringing Symfony Flex into Deployer. And from my perspective it's simple to require dotenv.

@magnetik
Copy link
Contributor

magnetik commented Feb 7, 2018

I defined my env vars in my Apache VirtualHost configuration... but when running a CLI command, obviously these env vars are inexistant, so I had to duplicate them in /etc/environment.

Same here but with multiple SF4 applications so I cannot use the same environment everywhere.

There should be a good practice on how to share the environment between web and console part of an application.

@fabpot
Copy link
Member

fabpot commented Feb 7, 2018

There is already a good practice. A .env in development which is automatically used for the console and web. For anything else, you can still use them by sourcing the file source .env. In production, the environment variables should be defined so that they are always defined.

@antonmedv
Copy link
Author

In production, the environment variables should be defined so that they are always defined.

Nice. But not helping. Why .env in production is bad (as in docs)? What to do with php-fpm and bash, crontab, etc?

@Tobion
Copy link
Contributor

Tobion commented Feb 7, 2018

If you don't want to use environment parameters in production, just use normal symfony parameters as before. And if you want to use .env, just move it from require-dev to require in your project. That is perfectly possible. There is no need to enforce that for everyone. So I'm in favor of closing this issue.

@bitgandtter
Copy link
Contributor

Yes you can do all that, but this is about DX and less steps in the way to qa,stage,production environments. There are so many ways on how one can archive this configuration, but the real question is:

What are the security risks or disadvantages of moving dotenv to require?

As implementation goes, a file checkout, if .env exist use it, if not fallback sell env variables.

I may be missing some important things here that is why i ask the question

@fabpot
Copy link
Member

fabpot commented Feb 7, 2018

I think @Tobion is right. There is nothing to fix here. Just an array of different options.

@fabpot fabpot closed this as completed Feb 7, 2018
@Tobion
Copy link
Contributor

Tobion commented Feb 7, 2018

What are the security risks or disadvantages of moving dotenv to require?

There is no security risk. But mostly environment params are set differently in production, e.g. using a cloud infrastructure. If you use .env file there as well, it just means you don't use the proper env infrastructure. So it highly depends on the deployment and .env should not be part of the default setup.

@javiereguiluz
Copy link
Member

@Tobion yes ... but please don't forget about users who don't use cloud services to deploy their projects. It's only a guess and I don't have data to prove it ... but I guess that the immense majority of Symfony developers don't use cloud services yet. Focusing on env vars too much complicates things for them a lot.

@bitgandtter
Copy link
Contributor

As i previously advice, as implementation goes, a file checkout, if .env exist use it, if not fallback shell env variables. There are programmatically ways that dotenv can be included by default and not be use. IMHO there are more advantages of allowing it that not, but the project managers have the final word and seems to be already expressed.

@Tobion
Copy link
Contributor

Tobion commented Feb 7, 2018

@javiereguiluz True, that's why I said for most people using symfony params as before might be the better solution. For the big project I'm working on for example, it's not practical to use env params for everything. We kept most things as symfony params when upgrading to SF 4.

@bitgandtter
Copy link
Contributor

Agreed @Tobion for old projects migration can be hard and parameters are a better solutions and there i nothing preventing them from using this approach. But SF4 its about evolution and how to deal with projects in a modern way, from micro-services, cloud providers, dockerized applications, and so on.

@bobvandevijver
Copy link
Contributor

The biggest problem I am currently experiencing with environment variables is that they appear in any $_SERVER dump. So when you have a phpinfo or any other programming that uses it (for example, we use a 500-error mailer which also includes the server variables), and you've defined you database password in there, you might have a problem as it might be send over the line.

Sure, you can block accounts for usage only on the machine etc etc, but it simply feels less secure than using the parameters.yml for sensitive data.

So, I will probably be using a combination of both, as the variable replacement without cache clear is great: but just not with sensitive data.

@nicolas-grekas
Copy link
Member

You might also want to store secrets in the filesystem, and reference them using the file: processor.
See https://symfony.com/blog/new-in-symfony-3-4-advanced-environment-variables

@bobvandevijver
Copy link
Contributor

bobvandevijver commented May 1, 2018

@nicolas-grekas Good suggestion! Instead of creating a file per secret, is there any way to retrieve a single value from the app.secrets array value in the parameters.yml? Something like:

env(json:file:SECRETS_FILE): '/var/www/.secrets.json'
app.secrets: '%env(json:file:SECRETS_FILE)%'
database_password: '%app.secrets[database_password]%'

with the secrets file:

{
  "database_password": "xxx",
  "oidc_client_secret": "xxx"
}

In the blog BoShurik suggests using expression language ('@=parameter("app.secrets")["dbSecret"]'), but that is not allowed within the parameters...

@nicolas-grekas
Copy link
Member

There is no way to do that yet, but we could do that by adding a processor allowing so I think. PR welcome :)

@nicolas-grekas
Copy link
Member

@bobvandevijver if you'd like to work on such a PR, I'd suggest using the existing mechanism for this, which means not creating any new syntax. Here, my advice would we to make it work this way:
%env(key:database_password:json:file:SECRETS_FILE)%
or %env(key:json:file:SECRETS_FILE:database_password)%
ie have a processor for "key" that uses the first (or last) argument as the key, and forwards the rest to other processors. See EnvVarProcessor.

@bobvandevijver
Copy link
Contributor

I will be looking into that (hopefully later this week)

nicolas-grekas added a commit that referenced this issue May 21, 2018
…r (bobvandevijver)

This PR was squashed before being merged into the 4.2-dev branch (closes #27157).

Discussion
----------

[DI] Select specific key from an array resolved env var

| Q             | A
| ------------- | ---
| Branch?       | master
| Bug fix?      | no
| New feature?  | yes <!-- don't forget to update src/**/CHANGELOG.md files -->
| BC breaks?    | no     <!-- see https://symfony.com/bc -->
| Deprecations? | no <!-- don't forget to update UPGRADE-*.md and src/**/CHANGELOG.md files -->
| Tests pass?   | yes    <!-- please add some, will be required by reviewers -->
| Fixed tickets | Discussed in #25643    <!-- #-prefixed issue number(s), if any -->
| License       | MIT
| Doc PR        | symfony/symfony-docs#9734 <!-- required for new features -->

<!--
Write a short README entry for your feature/bugfix here (replace this comment block.)
This will help people understand your PR and can be used as a start of the Doc PR.
Additionally:
 - Bug fixes must be submitted against the lowest branch where they apply
   (lowest branches are regularly merged to upper ones so they get the fixes too).
 - Features and deprecations must be submitted against the master branch.
-->

As discussed in #25643, it would be convenient to have an key processor for environment variables which have been read from for example a JSON file.

The main advantage of this feature lies in specifying a single file for your secrets, that can be directly used in your configuration, without leaking them into your env.

## Example
**.secrets.json**
```json
{
  "database_password": "xxx"
}
```
**.env**
```env
APP_SECRETS=/opt/application/.secrets.json
DATABASE_URL=mysql://myuser:%database_password%@localhost:3306/mydb
```
**services.yaml**
```yaml
parameters:
  database_password: '%env(key:database_password:json:file:APP_SECRETS)%'
```

This example configuration will result in a `database_password` parameter being filled with `xxx`, and due to Doctrine defaults, the database url will be resolved with the correct password.

Commits
-------

42186a2 [DI] Select specific key from an array resolved env var
@ryanotella
Copy link

Practically speaking managing environment variables through Apache SetEnv etc is quite tricky. You need to duplicate values those in the CLI as well, which either means regular environment variables + PassEnv or two duplicated configurations.

I imagine in fully hosted environments (like AWS) the httpd and worker environments are unified automatically, but that isn't the case with a regular Linux server setup.

The overhead of DotEnv loading a .env file with 6 variables is much less than 1ms in my testing.

Env vars are easy to change between deploys without changing any code... - 12factor

This is simply not true for Apache SetEnv and Nginx. It requires a server reload. Unless you use .htaccess in which case you have all the negatives of a .env file but worse because it's in the web root. So we've just made a load balancer a minimum requirement.

Is anyone recommending environment variables actually doing normal deployments? It's much better in theory than in practice.

I think the docs should be updated, to let people know it's fine to use .env or parameters.yml in production if you want to, and maybe show how you would set that up.

@bobvandevijver
Copy link
Contributor

I've just opened a PR to remove the line on usage from the docs. Using dotenv in production is just the best way to have your console and web environment identical and easily changeable, without reloading Apache on any change, or enable htaccess (which slows down you're whole environment as it will keep scanning for the file in the complete tree).

javiereguiluz added a commit to symfony/symfony-docs that referenced this issue Jun 25, 2018
This PR was submitted for the master branch but it was merged into the 3.4 branch instead (closes #9905).

Discussion
----------

Update dotenv usage

Based on the discussion in symfony/symfony#25643, I propose to remove this note. There is no downside of using dotenv in production, and it actually brings extra value when using a "simple" Linux installation, as the environments on console and web are automatically 'identical'.

Commits
-------

1dc95f9 Recommend to use real env vars in production
c82f04a Update dotenv usage
t2d referenced this issue in systemli/userli Nov 2, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests