Skip to content

[DSN] Add a new DSN component #32546

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Simperfit opened this issue Jul 15, 2019 · 17 comments
Closed

[DSN] Add a new DSN component #32546

Simperfit opened this issue Jul 15, 2019 · 17 comments
Labels
RFC RFC = Request For Comments (proposals about features that you want to be discussed)

Comments

@Simperfit
Copy link
Contributor

Description

Following this comment from @fabpot: #32392 (comment) and searching for information in the symfony repository
I've seen an attempt from @jderusse #24267. I still think the proposal was a good idea and we need a DSN component, we could even imagine a DSN contract, let me explain why:
I've looked into all of the symfony components to look for DSN usage, it seems that we use lot's of DSN and even in client project we use more and more DSN.

Lot's of symfony-linked librairies (doctrine, flysystem) use a DSN too, so instead of rechecking and reparsing the DSN all the time, we could have a simple dsn-parser that would do the job and return a DSN parsed as a PHP array/object.

If we talk about usage in symfony:

  • In the Cache 💾 component where are parsing two time the redis DSN:
    In the redis trait:

    if (0 === strpos($dsn, 'redis:')) {
    $scheme = 'redis';
    } elseif (0 === strpos($dsn, 'rediss:')) {
    $scheme = 'rediss';
    } else {
    throw new InvalidArgumentException(sprintf('Invalid Redis DSN: %s does not start with "redis:" or "rediss".', $dsn));
    }

    And in the adapter:
    if (0 === strpos($dsn, 'redis:') || 0 === strpos($dsn, 'rediss:')) {
    return RedisAdapter::createConnection($dsn, $options);
    }
    if (0 === strpos($dsn, 'memcached:')) {
    return MemcachedAdapter::createConnection($dsn, $options);
    }

    We are also parsing a memcache DSN in the adapter.
    Both of the redis check could be merged into one and an object could be passed between the two. The exception logic would be merge an a new exception would be thrown.

  • In the Messenger📮 component we are using DSN in connections classes and in the supports method, maybe a trait comming from the dns parser would help parsing this directly. I give you the exact lines of the code that each connection classes is doing but we know that's Doctrine AMQP, and redis. As an exemple we will only take doctrine.

    if (false === $components = parse_url($dsn)) {
    throw new InvalidArgumentException(sprintf('The given Doctrine Messenger DSN "%s" is invalid.', $dsn));
    }

  • In HttpKernel 📟 we parse a dsn too, that's the file DSN, it could be reworked to avoid parsing it in here and just getting the value without using substr:

    if (0 !== strpos($dsn, 'file:')) {
    throw new \RuntimeException(sprintf('Please check your configuration. You are trying to use FileStorage with an invalid dsn "%s". The expected format is "file:/path/to/the/storage/folder".', $dsn));
    }

  • In multiple places we parse PDO connection DSN with a throw if it's not a DSN, this could be all in the same places it will ease the maintenance and creating new adapter based on PDO:

    } elseif (\is_string($pdoOrDsn) && false !== strpos($pdoOrDsn, '://')) {
    $this->dsn = $this->buildDsnFromUrl($pdoOrDsn);
    with a method that fully parse de DSN.
    } elseif (\is_string($connOrDsn)) {
    $this->dsn = $connOrDsn;
    } else {

    And the PdoCache it uses the PdoTrait

  • In HttpFoundation 📡 we them found that big method that is parsing all kind of connection based sql DSN, this could totally be in the new DSN parser or in an dsn-sql-adapter ;).

    private function buildDsnFromUrl($dsnOrUrl)
    {
    // (pdo_)?sqlite3?:///... => (pdo_)?sqlite3?://localhost/... or else the URL will be invalid
    $url = preg_replace('#^((?:pdo_)?sqlite3?):///#', '$1://localhost/', $dsnOrUrl);
    $params = parse_url($url);
    if (false === $params) {
    return $dsnOrUrl; // If the URL is not valid, let's assume it might be a DSN already.
    }
    $params = array_map('rawurldecode', $params);
    // Override the default username and password. Values passed through options will still win over these in the constructor.
    if (isset($params['user'])) {
    $this->username = $params['user'];
    }
    if (isset($params['pass'])) {
    $this->password = $params['pass'];
    }
    if (!isset($params['scheme'])) {
    throw new \InvalidArgumentException('URLs without scheme are not supported to configure the PdoSessionHandler');
    }
    $driverAliasMap = [
    'mssql' => 'sqlsrv',
    'mysql2' => 'mysql', // Amazon RDS, for some weird reason
    'postgres' => 'pgsql',
    'postgresql' => 'pgsql',
    'sqlite3' => 'sqlite',
    ];
    $driver = isset($driverAliasMap[$params['scheme']]) ? $driverAliasMap[$params['scheme']] : $params['scheme'];
    // Doctrine DBAL supports passing its internal pdo_* driver names directly too (allowing both dashes and underscores). This allows supporting the same here.
    if (0 === strpos($driver, 'pdo_') || 0 === strpos($driver, 'pdo-')) {
    $driver = substr($driver, 4);
    }
    switch ($driver) {
    case 'mysql':
    case 'pgsql':
    $dsn = $driver.':';
    if (isset($params['host']) && '' !== $params['host']) {
    $dsn .= 'host='.$params['host'].';';
    }
    if (isset($params['port']) && '' !== $params['port']) {
    $dsn .= 'port='.$params['port'].';';
    }
    if (isset($params['path'])) {
    $dbName = substr($params['path'], 1); // Remove the leading slash
    $dsn .= 'dbname='.$dbName.';';
    }
    return $dsn;
    case 'sqlite':
    return 'sqlite:'.substr($params['path'], 1);
    case 'sqlsrv':
    $dsn = 'sqlsrv:server=';
    if (isset($params['host'])) {
    $dsn .= $params['host'];
    }
    if (isset($params['port']) && '' !== $params['port']) {
    $dsn .= ','.$params['port'];
    }
    if (isset($params['path'])) {
    $dbName = substr($params['path'], 1); // Remove the leading slash
    $dsn .= ';Database='.$dbName;
    }
    return $dsn;
    default:
    throw new \InvalidArgumentException(sprintf('The scheme "%s" is not supported by the PdoSessionHandler URL configuration. Pass a PDO DSN directly.', $params['scheme']));
    }
    }

  • In the Lock 🔒component we parse a DSN too in order to see what store we will instantiate:

    switch (true) {
    case 'flock' === $connection:
    return new FlockStore();
    case 0 === strpos($connection, 'flock://'):
    return new FlockStore(substr($connection, 8));
    case 'semaphore' === $connection:
    return new SemaphoreStore();
    case class_exists(AbstractAdapter::class) && preg_match('#^[a-z]++://#', $connection):
    return static::createStore(AbstractAdapter::createConnection($connection));
    default:
    throw new InvalidArgumentException(sprintf('Unsupported Connection: %s.', $connection));
    this could be refacoted to only getting the scheme and having a method that support or not the schema in each store for exemple.

  • And the Mailer components parses DSN too, the most widely used config param of the component is a DSN, all of this could simply in the DSN parser and will throw better exception when something is wrong

    if (false === $parsedDsn = parse_url($dsn)) {
    throw new InvalidArgumentException(sprintf('The "%s" mailer DSN is invalid.', $dsn));
    }
    if (!isset($parsedDsn['scheme'])) {
    throw new InvalidArgumentException(sprintf('The "%s" mailer DSN must contain a transport scheme.', $dsn));
    }
    if (!isset($parsedDsn['host'])) {
    throw new InvalidArgumentException(sprintf('The "%s" mailer DSN must contain a mailer name.', $dsn));
    }

As far I have looked into the code ans searched for DSN usages, I think we can really do something to avoid parsing all the DSN and having a system that will do it for us, to return the right types, the rights errors, the right scheme to have normalized error messages and interface on how we can parse DSN. It would avoid having the DSN parsed two times like in the RedisAdapter and in the RedisTrait.

Talking about symfony ecosystem (like doctrine for exemple) it will help too because since using DSN has been introduced widely when we use env var we can normalize on how we work with DSN.

Example

Like I said in the listing before, we could imagine the whole parsing logic being in the same place with a parser and them multiple connection like we've done with the mailer.
before would still be:

if (0 === strpos($dsn, 'redis:')) {
$scheme = 'redis';
} elseif (0 === strpos($dsn, 'rediss:')) {
$scheme = 'rediss';
} else {
throw new InvalidArgumentException(sprintf('Invalid Redis DSN: %s does not start with "redis:" or "rediss".', $dsn));
}

after it could looks like this (draaaft)
Either we inject the DsnParser or we use a static method that will return a Dsn Object:

Static version:

        $dsn = DsnParser::parse($dsn); // will return the correct Dsn object or throw if the Dsn is invalid
        $scheme = $dns->getSchema(); // will return the correct scheme

The object version is quite the same but we use the DependencyInjection to inject an instance of the DsnParser in the classe and to run parse on it.

Maybe we could even imagine a DSN component that contains all the connection that we have in symfony and all of them will be only in the DSN component, either we build a parser with different adapter either we build a whole connection object.

What do you think of this new component ?

@Koc
Copy link
Contributor

Koc commented Jul 15, 2019

See #31946 for some idea how to implement DSN for Mailer. I have plans do same for Messenger and after that consider extract it as separate component

@Simperfit
Copy link
Contributor Author

@Koc That's what I have in mind but for all of symfony, if your PR gets merged before we statued on this, we could take some of your work to begin the DSN component ;)

@xabbuh xabbuh added the RFC RFC = Request For Comments (proposals about features that you want to be discussed) label Jul 15, 2019
@csarrazi
Copy link
Contributor

-1 because there is no clear spec for DSNs, which are implementation-specific.

More precisely, ODBC, DBD, JDBC, native drivers, etc. handle DSNs in a different manner, meaning that there is a huge number of edge cases.

@fabpot
Copy link
Member

fabpot commented Jul 17, 2019

As a first step, perhaps having a Dsn class in each component would be enough (like done by @Koc on the Mailer and the Messenger components).

@francoispluchino
Copy link
Contributor

Why not create the Dsn component that will be used as you go by the other components? We will duplicate code unnecessarily, since we want the same implementation for all Symfony components using the Dsn configuration. In this case, the first component to use this new component will be Messenger.

@mdeboer
Copy link
Contributor

mdeboer commented Jul 17, 2019

-1 because there is no clear spec for DSNs, which are implementation-specific.

More precisely, ODBC, DBD, JDBC, native drivers, etc. handle DSNs in a different manner, meaning that there is a huge number of edge cases.

Well yes, as far as I understand it's not about creating one type of DSN but simply having a class that components can use to parse a DSN. Components that require a very specific DSN handle it on their own anyway as they are edge cases like you said. So it serves as a base, a quick way to implement DSN style configuration for components that benefit from it. Messenger and mailer are good examples.

@csarrazi
Copy link
Contributor

I agree with @fabpot's position: a Dsn class in each component should be good. What I meant is that I do not see the point in a Dsn component, living on its own.

@mdeboer
Copy link
Contributor

mdeboer commented Jul 17, 2019

I guess it could live in the framework bundle?

@francoispluchino
Copy link
Contributor

The interest of creating a small component is mainly to allow the reuse of this logic to facilitate the creation of services through a DSN (at least have a base, and add customization based on specific features). Given that there are at least 2 components (Mailer and Messenger) using the same logic, we should avoid duplicating code, but I understand very well that it is an additional component to maintain.

There was also a class in the Form component for pluralizing strings, this one ended up in a new symfony/inflector component, and it has only one class. We can still see that this component is mainly used for the Form component just by looking at the PHP doc of Inflector. Given that I had a component to do that had use this feature, but not using the Form component, I appreciate being able to use the Inflector component.

So concerning the creation or not of the DSN component, we have the same case, and as Symfony is moving more and more towards the DSN configuration, it would be interesting for all third-party components to have the same basis for also proposing the creation of service via a DSN configuration.

It will also be beneficial for the bug fix, because for example, a correction has been done on the URL decoding of the user and password parameters in the Mailer component, but this problem exists in all other classes creating a service from a DSN ; example with the Messenger transports: AmqpExt and RedisExt. In the end, the bug fix for the Mailer component is not reflected on the other components, while if they use the same basic component, this problem would have been immediately corrected for all components (by the way, it would be necessary to create an issue concerning this problem in the transports of the Messenger component when the username and/or the password has special characters and are therefore encoded).

@mdeboer If it is used by the Mailer and Messenger components, it will not be possible.

@fabpot
Copy link
Member

fabpot commented Jul 18, 2019

Don't get me wrong, I'm the one who suggested to have such a DSN component in the first place. But before coding, I really think we need to check that we will be able to cover all use cases.

@Simperfit
Copy link
Contributor Author

Simperfit commented Aug 5, 2019

@fabpot What do you think of starting it in the FrameworkBundle, using it in all components and check if we cover all our use case, if we do, then we can check if it can leave alone ?

@xabbuh
Copy link
Member

xabbuh commented Aug 5, 2019

@Simperfit I don't think having the FrameworkBundle as a dependency in our components is the way to go. This would completely contradict the direction of dependencies as they are right now (i.e. introducing circular dependencies).

@Simperfit
Copy link
Contributor Author

@xabbuh I don't see how we could not duplicate code if we don't make it live in a single place and not in a self working component.
So the first step will be to create a DSN object in every component that is using DSN ?
I guess if they are looking the same it will make more sense to create an only one. for specific use case I think the dsn object would be used as a base and we could use bridges to handle specific parsing.

@mdeboer
Copy link
Contributor

mdeboer commented Aug 5, 2019

Well configuration of components is done within the FrameworkBundle so I don't see why components would need to depend on the Framework bundle. In fact, they don't really need to know about a DSN component right now. So focus on adding DSN functionality for applications using the FrameworkBundle first then later move it to a component and make other components depend on that to bring functionality to component-only applications as well.

@carsonbot
Copy link

Thank you for this suggestion.
There has not been a lot of activity here for a while. Would you still like to see this feature?

@mdeboer
Copy link
Contributor

mdeboer commented Feb 19, 2021

Yes @carsonbot, I would ;)

@carsonbot carsonbot removed the Stalled label Feb 19, 2021
@wouterj
Copy link
Member

wouterj commented Feb 20, 2021

As far as I can see, the ideas of this issue have been proposed in #36999. Ultimately, that PR was closed as the number of different DSNs is too big to benefit from a single abstraction.

So I think it's best to close this issue (but of course, feel free to comment if you disagree after reading the PR discussion).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
RFC RFC = Request For Comments (proposals about features that you want to be discussed)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants