[Config][FrameworkBundle] Generate JSON schema for config #59620

valtzu · 2025-01-26T14:30:16Z

Q	A
Branch?	7.3
Bug fix?	no
New feature?	yes
Deprecations?	no
Issues	Fix #59603
License	MIT

Generate JSON schema in %kernel.build_dir%/config.schema.json for YAML config when the container is built to improve DX.

To discuss:

~~Is there any way to deal with PHP enums (!php/enum yaml tag)? At the moment they are just ignored/excluded~~ // just allowed a string !php/enum ...
~~Should we include # $schema: path/to/schema.json in generated yaml files?~~ probably no / up to Flex etc
Any edge-cases I missed?
~~Should we rather bump symfony/config version constraint in symfony/framework-bundle than check for class existence? This is kinda optional stuff so I thought we don't need to enforce it.~~ already bumped in [FrameworkBundle] bump symfony/config requirement to 7.3 #59649

GromNaN · 2025-01-26T19:46:13Z

Thank you very much for working on this feature.
I gave it a try and it works really well.

The per-environment configuration need to be added. This is the same as the current root definition (that need to be a type), with key following this pattern: when@[a-zA-Z0-9]+
The JSON schema file will be different for each environment, because the bundles are not the sames. The JSON schema must allow any additional root keys for when@ other environments.

Here is the files that are not validated by the generated JSON schema.

`doctrine.yaml`

Some doctrine configuration are not recognized.

`framework.yaml`

All configuration that canBeEnabled (like framework.esi) must accept a boolean value. All equivalentValues must be allowed.

`routing.yaml`

"Missing required property 'resource'. Since the configurations are merged from multiple files (and defaults) properties cannot be required in a single Yaml file.

`validator.yaml`

The framework.validation.auto_mapping.<class-name> value must be an object, but an empty array is accepted.

valtzu · 2025-01-26T21:23:44Z

Thanks for quick review/comment @GromNaN!

Resolved when@ (though it does not account for env-specific extensions yet), removed "required": [...] from the schema, added using equivalentValues.

The validator issue still on to-do list, not sure on what basis to allow the empty array – ~~maybe useAttributeAsKey~~ just added maxItems: 0 to non-prototyped arrays.

The doctrine issue seems to be caused by normalization in a closure, some possible solutions below 👇

always allow extra keys / have "additionalProperties": true in the schema (for every subschema), not to cause false-positives on keys that for whatever reason don't appear in the schema
do some manual override / manually defined schema to cover this special case.
convert (dynamically?) the xml schema to json schema (not sure if it matches yaml one-to-one)

GromNaN · 2025-01-27T10:34:17Z

Resolved when@ (though it does not account for env-specific extensions yet), removed "required": [...] from the schema, added using equivalentValues.

Thanks, perfect.

The validator issue still on to-do list, not sure on what basis to allow the empty array – maybe useAttributeAsKey 🤔

I think the JSON schema can be a bit more strict than the config validation rules. But this error may be hiding something else.

The doctrine issue seems to be caused by normalization in a closure, some possible solutions below 👇

always allow extra keys / have "additionalProperties": true in the schema (for every subschema), not to cause false-positives on keys that for whatever reason don't appear in the schema

Arbitrary additional properties are not allowed by the Config component. This would fail the purpose of the JSON schema.

do some manual override / manually defined schema to cover this special case.

It's the last option if you really can't do any better.

convert (dynamically?) the xml schema to json schema (not sure if it matches yaml one-to-one)

I'd prefer to be able to generate the xsd rather than use it.

Maybe we don't need to support this for a first version, since there's an alternative syntax (more verbose).

GromNaN

I did a quick code review.

src/Symfony/Component/Config/Definition/JsonSchemaGenerator.php

src/Symfony/Bundle/FrameworkBundle/CacheWarmer/ConfigSchemaCacheWarmer.php

src/Symfony/Component/Config/Tests/Definition/JsonSchemaGeneratorTest.php

src/Symfony/Component/Config/Definition/JsonSchemaGenerator.php

valtzu · 2025-01-29T19:03:17Z

Did a whole lot of cleanup based on the comments.

I also managed to workaround the Doctrine issue by allowing any type when $node->getNormalizedTypes() contains ExprBuilder::TYPE_ANY.

This of course means we don't get a schema for those, but I think it's better than false-positives when using example configuration.

Also added a deprecation message to the schema which shows up quite nicely now

GromNaN · 2025-01-29T19:43:32Z

Awesome, the deprecation message is exactly why I want such schema validation.

valtzu · 2025-02-03T17:40:01Z

I'm a little stuck with the support for env-specific extensions: since the schema is generated inside the kernel, it does not seem possible to generate schemas for other environments. And since the kernel class is not really exposed anywhere, it's not trivial to create multiple instances of the kernel with different envs (not that it would be a great idea anyway).

Best I could think of is

always only build schema for the current environment
in addition to saving the schema into single file inside env-specific directory, also insert a reference into common file outside env-specific dir

This could look something like

var/cache/dev/config.schema.json
var/cache/test/config.schema.json
var/cache/whatever/config.schema.json
var/config.schema.json # references all files above, like {"when@dev": { "$ref": "cache/dev/config.schema.json" } }

The downside with this solution is that unless you build the container for those other envs, you also don't get the schema for those. Also, I doubt that many people build the container locally for any env but dev & test.

If someone has a better approach, great!

GromNaN

We could look at bundles.php to find all the environments and bundles used, but that wouldn't cover generic use cases (if bundles are registered directly in Kernel.php, for example).

We could also scan all the directories and find all the classes that implement the BundleInterface, but that would require scanning composer autoloader and would take up too many bundles.

I think the idea of generating independent schema files for each env is more realist:

generate var/cache/dev/config.schema.json, var/cache/test/config.schema.json and var/cache/prod/config.schema.json.
when a file is generated, create or update the parent var/cache/config.schema.json to include the generated fields.

I tried this with PHPStorm, but the file loading doesn't work yet.

{
  "$schema": "http://json-schema.org/draft-06/schema#",
  "allOf": [
    { "$ref": "./dev/config.schema.json" },
    { "$ref": "./test/config.schema.json" }
  ]
}

GromNaN · 2025-01-29T19:47:48Z

src/Symfony/Component/Config/Definition/JsonSchemaGenerator.php

+            $allowNull = $allowNull || \in_array(ExprBuilder::TYPE_NULL, $normalizedTypes);
+            if (\in_array(ExprBuilder::TYPE_ANY, $normalizedTypes)) {
+                // This will make IDEs not complain about configurations containing complex beforeNormalization logic
+                $subSchemas[] = new \ArrayObject();


For an empty object, it's more correct to use stdClass. You are using the fact that any object that doesn't expose any property is encoded as an empty object.

Suggested change

$subSchemas[] = new \ArrayObject();

$subSchemas[] = new \stdClass();

I'm intentionally using \ArrayObject (just like Symfony Serializer does internally) so that I can perform array operations on it, f.e. $schema['default'] = $node->getDefaultValue() and \count($subSchema, \COUNT_RECURSIVE).

~~If \stdClass was used instead, I think the logic would be overall more complex.~~

Switched to \stdClass everywhere

GromNaN · 2025-01-29T19:48:18Z

src/Symfony/Component/Config/Definition/JsonSchemaGenerator.php

+            }
+        }
+
+        return $subSchemas ?: [new \ArrayObject()];


Suggested change

return $subSchemas ?: [new \ArrayObject()];

return $subSchemas ?: [new \stdClass()];

GromNaN · 2025-01-29T19:48:46Z

src/Symfony/Component/Config/Definition/JsonSchemaGenerator.php

+        $schema['type'][] = $type;
+    }
+
+    private function getReference(array|\ArrayObject $subSchema, array &$definitions): array|\ArrayObject


Suggested change

private function getReference(array|\ArrayObject $subSchema, array &$definitions): array|\ArrayObject

private function getReference(array|\stdClass $subSchema, array &$definitions): array|\stdClass

Using \stdClass for schemas everywhere now & thus got rid of array by-ref, reasoning in #59620 (comment)

GromNaN · 2025-02-18T16:02:03Z

src/Symfony/Bundle/FrameworkBundle/CacheWarmer/ConfigSchemaCacheWarmer.php

+
+    public function isOptional(): bool
+    {
+        return false;


I think this is optional. The app doesn't relies on it to run.

valtzu · 2025-02-18T21:21:50Z

I tried this with PHPStorm, but the file loading doesn't work yet.

Yeah, that is sad. I ended up putting everything in a single file, %kernel.build_dir%/../config.schema.json.

Also switched from array to \stdClass because after merging configs between envs I was getting [] where {} was expected, breaking the whole schema.

So now you'd need to run

bin/console --env=dev
bin/console --env=test
bin/console --env=prod

etc to populate all the configs – though in reality I think most projects have all the bundles & extensions available in dev, thus nothing extra except dev build should be needed.

carsonbot added Status: Needs Review Config Feature FrameworkBundle labels Jan 26, 2025

carsonbot added this to the 7.3 milestone Jan 26, 2025

valtzu force-pushed the dx-config-schema branch 2 times, most recently from 80efba3 to d7ab6dc Compare January 26, 2025 14:33

GromNaN mentioned this pull request Jan 26, 2025

[DX][Config] Generate JSON schema for Yaml configuration #59603

Open

valtzu force-pushed the dx-config-schema branch 2 times, most recently from 24fdcc8 to 2d3536a Compare January 26, 2025 21:17

valtzu force-pushed the dx-config-schema branch 2 times, most recently from 84f3965 to 008ee37 Compare January 27, 2025 22:23

GromNaN requested changes Jan 28, 2025

View reviewed changes

carsonbot added Status: Needs Work and removed Status: Needs Review labels Jan 28, 2025

valtzu force-pushed the dx-config-schema branch 3 times, most recently from 3cfc28e to de09b68 Compare January 29, 2025 18:55

GromNaN mentioned this pull request Feb 12, 2025

[Config] Add NodeDefinition::docUrl() #59762

Merged

GromNaN reviewed Feb 18, 2025

View reviewed changes

valtzu force-pushed the dx-config-schema branch 2 times, most recently from 8715b5d to 003b6f9 Compare February 18, 2025 21:16

Generate JSON schema for config

73e3fd8

valtzu force-pushed the dx-config-schema branch from 003b6f9 to 73e3fd8 Compare February 18, 2025 21:28

GromNaN mentioned this pull request Apr 11, 2025

[RFC] Deprecate and remove support for semantic XML configuration #60200

Open

	$subSchemas[] = new \ArrayObject();
	$subSchemas[] = new \stdClass();

	return $subSchemas ?: [new \ArrayObject()];
	return $subSchemas ?: [new \stdClass()];

	private function getReference(array\|\ArrayObject $subSchema, array &$definitions): array\|\ArrayObject
	private function getReference(array\|\stdClass $subSchema, array &$definitions): array\|\stdClass

Uh oh!

[Config][FrameworkBundle] Generate JSON schema for config #59620

Are you sure you want to change the base?

[Config][FrameworkBundle] Generate JSON schema for config #59620

Conversation

valtzu commented Jan 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

To discuss:

Uh oh!

GromNaN commented Jan 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

valtzu commented Jan 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GromNaN commented Jan 27, 2025

Uh oh!

GromNaN left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

valtzu commented Jan 29, 2025

Uh oh!

GromNaN commented Jan 29, 2025

Uh oh!

valtzu commented Feb 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GromNaN left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

valtzu Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

valtzu commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

valtzu commented Jan 26, 2025 •

edited

Loading

GromNaN commented Jan 26, 2025 •

edited

Loading

valtzu commented Jan 26, 2025 •

edited

Loading

valtzu commented Feb 3, 2025 •

edited

Loading

valtzu Feb 18, 2025 •

edited

Loading

valtzu commented Feb 18, 2025 •

edited

Loading