fix(declarative) lazy load full schema for validation and store it in partial schema

bungle · locao · commit fd4d94a66bb1 · 2021-01-09T03:12:04.000-03:00
### Summary It was reported on Kong#6547 that each time a new configuration is loaded to Kong the memory is leaked. On the issue thread there are information of the where the leak starts and where the actual leak happens. See: Kong#6547 (comment) and: Kong#6547 (comment) This PR changes the code so that we only (lazy) load the `full schema` that is needed for `full validation` only once and then we store that into partial schema. This means that we are not constantly re-creating the fullschema that seems to be cached differently inside `schema.lua` on each invocation as it uses tables as cache keys and the load seem to make new tables for fields each time we call `Declarative.load`. I am not 100% sure does it remove the leaks fully but here is a test that I did manually: #### Numbers without Patch Worker memory usage when started with example `kong.yaml`: 110 MB Worker memory usage after making 100 config updates to `:8001/config`: 714 MB #### Numbers with Patch (this commit) 110 MB (when started) 237 MB (after 100 updates) ### Issues Resolved Fix Kong#6547
diff --git a/kong/db/schema/others/declarative_config.lua b/kong/db/schema/others/declarative_config.lua
@@ -621,11 +621,13 @@ local function flatten(self, input)
     -- the error may be due entity validation that depends on foreign entity,
     -- and that is the reason why we try to validate the input again with the
     -- filled foreign keys
+    if not self.full_schema then
+      self.full_schema = DeclarativeConfig.load(self.plugin_set, true)
+    end
+
     local input_copy = utils.deep_copy(input, false)
     populate_ids_for_validation(input_copy, self.known_entities)
-    local schema = DeclarativeConfig.load(self.plugin_set, true)
-
-    local ok2, err2 = schema:validate(input_copy)
+    local ok2, err2 = self.full_schema:validate(input_copy)
     if not ok2 then
       local err3 = utils.deep_merge(err2, extract_null_errors(err))
       return nil, err3
