CLOUDP-333692: Re-design images building #303

Julien-Ben · 2025-07-29T15:28:56Z

Re-design images building

Note for review:

Since atomic_pipeline.py is largely a refactored version of pipeline.py, it’s much clearer to review their side-by-side diff than to wade through GitHub’s “all new lines” view.
Here's the diff:

https://gist.github.com/Julien-Ben/3698d532d17bafb380f2e4f05b5db153

You can also take a look at the related TD Section

Changes

The PR refactors our Docker image build system. Most notably by replacing pipeline.py along with other components, detailed below.

Usage of standalone Dockerfiles

Added in a previous PR, they eliminate the need for templating, and make it possible to retire Sonar once the Atomic Releases Epic is completed.

Building with docker buildx, multi-platform builds

In build_images.py we use docker buildx through a python API. It eliminates the need for building images separately for each platform (ARM/AMD), and then manually bundling them in a manifest.

Handle build environments explicitly

We’ve introduced a framework that centralizes build configuration by scenario (e.g local development, staging releases etc) so the pipeline automatically picks sensible defaults (registry, target platforms, signing flags, and more) based on where you’re running.

In pipeline_main.py (with support from build_configuration.py and build_context.py) we treat each execution context (local dev, merge to master, release etc...) as an explicit, top-level environment.
It infers defaults automatically but lets you override any value via CLI flags, ensuring all build parameters live in one single source of truth rather than scattered through pipeline scripts.

CLI usage

usage: pipeline_main.py [-h] [--parallel] [--debug] [--sign] [--scenario {BuildScenario.RELEASE,BuildScenario.PATCH,BuildScenario.STAGING,BuildScenario.DEVELOPMENT}]
                        [--platform PLATFORM] [--version VERSION] [--registry REGISTRY] [--parallel-factor PARALLEL_FACTOR]
                        image

Build container images.

positional arguments:
  image                 Image to build.

options:
  -h, --help            show this help message and exit
  --parallel            Build images in parallel.
  --debug               Enable debug logging.
  --sign                Sign images.
  --scenario {BuildScenario.RELEASE,BuildScenario.PATCH,BuildScenario.STAGING,BuildScenario.DEVELOPMENT}
                        Override the build scenario instead of inferring from environment. Options: release, patch, master, development
  --platform PLATFORM   Target platforms for multi-arch builds (comma-separated). Example: linux/amd64,linux/arm64. Defaults to linux/amd64.
  --version VERSION     Override the version/tag instead of resolving from build scenario
  --registry REGISTRY   Override the base registry instead of resolving from build scenario
  --parallel-factor PARALLEL_FACTOR
                        Number of builds to run in parallel, defaults to number of cores

Proof of work

CI is building images with the new pipeline, and tests pass.

Note

For the duration of the Atomic Releases epic, both pipelines will be in the repository, until we are done with the staging and promotion process. This new pipeline will only be used for Evergreen patches.
This PR also heavily depends on changes that are introduced by the agent matrix removal, and the multi-platform support epic.

The existing Evergreen function, that uses pipeline.py has been renamed legacy_pipeline, and is used for release and periodic builds tasks.
A new one has been created, calling the new pipeline.

Once the Atomic Release Epic is complete, we'll be able to remove:

Sonar
Inventories
Periodic builds
pipeline.py

Follow up ticket to this PR: https://jira.mongodb.org/browse/CLOUDP-335471

Checklist

Have you linked a jira ticket and/or is the ticket in the title?
Have you checked whether your jira ticket required DOCSP changes?
Have you added changelog file?
- use skip-changelog label if not needed
- refer to Changelog files and Release Notes section in CONTRIBUTING.md for more details

github-actions · 2025-07-29T15:29:41Z

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.3.0 Release Notes

Other Changes

Optional permissions for PersistentVolumeClaim moved to a separate role. When managing the operator with Helm it is possible to disable permissions for PersistentVolumeClaim resources by setting operator.enablePVCResize value to false (true by default). When enabled, previously these permissions were part of the primary operator role. With this change, permissions have a separate role.
subresourceEnabled Helm value was removed. This setting used to be true by default and made it possible to exclude subresource permissions from the operator role by specifying false as the value. We are removing this configuration option, making the operator roles always have subresource permissions. This setting was introduced as a temporary solution for this OpenShift issue. The issue has since been resolved and the setting is no longer needed.

Fix build scenario Remove create and push manifests Continue improvement to main Simplify main and build_context missed Pass Build Configuration object directly Use legacy and new pipeline Fix Remove --include Rename MCO test image Multi platform builds, with buildx TODOs Implement is_release_step_executed() Fix init appdb image Import sort black formatting Some cleaning and version adjustments Adapt main to new build config Add buildscenario to buildconfig Handle build env Renaming, usage of high level config All images build pass on EVG Lint Explicit image type, support custom build_path Replace old by new pipeline in EVG Add documentation Split in multiple files, cleanup WIP, passing builds on staging temp + multi arch manifests Replace usage of sonar Remove namespace Remove pin_at and build_id Copied pipeline, removed daily builds and --exclude

This reverts commit 426e522.

scripts/release/main.py

.evergreen-functions.yml

scripts/release/main.py

scripts/release/atomic_pipeline.py

MaciejKaras · 2025-08-08T07:40:00Z

How will this be merged with Maciej's changes (the release_info file)?

@lucian-tosa I already have integration working -> #317. At least for PRs.

Julien-Ben · 2025-08-08T07:53:31Z

How will this be merged with Maciej's changes (the release_info file)?

Maciej answered above, but I also plan to sync with him as soon as this is merged, to properly rebase and wire everything togetger

lucian-tosa · 2025-08-08T07:33:22Z

scripts/release/build_context.py

+    def infer_scenario_from_environment(cls) -> "BuildScenario":
+        """Infer the build scenario from environment variables."""
+        git_tag = os.getenv("triggered_by_git_tag")
+        is_patch = os.getenv("is_patch", "false").lower() == "true"


So this is true only when triggering manual patches or PR commits? Will it be false on merges to master?

I made it more explicit, and we'll make proper use of staging registry in a follow up:
2ec7587

I was referring to the is_patch variable from evergreen. This is what I found from the devprod docs

${is_patch} is "true" if the running task is in a patch build and undefined if it is not.

Does this mean that is_patch will be false in a merge to master?

scripts/release/atomic_pipeline.py

Julien-Ben · 2025-08-08T11:14:29Z

scripts/release/build_images.py

+        logger.info(f"Created new buildx builder: {builder_name}")
+    except DockerException as e:
+        # Check if this is a race condition (another process created the builder)
+        if hasattr(e, 'stderr') and 'existing instance for' in str(e.stderr):


@nammn @lucian-tosa @MaciejKaras

I'd like your opinion on two solutions here
This function is sometimes failing when building multiple agents concurrently.
I tried two alternatives:

Catching the exception

Use ensure_buildx_builder at top leven, in the pipeline_main

None of them are ideal. Using exception is not the right way of handling concurrency. Making the main aware of buildx increase coupling.
My knowledge of python multiprocessing is limited. I know that for multi threading it is easy to make a lock with a global variable and threading.Lock() . From what I've seen online, for multi processing we would need to pass the lock from the agent building method all the way down to here.
I feel like it adds a lot of complexity and optional variables.

What solutions do you prefer ?
Do you see an alternative ?

Here is the alternative solution with ensuring builder in main:
1badff0

scripts/release/atomic_pipeline.py

nammn · 2025-08-08T11:55:10Z

scripts/release/atomic_pipeline.py

+
+    logger.info(f"Building {image_name}, dockerfile args: {build_args}")
+    logger.debug(f"Build args: {build_args}")
+    logger.debug(f"Building {image_name} for platforms={build_configuration.platforms}")


since we have the span already setup here - can we save the span attributes as we already log them anyway?

I think it should really be done as a follow up, it might look like one minute but I have no idea how to properly setup spans.
The scope of this PR is getting quite large

It's listed in the follow up ticket
https://jira.mongodb.org/browse/CLOUDP-335471

you don't have to setup anything. All you need to do is already there.
its
span.set_attribute("mck.platforms", build_configuration.platforms)
span.set_attribute("mck.registry", registry)

and the same everywhere...

nammn · 2025-08-08T11:57:45Z

scripts/release/build_images.py

+    return builder_name
+
+
+def execute_docker_build(


or here - i think adding tracing costs 1 minute and saves us all the troubles later.

especially since you are logging the exact information we need as metrics anyway

I think it should really be done as a follow up, it might look like one minute but I have no idea how to properly setup spans.
The scope of this PR is getting quite large

scripts/release/atomic_pipeline.py

This reverts commit d2a6153.

This reverts commit c6fc163.

…ros propagation (#328) # Summary Fixes issue with conflicting `Dockerfile` files on master. Sonar is creating Dockerfiles in the directory where new Dockerfile existed already. They were added here -> #289. Additionally another PR #267 made changes to how the agent images were built and especially to agent Dockerfile ARGs: https://github.com/mongodb/mongodb-kubernetes/blob/89866b79233745e27ade51dc2d9fd0e73c6124e5/docker/mongodb-agent/Dockerfile#L3-L6 This conflicted with #303 where new `atomic_pipeline.py` was still depending on the old Dockerfile structure: https://github.com/mongodb/mongodb-kubernetes/blob/774bbaca930a1752060cfe5e5bf60f24cd6c999c/docker/mongodb-agent/Dockerfile#L25-L26 ## Proof of Work Passing CI (especially agent build task) ## Checklist - [ ] Have you linked a jira ticket and/or is the ticket in the title? - [ ] Have you checked whether your jira ticket required DOCSP changes? - [x] Have you added changelog file? - use `skip-changelog` label if not needed - refer to [Changelog files and Release Notes](https://github.com/mongodb/mongodb-kubernetes/blob/master/CONTRIBUTING.md#changelog-files-and-release-notes) section in CONTRIBUTING.md for more details

Julien-Ben mentioned this pull request Jul 29, 2025

[Draft] CLOUDP-333692: Re-design images building #209

Closed

Julien-Ben force-pushed the julienben/redesign-pipeline branch from d7ae339 to 6649987 Compare July 29, 2025 15:30

Julien-Ben self-assigned this Jul 29, 2025

Julien-Ben added 19 commits July 29, 2025 17:37

Remove file

675bee4

Put lib back in dependencies

833e25f

add todo

15e7f51

Fix

120c1af

Remove multi arch call, fix test image path

c9ceabf

Fix agent version for default case

fb87f4d

Lindt

c05e180

isort

747c4ba

Cleanup TODOs

03fd9b8

Rename arch -> platform

1fbb8d5

Don't rely on exception to check for builder existence

e9a524f

Remove unused variables

fa6b899

Pre commit

426e522

Cleanup

6890858

Correct build envs

aab9592

Lindt

33173bb

Update Makefile

74e867c

Add TODO

b13b054

Revert "Pre commit"

832ce61

This reverts commit 426e522.