Skip to content

Group variable create method does not set environment scope #3138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
amimas opened this issue Feb 23, 2025 · 6 comments
Open

Group variable create method does not set environment scope #3138

amimas opened this issue Feb 23, 2025 · 6 comments

Comments

@amimas
Copy link
Contributor

amimas commented Feb 23, 2025 •
edited
Loading

Description of the problem, including code/CLI snippet

The create method for group variables is not setting the environment_scope attribute. I ran into the issue when attempting to create multiple CI CD variables at the group level where variable keys are same but has different scope.

Below is a script that can reproduce the issue. It attempts to do the following:

  • Create 3 variables in a given group where key is same but has different environment_scope defined
  • List all variables in the group
  • Delete the 3 variables that were added in step 1

To run the script, first set the environment variables mentioned in the comments section and then execute python <path-to-script.py>.

import os
import time
import gitlab

# This script is to try reproduce a bug in GitLab
# Expect following environment variables are set when this script is run:
# - GITLAB_URL: GitLab URL
# - GITLAB_API_TOKEN: GitLab API token
# - GITLAB_GROUP_ID: GitLab group ID

GITLAB_URL = os.getenv('GITLAB_URL')
GITLAB_API_TOKEN = os.getenv('GITLAB_API_TOKEN')
GITLAB_GROUP_ID = os.getenv('GITLAB_GROUP_ID')

# Print environment variables for debugging
print(f"GITLAB_URL: {GITLAB_URL}")
print(f"GITLAB_API_TOKEN: {GITLAB_API_TOKEN}")
print(f"GITLAB_GROUP_ID: {GITLAB_GROUP_ID}")

# Initialize GitLab connection
gl = gitlab.Gitlab(url=GITLAB_URL, private_token=GITLAB_API_TOKEN)

def create_variable(key, value, environment_scope):
    group = gl.groups.get(GITLAB_GROUP_ID)
    group.variables.create({
        'key': key,
        'value': value,
        'variable_type': 'env_var',
        'protected': False,
        'masked': False,
        'environment_scope': environment_scope
    })
    print(f"Created variable: {key} with scope: {environment_scope}")

def delete_variable(key, environment_scope):
    group = gl.groups.get(GITLAB_GROUP_ID)
    group.variables.delete(key, filter={"environment_scope": environment_scope})
    print(f"Deleted variable: {key} with scope: {environment_scope}")

def list_variables():
    group = gl.groups.get(GITLAB_GROUP_ID)
    variables = group.variables.list(get_all=True)
    for var in variables:
        print(f"Variable: {var.key}, Value: {var.value}, Scope: {var.environment_scope}")

# Create variables with the same key but different environment scopes
print("Creating variables...")
create_variable("TEST_VARIABLE", "test_value_prod", "production")
print()
create_variable("TEST_VARIABLE", "test_value_stage", "staging")
print()
create_variable("TEST_VARIABLE", "test_value_dev", "development")
print()

# List variables to ensure the variables are created
print("Listing variables to ensure the variables are created...")
list_variables()
print()

# Delete the variables
print("Deleting variables...")
delete_variable("TEST_VARIABLE", "production")
print()
delete_variable("TEST_VARIABLE", "staging")
print()
delete_variable("TEST_VARIABLE", "development")
print()

# List variables to ensure the variables are deleted
print("Listing variables to ensure the variables are deleted...")
list_variables()
print()

Expected Behavior

When group.variables.create() method is called, it should set/pass all the attributes to the GitLab API so that the variable is configured/created correctly.

Actual Behavior

The group.variables.create() method is not configuring the environment_scope. When running the sample script, it fails when creating the 2nd variable in the group. The error message will indicate that the failure is due to another variable with same key/name already exists:

Traceback (most recent call last):
  File "group_variable_bug.py", line 50, in <module>
    create_variable("TEST_VARIABLE", "test_value_stage", "staging")
  File "group_variable_bug.py", line 25, in create_variable
    group.variables.create({
  File "venv/lib/python3.12/site-packages/gitlab/exceptions.py", line 346, in wrapped_f
    raise error(e.error_message, e.response_code, e.response_body) from e
gitlab.exceptions.GitlabCreateError: 400: {'key': ['(TEST_VARIABLE) has already been taken']}

If we manually check the variable in the GitLab group (Group Settings -> CI CD -> Variables), we'll see the first variable being created but it does not have the environment scope - it's set to All environments as that's the default.

Image

Specifications

  • python-gitlab version: 5.6.0
  • Gitlab server version (or gitlab.com): 17.9.0
@amimas
Copy link
Contributor Author

amimas commented Feb 23, 2025

I was checking out the code and noticed following differences between Group VS Project variables. The project variable manager is configured with the environment_scope in addition to other attributes.

python-gitlab/gitlab/v4/objects/variables.py

Lines 57 to 68 in cbc613d

class ProjectVariableManager(CRUDMixin[ProjectVariable]):
_path = "/projects/{project_id}/variables"
_obj_cls = ProjectVariable
_from_parent_attrs = {"project_id": "id"}
_create_attrs = RequiredOptional(
required=("key", "value"),
optional=("protected", "variable_type", "masked", "environment_scope"),
)
_update_attrs = RequiredOptional(
required=("key", "value"),
optional=("protected", "variable_type", "masked", "environment_scope"),
)

But, the group variable manager doesn't set the environment_scope.

python-gitlab/gitlab/v4/objects/variables.py

Lines 41 to 50 in cbc613d

class GroupVariableManager(CRUDMixin[GroupVariable]):
_path = "/groups/{group_id}/variables"
_obj_cls = GroupVariable
_from_parent_attrs = {"group_id": "id"}
_create_attrs = RequiredOptional(
required=("key", "value"), optional=("protected", "variable_type", "masked")
)
_update_attrs = RequiredOptional(
required=("key", "value"), optional=("protected", "variable_type", "masked")
)

I looked through the history of the above code and noticed the group variable manger was always configured this way (environment_scope is not set). So, if this is the issue then I would expect this issue to exist since the beginning. However, I was testing out the script in an environment where I had an older version of python-gitlab installed at some point and then upgraded to latest version (5.6.0). In this environment it appeared to have worked fine - Not sure why.

Also, GitLab supports more attributes for variables (i.e. description, masked_and_hidden, raw). I haven't tried it but looking at the above code snippet it seems those attributes are not supported? I'm probably not looking at the right place and this is probably a red herring. Would be great to get some clarification on this.

@amimas amimas mentioned this issue Feb 23, 2025
Open
@amimas
Copy link
Contributor Author

amimas commented Mar 9, 2025

@nejch - When you get a chance, could you please share your insight? Do you agree with my findings? I was thinking of opening a PR to fix it but wasn't sure about the tests. In this case, should it be in the unit test or functional test? My worry about unit test is that API responses are being mocked - so, the tests would easily pass even if I don't make any code change. Not sure what the general guideline is for testing.

@holysoles
Copy link
Contributor

@amimas I'm not able to reproduce this myself, your snippet works without issue for me. However I'm also testing against GitLab 16.7.10. The following was run against a subgroup under a top level group:

Image

Stopping halfway through, the variables in the UI look correct

Image

I also tested first with python-gitlab 5.0.0, then also tested 5.6.0, no change in behavior.

Is it possible this a GitLab licensing issue? Environment scope is only available (per the docs) for Premium and Ultimate customers

@amimas
Copy link
Contributor Author

amimas commented Mar 26, 2025

Hi @holysoles - thanks for trying to reproduce it. I don't think it's a gitlab license issue. I had checked that too. I think I mentioned earlier that I was having difficulty reproducing the issue at first. After blowing away my venv and doing a fresh setup, I was able to reproduce it. Not sure why though.

I linked another PR to this issue above. That's where I ran into it first and it's blocking that PR. That's why I tried to create a small reproducible script for this issue here.

@amimas
Copy link
Contributor Author

amimas commented Apr 19, 2025

Thought I'd try again with a fresh environment. Unfortunately I'm still seeing the issue on my end. Here's what my python venv has:

$ pip list                                      
Package            Version
------------------ ---------
certifi            2025.1.31
charset-normalizer 3.4.1
idna               3.10
pip                24.0
python-gitlab      5.6.0
requests           2.32.3
requests-toolbelt  1.0.0
urllib3            2.4.0

[notice] A new release of pip is available: 24.0 -> 25.0.1
[notice] To update, run: pip install --upgrade pip

I used GitLab version 17.9.0.

GitLab version:
{"version":"17.9.0-ee","revision":"f5041566b34","kas":{"enabled":false,"externalUrl":null,"externalK8sProxyUrl":null,"version":null},"enterprise":true}

@JohnVillalovos
Copy link
Member

As a note the _create_attrs and _update_attrs are only used so that the CLI knows about them. But when using the API they are not required. It is nice to have them of course, because then the CLI could use them.

Feel free to do a PR to add them, as long as the upstream GitLab docs show they are valid.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@JohnVillalovos @amimas @holysoles