Skip to content

feat: add a new type of hvcs for gitlab to support Gitlab Job Tokens #797

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from

Conversation

aborsu
Copy link

@aborsu aborsu commented Jan 6, 2024

This supports GITLAB_CI exclusively via CI_JOB_TOKEN.

Switching from PAT to JOB-TOKEN is relatively easy as it only requires switching the key used in the header from PRIVATE-TOKEN to JOB-TOKEN respectively.

But the the original Gitlab hvcs class supports running outside of gitlab-ci and thus queries the API to get information that is available in the environment of the gitlab-ci job.
Additionally, some of the queries used by the original Gitlab class are not allowed using a Job Token.
cfr. api-usage

# make an API request to create the gl.user object. This is not required but may be useful
# to validate your token authentication. Note that this will not work with job tokens.
gl.auth()

Supporting both running inside and outside Gitlab-CI within the same class is also possible, but we would have to make it explicit to avoid conflicts in the config resolution.

@aborsu
Copy link
Author

aborsu commented Jan 6, 2024

I'm still planning to write documentation for this, but I would appreciate a first feedback on the approach.
Thank you

This supports GITLAB_CI exclusively via CI_JOB_TOKEN
As this requires a different authentication mechanism than
using a Personal Access Token, I decided to create a
seperate hvcs as supporting both in a single class seemed
quite difficult.
@aborsu aborsu changed the title feat: add a new type of hvcs feat: add a new type of hvcs for gitlab to support Gitlab Job Tokens Jan 7, 2024
@aborsu
Copy link
Author

aborsu commented Jan 7, 2024

This should resolve #663 and #666, please tell me if I'm forgetting something?

@codejedi365
Copy link
Contributor

codejedi365 commented Jan 8, 2024

@aborsu, I appreciate you taking the initiative on this. I'll take a look more in depth tomorrow. I would rather not cause confusion between a GitLab CI and a regular GitLab as it is the intent each HVCS is to be able to be run in the CI and locally. If you can adapt the current GitLab Hvcs object that would be best.

I had the same thought to have it work with the CI_JOB_TOKEN which means we have to modify the GitLab(*_token) call accordingly. Personally I was thinking of implementing the function in python-gitlab's cli.py which will derive the correct token based on the environment which I think is what we need but also a merge between using the current configuration parsing for environment variables.

The other problem we have is the call to gl.auth(), this did not work for me with a CI_JOB_TOKEN. Were you able to get this to work?

Separately, when I tried to solve this problem previously, the tag publishing is the biggest limitation. If you run semantic-release version --push then it will try to execute git push --tags to publish the commit & or tag to the repository. The CI_JOB_TOKEN does not have enough scoped permissions for this. It can write to releases but it cannot write to the repository which is where the tags go. Secondly, I don't think you can make a vcs release without a tag to link it too. I have detailed this further in #768.

Were you able to solve these problems?

@aborsu
Copy link
Author

aborsu commented Jan 8, 2024

@codejedi365 I can try to bring the two together later this week.
As I wrote in my first message gl.auth() will not work with a CI_JOB_TOKEN as written in the documentation of python-gitlab api-usage.
This is because the CI_JOB_TOKEN has only access to a subset of the API, and on top of that there is no way to push using a CI_JOB_TOKEN cfr. 389060.

Using the CI_JOB_TOKEN with python-gitlab is broken as many methods do not work. At the same time all the information is available in the environment variables. Basically, I would be fine putting a choice job_token or pat_token in a single class, the difficulty is the fact that the current Gitlab class works from outside the CI, mine doesn't so that would be a breaking change.

My ideal workflow (but only works on gitlab ultimate or premium), is that we have a Project Access Token as a guest on the project with a token that only allows it to push to master. It cannot use the API, it cannot push to another branch.
We use those credentials to push the commit and tag.

We then use the CI_JOB_TOKEN to create the release in HVCS.

@aborsu
Copy link
Author

aborsu commented Jan 8, 2024

I added the following doc

Using a Job Token

In order to use gitlab-ci job token you need to configure python semantic release by adding the following to your
pyproject.toml file:

    [tool.semantic_release]
    remote.type = "gitlab-ci"
    remote.ignore_token_for_push = "true"

The following workflow is proposed to the reader.

A Project Access Token is created in Gitlab and given the right to push to the main branch.
This token is added as a secret variable in your Gitlab CI project under the name GL_PROJECT_TOKEN_NAME and
GL_PROJECT_TOKEN. This token should only have write access to the repository, it does not require API access.
If you are using Gitlab Premium or Ultimate, you can make this token a guest to further restrain the token's scope,
and specify it as the sole person allowed to push to the master branch.
Otherwise, you will have to grant that project access token a sufficiently high access privilege that it can push to
the main branch.

The following .gitlab-ci.yml should be enough to get you started:

    stages:
      - publish # Deploy new version of package to registry
    # Official language image. Look for the different tagged releases at:
    # https://hub.docker.com/r/library/python/tags/
    image: python:latest
    variables:
      PIP_CACHE_DIR: $CI_PROJECT_DIR/.cache/pip # Set folder in working dir for cache
    # Runs on commit in main branch that were not made by the semantic-release job
    # Using GITLAB_USER_EMAIL in the GIT_COMMIT_AUTHOR will display the person who
    # triggered the job either by clicking the merge button or pushing to master
    # as the author of the commit.
    version_and_publish:
      stage: publish
      image: python:latest
      variables:
        GIT_DEPTH: 0
        GIT_COMMIT_AUTHOR: "$GL_PROJECT_TOKEN_NAME <$GITLAB_USER_EMAIL>"
      before_script:
        - pip install python-semantic-release
        - pip install twine
      script:
        - git checkout "$CI_COMMIT_REF_NAME"
        - git remote set-url origin https://${GL_PROJECT_TOKEN_NAME}:${GL_PROJECT_TOKEN}@${CI_REPOSITORY_URL#*@}
        - semantic-release version
        - |
          if [ "dist" ]; then
            TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*
          fi
      cache:
        paths:
          - ${PIP_CACHE_DIR}
      rules:
        # Don't run on automatic commits
        - if: $CI_COMMIT_AUTHOR =~ /$GL_PROJECT_TOKEN_NAME.*/
          when: never
        # Only run on main/master branch
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

@sbates
Copy link

sbates commented Feb 9, 2024

I'm very interested in this PR. Does it also imply that a PAT doesn't actually work here? I've been grinding my face on the keyboard all week trying to figure out how to set up auth for Gitlab.

@codejedi365
Copy link
Contributor

codejedi365 commented Feb 10, 2024

Does it also imply that a PAT doesn't actually work here?

@sbates No, a Personal, Group or Project Access Token will work if your token permissions are correct.

It is the CI_JOB_TOKEN that does not work inherently due to the pushing of commits and/or tags. CI_JOB_TOKEN's have restricted api endpoint permissions that are not modifiable and unfortunately the ability to write to the remote git repository is not one of them.

Writing to the repository is one such permission that PSR requires.

Copy link

This PR is stale because it has not been confirmed or planned by the maintainers and had been open 60 days with no recent activity. It will be closed in 10 days,
if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label Apr 14, 2024
Copy link

This PR was closed because activity was dormant for 70 days.

@github-actions github-actions bot closed this Apr 25, 2024
@Exagone313
Copy link

Hello,

Is it possible to re-open this PR, and maybe add the confirmed label to prevent GitHub Actions from closing it again?

It's true that it is currently stale as some changes were requested (not having a new provider), but I believe that closing a PR with a good start doesn't help getting progress on the feature.

Thank you 😃

@codejedi365
Copy link
Contributor

codejedi365 commented Aug 9, 2024

@Exagone313, although I see your point, I don't agree with your solution. My issue is that PRs are restricted to the owner whom opened them and the maintainers. So even if a new contributor came along to complete it they would open a new PR. Even having a new contributor come along is unfortunately very unlikely based on my experience. But if they did and they wanted to use the code, they still can from a closed PR by using git fetch. It is closed because it's not going to be merged from its current state. And as time goes on the head of this branch diverges more and more from the current state of the project, which is more complicated to integrate over time.

In this specific PR, There are current issues that indicate this is still a feature request, so it is not forgotten. I have been monitoring this PR as I still value GitLab usage but I've been working other more significant efforts. Until recently job tokens were not possible as I stated above but luckily changed as identified in #977. I have extracted what code is valuable from this PR and kept it updated on my own branch. But given my feedback previously I can't say this is the code someone else should start with.

But all in all, a PR will be closed if it is not being worked, as it dilutes priorities of all the other progress of the project (remaining issues + PRs).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants