Skip to content

[py] New script to update Python dependencies #15845

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Jun 3, 2025
Merged

[py] New script to update Python dependencies #15845

merged 6 commits into from
Jun 3, 2025

Conversation

cgoldberg
Copy link
Contributor

@cgoldberg cgoldberg commented Jun 2, 2025 •
edited by qodo-merge-pro bot
Loading

User description

💥 What does this PR do?

This PR adds a new shell script (./scripts/update_py_dependencies.sh) that is used for keeping Python development dependencies up to date. It is not part of the build process, and should be run locally on occasion.

When you run it, it will:

  • create and activate a temporary virtual env
  • install the current package dependencies from py/requirements.txt
  • upgrade the package dependencies to the latest versions available on PyPI
  • run pip freeze to generate a new py/requirements.txt file
  • run bazel run //py:requirements.update to generate a new py/requirements_lock.txt file
  • deactivate and remove the temporary virtual env

This PR also contains updates to the package dependency versions.

🔄 Types of changes

  • development/packaging tooling

PR Type

Enhancement

Description

  • Add script to automate Python dependency updates

  • Upgrade multiple Python package versions in requirements

  • Regenerate requirements lock file for updated dependencies

  • Update dependency versions in pyproject.toml and tox.ini

Changes walkthrough 📝

Relevant files
Enhancement
update_py_dependencies.sh
Add script to automate Python dependency updates                 

scripts/update_py_dependencies.sh

  • Add new shell script to automate updating Python dependencies
  • Script creates a temp virtualenv, upgrades packages, regenerates
    requirements and lock files
  • Provides clear instructions and cleanup steps
    		• +73/-0   
    Dependencies
    requirements.txt
    Update Python dependency versions in requirements.txt       

    py/requirements.txt

  • Upgrade versions for multiple Python dependencies
  • Align requirements with latest available on PyPI
  • Minor format and version consistency adjustments
    		• +11/-11 
    requirements_lock.txt
    Regenerate requirements_lock.txt for new dependency versions

    py/requirements_lock.txt

  • Regenerate lock file to reflect updated dependency versions
  • Update hashes and package versions for consistency
    		• +169/-171
    pyproject.toml
    Update dependency version in pyproject.toml                           

    py/pyproject.toml

  • Bump typing_extensions version in dependencies
  • Ensure consistency with requirements.txt
    		• +1/-1     
    tox.ini
    Update test dependency versions in tox.ini                             

    py/tox.ini

  • Update mypy and lxml versions in mypy testenv
  • Bump types-urllib3 version for type checking
    		• +3/-3     
    Need help?
  • Type /help how to ... in the comments thread for any questions about Qodo Merge usage.
  • Check out the documentation for more information.
    •

    @selenium-ci selenium-ci added C-py Python Bindings B-build Includes scripting, bazel and CI integrations labels Jun 2, 2025
    @qodo-merge-pro Qodo Merge Pro
    Copy link
    Contributor

    qodo-merge-pro bot commented Jun 2, 2025

    PR Reviewer Guide 🔍

    Here are some key observations to aid the review process:

    ⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
    🧪 No relevant tests
    🔒 No security concerns identified
    ⚡ Recommended focus areas for review

    Error Handling

    The script doesn't handle errors that might occur during package upgrades or when generating new requirements files. Consider adding error checking after critical operations.

     
    echo "upgrading outdated dependencies ..."
echo
pip list --outdated | while read -r line; do
    if [[ ! "${line}" =~ "Version Latest" && ! "${line}" =~ "----" ]]; then
        read -ra fields <<< "${line}"
        echo "upgrading ${fields[0]} from ${fields[1]} to ${fields[2]}"
        pip install --upgrade "${fields[0]}==${fields[2]}" > /dev/null
    fi
done
    Dependency Filtering

    The script upgrades all outdated dependencies without any filtering mechanism. Consider adding a way to exclude certain packages that might need to be pinned at specific versions.

     
    pip list --outdated | while read -r line; do
    if [[ ! "${line}" =~ "Version Latest" && ! "${line}" =~ "----" ]]; then
        read -ra fields <<< "${line}"
        echo "upgrading ${fields[0]} from ${fields[1]} to ${fields[2]}"
        pip install --upgrade "${fields[0]}==${fields[2]}" > /dev/null
    fi
done

    @qodo-merge-pro Qodo Merge Pro
    Copy link
    Contributor

    qodo-merge-pro bot commented Jun 2, 2025 •
    edited
    Loading

    PR Code Suggestions ✨

    Explore these optional code suggestions:

    CategorySuggestion                                                                                                                                    Impact
    Possible issue
    ✅ Fix missing package extra
    Suggestion Impact:The commit directly implemented the suggestion by changing urllib3==2.4.0 to urllib3[socks]==2.4.0 in requirements.txt, ensuring consistency with pyproject.toml and including the necessary socks dependencies

    code diff:

    -urllib3==2.4.0
+urllib3[socks]==2.4.0

    The urllib3 package is specified without the [socks] extra in requirements.txt,
    but the pyproject.toml file specifies urllib3[socks]. This inconsistency could
    lead to missing dependencies when installing from requirements.txt directly.

    py/requirements.txt [59-60]

     typing_extensions==4.14.0
-urllib3==2.4.0
+urllib3[socks]==2.4.0

    [Suggestion processed]

    Suggestion importance[1-10]: 8

    __

    Why: The suggestion identifies a real inconsistency where urllib3[socks] in pyproject.toml differs from urllib3 in requirements.txt, which could lead to missing socks functionality when installing directly from requirements.txt.

    		Medium
    Use robust JSON parsing

    The script assumes a specific format for pip list --outdated output, but this
    format can vary between pip versions. The current implementation may fail if the
    output columns change or if package names contain spaces. Use a more robust
    approach to parse the output.

    scripts/update_py_dependencies.sh [49-55]

    -pip list --outdated | while read -r line; do
-    if [[ ! "${line}" =~ "Version Latest" && ! "${line}" =~ "----" ]]; then
-        read -ra fields <<< "${line}"
-        echo "upgrading ${fields[0]} from ${fields[1]} to ${fields[2]}"
-        pip install --upgrade "${fields[0]}==${fields[2]}" > /dev/null
-    fi
-done
+pip list --outdated --format=json | python -c "
+import json, sys
+for pkg in json.load(sys.stdin):
+    name = pkg['name']
+    old_version = pkg['version']
+    new_version = pkg['latest_version']
+    print(f\"upgrading {name} from {old_version} to {new_version}\")
+    import subprocess
+    subprocess.check_call(['pip', 'install', '--upgrade', f'{name}=={new_version}'], stdout=subprocess.DEVNULL)
+"
    • Apply / Chat
    Suggestion importance[1-10]: 7

    __

    Why: The suggestion correctly identifies a fragility in parsing pip list --outdated output that could fail with different pip versions or package names containing spaces. Using JSON format provides a more robust solution.

    		Medium
    General
    Improve virtual environment cleanup

    The script attempts to deactivate the virtual environment directly, which may
    fail in certain shell contexts. Use a more reliable approach to ensure the
    virtual environment is properly deactivated before removal.

    scripts/update_py_dependencies.sh [65-67]

     echo "deleting virtual env: ${VENV}"
-deactivate
+# Ensure virtual env is deactivated properly
+if [[ -n "${VIRTUAL_ENV}" ]]; then
+    deactivate || true
+fi
 rm -rf "${VENV}"
    • Apply / Chat
    Suggestion importance[1-10]: 5

    __

    Why: The suggestion provides defensive programming for virtual environment deactivation, though the issue is relatively minor since the script creates its own temporary environment and should have control over the activation state.

    		Low
    • Update

    @cgoldberg cgoldberg merged commit 0fe5ed2 into SeleniumHQ:trunk Jun 3, 2025
    17 of 18 checks passed
    @cgoldberg cgoldberg deleted the py-update-dependencies-script branch June 3, 2025 14:02
    @niklasmohrin niklasmohrin mentioned this pull request Jun 16, 2025
    Merged
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Reviewers
    No reviews
    Assignees
    No one assigned
    Labels
    B-build Includes scripting, bazel and CI integrations C-py Python Bindings Review effort 2/5
    Projects
    None yet
    Milestone
    No milestone
    Development

    Successfully merging this pull request may close these issues.
    2 participants
    @cgoldberg @selenium-ci