Skip to content

Pubsub msgs lost during redeployment when event_trigger_service is pubsub #419

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
TheRealVincentVanGogh opened this issue Jun 12, 2024 · 5 comments · Fixed by #420
Closed

Pubsub msgs lost during redeployment when event_trigger_service is pubsub #419

TheRealVincentVanGogh opened this issue Jun 12, 2024 · 5 comments · Fixed by #420
Labels
bug Something isn't working

Comments

@TheRealVincentVanGogh
Copy link

TL;DR

If the event_trigger_service is filled out for Gen 2 cloud function redeploy as event_trigger_service: pubsub.googleapis.com, the eventarc trigger is deleted and recreated. This causes pubsub message loss during the trigger recreation process, and any msgs that are processing during deployment may not retry on failure.

Expected behavior

If an existing eventarc subscription exists, and event_trigger_service: pubsub.googleapis.com, the existing eventarc subscription should be used.

Observed behavior

The existing eventarc subscription is deleted and recreated, causing pubsub message loss.

workaround: delete the event_trigger_service variable from the workflow file-- what is this for? it's not in the documentation but it is used in your action.

Action YAML

name: Cloud Functions Deploy
# uses: google-github-actions/deploy-cloud-functions@v2.1.0

on:
  push:
    branches:
      - deploy-staging
  workflow_dispatch:

jobs:
  job_id:
    runs-on: "ubuntu-latest"
    timeout-minutes: 30
    permissions:
      contents: "read"
      id-token: "write"

    steps:
      - uses: "actions/checkout@v4"

      - id: "auth"
        uses: "google-github-actions/auth@v2"
        with:
          credentials_json: "${{ secrets.GOOGLE_CREDENTIALS }}"

      - id: "deploy"
        uses: "google-github-actions/deploy-cloud-functions@v3"
        with:
          name: REDACTED
          environment: GEN_2
          entry_point: hello_pubsub
          region: northamerica-northeast1
          event_trigger_type: google.cloud.pubsub.topic.v1.messagePublished
          event_trigger_pubsub_topic: REDACTED
          event_trigger_service: pubsub.googleapis.com
          project_id: REDACTED
          runtime: python311
          cpu: 1
          max_instance_request_concurrency: 30
          min_instance_count: 0
          max_instance_count: 10
          memory: 1G
          service_account: REDACTED
          secrets: REDACTED
          environment_variables: |-
            CF_ENV=REDACTED
          service_timeout: 500
          vpc_connector: "REDACTED"
          vpc_connector_egress_settings: ALL_TRAFFIC
          ingress_settings: ALLOW_ALL
          event_trigger_retry: true
          labels: |-
            name=REDACTED
            environment=REDACTED

Log output

2024-06-12T18:14:52.5133147Z Current runner version: '2.317.0'
2024-06-12T18:14:52.5159085Z ##[group]Operating System
2024-06-12T18:14:52.5160030Z Ubuntu
2024-06-12T18:14:52.5160527Z 22.04.4
2024-06-12T18:14:52.5161094Z LTS
2024-06-12T18:14:52.5161603Z ##[endgroup]
2024-06-12T18:14:52.5162242Z ##[group]Runner Image
2024-06-12T18:14:52.5162841Z Image: ubuntu-22.04
2024-06-12T18:14:52.5163438Z Version: 20240609.1.0
2024-06-12T18:14:52.5164658Z Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240609.1/images/ubuntu/Ubuntu2204-Readme.md
2024-06-12T18:14:52.5166407Z Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240609.1
2024-06-12T18:14:52.5167471Z ##[endgroup]
2024-06-12T18:14:52.5168108Z ##[group]Runner Image Provisioner
2024-06-12T18:14:52.5168757Z 2.0.370.1
2024-06-12T18:14:52.5169303Z ##[endgroup]
2024-06-12T18:14:52.5170581Z ##[group]GITHUB_TOKEN Permissions
2024-06-12T18:14:52.5172384Z Contents: read
2024-06-12T18:14:52.5173086Z Metadata: read
2024-06-12T18:14:52.5174028Z ##[endgroup]
2024-06-12T18:14:52.5177378Z Secret source: Actions
2024-06-12T18:14:52.5178281Z Prepare workflow directory
2024-06-12T18:14:52.5803010Z Prepare all required actions
2024-06-12T18:14:52.5959066Z Getting action download info
2024-06-12T18:14:52.7734205Z Download action repository 'actions/checkout@v4' (SHA:a5ac7e51b41094c92402da3b24376905380afc29)
2024-06-12T18:14:52.8777472Z Download action repository 'google-github-actions/auth@v2' (SHA:71fee32a0bb7e97b4d33d548e7d957010649d8fa)
2024-06-12T18:14:53.0788264Z Download action repository 'google-github-actions/deploy-cloud-functions@v3' (SHA:043573caf530b0ceebcf854adf3acd12c1e03881)
2024-06-12T18:14:53.5150831Z Complete job name: job_id
2024-06-12T18:14:53.6096218Z ##[group]Run actions/checkout@v4
2024-06-12T18:14:53.6097071Z with:
2024-06-12T18:14:53.6097666Z   repository: REDACTED
2024-06-12T18:14:53.6098705Z   token: ***
2024-06-12T18:14:53.6099288Z   ssh-strict: true
2024-06-12T18:14:53.6099907Z   ssh-user: git
2024-06-12T18:14:53.6100504Z   persist-credentials: true
2024-06-12T18:14:53.6101235Z   clean: true
2024-06-12T18:14:53.6101780Z   sparse-checkout-cone-mode: true
2024-06-12T18:14:53.6102507Z   fetch-depth: 1
2024-06-12T18:14:53.6103035Z   fetch-tags: false
2024-06-12T18:14:53.6103654Z   show-progress: true
2024-06-12T18:14:53.6104198Z   lfs: false
2024-06-12T18:14:53.6104775Z   submodules: false
2024-06-12T18:14:53.6105326Z   set-safe-directory: true
2024-06-12T18:14:53.6106045Z ##[endgroup]
2024-06-12T18:14:53.8668800Z Syncing repository: REDACTED
2024-06-12T18:14:53.8672579Z ##[group]Getting Git version info
2024-06-12T18:14:53.8675275Z Working directory is '/home/runner/work/REDACTED'
2024-06-12T18:14:53.8678652Z [command]/usr/bin/git version
2024-06-12T18:14:53.8681258Z git version 2.45.2
2024-06-12T18:14:53.8710888Z ##[endgroup]
2024-06-12T18:14:53.8731238Z Temporarily overriding HOME='/home/runner/work/_temp/69fbf057-a51c-461b-89af-4d1cb29b02c3' before making global git config changes
2024-06-12T18:14:53.8735451Z Adding repository directory to the temporary git global config as a safe directory
2024-06-12T18:14:53.8748929Z [command]/usr/bin/git config --global --add safe.directory /home/runner/work/REDACTED
2024-06-12T18:14:53.8799197Z Deleting the contents of '/home/runner/work/REDACTED'
2024-06-12T18:14:53.8801550Z ##[group]Initializing the repository
2024-06-12T18:14:53.8803076Z [command]/usr/bin/git init /home/runner/work/REDACTED
2024-06-12T18:14:53.8875168Z hint: Using 'master' as the name for the initial branch. This default branch name
2024-06-12T18:14:53.8877486Z hint: is subject to change. To configure the initial branch name to use in all
2024-06-12T18:14:53.8879359Z hint: of your new repositories, which will suppress this warning, call:
2024-06-12T18:14:53.8880388Z hint:
2024-06-12T18:14:53.8881115Z hint: 	git config --global init.defaultBranch <name>
2024-06-12T18:14:53.8881976Z hint:
2024-06-12T18:14:53.8883103Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2024-06-12T18:14:53.8884484Z hint: 'development'. The just-created branch can be renamed via this command:
2024-06-12T18:14:53.8885414Z hint:
2024-06-12T18:14:53.8886073Z hint: 	git branch -m <name>
2024-06-12T18:14:53.8895516Z Initialized empty Git repository in /home/runner/work/REDACTED
2024-06-12T18:14:53.8901322Z [command]/usr/bin/git remote add origin https://github.com/REDACTED
2024-06-12T18:14:53.8938136Z ##[endgroup]
2024-06-12T18:14:53.8940363Z ##[group]Disabling automatic garbage collection
2024-06-12T18:14:53.8942274Z [command]/usr/bin/git config --local gc.auto 0
2024-06-12T18:14:53.8976152Z ##[endgroup]
2024-06-12T18:14:53.8978334Z ##[group]Setting up auth
2024-06-12T18:14:53.8985540Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-06-12T18:14:53.9021628Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-06-12T18:14:53.9336733Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-06-12T18:14:53.9370489Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-06-12T18:14:53.9671965Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-06-12T18:14:53.9684094Z ##[endgroup]
2024-06-12T18:14:53.9685559Z ##[group]Fetching the repository
2024-06-12T18:14:53.9697109Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --no-recurse-submodules --depth=1 origin +ff60e38401ff34fca227b48631ca781f5f180cab:refs/remotes/origin/deploy-staging
2024-06-12T18:14:54.2278960Z From https://github.com/REDACTED
2024-06-12T18:14:54.2281773Z  * [new ref]         ff60e38401ff34fca227b48631ca781f5f180cab -> origin/deploy-staging
2024-06-12T18:14:54.2319523Z ##[endgroup]
2024-06-12T18:14:54.2321748Z ##[group]Determining the checkout info
2024-06-12T18:14:54.2323910Z ##[endgroup]
2024-06-12T18:14:54.2328131Z [command]/usr/bin/git sparse-checkout disable
2024-06-12T18:14:54.2387743Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig
2024-06-12T18:14:54.2440668Z ##[group]Checking out the ref
2024-06-12T18:14:54.2444461Z [command]/usr/bin/git checkout --progress --force -B deploy-staging refs/remotes/origin/deploy-staging
2024-06-12T18:14:54.2589446Z Switched to a new branch 'deploy-staging'
2024-06-12T18:14:54.2598447Z branch 'deploy-staging' set up to track 'origin/deploy-staging'.
2024-06-12T18:14:54.2617716Z ##[endgroup]
2024-06-12T18:14:54.2672262Z [command]/usr/bin/git log -1 --format='%H'
2024-06-12T18:14:54.2711199Z 'ff60e38401ff34fca227b48631ca781f5f180cab'
2024-06-12T18:14:54.3176890Z ##[group]Run google-github-actions/auth@v2
2024-06-12T18:14:54.3177646Z with:
2024-06-12T18:14:54.3197130Z   credentials_json: ***
2024-06-12T18:14:54.3197813Z   create_credentials_file: true
2024-06-12T18:14:54.3198456Z   export_environment_variables: true
2024-06-12T18:14:54.3199190Z   universe: googleapis.com
2024-06-12T18:14:54.3199944Z   cleanup_credentials: true
2024-06-12T18:14:54.3200756Z   access_token_lifetime: 3600s
2024-06-12T18:14:54.3201662Z   access_token_scopes: https://www.googleapis.com/auth/cloud-platform
2024-06-12T18:14:54.3202616Z   id_token_include_email: false
2024-06-12T18:14:54.3203290Z ##[endgroup]
2024-06-12T18:14:54.4100207Z Created credentials file at "/home/runner/work/REDACTED"
2024-06-12T18:14:54.4331546Z ##[group]Run google-github-actions/deploy-cloud-functions@v3
2024-06-12T18:14:54.4332436Z with:
2024-06-12T18:14:54.4333016Z   name: REDACTED
2024-06-12T18:14:54.4334703Z   environment: GEN_2
2024-06-12T18:14:54.4335834Z   entry_point: hello_pubsub
2024-06-12T18:14:54.4336481Z   region: northamerica-northeast1
2024-06-12T18:14:54.4337444Z   event_trigger_type: google.cloud.pubsub.topic.v1.messagePublished
2024-06-12T18:14:54.4338638Z   event_trigger_pubsub_topic: REDACTED
2024-06-12T18:14:54.4339731Z   project_id: REDACTED
2024-06-12T18:14:54.4340421Z   runtime: python311
2024-06-12T18:14:54.4340947Z   cpu: 1
2024-06-12T18:14:54.4341499Z   max_instance_request_concurrency: 30
2024-06-12T18:14:54.4342160Z   min_instance_count: 0
2024-06-12T18:14:54.4342813Z   max_instance_count: 10
2024-06-12T18:14:54.4343343Z   memory: 1G
2024-06-12T18:14:54.4344119Z   service_account: REDACTED
2024-06-12T18:14:54.4348216Z   secrets: REDACTED
2024-06-12T18:14:54.4352284Z   environment_variables: CF_ENV=REDACTED
2024-06-12T18:14:54.4353095Z   service_timeout: 500
2024-06-12T18:14:54.4354159Z   vpc_connector: REDACTED
2024-06-12T18:14:54.4355459Z   vpc_connector_egress_settings: ALL_TRAFFIC
2024-06-12T18:14:54.4356194Z   ingress_settings: ALLOW_ALL
2024-06-12T18:14:54.4356871Z   event_trigger_retry: true
2024-06-12T18:14:54.4357516Z   labels: name=REDACTED
environment=REDACTED
2024-06-12T18:14:54.4358350Z   universe: googleapis.com
2024-06-12T18:14:54.4358908Z   source_dir: ./
2024-06-12T18:14:54.4359559Z   all_traffic_on_latest_revision: true
2024-06-12T18:14:54.4360213Z env:
2024-06-12T18:14:54.4361368Z   CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /home/runner/work/REDACTED
2024-06-12T18:14:54.4363202Z   GOOGLE_APPLICATION_CREDENTIALS: /home/runner/work/REDACTED
2024-06-12T18:14:54.4364993Z   GOOGLE_GHA_CREDS_PATH: /home/runner/work/REDACTED
2024-06-12T18:14:54.4366250Z   CLOUDSDK_CORE_PROJECT: REDACTED
2024-06-12T18:14:54.4367051Z   CLOUDSDK_PROJECT: REDACTED
2024-06-12T18:14:54.4367750Z   GCLOUD_PROJECT: REDACTED
2024-06-12T18:14:54.4368500Z   GCP_PROJECT: REDACTED
2024-06-12T18:14:54.4369293Z   GOOGLE_CLOUD_PROJECT: REDACTED
2024-06-12T18:14:54.4369983Z ##[endgroup]
2024-06-12T18:14:54.8864137Z Created zip file from './' at '/tmp/cfsrc-124c474c085477aeb063e516.zip'
2024-06-12T18:14:55.8158866Z Updating existing Cloud Functions deployment
2024-06-12T18:15:58.4162985Z Deploying Cloud Function...............
2024-06-12T18:15:58.4260310Z Post job cleanup.
2024-06-12T18:15:58.5080122Z Removed exported credentials at "/home/runner/work/REDACTED"
2024-06-12T18:15:58.5204568Z Post job cleanup.
2024-06-12T18:15:58.6149250Z [command]/usr/bin/git version
2024-06-12T18:15:58.6210965Z git version 2.45.2
2024-06-12T18:15:58.6268579Z Temporarily overriding HOME='/home/runner/work/_temp/8f498dc4-e92a-4dd7-a834-d8b693527ee5' before making global git config changes
2024-06-12T18:15:58.6271329Z Adding repository directory to the temporary git global config as a safe directory
2024-06-12T18:15:58.6286513Z [command]/usr/bin/git config --global --add safe.directory /home/runner/work/REDACTED
2024-06-12T18:15:58.6333308Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-06-12T18:15:58.6379280Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-06-12T18:15:58.6645481Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-06-12T18:15:58.6675332Z http.https://github.com/.extraheader
2024-06-12T18:15:58.6694179Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2024-06-12T18:15:58.6731561Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-06-12T18:15:58.7209411Z Cleaning up orphan processes

Additional information

No response
@TheRealVincentVanGogh TheRealVincentVanGogh added the bug Something isn't working label Jun 12, 2024
@sethvargo
Copy link
Member

Hi @TheRealVincentVanGogh - we're calling the Cloud Functions API here, which says "service" is for gen_1 functions only.

@TheRealVincentVanGogh
Copy link
Author

TheRealVincentVanGogh commented Jun 26, 2024 •
edited
Loading

Hi @TheRealVincentVanGogh - we're calling the Cloud Functions API here, which says "service" is for gen_1 functions only.

Ahh gotcha, thanks! I wonder if the event_trigger_service parameter should be ignored in the Gen 2 CF deploy action if it's only for Gen 1? I imagine folks migrating from Gen 1 to Gen 2 who leave the deprecated event_trigger_service param won't expect pubsub msg loss.

I see 2 references to event_trigger_service here in Gen 2 which causes the pubsub event trigger deletion and subsequent recreation:

deploy-cloud-functions/src/main.ts

Line 95 in 043573c

const eventTriggerService = presence(getInput('event_trigger_service'));

deploy-cloud-functions/src/main.ts

Line 153 in 043573c

service: eventTriggerService,

@sethvargo
Copy link
Member

Hi @TheRealVincentVanGogh - there are many changes between the v1 and v2 API (and gen1 vs gen2). Part of the reason we released a new major version was that the migration path and deprecation warnings were a minefield of untestable spaghetti code.

Are you able to confirm that things work as-expected when event_trigger_service is omitted? I can add some validations, but I want to make sure that it actually addresses the issue :)

@TheRealVincentVanGogh
Copy link
Author

TheRealVincentVanGogh commented Jun 27, 2024 •
edited
Loading

Hi @TheRealVincentVanGogh - there are many changes between the v1 and v2 API (and gen1 vs gen2). Part of the reason we released a new major version was that the migration path and deprecation warnings were a minefield of untestable spaghetti code.

Are you able to confirm that things work as-expected when event_trigger_service is omitted? I can add some validations, but I want to make sure that it actually addresses the issue :)

Yup 👍 everything works as-expected when event_trigger_service is omitted. I guess what I'm getting at is as someone moving from Gen 1 to Gen 2 CFs (v2 to v3 deploy action) it's a bit unexpected that the v3 action:

  1. Indicates the event_trigger_service parameter won't be used:

Warning: Unexpected input(s) 'event_trigger_service', valid inputs are...

  1. and has a readme that doesn't mention event_trigger_service

but that leaving that parameter in does have an undocumented effect on the gen 2 deployment process (loss of queued up / retrying pubsub msgs)

All in all though, it's a small thing-- just wanted to leave a clear paper trail in case someone else bumps into this issue. And yes! adding some validations sounds like a good idea.

Thanks!

@sethvargo sethvargo mentioned this issue Jun 27, 2024
Merged
@sethvargo
Copy link
Member

and has a readme that doesn't mention event_trigger_service

Oh, that's quite odd. GitHub Action is supposed to reject undeclared inputs in the action.yml.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove legacy event_trigger_service input google-github-actions/deploy-cloud-functions
2 participants
@sethvargo @TheRealVincentVanGogh