Skip to content

DO NOT MERGE: Debug grpc sending issue on 4.6.5 #15308

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

DO NOT MERGE: Debug grpc sending issue on 4.6.5 #15308

wants to merge 1 commit into from

Conversation

vikin91
Copy link
Contributor

@vikin91 vikin91 commented May 15, 2025 •
edited
Loading

Description

Some messages are causing marshaling errors (non-utf-8 characters) when sending message to Central. That causes the connection to crash without knowing what message caused this problem.

User-facing documentation

Testing and quality

  • the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
  • CI results are inspected

Automated testing

  • added unit tests
  • added e2e tests
  • added regression tests
  • added compatibility tests
  • modified existing tests

How I validated my change

Run it on a cluster with synthetic poison-pill message. The logs look like this:

kubernetes/clusterhealth: 2025/05/15 13:51:19.073553 updater.go:131: Info: Poison pill message sent
common/sensor: 2025/05/15 13:51:19.073800 central_receiver_impl.go:57: Info: Stopping with error: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8
common/sensor: 2025/05/15 13:51:19.073787 central_sender_impl.go:116: Error: on sending to stream: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8
common/sensor: 2025/05/15 13:51:19.073867 central_communication_impl.go:72: Error: Stopping connection due to error: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8
common/sensor: 2025/05/15 13:51:19.073961 central_communication_impl.go:210: Info: Communication with central ended.
common/sensor: 2025/05/15 13:51:19.074030 central_sender_impl.go:117: Error: message that caused the error: event:{id:"a1c73130-7b45-4297-a01d-55a08765aa6a"  action:CREATE_RESOURCE  sensor_hash:8040280699685460602  secret:{id:"a-secret"  name:"random secret"  namespace:"non_utf8_\xc5_\xc0"}}
common/sensor: 2025/05/15 13:51:19.074034 sensor.go:491: Info: Communication with Central stopped. Retrying.

@vikin91 vikin91 added the turbo-build Uses a faster path to images label May 15, 2025
@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented May 15, 2025

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@github-actions github-actions bot added area/sensor backport PR to backport changes from master to release branch labels May 15, 2025
@vikin91 vikin91 changed the title Piotr: Debug grpc 4.6.5 DO NOT MERGE: Debug grpc sending issue on 4.6.5 May 15, 2025
@vikin91 vikin91 removed the backport PR to backport changes from master to release branch label May 15, 2025
@rhacs-bot
Copy link
Contributor

Images are ready for the commit at c9348f7.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.6.5-1-gc9348f7101.

@codecov Codecov
Copy link

codecov bot commented May 15, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 48.43%. Comparing base (ec7e1c4) to head (c9348f7).
Report is 12 commits behind head on release-4.6.

Files with missing lines Patch % Lines
sensor/common/sensor/central_sender_impl.go 0.00% 2 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##           release-4.6   #15308   +/-   ##
============================================
  Coverage        48.42%   48.43%           
============================================
  Files             2470     2470           
  Lines           178524   178525    +1     
============================================
+ Hits             86458    86463    +5     
+ Misses           85113    85110    -3     
+ Partials          6953     6952    -1
Flag Coverage Δ
go-unit-tests 48.43% <0.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vikin91 vikin91 mentioned this pull request May 16, 2025
Merged
9 tasks
@vikin91
Copy link
Contributor Author

vikin91 commented Jun 5, 2025

This issue has been reproduced and the tests for that were added in #15270

@vikin91 vikin91 closed this Jun 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area/sensor do-not-merge/work-in-progress turbo-build Uses a faster path to images
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@vikin91 @rhacs-bot