Fix configuration of Metadata Agent daemon set #56576

kawych · 2017-11-29T14:31:22Z

What this PR does / why we need it:
Fixes small errors in Stackdriver Metadata Agent configuration: port number and default version.

Release note:

Fix port number and default Stackdriver Metadata Agent in daemon set configuration.

kawych · 2017-11-29T14:32:38Z

/assign @bmoyles0117
/assign @piosz
/unassign @vishh
/unassign @jszczepkowski

k8s-ci-robot · 2017-11-29T14:32:41Z

@kawych: GitHub didn't allow me to assign the following users: bmoyles0117.

Note that only kubernetes members can be assigned.

In response to this:

/assign @bmoyles0117
/assign @piosz
/unassign @vishh
/unassign @jszczepkowski

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kawych · 2017-11-29T15:52:43Z

/retest

kawych · 2017-11-29T17:28:40Z

cc @bmoyles0117 @igorpeshansky

igorpeshansky · 2017-11-29T18:03:30Z

cluster/addons/metadata-agent/stackdriver/metadata-agent.yaml

@@ -22,7 +22,7 @@ spec:
        name: metadata-agent
        ports:
        - containerPort: 8000
-          hostPort: 8000
+          hostPort: 8799


Don't you also need to change the logging agent configuration to point to this port?

Yes, but the logging agent doesn't support the host as a parameter. We should discuss if the port should be part of the same parameter, or a separate parameter entirely. Making it part of METADATA_AGENT_HOSTNAME would be easier, as we could check for the existence of a ":" in the hostname, and append :8000 if not found. WDYT?

I'm confused -- the logging agent uses the metadata_agent_url parameter, which can definitely include the port. Are you talking about the environment variables used by the container's entrypoint script? If so, then we can add a METADATA_AGENT_PORT variable or rename the current one to METADATA_AGENT_URL.

Yeah I'm referencing the container's entrypoint script.

This is where the sed replace occurs
https://github.com/GoogleCloudPlatform/google-fluentd/blob/master/docker/run.sh#L13

And this is what it's replacing "local-metadata-agent.stackdriver.com":8000, not the :8000. Easy enough to update.
https://github.com/GoogleCloudPlatform/google-fluentd/blob/master/docker/Dockerfile#L9

We can discuss the right naming for the env variable, but for now it's not modifiable in any existing docker image.

Are you OK with this being final configuration for Metadata Agent?

I'm ok with this configuration for the Metadata agent. However, in order for these changes to matter we still need to make changes to a few other places.

The input source for containers needs to incorporate the local_resource_id label. You can see how did it below.
The section that this needs to be added in is here: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-gcp/fluentd-gcp-configmap.yaml#L74

$ curl -s https://storage.googleapis.com/stackdriver-container-alpha/configs/stackdriver-agents.yaml | cat -n | grep -C 5 local_resource_id 205 <match reform.**> 206 @type record_reformer 207 enable_ruby true 208 <record> 209 "logging.googleapis.com/local_resource_id" ${"k8s_container.#{tag_parts[4].split('-')[-1]}"} 210 </record> 212 tag ${if record['stream'] == 'stderr' then 'raw.kubernetes.stderr' else 'raw.kubernetes.stdout' end} 213 </match>

You need to update the google_cloud output plugin for kubernetes.* to enable the metadata agent.

https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-gcp/fluentd-gcp-configmap.yaml#L351

You can do this by adding these lines. However you manage to get these lines in there, you'll need to ensure that the hostname is set to the spec.nodeName from the downwards API. We do this in our run.sh as you're familiar with here: https://github.com/GoogleCloudPlatform/google-fluentd/blob/master/docker/run.sh#L15

# Enable metadata agent lookups. enable_metadata_agent true metadata_agent_url "http://local-metadata-agent.stackdriver.com:8799"

@crassirostris
Mik, do you agree with these changes? Do you have this in your plans?

kawych · 2017-11-30T12:22:34Z

/assign @gmarek

piosz · 2017-11-30T13:56:33Z

/lgtm
/approve no-issue

bmoyles0117

Please note my comments as we'll need to make changes to the fluentd-gcp kubernetes manifests.

bmoyles0117 · 2017-12-01T23:01:08Z

cluster/addons/metadata-agent/stackdriver/metadata-agent.yaml

@@ -22,7 +22,7 @@ spec:
        name: metadata-agent
        ports:
        - containerPort: 8000
-          hostPort: 8000
+          hostPort: 8799


I'm ok with this configuration for the Metadata agent. However, in order for these changes to matter we still need to make changes to a few other places.

The input source for containers needs to incorporate the local_resource_id label. You can see how did it below.
The section that this needs to be added in is here: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-gcp/fluentd-gcp-configmap.yaml#L74

$ curl -s https://storage.googleapis.com/stackdriver-container-alpha/configs/stackdriver-agents.yaml | cat -n | grep -C 5 local_resource_id 205 <match reform.**> 206 @type record_reformer 207 enable_ruby true 208 <record> 209 "logging.googleapis.com/local_resource_id" ${"k8s_container.#{tag_parts[4].split('-')[-1]}"} 210 </record> 212 tag ${if record['stream'] == 'stderr' then 'raw.kubernetes.stderr' else 'raw.kubernetes.stdout' end} 213 </match>

You need to update the google_cloud output plugin for kubernetes.* to enable the metadata agent.

https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-gcp/fluentd-gcp-configmap.yaml#L351

You can do this by adding these lines. However you manage to get these lines in there, you'll need to ensure that the hostname is set to the spec.nodeName from the downwards API. We do this in our run.sh as you're familiar with here: https://github.com/GoogleCloudPlatform/google-fluentd/blob/master/docker/run.sh#L15

# Enable metadata agent lookups. enable_metadata_agent true metadata_agent_url "http://local-metadata-agent.stackdriver.com:8799"

k8s-github-robot · 2017-12-04T15:07:10Z

[MILESTONENOTIFIER] Milestone Pull Request Needs Attention

@gmarek @kawych @piosz @kubernetes/sig-instrumentation-misc

Action required: During code freeze, pull requests in the milestone should be in progress.
If this pull request is not being actively worked on, please remove it from the milestone.
If it is being worked on, please add the status/in-progress label so it can be tracked with other in-flight pull requests.

Note: This pull request is marked as priority/critical-urgent, and must be updated every 1 day during code freeze.

Example update:

ACK.  In progress
ETA: DD/MM/YYYY
Risks: Complicated fix required

Pull Request Labels

sig/instrumentation: Pull Request will be escalated to these SIGs if needed.
priority/critical-urgent: Never automatically move pull request out of a release milestone; continually escalate to contributor and SIG through all available channels.
kind/bug: Fixes a bug discovered during the current release.

Help

jberkus · 2017-12-04T20:07:14Z

Please update the status of this issue. Is this PR ready to merge? Does it require another PR for supporting changes?

igorpeshansky · 2017-12-04T20:42:02Z

This PR is ready to merge. Another PR will be needed to change configurations of other components accordingly.

igorpeshansky

LGTM

kawych · 2017-12-04T21:57:51Z

@gmarek PTAL

gmarek · 2017-12-05T01:13:18Z

cluster/gce/config-default.sh

@@ -158,7 +158,7 @@ ENABLE_METRICS_SERVER="${KUBE_ENABLE_METRICS_SERVER:-true}"
 ENABLE_METADATA_AGENT="${KUBE_ENABLE_METADATA_AGENT:-none}"

 # Version tag of metadata agent
-METADATA_AGENT_VERSION="${KUBE_METADATA_AGENT_VERSION:-0.2-0.0.13-5}"
+METADATA_AGENT_VERSION="${KUBE_METADATA_AGENT_VERSION:-0.2-0.0.13-5-watch}"


what's that? what's the versioning scheme of metadata agent?

This version will change in a bit once the watch branch is merged into master. This scheme is: <Dockerfile version>-<package version>[-<branch>]...

gmarek · 2017-12-05T01:13:52Z

/approve no-issue

k8s-github-robot · 2017-12-05T01:13:57Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gmarek, kawych, piosz

Associated issue requirement bypassed by: gmarek, piosz

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~cluster/addons/metadata-agent/OWNERS~~ [kawych,piosz]
~~cluster/gce/OWNERS~~ [gmarek]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-12-05T01:14:26Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2017-12-05T01:57:44Z

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

Fix configuration of Metadata Agent daemon set

b314d18

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Nov 29, 2017

k8s-github-robot assigned jszczepkowski and vishh Nov 29, 2017

k8s-ci-robot assigned piosz and unassigned vishh and jszczepkowski Nov 29, 2017

igorpeshansky suggested changes Nov 29, 2017

View reviewed changes

k8s-ci-robot assigned gmarek Nov 30, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 30, 2017

piosz added this to the v1.9 milestone Nov 30, 2017

k8s-github-robot added the milestone/incomplete-labels label Nov 30, 2017

piosz added kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. labels Nov 30, 2017

k8s-github-robot added milestone/needs-approval and removed milestone/incomplete-labels labels Nov 30, 2017

piosz added the status/approved-for-milestone label Nov 30, 2017

k8s-github-robot added milestone/removed and removed milestone/needs-approval labels Nov 30, 2017

k8s-github-robot removed this from the v1.9 milestone Nov 30, 2017

piosz added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Nov 30, 2017

piosz removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. milestone/removed labels Nov 30, 2017

piosz added this to the v1.9 milestone Nov 30, 2017

k8s-github-robot added the milestone/needs-attention label Nov 30, 2017

bmoyles0117 approved these changes Dec 1, 2017

View reviewed changes

igorpeshansky approved these changes Dec 4, 2017

View reviewed changes

gmarek reviewed Dec 5, 2017

View reviewed changes

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 5, 2017

k8s-github-robot merged commit 2175199 into kubernetes:master Dec 5, 2017

piosz removed the milestone/needs-attention label Dec 8, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix configuration of Metadata Agent daemon set #56576

Fix configuration of Metadata Agent daemon set #56576

kawych commented Nov 29, 2017

kawych commented Nov 29, 2017

k8s-ci-robot commented Nov 29, 2017

kawych commented Nov 29, 2017

kawych commented Nov 29, 2017

igorpeshansky Nov 29, 2017

bmoyles0117 Nov 29, 2017

igorpeshansky Nov 29, 2017

bmoyles0117 Nov 29, 2017

kawych Nov 30, 2017

bmoyles0117 Dec 1, 2017

kawych Dec 4, 2017

kawych commented Nov 30, 2017

piosz commented Nov 30, 2017

bmoyles0117 left a comment

bmoyles0117 Dec 1, 2017

k8s-github-robot commented Dec 4, 2017

jberkus commented Dec 4, 2017

igorpeshansky commented Dec 4, 2017

igorpeshansky left a comment

kawych commented Dec 4, 2017

gmarek Dec 5, 2017

igorpeshansky Dec 5, 2017 •

edited

Loading

gmarek commented Dec 5, 2017

k8s-github-robot commented Dec 5, 2017

k8s-github-robot commented Dec 5, 2017

k8s-github-robot commented Dec 5, 2017

Fix configuration of Metadata Agent daemon set #56576

Fix configuration of Metadata Agent daemon set #56576

Conversation

kawych commented Nov 29, 2017

kawych commented Nov 29, 2017

k8s-ci-robot commented Nov 29, 2017

kawych commented Nov 29, 2017

kawych commented Nov 29, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kawych commented Nov 30, 2017

piosz commented Nov 30, 2017

bmoyles0117 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-github-robot commented Dec 4, 2017

jberkus commented Dec 4, 2017

igorpeshansky commented Dec 4, 2017

igorpeshansky left a comment

Choose a reason for hiding this comment

kawych commented Dec 4, 2017

Choose a reason for hiding this comment

igorpeshansky Dec 5, 2017 • edited Loading

Choose a reason for hiding this comment

gmarek commented Dec 5, 2017

k8s-github-robot commented Dec 5, 2017

k8s-github-robot commented Dec 5, 2017

k8s-github-robot commented Dec 5, 2017

igorpeshansky Dec 5, 2017 •

edited

Loading