[aws-eks] Creating a ServiceAccount in a different stack than the EKS Cluster causes a circular dependency. #8884

roystchiang · 2020-07-03T00:52:26Z

I'm trying to create a ServiceAccount in a different stack than the EKS cluster. CDK throws a circular dependency error.
I tried to work around it by importing the cluster in my service account stack and creating a resource, but I can't because of ServiceAccount expects a Cluster instead of ICluster, which Cluster.fromClusterAttributes() returns.

Reproduction Steps

import {Cluster, ServiceAccount} from "@aws-cdk/aws-eks";

export class EKSCluster extends Stack {
  public eksCluster: Cluster;

  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);
    this.eksCluster = new Cluster(this, "EKSCluster");
  }
}

export class KubernetesApp extends Stack {
  constructor(scope: Construct, id: string, props: StackProps & { cluster: Cluster }) {
    super(scope, id, props);

    new ServiceAccount(this, 'testAccount', {cluster: props.cluster, name: 'test-account', namespace: 'test'})
  }
}

const app = new App();
const eksCluster = new EKSCluster(app, 'EKSCluster');
new KubernetesApp(app, "KubeApp", {cluster: eksCluster.eksCluster});

Error Log

Using cdk out from cdk.json: /Users/roychi/projects/cdk/src/cdk/build/cdk.out
/Users/roychi/projects/cdk/src/cdk/node_modules/@aws-cdk/core/lib/stack.js:413
            throw new Error(`'${target.node.path}' depends on '${this.node.path}' (${cycle.join(', ')}). Adding this dependency (${reason}) would create a cyclic reference.`);
            ^

Error: 'EKSCluster' depends on 'KubeApp' (EKSCluster -> KubeApp/testAccount/Role/Resource.Arn). Adding this dependency (KubeApp -> EKSCluster/EKSCluster/Resource/Resource/Default.OpenIdConnectIssuer) would create a cyclic reference.

Environment

CLI Version :1.45.0
Framework Version:1.45.0
Node.js Version:v12.18.1
OS :MacOS
Language (Version):TypeScript

Other

This is 🐛 Bug Report

The text was updated successfully, but these errors were encountered:

RicoToothless · 2020-07-10T07:14:04Z

I face the same issue.
the workaround is use cluster.add_service_account (python) in your KubernetesApp stack.
and add dependency stack like eks_cluster.add_dependency(kubernetes_app) (python).
but the problem is all the resources not in KubernetesApp cloudformation stack. Is on your EKSCluster cloudformation stack.

kingli-crypto · 2020-07-15T07:11:22Z

Similar problem with addAutoScalingGroup

Code to reproduce

import { Cluster, ServiceAccount, KubernetesVersion, EksOptimizedImage, NodeType } from "@aws-cdk/aws-eks";
import { Stack, Construct, StackProps, App } from "@aws-cdk/core";
import { AutoScalingGroup } from "@aws-cdk/aws-autoscaling";
import { InstanceType } from "@aws-cdk/aws-ec2";

export class EKSCluster extends Stack {
    public eksCluster: Cluster;

    constructor(scope: Construct, id: string, props?: StackProps) {
        super(scope, id, props);
        this.eksCluster = new Cluster(this, "EKSCluster", { version: KubernetesVersion.V1_16 });
    }
}

export class KubernetesApp extends Stack {
    constructor(scope: Construct, id: string, props: StackProps & { cluster: Cluster }) {
        super(scope, id, props);

        props.cluster.addAutoScalingGroup(new AutoScalingGroup(this, 'autoScaling', {
            instanceType: new InstanceType('t3.medium'),
            vpc: props.cluster.vpc,
            machineImage: new EksOptimizedImage({
                kubernetesVersion: KubernetesVersion.V1_16.version,
                nodeType: NodeType.STANDARD
            }),
        }), {})
    }
}

const app = new App();
const eksCluster = new EKSCluster(app, 'EKSCluster');
new KubernetesApp(app, "KubeApp", { cluster: eksCluster.eksCluster });
app.synth();

Error messages

'EKSCluster' depends on 'KubeApp' (EKSCluster -> KubeApp/autoScaling/InstanceRole/Resource.Arn). Adding this dependency (KubeApp -> EKSCluster/EKSCluster/Resource/Resource/Default.Ref) would create a cyclic reference.

Language: TypeScript
CDK version: v1.51.0
Node: v10.21.0

…Cluster` creates circular dependency between the two stacks (#9701) This PR changes a few scenarios with regards to circular dependencies in cases where some resources are created in a different stack than the cluster stack. > Reviewers, Please refer to this detail as a first response to PR questions :) ### ServiceAccount Previously, the `ServiceAccount` construct used `cluster.addManifest` to deploy the necessary resource. https://github.com/aws/aws-cdk/blob/25a9cc7fabbe3b70add48edfd01421f74429b97f/packages/%40aws-cdk/aws-eks/lib/service-account.ts#L81-L95 This means that the resource itself is added to the **cluster stack**. When the `ServiceAccount` is created in a different stack, it creates a dependency between the cluster stack and the service account stack. Since `ServiceAccount` also depends on the cluster, it creates a dependency between the service account stack and the cluster stack. And hence a circular dependency is formed. #### Solution There is no inherent reason to always add the `ServiceAccount` resource to the cluster stack. If it was added to the service account stack, the circular dependency could be avoided. The solution is to use `new KubernetesManifest(this, ...)` instead of `cluster.addResource` - creating the manifest in the service account stack, which is perfectly fine since that direction of dependency is the intended one. ### AutoScalingGroup Capacity When adding capacity to a cluster using an `AutoScalingGroup`, we add the role of the ASG to the `aws-auth` role mappings of the cluster: https://github.com/aws/aws-cdk/blob/25a9cc7fabbe3b70add48edfd01421f74429b97f/packages/%40aws-cdk/aws-eks/lib/cluster.ts#L914-L923 The ASG depends on the cluster because, among others, it requires to have a tag with the cluster name: https://github.com/aws/aws-cdk/blob/25a9cc7fabbe3b70add48edfd01421f74429b97f/packages/%40aws-cdk/aws-eks/lib/cluster.ts#L907-L909 This creates a dependency between the ASG stack and the cluster stack. In case the ASG role is defined in the ASG stack, the auth mapping now creates a dependency between the cluster stack and the ASG, forming a circular dependency between the stacks. #### Solution `AwsAuth` is a singleton of the cluster, which means it is always defined in the cluster stack. Since we have no control over the stack of the ASG and its role, a circular dependency may always be created. The solution is to simply disallow creating `userMappings` or `roleMappings` with a `User/Role` that is not part of the cluster stack. This might be a little more restrictive than necessary, but it has less exposure potential to edge cases and complex dependency cycles. We can always be less restrictive down the road if needed. -------------- Fixes #8884 Fixes #9325 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

roystchiang added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jul 3, 2020

github-actions bot added the @aws-cdk/aws-eks Related to Amazon Elastic Kubernetes Service label Jul 3, 2020

github-actions bot assigned eladb Jul 3, 2020

eladb added the p1 label Jul 8, 2020

eladb modified the milestone: EKS Dev Preview Jul 22, 2020

eladb added the effort/small Small work item – less than a day of effort label Aug 4, 2020

iliapolo removed the needs-triage This issue or PR still needs to be triaged. label Aug 9, 2020

iliapolo assigned iliapolo and unassigned eladb Aug 9, 2020

iliapolo added this to the EKS Dev Preview milestone Aug 10, 2020

iliapolo mentioned this issue Aug 11, 2020

[aws-eks] All eks.ServiceAccount constructs Are Added to Cluster Stack #9325

Closed

iliapolo added the in-progress This issue is being actively worked on. label Aug 11, 2020

iliapolo mentioned this issue Aug 14, 2020

fix(eks): creating a ServiceAccount in a different stack than the Cluster creates circular dependency between the two stacks #9701

Merged

iliapolo changed the title ~~[eks] Creating a ServiceAccount in a different stack than the EKS Cluster causes a circular dependency.~~ [aws-eks] Creating a ServiceAccount in a different stack than the EKS Cluster causes a circular dependency. Aug 16, 2020

iliapolo mentioned this issue Aug 23, 2020

[aws-eks] ServiceAccount property references do not result in dependencies #9910

Closed

mergify bot closed this as completed in #9701 Sep 2, 2020

iliapolo removed the in-progress This issue is being actively worked on. label Sep 9, 2020

fspaniol mentioned this issue Mar 10, 2021

[aws-eks]: Allow creating masters role in stacks other than the cluster stack #13529

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[aws-eks] Creating a ServiceAccount in a different stack than the EKS Cluster causes a circular dependency. #8884

[aws-eks] Creating a ServiceAccount in a different stack than the EKS Cluster causes a circular dependency. #8884

roystchiang commented Jul 3, 2020

RicoToothless commented Jul 10, 2020 •

edited

Loading

kingli-crypto commented Jul 15, 2020

[aws-eks] Creating a ServiceAccount in a different stack than the EKS Cluster causes a circular dependency. #8884

[aws-eks] Creating a ServiceAccount in a different stack than the EKS Cluster causes a circular dependency. #8884

Comments

roystchiang commented Jul 3, 2020

Reproduction Steps

Error Log

Environment

Other

RicoToothless commented Jul 10, 2020 • edited Loading

kingli-crypto commented Jul 15, 2020

Code to reproduce

Error messages

RicoToothless commented Jul 10, 2020 •

edited

Loading