Advanced container security

Jason Umiker
Solutions Architect, AWS

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Introduction

A (brief) refresher on security in AWS

Security considerations of Amazon ECS (in both Amazon EC2 and

AWS Fargate mode) and Amazon EKS

Container image best practices and scanning approaches

Conclusion

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenges of containers at scale
• More transient/dynamic
• More distributed and complex
• More services interdependent over a network
• Scheduling / Scaling / Resource Management
• Less isolated
• Share a kernel
• Often share a network and—in case of Amazon Elastic Container
Service for Kubernetes (Amazon EKS)—a network interface

All these new challenges have solutions or mitigations, though.

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
We give you the power to choose
ECS EKS
1. Choose your
orchestration tool

Coming
Soon!

2. Choose your
launch type

EC2 Fargate EC2 Fargate

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shared responsibility model

Customer Data
CUSTOMER

Responsible for
security “in” the Applications Platform Identity & Access Management
cloud

Operating System Network and Firewall Configuration

Compute Storage Database Networking

Responsible for
AWS

security “of” the Regions Availability Zones Edge Locations

cloud

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shared responsibility model

Customer Data
CUSTOMER

Responsible for
security “in” the Applications Platform Identity & Access Management
cloud

Operating System Network and Firewall Configuration

Compute Storage Database Networking

Responsible for
AWS

security “of” the Regions Availability Zones Edge Locations

cloud

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shared responsibility model

Customer Data
CUSTOMER

Responsible for
security “in” the Applications Platform Identity & Access Management
cloud

Operating System Network and Firewall Configuration

Compute Storage Database Networking

Responsible for
AWS

security “of” the Regions Availability Zones Edge Locations

cloud

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
IAM = Who can do what on the platform / in the cluster?
People Code/Pipelines

Photo & License Photo & License

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Invest in end-to-end automation via pipelines
• AWS infrastructure as code
• Code and container builds
• Security (DevSecOps)
• Deployments

Make it fast and easy for your

team to do the right thing!

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why DevSecOps via pipelines?
Because if you don’t make it
fast and easy to do the right
thing, people will often just go
around the rules/restrictions to
get their work done!

Photo &
License

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Identity and Access Management (IAM) vs.
Kubernetes RBAC
If using Amazon ECS, then that If running Kubernetes, either
is a native extension of the yourself or via Amazon EKS,
AWS solution and is fully you need to understand and
managed by IAM. configure BOTH IAM and
Kubernetes RBAC.

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EKS: IAM authentication + kubectl

K8s API IAM

kubectl Authentication

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EKS: IAM authentication + kubectl

Passes AWS identity

K8s API IAM

kubectl Authentication

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EKS: IAM authentication + kubectl

Passes AWS identity

Verifies AWS identity

K8s API IAM

kubectl Authentication

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EKS: IAM authentication + kubectl

Passes AWS identity

Verifies AWS identity

K8s API IAM

kubectl Authentication

Authorization of AWS identity

against Kubernetes RBAC

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EKS: IAM authentication + kubectl

Passes AWS identity

Verifies AWS identity

K8s action allowed/denied K8s API IAM

kubectl Authentication

Authorization of AWS identity

against Kubernetes RBAC

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kubernetes RBAC built-in ClusterRoles
Default
Description
ClusterRole

Allows superuser access to perform any action on any resource. When used in
a ClusterRoleBinding, it gives full control over every resource in the cluster and in all
cluster-admin
namespaces. When used in a RoleBinding, it gives full control over every resource in the
Rolebinding's namespace, including the namespace itself.

Allows admin access, intended to be granted within a namespace using a RoleBinding. If

used in a RoleBinding, allows read/write access to most resources in a namespace,
admin
including the ability to create roles and rolebindings within the namespace. It does not
allow write access to resource quota or to the namespace itself.

Allows read/write access to most objects in a namespace. It does not allow viewing or
edit
modifying roles or rolebindings.

Allows read-only access to see most objects in a namespace. It does not allow viewing
view
roles or rolebindings. It does not allow viewing secrets, since those are escalating.

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kubernetes RBAC basics
Kubernetes has roles, which are kind: ClusterRole
defined and apply within a single metadata:
namespace (a virtual cluster) and name: cluster-admin
ClusterRoles, which apply rules:

cluster-wide across all - apiGroups:

- '*'
namespaces. resources:
- '*'
verbs:
You define custom roles - '*'
describing resources (such as - nonResourceURLs:
pods and nodes); verbs (such as - '*'
get, update, and delete) are verbs:
allowed against them. - '*'

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
IAM role per task/pod?
Assigning an IAM role to an instance/task/function means the right
AWS access key and secret to call the AWS CLI/SDK are transparently
obtained and rotated.
Assigning an IAM role to an Amazon If running Kubernetes, you need to
ECS task is an included feature in the add either kube2iam or kiam to
AWS offering and “just works.” leverage this functionality.

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS container road map on GitHub
https://github.com/aws/containers-roadmap

Captured
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
5/19/2019
Shared responsibility model

Customer Data
CUSTOMER

Responsible for
security “in” the Applications Platform Identity & Access Management
cloud

Operating System Network and Firewall Configuration

Compute Storage Database Networking

Responsible for
AWS

security “of” the Regions Availability Zones Edge Locations

cloud

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Logging and auditing the control plane
Logging of the control plane, especially around an audit trail of API
actions, is an important aspect of security.
Amazon ECS is part of the AWS offering, Kubernetes’ control plane logs include an
and the control plane logs go to AWS audit trail. With Amazon EKS, these logs are
CloudTrail just like the rest of the solution. not exposed by default, but you can (and
should) enable sending them to Amazon
CloudWatch Logs.

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shared responsibility model

Customer Data
CUSTOMER

Responsible for
security “in” the Applications Platform Identity & Access Management
cloud

Operating System Network and Firewall Configuration

Compute Storage Database Networking

Responsible for
AWS

security “of” the Regions Availability Zones Edge Locations

cloud

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS security groups vs. Kube network policies

If using Amazon ECS, then that is If running Kubernetes yourself or

an extension of the AWS offering Amazon EKS, you need to
and you only need to understand understand and configure BOTH
and configure Amazon VPC and Amazon VPCs / AWS security
AWS security groups. groups as well as Kubernetes
network policies

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Networking with Amazon ECS
Default/Root Namespace

When using Amazon ECS with the

aws-vpc network mode (optional for
lo
EC2 mode but required for Fargate
mode), each task gets its own eth0
dedicated elastic network interface
(ENI).
Task Namespace

Since each task is 1:1 with an ENI

and each ENI is 1:1 with a security
lo
group (SG), any communication in/out
of each task goes through its SG on eth1
both ingress and egress.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Micro-segmenting with security groups

You can use a security group ID as both a source and a destination for
other security group rules—both to loop back to itself and to reference
other SGs.

This enables network segmentation without complex subnetting.

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Private Amazon EKS control plane API endpoints
Master VPC (AWS account)

etcd
Public == false
etcd

API Server API Server

Private == true AZ 1 AZ 2

Worker VPC (your account)

EKS-owned ENIs

Private hosted zone

prod-cluster-123.eks.amazonaws.com

Kubectl
Worker Worker
node node

Kubelet Kubelet

Kube-proxy Kube-proxy

AZ 1 AZ 2

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Networking with Amazon EKS
https://github.com/aws/amazon-vpc-cni-k8s
VPC
ec2.associateaddress()

ENI

10.0.0.20
10.0.0.1

10.0.0.2 10.0.0.22

Instance 1 ENI Instance 2

VPC Subnet – 10.0.0.0/24 Secondary IPs:

Secondary IPs:
10.0.0.1 10.0.0.20
10.0.0.2 10.0.0.22

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Installing a network policy provider on Kubernetes

You first need to add a network policy provider to Amazon EKS /

Kubernetes in order to use network policies. A popular one covered in
our documentation is Calico.
https://docs.aws.amazon.com/eks/latest/userguide/calico.html

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Network policies on Kubernetes
catsndogs-namespace

kind: NetworkPolicy
apiVersion: extensions/v1beta1
metadata:
Cats Dogs name: default-deny
spec:
podSelector:
matchLabels: {}

Frontend

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Network policies on Kubernetes
catsndogs-namespace

kind: NetworkPolicy
apiVersion: extensions/v1beta1
metadata:
Cats Dogs name: default-deny
spec:
podSelector:
matchLabels: {}

Frontend

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Network policies on Kubernetes
catsndogs-namespace

apiVersion: extensions/v1beta1
kind: NetworkPolicy
metadata:
name: public-to-frontend
spec:

Cats Dogs podSelector:

matchLabels:
role: frontend
ingress:
- from:
- ipBlock:
cidr: "0.0.0.0/0"
ports:
- protocol: TCP

Frontend
port: 80

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Network policies on Kubernetes
catsndogs-namespace

apiVersion: extensions/v1beta1
kind: NetworkPolicy
metadata:
name: public-to-frontend
spec:

Cats Dogs podSelector:

matchLabels:
role: frontend
ingress:
- from:
- ipBlock:
cidr: "0.0.0.0/0"
ports:
- protocol: TCP

Frontend
port: 80

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Network policies on Kubernetes
catsndogs-namespace

apiVersion: extensions/v1beta1
kind: NetworkPolicy
metadata:
name: frontend-to-cats
spec:

Cats Dogs
podSelector:
matchLabels:
role: cats
ingress:
- from:
- podSelector:
matchLabels:
role: "frontend"
ports:
- protocol: TCP
Frontend port: 80

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tigera Secure Cloud Edition (CE)
Features:
• Enterprise support from Tigera
• Host-to-host IPSEC encryption
• Flow logs enriched with Kubernetes workload metadata
• Integration between AWS security groups and network policies

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tigera Secure Cloud Edition (CE)

ElastiCache

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Alternative: Multiple NodeGroups or Clusters
One way that you can both assign Amazon EC2 instance-level IAM roles
(without kops or kiam) and fully trust security group-based micro-
segmentation without Tigera is to have a different set of worker nodes,
or even entirely separate Clusters, for different services or trust
boundaries.

Amazon EKS has the concept of a NodeGroup, which is a separate

automatically scaling group of worker nodes that can be labeled in
such a way that you can limit which pods/services can be run on them.
https://kubernetes.io/docs/concepts/configuration/assign-pod-node

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Wildcard: Service mesh instead?
Instead of imposing network-level restrictions like AWS security groups
or Kubernetes network policies, you can use a service mesh to both
encrypt and authenticate all of your services, allowing for a flatter
more unsegmented underlying network, while still staying safe.

AWS App Mesh Istio

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why a service mesh?
How to generate uniform logs,
metrics, and traces
Common need: Manage interservice traffic How to load balance traffic
How to shift traffic between
deployments
http/tcp
How to decouple service teams
How to minimize impact to
application code
How to ensure all traffic is
Service Service
encrypted
team A team B How to ensure that the service
making the request is the one
that SHOULD be making it

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why AWS App Mesh?

Control plane
Proxy

Sits between all services

http/tcp
Manages and observes traffic

Control plane
Service Service
Translates intent to proxy config
team A team B Distributes proxy config

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sidecar proxy with containers

Task or pod

Application
code as a
External traffic container

Proxy runs
as a container

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS App Mesh and Istio both use Envoy

OSS project

Wide community support, numerous integrations

Stable and production-proven

Graduated Project in Cloud Native Computing Foundation

Started at Lyft in 2016

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
App Mesh control plane configures every proxy

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Mutual TLS authentication (mTLS)
Certificate Authority Client Validates Server Cert
Certificate Generation /
and
Signing for Client &
Server Validates Client Cert
Server Client Cert
Server Cert

Client Server

Private Key Private Key

Certificate Exchange
Client Cert Client Cert

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS App Mesh vs. Istio
App Mesh Istio (on Amazon EKS)
• Orchestrates Envoy sidecar • Orchestrates Envoy sidecar
• Integrated with Kubernetes • Integrated with Kubernetes
• Does not yet support • Supports encryption and
encryption and mTLS mTLS
• Wider integration with AWS • Requires Kubernetes
• Regional managed service • State stored in Kubernetes’
• Fully supported by AWS control plane and etcd via
custom resources
• Best-effort support by AWS

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS AppMesh road map on GitHub
https://github.com/aws/aws-app-mesh-roadmap

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Captured
19/5/2019
Shared responsibility model

Customer Data
CUSTOMER

Responsible for
security “in” the Applications Platform Identity & Access Management
cloud

Operating System Network and Firewall Configuration

Compute Storage Database Networking

Responsible for
AWS

security “of” the Regions Availability Zones Edge Locations

cloud

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EC2 mode: Customer responsibilities
• Instance type and quantity to choose?
• What is the CPU-to-RAM ratio?
• Excess capacity for scaling and
availability?
• Which OS to choose?
• If Amazon Linux, we provide AMIs
• Hardening the OS (e.g., against CIS
benchmark)
• Patching of the OS, Docker, Amazon
ECS agent or kubelet, etc.

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Photo & License
Security benefits of AWS Fargate
We do more, you do less.

• Patching (OS, Docker, Amazon ECS agent, etc.)

• Task isolation (via separate Clusters)
• No --privileged mode for containers
• Requires awsvpc network mode, so ENI/SG per task
• No runtime access for users (ssh or interactive
Docker)

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EC2-mode ECS shared responsibility model

Customer Data
CUSTOMER

Images ECS Config Identity & Access Management

Instance Scaling Operating System Network and Firewall Configuration

ECS Control Plane Compute Storage Database Networking

AWS

Regions Availability Zones Edge Locations

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon ECS and AWS Fargate shared responsibility
model
Customer Data
CUSTOMER

Images ECS Config Identity & Access Management

Network and Firewall Configuration

Operating System Compute Storage Database Networking

AWS

Instance Scaling Regions Availability Zones Edge Locations

ECS Control Plane

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Updating Amazon EKS
• Kubernetes has a new major version every quarter
• Kubernetes has a new minor version quite regularly
• Sometimes Kubernetes updates are security-related
• Amazon EKS has APIs to trigger an update of the
control plane
• You then need to update the worker nodes—re:
Kubernetes as well as Docker and OS
•Often the workers are in an automatically scaling
group, so this means building/updating AMIs
•We provide a regularly updated Amazon EKS node AMI
as well as scripts to build your own

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shared responsibility model

Customer Data
CUSTOMER

Responsible for
security “in” the Applications Platform Identity & Access Management
cloud

Operating System Network and Firewall Configuration

Compute Storage Database Networking

Responsible for
AWS

security “of” the Regions Availability Zones Edge Locations

cloud

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Parameter Store, AWS Secrets Manager, and
Kubernetes Secrets
AWS has both Parameter Store and Secrets Kubernetes’ built-in Secrets functionality
Manager to store your secrets. They are stores secrets in its control plane and puts
integrated into Amazon ECS, but you’ll them into running pods via environment
need to call them within the pod on variables or files in the file system. You
Kubernetes via our CLI or SDK. can’t use these outside of the Kubernetes
cluster.

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shared responsibility model

Customer Data
CUSTOMER

Responsible for
security “in” the Applications Platform Identity & Access Management
cloud

Operating System Network and Firewall Configuration

Compute Storage Database Networking

Responsible for
AWS

security “of” the Regions Availability Zones Edge Locations

cloud

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security best practices for container images

Less is more (secure)

• No secrets in them Container

• One service per container Image References

parent
- Use sidecars within task/pod Image
image

• Minimize container footprint Base image

- Include only what is needed at runtime
bootfs

kernel

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security best practices for container images

• Use known and trusted base images

- Official on Docker Hub
- Read the Dockerfiles Container

- Scan the image for CVEs Image References

• Specify USER in Dockerfile parent

image
Image
(otherwise it’s root)
• Unique and informative image tags Base image

- Be able to tell which commit at a bootfs

glance kernel

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Image scanning
• Scan images in your registry
Docker Hub does this
On our road map to do with Amazon Elastic Container Registry
(Amazon ECR) as well
• Scan images in your build pipeline
•Clair - https://github.com/coreos/clair
•Aqua Microscanner - https://github.com/aquasecurity/microscanner
• Scan images at runtime / running containers (these can also scan in
pipeline)
•Aqua - https://www.aquasec.com/solutions/aws-container-security/
•Twistlock - https://www.twistlock.com/solutions/aws-security/

All of the above?

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Runtime container security
You can protect against zero-day vulnerabilities not yet even in a CVE database
via APN Partner products from the likes of Aqua and Twistlock.

• Limit what can execute within container(s) via rules engine

e.g., “Do not run things that were not in the image” or
“Do not run things that are not on this whitelist”
• Ensure only trusted images can be deployed/run in your cluster
• Get visibility into the runtime behavior of the entire environment
• Detect vulnerable running containers as soon as a CVE is made public

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
https://aws.amazon.com/containers/partner-solutions/

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summing up
The customer has many responsibilities when running containers securely in AWS.

This is even more true when running Kubernetes on the platform than when
running Amazon ECS.

The key areas to delve into include:

• Identity and access management
• Network topology and firewalling
• Logging and auditing
• Encryption and mutual authentication between tasks/pods
• Patching (container images, container hosts, and the Kubernetes control plane)
• Secrets management
• What is in, and what isn’t in, each container image you run

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 Learn from AWS experts. Advance your skills and
knowledge. Build your future in the AWS Cloud.

Digital Training Classroom Training AWS Certification

Free, self-paced online Classes taught by accredited Exams to validate expertise
courses built by AWS AWS instructors with an industry-recognized
experts credential
Ready to begin building your cloud skills?
Get started at: https://www.aws.training/
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why work with an APN Partner?
APN Partners are uniquely positioned APN Partners with deep expertise in
to help your organization at any AWS services:
stage of your cloud adoption journey, AWS Managed Service Provider (MSP)
and they:
Partners
• Share your goals—focused on your APN Partners with cloud infrastructure and
success application migration expertise

• Help you take full advantage of all the AWS Competency Partners
business benefits that AWS has to offer APN Partners with verified, vetted, and validated
specialized offerings
• Provide services and solutions to
support any AWS use case across your AWS Service Delivery Partners
full customer life cycle APN Partners with a track record of delivering
specific AWS services to customers

Find the right APN Partner for your needs: https://aws.amazon.com/partners/find/

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 Thank you for attending AWS Innovate
We hope you found it interesting! A kind reminder to complete the survey.
Let us know what you thought of today’s event and how we can improve the event
experience for you in the future.

aws-apac-marketing@amazon.com
twitter.com/AWSCloud
facebook.com/AmazonWebServices
youtube.com/user/AmazonWebServices
slideshare.net/AmazonWebServices
twitch.tv/aws
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.