Kubernetes in Docker
Alex Mavrogiannis
Daniel Hiltgen
Docker EE Engineering
Agenda
1. Recap EE demo from Keynote
2. General CE/EE Architectures
3. EE: Topics on mixed workloads
4. EE: AuthN/AuthZ
5. Q&A
Demo Recap
General CE/EE Architecture
Docker CE to include Kubernetes (Windows and Mac)
Docker CLI Kubernetes CLI
Linuxkit VM
Stacks
CRD
(Swarm-Mode) kubeadm Kubernetes
etcd
Single Docker Engine
Host fs mounts hyperkit / hyperv vpnkit
Docker EE to include Kubernetes
Docker Enterprise Edition
Private Image Registry Image Security Scanning Content Trust and Verification
Secure Access and User
App and Cluster Management Policy Management
Management
Pods, batch jobs, blue-green deployments,
Production Ready Windows and IBM P/Z Support
horizontal pod auto-scaling
Docker Swarm Swarm-Mode Kubernetes
Orchestrator: Docker Swarm
● github.com/docker/swarm
● Cluster-wide imperative API based on the Single-node API of the Docker Engine
● High Availability and peer discovery managed through a pluggable discovery backend:
etcd, consul
● Leader caches cluster state: containers, volumes, networks etc.
● Scheduling decisions based on the reservations and limits of all cached Docker Containers.
Orchestrator: Docker Engine with Swarm-Mode Enabled
● github.com/docker/swarmkit
● Declarative State through the “Service” construct
● Built-in Routing Mesh & Overlay networking
● Scheduling decisions based on all the reservations of all swarm services across all nodes.
● Built-in in-memory Raft Store for all state (persisted to disk)
● Built in CA, per-node cryptographic node identity, mTLS between all endpoints
Orchestrator: Kubernetes
● github.com/kubernetes/kubernetes
● Scheduling: Pods
● Declarative State through “Controllers”: Deployment, ReplicaSet, DaemonSet …
● Flat Networking model delegated to plugins
● Scheduling decisions based on usage, reservations and limits of all kubernetes workloads.
○ Usage monitored through “cadvisor”, a cgroup monitoring tool
Docker EE to include Kubernetes
GUI Trusted Registry Docker CLI Kubernetes CLI
Universal Control Plane
CA OIDC Provider Agent Reconciler
Docker Swarm Kubernetes
Swarm-Mode
etcd
Docker Engine
Docker EE Architectural Highlights
● Unmodified Kubernetes components run as Docker containers
● Swarm Managers are Kubernetes Masters
● Swarmkit node inventory is source of truth
● Cryptographic Node Identity and mTLS used throughout
● UCP Agent/Reconciler manages component lifecycle
○ Manager / Worker states
○ Certificate validity
Plugin Interfaces
● General: Native API extensibility supported
○ API server and kubelet flags not modifiable
● Networking:
○ Support for CNI plugin during install
○ Ingress
● Storage: Docker Volume Plugins supported via built-in flexvolume driver, CSI in future
● Metrics: Heapster Storage Backends or Prometheus
Topics on Mixed Workloads
Resource Contention
● Allocatable Resources: The set of CPU and Memory resources available for scheduling by
an orchestrator
● Multiple orchestrators = Different definitions of allocatable resources
○ Docker Swarm: Respectful of CPU/Memory limits, but container cache may be stale
○ Docker Engine with Swarm-Mode: Only aware of its own reservations
○ Kubernetes: Effective handling of out-of-resource situations, but only for kubernetes
workloads
● When a node is at/near capacity:
○ All CPU shares throttled equally
○ The OS’s OOM killer kills processes
○ All orchestrators will reschedule on OOM, but potential workload interruption
Resource Contention (cont.)
For production workloads
● For now we recommended one orchestrator per node
● Working on UX to provide simple orchestrator selection per node
Future:
● Working to address shortcomings to better support mixed orchestration
Workload Interoperability
● Networking
○ Layer 3 not connected between kubernetes & swarm
○ Batteries-included kubernetes ingress controller
○ Layer 7 routing for swarm workloads
○ Configure external DNS
● Storage: Kubernetes workloads with docker volumes via flexvolume
AuthN / AuthZ
AuthN
AuthZ
In Summary...
● Docker will include an unmodified Kubernetes
distribution.
● Resource Contention mitigated via workload
separation
● In EE, Authentication and Authorization integrated
via standard plugin interfaces.
Thank You!
alexmavr
dhiltgen
