K U B E R N E T E S B E N C H M A R K R E P O R T 202 3

Security, Cost and

Reliability Workload
Results
See how your Kubernetes workloads compare
to other organizations
 DATASHEET // DATADOG & FAIRWINDS INSIGHTS: A POWERFUL COMBINATION
KUBERNETES BENCHMARK REPORT // 2

INTRODUCTION
Kubernetes security, cost efficiency and reliability are top employ Kubernetes governance and guardrails to ensure that
concerns to cloud native users. Unfortunately, many DevOps whether you are developing with cloud native technologies
team leaders lack visibility into what’s happening within or on the DevOps and platform engineering team building
clusters. The result is unnecessary risk, cloud cost overruns the infrastructure, the platform is secure, highly performant
and lost customers due to poor application performance. and cost effective. Otherwise, the benefits of the investment
in container and Kubernetes technology will never be
The Kubernetes Benchmark Report 2023 evaluates results fully realized by the business, nor will the leaders of your
from over 150,000 workloads and hundreds of organizations organization understand the full value it can bring.
using the Fairwinds Insights platform. The report serves
as a tool for Kubernetes users to benchmark their clusters On the positive side, there are areas where Kubernetes
against reliability, security and cost efficiency. guardrails are helping organizations accelerate remediation.
Fairwinds evaluated organizations who are using extensive
Things are getting worse, not better. Over the last year, guardrails vs. those who are not. Organizations who
the benchmark data shows that people are not configuring implement guardrails in shift-left scenarios (e.g., catching
Kubernetes according to best practices. In the previous misconfigurations before they are released), or at time of
report we saw many results with less than 10% of workloads deployment (e.g., via Admission Controllers) are able to fix
impacted, but this year the spread has become more varied 36% more issues where CPU and Memory configurations
across all the core issues: reliability, security and cost are missing than those who do not use guardrails. Features
efficiency. like Fairwinds’ application right-sizing recommendations
help provide these recommendations in a self-service way,
It’s clear DevOps teams are outnumbered and we need to
removing the DevOps team as a barrier. Similarly, companies
do better as a community to support them. As Kubernetes
leveraging guardrails are fixing 15% more image vulnerabilities
usage expands, it’s harder for DevOps to manage
than those without guardrails.
configuration risk introduced by new teams. We need to

RELIABILITY SECURITY COST

EFFICIENCY

Use this report to understand your cluster deficiencies, where Fairwinds is the trusted partner for Kubernetes governance
to make investments and how to configure Kubernetes to have and guardrails. With Fairwinds, customers ship cloud native
a positive business impact. Data from the benchmark results is applications faster, more cost-effectively and with less risk.
anonymous and sourced from users of Fairwinds Insights. Fairwinds provides a unified view between dev, sec and ops,
removing friction between those teams with software that
simplifies complexity.
 DATASHEET // DATADOG & FAIRWINDS INSIGHTS: A POWERFUL COMBINATION
KUBERNETES BENCHMARK REPORT // 3

RELIABILITY
Missing Memory Limits & Missing Memory Requests
While Kubernetes best practices dictate that you should always set resource limits and requests on your workloads, it is not always
easy to know what values to use for each application. As a result, some teams never set requests or limits at all, while others set
them too high during initial testing and then never course correct.

Last year’s benchmark report showed that nearly half of organizations had set memory requests and limits for over 90% of their
workloads. In our latest findings, that number is down to around 20%. The spread varies after that, but we are seeing that more
workloads are impacted compared to previous years.

This could be because developers and DevOps teams do not know what limits to set or because Kubernetes consumption is
growing and there is a loss of control and visibility into how it is being configured, or both.

The key to ensuring scaling actions work properly is dialing in your memory limits and requests on each pod so workloads run
efficiently. Setting memory limits and requests is essential to operating applications on Kubernetes clusters as efficiently and
reliably as possible.

Percentage of organizations 2022 2023

Percentage of organizations 2022 2023

 DATASHEET // DATADOG & FAIRWINDS INSIGHTS: A POWERFUL COMBINATION
KUBERNETES BENCHMARK REPORT // 4

Missing Liveness and Readiness Probes

A liveness probe indicates whether or not the container is running, a fundamental indicator of how a Kubernetes workload should
function. If this liveness probe moves into a failing state, then Kubernetes automatically sends a signal to restart the container in an
attempt to restore your service to an operational state. If each container in the pod does not have a liveness probe, then a faulty or
non-functioning pod will continue to run indefinitely, using up valuable resources and potentially causing application errors.

Eighty-three percent of organizations are not setting liveness or readiness probes for more than 10% of workloads. That is an
increase of 18 percentage points year over year against the previous benchmark report. Based on the reviewed data from the past
year, we see indications that this issue is getting worse, not better.

Percentage of organizations 2022 2023

Percentage of organizations 2022 2023

 DATASHEET // DATADOG & FAIRWINDS INSIGHTS: A POWERFUL COMBINATION
KUBERNETES BENCHMARK REPORT // 5

Pull Policy Not Always

Relying on cached versions of a Docker image can become a reliability issue. By default, an image will be pulled if it isn’t already
cached on the node attempting to run it. This issue can cause variations in images that are running per node, or potentially provide
a way to gain access to an image without having direct access to the ImagePullSecret.

Again, we see an increase this year in all workloads impacted. While the distribution of percentage varies, we see that 25% of
organizations are relying on cached images for nearly all their workloads - up 10 percentage points from the previous report, which
is impacting the reliability of applications.

Percentage of organizations 2022 2023

Deployment Missing Replicas

This year we added a new check for deployments that only have a single replica. In many cases, workloads have not been
configured to use multiple replicas. 25% of organizations are running over half their workloads without replicas.

Deployments help maintain the stability and high availability of containers. Without these in place, if a node crashes, a
Deployment will still replace pods if the replica count is 1, but during that time there will be 0 replicas, which may cause
unavailability of that application.

Percentage of organizations
 DATASHEET // DATADOG & FAIRWINDS INSIGHTS: A POWERFUL COMBINATION
KUBERNETES BENCHMARK REPORT // 6

CPU Limits Missing

In the last benchmark report, we were impressed to see that 36% of organizations were missing CPU limits on fewer than 10% of
their workloads. But this year, the number of impacted workloads increased across the board. The number of organizations with
more than 10% of workloads impacted has risen from 64% to 86%. If you do not specify CPU limits, the container will not have any
upper bound. This can impact reliability as the CPU-intensive container slows down and could exhaust all CPU available on the node.

Percentage of organizations 2022 2023

CPU Requests Missing

Similarly to CPU limits, we’ve seen an increase in the percentage of organizations with missing CPU requests. Previously, we
saw that only 50% of organizations were missing requests on at least 10% of their workloads. Now 78% of organizations have
greater than 10% of workloads impacted. We also saw a significant increase (up from 0% to 17%) of organizations with 71-80%
of workloads missing CPU requests.

If a single pod is allowed to consume all of the node CPU and memory, then other pods will be starved for resources. Setting
resource requests increases reliability by guaranteeing the pod will have access to those resources—and preventing other
pods from consuming all of the available resources on a node (this is referred to as the “noisy neighbor problem”).

Percentage of organizations 2022 2023

 DATASHEET // DATADOG & FAIRWINDS INSIGHTS: A POWERFUL COMBINATION
KUBERNETES BENCHMARK REPORT // 7

SECURITY
Insecure Capabilities
Certain Linux capabilities are enabled by default for Kubernetes workloads, though most workloads don’t really need these
capabilities. The effort organizations made to pare back these capabilities has dropped. Whereas in 2021 42% of organizations
were turning off these capabilities for the vast majority of workloads, that number is only 10% now.

Instead, now we see 33% of organizations with more than 90% of workloads running with insecure capabilities. This is an
unexpected change as it shows that more organizations are now running workloads insecurely.

Percentage of organizations 2022 2023

33% of organizations have more than 90% of

workloads running with insecure capabilities.
An unexpected change as more workloads are
running insecurely.
 DATASHEET // DATADOG & FAIRWINDS INSIGHTS: A POWERFUL COMBINATION
KUBERNETES BENCHMARK REPORT // 8

Writeable File Systems

readOnlyRootFilesystem is a security setting that prevents a container from writing to its filesystem. It is a feature most organizations
want enabled in the event of a hack. If an attacker gets in, they will not be able to tamper with the application or write foreign
executables to disk. Unfortunately, Kubernetes workloads do not set this to true by default, which means teams need to explicitly
change this setting to get the most secure configuration possible.

Whereas we previously saw a binary distribution of organizations locking down filesystems inside their containers, this year we see
a huge increase of workloads impacted. Today we see 56% of organizations seemingly unaware of the need to override the insecure
defaults for the majority (71%-100%) of their workloads compared to only 23% in 2021. This is not a good trend as Kubernetes usage
and adoption grows.

Percentage of organizations 2022 2023

Privilege Escalation Allowed

Under particular configurations, a container may be able to escalate its privileges. Setting allowPrivilegeEscalation to false will set
the no_new_privs flag on the container process, preventing setuid binaries from changing the effective user ID. Setting this flag is
particularly important when using runAsNonRoot, which can otherwise be circumvented. Because this, too, is not set by default,
security-conscious teams need to explicitly set it.

What is unsettling in this year’s findings is the increase in workloads open to privilege escalation. In 2021, we saw that 42% of
organizations managed to lock down the vast majority of workloads. That has dipped to just 10%. Whereas only 18% of organizations
had 91% or greater impacted workloads in 2021, that number has risen to 29% of organizations. It is a dangerous trend.

Percentage of organizations 2022 2023

 DATASHEET // DATADOG & FAIRWINDS INSIGHTS: A POWERFUL COMBINATION
KUBERNETES BENCHMARK REPORT // 9

Runs as Privileged
The command, privileged, determines if any container in a pod can enable privileged mode. By default, a container is not allowed to
access any devices on the host, but a privileged container is given access to all devices on the host. This feature allows the container
nearly all the same access as processes running on the host, which is useful for containers that need to to use Linux capabilities,
such as manipulating the network stack and accessing devices.

Fortunately, the privileged flag is off by default, and 87% of organizations have stuck with that default, helping to increase the
security of their workloads. We’ve seen very little change in this over the last year, though like other checks, it’s trending in the
wrong direction.

Percentage of organizations 2022 2023

Run as Root Allowed

Running containers as root is not secure. Unfortunately, many workloads are allowed to do this. We are seeing an increase in
workloads running as root allowed. 44% of organizations are running 71% or more of their workloads as allowing root access.
That’s up by 22 points in the last year. It’s surprising that with the known CVEs targeting this issue we are seeing increasing
misconfigurations in this area.

Percentage of organizations 2022 2023

 DATASHEET // DATADOG & FAIRWINDS KUBERNETES
INSIGHTS: A POWERFUL COMBINATION
BENCHMARK REPORT // 10

Image Vulnerability
Workloads impacted by image vulnerabilities (such as last year’s log4j vulnerability) grew significantly this year. The 2021-2022
data showed that 40% of organizations had less than 10% of workloads impacted; in the last year that number has fallen to only
12% of organizations. There was a spike from 9% to 25% of organizations seeing greater than 90% of workloads impacted. Known
vulnerabilities can be exploited by malicious actors and need to be patched or remediated.

62% of organizations have more than 50% of workloads impacted by vulnerabilities. It is likely your organization is running images
with vulnerabilities.

Percentage of organizations 2022 2023

Outdated Helm Charts

It can be hard to keep up with the latest releases for all your cluster add-ons, so outdated Helm charts are a pervasive issue
across most organizations. This year, the percentage of organizations running workloads from outdated Helm charts has
increased. 46% of organizations have 50% or greater workloads impacted. Worse, if we narrow in, there are far too many
organizations with almost 100% of workloads impacted. The add-ons running your cluster are most likely installed by Helm. Each
add-on has its own release cadence, and some updates may come with critical security patches. Helm charts must be updated,
but they are increasingly hard to monitor and predict. That’s why so many teams are running outdated Helm charts. Nova cross-
checks Helm charts running in the cluster with the latest version available, making it easy to see when they need to be updated.

Percentage of organizations 2022 2023

 DATASHEET // DATADOG & FAIRWINDS INSIGHTS: A POWERFUL COMBINATION
KUBERNETES BENCHMARK REPORT // 11

Outdated Container Images

New this year, we now benchmark how many organizations are running outdated container images. What’s fascinating in these
results is that we see extremes: either less than 10% or greater than 90% of workloads are impacted. Nova users can run a flag
called “containers,” which examines all container images in a Kubernetes cluster and notifies users if a new version is available.
Nova provides three different alternatives for updating images: the absolute latest version, the latest minor version and the most
recent patch version.

When benchmarking yourself against these findings, the important takeaway is to update outdated helm charts or container
images. While you might be on the lower spectrum of impacted workloads, it still could introduce avoidable risk.

API Version Deprecated

In more positive news, most organizations appear to have only a few workloads with deprecated API versions. But we’re
still seeing a negative trend here: in 2021, 82% of organizations were up-to-date for the vast majority of their workloads,
decreasing to 74% this year. The vast majority had almost no deprecated APIs. Monitoring for deprecated APIs remains an
important step in de-risking Kubernetes upgrades.

Percentage of organizations 2022 2023

 DATASHEET // DATADOG & FAIRWINDS KUBERNETES
INSIGHTS: A POWERFUL COMBINATION
BENCHMARK REPORT // 12

COST EFFICIENCY
CPU Requests and Limits
Organizations appear to be setting CPU limits and requests adequately. Data shows that only 72% of organizations are setting
0-10% of workload limits too high. At the other end of the spectrum, only 94% of organizations are setting 0-10% of workload
limits too low (CPU Limits set too high chart below, too low chart not shown).

Percentage of organizations

Memory Limits too High

Similar to the previous benchmark report, almost 50% of workloads have memory limits set too high. However, this year we
are seeing an increase in the percentage of workloads impacted. In 2021, we saw that only 3% of organizations had 51-60% of
workloads impacted. That number has grown dramatically to 30% of organizations having at least 50% of workloads impacted.
That is a lot of wasted cloud resources. Rightsizing these memory limits will help your organization reduce waste and the inflated
cloud bill.

Percentage of organizations 2022 2023

 DATASHEET // DATADOG & FAIRWINDS KUBERNETES
INSIGHTS: A POWERFUL COMBINATION
BENCHMARK REPORT // 13

Memory Limits too Low

Conversely, 67% of organizations are setting memory limits too low on at least 10% of their workloads. This is a slight decrease on
the last report (70%). When limits are set too low, the reliability of clusters can be reduced. Again, rightsizing applications can help
ensure that applications don’t fail under pressure, while also ensuring that you are not wasting precious cloud resources.

To mitigate the event of an Out-of-Memory error (OOMKill), Fairwinds Insights provides users with a tool that detects and reports
OOMKills. This has helped SREs reduce pager fatigue.

Percentage of organizations 2022 2023

Memory Requests too Low

When memory requests are set too low, ensuring application reliability becomes a challenge. 59% of organizations show that this
only impacts up to 10% of their workloads, similar to 2021.

Avoid issues with too high or low memory requests by using tools that analyze usage and make suggestions to help you right-size
memory requests. Goldilocks, an open source tool, can help, or use the Fairwinds Insights platform if you are running multiple
clusters across multiple teams.

Percentage of organizations 2022 2023

 DATASHEET // DATADOG & FAIRWINDS KUBERNETES
INSIGHTS: A POWERFUL COMBINATION
BENCHMARK REPORT // 14

Memory Requests too High

In 2021, 34% of organizations had memory requests set too high on at least 10% of their workloads. That number has increased
significantly; now 82% of organizations are setting memory requests too high on workloads. 53% of organizations have between
30 and 60% of workloads impacted by this issue.

Percentage of organizations 2022 2023

82% of organizations are setting memory requests

too high on workloads.
 DATASHEET // DATADOG & FAIRWINDS KUBERNETES
INSIGHTS: A POWERFUL COMBINATION
BENCHMARK REPORT // 15

CONCLUSION
Fairwinds will continue to update the results of this benchmark data to help the cloud native community understand how they
stack up against peers. The important takeaway is this: it’s hard to configure Kubernetes to keep security, cost efficiency and
reliability consistent.

Misconfigurations present too many risks to organizations increasingly moving workloads to cloud native infrastructure.
Empower your team to deploy confidently by applying Kubernetes governance and guardrails to ensure Kubernetes
configurations are done right. Kubernetes governance solutions, such as Fairwinds Insights, can provide Dev, Sec and Ops
the ability to migrate to Kubernetes more quickly, ship applications faster, optimize cloud spend and achieve the application
performance customers expect.

Fairwinds Insights
Fairwinds Insights is Kubernetes governance software
“Insights saves the Variant platform team
that provides DevOps teams with the ability to set cost one to two weeks of work —and developers
and security guardrails. Users reduce risk, optimize cloud even more time. We’ve gone from a manual
spending and enhance application reliability all while
process over weeks to a fully automated
enabling developers to innovate and ship applications faster.
process saving significant time and
Insights provides a centralized view of multiple clusters and resources. My team can now focus on our
infrastructure as code, consistently aligns dev, sec and ops
and helps teams prioritize what matters most. Kubernetes
sprints, the R&D work we prefer.”
guardrails are implemented in shift-left scenarios (e.g.,
Vibin Daniel, Manager, Platform Engineering, Variant
catching misconfigurations before they are released), or
at time of deployment (e.g., via Admission Controllers) to
eliminate security vulnerabilities and blind spots, monitor
and optimize cost, equip developers to fix issues and
achieve compliance.

DevOps teams using Insights can stop serving as a “The cloud will hammer you to death with
Kubernetes help desk. bills. A major benefit of Fairwinds Insights is
its resource utilization and cost optimization.
Using Fairwinds Insights, we were able
to right-size the number of nodes and
databases per cluster. That understanding
Organizations of all sizes can use
helped us reduce our cost per cluster by
Fairwinds Insights. Learn more at 25%. With 25+ clusters in production and
www.fairwinds.com/insights. growing, that cost saving is significant.”
G
 len Zangirolami, Principal DevOps Architect at
Decisio Health
WHY FAIRWINDS
Fairwinds is your trusted partner for Kubernetes governance and guardrails. With Fairwinds, customers ship
cloud-native applications faster, more cost effectively, and with less risk. We provide a unified view between
dev, sec, and ops, removing friction between those teams with software that simplifies complexity. Fairwinds
Insights is built on Kubernetes expertise and integrates our leading open source tools to help you save time,
reduce risk, and deploy with confidence.

WWW.FAIRWINDS.COM