Open Source and Container Security Are Fundamentally Broken
Containers have become the plumbing of modern engineering. They are everywhere, silently powering applications, CI/CD pipelines, and entire cloud native infrastructures. But, like the plumbing in your home, most engineers don’t think about how they work — until something goes catastrophically wrong. The ease of pulling a public image from Docker Hub or another registry has made containers the default building block for modern applications. However, this abstraction comes with a significant cost: a complete lack of visibility into the underlying security risks.
The problem is twofold: an industry-wide gap in foundational knowledge and an overwhelming flood of vulnerabilities. Most engineers today work at the application level and have never needed to understand what happens under the hood. As AI accelerates development and infrastructure abstraction, the number of engineers with deep systems expertise continues to shrink. This creates a world where, when container security issues arise that don’t have immediate and obvious fixes, understanding the underlying plumbing to get them patched has become a significant challenge for organizations running containers at scale.
This post explores why container security has become a growing source of risk and complexity in modern engineering, but doesn’t have to be. We’ll examine the obstacles organizations face when trying to track, triage, and remediate vulnerabilities, why current approaches are falling short, and what can be done to move beyond the cycle of endless patching and firefighting.
The Never-Ending Vulnerability Avalanche
Container security today is broken because it forces organizations to engage in a constant, reactive game of whack-a-mole. It starts with a single base image. That image gets deployed across thousands of instances, each supporting its own set of applications — each a delicate snowflake stack of dependencies, configurations, and quirks.
Then, a CVE hits.
That one vulnerability doesn’t just affect a single container — it’s now multiplied across every instance running that base image. Security teams scramble to track which workloads are vulnerable, which applications are at risk, and whether there’s even a viable fix. And with every new CVE that drops, the list of tracked vulnerabilities grows exponentially.
The result?
A never-ending triage nightmare, where teams are drowning in security debt, forced into an exhausting cycle of patching, breaking, and firefighting.
Take the recent CVE-2024-10963 vulnerability in libpam. A flaw in access control mechanisms allowed attackers with root privileges to bypass local security policies through hostname spoofing. Despite its high severity rating (CVSS 7.4), some major Linux distributions remained vulnerable weeks after the disclosure. The security teams relying on those distributions were stuck in limbo, waiting for upstream patches while exposed to potential attacks. This is the daily reality of container security management: constant firefighting, unpredictable patch availability, and a never-ending backlog of vulnerabilities.
Why Current Approaches Fail
Finding a security vulnerability is only the beginning of the nightmare. The real chaos starts when teams attempt to patch it. A fix is often available, but applying it isn’t as simple as swapping out a single package. Instead, it requires upgrading the entire OS or switching to a new version of a critical dependency.
With thousands of containers in production, each tied to specific configurations and application requirements, this becomes a game of Jenga, where one wrong move could bring entire services crashing down. Organizations have tried to address these problems with a variety of security platforms, from traditional vulnerability scanners to newer ASPM (Application Security Posture Management) solutions. But these tools, while helpful in tracking vulnerabilities, don’t solve the root issue: fixing them. Most scanning tools generate triage lists that quickly become overwhelming. When a base image is found to have a new vulnerability, security teams must ask themselves:
- Is there an available fix?
- If so, does it require upgrading the OS or significant dependencies?
- Will applying the fix break other applications?
- If no fix exists, what mitigations can be applied?
Even worse, many of these patches land in future OS releases, leaving teams with two equally bad options: wait for the upstream fix and remain exposed, or take on the risk of breaking production. And this cycle never stops — new vulnerabilities emerge every week, and every “fix” threatens to create just as much chaos as the vulnerability itself.
They either have to accept risk, wait for an upstream fix that may take weeks or months, or attempt to patch the issue themselves — a process that requires deep Linux expertise and careful testing. This leads to security bottlenecks where teams are forced to choose between security and stability.
Automate All the Security Things!
The reality of container security today is that most teams start vulnerable by default. Public base images are bloated with unnecessary dependencies, riddled with known CVEs, and updated on unpredictable schedules. Even when teams proactively track vulnerabilities, they are often stuck waiting for upstream fixes that may never arrive or are forced into full OS upgrades that introduce more risk than they resolve. There is an alternative: a security model that doesn’t rely on distributions to keep up.
By applying patches immediately, backporting fixes when necessary, and rebuilding images with precision, organizations can break free from the endless patch-and-pray cycle. Instead of reacting to security issues after they disrupt operations, teams can maintain hardened images from the start without compromising stability or developer velocity.
Here’s where emerging solutions, like our community-driven image catalog, come into play. Instead of relying on upstream patches or waiting for a distribution’s next release cycle, it’s possible to:
- Provide immediate patches for critical vulnerabilities, including those with no upstream fix yet.
- Backport security fixes to existing versions, so teams don’t have to upgrade entire distributions.
- Rebuild base images with minimal changes to ensure fixes are applied without introducing breaking changes.
- Automate deployment of patched images, reducing vulnerability exposure time from weeks to hours.
This approach drastically reduces the vulnerability backlog, preventing it from growing exponentially with every new CVE. By systematically patching known vulnerabilities before they ever reach production, security and DevOps teams eliminate much of the noise that overwhelms traditional vulnerability management workflows.
Instead of drowning in an unmanageable queue of issues — most of which already have fixes — developers and security teams can focus their attention on vulnerabilities that don’t have clear remediation paths, prioritizing actual risk rather than getting lost in an endless triage cycle. This results in a security posture that is both stronger and more sustainable, allowing teams to remediate threats at a reasonable, controlled pace.
Containers Need to Be Invisible — Until It Matters
The current state of open source and container security is unsustainable. With vulnerabilities emerging faster than organizations can fix them, and a growing skills gap in systems engineering fundamentals, the industry is headed toward a crisis of unmanageable security debt. The only viable path forward is to rethink how container security is handled, shifting from reactive patching to seamless, automated remediation.
Security should be like plumbing: invisible when everything is working, but capable of immediate response when something goes wrong. By integrating preemptive security into the container lifecycle — ensuring base images are always patched before vulnerabilities become crises — we can finally start breaking the cycle of endless firefighting and move our container security management from reactive to proactive.
The question is no longer if the system is broken. Still, it’s whether we’re prepared to break free from endless triage and rethink container vulnerability management from the ground up.
