2019-033-Part 2 of the Kubernetes security audit discussion (Jay Beale & Aaron Small)
Published September 16, 2019
44 min
    Add to queue
    Copy URL
    Show notes



    Infosec Campout report


    Jay Beale (co-lead for audit) *Bust-a-Kube*  

    Aaron Small (product mgr at GKE/Google)


    Atreides Partners

    Trail of Bits


    What was the Audit? 

    How did it come about? 


    Who were the players?

        Kubernetes Working Group

            Aaron, Craig, Jay, Joel

        Outside vendors:

            Atredis: Josh, Nathan Keltner

            Trail of Bits: Stefan Edwards, Bobby Tonic , Dominik

        Kubernetes Project Leads/Devs

            Interviewed devs -- this was much of the info that went into the threat model

            Rapid Risk Assessments - let’s put the GitHub repository in the show notes


    What did it produce?

        Vuln Report

        Threat Model - https://github.com/kubernetes/community/blob/master/wg-security-audit/findings/Kubernetes%20Threat%20Model.pdf

        White Papers



        Discuss the results:

            Threat model findings

                Controls silently fail, leading to a false sense of security

                    Pod Security Policies, Egress Network Rules

                Audit model isn’t strong enough for non-repudiation

                    By default, API server doesn’t log user movements through system

                TLS Encryption weaknesses

                    Most components accept cleartext HTTP

                    Boot strapping to add Kubelets is particularly weak       

                    Multiple components do not check certificates and/or use self-signed certs

                    HTTPS isn’t enforced

                    Certificates are long-lived, with no revocation capability

                    Etcd doesn’t authenticate connections by default

                Controllers all Bundled together

                    Confused Deputy: b/c lower priv controllers bundled in same binary as higher

                Secrets not encrypted at rest by default

                Etcd doesn’t have signatures on its write-ahead log

                DoS attack: you can set anti-affinity on your pods to get nothing else scheduled on their nodes


                Port 10255 has an unauthenticated HTTP server for status and health checking


            Vulns / Findings (not complete list, but interesting)

                Hostpath pod security policy bypass via persistent volumes

                TOCTOU when moving PID to manager’s group

                Improperly patched directory traversal in kubectl cp

                Bearer tokens revealed in logs

                Lots of MitM risk:

                SSH not checking fingerprints: InsecureIgnoreHostKey

                gRPC transport seems all set to WithInsecure()

    HTTPS connections not checking certs 

                Some HTTPS connections are unauthenticated

                Output encoding on JSON construction

                    This might lead to further work, as JSON can get written to logs that may be consumed elsewhere.

                Non-constant time check on passwords

    Lack of re-use / library-ification of code


        Who will use these findings and how? Devs, google, bad guys? 

        Any new audit tools created from this? 


    Brad geesaman “Hacking and Hardening Kubernetes Clusters by Example [I] - Brad Geesaman, Symantec   https://www.youtube.com/watch?v=vTgQLzeBfRU


    Aaron Small: 





    CNCF:  https://www.youtube.com/watch?v=90kZRyPcRZw 





    Scope for testing:

            Source code review (what languages did they have to review?)

                Golang, shell, ...


    Networking (discuss the networking *internal* *external*

    Cryptography (TLS, data stores)


    RBAC (which roles were tested? Just admin/non-admin *best practice is no admin/least priv*)


    Namespace traversals

    Namespace claims




    Setup a bunch of environments?

        Primarily set up a single environment IIRC

        Combination of code audit and active ?fuzzing?

            What does one fuzz on a K8s environment?

    Tested with latest alpha or production versions?

        Version 1.13 or 1.14 - version locked at whatever was current - K8S releases a new version every 3 months, so this is a challenge and means we have to keep auditing.

    Tested mulitple different types of k8s implementations?

        Tested primarily against kubespray (https://github.com/kubernetes-sigs/kubespray)


    Bug Bounty program:



    Check out our Store on Teepub! https://brakesec.com/store

    Join us on our #Slack Channel! Send a request to @brakesec on Twitter or email bds.podcast@gmail.com

    #Brakesec Store!:https://www.teepublic.com/user/bdspodcast



    #Youtube Channel:  http://www.youtube.com/c/BDSPodcast

    #iTunes Store Link: https://brakesec.com/BDSiTunes

    #Google Play Store: https://brakesec.com/BDS-GooglePlay

    Our main site:  https://brakesec.com/bdswebsite

    #iHeartRadio App:  https://brakesec.com/iHeartBrakesec


    Comments, Questions, Feedback: bds.podcast@gmail.com

    Support Brakeing Down Security Podcast by using our #Paypalhttps://brakesec.com/PaypalBDS OR our #Patreon


    #Twitter@brakesec @boettcherpwned @bryanbrake @infosystir

    #Player.FM : https://brakesec.com/BDS-PlayerFM

    #Stitcher Network: https://brakesec.com/BrakeSecStitcher

    #TuneIn Radio App: https://brakesec.com/TuneInBrakesec


        0:00:00 / 0:00:00