Show notes
Brenden Blanco is one of the most prolific developers working on the IO Visor Project, a Linux Foundation Collaborative Project. Brenden was an employee at PLUMgrid, the startup behind IO Visor, until it was acquired by VMware. The interview begins with a history of the layers that stacked up to form IO Visor. The history begins with the publication of The BSD Packet Filter: A New Architecture for User-Level Packet Capture at the 1993 Winter USENIX Conference, in January 1993. This paper introduced BPF, short for Berkeley Packet Filter, for selecting packets to be copied to userspace for analysis. BPF was a register-based, RISC-like virtual machine (analogous to the Java virtual machine). In Brenden's words:
Two of the major points really seemed very wise to me. It must be protocol-independent: the kernel should not have to be modified to add more protocol support... It must be general: the instruction set should be rich enough to handle unforeseen uses. They have a few more points about efficiency and generality, but those two first ones have really ended up creating kind of a platform that has stood the test of time.Safety was also critical:
Any operating system developer has an inherent distrust of the userspace, and the system calls that they're defining need to be secure against attack, as well as ignorance or bad programming. The API they expose lets a code be uploaded but the kernel has to protect itself against bad code. It doesn't allow loops, so you can't create an infinite loop in this virtual machine, and it doesn't allow access to memory that's out of bounds... That produces something that's limited, but safe. You can't do everything that a programmer would want, but you can do so efficiently.BPF came to Linux during the development of the 2.5 release series, according to LWN. A second fork of the history comes from academic research on operating system extensibility, which was a theme that grew to prominence during the 1990s with extensible research operating systems such as SPIN and the Exokernel. Many different approaches were proposed, including those based on safe languages like Modula-3 and Java. In addition, Brenden draws an analogy between BPF and the development of instruction set-based GPUs. Brenden says that the movement to software-defined networking has not been able to speed up operating system evolution. Part of the goal of IO Visor is to help speed up this evolution, by allowing the networking subsystem to evolve independent of the rest of the operating system. In Brenden's words:
Five years ago or so, when we started seeing a move toward software-defined networking, the hope was that some of this paradigm would change a little bit. But what I've seen in some of the solutions is that some of this actually isn't the case. You have a movement of network functionality from hardware to software but it's still locked into kind of the operating system life cycle, which is maybe faster than hardware but isn't in data centers where we've been trying to address this, it's still slower than the applications change, so the use cases still change faster than the infrastructure.Around 2010, Alexei Starovoitov started bringing together these two branches by extending BPF to form eBPF, which added numerous features such as extending registers to 32 to 64 bits, increasing the number of registers from 2 to 10, additional instructions, and the ability to jump forward and backward (with some restrictions) and most importantly, the ability to call a restricted set of kernel helper functions and the ability to work with data structures (“maps”). These changes made the platform a better target for compiling high-level language code and better able to interact with its environments. In addition, compilers for eBPF started being integrated into the kernel, to make eBPF execution faster on key architectures. Brenden introduces the BCC, or BPF Compiler Collection, one of the primary sub-projects within IO Visor. BCC provides a set of tools for building software in high-level languages, such as C and Python, into BPF program objects, loading those programs into running kernels, and interacting with them once they are loaded. BCC also includes a large suite of example programs. BPF programs are loaded into a kernel by attaching them to “hook points.” Typically, the programs attached to a hook point are invoked when some particular event occurs. For example, in networking, a BPF program might be invoked whenever a packet is received on a particular device, or for performance monitoring a BPF program might be attached to a “kprobe” point. Brenden explains how to use BCC to load a BPF program into the kernel using a 1-line Python program. A more sophisticated program can retain handles and use them to interact with the program at runtime. Brenden describes how restrictions on BPF are reflected in what a C programmer can include in a program. Keeping the in-kernel safety verifier simple and correct is paramount, which tends to reduce the maximum complexity of programs that can be loaded. The verifier is a limiting factor that continues to evolve to make BPF more useful over time. Currently the Open vSwitch community is considering whether to replace the use of the Open vSwitch kernel module by BPF. Brenden is in favor of the idea and offers some of his thoughts. First, his experience at PLUMgrid shows that BPF is flexible enough to support a wide variety of network applications, including those that Open vSwitch implements. Second, it's an enjoyable experience for a single developer to be able to cover the entire infrastructure for an application. In the future, Brenden is looking forward to BPF usage becoming ubiquitous for Linux performance monitoring and other non-networking use cases, such as storage acceleration and security. BPF is already, for example, used for sandboxing in Chrome. He's also looking forward to BPF for hardware offload; it can already be used for hardware offload on Netronome NICs. For more information on IO Visor, please visit iovisor.org. To talk to Brenden and other IO Visor developers, visit the #iovisor channel on the oftc.net IRC network. Brenden's nick is bblanco. You can also tweet to Brenden at @BrendenBlanco. More about BPF:
- In episode 11, John Fastabend from Intel talks about BPF on network edge nodes.
- In episode 4, Thomas Graf from Cisco talks about Cilium, which uses BPF to address the question of how to address policy in a legacy-free container environment that scales to millions of endpoints.
- Packet Pushers PQ Show 60, from 2015, interviewed Pere Monclus from PLUMgrid about the IO Visor Project and Linux networking.