On Aug. 4 and 5, I attended APSys 2016, the Asia-Pacific Workshop on Systems. I was impressed with how many of the papers presented there were relevant to Open vSwitch and virtualization in general. This episode is an interview with Jia Rao and Kun (Tony) Suo of the University of Texas at Arlington, to talk about their APSys paper, Time Capsule:

Tracing Packet Latency across Different Layers in Virtualized

Systems, which received the conference's Best Paper award.

The paper's abstract is:

Latency monitoring is important for improving user experience and

guaranteeing quality-of-service (QoS). Virtualized systems, which have

complex I/O stacks spanning multiple layers and often with unpredictable

performance, present more challenges in monitoring packet latency and

diagnosing performance abnormalities compared to traditional systems.

Existing tools either trace network latency at a coarse granularity, or

incur considerable overhead, or lack the ability to trace across

different boundaries in virtualized environments. To address this issue,

we propose Time Capsule (TC), an in-band profiler to trace packet level

latency in virtualized systems with acceptable overhead. TC timestamps

packets at predefined tracepoints and embeds the timing information into

packet payloads. TC decomposes and attributes network latency to various

layers in the virtualized network stack, which can help monitor network

latency, identify bottlenecks, and locate performance problems.

The interview covers the basic idea behind Time Capsule, the mechanism that it uses, techniques for comparing clocks of different machines across a network, and how it helps users and administrators track down latency issues in a virtual network, with reference to a specific example in the paper that shows the advantage of the fine-grained latency monitoring available in Time Capsule. “You can find some interesting results that are totally different from the results you get from coarse-grained monitoring.”

Other topics include comparison against whole-system profilers such as Perf or Xenoprof, the overhead of using Time Capsule, how many tracepoints are usually needed, how to decide where to put them, and how to insert a tracepoint.

There is a brief discussion of the relationship between Time Capsule and In-Band Network

Telemetry (INT). Time Capsule focuses on virtualization, timing, and network processing within computer systems, whereas INT tends to focus more on switching and properties of the network such as queue lengths.

Time Capsule has not yet been released but it will be made available in the future. For now, the best way to learn more is to read the paper. Readers who want to know more can contact the authors at the email addresses listed in the paper.

The authors are using Time Capsule as the basis for continuing research into the performance of virtualized systems.

Time Capsule has some limitations. For example, it is limited to measurements of latency, and it cannot record packet drops. It also, currently, requires tracepoints to be inserted manually, although eBPF might be usable in the future.

OVS Orbit is produced by Ben Pfaff. The intro music in this episode is Drive, featuring cdk and DarrylJ, copyright 2013, 2016 by Alex. The bumper music is Yeah Ant featuring Wired Ant and Javolenus, copyright 2013 by Speck. The outro music is Space

Bazooka featuring Doxen Zsigmond, copyright 2013 by Kirkoid. All content is licensed under a Creative Commons Attribution 3.0

Unported (CC BY 3.0) license.