myths-that-refuse-to-die-scalability-of.html

---
date: 2015-02-16T08:38:00.000+01:00
tags:
- scalability
- overlay networks
title: 'Myths That Refuse to Die: Scalability of Overlay Virtual Networking'
url: /2015/02/myths-that-refuse-to-die-scalability-of/
---

<p>If you watched the Network Field Day videos, you might have noticed an <ahref="https://vimeo.com/119471817">interesting (somewhat one-sided) argument</a> I had with Sunay Tripathi, CTO and co-founder of Pluribus Networks (start watching at around 32:00 to get the context). Let’s try to get the record straight.<!--more--></p>
<h4>TL&amp;DR Summary</h4><p><em>Data plane performance</em> of overlay virtual networks run with well-implemented virtual switches does not vary based on number of tenants, remote hypervisors (VTEPs), or virtual machines (MAC addresses or IP host routes). Hypervisor-based overlay virtual networks might have other scalability concerns, but forwarding performance is not one of them.</p>
<h4>Basics</h4><p>The crucial part of my argument was the difference between <em>performance </em>(how fast can you do something in software versus hardware) and <em>scalability </em>(how large can something grow). That nuance somehow got lost in the translation.</p>
<p>Early host-based VXLAN implementations actually had significant performance limitations (when someone <ahref="/2014/11/open-vswitch-performance-revisited/">calls 1Gbps <em>ludicrous speed </em>in 2014</a> you have to wonder how bad it was before that), and VMware clearly documented them in their <ahref="http://www.vmware.com/files/pdf/techpaper/VMware-vSphere-VXLAN-Perf.pdf">technical white paper</a>… but that topic will have to wait for another blog post.</p>
<h4>Overlay Virtual Networking Scalability</h4><p>In November 2014 I did a <ahref="http://www.ipspace.net/Scaling_Overlay_Virtual_Networks">2-hour public webinar on scalability challenges of overlay virtual networks</a>, so you might want to <ahref="http://content.ipspace.net/get/OverlayScale">watch that one</a> (or at least the <ahref="http://content.ipspace.net/get/2 - Distributed Data Plane.mp4">Distributed Data Plane video</a>) first.</p>
<pclass="more"><strong>Disclosure</strong>: As you’ll notice in the introduction to each video, Nuage Networks sponsored the webinar, but I would never accept to work on a sponsored webinar if I didn’t believe in what I would be telling you. It’s impossible to buy integrity once you compromise it.</p>
<p>The scalability aspect of overlay virtual networking we’re discussing here is the scalability of hypervisor data plane – how many <ahref="/2014/06/is-openflow-best-tool-for-overlay/">MACs, IPs and remote VTEPs</a> can a hypervisor have and what’s the impact of large-scale environment on forwarding performance.</p>
<p>Layer-2 forwarding in hardware or software is extremely simple:</p>
<ulclass="ListParagraph"><li>Extract destination MAC address from the packet;</li>
<li>Look up destination MAC address in a <ahref="http://en.wikipedia.org/wiki/Hash_table">hash table</a>. Hash table lookups have almost linear time when the table is sparsely populated, which is easier to achieve in software than in hardware;</li>
<li>Send the packet to output port specified in the hash table entry, potentially adding tunnel encapsulation (VXLAN, PBB, VPLS…)</li>
</ul>
<p>With proper implementation, the number of MAC addresses has absolutely no impact on the forwarding performance until the MAC hash table overflows, and the number of VTEPs doesn’t matter at all (VTEP information needed by encapsulation headers is referred to in the MAC address entries).</p>
<p>Layer-3 forwarding is similar to layer-2 forwarding, but requires more complex data structures… or not. In the case of <ahref="/2013/12/hyper-v-network-virtualization-packet/">distributed L3 forwarding</a> one could <ahref="/2014/02/this-is-not-host-route-youre-looking-for/">combine ARP entries and connected subnets into host routes</a> (that’s what most ToR switches do these days) and do a simple hash-based lookup on destination IP addresses. Longest-prefix matches (for non-connected destinations) would still require a walk down an optimized tree structure.</p>
<p>It’s obvious that the number of tenants present on a hypervisor has zero impact on performance (every tenant has an independent forwarding table), the number of hosts in tenant virtual network has almost no impact on performance (see the <em>hosts routes </em>and <em>layer-2 forwarding </em>above), and the longest-prefix match can usually be done in two to four lookups (or more for IPv6). In many implementations the number of lookups doesn’t depend on the size of the forwarding table.</p>
<p><strong>Summary</strong></p>
<p>From the forwarding performance standpoint a properly implemented virtual switch remains (almost) infinitely scalable. Suboptimal implementations might have scalability challenges, and every implementation eventually runs into controller scalability issues, which some vendors like Juniper (Contrail), Nuage (VSP) and Cisco (Nexus 1000V) solved with scale-out controller architecture.</p>
<p>Scalability of hypervisor-based overlay virtual networking might have been an issue in early days of technology. Talking about its challenges in 2015 is mostly FUD (physical-to-virtual connectivity is a different story).</p>
<p>Finally, the hardware table sizes (primarily the MAC and ARP table sizes) limit the scalability of hardware-based forwarding. Software-based forwarding has significantly higher limits (how many MAC addresses can you cram into 1GB of RAM?).</p>
<h4>Want to know more?</h4><ulclass="ListParagraph"><li>Read all the linked-to blog posts. Repeat for 2-3 levels of indirection ;)</li>
<li>Watch the <ahref="http://content.ipspace.net/get/NSXArch">VMware NSX Architecture</a> and <ahref="http://content.ipspace.net/get/OverlayScale">Scaling Overlay Virtual Networks</a> webinar;</li>
<li>For even more details, watch the <ahref="http://www.ipspace.net/Overlay_Virtual_Networking">Overlay Virtual Networking</a> webinar (which includes packet walks for all major hypervisor-based overlay virtual networking solutions).</li>
</ul>
<pclass="more"><strong>More disclosure</strong>: Pluribus Networks presented @ NFD9.Presenting companies indirectly cover part of my travel expenses, but that never stopped me from expressing my own opinions.</p>