VXLAN: Difference between revisions

No edit summary
 
(12 intermediate revisions by the same user not shown)
Line 1: Line 1:
The Virtual eXtensible Local Area Network protocol is used to encapsulate virtual Layer 2 networks over Layer 3+4, designed for use among multitenant hypervisors in (potentially multi-DC) cloud networks. It was formalized in 2014's [https://tools.ietf.org/html/rfc7348 RFC 7348]. It avoids use of 802.1D's Spanning Tree Protocol while facilitating a full broadcast domain, superseding 802.1Q VLANs and their 12-bit VLAN IDs (VXLAN uses a 24-bit ID, the VXLAN Network Identifier (VNI)). The virtual layer 2 network thus created is known as a "VXLAN segment" or "VXLAN overlay network". The agents adding or removing VXLAN encapsulation (commonly hypervisors or switches) are referred to as "VXLAN Tunnel Endpoints" or VTEPs, and play roles similar to bridges, learning MACs and selectively forwarding frames.
The Virtual eXtensible Local Area Network protocol is used to encapsulate virtual Layer 2 networks over Layer 3+4, designed for use among multitenant hypervisors in (potentially multi-DC) cloud networks. It was formalized in 2014's [https://tools.ietf.org/html/rfc7348 RFC 7348]. It avoids use of 802.1D's Spanning Tree Protocol while facilitating a full broadcast domain, superseding 802.1Q VLANs and their 12-bit VLAN IDs (VXLAN uses a 24-bit ID, the VXLAN Network Identifier aka VNI). The virtual layer 2 network thus created is known as a "VXLAN segment" or "VXLAN overlay network". The agents adding or removing VXLAN encapsulation (commonly hypervisors or switches) are referred to as "VXLAN Tunnel Endpoints" or VTEPs, and play roles similar to bridges, learning MACs and selectively forwarding frames.


The clients within a VXLAN segment needn't know that VXLAN is in use, and use standard unicast/broadcast traffic to talk to other hosts within the segment. Upon receipt of a frame, the VTEP looks up the VTEP with which this destination MAC is associated. If the client does not know the destination's L2 address, ARP is performed via normal broadcast. Broadcasts within a VXLAN are carried over a multicast address (multicast also uses this same tree).
The clients within a VXLAN segment needn't know that VXLAN is in use, and use standard unicast/broadcast traffic to talk to other hosts within the segment. Upon receipt of a frame, the VTEP looks up the VTEP with which this destination MAC is associated. If the client does not know the destination's L2 address, ARP is performed via normal broadcast. Broadcasts within a VXLAN are carried over a multicast address (multicast also uses this same tree).
Line 5: Line 5:
VTEPs are not allowed to fragment packets. It is thus important to choose a network MTU which allows the VXLAN header to be inserted without exceeding physical MTUs.
VTEPs are not allowed to fragment packets. It is thus important to choose a network MTU which allows the VXLAN header to be inserted without exceeding physical MTUs.


VXLAN runs over udp/4789 by default. VXLANs can be stacked.
VXLAN runs over udp/4789 by default (some vendors used 8472 prior to standardization). VXLANs can be stacked.


==VXLAN encapsulation==
==VXLAN encapsulation==
Line 21: Line 21:


==Use on Linux==
==Use on Linux==
Note that Openvswitch has its own, distinct VXLAN implementation. This describes the Linux kernel [[tunneling]] device.
VXLAN was added to the Linux kernel in 3.7, with the DOVE extensions added in 3.8. Note that Openvswitch has its own, distinct VXLAN implementation. This describes the Linux kernel [[tunneling]] device.
* Create the interface using <tt>ip</tt>: <tt>ip l a VXLANDEV type vxlan id VNI dstport DSTPORT</tt>
* Create the interface using <tt>ip</tt>: <tt>ip l a VXLANDEV type vxlan id VNI dstport DSTPORT local VTEPIP</tt>
** Add <tt>group MCASTIP dev MCASTDEV</tt> for a multicast-based VXLAN (see below)
** Add <tt>group MCASTIP dev MCASTDEV</tt> for a multicast-based VXLAN (see below)
*** The default multicast TTL is 1; change it by adding <tt>ttl MCASTTTL</tt> if necessary
** Add the <tt>nolearning</tt> flag to disable source address-based FDB learning
* Destroy the interface with <tt>ip l d VXLANDEV</tt>
* Destroy the interface with <tt>ip l d VXLANDEV</tt>
* Get information with <tt>ip -d l s VXLANDEV</tt>
* Get information with <tt>ip -d -s l sh VXLANDEV</tt>
** Note: <tt>ip l s</tt> will ''not'' work, as <tt>s</tt> will be interpreted as <tt>set</tt>
 
===[[systemd]]===
Minimal <tt>systemd-networkd</tt> netdev file:
<pre>
[NetDev]
Name=vx0
Kind=vxlan
 
[VXLAN]
VNI=whatever
Independent=yes
</pre>
 
Without <tt>Independent=yes</tt>, the device must be bound to some other device via a <tt>network</tt> file, or it will be more or less silently ignored by <tt>systemd-networkd</tt>.


===Forwarding tables===
===Forwarding tables===
* On Linux, the forwarding tables can be built up two ways:
* On Linux, the forwarding tables can be built up four different ways:
** Automatically using multicast: supply the multicast IP as the <tt>group</tt> argument when creating the VXLAN interface
** Automatically using multicast: supply the multicast IP as the <tt>group</tt> argument when creating the VXLAN interface
** Manually using <tt>bridge</tt>: add the entries using <tt>bridge fdb add to L2ADDR dst L3ADDR dev VXLANDEV</tt>
** Manually using <tt>bridge</tt>: add the entries using <tt>bridge fdb add to L2ADDR dst L3ADDR dev VXLANDEV</tt>
** Automatically using flooding: <tt>bridge fdb append 00:00:00:00:00:00 dev VXLANDEV dst RVTEPIP</tt>
*** Unknown l2 destinations will see [[ARP]] flooded to all 00:00:00:00:00:00 peers
** BGP-EVPN (see [https://tools.ietf.org/html/rfc7432 RFC 7432])
* Delete an entry: <tt>bridge fdb delete L2ADDR dev VXLANDEV</tt>
* Delete an entry: <tt>bridge fdb delete L2ADDR dev VXLANDEV</tt>
* Dump the forwarding table: <tt>bridge fdb show dev VXLANDEV</tt>
* Dump the forwarding table: <tt>bridge fdb show dev VXLANDEV</tt>
===Offloading===
Several NICs provide hardware offloading functionality, typically configured with <tt>[[ethtool]]</tt>. VXLAN offloading mainly means that other NIC offloading (UDP checksums, GRO, etc.) can be performed in the presence of VXLAN.
* [https://community.mellanox.com/s/article/vxlan-considerations-for-connectx-3-pro Mellanox Connect-X 3]
* Telefonica's "[https://engineering.telefonica.com/maximizing-performance-in-vxlan-overlay-networks-ec35ebe29440 Maximizing Performance in VXLAN Networks]", 2017-01-25


==External links==
==External links==
* [[Linux]] kernel [https://www.kernel.org/doc/Documentation/networking/vxlan.txt vxlan] documentation
* [[Linux]] kernel [https://www.kernel.org/doc/Documentation/networking/vxlan.txt vxlan] documentation
* Vincent Bernat's "[https://vincent.bernat.ch/en/blog/2017-vxlan-linux VXLAN & Linux]", 2017-05-03
* Vincent Bernat's "[https://vincent.bernat.ch/en/blog/2017-vxlan-linux VXLAN & Linux]", 2017-05-03