[pve-devel] [PATCH pve-docs 0/1] vxlan l3 routing

[Alexandre Derumier]

This add documentation for inter vxlan routing, with frr and anycast gateway.
External router documentation will come later.

http://odisoweb1.odiso.net/vxlan-l3-asymmetric.svg
http://odisoweb1.odiso.net/vxlan-l3-symmetric.svg

Alexandre Derumier (1):
add vxlan l3 routing

vxlan-and-evpn.adoc | 604 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 604 insertions(+)

[Alexandre Derumier]

This add documentation for inter vxlan routing, with frr and anycast gateway.
---
vxlan-and-evpn.adoc | 604 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 604 insertions(+)

diff --git a/vxlan-and-evpn.adoc b/vxlan-and-evpn.adoc
index 73ae4a6..703cd8b 100644
--- a/vxlan-and-evpn.adoc
+++ b/vxlan-and-evpn.adoc
@@ -849,3 +849,607 @@ router bgp 1234
line vty
!
----
+
+VXLAN layer3 routing with anycast gateway
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+With this need, each vmbr bridge will be the gateway for the vm.
+Same vmbr on different node, will have same ip address and same mac address,
+to have working vm live migration and no network disruption.
+
+VXLAN layer3 routing only work with FRR and non-aware bridge.
+(vlan aware bridge support is buggy currently).
+
+
+sysctl.conf tuning
+
+----
+#enable routing
+net.ipv4.ip_forward=1
+net.ipv6.conf.all.forwarding=1
+#disable reverse path filtering
+net.ipv4.conf.default.rp_filter=0
+net.ipv4.conf.all.rp_filter=0
+#allow frr to work with vrf
+net.ipv4.tcp_l3mdev_accept=1
+----
+
+asymmetric model
+^^^^^^^^^^^^^^^^
+
+This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
+
+The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
+but only bridging on the egress.
+This results in bi-directional VXLAN traffic traveling on different VNIs
+in each direction (always the destination VNI) across the routed infrastructure.
+
+image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
+
+
+* node1
+
+----
+auto eno1
+iface eno1 inet manual
+
+auto vmbr0
+iface vmbr0 inet static
+ address 192.168.0.1
+ netmask 255.255.255.0
+ bridge_ports eno1
+ bridge_stp off
+ bridge_fd 0
+
+auto vxlan2
+iface vxlan2 inet manual
+ vxlan-local-tunnelip 192.168.0.1
+ bridge-learning off
+ bridge-arp-nd-suppress on
+ bridge-unicast-flood off
+ bridge-multicast-flood off
+
+
+auto vmbr2
+iface vmbr2 inet static
+ address 10.0.2.254
+ netmask 255.255.255.0
+ hwaddress 44:39:39:FF:40:94
+ bridge_ports vxlan2
+ bridge_stp off
+ bridge_fd 0
+
+
+auto vxlan3
+iface vxlan3 inet manual
+ vxlan-local-tunnelip 192.168.0.1
+ bridge-learning off
+ bridge-arp-nd-suppress on
+ bridge-unicast-flood off
+ bridge-multicast-flood off
+
+
+auto vmbr3
+iface vmbr3 inet static
+ address 10.0.3.254
+ netmask 255.255.255.0
+ hwaddress 44:39:39:FF:40:94
+ bridge_ports vxlan3
+ bridge_stp off
+ bridge_fd 0
+----
+
+
+frr.conf
+
+----
+router bgp 1234
+ bgp router-id 192.168.0.1
+ no bgp default ipv4-unicast
+ coalesce-time 1000
+ neighbor 192.168.0.2 remote-as 1234
+ neighbor 192.168.0.3 remote-as 1234
+ !
+ address-family l2vpn evpn
+ neighbor 192.168.0.2 activate
+ neighbor 192.168.0.3 activate
+ advertise-all-vni
+ exit-address-family
+!
+line vty
+!
+----
+
+
+* node2
+
+----
+auto eno1
+iface eno1 inet manual
+
+auto vmbr0
+iface vmbr0 inet static
+ address 192.168.0.2
+ netmask 255.255.255.0
+ bridge_ports eno1
+ bridge_stp off
+ bridge_fd 0
+
+auto vxlan2
+iface vxlan2 inet manual
+ vxlan-local-tunnelip 192.168.0.2
+ bridge-learning off
+ bridge-arp-nd-suppress on
+ bridge-unicast-flood off
+ bridge-multicast-flood off
+
+
+auto vmbr2
+iface vmbr2 inet static
+ address 10.0.2.254
+ netmask 255.255.255.0
+ hwaddress 44:39:39:FF:40:94
+ bridge_ports vxlan2
+ bridge_stp off
+ bridge_fd 0
+
+
+auto vxlan3
+iface vxlan3 inet manual
+ vxlan-local-tunnelip 192.168.0.2
+ bridge-learning off
+ bridge-arp-nd-suppress on
+ bridge-unicast-flood off
+ bridge-multicast-flood off
+
+
+auto vmbr3
+iface vmbr3 inet static
+ address 10.0.3.254
+ netmask 255.255.255.0
+ hwaddress 44:39:39:FF:40:94
+ bridge_ports vxlan3
+ bridge_stp off
+ bridge_fd 0
+----
+
+
+frr.conf
+
+----
+router bgp 1234
+ bgp router-id 192.168.0.2
+ no bgp default ipv4-unicast
+ coalesce-time 1000
+ neighbor 192.168.0.1 remote-as 1234
+ neighbor 192.168.0.3 remote-as 1234
+ !
+ address-family l2vpn evpn
+ neighbor 192.168.0.1 activate
+ neighbor 192.168.0.3 activate
+ advertise-all-vni
+ exit-address-family
+!
+line vty
+!
+----
+
+
+* node3
+
+----
+auto eno1
+iface eno1 inet manual
+
+auto vmbr0
+iface vmbr0 inet static
+ address 192.168.0.3
+ netmask 255.255.255.0
+ bridge_ports eno1
+ bridge_stp off
+ bridge_fd 0
+
+auto vxlan2
+iface vxlan2 inet manual
+ vxlan-local-tunnelip 192.168.0.3
+ bridge-learning off
+ bridge-arp-nd-suppress on
+ bridge-unicast-flood off
+ bridge-multicast-flood off
+
+
+auto vmbr2
+iface vmbr2 inet static
+ address 10.0.2.254
+ netmask 255.255.255.0
+ hwaddress 44:39:39:FF:40:94
+ bridge_ports vxlan2
+ bridge_stp off
+ bridge_fd 0
+
+
+auto vxlan3
+iface vxlan3 inet manual
+ vxlan-local-tunnelip 192.168.0.3
+ bridge-learning off
+ bridge-arp-nd-suppress on
+ bridge-unicast-flood off
+ bridge-multicast-flood off
+
+
+auto vmbr3
+iface vmbr3 inet static
+ address 10.0.3.254
+ netmask 255.255.255.0
+ hwaddress 44:39:39:FF:40:94
+ bridge_ports vxlan3
+ bridge_stp off
+ bridge_fd 0
+----
+
+
+frr.conf
+
+----
+router bgp 1234
+ bgp router-id 192.168.0.3
+ no bgp default ipv4-unicast
+ coalesce-time 1000
+ neighbor 192.168.0.1 remote-as 1234
+ neighbor 192.168.0.2 remote-as 1234
+ !
+ address-family l2vpn evpn
+ neighbor 192.168.0.1 activate
+ neighbor 192.168.0.2 activate
+ advertise-all-vni
+ exit-address-family
+!
+line vty
+!
+----
+
+
+symmetric model
+^^^^^^^^^^^^^^^
+
+With this model, you don't need to have all vxlan on all nodes.
+This model will also be needed to route traffic to an external router.
+
+The symmetric model routes and bridges on both the ingress and the egress leafs.
+This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
+However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
+All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
+routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
+
+A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
+
+image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
+
+
+* node1
+
+----
+auto vrf1
+iface vrf1
+ vrf-table auto
+
+auto eno1
+iface eno1 inet manual
+
+auto vmbr0
+iface vmbr0 inet static
+ address 192.168.0.1
+ netmask 255.255.255.0
+ bridge_ports eno1
+ bridge_stp off
+ bridge_fd 0
+
+auto vxlan2
+iface vxlan2 inet manual
+ vxlan-local-tunnelip 192.168.0.1
+ bridge-learning off
+ bridge-arp-nd-suppress on
+ bridge-unicast-flood off
+ bridge-multicast-flood off
+
+auto vmbr2
+iface vmbr2 inet static
+ bridge_ports vxlan2
+ bridge_stp off
+ bridge_fd 0
+ address 10.0.2.254
+ netmask 255.255.255.0
+ hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
+ vrf vrf1
+
+auto vxlan3
+iface vxlan3 inet manual
+ vxlan-local-tunnelip 192.168.0.1
+ bridge-learning off
+ bridge-arp-nd-suppress on
+ bridge-unicast-flood off
+ bridge-multicast-flood off
+
+auto vmbr3
+iface vmbr3 inet static
+ bridge_ports vxlan3
+ bridge_stp off
+ bridge_fd 0
+ address 10.0.3.254
+ netmask 255.255.255.0
+ hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
+ vrf vrf1
+
+#interconnect vxlan-vfr l3vni
+auto vxlan4000
+iface vxlan4000 inet manual
+ vxlan-local-tunnelip 192.168.0.1
+ bridge-learning off
+ bridge-arp-nd-suppress on
+ bridge-unicast-flood off
+ bridge-multicast-flood off
+
+
+auto vmbr4000
+iface vmbr4000 inet manual
+ bridge_ports vxlan4000
+ bridge_stp off
+ bridge_fd 0
+ hwaddress 44:39:39:FF:40:90 #must be different on each node
+ vrf vrf1
+----
+
+frr.conf
+
+----
+vrf vrf1
+ vni 4000
+!
+router bgp 1234
+ bgp router-id 192.168.0.1
+ no bgp default ipv4-unicast
+ coalesce-time 1000
+ neighbor 192.168.0.2 remote-as 1234
+ neighbor 192.168.0.3 remote-as 1234
+ !
+ address-family l2vpn evpn
+ neighbor 192.168.0.2 activate
+ neighbor 192.168.0.3 activate
+ advertise-all-vni
+ exit-address-family
+!
+router bgp 1234 vrf vrf1
+!
+ bgp router-id 192.168.0.1
+ !
+ address-family ipv4 unicast
+ redistribute connected
+ exit-address-family
+ !
+ address-family l2vpn evpn
+ advertise ipv4 unicast
+ exit-address-family
+!
+line vty
+!
+----
+
+
+* node2
+
+----
+auto vrf1
+iface vrf1
+ vrf-table auto
+
+auto eno1
+iface eno1 inet manual
+
+auto vmbr0
+iface vmbr0 inet static
+ address 192.168.0.2
+ netmask 255.255.255.0
+ bridge_ports eno1
+ bridge_stp off
+ bridge_fd 0
+
+auto vxlan2
+iface vxlan2 inet manual
+ vxlan-local-tunnelip 192.168.0.2
+ bridge-learning off
+ bridge-arp-nd-suppress on
+ bridge-unicast-flood off
+ bridge-multicast-flood off
+
+auto vmbr2
+iface vmbr2 inet static
+ bridge_ports vxlan2
+ bridge_stp off
+ bridge_fd 0
+ address 10.0.2.254
+ netmask 255.255.255.0
+ hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
+ vrf vrf1
+
+auto vxlan3
+iface vxlan3 inet manual
+ vxlan-local-tunnelip 192.168.0.2
+ bridge-learning off
+ bridge-arp-nd-suppress on
+ bridge-unicast-flood off
+ bridge-multicast-flood off
+
+auto vmbr3
+iface vmbr3 inet static
+ bridge_ports vxlan3
+ bridge_stp off
+ bridge_fd 0
+ address 10.0.3.254
+ netmask 255.255.255.0
+ hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
+ vrf vrf1
+
+#interconnect vxlan-vfr l3vni
+auto vxlan4000
+iface vxlan4000 inet manual
+ vxlan-local-tunnelip 192.168.0.2
+ bridge-learning off
+ bridge-arp-nd-suppress on
+ bridge-unicast-flood off
+ bridge-multicast-flood off
+
+
+auto vmbr4000
+iface vmbr4000 inet manual
+ bridge_ports vxlan4000
+ bridge_stp off
+ bridge_fd 0
+ hwaddress 44:39:39:FF:40:91 #must be different on each node
+ vrf vrf1
+----
+
+
+frr.conf
+
+----
+vrf vrf1
+ vni 4000
+!
+router bgp 1234
+ bgp router-id 192.168.0.2
+ no bgp default ipv4-unicast
+ coalesce-time 1000
+ neighbor 192.168.0.1 remote-as 1234
+ neighbor 192.168.0.3 remote-as 1234
+ !
+ address-family l2vpn evpn
+ neighbor 192.168.0.1 activate
+ neighbor 192.168.0.3 activate
+ advertise-all-vni
+ exit-address-family
+!
+router bgp 1234 vrf vrf1
+!
+ bgp router-id 192.168.0.2
+ !
+ address-family ipv4 unicast
+ redistribute connected
+ exit-address-family
+ !
+ address-family l2vpn evpn
+ advertise ipv4 unicast
+ exit-address-family
+!
+line vty
+!
+----
+
+
+* node3
+
+----
+auto vrf1
+iface vrf1
+ vrf-table auto
+
+auto eno1
+iface eno1 inet manual
+
+auto vmbr0
+iface vmbr0 inet static
+ address 192.168.0.3
+ netmask 255.255.255.0
+ bridge_ports eno1
+ bridge_stp off
+ bridge_fd 0
+
+auto vxlan2
+iface vxlan2 inet manual
+ vxlan-local-tunnelip 192.168.0.3
+ bridge-learning off
+ bridge-arp-nd-suppress on
+ bridge-unicast-flood off
+ bridge-multicast-flood off
+
+auto vmbr2
+iface vmbr2 inet static
+ bridge_ports vxlan2
+ bridge_stp off
+ bridge_fd 0
+ address 10.0.2.254
+ netmask 255.255.255.0
+ hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
+ vrf vrf1
+
+auto vxlan3
+iface vxlan3 inet manual
+ vxlan-local-tunnelip 192.168.0.3
+ bridge-learning off
+ bridge-arp-nd-suppress on
+ bridge-unicast-flood off
+ bridge-multicast-flood off
+
+auto vmbr3
+iface vmbr3 inet static
+ bridge_ports vxlan3
+ bridge_stp off
+ bridge_fd 0
+ address 10.0.3.254
+ netmask 255.255.255.0
+ hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
+ vrf vrf1
+
+#interconnect vxlan-vfr l3vni
+auto vxlan4000
+iface vxlan4000 inet manual
+ vxlan-local-tunnelip 192.168.0.3
+ bridge-learning off
+ bridge-arp-nd-suppress on
+ bridge-unicast-flood off
+ bridge-multicast-flood off
+
+
+auto vmbr4000
+iface vmbr4000 inet manual
+ bridge_ports vxlan4000
+ bridge_stp off
+ bridge_fd 0
+ hwaddress 44:39:39:FF:40:92 #must be different on each node
+ vrf vrf1
+----
+
+
+frr.conf
+
+----
+vrf vrf1
+ vni 4000
+!
+router bgp 1234
+ bgp router-id 192.168.0.3
+ no bgp default ipv4-unicast
+ coalesce-time 1000
+ neighbor 192.168.0.1 remote-as 1234
+ neighbor 192.168.0.2 remote-as 1234
+ !
+ address-family l2vpn evpn
+ neighbor 192.168.0.1 activate
+ neighbor 192.168.0.2 activate
+ advertise-all-vni
+ exit-address-family
+!
+router bgp 1234 vrf vrf1
+!
+ bgp router-id 192.168.0.3
+ !
+ address-family ipv4 unicast
+ redistribute connected
+ exit-address-family
+ !
+ address-family l2vpn evpn
+ advertise ipv4 unicast
+ exit-address-family
+!
+line vty
+!
+----

[Dietmar Maurer]

Post by Alexandre Derumier
+sysctl.conf tuning
+
+----
+#enable routing
+net.ipv4.ip_forward=1
+net.ipv6.conf.all.forwarding=1
+#disable reverse path filtering
+net.ipv4.conf.default.rp_filter=0
+net.ipv4.conf.all.rp_filter=0

rp_filter is essential for security. Why do we
need to turn this off?

Post by Alexandre Derumier
+#allow frr to work with vrf
+net.ipv4.tcp_l3mdev_accept=1
+----

[Alexandre DERUMIER]

Post by Alexandre Derumier

Post by Dietmar Maurer
rp_filter is essential for security. Why do we
need to turn this off?

For example, I had problem with live migration, and symmetric model , timeout of 30-60s.
https://github.com/FRRouting/frr/issues/2129



----- Mail original -----
De: "dietmar" <***@proxmox.com>
À: "pve-devel" <pve-***@pve.proxmox.com>, "aderumier" <***@odiso.com>
Envoyé: Samedi 11 Août 2018 10:01:37
Objet: Re: [pve-devel] [PATCH pve-docs 1/1] add vxlan l3 routing

Post by Alexandre Derumier
+sysctl.conf tuning
+
+----
+#enable routing
+net.ipv4.ip_forward=1
+net.ipv6.conf.all.forwarding=1
+#disable reverse path filtering
+net.ipv4.conf.default.rp_filter=0
+net.ipv4.conf.all.rp_filter=0

rp_filter is essential for security. Why do we
need to turn this off?

Post by Alexandre Derumier
+#allow frr to work with vrf
+net.ipv4.tcp_l3mdev_accept=1
+----

[Dietmar Maurer]

Post by Alexandre DERUMIER

Post by Dietmar Maurer
rp_filter is essential for security. Why do we
need to turn this off?

For example, I had problem with live migration, and symmetric model , timeout of 30-60s.
https://github.com/FRRouting/frr/issues/2129

But I think we cannot simply turn off rp_filter, see

https://vincent.bernat.im/en/blog/2017-linux-bridge-isolation

Maybe we can use vrf (instead of rp_filter) to isolate our bridges??

[Alexandre DERUMIER]

Post by Alexandre DERUMIER

Post by Dietmar Maurer
But I think we cannot simply turn off rp_filter, see
https://vincent.bernat.im/en/blog/2017-linux-bridge-isolation
Maybe we can use vrf (instead of rp_filter) to isolate our bridges??

with symmetric routing, all bridges are in a vrf.
so you can't access to host ip if it's defined on an interface not in this vrf.


But, because of a bug in >= kernel 4.14 (https://github.com/FRRouting/frr/issues/2460),
we need net.ipv4.tcp_l3mdev_accept=1, which allow to access from a vrf, to a service listeting in the default vrf.


(Note that rp_filter=0 on all interfaces is maybe a little bit too large, I think it could be done only on
specific interfaces, but I need to do tests again to very which interfaces really need it)



----- Mail original -----
De: "dietmar" <***@proxmox.com>
À: "aderumier" <***@odiso.com>
Cc: "pve-devel" <pve-***@pve.proxmox.com>
Envoyé: Dimanche 12 Août 2018 07:46:38
Objet: Re: [pve-devel] [PATCH pve-docs 1/1] add vxlan l3 routing

Post by Alexandre DERUMIER

Post by Dietmar Maurer

Post by Dietmar Maurer
rp_filter is essential for security. Why do we
need to turn this off?

For example, I had problem with live migration, and symmetric model , timeout of 30-60s.
https://github.com/FRRouting/frr/issues/2129

But I think we cannot simply turn off rp_filter, see

https://vincent.bernat.im/en/blog/2017-linux-bridge-isolation

Maybe we can use vrf (instead of rp_filter) to isolate our bridges??

[Dietmar Maurer]

Post by Alexandre DERUMIER

Post by Dietmar Maurer
But I think we cannot simply turn off rp_filter, see
https://vincent.bernat.im/en/blog/2017-linux-bridge-isolation
Maybe we can use vrf (instead of rp_filter) to isolate our bridges??

with symmetric routing, all bridges are in a vrf.

What about other bridges in the system which does not use vxlan at all (firewall
bridges)?

[Alexandre DERUMIER]

Post by Alexandre DERUMIER

Post by Dietmar Maurer
What about other bridges in the system which does not use vxlan at all (firewall
bridges)?

mmm, good question. I think you can put it in the vrf or not.
as they don't have any ip address, and it's only layer2, it's not a problem.


Vrf is mandatory on bridge with symetric routing, because they are the router for the bridge,
and they need to have their routing table from the vrf.



----- Mail original -----
De: "dietmar" <***@proxmox.com>
À: "aderumier" <***@odiso.com>
Cc: "pve-devel" <pve-***@pve.proxmox.com>
Envoyé: Dimanche 12 Août 2018 13:53:27
Objet: Re: [pve-devel] [PATCH pve-docs 1/1] add vxlan l3 routing

Post by Alexandre DERUMIER

Post by Dietmar Maurer

Post by Dietmar Maurer
But I think we cannot simply turn off rp_filter, see
https://vincent.bernat.im/en/blog/2017-linux-bridge-isolation
Maybe we can use vrf (instead of rp_filter) to isolate our bridges??

with symmetric routing, all bridges are in a vrf.

What about other bridges in the system which does not use vxlan at all (firewall
bridges)?

[Dietmar Maurer]

Post by Alexandre DERUMIER

Post by Dietmar Maurer
What about other bridges in the system which does not use vxlan at all (firewall
bridges)?

mmm, good question. I think you can put it in the vrf or not.
as they don't have any ip address, and it's only layer2, it's not a problem.

See: https://vincent.bernat.im/en/blog/2017-linux-bridge-isolation

Such bridge accepts packets anyways - not sure how severe that problem is...

[Dietmar Maurer]

I wonder if it is necessary to use three nodes for your examples.
Wouldn't it be enough to use two nodes? That would make configuration
files smaller.

[Alexandre DERUMIER]

Post by Dietmar Maurer
I wonder if it is necessary to use three nodes for your examples.
Wouldn't it be enough to use two nodes? That would make configuration
files smaller.

This was mainly for l2 unicast mode and also frr, to show multiple peers configuration.



----- Mail original -----
De: "dietmar" <***@proxmox.com>
À: "pve-devel" <pve-***@pve.proxmox.com>, "aderumier" <***@odiso.com>
Envoyé: Samedi 11 Août 2018 10:08:28
Objet: Re: [pve-devel] [PATCH pve-docs 1/1] add vxlan l3 routing

I wonder if it is necessary to use three nodes for your examples.
Wouldn't it be enough to use two nodes? That would make configuration
files smaller.

[Dietmar Maurer]

Post by Alexandre DERUMIER

Post by Dietmar Maurer
I wonder if it is necessary to use three nodes for your examples.
Wouldn't it be enough to use two nodes? That would make configuration
files smaller.

This was mainly for l2 unicast mode and also frr, to show multiple peers configuration.

OK

[Dietmar Maurer]

BTW, the existing vxlan-and-evpn.adoc explain howto setup vxlan on vlan.
I can see that this works, but why would someone do that?

Post by Alexandre Derumier
This add documentation for inter vxlan routing, with frr and anycast gateway.
External router documentation will come later.
http://odisoweb1.odiso.net/vxlan-l3-asymmetric.svg
http://odisoweb1.odiso.net/vxlan-l3-symmetric.svg
add vxlan l3 routing
vxlan-and-evpn.adoc | 604
++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 604 insertions(+)
--
2.11.0
_______________________________________________
pve-devel mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[Alexandre DERUMIER]

Post by Alexandre Derumier

Post by Dietmar Maurer
BTW, the existing vxlan-and-evpn.adoc explain howto setup vxlan on vlan.
I can see that this works, but why would someone do that?

Maybe because it's reduce the configuration, don't need to create 1bridge for 1vxlan.
I have added the doc if they want it, but we could remove it if you want, to have less differents setup and remove complexity for users.


I known that cumulus is working on lwt vxlan (lightway tunnel),
where you can have 1 vxlan interface - 1 vlan aware bridge, and vlan are mapped to vxlan when going out.
But that don't work with frr currently, and with l3 routing, anyway, you need to define interface with gateway ip on each vlan.


----- Mail original -----
De: "dietmar" <***@proxmox.com>
À: "pve-devel" <pve-***@pve.proxmox.com>, "Alexandre Derumier" <***@odiso.com>
Envoyé: Vendredi 10 Août 2018 13:18:11
Objet: Re: [pve-devel] [PATCH pve-docs 0/1] vxlan l3 routing

BTW, the existing vxlan-and-evpn.adoc explain howto setup vxlan on vlan.
I can see that this works, but why would someone do that?

Post by Alexandre Derumier
This add documentation for inter vxlan routing, with frr and anycast gateway.
External router documentation will come later.
http://odisoweb1.odiso.net/vxlan-l3-asymmetric.svg
http://odisoweb1.odiso.net/vxlan-l3-symmetric.svg
add vxlan l3 routing
vxlan-and-evpn.adoc | 604
++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 604 insertions(+)
--
2.11.0
_______________________________________________
pve-devel mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[Dietmar Maurer]

Post by Alexandre DERUMIER

Post by Dietmar Maurer
BTW, the existing vxlan-and-evpn.adoc explain howto setup vxlan on vlan.
I can see that this works, but why would someone do that?

Maybe because it's reduce the configuration, don't need to create 1bridge for 1vxlan.

OH, I see. I guess the want/need to avoid multiple bridges on those hardware
switches.

Post by Alexandre DERUMIER
I have added the doc if they want it, but we could remove it if you want, to
have less differents setup and remove complexity for users.

I think it is an interesting lesson, but the setup without vlan is much easier
inside our environment. I guess it is also easier to debug.

Maybe we move those VLAN setup into wiki-artikels, but remove them from
the main docu?

Post by Alexandre DERUMIER
I known that cumulus is working on lwt vxlan (lightway tunnel),
where you can have 1 vxlan interface - 1 vlan aware bridge, and vlan are
mapped to vxlan when going out.
But that don't work with frr currently, and with l3 routing, anyway, you need
to define interface with gateway ip on each vlan.

My impression is that lwt was there old setup, and they now
move on with frr?

[Alexandre DERUMIER]

Post by Alexandre DERUMIER

Post by Dietmar Maurer
OH, I see. I guess the want/need to avoid multiple bridges on those hardware
switches.

yes, indeed, on a hardware switch, you have only 1 bridge (the switch)

Post by Alexandre DERUMIER

Post by Dietmar Maurer
think it is an interesting lesson, but the setup without vlan is much easier
inside our environment. I guess it is also easier to debug.
Maybe we move those VLAN setup into wiki-artikels, but remove them from
the main docu?

Sure, no problem. It's indeed more simplier.
we have network reload with ifupdown2, so we can add vxlan/vmbr without reboot easily in config
if we have a lot of bridge/vxlan, we can also use template

%for v in [11,12]:
auto vxlan${v}
iface vxlan${v}
....
%endfor

Post by Alexandre DERUMIER

Post by Dietmar Maurer
My impression is that lwt was there old setup, and they now
move on with frr?

No, it's different. lwt is not yet supported by ifdown2 (it need a recent kernel 4.13 or 4.15, don't remember)
roopa from cumulus talk about this at netdev 2.2

https://www.netdevconf.org/2.2/slides/prabhu-linuxbridge-tutorial.pdf

They are trying to reduce the number of interfaces, because on hardware switch, it could take really a long time
to load. (4000 bridge + 4000 vxlan for example).
That's also why ifupdown2 use netlink directly, to speedup configuration.

That's works fine at layer2, but at layer3, if you need an ip for each vxlan for gateway, you'll need an extra exterface.



----- Mail original -----
De: "dietmar" <***@proxmox.com>
À: "Alexandre Derumier" <***@odiso.com>
Cc: "pve-devel" <pve-***@pve.proxmox.com>
Envoyé: Samedi 11 Août 2018 09:20:16
Objet: Re: [pve-devel] [PATCH pve-docs 0/1] vxlan l3 routing

Post by Alexandre DERUMIER

Post by Dietmar Maurer

Post by Dietmar Maurer
BTW, the existing vxlan-and-evpn.adoc explain howto setup vxlan on vlan.
I can see that this works, but why would someone do that?

Maybe because it's reduce the configuration, don't need to create 1bridge for 1vxlan.

OH, I see. I guess the want/need to avoid multiple bridges on those hardware
switches.

Post by Alexandre DERUMIER
I have added the doc if they want it, but we could remove it if you want, to
have less differents setup and remove complexity for users.

I think it is an interesting lesson, but the setup without vlan is much easier
inside our environment. I guess it is also easier to debug.

Maybe we move those VLAN setup into wiki-artikels, but remove them from
the main docu?

Post by Alexandre DERUMIER
I known that cumulus is working on lwt vxlan (lightway tunnel),
where you can have 1 vxlan interface - 1 vlan aware bridge, and vlan are
mapped to vxlan when going out.
But that don't work with frr currently, and with l3 routing, anyway, you need
to define interface with gateway ip on each vlan.

My impression is that lwt was there old setup, and they now
move on with frr?

[Dietmar Maurer]

Post by Alexandre DERUMIER

Post by Dietmar Maurer
Maybe we move those VLAN setup into wiki-artikels, but remove them from
the main docu?

Sure, no problem. It's indeed more simplier.
we have network reload with ifupdown2, so we can add vxlan/vmbr without reboot
easily in config
if we have a lot of bridge/vxlan, we can also use template
auto vxlan${v}
iface vxlan${v}
....
%endfor

Those templates are interesting, but I am not sure if we can handle them
(parser, GUI) ...

[Alexandre DERUMIER]

Post by Alexandre DERUMIER

Post by Dietmar Maurer
Those templates are interesting, but I am not sure if we can handle them
(parser, GUI) ...

yes, I'm not sure too.


I was also thinking about vxlan/gateway configuration, as we need to define them on all nodes.
it could be great to be able to define them globally. (to avoid errors and faster setup)

maybe a symlink from /etc/network/interfaces.d/vxlan /etc/pve/... ? (not sure if it'll work at boot, as we don't have network yet)

or define them somewhere in /etc/pve/network.cfg for example and have a local daemon which write to /etc/network/interfaces and reload config.

What do you think about this ?


----- Mail original -----
De: "dietmar" <***@proxmox.com>
À: "aderumier" <***@odiso.com>
Cc: "pve-devel" <pve-***@pve.proxmox.com>
Envoyé: Lundi 13 Août 2018 08:38:43
Objet: Re: [pve-devel] [PATCH pve-docs 0/1] vxlan l3 routing

Post by Alexandre DERUMIER

Post by Dietmar Maurer

Post by Dietmar Maurer
Maybe we move those VLAN setup into wiki-artikels, but remove them from
the main docu?

Sure, no problem. It's indeed more simplier.
we have network reload with ifupdown2, so we can add vxlan/vmbr without reboot
easily in config
if we have a lot of bridge/vxlan, we can also use template
auto vxlan${v}
iface vxlan${v}
....
%endfor

Those templates are interesting, but I am not sure if we can handle them
(parser, GUI) ...

[Dietmar Maurer]

Post by Alexandre DERUMIER

Post by Dietmar Maurer
Those templates are interesting, but I am not sure if we can handle them
(parser, GUI) ...

yes, I'm not sure too.

I guess it would be very difficult ...

Post by Alexandre DERUMIER
I was also thinking about vxlan/gateway configuration, as we need to define
them on all nodes.
it could be great to be able to define them globally. (to avoid errors and faster setup)
maybe a symlink from /etc/network/interfaces.d/vxlan /etc/pve/... ? (not sure
if it'll work at boot, as we don't have network yet)
or define them somewhere in /etc/pve/network.cfg for example and have a local
daemon which write to /etc/network/interfaces and reload config.
What do you think about this ?

Not sure if the vxlan config is most times node related, so you always have node
related config anyways?

How many vxlan will you deploy in you setup?

[Alexandre DERUMIER]

Post by Alexandre DERUMIER

Post by Dietmar Maurer
Not sure if the vxlan config is most times node related, so you always have node
related config anyways?

with frr, The vxlan interface indeed have the local ip vxlan-local-tunnelip. (this could be found in frr.conf).
This could be managed with ifupdown2 template (as vxlan-local-tunnelip is always the same).

I was thinking of vmbr definition, as it's exactly the same on all nodes. (and can't be managed easily with ifupdown2 template).
We can manage this with puppet without problem, but I was thinking for users which only manage this through proxmox.

I don't have looked yet at vmware nsx to see how they manage this. (as it's almost the same setup).

Post by Alexandre DERUMIER

Post by Dietmar Maurer
How many vxlan will you deploy in you setup?

1 vxlan by customer, so if I migrate all my current vlans, around 300 currently ;)


----- Mail original -----
De: "dietmar" <***@proxmox.com>
À: "aderumier" <***@odiso.com>
Cc: "pve-devel" <pve-***@pve.proxmox.com>
Envoyé: Lundi 13 Août 2018 11:22:16
Objet: Re: [pve-devel] [PATCH pve-docs 0/1] vxlan l3 routing

Post by Alexandre DERUMIER

Post by Dietmar Maurer

Post by Dietmar Maurer
Those templates are interesting, but I am not sure if we can handle them
(parser, GUI) ...

yes, I'm not sure too.

I guess it would be very difficult ...

Post by Alexandre DERUMIER
I was also thinking about vxlan/gateway configuration, as we need to define
them on all nodes.
it could be great to be able to define them globally. (to avoid errors and faster setup)
maybe a symlink from /etc/network/interfaces.d/vxlan /etc/pve/... ? (not sure
if it'll work at boot, as we don't have network yet)
or define them somewhere in /etc/pve/network.cfg for example and have a local
daemon which write to /etc/network/interfaces and reload config.
What do you think about this ?

Not sure if the vxlan config is most times node related, so you always have node
related config anyways?

How many vxlan will you deploy in you setup?

[Dietmar Maurer]

Post by Alexandre DERUMIER

Post by Dietmar Maurer
Not sure if the vxlan config is most times node related, so you always have node
related config anyways?

with frr, The vxlan interface indeed have the local ip vxlan-local-tunnelip.
(this could be found in frr.conf).
This could be managed with ifupdown2 template (as vxlan-local-tunnelip is always the same).
I was thinking of vmbr definition, as it's exactly the same on all nodes. (and
can't be managed easily with ifupdown2 template).
We can manage this with puppet without problem, but I was thinking for users
which only manage this through proxmox.

Just try - maybe you can find a way to define a cluster wide config
(etc/pve/network.cfg) for those bridges.

vnet1000: vxlan # VNI 1000
device vmbr0 # => vxlan-local-tunnelip
group 239.0.0.1

above would create the vtep and the bridge called 'vnet1000'?

...

[Alexandre DERUMIER]

Post by Alexandre DERUMIER

Post by Dietmar Maurer
Just try - maybe you can find a way to define a cluster wide config
(etc/pve/network.cfg) for those bridges.
vnet1000: vxlan # VNI 1000
device vmbr0 # => vxlan-local-tunnelip
group 239.0.0.1
above would create the vtep and the bridge called 'vnet1000'?

Hi Dietmar,

I think I have found a way.

ifupdown2 support "alias" options

"
auto eno1
iface eno1
alias hypervisor_port_1
"


ip link set dev ${interface name} alias "${description}"

Examples:
ip link set dev eno1 alias "hypervisor_port_1"
Link aliases show up in "ip link show" output, like:
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
link/ether 22:ce:e0:99:63:6f brd ff:ff:ff:ff:ff:ff
alias hypervisor_port_1



Like this, we could define a common alias on differents hosts, with differents interfaces,
and no need to use a vmbr0. (I'm thinking of layer3 underlay, with multiple interfaces with ecmp)



Some examples:
---------------

multicast setup
---------------
vnet1000: vxlan # VNI 1000
device hypervisor_port_1
type multicast
group 239.0.0.1



this will add a vxlan1000 interface with "vxlan-svcnodeip ..." option (with multicast address)
and vxlan-physdev (interface enX with hypervisor_port_1 alias)



node1
-----
auto eno1
iface eno1 inet static
address 192.168.0.1/24
alias hypervisor_port_1

auto vxlan1000
iface vxlan1000 inet manual
vxlan-svcnodeip 239.0.0.1
vxlan-physdev eno1

auto vmbr1000
iface vmbr1000 inet manual
bridge_ports vxlan1000
bridge_stp off
bridge_fd 0

node2
-----
auto eno2
iface eno2 inet static
address 192.168.0.2/24
alias hypervisor_port_1

auto vxlan1000
iface vxlan1000 inet manual
vxlan-svcnodeip 239.0.0.1
vxlan-physdev eno2

auto vmbr1000
iface vmbr1000 inet manual
bridge_ports vxlan1000
bridge_stp off
bridge_fd 0


unicast setup
-------------
vnet1000: vxlan # VNI 1000
device hypervisor_port_1
type unicast
group 192.168.0.1,192.168.0.2,192.168.0.3,ip4,ip5,.. (all proxmox nodes ips or others ip)


this will add a vxlan1000 interface with "vxlan_remoteip ..." option, with all ips from "group", excluding ip of hypervisor_port_1 on each host,
and a vmbr1000 with 192.168.0.1 (anycast gateway)

node1
-----
auto eno1
iface eno1 inet static
address 192.168.0.1/24
alias hypervisor_port_1

auto vxlan1000
iface vxlan1000 inet manual
vxlan_remoteip 192.168.0.2
vxlan_remoteip 192.168.0.3

auto vmbr1000
iface vmbr1000 inet manual
bridge_ports vxlan1000
bridge_stp off
bridge_fd 0

node2
-----
auto eno2
iface eno2 inet static
address 192.168.0.2/24
alias hypervisor_port_1

auto vxlan1000
iface vxlan1000 inet manual
vxlan_remoteip 192.168.0.1
vxlan_remoteip 192.168.0.3

auto vmbr1000
iface vmbr1000 inet manual
bridge_ports vxlan1000
bridge_stp off
bridge_fd 0




evpn-bgp setup
--------------
vnet1000: vxlan # VNI 1000
device hypervisor_port_1
type evpn-bgp
gateway 10.0.0.1

this will add a vxlan1000 interface "vxlan-local-tunnelip ..." with ip address of hypervisor_port_1 on each host,
and a vmbr1000 with 192.168.0.1 (anycast gateway)


node1
-----
auto eno1
iface eno1 inet static
address 192.168.0.1/24
alias hypervisor_port_1

auto vxlan1000
iface vxlan1000 inet manual
vxlan-id 1000
vxlan-local-tunnelip 192.168.0.1
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off

auto vmbr1000
iface vmbr1000 inet static
address 10.0.0.1/24
bridge_ports vxlan1000
bridge_stp off
bridge_fd 0
node2
-----
auto eno2
iface eno2 inet static
address 192.168.0.2/24
alias hypervisor_port_1

auto vxlan1000
iface vxlan1000 inet manual
vxlan-id 1000
vxlan-local-tunnelip 192.168.0.2
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off

auto vmbr1000
iface vmbr1000 inet static
address 10.0.0.1/24
bridge_ports vxlan1000
bridge_stp off
bridge_fd 0





What do you think about this ?




----- Mail original -----
De: "dietmar" <***@proxmox.com>
À: "aderumier" <***@odiso.com>
Cc: "pve-devel" <pve-***@pve.proxmox.com>
Envoyé: Mardi 14 Août 2018 12:09:20
Objet: Re: [pve-devel] [PATCH pve-docs 0/1] vxlan l3 routing

Post by Alexandre DERUMIER

Post by Dietmar Maurer

Post by Dietmar Maurer
Not sure if the vxlan config is most times node related, so you always have node
related config anyways?

with frr, The vxlan interface indeed have the local ip vxlan-local-tunnelip.
(this could be found in frr.conf).
This could be managed with ifupdown2 template (as vxlan-local-tunnelip is always the same).
I was thinking of vmbr definition, as it's exactly the same on all nodes. (and
can't be managed easily with ifupdown2 template).
We can manage this with puppet without problem, but I was thinking for users
which only manage this through proxmox.

Just try - maybe you can find a way to define a cluster wide config
(etc/pve/network.cfg) for those bridges.

vnet1000: vxlan # VNI 1000
device vmbr0 # => vxlan-local-tunnelip
group 239.0.0.1

above would create the vtep and the bridge called 'vnet1000'?

...

[Alexandre DERUMIER]

Hi Dietmar,

I'll have time to work again on /etc/pve/network.cfg idea.

I don't known if you have some time to check my idea about using ifupdown2 "alias"


BTW,talking with students on last training (a lot coming from vmware world),
they think it could be great to be able to define global network (distributed virtual switch/ dvs in vmware),
for vlan or vxlan, and be able to make permissions on it. (maybe be able to add them in a proxmox pool).


----- Mail original -----
De: "aderumier" <***@odiso.com>
À: "dietmar" <***@proxmox.com>
Cc: "pve-devel" <pve-***@pve.proxmox.com>
Envoyé: Samedi 25 Août 2018 02:10:33
Objet: Re: [pve-devel] [PATCH pve-docs 0/1] vxlan l3 routing

Post by Alexandre DERUMIER

Post by Dietmar Maurer
Just try - maybe you can find a way to define a cluster wide config
(etc/pve/network.cfg) for those bridges.
vnet1000: vxlan # VNI 1000
device vmbr0 # => vxlan-local-tunnelip
group 239.0.0.1
above would create the vtep and the bridge called 'vnet1000'?

Hi Dietmar,

I think I have found a way.

ifupdown2 support "alias" options

"
auto eno1
iface eno1
alias hypervisor_port_1
"


ip link set dev ${interface name} alias "${description}"

Examples:
ip link set dev eno1 alias "hypervisor_port_1"
Link aliases show up in "ip link show" output, like:
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
link/ether 22:ce:e0:99:63:6f brd ff:ff:ff:ff:ff:ff
alias hypervisor_port_1



Like this, we could define a common alias on differents hosts, with differents interfaces,
and no need to use a vmbr0. (I'm thinking of layer3 underlay, with multiple interfaces with ecmp)



Some examples:
---------------

multicast setup
---------------
vnet1000: vxlan # VNI 1000
device hypervisor_port_1
type multicast
group 239.0.0.1



this will add a vxlan1000 interface with "vxlan-svcnodeip ..." option (with multicast address)
and vxlan-physdev (interface enX with hypervisor_port_1 alias)



node1
-----
auto eno1
iface eno1 inet static
address 192.168.0.1/24
alias hypervisor_port_1

auto vxlan1000
iface vxlan1000 inet manual
vxlan-svcnodeip 239.0.0.1
vxlan-physdev eno1

auto vmbr1000
iface vmbr1000 inet manual
bridge_ports vxlan1000
bridge_stp off
bridge_fd 0

node2
-----
auto eno2
iface eno2 inet static
address 192.168.0.2/24
alias hypervisor_port_1

auto vxlan1000
iface vxlan1000 inet manual
vxlan-svcnodeip 239.0.0.1
vxlan-physdev eno2

auto vmbr1000
iface vmbr1000 inet manual
bridge_ports vxlan1000
bridge_stp off
bridge_fd 0


unicast setup
-------------
vnet1000: vxlan # VNI 1000
device hypervisor_port_1
type unicast
group 192.168.0.1,192.168.0.2,192.168.0.3,ip4,ip5,.. (all proxmox nodes ips or others ip)


this will add a vxlan1000 interface with "vxlan_remoteip ..." option, with all ips from "group", excluding ip of hypervisor_port_1 on each host,
and a vmbr1000 with 192.168.0.1 (anycast gateway)

node1
-----
auto eno1
iface eno1 inet static
address 192.168.0.1/24
alias hypervisor_port_1

auto vxlan1000
iface vxlan1000 inet manual
vxlan_remoteip 192.168.0.2
vxlan_remoteip 192.168.0.3

auto vmbr1000
iface vmbr1000 inet manual
bridge_ports vxlan1000
bridge_stp off
bridge_fd 0

node2
-----
auto eno2
iface eno2 inet static
address 192.168.0.2/24
alias hypervisor_port_1

auto vxlan1000
iface vxlan1000 inet manual
vxlan_remoteip 192.168.0.1
vxlan_remoteip 192.168.0.3

auto vmbr1000
iface vmbr1000 inet manual
bridge_ports vxlan1000
bridge_stp off
bridge_fd 0




evpn-bgp setup
--------------
vnet1000: vxlan # VNI 1000
device hypervisor_port_1
type evpn-bgp
gateway 10.0.0.1

this will add a vxlan1000 interface "vxlan-local-tunnelip ..." with ip address of hypervisor_port_1 on each host,
and a vmbr1000 with 192.168.0.1 (anycast gateway)


node1
-----
auto eno1
iface eno1 inet static
address 192.168.0.1/24
alias hypervisor_port_1

auto vxlan1000
iface vxlan1000 inet manual
vxlan-id 1000
vxlan-local-tunnelip 192.168.0.1
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off

auto vmbr1000
iface vmbr1000 inet static
address 10.0.0.1/24
bridge_ports vxlan1000
bridge_stp off
bridge_fd 0
node2
-----
auto eno2
iface eno2 inet static
address 192.168.0.2/24
alias hypervisor_port_1

auto vxlan1000
iface vxlan1000 inet manual
vxlan-id 1000
vxlan-local-tunnelip 192.168.0.2
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off

auto vmbr1000
iface vmbr1000 inet static
address 10.0.0.1/24
bridge_ports vxlan1000
bridge_stp off
bridge_fd 0





What do you think about this ?




----- Mail original -----
De: "dietmar" <***@proxmox.com>
À: "aderumier" <***@odiso.com>
Cc: "pve-devel" <pve-***@pve.proxmox.com>
Envoyé: Mardi 14 Août 2018 12:09:20
Objet: Re: [pve-devel] [PATCH pve-docs 0/1] vxlan l3 routing

Post by Alexandre DERUMIER

Post by Dietmar Maurer

Post by Dietmar Maurer
Not sure if the vxlan config is most times node related, so you always have node
related config anyways?

with frr, The vxlan interface indeed have the local ip vxlan-local-tunnelip.
(this could be found in frr.conf).
This could be managed with ifupdown2 template (as vxlan-local-tunnelip is always the same).
I was thinking of vmbr definition, as it's exactly the same on all nodes. (and
can't be managed easily with ifupdown2 template).
We can manage this with puppet without problem, but I was thinking for users
which only manage this through proxmox.

Just try - maybe you can find a way to define a cluster wide config
(etc/pve/network.cfg) for those bridges.

vnet1000: vxlan # VNI 1000
device vmbr0 # => vxlan-local-tunnelip
group 239.0.0.1

above would create the vtep and the bridge called 'vnet1000'?

...

_______________________________________________
pve-devel mailing list
pve-***@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[Dietmar Maurer]

applied

Post by Alexandre Derumier
This add documentation for inter vxlan routing, with frr and anycast gateway.
External router documentation will come later.
http://odisoweb1.odiso.net/vxlan-l3-asymmetric.svg
http://odisoweb1.odiso.net/vxlan-l3-symmetric.svg