267 lines
9.5 KiB
Markdown
267 lines
9.5 KiB
Markdown
|
# Test of VLAN on openvswitch
|
|||
|
|
|
|||
|
|
## Note 1
|
|||
|
|
基本操作，建网桥，配置地址，启动网桥
|
|||
|
|
|
|||
|
|
ovs-vsctl add-br br0
|
|||
|
|
ip address add 172.0.0.1/8 dev br0
|
|||
|
|
ip link set br0 up
|
|||
|
|
|
|||
|
|
## Note 2
|
|||
|
|
LXC conf 中指定 pair 的名称，从而方便控制 网络链接
|
|||
|
|
|
|||
|
|
所以，需要修改 conf 文件来实现这一点
|
|||
|
|
|
|||
|
|
lxc.network.type = veth
|
|||
|
|
lxc.network.name = eth0
|
|||
|
|
lxc.network.script.up = Bridge=br0 /home/leebaok/Container/lxc-ifup
|
|||
|
|
lxc.network.script.down = Bridge=br0 /home/leebaok/Container/lxc-ifdown
|
|||
|
|
lxc.network.veth.pair = base
|
|||
|
|
lxc.network.ipv4 = 172.0.0.10/8
|
|||
|
|
lxc.network.ipv4.gateway = 172.0.0.1
|
|||
|
|
lxc.network.flags = up
|
|||
|
|
lxc.network.mtu = 1420
|
|||
|
|
|
|||
|
|
我们对上面的配置解释一下：
|
|||
|
|
* lxc.network.link 现在不需要了
|
|||
|
|
* lxc.network.script.up/down 来指定container启动前和关闭后的网络准备和释放，这个脚本的路径是物理机的路径，因为这个脚本是由物理机来执行的，“Bridge=br0” 是为了传参数给后面的脚本
|
|||
|
|
* lxc.network.veth.pair 是网络连接的名字，即container和物理机的哪个口连接
|
|||
|
|
|
|||
|
|
配置了网络设置的脚本路径，我们还需要实现这两个具体的脚本：
|
|||
|
|
* /home/leebaok/Container/lxc-ifup
|
|||
|
|
|
|||
|
|
|
|||
|
|
#!/bin/bash
|
|||
|
|
# $1 : name of container ( name in lxc-start with -n )
|
|||
|
|
# $2 : net
|
|||
|
|
# $3 : network flags, up or down
|
|||
|
|
# $4 : network type, for example, veth
|
|||
|
|
# $5 : value of lxc.network.veth.pair
|
|||
|
|
ovs-vsctl --may-exist add-port $Bridge $5
|
|||
|
|
# ovs-vsctl set port $5 tag=$Tag
|
|||
|
|
|
|||
|
|
* /home/leebaok/Container/lxc-ifdown
|
|||
|
|
|
|||
|
|
|
|||
|
|
#!/bin/bash
|
|||
|
|
# $1 : name of container ( name in lxc-start with -n )
|
|||
|
|
# $2 : net
|
|||
|
|
# $3 : network flags, up or down
|
|||
|
|
# $4 : network type, for example, veth
|
|||
|
|
# $5 : value of lxc.network.veth.pair
|
|||
|
|
ovs-vsctl --if-exists del-port $Bridge $5
|
|||
|
|
|
|||
|
|
## Note 3
|
|||
|
|
VLAN tag 操作：
|
|||
|
|
|
|||
|
|
ovs-vsctl set port <port-name> tag=<tag-id>
|
|||
|
|
ovs-vsctl clear port <port-name> tag
|
|||
|
|
|
|||
|
|
patch 是用来连接两个网桥的，操作如下：
|
|||
|
|
|
|||
|
|
ovs-vsctl add-br br0
|
|||
|
|
ovs-vsctl add-br br1
|
|||
|
|
ovs-vsctl add-port br0 patch0 -- set interface patch0 type=patch options:peer=patch1
|
|||
|
|
ovs-vsctl add-port br1 patch1 -- set interface patch1 type=patch options:peer=patch0
|
|||
|
|
# NOW : two bridges are connected by patch
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Note 4
|
|||
|
|
一台机器上一个域的网桥只有一个，比如在 host-0 上，建两个网桥：
|
|||
|
|
|
|||
|
|
ovs-vsctl add-br br0
|
|||
|
|
ip address add 172.0.0.1/8 dev br0
|
|||
|
|
ip link set br0 up
|
|||
|
|
|
|||
|
|
ovs-vsctl add-br br1
|
|||
|
|
ip address add 172.0.0.2/8 dev br1
|
|||
|
|
ip link set br1 up
|
|||
|
|
|
|||
|
|
则，后配置的那个网桥会失效
|
|||
|
|
|
|||
|
|
因为系统认为，172.0.0.1/8 内的机器都应该在 br0 中
|
|||
|
|
|
|||
|
|
而以下配置是正确的：
|
|||
|
|
|
|||
|
|
ovs-vsctl add-br br0
|
|||
|
|
ip address add 172.0.0.1/24 dev br0
|
|||
|
|
ip link set br0 up
|
|||
|
|
|
|||
|
|
ovs-vsctl add-br br1
|
|||
|
|
ip address add 172.0.1.1/24 dev br1
|
|||
|
|
ip link set br1 up
|
|||
|
|
|
|||
|
|
## Note 5
|
|||
|
|
关于网关，网桥/交换机是二层设备，网关是三层组件，我们可以将网桥连接起来，多个网桥共用一个网关
|
|||
|
|
|
|||
|
|
ovs-vsctl add-br br0
|
|||
|
|
ip link set br0 up
|
|||
|
|
ovs-vsctl add-br br1
|
|||
|
|
ip address add 172.0.0.1/24 dev br1
|
|||
|
|
ip link set br1 up
|
|||
|
|
ovs-vsctl add-port br0 patch0 -- set interface patch0 type=patch options:peer=patch1
|
|||
|
|
ovs-vsctl add-port br1 patch1 -- set interface patch1 type=patch options:peer=patch0
|
|||
|
|
|
|||
|
|
# lxc config :
|
|||
|
|
# ip -- 172.0.0.11/24
|
|||
|
|
# gateway -- 172.0.0.1
|
|||
|
|
# lxc.network.veth.pair -- base , base is connected on br0
|
|||
|
|
lxc-start -f container.conf -n base -F -- /bin/bash
|
|||
|
|
# NOW ： lxc network is running ok
|
|||
|
|
|
|||
|
|
## Note 6
|
|||
|
|
基于多个网桥实现VLAN
|
|||
|
|
|
|||
|
|
### 方案一
|
|||
|
|
|
|||
|
|
ovs-vsctl add-br br0
|
|||
|
|
ip link set br0 up
|
|||
|
|
ovs-vsctl add-br br1
|
|||
|
|
ip address add 172.0.0.1/24 dev br1
|
|||
|
|
ip link set br1 up
|
|||
|
|
ovs-vsctl add-port br0 patch0 -- set interface patch0 type=patch options:peer=patch1
|
|||
|
|
ovs-vsctl add-port br1 patch1 -- set interface patch1 type=patch options:peer=patch0
|
|||
|
|
|
|||
|
|
# lxc config :
|
|||
|
|
# ip -- 172.0.0.11/24
|
|||
|
|
# gateway -- 172.0.0.1
|
|||
|
|
# lxc.network.veth.pair -- base , base is connected on br0
|
|||
|
|
lxc-start -f container.conf -n base -F -- /bin/bash
|
|||
|
|
# NOW ： lxc network is running ok
|
|||
|
|
## above is the same as before
|
|||
|
|
|
|||
|
|
ovs-vsctl set port base tag=5
|
|||
|
|
ovs-vsctl set port patch0 tag=5
|
|||
|
|
# NOW : lxc network is running ok
|
|||
|
|
|
|||
|
|
# ARCH
|
|||
|
|
+-----------------------+ +----------------------+
|
|||
|
|
| br0 | | br1 : 172.0.0.1/24 |
|
|||
|
|
+--+-----tag=5---tag=5--+ +---+-------+----------+
|
|||
|
|
| | | patch | |
|
|||
|
|
| | +-------------------+ |
|
|||
|
|
| | |
|
|||
|
|
internal base:172.0.0.11/24 internal
|
|||
|
|
(gateway:172.0.0.1)
|
|||
|
|
|
|||
|
|
# flow : base --> patch --> br1/internal
|
|||
|
|
|
|||
|
|
* 方案可行
|
|||
|
|
* 但是，每个 VLAN 需要一个网关
|
|||
|
|
|
|||
|
|
### 方案二 （不可行）
|
|||
|
|
|
|||
|
|
# ARCH
|
|||
|
|
+-------------------------------------------------------------+
|
|||
|
|
| br0 |
|
|||
|
|
+--+-----tag=5---tag=5---------+-----tag=6---tag=6---------+--+
|
|||
|
|
| | | +-----+ | | | +-----+ |
|
|||
|
|
| | +--| br1 |--+ | +--| br2 |--+
|
|||
|
|
| | +-----+ | +-----+
|
|||
|
|
internal base1:172.0.0.11/24 base2:172.0.0.12/24
|
|||
|
|
|
|||
|
|
# flow 1 : base1 --> br1 --> internal
|
|||
|
|
# flow 2 : base1 --> br1 --> br2 --> base2
|
|||
|
|
|
|||
|
|
* 方案不可行，因为上面的 flow 可以使得 base1、base2 在二层通信，无法隔离
|
|||
|
|
|
|||
|
|
## Note 7
|
|||
|
|
上述可行方案的简化版
|
|||
|
|
### 简化版一
|
|||
|
|
|
|||
|
|
ovs-vsctl add-br br0
|
|||
|
|
ip link set br0 up
|
|||
|
|
# add a fake bridge connected to br0 with vlan tag=5
|
|||
|
|
ovs-vsctl add-br fakebr br0 5
|
|||
|
|
ip address add 172.0.0.1/24 dev fakebr
|
|||
|
|
ip link set fakebr up
|
|||
|
|
|
|||
|
|
# lxc config:
|
|||
|
|
# ip : 172.0.0.11/24
|
|||
|
|
# gateway : 172.0.0.1/24
|
|||
|
|
# lxc.network.veth.pair -- base , base is connected on br0
|
|||
|
|
lxc-start -f container.conf -n base -F -- /bin/bash
|
|||
|
|
|
|||
|
|
ovs-vsctl set port base tag=5
|
|||
|
|
|
|||
|
|
# ARCH
|
|||
|
|
+-----------------------+
|
|||
|
|
| br0 |
|
|||
|
|
+--+-----tag=5---tag=5--+
|
|||
|
|
| | |
|
|||
|
|
| | fakebr:172.0.0.1/24
|
|||
|
|
| |
|
|||
|
|
internal base:172.0.0.11/24
|
|||
|
|
(gateway:172.0.0.1)
|
|||
|
|
|
|||
|
|
# flow : base --> fakebr
|
|||
|
|
|
|||
|
|
### 简化版二
|
|||
|
|
|
|||
|
|
ovs-vsctl add-br br0
|
|||
|
|
ip link set br0 up
|
|||
|
|
# add an internal interface for vlan
|
|||
|
|
ovs-vsctl add-port br0 vlanif tag=5 -- set interface vlanif type=internal
|
|||
|
|
ip address add 172.0.0.1/24 dev vlanif
|
|||
|
|
ip link set vlanif up
|
|||
|
|
|
|||
|
|
# lxc config:
|
|||
|
|
# ip : 172.0.0.11/24
|
|||
|
|
# gateway : 172.0.0.1/24
|
|||
|
|
# lxc.network.veth.pair -- base , base is connected on br0
|
|||
|
|
lxc-start -f container.conf -n base -F -- /bin/bash
|
|||
|
|
|
|||
|
|
ovs-vsctl set port base tag=5
|
|||
|
|
|
|||
|
|
# ARCH
|
|||
|
|
+-----------------------+
|
|||
|
|
| br0 |
|
|||
|
|
+--+-----tag=5---tag=5--+
|
|||
|
|
| | |
|
|||
|
|
| | vlanif:172.0.0.1/24
|
|||
|
|
| |
|
|||
|
|
internal base:172.0.0.11/24
|
|||
|
|
(gateway:172.0.0.1)
|
|||
|
|
|
|||
|
|
# flow : base --> vlanif
|
|||
|
|
|
|||
|
|
### 简化版一 & 简化版二
|
|||
|
|
使用 ovs-vsctl show 查看的时候，上述两个版本显示的信息是一样的，说明 fakebr 其实本质上可能就是一个 internal interface
|
|||
|
|
|
|||
|
|
其实，方案一中，对 br1 的 IP（172.0.0.1/24）的配置，其实就是对 br1 的 internal 的 interface 的配置，所以其实多余的网桥不是必须的，而 interface 才是真正需要的。
|
|||
|
|
|
|||
|
|
而，internal interface 相当于是连接着本地Linux的虚拟网卡，这块网卡的另一端连着OVS的虚拟网桥。
|
|||
|
|
|
|||
|
|
而，Linux 的网络栈又管理着物理网卡、虚拟网卡，以及对这些网卡的包进行转发、路由等处理。
|
|||
|
|
|
|||
|
|
似乎，Linux 的网络栈又成了一个大的交换机/网桥，上面连接着 internal interface 和 物理网卡。
|
|||
|
|
|
|||
|
|
## Note 8
|
|||
|
|
基于上述的实践和探索，其实 **我们需要给一个VLAN配置一个可以出去的网关、网卡。**
|
|||
|
|
|
|||
|
|
那么，我们一个简单可行的方案可以这样：
|
|||
|
|
|
|||
|
|
+------------------------------------------------------------------------------+
|
|||
|
|
| bridge |
|
|||
|
|
| <------- VLAN ID=5 ---------> <---- VLAN ID=6 ------> |
|
|||
|
|
+--+-----tag=5---tag=5------------tag=5-------------tag=6-------------tag=6----+
|
|||
|
|
| | | | | |
|
|||
|
|
| | lxc-2:172.0.0.12/24 | | |
|
|||
|
|
internal | (gateway:172.0.0.1) | | |
|
|||
|
|
| | | |
|
|||
|
|
lxc-1:172.0.0.11/24 gw5:172.0.0.1/24 lxc-3:172.0.1.11/24 gw6:172.0.1.1/24
|
|||
|
|
(gateway:172.0.0.1) internal (gateway:172.0.1.1) internal
|
|||
|
|
| |
|
|||
|
|
| |
|
|||
|
|
+----------- NAT / iptables --------+
|
|||
|
|
||||
|
|||
|
|
||||
|
|||
|
|
\\\///
|
|||
|
|
\\//
|
|||
|
|
\/
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
# end
|