The Geek Diary

How to Configure Network Interface Teaming in CentOS/RHEL 7 and 8

by

Network interface teaming was introduced from CentOS/RHEL 7 as a more extensible and scalable alternative to network bonding. This post describes how to configure network teaming on CentOS/RHEL 7/8.

The examples provided are based on Oracle Linux 8.2 system (Oracle VirtualBox 6.1 guest virtual machine) with two network interfaces using NetworkManager.
Specifically, the Network Manager Command Line Interface (nmcli) is primarily used with options specified in abbreviated/shortened form.

1. Original Pre-Team Configuration

The following denotes the original, pre-team network configuration:

# dnf list | grep team | grep anaconda
NetworkManager-team.x86_64        1:1.22.8-4.el8                 @anaconda
libteam.x86_64                    1.29-1.el8                     @anaconda
teamd.x86_64                      1.29-1.el8                     @anaconda
# lspci | grep -i eth
00:03.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)
00:08.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)
# nmcli dev
DEVICE  TYPE      STATE      CONNECTION
enp0s3  ethernet  connected  enp0s3
enp0s8  ethernet  connected  enp0s8
lo      loopback  unmanaged  --
# nmcli con
NAME    UUID                                  TYPE      DEVICE
enp0s3  XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX  ethernet  enp0s3
enp0s8  XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX  ethernet  enp0s8
# ls -l /etc/sysconfig/network-scripts/*
-rw-r--r-- 1 root root 334 Aug 21 13:29 /etc/sysconfig/network-scripts/ifcfg-enp0s3
-rw-r--r-- 1 root root 334 Aug 21 13:30 /etc/sysconfig/network-scripts/ifcfg-enp0s8
# cat /etc/sysconfig/network-scripts/ifcfg-enp0s3
  1 MACADDR=[MAC_ADDR1]
  2 MTU=1500
  3 TYPE=Ethernet
  4 PROXY_METHOD=none
  5 BROWSER_ONLY=no
  6 BOOTPROTO=dhcp
  7 DEFROUTE=yes
  8 IPV4_FAILURE_FATAL=no
  9 IPV6INIT=yes
  10 IPV6_AUTOCONF=yes
  11 IPV6_DEFROUTE=yes
  12 IPV6_FAILURE_FATAL=no
  13 IPV6_ADDR_GEN_MODE=stable-privacy
  14 NAME="enp0s3"
  15 UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  16 DEVICE=enp0s3
  17 ONBOOT=yes
# cat /etc/sysconfig/network-scripts/ifcfg-enp0s8
  1 MACADDR=[MAC_ADDR2]
  2 MTU=1500
  3 TYPE=Ethernet
  4 PROXY_METHOD=none
  5 BROWSER_ONLY=no
  6 BOOTPROTO=dhcp
  7 DEFROUTE=yes
  8 IPV4_FAILURE_FATAL=no
  9 IPV6INIT=yes
  10 IPV6_AUTOCONF=yes
  11 IPV6_DEFROUTE=yes
  12 IPV6_FAILURE_FATAL=no
  13 IPV6_ADDR_GEN_MODE=stable-privacy
  14 NAME="enp0s8"
  15 UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  16 DEVICE=enp0s8
  17 ONBOOT=yes
# ip addr
...
2: enp0s3:  mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether  brd ff:ff:ff:ff:ff:ff
    inet /24 brd  scope global dynamic noprefixroute enp0s3
       valid_lft 86059sec preferred_lft 86059sec
    inet6 fe80::ca99:46d3:1765:f02b/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: enp0s8:  mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether  brd ff:ff:ff:ff:ff:ff
    inet /24 brd  scope global dynamic noprefixroute enp0s8
       valid_lft 86121sec preferred_lft 86121sec
    inet6 fe80::36d0:6bd3:5152:83dc/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

2. Delete existing network connections

Delete the existing enp0s3 and enp0s8 connections as follows. These are recreated as team slaves in the following steps.

# nmcli con show
NAME    UUID                                  TYPE      DEVICE
enp0s3  XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX  ethernet  enp0s3
enp0s8  XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX  ethernet  enp0s8
# nmcli con del XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
Connection 'enp0s3' (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX) successfully deleted.
# nmcli con del XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
Connection 'enp0s8' (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX) successfully deleted.
# nmcli dev
DEVICE  TYPE      STATE         CONNECTION
enp0s3  ethernet  disconnected  --
enp0s8  ethernet  disconnected  --
lo      loopback  unmanaged     --
# ls -l /etc/sysconfig/network-scripts/
total 0
#

3. Create master team connection

Create the master team connection. For example:

# nmcli con add type team con-name team0 ifname team0 config '{"runner": {"name": "activebackup"}, "link_watch": {"name": "ethtool"}}'
Connection 'team0' (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX) successfully added.

Above, the team running mode (runner) is activebackup and the team link monitor/watcher (link_watch) is ethtool. Alternative runner and link_watch values include:

  • runner: loadbalance, roundrobin, lacp, broadcast, random.
  • link_watch: arp_ping, nsna_ping

NetworkManager creates the following interface configuration file:

# cat /etc/sysconfig/network-scripts/ifcfg-team0
  1 TEAM_CONFIG="{\"runner\": {\"name\": \"activebackup\"}, \"link_watch\": {\"name\": \"ethtool\"}}"
  2 PROXY_METHOD=none
  3 BROWSER_ONLY=no
  4 BOOTPROTO=dhcp
  5 DEFROUTE=yes
  6 IPV4_FAILURE_FATAL=no
  7 IPV6INIT=yes
  8 IPV6_AUTOCONF=yes
  9 IPV6_DEFROUTE=yes
 10 IPV6_FAILURE_FATAL=no
 11 IPV6_ADDR_GEN_MODE=stable-privacy
 12 NAME=team0
 13 UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
 14 DEVICE=team0
 15 ONBOOT=yes
 16 DEVICETYPE=Team
# nmcli con
NAME   UUID                                  TYPE  DEVICE
team0  XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX  team  team0
# nmcli dev
DEVICE  TYPE      STATE                                  CONNECTION
team0   team      connecting (getting IP configuration)  team0
enp0s3  ethernet  disconnected                           --
enp0s8  ethernet  disconnected                           --
lo      loopback  unmanaged

4. Optionally assign a static IP address to the team

Optionally assign a static IP address, gateway, DNS, etc. to the team connection. For example:

# nmcli con mod team0 ipv4.addresses [IP3]/24
# nmcli con mod team0 ipv4.gateway [IP4]
# nmcli con mod team0 ipv4.dns [IP5]
# nmcli con mod team0 ipv4.method manual
# nmcli con mod team0 connection.autoconnect yes

NetworkManager modifies the following team interface configuration file:

# cat /etc/sysconfig/network-scripts/ifcfg-team0
  1 TEAM_CONFIG="{\"runner\": {\"name\": \"activebackup\"}, \"link_watch\": {\"name\": \"ethtool\"}}"
  2 PROXY_METHOD=none
  3 BROWSER_ONLY=no
  4 BOOTPROTO=none
  5 DEFROUTE=yes
  6 IPV4_FAILURE_FATAL=no
  7 IPV6INIT=yes
  8 IPV6_AUTOCONF=yes
  9 IPV6_DEFROUTE=yes
 10 IPV6_FAILURE_FATAL=no
 11 IPV6_ADDR_GEN_MODE=stable-privacy
 12 NAME=team0
 13 UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
 14 DEVICE=team0
 15 ONBOOT=yes
 16 DEVICETYPE=Team
 17 IPADDR=[IP3]
 18 PREFIX=24
 19 GATEWAY=[IP4]
 20 DNS1=[IP5]

The team will utilise DHCP if no static IP address is assigned.

5. Configure and add slaves to the team

Configure and add slaves to team. For example:

# nmcli con add type team-slave con-name team0-slave0 ifname enp0s3 master team0
Connection 'team0-slave0' (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX) successfully added.
# nmcli con add type team-slave con-name team0-slave1 ifname enp0s8 master team0
Connection 'team0-slave1' (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX) successfully added.
# nmcli conn
NAME          UUID                                  TYPE      DEVICE
team0         XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX  team      team0
team0-slave0  XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX  ethernet  enp0s3
team0-slave1  XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX  ethernet  enp0s8
# nmcli dev
DEVICE  TYPE      STATE      CONNECTION
team0   team      connected  team0
enp0s3  ethernet  connected  team0-slave0
enp0s8  ethernet  connected  team0-slave1
lo      loopback  unmanaged  --

NetworkManager creates the following team slave interface configuration files:

# cat /etc/sysconfig/network-scripts/ifcfg-team0-slave0
  1 NAME=team0-slave0
  2 UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  3 DEVICE=enp0s3
  4 ONBOOT=yes
  5 TEAM_MASTER=team0
  6 DEVICETYPE=TeamPort
# cat /etc/sysconfig/network-scripts/ifcfg-team0-slave1
  1 NAME=team0-slave1
  2 UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  3 DEVICE=enp0s8
  4 ONBOOT=yes
  5 TEAM_MASTER=team0
  6 DEVICETYPE=TeamPort

When at least one slave is added to the team, the interface (team0) is brought up and becomes accessible. 

# ip addr
...
2: enp0s3: [BROADCAST,MULTICAST,UP,LOWER_UP] mtu 1500 qdisc fq_codel master team0 state UP group default qlen 1000
    link/ether [MAC1] brd ff:ff:ff:ff:ff:ff
3: enp0s8: [BROADCAST,MULTICAST,UP,LOWER_UP] mtu 1500 qdisc fq_codel master team0 state UP group default qlen 1000
    link/ether [MAC1] brd ff:ff:ff:ff:ff:ff
6: team0: [BROADCAST,MULTICAST,UP,LOWER_UP] mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether [MAC1] brd ff:ff:ff:ff:ff:ff
    inet [IP3]/24 brd [IP6] scope global dynamic noprefixroute team0
       valid_lft 86045sec preferred_lft 86045sec
    inet6 fe80::5b1f:554a:1928:8575/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

Note that all team/teamed interfaces utilise the same MAC address i.e. that of the first team slave member of the team.

6. Restart the team

Restart the team for the static IP address, etc. to take effect. For example:

# nmcli con down team0 && nmcli con up team0

7. Identify the current active/inactive slave interfaces

Identify the current active and inactive slave interfaces using teamdctl(8). For example:

# teamdctl team0 state
setup:
  runner: activebackup
ports:
  enp0s3
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up
        down count: 0
  enp0s8
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up
        down count: 0
runner:
  active port: enp0s3

8. Enable promiscuous mode for network interfaces

Some virtualisation technologies, such as Oracle VM VirtualBox, require promiscuous mode to be enabled on network interfaces assigned to guests, as well as within guests in order for slave failover/failback to behave correctly. Enabling promiscuous mode on network interfaces on physical systems may also be required.

For Oracle VM VirtualBox, promiscuous mode can be enabled for guest interfaces as follows:
– Oracle VM VirtualBox Manager > [GUEST] > Settings > Network > Adapter 1|… ] Advanced ] Promiscuous Mode: Allow All

Promiscuous mode can be enabled within CentOS/RHEL 7 systems dynamically and statically using a custom service as follows:

Dynamic, non-persistent:

# ip link set enp0s3 promisc on
# ip link set enp0s8 promisc on

Static, persistent:

Create a custom systemd unit file with the following contents e.g.:

# cat /usr/lib/systemd/system/promiscuous.service
  1 [Unit]
  2 Description=Bring up network interfaces in promiscuous mode upon boot
  3 After=network.target
  4
  5 [Service]
  6 Type=oneshot
  7 ExecStart=/usr/sbin/ip link set dev enp0s3 promisc on
  8 ExecStart=/usr/sbin/ip link set dev enp0s8 promisc on
  9 ExecStop=/usr/sbin/ip link set dev enp0s3 promisc off
 10 ExecStop=/usr/sbin/ip link set dev enp0s8 promisc off
 11 TimeoutStartSec=0
 12 RemainAfterExit=yes
 13
 14 [Install]
 15 WantedBy=default.target

Inform systemd of the new service. For example:

# systemctl daemon-reload

Enable and start the new service/unit e.g.:

# systemctl enable promiscuous
Created symlink /etc/systemd/system/default.target.wants/promiscuous.service → /usr/lib/systemd/system/promiscuous.service.
# systemctl start promiscuous
# systemctl status promiscuous
● promiscuous.service - Bring up network interfaces in promiscuous mode upon boot
   Loaded: loaded (/usr/lib/systemd/system/promiscuous.service; enabled; vendor preset: disabled)
   Active: active (exited) since Fri 2020-08-21 16:14:53 AEST; 17s ago
  Process: 8088 ExecStart=/usr/sbin/ip link set dev enp0s8 promisc on (code=exited, status=0/SUCCESS)
  Process: 8086 ExecStart=/usr/sbin/ip link set dev enp0s3 promisc on (code=exited, status=0/SUCCESS)
 Main PID: 8088 (code=exited, status=0/SUCCESS)

Aug 21 16:14:53 [HOST] systemd[1]: Starting Bring up network interfaces in promiscuous mode upon boot...
Aug 21 16:14:53 [HOST] systemd[1]: Started Bring up network interfaces in promiscuous mode upon boot.

Verify promiscuous mode is enabled on all slave interfaces. For example:

# ip addr | grep enp
2: enp0s3: [BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP] mtu 1500 qdisc fq_codel master team0 state UP group default qlen 1000
3: enp0s8: [BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP] mtu 1500 qdisc fq_codel master team0 state UP group default qlen 1000

9. Testing Team network connectivity resilience – slave failover/failback

A. From a remote client, initiate continuous ping(8) of the server for which network teaming has been configured e.g.:

[CLIENT]$ ping [SERVER]
PING [IP3] ([IP3]) 56(84) bytes of data.
64 bytes from [IP3]: icmp_seq=1 ttl=64 time=0.025 ms
64 bytes from [IP3]: icmp_seq=2 ttl=64 time=0.034 ms
64 bytes from [IP3]: icmp_seq=3 ttl=64 time=0.039 ms
...

B. Disconnect the currently active team slave.

Temporarily disconnect the physical network cable connected to slave interface e.g. enp0s3. Note that disabling an interface programmatically does not emulate actual physical link connectivity loss.

# teamdctl team0 state
setup:
  runner: activebackup
ports:
  enp0s3
    link watches:
      link summary: down
      instance[link_watch_0]:
        name: ethtool
        link: down
        down count: 1
  enp0s8
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up
        down count: 0
runner:
  active port: enp0s8

Once disconnected, teamdctl denotes slave interface enp0s3 as down and former backup slave enp0s8 is now the current active slave. Despite slave failover, ping(8) from the remote client continues uninterrupted.

Running the ip(8) command further confirms all network traffic to the team occurs via new active slave enp0s8 i.e.:

# ip -s link
...
2: enp0s3: [BROADCAST,MULTICAST,PROMISC] mtu 1500 qdisc fq_codel master team0 state DOWN mode DEFAULT group default qlen 1000
    link/ether [MAC1] brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets  errors  dropped overrun mcast
    3007606561 5014092  0       27008   0       91263
    TX: bytes  packets  errors  dropped carrier collsns
    787749     7568     0       0       0       0
3: enp0s8: [BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP] mtu 1500 qdisc fq_codel master team0 state UP mode DEFAULT group default qlen 1000
    link/ether [MAC1] brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors dropped overrun mcast
    3829373093 6116137  0      26680   0       111948
    TX: bytes  packets  errors dropped carrier collsns
    179163     1337     0      0       0       0
7: team0: [BROADCAST,MULTICAST,UP,LOWER_UP] mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether [MAC1] brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors dropped overrun mcast
    2106463511 1969187  0      49615   0       60559
    TX: bytes  packets  errors dropped carrier collsns
    176341     1378     0      0       0       0

C. Disconnect new current active team slave.

Temporarily disconnect the physical network cable connected to the newly promoted active slave interface e.g. enp0s8. Once disconnected, teamdctl denotes both slave interfaces enp0s3 and enp0s8 as down i.e.:

# teamdctl team0 state
setup:
  runner: activebackup
ports:
  enp0s3
    link watches:
      link summary: down
      instance[link_watch_0]:
        name: ethtool
        link: down
        down count: 1
  enp0s8
    link watches:
      link summary: down
      instance[link_watch_0]:
        name: ethtool
        link: up
        down count: 1
runner:
  active port:

At this point, with both team slaves disconnected, ping(8) from the remote client ceases e.g.

...
64 bytes from [IP3]: icmp_seq=1253 ttl=64 time=0.207 ms
64 bytes from [IP3]: icmp_seq=1254 ttl=64 time=0.131 ms
64 bytes from [IP3]: icmp_seq=1255 ttl=64 time=0.227 ms
64 bytes from [IP3]: icmp_seq=1256 ttl=64 time=0.218 ms
64 bytes from [IP3]: icmp_seq=1257 ttl=64 time=0.198 ms
From [IP3] icmp_seq=1258 Destination Host Unreachable
From [IP3] icmp_seq=1259 Destination Host Unreachable
...

D. Reconnect disconnected slave

Reconnect the physical network cable to one of the slave interfaces e.g. enp0s3. Once reconnected, teamdctl denotes slave interface enp0s3 as up i.e.:

# teamdctl team0 state
setup:
  runner: activebackup
ports:
  enp0s3
    link watches:
    link summary: up
      instance[link_watch_0]:
      name: ethtool
      link: up
      down count: 1
  enp0s8
    link watches:
    link summary: down
      instance[link_watch_0]:
      name: ethtool
      link: down
      down count: 1
runner:
  active port: enp0s3

At this point, with enp0s3 the current active slave, ping(8) from the remote client resumes e.g.

...
From [IP3] icmp_seq=1392 Destination Host Unreachable
From [IP3] icmp_seq=1393 Destination Host Unreachable
From [IP3] icmp_seq=1394 Destination Host Unreachable
From [IP3] icmp_seq=1395 Destination Host Unreachable
64 bytes from [IP3]: icmp_seq=1396 ttl=64 time=1258180 ms
64 bytes from [IP3]: icmp_seq=1397 ttl=64 time=1257180 ms
64 bytes from [IP3]: icmp_seq=1398 ttl=64 time=1256181 ms
64 bytes from [IP3]: icmp_seq=1399 ttl=64 time=1255181 ms
...

E. Reconnect remaining disconnected slave.

Reconnect the physical network cable to the remaining slave interface e.g. enp0s8. Once reconnected, teamdctl denotes remaining slave interface enp0s8 as up i.e.:

# teamdctl team0 state
setup:
  runner: activebackup
ports:
  enp0s3
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up
        down count: 1
  enp0s8
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up
        down count: 1
runner:
  active port: enp0s3

You May Also Like