The Setup
This configuration uses below OS, Storage and Server Hardware –
OS release – “RHEL 7.1”
Kernel – 3.8.13-55.1.6.el7.x86_64
Hardware: HP ProLiant BL460c Gen8
Ethernet Interfaces with FCoE capability –
04:00.0 Ethernet controller: Broadcom Corporation BCM57840 NetXtreme II 10/20-Gigabit Ethernet (rev 11)
04:00.1 Ethernet controller: Broadcom Corporation BCM57840 NetXtreme II 10/20-Gigabit Ethernet (rev 11)
04:00.2 Ethernet controller: Broadcom Corporation BCM57840 NetXtreme II 10/20-Gigabit Ethernet (rev 11)
04:00.3 Ethernet controller: Broadcom Corporation BCM57840 NetXtreme II 10/20-Gigabit Ethernet (rev 11)
04:00.4 Ethernet controller: Broadcom Corporation BCM57840 NetXtreme II 10/20-Gigabit Ethernet (rev 11)
04:00.5 Ethernet controller: Broadcom Corporation BCM57840 NetXtreme II 10/20-Gigabit Ethernet (rev 11)
Storage – HP 3PAR StoreServ Storage server (FCoE target).
Switch – FCoE-enabled switch.
The Steps
Follow the steps outlined below to configure the FCoE interface in a CentOS/RHEL system.
1. Configure FCoE using system BIOS – Consult your Server HW documentation for instructions of how to enable FCoE on available interfaces.
2. Install the required packages:
# yum install fcoe-utils lldpad
3. Find out the interfaces configured for FCoE – In this example its eno3 (6C:C2:17:2B:4D:11) and eno4 (6C:C2:17:2B:4D:19).
4. Configure Ethernet interface to use FCoE – bring down the interfaces and fcoe service if running:
# ifconfig eno3 down # ifconfig eno4 down
and
# systemctl stop fcoe ### if running
5. Configure a new VLAN by copying an existing network script i.e. vi /etc/fcoe/cfg-eth0 to the name of the Ethernet device that supports FCoE. The configuration file /etc/fcoe/cfg-eth0 will give you the default settings to configure. Given that the FCoE device is ethX, run:
# cp /etc/fcoe/cfg-eth0 /etc/fcoe/cfg-ethX
Modify the FCoE config file as follows:
# cp /etc/fcoe/cfg-eth0 /etc/fcoe/cfg-eno3 # cp /etc/fcoe/cfg-eth0 /etc/fcoe/cfg-eno4
6. Modify /etc/fcoe/cfg-[interface] by setting DCB_REQUIRED to “no“. DCB_REQUIRED should be set to no for networking interfaces that implement a hardware DCBX client. Here we are not using any hardware DCBX client. More information on Data Center Bridging can be read at – http://linux.die.net/man/8/fcoemon.
For example:
$ cat /etc/fcoe/cfg-eno3
## Type: yes/no
## Default: no
# Enable/Disable FCoE service at the Ethernet port
# Normally set to "yes"
FCOE_ENABLE="yes"
## Type: yes/no
## Default: no
# Indicate if DCB service is required at the Ethernet port
# Normally set to "yes"
DCB_REQUIRED="no"
#DCB_REQUIRED="yes"
## Type: yes/no
## Default: no
# Indicate if VLAN discovery should be handled by fcoemon
# Normally set to "yes"
AUTO_VLAN="yes"
## Type: fabric/vn2vn
## Default: fabric
# Indicate the mode of the FCoE operation, either fabric or vn2vn
# Normally set to "fabric"
MODE="fabric"
## Type: yes/no
## Default: no
# Indicate whether to run a FIP responder for VLAN discovery in vn2vn mode
#FIP_RESP="yes"
Do the same for eno4 interface.
7. Activate Interface on boot and enable Jumbo Frames – Set “ONBOOT=yes” in the corresponding /etc/sysconfig/network-scripts/ifcfg-eno3 and /etc/sysconfig/network-scripts/ifcfg-eno4 file to activate automatically during boot time. Also enable jumbo frames (MTU=9000) on the FCoE enabled interfaces. With jumbo frames enabled on the interfaces, it will avoid unnecessary IP fragmentation.
# cat /etc/sysconfig/network-scripts/ifcfg-eno3 TYPE=Ethernet BOOTPROTO=dhcp DEFROUTE=yes PEERDNS=yes PEERROUTES=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_PEERDNS=yes IPV6_PEERROUTES=yes IPV6_FAILURE_FATAL=no NAME=eno3 UUID=fae9a6da-f05b-497e-b625-ba5826db0d78 DEVICE=eno3 #ONBOOT=no ONBOOT=yes MTU=9000
# cat /etc/sysconfig/network-scripts/ifcfg-eno4 TYPE=Ethernet BOOTPROTO=dhcp DEFROUTE=yes PEERDNS=yes PEERROUTES=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_PEERDNS=yes IPV6_PEERROUTES=yes IPV6_FAILURE_FATAL=no NAME=eno4 UUID=63157ec1-3013-4895-8ea4-079c5865d364 DEVICE=eno4 #ONBOOT=no ONBOOT=yes MTU=9000
8. For interfaces that require a software DCBX client – Enable data center bridging on the Ethernet interface using the following commands:
# dcbtool sc ethX dcb on
Then, enable FCoE on the Ethernet interface by running:
# dcbtool sc ethX app:fcoe e:1
Enable Data Center Bridging on the Ethernet interface using the following commands:
# dcbtool sc eno3 dcb on # dcbtool sc eno4 dcb on
Then, enable FCoE on the Ethernet interface by running:
# dcbtool sc eno3 app:fcoe e:1 # dcbtool sc eno4 app:fcoe e:1
9. Bring up interfaces and start fcoe service:
# ifconfig eno3 up # ifconfig eno4 up
10. Start FCoE using:
# systemctl start fcoe
11. Start the data center bridging daemon (dcbd) using the following command:
# /etc/init.d/lldpad start
12. View configured FCoE devices:
# fcoeadm -i
After correctly configuring the Ethernet interface to use FCoE, set “fcoe” and “lldpad” to run at startup. To do so, use chkconfig in CentOS/RHEL 6.x, as in:
# chkconfig lldpad on # chkconfig fcoe on
Or in CentOS/RHEL 7.x:
# systemctl enable fcoe # systemctl enable lldpad