How to enable NVMe/FC BFS full-offload with Emulex lpfc



First of all we have to install RHEL on the remote nvme device; considering
that at the moment the RHEL installer is unable to discover an NVMe/FC target,
we will have to use qemu to carry out the installation on the target device.

  • Start the installation by using the following command:

    #qemu-system-x86_64 --enable-kvm -bios OVMF-pure-efi.fd -m 4G
    -drive file=/dev/nvme0n1,if=none,id=NVME1 -device nvme,drive=NVME1,serial=nvme-1
    -cdrom RHEL-8.5.iso -net user -net nic -boot d

    IMPORTANT: select "Standard Partitioning", not "LVM" when creating the disk partitions
    You can download the bios image here

  • When the installation is complete, restart the system again with qemu and fix the hostnqn and hostid files
    in the /etc/nvme/ directory so the system will be able to discover the remote target.
    Create a discovery.conf file in the /etc/nvme/ directory that nvme-cli can use to discover the target at boot.
    Example:

    • #cat /etc/nvme/discovery.conf
    --transport=fc --traddr=nn-0x200200a098d851c8:pn-0x202200a098d851c8 --host-traddr=nn-0x200000109bc1e4e7:pn-0x100000109bc1e4e7
    --transport=fc --traddr=nn-0x200200a098d851c8:pn-0x202300a098d851c8 --host-traddr=nn-0x200000109bc1e4e8:pn-0x100000109bc1e4e8
    --transport=fc --traddr=nn-0x200200a098d851c8:pn-0x201300a098d851c8 --host-traddr=nn-0x200000109bc1e4e8:pn-0x100000109bc1e4e8
    --transport=fc --traddr=nn-0x200200a098d851c8:pn-0x201200a098d851c8 --host-traddr=nn-0x200000109bc1e4e7:pn-0x100000109bc1e4e7
  • Apply this patch to the /lib/dracut/modules.d/95nvmf/parse-nvmf-boot-connections.sh file
    (this is only necessary for RHEL8)

  • Update the initramfs, forcing dracut to install the nvmf module and the lpfc driver:

    #dracut -f -v --add nvmf --add-driver lpfc

  • Shutdown the qemu virtual machine, reboot the server and enter the BIOS settings

  • Enter the "Device Settings" menu in the BIOS and select the Fibre Channel
    Adapter you want to use for booting.




  • Set "Boot from SAN" to "Enable"




  • Enter the "Emulex NVMe over FC Boot Settings" menu, make sure the
    Initiator NQN is allowed to connect to the target and enable the NVMe over FC Boot.
    You may have to reboot the machine to apply the changes.




  • Select "Scan for NVMe over FC Devices" and check if the controllers have been discovered




  • Go back to the previous menu and select "Add NVMe over FC Boot Devices",
    select the device you want to boot from and then select "Commit Changes".




  • Return to the main menu and enter the "System BIOS" settings, then enter the
    the "Boot Settings" menu




  • Enable the "UEFI Boot Mode", you may have to reboot the server for the change to be applied




  • Enter again the "Boot Settings" and move the Fibre Channel Adapter on top
    of the Boot Sequence list




  • Reboot the system, the Fibre Channel Adapter should now connect to the target and load the GRUB bootloader: