Introduction
This document provides technical guidance for NVMe over Fabrics (NVMe-oF), a protocol designed to extend the performance and efficiency of NVMe beyond local PCIe connections into a variety of network fabrics. By transporting NVMe commands over high-speed networks, NVMe-oF enables remote storage to perform as if it were locally attached, delivering ultra-low latency and scalable throughput that meets the demands of modern applications.
What is NVMe-oF #
NVMe-oF (Non-Volatile Memory Express over Fabrics) is an extension of the NVMe protocol that allows a host to access remote NVMe devices across a network fabric with the same efficiency as a direct PCIe connection. Unlike legacy storage protocols such as iSCSI or Fibre Channel carrying SCSI commands, NVMe-oF transmits native NVMe commands directly through supported transports like Fibre Channel (FC), RDMA-based Ethernet (RoCE), and TCP/IP.
By mapping NVMe queues onto the transport layer, NVMe-oF minimizes protocol overhead and achieves microsecond-level latency. This design allows enterprises to unlock the full potential of NVMe SSDs not only within a server, but across a distributed IT environment, making it an essential technology for next-generation infrastructures such as AI, virtualization, real-time analytics, and media production.
Figure – NVMe-oF Data Transfer Diagram
Key Features of NVMe-oF #
NVMe-oF introduces several critical features that differentiate it from traditional block storage protocols:
- Lower Latency and Higher IOPS
NVMe-oF reduces the command processing path by eliminating SCSI translation layers. Combined with parallel NVMe queues, this architecture delivers microsecond-level latency and millions of IOPS, significantly outperforming iSCSI-based storage. - Flexible Transport Options
NVMe-oF is transport-agnostic, supporting multiple network options to meet different infrastructure needs. Among these, NVMe/TCP and NVMe/RoCE are widely adopted and fully supported in QSAN storage solutions: - NVMe/TCP: Offers broad compatibility and simple deployment on standard Ethernet infrastructure without requiring special hardware or lossless networks. It delivers higher performance than iSCSI with lower cost and complexity, making it ideal for enterprises seeking rapid adoption.
- NVMe/RoCE: Leverages RDMA technology to bypass kernel and CPU overhead, providing ultra-low latency and efficient data transfers. It is especially suited for demanding workloads such as AI/ML training, HPC, and high-frequency databases where every microsecond matters.
INFORMATION:
When adopting an NVMe-oF environment, please check your storage, Ethernet, and the host configuration in terms of compatibility.
Theory of Operations
This chapter introduces the operational theory of NVMe-oF and explains how its transport mechanisms enable remote NVMe storage to deliver near-local performance.
Mechanism of NVMe-oF #
NVMe-oF enables communication between a host and a remote NVMe storage target by encapsulating NVMe commands within a chosen transport protocol. Unlike legacy block storage protocols such as iSCSI that translate SCSI commands over TCP/IP, NVMe-oF transmits NVMe commands natively across a fabric. This direct command mapping removes translation overhead and preserves the full efficiency of NVMe’s parallel queue architecture.
To break down the NVMe-oF architecture consists of:
- Host: The initiator that submits NVMe commands.
- Fabric (Transport Layer): The medium that encapsulates NVMe commands (e.g., TCP, and RDMA).
- Target: The NVMe storage device that processes commands as if they originated from a local PCIe bus, including the NVMe controller and NVMe Namespace.
Figure – Data I/O Task under NVMe-oF
This design ensures that applications interacting with NVMe-oF targets experience performance comparable to locally attached NVMe SSDs, while retaining the scalability and flexibility of a networked solution.
Transport Options #
NVMe-oF is designed to be transport-agnostic, offering multiple options to meet different infrastructure requirements. Each transport has unique trade-offs in terms of performance, deployment complexity, and hardware needs.
NVMe/TCP #
NVMe/TCP encapsulates NVMe commands within standard TCP/IP packets, allowing organizations to deploy NVMe-oF on their existing Ethernet infrastructure without requiring specialized hardware. This broad compatibility and ease of deployment make it an attractive choice for enterprises seeking cost-effective NVMe adoption. Compared with iSCSI, NVMe/TCP reduces protocol overhead and delivers significantly higher performance while maintaining simple scalability.
NVMe/FC #
NVMe over Fibre Channel (NVMe/FC) enables high-performance NVMe storage access through established Fibre Channel SAN infrastructure. By leveraging dedicated storage networks, it delivers predictable low latency, consistent throughput, and reliable data transmission. NVMe/FC also allows organizations to maximize existing Fibre Channel investments while transitioning from traditional SCSI-based storage to NVMe, making it well-suited for mission-critical enterprise databases, virtualization, and latency-sensitive workloads.
NVMe/RoCE #
RoCE (RDMA over Converged Ethernet) is designed to achieve the lowest possible latency by bypassing the traditional TCP/IP stack. By enabling direct memory access between nodes, it reduces CPU utilization and improves throughput, making it ideal for latency-sensitive applications such as AI/ML training, real-time financial trading, HPC clusters, and high-performance databases.
NVMe-oF Configuration
The following are tutorials for configuring NVMe-oF environments. Follow the steps and you can learn them quickly.
Setup Tutorial
Performance Test Results
This test verifies the dramatic performance improvement offered by NVMe-oF. We test the the NVMe-oF in TCP and RoCE environment. Please check the following test result for a clear vision of the benefit from NVMe-oF.
Test Equipment and Configurations
Server
Model: GIGABYTE R181-342
Memory: 64 GB
iSCSI HBA: Mellanox ConnectX-4 Lx
OS: Ubuntu 22.02Equipment
Storage
Model: XN4226D-8C
Memory: 32 GB per controller
Host Connection: iSCSI: HCQ25GS3-40
Firmware: QSM 4.1.6
SSD: QSAN SD43T840-00 3.84 TB * 26
NVMe/TCP Test Result #
The first test item is the widely adopted TCP environment data transfer. The NVMe/TCP test showed a maximum 95% performance gain, with a notable improvement in read-intensive workloads.
Table – XN4226D NVMe/TCP Test Matrix
| Test Item | NVMe/TCP | iSCSI | Improvement |
| 4K Random Read (IOPS) | 916,059 | 551,722 | 66% |
| 4K Random Write (IOPS) | 485,000 | 401,957 | 21% |
| 128K Sequential Read (MB/s) | 12,626 | 6,481 | 95% |
| 128K Sequential Write (MB/s) | 5,817 | 4,821 | 21% |
NVMe/RoCE Test Configuration #
The next test evaluates the NVMe/RoCE data transfer method for the all-flash model in RDMA environment. The most significant improvement was observed in 128K sequential read throughput, which increased by 82% with in NVMe-oF protocol, demonstrating its advantage in accelerating large block sequential transfers.
Table – XN4226D NVMe/RoCE Protocol Test Matrix
| Test Item | NVMe/RoCE | iSCSI over RDMA | Improvement |
| 4K Random Read (IOPS) | 967,357 | 591,306 | 64% |
| 4K Random Write (IOPS) | 521,900 | 445,308 | 17% |
| 128K Sequential Read (MB/s) | 14,094 | 7,764 | 82% |
| 128K Sequential Write (MB/s) | 9,049 | 7,721 | 17% |
Conclusion
substantial performance improvements compared to traditional block storage protocols, with clear benefits across random and sequential workloads. NVMe/RoCE excels in environments requiring ultra-low latency and maximum throughput, showing up to 82% gain in sequential read performance. On the other hand, NVMe/TCP offers remarkable scalability and ease of deployment while still delivering up to 95% performance improvement in sequential reads over legacy approaches.
By supporting both transports, QSAN enables users to select the optimal balance between performance, deployment simplicity, and infrastructure investment. This flexibility ensures enterprises are well-prepared to meet the demands of next-generation workloads, from virtualization and media production to AI-driven analytics and beyond.
