NVLink is a wire-based serial, multi-lane, near-range, communications link developed by Nvidia. Unlike PCI Express, a device can consist of multiple NVLinks, and devices can use mesh networking to communicate instead of a central hub/switch. The protocol was first announced in March 2014 and uses a proprietary high-speed signaling interconnect (NVHS).
NVLink is a wire-based serial, multi-lane, near-range, communications link developed by Nvidia. Unlike PCI Express, a device can consist of multiple NVLinks, and devices can use mesh networking to communicate instead of a central hub/switch. The protocol was first announced in March 2014 and uses a proprietary high-speed signaling interconnect (NVHS).
For small numbers of GPUs, the NVLink lanes on a single device are sufficient for an all-to-all mesh connectivity. To accommodate higher GPU counts, NVLink since 2018 use a packet-switched architecture, where a central switch can serve up to 32 two-lane ports. The NVSwitch for NVLink 4.0 can produce some simple computation of its own (e.g., sum, broadcast) to reduce the need for communication thanks to the "SHARP" accelerator.
Discovered by embedding cosine similarity (sentence-transformers MiniLM, 384-dim).