What is a PCI-Express Lane?

Question

I am reading an article bemoaning i7-5820K Will Only Have 28 PCI-Express Lanes compared to its sibling processors having 40 lanes.

Isn't 28 lanes already too many? How many lanes would a normal home PC actually need and for what purposes?

I don't know how the following would be connected to PCIe, but do they even number 28?

2 HDs, 1 SSD, 1 CD-DVD-BR, card reader, printer, wifi or lan but seldom both, joystick, keyboard, mouse, graphics.

What else possibilities would need direct access to PCIe for a home/office PC? Or even a server.

Answer 1

Many devices use more than 1 lane.

For example - gaming graphics cards use 16 lanes. Some powerful gaming computers have two graphics cards - that's 32 PCIe lanes (two x16 ports).

Intel i7-5820K can't handle two x16 graphics cards. For some gaming enthusiasts or some engineers, that may be a serious problem. They may have to choose different CPU (maybe some Xeon) if they need more than 4 cores and two x16 graphics cards.

PCIe SSD drives use multiple PCIe lanes too (x4 or x8).

Many gigabit network adapters use PCIe x4, there are also 10-gigabit server adapters and they use PCIe x8.

28 lanes is not that much. If a motherboard manufacturer puts one x16 slot, one x8 slot and one x4 slot (x28 total) - you can use only 3 devices there and... that's all.

Here is an image from Wikipedia PCIe article. I added information about lanes on these PCIe slots.

You can read more in another answer written by reirab.

Answer 2

A PCIe 'lane' consists of 2 differential pairs of signals. One differential pair is used for sending and the other is used for receiving, which allows simultaneous bi-directional communication. Each lane is point-to-point. That is, each lane directly attaches a single host to a single device. PCIe switches can, however, be used when a host lane needs to be shared between multiple devices. Per Wikipedia, the bandwidth of a single PCIe lane (in each direction) is as follows:

• PCIe 1.x: 250 MB/s
• PCIe 2.x: 500 MB/s
• PCIe 3.0: 985 MB/s
• PCIe 4.0: 1969 MB/s
• PCIe 5.0: 3.9 GB/s

As Kamil said, most PCIe devices use multiple lanes. Some devices, such as NICs, sound cards, and other relatively low-bandwidth devices just use 1 lane. SSDs, RAID controllers, and other medium-bandwidth devices typically use 4 or 8 lanes. Graphics cards and other high-bandwidth devices (FPGAs, for instance) typically use 16 lanes. At system boot, the host and device will negotiate the number of lanes that will be used for a particular connection. Typically, the smaller of either the number of lanes that the card is wired for and the number of lanes that the slot it's installed in is wired for (i.e. the maximum physically possible) will be negotiated, though the number may be less in cases where so many PCIe devices are installed that the host does not have enough lanes to give each of them its maximum. The physical slots are designed such that devices with connectors for a smaller number of physical lanes will fit in and function properly in larger slots (e.g. a PCIe x4 card will fit in a PCIe x16 slot and will negotiate to run with 4 lanes.)

Also, some chipsets use some of the PCIe lanes to attach the Southbridge. This was how the Intel x58 chipset worked (the chipset for the Bloomfield chips, high-end of the first-generation Core i7 processors.) It used 4 lanes to attach the Southbridge, leaving 36 lanes for everything else. This typically was divided up as 2 16-lane links for graphics cards and 4 lanes for any other devices. Boards that supported 3 or 4 graphics cards would have to reduce some or all of the graphics cards to 8 lanes when 3 or 4 graphics boards were installed.

Having 2 graphics cards is very common in gaming systems and many gaming systems actually have 3 or 4 graphics cards. Even in a 2-card setup, at least one card will have to fall back to x8 mode in a system with only 28 lanes available. Additionally, systems that use graphics cards as computational accelerators often have 2-4 graphics cards installed. For these situations, only having 28 lanes is a problem, as that greatly limits the amount of host-to-device (and device-to-host) bandwidth available to each card. CUDA in particular has been gaining widespread popularity over the last several years, especially in the high-performance computing community. The PCIe bus can very easily become the bottleneck in GPGPU (General-Purpose computing on Graphics Processing Units) applications, so having as many lanes per card as possible is highly desirable in GPGPU systems.

Answer 3

A PCIe lane is a pair of high speed differential serial connections, one in each direction. A link between devices can be and often is made up of multiple lanes for higher data rates. The data rates of individual lanes also varies by generation, roughly speaking one lane of Gen x provides about the same data rate as two lanes of Gen x-1.

On modern Intel systems some PCIe lanes are provided by the CPU directly, while others are provided by the PCH in the chipset. The link from CPU to chipset is similar to PCIe but there are differences in the details.

Motherboard vendors have to decide how to allocate the lanes provided by the CPU and PCH to the onboard devices and slots. They can and often do include signal switches to give the user some options but there is a limit to how much signal switching can be affordably implemented.

Intels "mainstream desktop" platforms currently have 16 lanes from the CPU plus up to 24 (depending on which chipset is selected) from the chipset. However the lanes from the chipset are limited by the total bandwidth available from the CPU to the chipset (roughly equivalent to PCIe 3.0 x4 IIRC).

16 lanes from the CPU and 24 from the chipset more than enough for a normal desktop or small server, you can put your graphics card on the 16 lanes from the CPU and then the lanes from the chipset plus the integrated controllers in the chipset are generally enough for storage, networking etc. Even with two GPUs 8 lanes per GPU is enough most of the time.

However when building a high end system with 3+GPUs (or possiblly two top of the lines GPUs), lots of fast storage and/or very fast network interfaces more lanes are desirable. If you want to give each device it's maximum possible capacity you are looking at 16 lanes per GPU,

So for those with higher end needs Intel has a high end desktop socket, currently LGA2066. This socket also covers single socket workstation/server systems, though it seems officially at least you can't use the workstation/server processors in most desktop boards.

Unfortunately while with previous generations of high end desktop the number of PCIe lanes and ram channels was fixed, with LGA2066 the number varies by the processor you select. A desktop LGA2066 CPU can have 16, 28 or 44 PCIe lanes.

This puts motherboard vendors in a tricky position, they have to decide how they will handle giving the true high end customers the full functionality of their CPU while deciding what to disable or throttle for those with lower-end CPUs. System builders in turn must carefully read the manuals for the motherboards to find out what the limitations are before buying.

Grabbing the manual for one of the cheaper X299 boards https://dlcdnets.asus.com/pub/ASUS/mb/LGA2066/TUF_X299_MARK2/E12906_TUF_X299_MARK2_UM_WEB.pdf shows that the main limitation is the x16 slots, On a 44 lane CPU all three slots are usable with two running in x16 mode and one running in x8 mode. On the other hand on a 28 lane CPU you get one x16 one x8 and one unusable and on a 16 lane CPU you only get one x16 or two x8.

Grabbing the manual for a high end X299 board https://dlcdnets.asus.com/pub/ASUS/mb/LGA2066/ROG_RAMPAGE_VI_EXTREME_OMEGA/E15119_ROG_RAMPAGE_VI_EXTREME_OMEGA_UM_V2_WEB.pdf it seems they have decided not to support the 16 lane parts at all. This board does let you use three GPUs on a 28 lane CPU, but the second m.2 slot and the u.2 connector are only available with 44 lane CPUs