Rechenzentrum

QSFP28 breakout: how to turn one 100G port into four 25G links

One 100G port, four 25G servers or switches? When to pick a breakout DAC or AOC, when SR4 or PSM4 with an MPO fanout, and what to configure first.

5 Min. LesezeitVon Yvan Watchman

One 100G port, four devices at 25G. QSFP28 (four lanes of 25G) naar 25G, 25G, 25G, 25G. Works: breakout DAC, breakout AOC, SR4 or PSM4 with an MPO fanout. Doesn't: LR4 and CWDM4. Breakout mode on the switch first, then FEC per leg.

You have a spare 100G port and four servers or leaf switches with 25G NICs that need connecting. Can one port do that, and if so, with which cable or optics? It can, with a breakout. This post covers the three ways to do it, which optics can and can't be split, what to set on the switch, and what to tell us when you order.

What a breakout actually does

A QSFP28 port (SFF-8665) is electrically four 25G lanes side by side. As a 100G port, the switch bundles those lanes into one interface. In breakout mode it lets go of them: each lane becomes its own 25G port with its own interface name, its own configuration and its own link state. At the far end, each lane lands on an SFP28 (SFF-8402): one 25G lane.

40G works the same way: QSFP+ is four times 10G. The cable or optic just gives each lane its own physical exit, and that only works if every lane really has its own path.

Three routes, and the optics that drop out

  • Breakout DAC — copper, with a QSFP28 on one end and four SFP28s on the other. Vendors sell them from 1 to 5 m: within the rack or to the next one.
  • Breakout AOC — the same idea with fiber and fixed optics on the ends. Thinner and longer: NVIDIA makes them from 3 to 30 m, for example.
  • Parallel optics with an MPO fanout — a 100G SR4 (multimode, 70 m on OM3 and 100 m on OM4) or PSM4 (single-mode, 500 m) in the switch, a fanout from MPO to four duplex LC legs, and a separate 25G optic on each leg. For longer distances or existing structured cabling.

What can't be split: LR4 and CWDM4. They put the four lanes on four wavelengths and send them together over one duplex pair. There is no fiber per lane, so there is nothing to fan out. Juniper accordingly lists its 100G LR4 as not breakout-capable, and SR4 and PSM4 as capable. The same goes for a single-lane 100G DR: that is one 100G lane, not four 25G ones. A 400G DR4 can break out into four 100G DR links, but 400G is on quote with us.

Even parallel optics aren't a given: Cisco's QSFP-40G-SR4 can run as four 10G SR links, the almost identically named QSFP-40G-SR4-S can't. Check the datasheet.

Configure the switch first, then plug in the cable

A QSFP28 port runs as one 100G interface by default. Plug in a breakout cable without changing the config and not a single 25G link comes up. You put the port into breakout mode explicitly, and how depends on the platform and software release.

Mind the port groups. On some switches not every port can break out, and splitting one costs you the ports next to it. On most ports of Juniper's PTX10002-60MR, 4x 25G is only available on every fourth port, and the three ports after it must stay unused. Some Cisco Nexus models only break out on specific ports, and some need a reload.

  • Look up in your switch datasheet or configuration guide which ports support breakout, and in which modes.
  • Check which neighbouring ports go out of service when you split a port.
  • Schedule the change in a maintenance window if your platform needs a reload or a port re-initialisation.
  • Set the same FEC on each 25G leg as on the device at the other end.

FEC per leg, and why length matters

After the breakout, every leg is its own 25G link with its own FEC setting. Both ends have to run the same FEC or the link won't work properly; Cisco even wrote a blog post called "Don't Mix Up Your FECs".

A 25GBASE-SR optic requires RS-FEC. With DACs, the cable class decides: short cables (CA-25G-N) can run without FEC, longer ones need BASE-R FEC or RS-FEC. In NVIDIA's breakout DAC range, cables up to 2 m are CA-25G-N and the 3.5 and 5 m ones are CA-25G-L, with RS-FEC; 2.5 and 3 m come in both classes. So don't order a longer cable "just to be safe": it comes with a FEC requirement that the server NIC has to support too.

Coding: two ends, two platforms

A breakout DAC has five EEPROMs: one in the QSFP28 head and one in each SFP28. On a breakout AOC, each end has its own management interface too. Each host only reads its own end. Your switch reads the QSFP28, the devices at the far end read the SFP28s. If there are different brands on each side, the coding has to be right on both ends.

So always tell us both platforms: make and model of the switch on the 100G side, and what sits on the 25G side. We'll check what is possible for that combination, so you don't find out during the install.

How to order it

Breakout cables aren't in our webshop at the moment, and the QSFP28 optics that are (LR4, ER4, ZR4 and DR) are all duplex optics that don't break out. MPO cabling, SR4 and PSM4 are on quote.

What is in the shop sits under DAC cables, AOC cables, 25G SFP28 and 100G QSFP28. For a breakout, send us your request with both platforms, the distances and the number of legs, and we'll send you a quote. Installation and testing go through our services.

Yvan Watchman

Über den Autor

Yvan Watchman

Gründer von FiberNxt und Xyphen IT

Yvan Watchman ist Gründer von FiberNxt und Xyphen IT. Er baut und betreibt seit Jahren Netzwerke und Rechenzentrumsinfrastruktur in und um Amsterdam.

LinkedIn-Profil

Passende Produkte und Dienstleistungen

Häufig gestellte Fragen

Can I split a 100G LR4 or CWDM4 into four 25G links?

No. LR4 and CWDM4 put the four lanes on four wavelengths over a single duplex pair. There is no separate fiber per lane to fan out. For breakout you need parallel optics such as SR4 or PSM4, or a breakout DAC or AOC.

Do I have to configure the switch for breakout?

Yes. A QSFP28 port runs as a single 100G interface by default. You put it into breakout mode per port, and on some platforms that takes neighbouring ports out of service or needs a reload. Check the documentation for your switch and software release.

Does the same apply to 40G into 4x 10G?

Yes, with the same rules. A QSFP+ port has four 10G lanes. A breakout DAC or AOC to four SFP+ ends works, and so does a 40G SR4 with an MPO fanout. A 40G LR4 does not, and not every SR4 variant supports breakout either, so check the datasheet.

Weiterlesen