SerDes, Topology, and Compute Locality
April 30, 2023
Raw core throughput cannot compensate for an architecture that burns energy and time moving data between isolated islands. SerDes links, memory traffic, latency, topology, and programming overhead determine whether a massive computer actually scales.
Brian Greenforest argues for a return to meshes and lattices where local communication keeps computation close to the state it changes.
Count the Cost of Every Data Highway
A high-speed SerDes consumes power at each endpoint, adds encoding and clock recovery, and introduces distance into every exchange. Large memory writes and irregular dependencies can turn an accelerator’s arithmetic units into waiting machinery.
Island-style systems also force developers to partition programs around drivers, memory copies, kernels, and device boundaries. The software inherits the topology’s fragmentation.
Design the Fabric Around Locality
A mesh or lattice gives neighboring processes direct paths and lets pipelines advance work continuously. Repetition creates scale without routing every interaction through a distant memory or centralized switch.
Chip architects and HPC teams can quantify link energy, latency, utilization, and programming cost together. Build the topology that minimizes the total system burden instead of celebrating the hottest core.
LinkedIn status when archived: Visible to anyone on or off LinkedIn.
I struggle to fight against this trend of burning I/O and melting down the core. Have anyone ever care to quantify the ineffieciency of SerDeses, waste of energy, and introduced latencies? The massive compute topology went from meshes and lattices into these odd lonely islands connected with data highways. Writing scalable code for such architecture is not only a mess, and doesn't scale, but a waste of time and resources. Driver? Nvidia, of course, commoditizing V100 like if these were barrels of oil. We all know the oil companies still resist green energy change. And so Nvidia.
