Navigating the Complexity of Modern Chip Interconnects: NoCs, Chiplets, and the Future of Data Flow

The semiconductor industry is facing escalating demands for performance and efficiency in high-performance computing and artificial intelligence. As traditional monolithic chips reach their scaling limits, a shift towards multi-die architectures using chiplets is gaining momentum. This transition necessitates sophisticated on-chip network (NoC) designs capable of managing increasingly complex data flows. This article explores the challenges and innovations in NoC technology, particularly in the context of chiplet-based systems, and how designers are adapting to meet these evolving needs.

The Growing Complexity of Data Management

Modern on-chip networks are akin to intricate freeway systems, handling a massive influx of data generated and analyzed by today’s processors and memory. Real-time AI analytics place significant pressure on these networks, requiring rapid data transfer between processing units and memory. This demand is driving innovation in NoC topologies and inter-die fabrics to optimize data delivery and timing. As Priyank Shukla, director of product management for interface IP at Synopsys, notes, “This space is growing since we have a lot of data,” leading to new technologies for connecting data in and out of accelerators and to memory [1].

Challenges Across Different Architectures

The architectural challenges associated with NoC design vary depending on whether the focus is on Systems-on-Chip (SoCs), multi-die systems, or chiplets. However, some common concerns persist across all configurations. William Wang, CEO of ChipAgents, highlights key challenges: “scalability, congestion management, traffic fairness, latency predictability, and achieving timing closure across increasingly heterogeneous IP blocks.” [1]

SoC Fabrics: A Scaling Challenge

In SoCs, the complexity scales with size. Andy Nightingale, vice president of product management and marketing at Arteris, explains that as SoCs grow to encompass thousands of endpoints, managing traffic becomes a critical task. “Wiring congestion, timing closure, and performance are inseparable from topology and placement,” he states. [1] Heterogeneous integration—combining CPUs, GPUs, NPUs, and accelerators—further complicates matters, impacting clocking, power consumption, and protocol domains.

AI Designs: Burst Traffic and Congestion

AI designs introduce unique challenges related to bursty, high-fan-in traffic. Nightingale emphasizes that modern NoCs must evolve beyond simple crossbars or rings to handle this type of load effectively. The fabric needs to be designed as a scalable system, not merely as “glue logic” added after the core IP is finalized. [1]

Heterogeneity and Application-Specific Design

Due to limitations in silicon scaling, engineers are increasingly customizing network solutions for specific applications rather than pursuing general-purpose designs. Kent Orthner, principal solutions architect at Baya Systems, points out that heterogeneity aids this process. “You’re coming up with different types of processors, and different types of compute, and different types of networks and topologies, maybe all within a single SoC to solve different flavors of problems.” [1]

Evolving NoC Topologies

To meet changing data needs, NoC topologies have evolved from basic structures like crossbars and stars to more complex formations such as ringtrees, meshes, and Torus networks. Complex systems often employ multiple NoCs with varying topologies. ChipAgents’ Wang notes the trend towards hybrid fabrics mixing mesh, Torus, and hierarchical clusters with configurable coherency islands and adaptive routing to balance bandwidth and power. [1] Future fabrics are expected to be dynamic and self-optimizing, with agent-driven traffic tuning and runtime topology morphing based on workload patterns.

Arteris envisions fabrics that “disappear as a problem,” becoming transparent to the user even as chip size continues to increase. [1]

Algorithmic and Software-Defined Approaches

Saurabh Gayen, chief solutions architect at Baya Systems, highlights the emergence of algorithmic, software-based hardware design, enabling the creation of fundamentally new topologies. This approach involves a holistic system view combined with a bottom-up component-level understanding. [1]

Chiplet-Specific Considerations

Chiplets introduce unique challenges related to data management and interconnect. Cadence’s Mick Posner explains that within a chiplet, subsystems communicate via their own NoCs, with a higher-level NoC facilitating inter-subsystem communication. The need for cache coherency depends on the type of chiplets being connected. CPU chiplets require coherent NoCs for performance scaling, while connections between CPUs and accelerators often only require I/O coherency. [1]

Hee Soo Lee, high-speed digital design segment lead at Keysight EDA, emphasizes the importance of maintaining bandwidth and minimizing latency between chiplets, as well as ensuring signal integrity across the numerous tiny connections. [1]

Conclusion

The design of NoCs and interconnect fabrics is becoming increasingly critical as the semiconductor industry moves towards multi-die architectures and tackles the demands of AI workloads. Addressing challenges related to scalability, congestion, and heterogeneity requires innovative topologies, application-specific designs, and a shift towards more dynamic and adaptable systems. As AI evolves and integrates more closely with the physical world, ensuring deterministic, observable, and adaptable data movement will be paramount for achieving reliable and scalable performance. Efficiency will follow discipline in the design and management of these complex interconnects.