20 KiB
Memory Bandwidth
Scope
This document investigates memory bandwidth requirements, limits, and implementation strategies for the XH-1 processor, a proposed 128-core RISC-V design. The discussion covers off-chip DRAM bandwidth, on-chip interconnect bandwidth, cache hierarchy bandwidth, and the interactions between bandwidth, coherence traffic, and the memory system.
This document is part of the XH-1 research series (area 05-memory). Statements about XH-1 implementation are proposals unless explicitly marked as decided.
1. Problem Statement
A 128-core processor generates substantially more memory traffic than single-core or small-count-core designs. Even if each core issues a modest number of outstanding memory requests, the aggregate demand can quickly exceed the bandwidth available from a realistic DRAM subsystem.
The XH-1 design must therefore answer several related questions:
- How much DRAM bandwidth is physically available on the target package?
- How much of that bandwidth is consumed by coherence traffic?
- How much remains for useful application work?
- How is that remaining bandwidth shared between cores without starvation or severe unfairness?
- Where do bandwidth bottlenecks arise in the on-chip interconnect, cache hierarchy, and memory controllers?
The remainder of this document analyses each of these questions.
2. Background
2.1 DRAM Bandwidth Fundamentals
Modern DDR systems provide bandwidth through wide channels and high transfer rates:
- DDR5 devices use 32-bit sub-channels per module rank, with transfer rates up to 8400 MT/s in JEDEC-published modules.
- A single DDR5 channel (64 bits) at 6400 MT/s delivers approximately 51.2 GB/s peak; a dual-channel configuration roughly doubles this.
- HBM3 stacks deliver substantially higher bandwidth per package (around 819 GB/s for a 6-high stack per HBM3 specification).
- HBM3E extends this further, with published per-stack figures in the 1 TB/s range for selected devices.
The XH-1 design has not fixed a memory technology. Both DDR5 and HBM3-class options are considered below as proposals.
2.2 RISC-V Memory Model
RISC-V defines a weak memory ordering model, the RISC-V Weak Memory Ordering (RVWMO), in the architecture specification. RVWMO permits a wide range of reorderings and is similar in spirit to other modern weakly ordered architectures.
Key consequences for bandwidth analysis:
- Loads may bypass earlier stores under default rules when addresses differ.
- Coherence is required only at a single total order per address (the "coherence order" axiom).
- Fence instructions (
fence,fence.i,fence.vma) are required to constrain ordering where software demands it.
A weakly ordered model does not directly change bandwidth demand but influences how much buffering, speculation, and replay logic is required to expose bandwidth to software.
2.3 Bandwidth vs. Latency
Bandwidth and latency are distinct metrics. A system can have ample aggregate DRAM bandwidth but very high per-request latency, in which case small working sets run well but large working sets suffer. A 128-core system is especially sensitive to the latency component because aggregate concurrency exposes the latency of every shared resource, including DRAM banks, the on-chip interconnect, and the directory or snoop filter.
3. Bandwidth Demand Estimation
3.1 Per-Core Demand
A modern out-of-order core can issue multiple outstanding cache misses. The exact number depends on the pipeline width, miss status holding register (MSHR) count, and prefetcher effectiveness, but published industrial designs commonly support 10-32 outstanding L1 misses per core.
If each outstanding request ultimately consumes 64 bytes from DRAM, and a core sustains 10 outstanding misses, the per-core demand is:
10 misses * 64 B / core / (memory round-trip latency)
For a 200 ns round-trip DRAM latency, this is roughly:
10 * 64 B / 200 ns = 3.2 GB/s per core
If all 128 cores simultaneously sustain this rate, the aggregate is approximately 410 GB/s, which already exceeds what a single DDR5 channel can deliver.
This calculation is intentionally rough. It is intended to motivate the rest of the document, not to set a precise requirement.
3.2 Realistic Demand
In practice, applications rarely sustain peak miss rates on every core simultaneously. Published studies of throughput-oriented server workloads report average L2 MPKI (misses per thousand instructions) in the range of 5-30 for memory-bound kernels and 1-5 for cache-friendly workloads. These values vary significantly across workloads.
Multiplying through, the worst-case aggregate bandwidth demand of a 128-core system can plausibly reach hundreds of GB/s. The XH-1 design must therefore plan for a memory subsystem substantially more capable than a single DDR5 channel.
3.3 Coherence Traffic Overhead
Cache coherence protocols add traffic on top of application reads and writes. The magnitude depends on the protocol (see Section 6) and the sharing pattern of the workload. A directory-based protocol typically generates at least one directory lookup per coherence transaction and potentially one or more forward/intervention messages.
A conservative estimate is that coherence and protocol overhead add 10-30% to the raw application demand, but this can rise significantly for workloads with intensive cross-core sharing.
4. On-Chip Bandwidth
DRAM bandwidth is only one part of the system. The on-chip interconnect and cache hierarchy must also carry:
- Coherence messages between cores and directories.
- Cache line fills from memory controllers to L2/L3 slices.
- Write-backs from L1 to L2.
- Snoop responses and invalidation acknowledgements.
- Interrupt and configuration traffic (typically negligible).
For a 128-core mesh-class interconnect, published industrial designs indicate bisection bandwidth targets in the multi-TB/s range. Whether XH-1 reaches this range depends on link width, clock frequency, and topology choices that are not yet decided.
5. Interconnect Topology and Bandwidth
The interconnect determines how effectively aggregate on-chip bandwidth is delivered to the DRAM controllers. Three common topologies for many-core designs are:
- Crossbar
- Ring
- Mesh
- Concentrated mesh (cmesh)
| Topology | Typical use | Bandwidth scaling | Notes |
|---|---|---|---|
| Crossbar | small core counts | linear in core count | poor scalability beyond ~16 cores |
| Ring | mid counts | limited by ring hops | simple but latency grows |
| Mesh | mid-to-high counts | roughly linear with links | common in tiled designs |
| Cmesh | high counts | similar to mesh, fewer routers | reduces router count |
A crossbar at 128 cores is generally impractical due to area and wire count. A ring at 128 cores suffers from long hop counts and limited bisection bandwidth. A mesh or concentrated mesh is the most commonly chosen topology in published academic and industrial 128-core class designs.
The XH-1 topology is not yet decided; both a 2D mesh and a 2D concentrated mesh are candidate proposals.
6. Coherence Protocol Bandwidth
6.1 Protocol Options
Two main families of coherence protocol exist:
- Snooping protocols (broadcast-based).
- Directory protocols (tracking-based).
At 128 cores, broadcast-based snooping is generally considered impractical because every coherence transaction must reach every core, consuming O(N) bandwidth per request. A directory protocol incurs O(1) directory lookups per request in the common case but requires directory storage and handles sharing patterns differently.
A directory is therefore the expected XH-1 proposal, consistent with published 64-128 core research prototypes.
6.2 Directory Organization
Directories can be implemented in several ways:
- Full map directory: tracks sharers exactly. Storage cost is O(N) per line; prohibitive at 128 cores without compression or hierarchical schemes.
- Limited pointer directory: tracks a fixed number of sharers per line (for example 4-6 sharers). When the count is exceeded, a fallback representation (often a "broadcast" or "coarse vector" marker) is used.
- Coarse vector directory: groups cores into regions (e.g. 8-core tiles) and tracks presence at region granularity. Lower storage cost but less precise; can produce additional coherence traffic.
- Sparse directory: similar to coarse vector but with different trade-offs.
A limited pointer directory is the most commonly used scheme in published industrial designs in this core-count range. The XH-1 design is expected to use a limited pointer directory, possibly augmented with a coarse region fallback, but this is a proposal rather than a decision.
6.3 Coherence Bandwidth Cost
For a limited pointer directory with P pointers, the common case is one directory lookup per coherence transaction. Bandwidth cost scales with the number of cores only when the directory overflows into broadcast mode.
Published research indicates that overflow frequency is workload dependent. For read-mostly shared data, pointer counts typically stay small. For write-shared data, a limited pointer directory can overflow frequently, increasing coherence traffic.
7. Cache Hierarchy Bandwidth
The cache hierarchy must sustain:
- L1 to L2 traffic (fills, write-backs, invalidations).
- L2 to L3 traffic.
- L3 to DRAM traffic.
- Snoop and probe traffic at every level.
Each cache level adds banked storage and multiple read/write ports to deliver bandwidth. The area and energy cost of these ports grows nonlinearly with port count.
For XH-1, a three-level hierarchy is the expected proposal:
- Private L1 per core.
- Private L2 per core (or per core pair) with a coherence controller.
- Shared L3 distributed across tiles.
The exact partition of L2 vs. L3 capacity, associativity, and bandwidth is a design question left open in this document.
8. Bandwidth Allocation and QoS
When 128 cores compete for limited DRAM bandwidth, some form of allocation policy is required. Options include:
- Strict priority ordering.
- Weighted fair queuing.
- Per-core bandwidth caps.
- Per-class (e.g. memory-mapped I/O vs. CPU) priorities.
Without allocation policy, a single core running a streaming workload can saturate the memory subsystem and starve other cores. This is a documented failure mode in many-core research literature.
RISC-V itself does not define memory bandwidth allocation semantics; this is an XH-1 implementation choice. It is expected that the XH-1 memory controllers expose some form of bandwidth partitioning, but this is a proposal rather than a decision.
9. Memory Controller Architecture
The memory controller is the point at which on-chip bandwidth meets off-chip DRAM. Key design choices include:
- Number of independent channels.
- Channel-to-core mapping.
- Scheduling policy (FR-FCFS, parallelism-aware batch scheduling, etc.).
- Address mapping policy.
- Refresh and power management handling.
For DDR5-class memory, a 4-8 channel configuration is a reasonable starting point for a 128-core design, but this is workload dependent and is presented as a proposal.
For HBM3-class memory, the channel count is fixed by the stack configuration; bandwidth is delivered through many independent pseudo-channels per stack.
10. Quantitative Summary
The following table summarises representative bandwidth figures from published sources. These are not XH-1 targets; they are external reference points.
| Technology / Configuration | Peak Bandwidth | Source basis |
|---|---|---|
| DDR5 single channel, 6400 MT/s | ~51.2 GB/s | JEDEC DDR5 spec |
| DDR5 dual channel, 6400 MT/s | ~102.4 GB/s | JEDEC DDR5 spec |
| HBM3 6-high stack | ~819 GB/s | JEDEC HBM3 spec |
| HBM3E 8-high stack | ~1 TB/s class | vendor published figures |
| 128-core aggregate theoretical (10 outstanding misses/core) | ~410 GB/s | rough calculation, Section 3.1 |
11. Advantages and Disadvantages of Candidate Memory Systems
| Option | Advantages | Disadvantages |
|---|---|---|
| DDR5 multi-channel | Standard, widely available, lower cost per GB | Lower peak bandwidth per channel, more channels increase board complexity |
| HBM3 / HBM3E | Very high bandwidth, high channel count per stack | Higher cost, 2.5D/3D packaging required, lower capacity per stack |
| Hybrid (HBM + DDR) | Combine high bandwidth with high capacity | Complex memory hierarchy, two coherence domains |
The XH-1 memory technology choice is not yet decided.
12. Scalability Problems at 128 Cores
The following scalability problems are well documented in the many-core literature and are directly relevant to XH-1:
- DRAM bandwidth saturation: aggregate demand can exceed realistic DRAM bandwidth.
- Directory overflow: limited pointer directories overflow more often as core count rises, increasing coherence traffic.
- Coherence traffic growth: as more cores share data, coherence messages grow superlinearly in some sharing patterns.
- Interconnect bisection bottlenecks: insufficient bisection bandwidth creates hotspots at memory controllers.
- Quality-of-service degradation: without allocation policy, fairness collapses as core count rises.
- Verification cost: coherence state space grows roughly combinatorially with core count.
These are not unique to XH-1 but apply to any 128-core coherent design.
13. Interactions With Other Subsystems
13.1 Pipeline
A deeper pipeline can support higher clock frequencies but increases the penalty of cache misses. Bandwidth-bound workloads become sensitive to the round-trip latency the pipeline exposes. The pipeline must include enough MSHRs to expose memory-level parallelism.
13.2 Cache Hierarchy
Cache capacity and associativity directly affect miss rate and hence DRAM bandwidth demand. Larger caches reduce demand at the cost of area and access time. Bandwidth and capacity are coupled; both must be planned together.
13.3 Memory System
The memory controller scheduling policy, address mapping, and refresh handling all affect effective bandwidth. Closed-page vs. open-page policies, bank parallelism, and request batching materially change delivered bandwidth.
13.4 Interconnect
The interconnect must deliver at least as much aggregate bandwidth as the DRAM subsystem, plus coherence overhead. Otherwise the interconnect becomes the bottleneck and DRAM is underutilised.
13.5 Coherence
Coherence traffic is a first-class component of bandwidth demand. Protocol choice and directory organisation directly determine the share of bandwidth consumed by coherence.
13.6 Interrupts
Interrupts are typically not bandwidth-limited, but interrupt delivery traffic and any interrupt-cause queues add minor traffic to the system fabric.
13.7 Operating System
The OS schedules processes across cores, controls page placement, and may implement memory allocation policies that affect locality. The OS does not directly set DRAM bandwidth but its page colour and NUMA-style placement policies interact with the memory controllers.
13.8 Verification
Coherence protocols at 128 cores have a large state space. Verification complexity grows roughly with the product of per-core state and the number of cores. Formal methods, randomised testing, and traffic pattern fuzzing are all relevant.
13.9 Performance
Bandwidth is one of the dominant performance constraints for memory-bound workloads. Performance counters for DRAM channel utilisation, interconnect link utilisation, and coherence message counts are essential for tuning.
14. RISC-V Architectural Requirements
The following RISC-V architectural elements are relevant to bandwidth analysis:
- The RISC-V ISA specifies
fence,fence.i, andfence.vmainstructions that order memory operations. - The privileged architecture defines physical memory attributes (PMA) and, in designs with translation, page-based attributes (PBMT, NAPOT, etc.) that can affect memory behaviour.
- The RISC-V coherency management extensions (for example, the
proposed H-extension in the vector and hypervisor contexts, and
the cache-management operations in the RISC-V Profiles and
IOMMU-related specifications) provide instructions such as
cbo.clean,cbo.flush, andcbo.invalthat interact with cache bandwidth.
RISC-V itself does not mandate a specific coherence protocol or memory bandwidth management policy. These are XH-1 implementation choices.
15. Open Questions and Unresolved Design Decisions
The following items are not yet decided and require further research or design effort:
- Choice of DRAM technology (DDR5 multi-channel vs. HBM3-class).
- Number and placement of memory controllers.
- Interconnect topology (mesh vs. concentrated mesh, dimensional order, link width).
- Coherence protocol variant (e.g. MESI vs. MOESI and the directory structure).
- Bandwidth allocation policy and per-core fairness guarantees.
- L2 vs. L3 capacity split and per-tile bandwidth targets.
- Quality-of-service hooks exposed to the operating system.
- Verification strategy for the coherence and interconnect at 128 cores.
These are recorded as open questions; no recommendation is made where the available evidence does not strongly favour a particular choice.
16. Recommendations
Recommendations are made only where published evidence and engineering practice converge.
- Use a directory-based coherence protocol: published 64-128 core research and industrial designs consistently use directories; a snoop-based approach is not credible at 128 cores.
- Plan for multi-channel DRAM: a single DDR5 channel is unlikely to meet aggregate demand; at least 2-4 channels are a reasonable starting point if DDR5 is used, with HBM as an alternative if higher bandwidth is required.
- Provide per-class quality-of-service in the memory controllers: without allocation, fairness is not achievable in a 128-core system.
- Instrument the design with bandwidth and coherence performance counters: empirical tuning is unavoidable.
- Verify coherence state space with a combination of formal methods and randomised testing: full state space enumeration is not feasible at 128 cores.
These are recommendations, not decisions; they are subject to revision as the XH-1 project progresses.
17. References
- RISC-V International, The RISC-V Instruction Set Manual, Volume I: Unprivileged ISA, current ratified version. RVWMO is described in Volume I.
- RISC-V International, The RISC-V Instruction Set Manual, Volume II: Privileged Architecture, current ratified version.
- JEDEC, DDR5 SDRAM Standard (JESD79-5), published release.
- JEDEC, High Bandwidth Memory DRAM (HBM3), JESD238A.
- JEDEC, High Bandwidth Memory DRAM (HBM3E), JESD238B-1 (where applicable).
- Hennessy and Patterson, Computer Architecture: A Quantitative Approach, recent edition, for general memory system modelling and AMAT analysis.
- Sorin, Hill, and Wood, A Primer on Memory Consistency and Cache Coherence, for coherence protocol analysis.
- Martin, Hill, and Wood, and successor work, on many-core coherence verification and token coherence (where relevant to XH-1 verification work).
Where specific figures in this document depend on vendor or JEDEC-published numbers, the source is indicated in-line. Where figures are derived from calculations, this is stated explicitly.
18. Document Status
This document is a research proposal and analysis. It is not a description of an implemented design. Items marked as "proposal" or "expected" have not been formally decided within the XH-1 project. Items marked as "open" represent unresolved design questions. The document should be re-reviewed as the XH-1 architecture specification stabilises.