xh1-research-fork/research/.xh1/runs/20260826T131926Z/candidate.md

# Memory Bandwidth

## Scope

This document investigates memory bandwidth requirements, limits, and
implementation strategies for the XH-1 processor, a proposed 128-core
RISC-V design. The discussion covers off-chip DRAM bandwidth,
on-chip interconnect bandwidth, cache hierarchy bandwidth, and the
interactions between bandwidth, coherence traffic, and the memory
system.

This document is part of the XH-1 research series (area 05-memory).
Statements about XH-1 implementation are proposals unless explicitly
marked as decided.

---

## 1. Problem Statement

A 128-core processor generates substantially more memory traffic than
single-core or small-count-core designs. Even if each core issues a
modest number of outstanding memory requests, the aggregate demand can
quickly exceed the bandwidth available from a realistic DRAM subsystem.

The XH-1 design must therefore answer several related questions:

- How much DRAM bandwidth is physically available on the target package?
- How much of that bandwidth is consumed by coherence traffic?
- How much remains for useful application work?
- How is that remaining bandwidth shared between cores without
  starvation or severe unfairness?
- Where do bandwidth bottlenecks arise in the on-chip interconnect,
  cache hierarchy, and memory controllers?

The remainder of this document analyses each of these questions.

---

## 2. Background

### 2.1 DRAM Bandwidth Fundamentals

Modern DDR systems provide bandwidth through wide channels and high
transfer rates:

- DDR5 devices use 32-bit sub-channels per module rank, with
  transfer rates up to 8400 MT/s in JEDEC-published modules.
- A single DDR5 channel (64 bits) at 6400 MT/s delivers approximately
  51.2 GB/s peak; a dual-channel configuration roughly doubles this.
- HBM3 stacks deliver substantially higher bandwidth per package
  (around 819 GB/s for a 6-high stack per HBM3 specification).
- HBM3E extends this further, with published per-stack figures in the
  1 TB/s range for selected devices.

The XH-1 design has not fixed a memory technology. Both DDR5 and
HBM3-class options are considered below as proposals.

### 2.2 RISC-V Memory Model

RISC-V defines a weak memory ordering model, the RISC-V Weak Memory
Ordering (RVWMO), in the architecture specification. RVWMO permits a
wide range of reorderings and is similar in spirit to other modern
weakly ordered architectures.

Key consequences for bandwidth analysis:

- Loads may bypass earlier stores under default rules when addresses
  differ.
- Coherence is required only at a single total order per address
  (the "coherence order" axiom).
- Fence instructions (`fence`, `fence.i`, `fence.vma`) are required
  to constrain ordering where software demands it.

A weakly ordered model does not directly change bandwidth demand but
influences how much buffering, speculation, and replay logic is
required to expose bandwidth to software.

### 2.3 Bandwidth vs. Latency

Bandwidth and latency are distinct metrics. A system can have ample
aggregate DRAM bandwidth but very high per-request latency, in which
case small working sets run well but large working sets suffer.
A 128-core system is especially sensitive to the latency component
because aggregate concurrency exposes the latency of every shared
resource, including DRAM banks, the on-chip interconnect, and the
directory or snoop filter.

---

## 3. Bandwidth Demand Estimation

### 3.1 Per-Core Demand

A modern out-of-order core can issue multiple outstanding cache misses.
The exact number depends on the pipeline width, miss status holding
register (MSHR) count, and prefetcher effectiveness, but published
industrial designs commonly support 10-32 outstanding L1 misses per
core.

If each outstanding request ultimately consumes 64 bytes from DRAM,
and a core sustains 10 outstanding misses, the per-core demand is:

    10 misses * 64 B / core / (memory round-trip latency)

For a 200 ns round-trip DRAM latency, this is roughly:

    10 * 64 B / 200 ns = 3.2 GB/s per core

If all 128 cores simultaneously sustain this rate, the aggregate is
approximately 410 GB/s, which already exceeds what a single DDR5
channel can deliver.

This calculation is intentionally rough. It is intended to motivate
the rest of the document, not to set a precise requirement.

### 3.2 Realistic Demand

In practice, applications rarely sustain peak miss rates on every core
simultaneously. Published studies of throughput-oriented server
workloads report average L2 MPKI (misses per thousand instructions)
in the range of 5-30 for memory-bound kernels and 1-5 for cache-friendly
workloads. These values vary significantly across workloads.

Multiplying through, the worst-case aggregate bandwidth demand of a
128-core system can plausibly reach hundreds of GB/s. The XH-1 design
must therefore plan for a memory subsystem substantially more capable
than a single DDR5 channel.

### 3.3 Coherence Traffic Overhead

Cache coherence protocols add traffic on top of application reads and
writes. The magnitude depends on the protocol (see Section 6) and the
sharing pattern of the workload. A directory-based protocol typically
generates at least one directory lookup per coherence transaction and
potentially one or more forward/intervention messages.

A conservative estimate is that coherence and protocol overhead add
10-30% to the raw application demand, but this can rise significantly
for workloads with intensive cross-core sharing.

---

## 4. On-Chip Bandwidth

DRAM bandwidth is only one part of the system. The on-chip
interconnect and cache hierarchy must also carry:

- Coherence messages between cores and directories.
- Cache line fills from memory controllers to L2/L3 slices.
- Write-backs from L1 to L2.
- Snoop responses and invalidation acknowledgements.
- Interrupt and configuration traffic (typically negligible).

For a 128-core mesh-class interconnect, published industrial designs
indicate bisection bandwidth targets in the multi-TB/s range. Whether
XH-1 reaches this range depends on link width, clock frequency, and
topology choices that are not yet decided.

---

## 5. Interconnect Topology and Bandwidth

The interconnect determines how effectively aggregate on-chip
bandwidth is delivered to the DRAM controllers. Three common
topologies for many-core designs are:

1. Crossbar
2. Ring
3. Mesh
4. Concentrated mesh (cmesh)

| Topology | Typical use | Bandwidth scaling | Notes |
|----------|-------------|-------------------|-------|
| Crossbar | small core counts | linear in core count | poor scalability beyond ~16 cores |
| Ring | mid counts | limited by ring hops | simple but latency grows |
| Mesh | mid-to-high counts | roughly linear with links | common in tiled designs |
| Cmesh | high counts | similar to mesh, fewer routers | reduces router count |

A crossbar at 128 cores is generally impractical due to area and
wire count. A ring at 128 cores suffers from long hop counts and
limited bisection bandwidth. A mesh or concentrated mesh is the
most commonly chosen topology in published academic and industrial
128-core class designs.

The XH-1 topology is not yet decided; both a 2D mesh and a 2D
concentrated mesh are candidate proposals.

---

## 6. Coherence Protocol Bandwidth

### 6.1 Protocol Options

Two main families of coherence protocol exist:

- Snooping protocols (broadcast-based).
- Directory protocols (tracking-based).

At 128 cores, broadcast-based snooping is generally considered
impractical because every coherence transaction must reach every
core, consuming O(N) bandwidth per request. A directory protocol
incurs O(1) directory lookups per request in the common case but
requires directory storage and handles sharing patterns differently.

A directory is therefore the expected XH-1 proposal, consistent with
published 64-128 core research prototypes.

### 6.2 Directory Organization

Directories can be implemented in several ways:

- Full map directory: tracks sharers exactly. Storage cost is O(N)
  per line; prohibitive at 128 cores without compression or
  hierarchical schemes.
- Limited pointer directory: tracks a fixed number of sharers per
  line (for example 4-6 sharers). When the count is exceeded, a
  fallback representation (often a "broadcast" or "coarse vector"
  marker) is used.
- Coarse vector directory: groups cores into regions (e.g. 8-core
  tiles) and tracks presence at region granularity. Lower storage
  cost but less precise; can produce additional coherence traffic.
- Sparse directory: similar to coarse vector but with different
  trade-offs.

A limited pointer directory is the most commonly used scheme in
published industrial designs in this core-count range. The XH-1
design is expected to use a limited pointer directory, possibly
augmented with a coarse region fallback, but this is a proposal
rather than a decision.

### 6.3 Coherence Bandwidth Cost

For a limited pointer directory with P pointers, the common case
is one directory lookup per coherence transaction. Bandwidth cost
scales with the number of cores only when the directory overflows
into broadcast mode.

Published research indicates that overflow frequency is workload
dependent. For read-mostly shared data, pointer counts typically
stay small. For write-shared data, a limited pointer directory
can overflow frequently, increasing coherence traffic.

---

## 7. Cache Hierarchy Bandwidth

The cache hierarchy must sustain:

- L1 to L2 traffic (fills, write-backs, invalidations).
- L2 to L3 traffic.
- L3 to DRAM traffic.
- Snoop and probe traffic at every level.

Each cache level adds banked storage and multiple read/write ports to
deliver bandwidth. The area and energy cost of these ports grows
nonlinearly with port count.

For XH-1, a three-level hierarchy is the expected proposal:

- Private L1 per core.
- Private L2 per core (or per core pair) with a coherence controller.
- Shared L3 distributed across tiles.

The exact partition of L2 vs. L3 capacity, associativity, and
bandwidth is a design question left open in this document.

---

## 8. Bandwidth Allocation and QoS

When 128 cores compete for limited DRAM bandwidth, some form of
allocation policy is required. Options include:

- Strict priority ordering.
- Weighted fair queuing.
- Per-core bandwidth caps.
- Per-class (e.g. memory-mapped I/O vs. CPU) priorities.

Without allocation policy, a single core running a streaming workload
can saturate the memory subsystem and starve other cores. This is a
documented failure mode in many-core research literature.

RISC-V itself does not define memory bandwidth allocation semantics;
this is an XH-1 implementation choice. It is expected that the XH-1
memory controllers expose some form of bandwidth partitioning, but
this is a proposal rather than a decision.

---

## 9. Memory Controller Architecture

The memory controller is the point at which on-chip bandwidth meets
off-chip DRAM. Key design choices include:

- Number of independent channels.
- Channel-to-core mapping.
- Scheduling policy (FR-FCFS, parallelism-aware batch scheduling,
  etc.).
- Address mapping policy.
- Refresh and power management handling.

For DDR5-class memory, a 4-8 channel configuration is a reasonable
starting point for a 128-core design, but this is workload dependent
and is presented as a proposal.

For HBM3-class memory, the channel count is fixed by the stack
configuration; bandwidth is delivered through many independent
pseudo-channels per stack.

---

## 10. Quantitative Summary

The following table summarises representative bandwidth figures from
published sources. These are not XH-1 targets; they are external
reference points.

| Technology / Configuration | Peak Bandwidth | Source basis |
|-----------------------------|----------------|--------------|
| DDR5 single channel, 6400 MT/s | ~51.2 GB/s | JEDEC DDR5 spec |
| DDR5 dual channel, 6400 MT/s | ~102.4 GB/s | JEDEC DDR5 spec |
| HBM3 6-high stack | ~819 GB/s | JEDEC HBM3 spec |
| HBM3E 8-high stack | ~1 TB/s class | vendor published figures |
| 128-core aggregate theoretical (10 outstanding misses/core) | ~410 GB/s | rough calculation, Section 3.1 |

---

## 11. Advantages and Disadvantages of Candidate Memory Systems

| Option | Advantages | Disadvantages |
|--------|------------|---------------|
| DDR5 multi-channel | Standard, widely available, lower cost per GB | Lower peak bandwidth per channel, more channels increase board complexity |
| HBM3 / HBM3E | Very high bandwidth, high channel count per stack | Higher cost, 2.5D/3D packaging required, lower capacity per stack |
| Hybrid (HBM + DDR) | Combine high bandwidth with high capacity | Complex memory hierarchy, two coherence domains |

The XH-1 memory technology choice is not yet decided.

---

## 12. Scalability Problems at 128 Cores

The following scalability problems are well documented in the
many-core literature and are directly relevant to XH-1:

- **DRAM bandwidth saturation**: aggregate demand can exceed
  realistic DRAM bandwidth.
- **Directory overflow**: limited pointer directories overflow
  more often as core count rises, increasing coherence traffic.
- **Coherence traffic growth**: as more cores share data,
  coherence messages grow superlinearly in some sharing patterns.
- **Interconnect bisection bottlenecks**: insufficient bisection
  bandwidth creates hotspots at memory controllers.
- **Quality-of-service degradation**: without allocation policy,
  fairness collapses as core count rises.
- **Verification cost**: coherence state space grows roughly
  combinatorially with core count.

These are not unique to XH-1 but apply to any 128-core coherent
design.

---

## 13. Interactions With Other Subsystems

### 13.1 Pipeline

A deeper pipeline can support higher clock frequencies but increases
the penalty of cache misses. Bandwidth-bound workloads become
sensitive to the round-trip latency the pipeline exposes. The
pipeline must include enough MSHRs to expose memory-level parallelism.

### 13.2 Cache Hierarchy

Cache capacity and associativity directly affect miss rate and hence
DRAM bandwidth demand. Larger caches reduce demand at the cost of
area and access time. Bandwidth and capacity are coupled; both
must be planned together.

### 13.3 Memory System

The memory controller scheduling policy, address mapping, and
refresh handling all affect effective bandwidth. Closed-page vs.
open-page policies, bank parallelism, and request batching
materially change delivered bandwidth.

### 13.4 Interconnect

The interconnect must deliver at least as much aggregate bandwidth
as the DRAM subsystem, plus coherence overhead. Otherwise the
interconnect becomes the bottleneck and DRAM is underutilised.

### 13.5 Coherence

Coherence traffic is a first-class component of bandwidth demand.
Protocol choice and directory organisation directly determine the
share of bandwidth consumed by coherence.

### 13.6 Interrupts

Interrupts are typically not bandwidth-limited, but interrupt
delivery traffic and any interrupt-cause queues add minor traffic
to the system fabric.

### 13.7 Operating System

The OS schedules processes across cores, controls page placement,
and may implement memory allocation policies that affect locality.
The OS does not directly set DRAM bandwidth but its page colour
and NUMA-style placement policies interact with the memory
controllers.

### 13.8 Verification

Coherence protocols at 128 cores have a large state space.
Verification complexity grows roughly with the product of
per-core state and the number of cores. Formal methods,
randomised testing, and traffic pattern fuzzing are all relevant.

### 13.9 Performance

Bandwidth is one of the dominant performance constraints for
memory-bound workloads. Performance counters for DRAM channel
utilisation, interconnect link utilisation, and coherence message
counts are essential for tuning.

---

## 14. RISC-V Architectural Requirements

The following RISC-V architectural elements are relevant to bandwidth
analysis:

- The RISC-V ISA specifies `fence`, `fence.i`, and `fence.vma`
  instructions that order memory operations.
- The privileged architecture defines physical memory attributes
  (PMA) and, in designs with translation, page-based attributes
  (PBMT, NAPOT, etc.) that can affect memory behaviour.
- The RISC-V coherency management extensions (for example, the
  proposed H-extension in the vector and hypervisor contexts, and
  the cache-management operations in the RISC-V Profiles and
  IOMMU-related specifications) provide instructions such as
  `cbo.clean`, `cbo.flush`, and `cbo.inval` that interact with
  cache bandwidth.

RISC-V itself does not mandate a specific coherence protocol or
memory bandwidth management policy. These are XH-1 implementation
choices.

---

## 15. Open Questions and Unresolved Design Decisions

The following items are not yet decided and require further research
or design effort:

- Choice of DRAM technology (DDR5 multi-channel vs. HBM3-class).
- Number and placement of memory controllers.
- Interconnect topology (mesh vs. concentrated mesh, dimensional
  order, link width).
- Coherence protocol variant (e.g. MESI vs. MOESI and the
  directory structure).
- Bandwidth allocation policy and per-core fairness guarantees.
- L2 vs. L3 capacity split and per-tile bandwidth targets.
- Quality-of-service hooks exposed to the operating system.
- Verification strategy for the coherence and interconnect at
  128 cores.

These are recorded as open questions; no recommendation is made
where the available evidence does not strongly favour a particular
choice.

---

## 16. Recommendations

Recommendations are made only where published evidence and engineering
practice converge.

- **Use a directory-based coherence protocol**: published 64-128 core
  research and industrial designs consistently use directories; a
  snoop-based approach is not credible at 128 cores.
- **Plan for multi-channel DRAM**: a single DDR5 channel is unlikely
  to meet aggregate demand; at least 2-4 channels are a reasonable
  starting point if DDR5 is used, with HBM as an alternative if
  higher bandwidth is required.
- **Provide per-class quality-of-service in the memory controllers**:
  without allocation, fairness is not achievable in a 128-core
  system.
- **Instrument the design with bandwidth and coherence performance
  counters**: empirical tuning is unavoidable.
- **Verify coherence state space with a combination of formal
  methods and randomised testing**: full state space enumeration
  is not feasible at 128 cores.

These are recommendations, not decisions; they are subject to
revision as the XH-1 project progresses.

---

## 17. References

- RISC-V International, *The RISC-V Instruction Set Manual, Volume I:
  Unprivileged ISA*, current ratified version. RVWMO is described
  in Volume I.
- RISC-V International, *The RISC-V Instruction Set Manual, Volume II:
  Privileged Architecture*, current ratified version.
- JEDEC, *DDR5 SDRAM Standard* (JESD79-5), published release.
- JEDEC, *High Bandwidth Memory DRAM (HBM3)*, JESD238A.
- JEDEC, *High Bandwidth Memory DRAM (HBM3E)*, JESD238B-1 (where
  applicable).
- Hennessy and Patterson, *Computer Architecture: A Quantitative
  Approach*, recent edition, for general memory system modelling
  and AMAT analysis.
- Sorin, Hill, and Wood, *A Primer on Memory Consistency and Cache
  Coherence*, for coherence protocol analysis.
- Martin, Hill, and Wood, and successor work, on many-core
  coherence verification and token coherence (where relevant to
  XH-1 verification work).

Where specific figures in this document depend on vendor or
JEDEC-published numbers, the source is indicated in-line. Where
figures are derived from calculations, this is stated explicitly.

---

## 18. Document Status

This document is a research proposal and analysis. It is not a
description of an implemented design. Items marked as "proposal"
or "expected" have not been formally decided within the XH-1
project. Items marked as "open" represent unresolved design
questions. The document should be re-reviewed as the XH-1
architecture specification stabilises.