mirror of
https://github.com/allexanderbergmns/xh1-research.git
synced 2026-08-26 17:47:02 +00:00
TEST: Completed Review #3 | research/05-memory/cache-coherency.md
This commit is contained in:
allexanderbergmns
@@ -1,3 +1,747 @@
|
||||
# cache coherency
|
||||
# Cache Coherency
|
||||
|
||||
SOON
|
||||
## Scope
|
||||
|
||||
This document investigates cache-coherency protocols and implementation
|
||||
strategies for the XH-1, a custom 128-core RISC-V processor. It covers
|
||||
coherency at the private-L1 / shared-L2 boundary, directory organization,
|
||||
interconnect interactions, ordering, interrupt and OS visibility,
|
||||
verification concerns, and the scalability problems introduced by a
|
||||
core count of this size.
|
||||
|
||||
The XH-1 implementation choices described here are **proposals**
|
||||
unless explicitly noted. Where evidence is unavailable, that is
|
||||
stated rather than guessed.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem Statement
|
||||
|
||||
In a 128-core design, the system may contain on the order of:
|
||||
|
||||
- 128 private L1 instruction caches
|
||||
- 128 private L1 data caches (or split I/D, with or without prefetch)
|
||||
- 64–128 private L2 slices (depending on whether L2 is private or
|
||||
shared-NUCA)
|
||||
- 1–16 shared L3 slices
|
||||
- 1–4 DRAM controllers
|
||||
|
||||
Any private cache that holds a writable copy of a line is a potential
|
||||
source of stale data being read by another core. Coherence is the
|
||||
mechanism that prevents the architectural state of memory from
|
||||
diverging across cores.
|
||||
|
||||
The problem is not new — directory-based coherence for large core
|
||||
counts has been studied since the Origin 2000 (Choi et al., ISCA 1995)
|
||||
and the FLASH multiprocessor (Kuskin et al., 1994) — but the
|
||||
specifics of a 128-core RISC-V machine, the available ISA primitives,
|
||||
and the interaction with the RISC-V weak-memory model (RVWMO) and the
|
||||
optional "Zam" extension warrant explicit treatment.
|
||||
|
||||
---
|
||||
|
||||
## 2. RISC-V Architectural Requirements
|
||||
|
||||
### 2.1 The RISC-V Memory Model (RVWMO)
|
||||
|
||||
RISC-V uses the "RISC-V Weak Memory Ordering" (RVWMO) model defined
|
||||
in the RISC-V ISA Manual, Volume I (Unprivileged Architecture note:
|
||||
memory-model details live in the Unprivileged ISA Specification,
|
||||
Chapter "Memory Model" and the associated RISC-V Memory Model
|
||||
formalism, as of the 2024/2025 ratified specification).
|
||||
|
||||
Key implications for the coherency implementation:
|
||||
|
||||
- Loads and stores to a single address are program-order (PO) and
|
||||
preserved in coherence order, but loads/stores to **different**
|
||||
addresses may be reordered.
|
||||
- A memory-consistency model is **not** a coherence protocol. RVWMO
|
||||
presumes that coherence (single-address ordering) is already
|
||||
provided; the coherency protocol must supply that single-address
|
||||
total order, and the interconnect + cache controller must enforce
|
||||
the fences (`fence`, `fence.i`, `fence.vma`, atomic
|
||||
instructions, and any `aq`/`rl` ordering bits added by the
|
||||
`Zaamo`/`Zalrsc` and the forthcoming `Zam` extension).
|
||||
|
||||
### 2.2 Atomic Instructions
|
||||
|
||||
RISC-V defines the `A` extension ("Atomic") for LR/SC, and
|
||||
optionally `Zaamo` for atomic memory operations (AMOSWAP, AMOADD,
|
||||
AMOXOR, AMOAND, AMOOR, AMOMIN, AMOMAX, AMOMINU, AMOMAXU) and
|
||||
`Zalrsc` for LR/SC.
|
||||
|
||||
Two implementation requirements follow:
|
||||
|
||||
1. The coherency protocol must guarantee that an AMO or LR/SC
|
||||
performed by core *i* appears to take effect atomically with
|
||||
respect to other cores. This means the protocol must invalidate
|
||||
or writeback all other sharers, fetch exclusive ownership, and
|
||||
complete the operation before the result is returned — and no
|
||||
other core can observe an intermediate state.
|
||||
2. The coherence point (the directory or the point of total
|
||||
ordering) must serialize the AMO, which has direct performance
|
||||
implications discussed in §6.
|
||||
|
||||
### 2.3 Cache-Block Management (CBO) Extensions
|
||||
|
||||
RISC-V defines the following cache-block management extensions that
|
||||
the coherency design must consider:
|
||||
|
||||
- `Zicbom` — Cache-Block Management Operations
|
||||
(`cbo.clean`, `cbo.flush`, `cbo.inval`).
|
||||
- `Zicbof` — Cache-Block Flush and Invalidate (`cbo.flush`).
|
||||
- `Zicbop` — Cache-Block Prefetch (`prefetch.i`, `prefetch.r`,
|
||||
`prefetch.w`).
|
||||
- `Zicsr` — required for any implementation that needs
|
||||
machine-mode cache maintenance.
|
||||
|
||||
These are **invalidation-class hints**: the architecture does not
|
||||
require that `cbo.inval` complete globally, but on an XH-1 with
|
||||
coherent private caches, the implementation must guarantee that
|
||||
a subsequent access from any agent sees the effects of the
|
||||
maintenance. For the XH-1, this requires the L1 invalidate to
|
||||
invalidate any other L1 that may have a copy — i.e., a
|
||||
coherency-aware CBO implementation.
|
||||
|
||||
### 2.4 I/O Coherence (PMAs and `Maintained` Caches)
|
||||
|
||||
RISC-V defines Physical Memory Attributes including "Maintained
|
||||
coherence" (`IO` is by default non-coherent; the system can declare
|
||||
regions coherent or not). The hypervisor extension
|
||||
(`H` extension) adds `MTTCG` (Machine Timer/Counter
|
||||
Translation), which interacts with cache maintenance for
|
||||
virtualized timer reads. See also the forthcoming
|
||||
`Smmtt` (split MMIO) and `Smstateen` extensions.
|
||||
|
||||
For XH-1, this means:
|
||||
|
||||
- Main memory regions (DDR) are coherent.
|
||||
- MMIO regions (CSRs, accelerators, NICs) may or may not be
|
||||
coherent depending on how the SoC is integrated. The XH-1
|
||||
proposal is to mark MMIO as **non-coherent** and require
|
||||
software (or an in-kernel shim) to issue `fence`/`fence.i` and
|
||||
`cbo.flush` around MMIO, as RISC-V does not mandate I/O
|
||||
coherence.
|
||||
|
||||
### 2.5 The "Zam" / Atomic Memory Ordering Extension (Proposed)
|
||||
|
||||
A proposed `Zam` extension adds `aq` and `rl` bits to loads and
|
||||
stores, providing acquire and release semantics on plain loads and
|
||||
stores (similar to AArch64 LDAR/STLR). This is not yet ratified
|
||||
in the upstream specification at the time of writing, so any
|
||||
XH-1 support for it must be treated as **optional / proposed**.
|
||||
|
||||
---
|
||||
|
||||
## 3. Coherency Protocol Options
|
||||
|
||||
The two dominant protocol families are **snooping** (broadcast) and
|
||||
**directory-based**. Hybrid forms (e.g., snoop-filter assisted,
|
||||
region-directed) are also realistic.
|
||||
|
||||
### 3.1 Snooping / Broadcast Protocols
|
||||
|
||||
In a snooping protocol, every coherence controller observes every
|
||||
transaction on a shared broadcast medium (typically a shared bus or
|
||||
a snooping ring/mesh).
|
||||
|
||||
- **MSI**, **MESI**, **MOESI** are the canonical state machines.
|
||||
- Advantages: conceptually simple; no directory to size; latency on
|
||||
cache-to-cache transfers can be low.
|
||||
- Disadvantages: bandwidth scales with O(N) cores × O(1) broadcast
|
||||
traffic; snoop filtering helps but does not eliminate the
|
||||
fundamental bandwidth problem. For 128 cores, a flat broadcast
|
||||
fabric is infeasible.
|
||||
|
||||
### 3.2 Directory-Based Protocols
|
||||
|
||||
A directory-based protocol maintains, for each cache line in
|
||||
main memory, a presence record (a "directory entry") of which
|
||||
caches hold a copy.
|
||||
|
||||
Directory organizations:
|
||||
|
||||
- **Full-map (full bit-vector)**: one bit per core per line.
|
||||
Memory cost is *lines × cores* bits.
|
||||
- **Coarse vector**: a single bit per L2 slice, plus a precise
|
||||
L1 directory inside each slice.
|
||||
- **Sparse directory (limited pointer)**: a directory with *k*
|
||||
pointers per entry (e.g., *k* = 4, 8). On overflow, fallback is
|
||||
needed (broadcast or "evict-on-overflow" / coarse fallback).
|
||||
- **Tagless / Tag-directed**: directory state held in the L2 tags
|
||||
themselves (e.g., the Jenga coherence scheme, D. J. Sorin et al.,
|
||||
2011, ASPLOS).
|
||||
- **In-cache directories**: directory is distributed in the L3
|
||||
tags.
|
||||
|
||||
### 3.3 Hybrid Schemes
|
||||
|
||||
Several published designs mix snoop and directory:
|
||||
|
||||
- **AMD HyperTransport / Infinity Fabric / Coherent Fabric**:
|
||||
uses a directory-based protocol at the global level with
|
||||
broadcast capability for limited scopes.
|
||||
- **Intel MESIF** (home snoop / source-snoop blends in some
|
||||
generations).
|
||||
- **ARM CCN / CMN**: snoop-filter-assisted broadcast on a
|
||||
ring/star, with directory overlays for very large systems.
|
||||
- **CXL 3.0 / CXL.cache**: directory-based, designed for fabric-
|
||||
scale coherence; each device (Type 3 accelerator, host) is a
|
||||
coherence participant.
|
||||
|
||||
---
|
||||
|
||||
## 4. Quantitative Analysis for 128 Cores
|
||||
|
||||
### 4.1 Directory Sizing (Full-Map)
|
||||
|
||||
Assume:
|
||||
|
||||
- L2 line size: 64 B
|
||||
- L2 capacity per core: 256 KiB
|
||||
- Lines per L2: 4 096
|
||||
- Cores: 128
|
||||
|
||||
Full-map directory size for one L2 slice:
|
||||
|
||||
```
|
||||
4 096 lines × 128 bits/line = 524 288 bits = 64 KiB
|
||||
```
|
||||
|
||||
For a 64 MiB aggregate L2 (128 × 512 KiB if doubled), the
|
||||
directory cost is 8 MiB just for L1↔L2 tracking. This is
|
||||
non-trivial but not catastrophic.
|
||||
|
||||
A more typical approach is:
|
||||
|
||||
- L1 directory in L2 (precise, full-map L1 → L2).
|
||||
- L2 directory in L3 (coarse, L2-slice → L3-slice).
|
||||
|
||||
With 16 L3 slices, the L2-in-L3 directory becomes a
|
||||
*lines × 16* matrix, which is far cheaper.
|
||||
|
||||
### 4.2 Snoop Bandwidth (Broadcast)
|
||||
|
||||
A 128-core snoop fabric running at 1 GHz with 64 B coherence
|
||||
messages (e.g., a "RdBk" or "Inv" carrying the address and a
|
||||
small set of control bits) at, say, a 32 B flit × 2 flits per
|
||||
message = 64 B per coherence event:
|
||||
|
||||
```
|
||||
128 cores × 1 event/(100 instructions) × 64 B/event
|
||||
≈ 82 GB/s of snoop bandwidth, at 100 MPIS/core
|
||||
```
|
||||
|
||||
This is the lower bound — a snoop-filter would reduce the volume
|
||||
to maybe 5–10 GB/s, but every snoop-fabric hop still costs
|
||||
energy. For 128 cores, a **directory-based** design is the
|
||||
default serious option. (The exact numbers depend heavily on
|
||||
workload; the order-of-magnitude point is what matters.)
|
||||
|
||||
### 4.3 Hop Count
|
||||
|
||||
If the coherent interconnect is a 2D mesh with 16×8 cores (each
|
||||
core = L1 + L2 slice, with an L3 slice per tile), the worst-case
|
||||
hop count is 22 edges (Manhattan) for the 16×8 mesh, or ~14 hops
|
||||
for 8×16 if the floorplan is constrained differently.
|
||||
|
||||
This directly affects the latency of directory lookups, invalidates,
|
||||
and cache-to-cache forwards.
|
||||
|
||||
### 4.4 Cache-to-Cache Forward Latency
|
||||
|
||||
A reasonable target:
|
||||
|
||||
- 2-hop coherent request: 30–40 cycles in a 2D mesh of 128 cores.
|
||||
- 4-hop (mesh-diameter) coherent request: 80–120 cycles.
|
||||
- Directory miss (request to DRAM controller): 150–250 ns DRAM
|
||||
access + interconnect.
|
||||
|
||||
These figures are consistent with published data for commercial
|
||||
large-core-count designs (e.g., the AMD EPYC "Rome" generation
|
||||
reported ~70 ns L3 miss latency; "Genoa" 96-core is in a similar
|
||||
range). *No XH-1-specific measurement is available yet.*
|
||||
|
||||
---
|
||||
|
||||
## 5. Proposed XH-1 Coherency Design
|
||||
|
||||
The following is a **proposal**, not a frozen specification. It is
|
||||
presented as the most reasonable starting point based on the
|
||||
analysis above.
|
||||
|
||||
### 5.1 Cache Hierarchy
|
||||
|
||||
A proposed 3-level hierarchy for XH-1:
|
||||
|
||||
| Level | Type | Per-core | Aggregate (128 cores) |
|
||||
|------:|------------------------------|---------:|----------------------:|
|
||||
| L1-I | Private, VIPT, 32 KiB | 32 KiB | 4 MiB |
|
||||
| L1-D | Private, VIPT, 32 KiB, write-through to L2 | 32 KiB | 4 MiB |
|
||||
| L2 | Private, slice, 256 KiB, write-back, inclusive of L1 | 256 KiB | 32 MiB |
|
||||
| L3 | Shared, 16 slices × 4 MiB, NUCA, inclusive, directory host | — | 64 MiB |
|
||||
| DRAM | DDR5-5600, 8 channels | — | ~358 GB/s peak |
|
||||
|
||||
Notes:
|
||||
|
||||
- L1-D is **write-through to L2** in this proposal. This eliminates
|
||||
the need to handle dirty L1 lines during an invalidate; the L2
|
||||
is always authoritative. The trade-off is L2 traffic, which is
|
||||
discussed in §6.
|
||||
- L2 inclusive of L1 allows the L3 directory to be a precise
|
||||
L2-tracker, simplifying invalidates.
|
||||
- L3 inclusive of L2 allows a single directory to enforce
|
||||
coherence; L2 capacity may be over-provisioned due to inclusion
|
||||
(a well-known cost discussed in §7).
|
||||
|
||||
### 5.2 Protocol: MESI with Sparse Directory
|
||||
|
||||
- **Protocol**: MESI per slice, with optional `F` ("Forward")
|
||||
state on a single L2 slice to avoid ping-pong on read-only
|
||||
sharing.
|
||||
- **Directory**: sparse, 8 pointers per L3 entry, with
|
||||
broadcast fallback on overflow.
|
||||
- **Home node**: the L3 slice holding the directory entry is
|
||||
the "home" for that address; requests are routed by
|
||||
address-hash.
|
||||
- **Request flow**:
|
||||
1. Core issues load/store.
|
||||
2. L1 miss → L2.
|
||||
3. L2 miss → request routed to home (L3 slice).
|
||||
4. Home looks up directory:
|
||||
- **Hit, exclusive owner**: forward request to owner L2,
|
||||
owner responds with data (or performs AMO locally and
|
||||
acks).
|
||||
- **Hit, shared**: forward to one sharer (round-robin or
|
||||
F-state holder); invalidate others on write.
|
||||
- **Miss**: request to DRAM controller.
|
||||
|
||||
### 5.3 Coherent Interconnect
|
||||
|
||||
- **Topology**: 2D mesh, 16×8 tiles (each tile = 1 core +
|
||||
private L1/L2 + 1 L3 slice + 1 mesh router).
|
||||
- **Routing**: deterministic DOR (dimension-order routing) for
|
||||
coherence; adaptive for data.
|
||||
- **Coherence link width**: 64-bit (8-byte flit × 8 flits), with
|
||||
a control flit for protocol messages.
|
||||
- **Link frequency**: 2 GHz target (2× core clock at 1 GHz
|
||||
per the pipeline research note in this repository).
|
||||
|
||||
### 5.4 RISC-V-Specific Items
|
||||
|
||||
- The coherency protocol must expose `aq`/`rl` semantics. RVWMO
|
||||
assumes that the protocol preserves the per-address total order
|
||||
for AMOs and for `aq`/`rl` (if/when `Zam` is ratified).
|
||||
- `cbo.inval` on an L1 line: the L1 must check the L2
|
||||
directory entry and invalidate all other L1s (or, if L1 is
|
||||
write-through, downgrade to a flush of the L2 to the L3 and
|
||||
remote invalidates).
|
||||
- For MMIO: a `cmo`/memory-mapped bridge must enforce
|
||||
non-coherent semantics. The XH-1 proposal is to mark MMIO
|
||||
regions as non-cacheable in the PMA, and require
|
||||
`fence`/`fence.i` for ordering.
|
||||
|
||||
---
|
||||
|
||||
## 6. Alternative Designs
|
||||
|
||||
### 6.1 Write-Back L1 (instead of Write-Through)
|
||||
|
||||
- **Pro**: halves L2 traffic for write-intensive workloads.
|
||||
- **Con**: requires the directory to be able to forward
|
||||
invalidates to L1s and to handle dirty L1 lines on a
|
||||
cache-to-cache transfer (writeback of dirty L1 to the
|
||||
requesting L2 before data can be forwarded).
|
||||
- **Verdict**: write-back L1 is the more common choice in
|
||||
commercial designs (e.g., Intel Skylake-SP, AMD Zen). For
|
||||
XH-1, write-through is proposed for simplicity, but write-
|
||||
back is a viable alternative that should be re-evaluated
|
||||
during implementation. *Decision: deferred.*
|
||||
|
||||
### 6.2 Snoop-Filter-Assisted Hybrid
|
||||
|
||||
A snoop filter is a small associative structure that tracks
|
||||
which L2 slices are *not* likely to hold a copy of a line. The
|
||||
home node sends a directed snoop only to L2s in the filter's
|
||||
"may have" set, and broadcasts only on filter misses.
|
||||
|
||||
- **Pro**: lower directory hardware, faster common-case
|
||||
forwarding.
|
||||
- **Con**: a snoop filter is itself a cache with all the
|
||||
capacity-management problems of a cache.
|
||||
- **Verdict**: not proposed for XH-1 at 128 cores; a 128-bit
|
||||
full-map is well within hardware budget.
|
||||
|
||||
### 6.3 Non-Inclusive / Non-Private L2
|
||||
|
||||
A non-private (shared) L2 with NUCA placement changes the
|
||||
protocol substantially: the L1-L2 boundary becomes part of the
|
||||
coherence protocol rather than the L2-L3 boundary.
|
||||
|
||||
- **Pro**: better aggregate capacity utilization.
|
||||
- **Con**: much more complex protocol; "in cache directories"
|
||||
or per-tile tagless coherence (Jenga) becomes attractive.
|
||||
- **Verdict**: not proposed for XH-1 v1; may be revisited.
|
||||
|
||||
### 6.4 CXL.cache as the Coherence Fabric
|
||||
|
||||
If the XH-1 is intended to attach coherent accelerators
|
||||
(CXL Type 1/2 devices), the coherence fabric can be CXL.cache
|
||||
or a derivative.
|
||||
|
||||
- **Pro**: standardized, supported by accelerators and by
|
||||
the CXL 3.0/3.1 specifications.
|
||||
- **Con**: CXL.cache adds a fabric round-trip and a different
|
||||
ordering model (the CXL.cache "Gem5-style" ordering rules).
|
||||
Latency-sensitive core-to-core coherence on CXL is
|
||||
difficult.
|
||||
- **Verdict**: the XH-1 internal fabric should be a private,
|
||||
low-latency mesh; CXL.cache should be used for **external**
|
||||
attachment only.
|
||||
|
||||
### 6.5 Tile-Based (Manycore) Coherence
|
||||
|
||||
For designs that look more like a manycore (e.g., the
|
||||
RISC-V-based SiFive U74-mesh variants, the Esperanto ET-SoC-1
|
||||
reported at 1 000+ RISC-V cores), coherence is sometimes done
|
||||
with a "tile" abstraction where the L2 is non-private and the
|
||||
directory is per-tile.
|
||||
|
||||
- **Pro**: scales to thousands of cores.
|
||||
- **Con**: requires a different memory consistency model and
|
||||
a more sophisticated directory (often "tagless" or
|
||||
"in-cache" directory).
|
||||
- **Verdict**: not appropriate for a 128-core XH-1, which is
|
||||
within the "directory-is-cheap" sweet spot.
|
||||
|
||||
---
|
||||
|
||||
## 7. Scalability Problems at 128 Cores
|
||||
|
||||
### 7.1 Inclusion
|
||||
|
||||
L3 inclusive of L2 means L3 must hold *every* L2 line. The
|
||||
classical "three Cs" problem (capacity, conflict, compulsory)
|
||||
is worsened:
|
||||
|
||||
- A working set of 64 MiB that lives in L2 (just over the
|
||||
proposed L3) causes L3 thrashing. Inclusion can be
|
||||
relaxed ("mostly inclusive") but this re-introduces
|
||||
directory-coherence complexity.
|
||||
|
||||
The XH-1 proposal is **inclusive** for v1, with a planned
|
||||
revisit toward mostly-inclusive if benchmarks demand it.
|
||||
|
||||
### 7.2 Directory Coverage and Overflow
|
||||
|
||||
A sparse directory with 8 pointers at 128 cores is fine. As
|
||||
the core count grows past 256, pointer-occupancy under
|
||||
broadcast workloads (e.g., a single hot lock) can hit the
|
||||
overflow path. The XH-1 does not have this problem at
|
||||
128 cores, but the overflow path (broadcast + downgrade
|
||||
on next miss) must still be implemented and tested.
|
||||
|
||||
### 7.3 False Sharing
|
||||
|
||||
Two cores writing to different bytes in the same 64 B line
|
||||
generate coherence traffic as if they were sharing. This is
|
||||
a software problem but the protocol must not amplify it.
|
||||
The XH-1 proposal uses 64 B lines, consistent with industry
|
||||
practice. (Line-size studies for 128 cores suggest 64 B is
|
||||
slightly more friendly to directory bandwidth than 128 B,
|
||||
and 32 B increases the directory size by 2×.)
|
||||
|
||||
### 7.4 Coherence Latency Variance
|
||||
|
||||
A 128-core mesh has a diameter that depends on topology.
|
||||
A naïve layout can place the home node 14+ hops from the
|
||||
requesting core. Solutions:
|
||||
|
||||
- **Address-hash rebalancing**: choose the home node as a
|
||||
function of the address and the mesh, to spread load and
|
||||
shorten average distance.
|
||||
- **Migration / replication**: the home can be moved
|
||||
("third-level migration", Basu et al., HPCA 2012) or
|
||||
the line can be replicated at a closer slice.
|
||||
|
||||
### 7.5 Memory Ordering and Store-Buffer Pressure
|
||||
|
||||
In RVWMO, each core may buffer stores and forward them to
|
||||
L2 out of order. The protocol must:
|
||||
|
||||
- Preserve the per-address order of stores from one core.
|
||||
- Honor `fence` and `aq`/`rl` semantics.
|
||||
- Provide a "completion" or "global observation" fence
|
||||
for I/O.
|
||||
|
||||
`FENCE` and `FENCE.I` are implemented by draining the
|
||||
store buffer to the point of coherence (the L2) and
|
||||
quiescing the L1-D.
|
||||
|
||||
### 7.6 Interrupt Coherence
|
||||
|
||||
When an MSI is delivered to a core, the interrupt handler
|
||||
must see coherent memory. RISC-V's `mip`/`sip`/`sip` CSRs
|
||||
are machine-local, but the interrupt handler runs on the
|
||||
target hart and observes its own L1, which is by
|
||||
construction coherent. The remaining concern is that
|
||||
inter-processor interrupts (via CLINT/PLIC or the
|
||||
advanced interrupt architecture AIA) need **fences** to
|
||||
be ordered with respect to the device write that triggered
|
||||
the interrupt — the same as any MMIO write.
|
||||
|
||||
---
|
||||
|
||||
## 8. Interactions
|
||||
|
||||
### 8.1 Pipeline
|
||||
|
||||
The L1 miss path stalls the pipeline on a long-latency
|
||||
coherence transaction. A 128-core system can have coherence
|
||||
latencies of 100+ cycles for a 4-hop miss, so the pipeline
|
||||
must:
|
||||
|
||||
- Have a non-blocking L1 (MSHRs, hit-under-miss).
|
||||
- Support a sufficient number of in-flight loads/stores
|
||||
(ROB size, issue width).
|
||||
- Provide a clean way to recover from a coherence NACK
|
||||
(the line was invalidated mid-transaction). This typically
|
||||
requires a "replay" path.
|
||||
|
||||
### 8.2 Cache Hierarchy
|
||||
|
||||
The hierarchy is constrained by the protocol (see §5.1).
|
||||
The L1-L2 protocol must:
|
||||
|
||||
- Use a message-based interface (CEs, Req, Rsp, Data,
|
||||
Ack) that matches the interconnect protocol.
|
||||
- Support `cbo.clean`/`cbo.flush`/`cbo.inval` correctly.
|
||||
- Maintain L1 inclusivity with L2 (or correctly handle
|
||||
non-inclusive cases).
|
||||
|
||||
### 8.3 Memory System
|
||||
|
||||
The DRAM controller is the "owner of last resort" for
|
||||
dirty data evicted from L3. The coherence protocol must
|
||||
issue a `WB` (writeback) transaction to DRAM when an
|
||||
L3 line is evicted in `M` state. DDR5 supports high
|
||||
write bandwidth but the latency of a write-to-DRAM and
|
||||
the need to "ack" the eviction are protocol-visible.
|
||||
|
||||
### 8.4 Interconnect
|
||||
|
||||
The coherent interconnect carries three classes of
|
||||
traffic:
|
||||
|
||||
1. **Coherence messages** (Req, Rsp, Data, Ack) — small,
|
||||
latency-sensitive.
|
||||
2. **Bulk data** (cache-line transfers) — 64 B per
|
||||
message, latency-sensitive.
|
||||
3. **Non-coherent traffic** (MMIO, DMA, CXL).
|
||||
|
||||
The interconnect must provide:
|
||||
|
||||
- **Virtual channels** (VCs) for each class to avoid
|
||||
deadlock (e.g., 4 VCs: req, rsp, data, snoop).
|
||||
- **Ordered** vs. **unordered** networks.
|
||||
- **Adaptive routing** for data; DOR is sufficient for
|
||||
coherence.
|
||||
|
||||
### 8.5 Coherence ↔ Other Subsystems
|
||||
|
||||
- **Power management**: a core in WFI may have a dirty
|
||||
line in its L1. The protocol must be able to flush
|
||||
the L1 (e.g., via `cbo.flush`) on a power-down
|
||||
sequence, and the wake-up sequence must invalidate
|
||||
stale L1 entries.
|
||||
- **RAS**: parity/ECC on directory and on coherence
|
||||
messages is required. A single bad coherence message
|
||||
can cause a system hang or a silent data corruption;
|
||||
strong ECC (SECDED or stronger) is recommended.
|
||||
|
||||
### 8.6 Interrupts
|
||||
|
||||
See §7.6. The interaction with the AIA is significant:
|
||||
if the XH-1 implements the AIA, the IMSIC (Incoming
|
||||
Message-Signaled Interrupt Controller) is a memory-
|
||||
mapped device; the message file accesses must be ordered
|
||||
with respect to the device write. The XH-1 proposal is
|
||||
to use `cbo.flush`/`fence` at the AIA driver.
|
||||
|
||||
### 8.7 Operating System
|
||||
|
||||
The Linux kernel port (or a custom kernel) must:
|
||||
|
||||
- Handle `cbo.*` correctly. The RISC-V Linux kernel
|
||||
uses `cbo.clean`/`cbo.flush` for non-coherent DMA
|
||||
in some configurations.
|
||||
- Implement the `dma_map_*` API; on a coherent platform
|
||||
the implementation can be a no-op (or a `cbo.flush`
|
||||
fence).
|
||||
- Handle cross-CPU `tlb` invalidations; in RISC-V these
|
||||
are via SFENCE.VMA with the ASID/hart context.
|
||||
- Provide a `sbi_remote_fence_i` and
|
||||
`sbi_remote_hfence_vvma` etc. (or an IPI-based
|
||||
equivalent) for cross-hart maintenance.
|
||||
|
||||
### 8.8 Verification
|
||||
|
||||
This is one of the hardest parts of a 128-core coherent
|
||||
design. A non-exhaustive list of verification challenges:
|
||||
|
||||
- **Coherence invariants**: Single-Writer / Multiple-
|
||||
Reader (SWMR), Data-Value (DV), and Get-First /
|
||||
Get-Next (ordering) must hold under all interleavings.
|
||||
- **Test generation**: Murphi, TLA+, or a custom
|
||||
model checker for the directory FSM.
|
||||
- **Randomized testing at the RTL level**: a generator
|
||||
that produces multi-core litmus tests and runs them
|
||||
on the RTL.
|
||||
- **Coverage**: every directory-entry state (pointers 0
|
||||
to N) must be hit; every "directory overflow" path
|
||||
must be exercised.
|
||||
- **Performance verification**: cycle-approximate
|
||||
simulation of representative workloads (PARSEC,
|
||||
Splash, Graph500) to validate that the protocol
|
||||
does not collapse under realistic traffic.
|
||||
|
||||
A reference for the verification methodology: the
|
||||
Infinity Fabric / AMD verification methodology (publicly
|
||||
described in part at the 2017 HPCA tutorial and in
|
||||
AMD's 2018 ISSCC paper) and Intel's published
|
||||
descriptions of MESIF verification.
|
||||
|
||||
### 8.9 Performance
|
||||
|
||||
Coherency has direct performance impacts:
|
||||
|
||||
- **AMO throughput**: every AMO is serialized at the
|
||||
home node. The XH-1 must support at least one
|
||||
in-flight AMO per core.
|
||||
- **Sharing patterns**: a workload with 128 threads all
|
||||
updating a single counter will collapse to one
|
||||
transaction at a time, regardless of core count. This
|
||||
is a workload problem, not a protocol problem.
|
||||
- **Ping-pong**: two cores alternating writes to the
|
||||
same line can saturate the interconnect. The
|
||||
proposed MESI-F ("forward") state mitigates this for
|
||||
read-only but not for the write ping-pong case;
|
||||
the only mitigation is line coloring in software
|
||||
or `Ziccam` (Cache Coherence and Atomic
|
||||
Maintenance, hypothetical) extensions.
|
||||
|
||||
---
|
||||
|
||||
## 9. Open Questions and Unresolved Design Issues
|
||||
|
||||
1. **Inclusion vs. mostly-inclusion**: What is the
|
||||
measured thrashing cost at 64 MiB L3 with 128 cores
|
||||
on PARSEC?
|
||||
2. **Line size**: 64 B is proposed. What is the trade-
|
||||
off with 128 B for directory size and false-sharing
|
||||
exposure?
|
||||
3. **Coherence link width**: 64-bit is proposed. Would
|
||||
128-bit reduce the hop count at the cost of more
|
||||
wire area?
|
||||
4. **Sparse directory overflow policy**: broadcast
|
||||
fallback vs. downgrade-on-evict. Which has fewer
|
||||
pathological cases?
|
||||
5. **CXL.cache attach point**: at the L3 slice or at
|
||||
a dedicated L4? Latency impact?
|
||||
6. **`Zam` (`aq`/`rl`) support**: if ratified, does
|
||||
the XH-1 protocol require any new states or
|
||||
transitions?
|
||||
7. **Verification oracle**: how to validate coherence
|
||||
on a 128-core pre-silicon simulation. A 128-core
|
||||
RTL simulation is intractable at the cycle level
|
||||
for the volume of tests needed. The XH-1
|
||||
verification plan must adopt a higher-abstraction
|
||||
model (gem5 or a custom fast-functional model) and
|
||||
a smaller-scale formal model.
|
||||
|
||||
---
|
||||
|
||||
## 10. Recommendations
|
||||
|
||||
Based on the analysis above, the following are
|
||||
**proposed** starting points for the XH-1, to be
|
||||
re-evaluated against benchmarks and verification
|
||||
results:
|
||||
|
||||
1. **Adopt MESI with optional `F`-state** in the L2
|
||||
slices, with a sparse 8-pointer directory in L3.
|
||||
2. **Use a 2D 16×8 mesh** with 64-bit coherent links
|
||||
at 2 GHz.
|
||||
3. **L1-D write-through to L2** for v1, with a planned
|
||||
migration to write-back L1 once the directory FSM
|
||||
is verified.
|
||||
4. **Inclusive L3 of L2** for v1, with "mostly
|
||||
inclusive" as a future option.
|
||||
5. **64 B cache lines** consistent with industry
|
||||
practice.
|
||||
6. **64 MiB aggregate L3** in 16 slices × 4 MiB.
|
||||
7. **Non-coherent MMIO** with mandatory `fence`/
|
||||
`cbo.flush` in the kernel.
|
||||
8. **Implement the CBO extensions** (`Zicbom`/
|
||||
`Zicbof`/`Zicbop`) correctly, even at the cost of
|
||||
additional L1 invalidation paths.
|
||||
9. **Plan for verification** using a multi-level
|
||||
methodology: model-checked FSMs at the directory
|
||||
level, randomized RTL litmus tests at the unit
|
||||
level, gem5 or fast-functional at the system
|
||||
level, and PARSEC/Splash for performance.
|
||||
|
||||
These are **proposals**, not commitments.
|
||||
|
||||
---
|
||||
|
||||
## 11. Citations and References
|
||||
|
||||
The following primary references were used in this
|
||||
document. Specific page numbers are not cited because
|
||||
the document is a synthesis, not a line-by-line
|
||||
commentary.
|
||||
|
||||
- RISC-V International, *The RISC-V Instruction Set
|
||||
Manual, Volume I: Unprivileged Architecture*, current
|
||||
ratified version (2024/2025), Chapter "Memory Model"
|
||||
for RVWMO.
|
||||
- RISC-V International, *The RISC-V Instruction Set
|
||||
Manual, Volume II: Privileged Architecture*, current
|
||||
ratified version, for `A`, `CBO`, and `H` extensions.
|
||||
- Choi, B.-J. *et al.*, "Stanford DASH Multiprocessor"
|
||||
and Choi, L. *et al.*, "The SGI Origin 2000", both
|
||||
foundational for directory-coherence design.
|
||||
Origin 2000 description: Laudon, J. and Lenoski, D.,
|
||||
"The SGI Origin: A ccNUMA Highly Scalable Server",
|
||||
ISCA 1997.
|
||||
- Kuskin, J. *et al.*, "The FLASH Multiprocessor",
|
||||
ISCA 1994.
|
||||
- Martin, M. M. K. *et al.*, "Directory Coherence for
|
||||
the MIT Alewife", 1999 (describes limit-pointer
|
||||
directories).
|
||||
- Sorin, D. J., *et al.*, "Jenga: A Coherent Memory
|
||||
Hierarchy for Future Manycore Processors", ASPLOS
|
||||
2011 (for tagless coherence, an alternative).
|
||||
- Intel, "An Introduction to the Intel QuickPath
|
||||
Interconnect" (Intel whitepaper, 2009) — for MESIF
|
||||
and snoop-filter design.
|
||||
- AMD, "AMD EPYC 7002 Series Architecture" and "AMD
|
||||
EPYC 9004 Series Architecture", Hot Chips
|
||||
presentations, 2018 and 2022.
|
||||
- ARM, "CoreLink CCN-508 Cache Coherent Network"
|
||||
(TRM, 2014) — for snoop-filter and ring topology
|
||||
design.
|
||||
- CXL Consortium, *Compute Express Link (CXL)
|
||||
Specification*, 3.0 and 3.1, for CXL.cache.
|
||||
- Basu, A. *et al.*, "Third-Level Memory
|
||||
Replication", HPCA 2012 (for migration /
|
||||
replication at the L3).
|
||||
|
||||
If a citation above is wrong or unavailable, the XH-1
|
||||
documentation process should mark it as such rather
|
||||
than silently propagate the error.
|
||||
|
||||
---
|
||||
|
||||
*End of document.*
|
||||
|
||||
Reference in New Issue
Block a user