# Cache Coherency

## Scope

This document investigates cache-coherency protocols and implementation
strategies for the XH-1, a custom 128-core RISC-V processor. It covers
coherency at the private-L1 / shared-L2 boundary, directory organization,
interconnect interactions, ordering, interrupt and OS visibility,
verification concerns, and the scalability problems introduced by a
core count of this size.

The XH-1 implementation choices described here are **proposals**
unless explicitly noted. Where evidence is unavailable, that is
stated rather than guessed.

---

## 1. Problem Statement

In a 128-core design, the system may contain on the order of:

- 128 private L1 instruction caches
- 128 private L1 data caches (or split I/D, with or without prefetch)
- 64–128 private L2 slices (depending on whether L2 is private or
  shared-NUCA)
- 1–16 shared L3 slices
- 1–4 DRAM controllers

Any private cache that holds a writable copy of a line is a potential
source of stale data being read by another core. Coherence is the
mechanism that prevents the architectural state of memory from
diverging across cores.

The problem is not new — directory-based coherence for large core
counts has been studied since the Origin 2000 (Choi et al., ISCA 1995)
and the FLASH multiprocessor (Kuskin et al., 1994) — but the
specifics of a 128-core RISC-V machine, the available ISA primitives,
and the interaction with the RISC-V weak-memory model (RVWMO) and the
optional "Zam" extension warrant explicit treatment.

---

## 2. RISC-V Architectural Requirements

### 2.1 The RISC-V Memory Model (RVWMO)

RISC-V uses the "RISC-V Weak Memory Ordering" (RVWMO) model defined
in the RISC-V ISA Manual, Volume I (Unprivileged Architecture note:
memory-model details live in the Unprivileged ISA Specification,
Chapter "Memory Model" and the associated RISC-V Memory Model
formalism, as of the 2024/2025 ratified specification).

Key implications for the coherency implementation:

- Loads and stores to a single address are program-order (PO) and
  preserved in coherence order, but loads/stores to **different**
  addresses may be reordered.
- A memory-consistency model is **not** a coherence protocol. RVWMO
  presumes that coherence (single-address ordering) is already
  provided; the coherency protocol must supply that single-address
  total order, and the interconnect + cache controller must enforce
  the fences (`fence`, `fence.i`, `fence.vma`, atomic
  instructions, and any `aq`/`rl` ordering bits added by the
  `Zaamo`/`Zalrsc` and the forthcoming `Zam` extension).

### 2.2 Atomic Instructions

RISC-V defines the `A` extension ("Atomic") for LR/SC, and
optionally `Zaamo` for atomic memory operations (AMOSWAP, AMOADD,
AMOXOR, AMOAND, AMOOR, AMOMIN, AMOMAX, AMOMINU, AMOMAXU) and
`Zalrsc` for LR/SC.

Two implementation requirements follow:

1. The coherency protocol must guarantee that an AMO or LR/SC
   performed by core *i* appears to take effect atomically with
   respect to other cores. This means the protocol must invalidate
   or writeback all other sharers, fetch exclusive ownership, and
   complete the operation before the result is returned — and no
   other core can observe an intermediate state.
2. The coherence point (the directory or the point of total
   ordering) must serialize the AMO, which has direct performance
   implications discussed in §6.

### 2.3 Cache-Block Management (CBO) Extensions

RISC-V defines the following cache-block management extensions that
the coherency design must consider:

- `Zicbom` — Cache-Block Management Operations
  (`cbo.clean`, `cbo.flush`, `cbo.inval`).
- `Zicbof` — Cache-Block Flush and Invalidate (`cbo.flush`).
- `Zicbop` — Cache-Block Prefetch (`prefetch.i`, `prefetch.r`,
  `prefetch.w`).
- `Zicsr` — required for any implementation that needs
  machine-mode cache maintenance.

These are **invalidation-class hints**: the architecture does not
require that `cbo.inval` complete globally, but on an XH-1 with
coherent private caches, the implementation must guarantee that
a subsequent access from any agent sees the effects of the
maintenance. For the XH-1, this requires the L1 invalidate to
invalidate any other L1 that may have a copy — i.e., a
coherency-aware CBO implementation.

### 2.4 I/O Coherence (PMAs and `Maintained` Caches)

RISC-V defines Physical Memory Attributes including "Maintained
coherence" (`IO` is by default non-coherent; the system can declare
regions coherent or not). The hypervisor extension
(`H` extension) adds `MTTCG` (Machine Timer/Counter
Translation), which interacts with cache maintenance for
virtualized timer reads. See also the forthcoming
`Smmtt` (split MMIO) and `Smstateen` extensions.

For XH-1, this means:

- Main memory regions (DDR) are coherent.
- MMIO regions (CSRs, accelerators, NICs) may or may not be
  coherent depending on how the SoC is integrated. The XH-1
  proposal is to mark MMIO as **non-coherent** and require
  software (or an in-kernel shim) to issue `fence`/`fence.i` and
  `cbo.flush` around MMIO, as RISC-V does not mandate I/O
  coherence.

### 2.5 The "Zam" / Atomic Memory Ordering Extension (Proposed)

A proposed `Zam` extension adds `aq` and `rl` bits to loads and
stores, providing acquire and release semantics on plain loads and
stores (similar to AArch64 LDAR/STLR). This is not yet ratified
in the upstream specification at the time of writing, so any
XH-1 support for it must be treated as **optional / proposed**.

---

## 3. Coherency Protocol Options

The two dominant protocol families are **snooping** (broadcast) and
**directory-based**. Hybrid forms (e.g., snoop-filter assisted,
region-directed) are also realistic.

### 3.1 Snooping / Broadcast Protocols

In a snooping protocol, every coherence controller observes every
transaction on a shared broadcast medium (typically a shared bus or
a snooping ring/mesh).

- **MSI**, **MESI**, **MOESI** are the canonical state machines.
- Advantages: conceptually simple; no directory to size; latency on
  cache-to-cache transfers can be low.
- Disadvantages: bandwidth scales with O(N) cores × O(1) broadcast
  traffic; snoop filtering helps but does not eliminate the
  fundamental bandwidth problem. For 128 cores, a flat broadcast
  fabric is infeasible.

### 3.2 Directory-Based Protocols

A directory-based protocol maintains, for each cache line in
main memory, a presence record (a "directory entry") of which
caches hold a copy.

Directory organizations:

- **Full-map (full bit-vector)**: one bit per core per line.
  Memory cost is *lines × cores* bits.
- **Coarse vector**: a single bit per L2 slice, plus a precise
  L1 directory inside each slice.
- **Sparse directory (limited pointer)**: a directory with *k*
  pointers per entry (e.g., *k* = 4, 8). On overflow, fallback is
  needed (broadcast or "evict-on-overflow" / coarse fallback).
- **Tagless / Tag-directed**: directory state held in the L2 tags
  themselves (e.g., the Jenga coherence scheme, D. J. Sorin et al.,
  2011, ASPLOS).
- **In-cache directories**: directory is distributed in the L3
  tags.

### 3.3 Hybrid Schemes

Several published designs mix snoop and directory:

- **AMD HyperTransport / Infinity Fabric / Coherent Fabric**:
  uses a directory-based protocol at the global level with
  broadcast capability for limited scopes.
- **Intel MESIF** (home snoop / source-snoop blends in some
  generations).
- **ARM CCN / CMN**: snoop-filter-assisted broadcast on a
  ring/star, with directory overlays for very large systems.
- **CXL 3.0 / CXL.cache**: directory-based, designed for fabric-
  scale coherence; each device (Type 3 accelerator, host) is a
  coherence participant.

---

## 4. Quantitative Analysis for 128 Cores

### 4.1 Directory Sizing (Full-Map)

Assume:

- L2 line size: 64 B
- L2 capacity per core: 256 KiB
- Lines per L2: 4 096
- Cores: 128

Full-map directory size for one L2 slice:

```
4 096 lines × 128 bits/line = 524 288 bits = 64 KiB
```

For a 64 MiB aggregate L2 (128 × 512 KiB if doubled), the
directory cost is 8 MiB just for L1↔L2 tracking. This is
non-trivial but not catastrophic.

A more typical approach is:

- L1 directory in L2 (precise, full-map L1 → L2).
- L2 directory in L3 (coarse, L2-slice → L3-slice).

With 16 L3 slices, the L2-in-L3 directory becomes a
*lines × 16* matrix, which is far cheaper.

### 4.2 Snoop Bandwidth (Broadcast)

A 128-core snoop fabric running at 1 GHz with 64 B coherence
messages (e.g., a "RdBk" or "Inv" carrying the address and a
small set of control bits) at, say, a 32 B flit × 2 flits per
message = 64 B per coherence event:

```
128 cores × 1 event/(100 instructions) × 64 B/event
    ≈ 82 GB/s of snoop bandwidth, at 100 MPIS/core
```

This is the lower bound — a snoop-filter would reduce the volume
to maybe 5–10 GB/s, but every snoop-fabric hop still costs
energy. For 128 cores, a **directory-based** design is the
default serious option. (The exact numbers depend heavily on
workload; the order-of-magnitude point is what matters.)

### 4.3 Hop Count

If the coherent interconnect is a 2D mesh with 16×8 cores (each
core = L1 + L2 slice, with an L3 slice per tile), the worst-case
hop count is 22 edges (Manhattan) for the 16×8 mesh, or ~14 hops
for 8×16 if the floorplan is constrained differently.

This directly affects the latency of directory lookups, invalidates,
and cache-to-cache forwards.

### 4.4 Cache-to-Cache Forward Latency

A reasonable target:

- 2-hop coherent request: 30–40 cycles in a 2D mesh of 128 cores.
- 4-hop (mesh-diameter) coherent request: 80–120 cycles.
- Directory miss (request to DRAM controller): 150–250 ns DRAM
  access + interconnect.

These figures are consistent with published data for commercial
large-core-count designs (e.g., the AMD EPYC "Rome" generation
reported ~70 ns L3 miss latency; "Genoa" 96-core is in a similar
range). *No XH-1-specific measurement is available yet.*

---

## 5. Proposed XH-1 Coherency Design

The following is a **proposal**, not a frozen specification. It is
presented as the most reasonable starting point based on the
analysis above.

### 5.1 Cache Hierarchy

A proposed 3-level hierarchy for XH-1:

| Level | Type                         | Per-core | Aggregate (128 cores) |
|------:|------------------------------|---------:|----------------------:|
| L1-I  | Private, VIPT, 32 KiB         | 32 KiB   | 4 MiB                |
| L1-D  | Private, VIPT, 32 KiB, write-through to L2 | 32 KiB   | 4 MiB                |
| L2    | Private, slice, 256 KiB, write-back, inclusive of L1 | 256 KiB  | 32 MiB               |
| L3    | Shared, 16 slices × 4 MiB, NUCA, inclusive, directory host | —         | 64 MiB               |
| DRAM  | DDR5-5600, 8 channels         | —         | ~358 GB/s peak       |

Notes:

- L1-D is **write-through to L2** in this proposal. This eliminates
  the need to handle dirty L1 lines during an invalidate; the L2
  is always authoritative. The trade-off is L2 traffic, which is
  discussed in §6.
- L2 inclusive of L1 allows the L3 directory to be a precise
  L2-tracker, simplifying invalidates.
- L3 inclusive of L2 allows a single directory to enforce
  coherence; L2 capacity may be over-provisioned due to inclusion
  (a well-known cost discussed in §7).

### 5.2 Protocol: MESI with Sparse Directory

- **Protocol**: MESI per slice, with optional `F` ("Forward")
  state on a single L2 slice to avoid ping-pong on read-only
  sharing.
- **Directory**: sparse, 8 pointers per L3 entry, with
  broadcast fallback on overflow.
- **Home node**: the L3 slice holding the directory entry is
  the "home" for that address; requests are routed by
  address-hash.
- **Request flow**:
  1. Core issues load/store.
  2. L1 miss → L2.
  3. L2 miss → request routed to home (L3 slice).
  4. Home looks up directory:
     - **Hit, exclusive owner**: forward request to owner L2,
       owner responds with data (or performs AMO locally and
       acks).
     - **Hit, shared**: forward to one sharer (round-robin or
       F-state holder); invalidate others on write.
     - **Miss**: request to DRAM controller.

### 5.3 Coherent Interconnect

- **Topology**: 2D mesh, 16×8 tiles (each tile = 1 core +
  private L1/L2 + 1 L3 slice + 1 mesh router).
- **Routing**: deterministic DOR (dimension-order routing) for
  coherence; adaptive for data.
- **Coherence link width**: 64-bit (8-byte flit × 8 flits), with
  a control flit for protocol messages.
- **Link frequency**: 2 GHz target (2× core clock at 1 GHz
  per the pipeline research note in this repository).

### 5.4 RISC-V-Specific Items

- The coherency protocol must expose `aq`/`rl` semantics. RVWMO
  assumes that the protocol preserves the per-address total order
  for AMOs and for `aq`/`rl` (if/when `Zam` is ratified).
- `cbo.inval` on an L1 line: the L1 must check the L2
  directory entry and invalidate all other L1s (or, if L1 is
  write-through, downgrade to a flush of the L2 to the L3 and
  remote invalidates).
- For MMIO: a `cmo`/memory-mapped bridge must enforce
  non-coherent semantics. The XH-1 proposal is to mark MMIO
  regions as non-cacheable in the PMA, and require
  `fence`/`fence.i` for ordering.

---

## 6. Alternative Designs

### 6.1 Write-Back L1 (instead of Write-Through)

- **Pro**: halves L2 traffic for write-intensive workloads.
- **Con**: requires the directory to be able to forward
  invalidates to L1s and to handle dirty L1 lines on a
  cache-to-cache transfer (writeback of dirty L1 to the
  requesting L2 before data can be forwarded).
- **Verdict**: write-back L1 is the more common choice in
  commercial designs (e.g., Intel Skylake-SP, AMD Zen). For
  XH-1, write-through is proposed for simplicity, but write-
  back is a viable alternative that should be re-evaluated
  during implementation. *Decision: deferred.*

### 6.2 Snoop-Filter-Assisted Hybrid

A snoop filter is a small associative structure that tracks
which L2 slices are *not* likely to hold a copy of a line. The
home node sends a directed snoop only to L2s in the filter's
"may have" set, and broadcasts only on filter misses.

- **Pro**: lower directory hardware, faster common-case
  forwarding.
- **Con**: a snoop filter is itself a cache with all the
  capacity-management problems of a cache.
- **Verdict**: not proposed for XH-1 at 128 cores; a 128-bit
  full-map is well within hardware budget.

### 6.3 Non-Inclusive / Non-Private L2

A non-private (shared) L2 with NUCA placement changes the
protocol substantially: the L1-L2 boundary becomes part of the
coherence protocol rather than the L2-L3 boundary.

- **Pro**: better aggregate capacity utilization.
- **Con**: much more complex protocol; "in cache directories"
  or per-tile tagless coherence (Jenga) becomes attractive.
- **Verdict**: not proposed for XH-1 v1; may be revisited.

### 6.4 CXL.cache as the Coherence Fabric

If the XH-1 is intended to attach coherent accelerators
(CXL Type 1/2 devices), the coherence fabric can be CXL.cache
or a derivative.

- **Pro**: standardized, supported by accelerators and by
  the CXL 3.0/3.1 specifications.
- **Con**: CXL.cache adds a fabric round-trip and a different
  ordering model (the CXL.cache "Gem5-style" ordering rules).
  Latency-sensitive core-to-core coherence on CXL is
  difficult.
- **Verdict**: the XH-1 internal fabric should be a private,
  low-latency mesh; CXL.cache should be used for **external**
  attachment only.

### 6.5 Tile-Based (Manycore) Coherence

For designs that look more like a manycore (e.g., the
RISC-V-based SiFive U74-mesh variants, the Esperanto ET-SoC-1
reported at 1 000+ RISC-V cores), coherence is sometimes done
with a "tile" abstraction where the L2 is non-private and the
directory is per-tile.

- **Pro**: scales to thousands of cores.
- **Con**: requires a different memory consistency model and
  a more sophisticated directory (often "tagless" or
  "in-cache" directory).
- **Verdict**: not appropriate for a 128-core XH-1, which is
  within the "directory-is-cheap" sweet spot.

---

## 7. Scalability Problems at 128 Cores

### 7.1 Inclusion

L3 inclusive of L2 means L3 must hold *every* L2 line. The
classical "three Cs" problem (capacity, conflict, compulsory)
is worsened:

- A working set of 64 MiB that lives in L2 (just over the
  proposed L3) causes L3 thrashing. Inclusion can be
  relaxed ("mostly inclusive") but this re-introduces
  directory-coherence complexity.

The XH-1 proposal is **inclusive** for v1, with a planned
revisit toward mostly-inclusive if benchmarks demand it.

### 7.2 Directory Coverage and Overflow

A sparse directory with 8 pointers at 128 cores is fine. As
the core count grows past 256, pointer-occupancy under
broadcast workloads (e.g., a single hot lock) can hit the
overflow path. The XH-1 does not have this problem at
128 cores, but the overflow path (broadcast + downgrade
on next miss) must still be implemented and tested.

### 7.3 False Sharing

Two cores writing to different bytes in the same 64 B line
generate coherence traffic as if they were sharing. This is
a software problem but the protocol must not amplify it.
The XH-1 proposal uses 64 B lines, consistent with industry
practice. (Line-size studies for 128 cores suggest 64 B is
slightly more friendly to directory bandwidth than 128 B,
and 32 B increases the directory size by 2×.)

### 7.4 Coherence Latency Variance

A 128-core mesh has a diameter that depends on topology.
A naïve layout can place the home node 14+ hops from the
requesting core. Solutions:

- **Address-hash rebalancing**: choose the home node as a
  function of the address and the mesh, to spread load and
  shorten average distance.
- **Migration / replication**: the home can be moved
  ("third-level migration", Basu et al., HPCA 2012) or
  the line can be replicated at a closer slice.

### 7.5 Memory Ordering and Store-Buffer Pressure

In RVWMO, each core may buffer stores and forward them to
L2 out of order. The protocol must:

- Preserve the per-address order of stores from one core.
- Honor `fence` and `aq`/`rl` semantics.
- Provide a "completion" or "global observation" fence
  for I/O.

`FENCE` and `FENCE.I` are implemented by draining the
store buffer to the point of coherence (the L2) and
quiescing the L1-D.

### 7.6 Interrupt Coherence

When an MSI is delivered to a core, the interrupt handler
must see coherent memory. RISC-V's `mip`/`sip`/`sip` CSRs
are machine-local, but the interrupt handler runs on the
target hart and observes its own L1, which is by
construction coherent. The remaining concern is that
inter-processor interrupts (via CLINT/PLIC or the
advanced interrupt architecture AIA) need **fences** to
be ordered with respect to the device write that triggered
the interrupt — the same as any MMIO write.

---

## 8. Interactions

### 8.1 Pipeline

The L1 miss path stalls the pipeline on a long-latency
coherence transaction. A 128-core system can have coherence
latencies of 100+ cycles for a 4-hop miss, so the pipeline
must:

- Have a non-blocking L1 (MSHRs, hit-under-miss).
- Support a sufficient number of in-flight loads/stores
  (ROB size, issue width).
- Provide a clean way to recover from a coherence NACK
  (the line was invalidated mid-transaction). This typically
  requires a "replay" path.

### 8.2 Cache Hierarchy

The hierarchy is constrained by the protocol (see §5.1).
The L1-L2 protocol must:

- Use a message-based interface (CEs, Req, Rsp, Data,
  Ack) that matches the interconnect protocol.
- Support `cbo.clean`/`cbo.flush`/`cbo.inval` correctly.
- Maintain L1 inclusivity with L2 (or correctly handle
  non-inclusive cases).

### 8.3 Memory System

The DRAM controller is the "owner of last resort" for
dirty data evicted from L3. The coherence protocol must
issue a `WB` (writeback) transaction to DRAM when an
L3 line is evicted in `M` state. DDR5 supports high
write bandwidth but the latency of a write-to-DRAM and
the need to "ack" the eviction are protocol-visible.

### 8.4 Interconnect

The coherent interconnect carries three classes of
traffic:

1. **Coherence messages** (Req, Rsp, Data, Ack) — small,
   latency-sensitive.
2. **Bulk data** (cache-line transfers) — 64 B per
   message, latency-sensitive.
3. **Non-coherent traffic** (MMIO, DMA, CXL).

The interconnect must provide:

- **Virtual channels** (VCs) for each class to avoid
  deadlock (e.g., 4 VCs: req, rsp, data, snoop).
- **Ordered** vs. **unordered** networks.
- **Adaptive routing** for data; DOR is sufficient for
  coherence.

### 8.5 Coherence ↔ Other Subsystems

- **Power management**: a core in WFI may have a dirty
  line in its L1. The protocol must be able to flush
  the L1 (e.g., via `cbo.flush`) on a power-down
  sequence, and the wake-up sequence must invalidate
  stale L1 entries.
- **RAS**: parity/ECC on directory and on coherence
  messages is required. A single bad coherence message
  can cause a system hang or a silent data corruption;
  strong ECC (SECDED or stronger) is recommended.

### 8.6 Interrupts

See §7.6. The interaction with the AIA is significant:
if the XH-1 implements the AIA, the IMSIC (Incoming
Message-Signaled Interrupt Controller) is a memory-
mapped device; the message file accesses must be ordered
with respect to the device write. The XH-1 proposal is
to use `cbo.flush`/`fence` at the AIA driver.

### 8.7 Operating System

The Linux kernel port (or a custom kernel) must:

- Handle `cbo.*` correctly. The RISC-V Linux kernel
  uses `cbo.clean`/`cbo.flush` for non-coherent DMA
  in some configurations.
- Implement the `dma_map_*` API; on a coherent platform
  the implementation can be a no-op (or a `cbo.flush`
  fence).
- Handle cross-CPU `tlb` invalidations; in RISC-V these
  are via SFENCE.VMA with the ASID/hart context.
- Provide a `sbi_remote_fence_i` and
  `sbi_remote_hfence_vvma` etc. (or an IPI-based
  equivalent) for cross-hart maintenance.

### 8.8 Verification

This is one of the hardest parts of a 128-core coherent
design. A non-exhaustive list of verification challenges:

- **Coherence invariants**: Single-Writer / Multiple-
  Reader (SWMR), Data-Value (DV), and Get-First /
  Get-Next (ordering) must hold under all interleavings.
- **Test generation**: Murphi, TLA+, or a custom
  model checker for the directory FSM.
- **Randomized testing at the RTL level**: a generator
  that produces multi-core litmus tests and runs them
  on the RTL.
- **Coverage**: every directory-entry state (pointers 0
  to N) must be hit; every "directory overflow" path
  must be exercised.
- **Performance verification**: cycle-approximate
  simulation of representative workloads (PARSEC,
  Splash, Graph500) to validate that the protocol
  does not collapse under realistic traffic.

A reference for the verification methodology: the
Infinity Fabric / AMD verification methodology (publicly
described in part at the 2017 HPCA tutorial and in
AMD's 2018 ISSCC paper) and Intel's published
descriptions of MESIF verification.

### 8.9 Performance

Coherency has direct performance impacts:

- **AMO throughput**: every AMO is serialized at the
  home node. The XH-1 must support at least one
  in-flight AMO per core.
- **Sharing patterns**: a workload with 128 threads all
  updating a single counter will collapse to one
  transaction at a time, regardless of core count. This
  is a workload problem, not a protocol problem.
- **Ping-pong**: two cores alternating writes to the
  same line can saturate the interconnect. The
  proposed MESI-F ("forward") state mitigates this for
  read-only but not for the write ping-pong case;
  the only mitigation is line coloring in software
  or `Ziccam` (Cache Coherence and Atomic
  Maintenance, hypothetical) extensions.

---

## 9. Open Questions and Unresolved Design Issues

1. **Inclusion vs. mostly-inclusion**: What is the
   measured thrashing cost at 64 MiB L3 with 128 cores
   on PARSEC?
2. **Line size**: 64 B is proposed. What is the trade-
   off with 128 B for directory size and false-sharing
   exposure?
3. **Coherence link width**: 64-bit is proposed. Would
   128-bit reduce the hop count at the cost of more
   wire area?
4. **Sparse directory overflow policy**: broadcast
   fallback vs. downgrade-on-evict. Which has fewer
   pathological cases?
5. **CXL.cache attach point**: at the L3 slice or at
   a dedicated L4? Latency impact?
6. **`Zam` (`aq`/`rl`) support**: if ratified, does
   the XH-1 protocol require any new states or
   transitions?
7. **Verification oracle**: how to validate coherence
   on a 128-core pre-silicon simulation. A 128-core
   RTL simulation is intractable at the cycle level
   for the volume of tests needed. The XH-1
   verification plan must adopt a higher-abstraction
   model (gem5 or a custom fast-functional model) and
   a smaller-scale formal model.

---

## 10. Recommendations

Based on the analysis above, the following are
**proposed** starting points for the XH-1, to be
re-evaluated against benchmarks and verification
results:

1. **Adopt MESI with optional `F`-state** in the L2
   slices, with a sparse 8-pointer directory in L3.
2. **Use a 2D 16×8 mesh** with 64-bit coherent links
   at 2 GHz.
3. **L1-D write-through to L2** for v1, with a planned
   migration to write-back L1 once the directory FSM
   is verified.
4. **Inclusive L3 of L2** for v1, with "mostly
   inclusive" as a future option.
5. **64 B cache lines** consistent with industry
   practice.
6. **64 MiB aggregate L3** in 16 slices × 4 MiB.
7. **Non-coherent MMIO** with mandatory `fence`/
   `cbo.flush` in the kernel.
8. **Implement the CBO extensions** (`Zicbom`/
   `Zicbof`/`Zicbop`) correctly, even at the cost of
   additional L1 invalidation paths.
9. **Plan for verification** using a multi-level
   methodology: model-checked FSMs at the directory
   level, randomized RTL litmus tests at the unit
   level, gem5 or fast-functional at the system
   level, and PARSEC/Splash for performance.

These are **proposals**, not commitments.

---

## 11. Citations and References

The following primary references were used in this
document. Specific page numbers are not cited because
the document is a synthesis, not a line-by-line
commentary.

- RISC-V International, *The RISC-V Instruction Set
  Manual, Volume I: Unprivileged Architecture*, current
  ratified version (2024/2025), Chapter "Memory Model"
  for RVWMO.
- RISC-V International, *The RISC-V Instruction Set
  Manual, Volume II: Privileged Architecture*, current
  ratified version, for `A`, `CBO`, and `H` extensions.
- Choi, B.-J. *et al.*, "Stanford DASH Multiprocessor"
  and Choi, L. *et al.*, "The SGI Origin 2000", both
  foundational for directory-coherence design.
  Origin 2000 description: Laudon, J. and Lenoski, D.,
  "The SGI Origin: A ccNUMA Highly Scalable Server",
  ISCA 1997.
- Kuskin, J. *et al.*, "The FLASH Multiprocessor",
  ISCA 1994.
- Martin, M. M. K. *et al.*, "Directory Coherence for
  the MIT Alewife", 1999 (describes limit-pointer
  directories).
- Sorin, D. J., *et al.*, "Jenga: A Coherent Memory
  Hierarchy for Future Manycore Processors", ASPLOS
  2011 (for tagless coherence, an alternative).
- Intel, "An Introduction to the Intel QuickPath
  Interconnect" (Intel whitepaper, 2009) — for MESIF
  and snoop-filter design.
- AMD, "AMD EPYC 7002 Series Architecture" and "AMD
  EPYC 9004 Series Architecture", Hot Chips
  presentations, 2018 and 2022.
- ARM, "CoreLink CCN-508 Cache Coherent Network"
  (TRM, 2014) — for snoop-filter and ring topology
  design.
- CXL Consortium, *Compute Express Link (CXL)
  Specification*, 3.0 and 3.1, for CXL.cache.
- Basu, A. *et al.*, "Third-Level Memory
  Replication", HPCA 2012 (for migration /
  replication at the L3).

If a citation above is wrong or unavailable, the XH-1
documentation process should mark it as such rather
than silently propagate the error.

---

*End of document.*
