xh1-research-fork/research/.xh1/runs/20260826T123303Z/candidate.md

# Memory Ordering

> **Status:** Research proposal for the XH-1 128-core RISC-V processor.
> Nothing in this document is a finalized architectural commitment unless
> explicitly stated. Items marked **[Proposed]** are engineering
> recommendations under evaluation; items marked **[Open]** are unresolved
> design questions.

---

## 1. Scope and Definitions

**Memory ordering** defines the set of rules that determine when memory
operations issued by one or more hardware threads become visible to other
hardware threads, and in what order such visibility is required to occur.

In a single-core, single-threaded system, "ordering" is trivial: program
order matches execution order matches visibility order. As soon as a
processor exposes:

- out-of-order execution,
- multiple hardware threads (SMT),
- multiple cores with private caches,
- coherent or non-coherent interconnects, or
- devices that perform DMA,

…memory ordering becomes a first-class architectural and microarchitectural
concern. For a 128-core design like XH-1, it is one of the dominant
correctness and performance constraints.

The RISC-V ISA delegates most ordering policy to a **Memory Consistency
Model (MCM)** specification rather than baking it into the base ISA. The
current published MCM is documented in the RISC-V specification volume
*“RISC-V Instruction Set Manual, Volume I: Unprivileged ISA”*, Appendix A —
*"RVWMO: The RISC-V Weak Memory Ordering Model"*. RVWMO is the canonical
model; "Zam" (the "A" extension candidate) is a candidate formalization
of the same weak-ordering contract with additional operational
clarifications.

This document covers:

1. The RVWMO contract XH-1 must honor.
2. Microarchitectural mechanisms needed to enforce that contract.
3. Scalability concerns specific to 128 cores.
4. Interactions with coherence, the interconnect, the pipeline, the
   OS, and verification.

---

## 2. The RVWMO Contract XH-1 Must Honor

RVWMO is defined by:

- **Preserved Program Order (PPO):** a set of (microarchitectural)
  orderings between pairs of memory operations from the same hart that
  the implementation must never violate. PPO rules include the obvious
  ones (a load following a store to the same address, register
  dependencies, fences such as `FENCE`, `FENCE.I`, etc.) and the
  subtler "axiomatic" PPO rules covering acquire/release semantics on
  ordinary AMOs.
- **Memory Model Axioms:** a set of forbidden load–load, load–store,
  and store–store reorderings when no PPO rule requires otherwise
  (e.g., "no load may be reordered with a store to a different
  location" except under specific conditions).
- **Syntactic Dependencies:** register dependencies via the
  destination register of a load are sufficient to constrain certain
  reorderings, mirroring the original SPARC RMO/TSO literature.

The base RVWMO does not mandate TSO; an implementation may run hart-local
memory accesses in any order consistent with the PPO rules, with a few
mandated orderings (e.g., loads after stores to the *same* address are
preserved).

Key architectural levers available to software:

| Construct                  | Effect (informal)                                       |
| -------------------------- | ------------------------------------------------------- |
| `FENCE` (various `pred`/`succ`/fm) | Orders memory operations as specified.       |
| `FENCE.I`                  | Orders stores to instruction memory w.r.t. subsequent instruction fetches on the same hart. |
| `FENCE.VMA`                | Orders SFENCE.VMA-style TLB management.                 |
| `SFENCE.VMA`               | Orders stores w.r.t. subsequent TLB refills on the same hart. |
| Acquire/release on AMOs    | Provides PPO ordering for paired RMW operations.        |
| `LR`/`SC` (LR/SC pair)    | Ordering anchored to the reservation.                   |
| `Zalrsc` / `Zaamo`         | Standard atomic extensions.                             |
| `Zifencei`                 | Required for `FENCE.I`.                                 |
| `Zihintpause`              | Hint only; no ordering effect.                          |
| RV32/64 `fence` on devices | Device ordering via PTE bits (see §10.3).               |

**XH-1 requirement:** the ISA-visible contract of every memory and
synchronization instruction listed in the *Unprivileged ISA* and any
extension XH-1 implements (`A`, `Zifencei`, `Zihintpause`, `Zalrsc`,
`Zaamo`, `Ztso` if adopted) must be honored by the combined
core/LSU/coherence/interconnect subsystem.

> **Open:** Whether XH-1 advertises `Ztso` (an optional RVWMO relaxation
> that grants TSO at every hart) or only RVWMO. `Ztso` simplifies
> software reasoning but slightly constrains hardware ordering
> flexibility. Recommendation deferred until microarchitecture is
> fixed.

---

## 3. Required Fences and Their Microarchitectural Meaning

The `FENCE` instruction is parameterized by:

- `pred`: which predecessor operations are ordered (R, W).
- `succ`: which successor operations are ordered (R, W).
- `fm` field: indicates "fence-mode"; value `0` is the standard fence,
  value `>=1` is reserved for implementations and is treated as a no-op
  by standard tools. Custom XH-1 fence-mode encodings are **[Open]**.

XH-1 must ensure that a `FENCE` with predecessor set *P* and successor
set *S*:

- All memory operations of type *P* issued by this hart prior to the
  `FENCE` are ordered before all memory operations of type *S* issued by
  this hart after the `FENCE`, in the global memory order, to the extent
  that the coherence protocol is willing to serialize them.

Microarchitecturally, the `FENCE` typically drains in-flight operations
of the predecessor classes through the load/store unit (LSU), the
coherence point, and the write-completion path. On XH-1 with private L1
and a shared L2 slice, "the coherence point" is the directory controller
on the L2 slice; see §7.

---

## 4. Why 128 Cores Changes the Problem

A small-count many-core can often "get away" with conservative
ordering: a single shared bus or ring, broadcast snoops, and a
full-fence on every store buffer drain. At 128 cores these approaches
do not scale:

1. **Bandwidth:** A serialized global fence requires all in-flight
   traffic from the issuing core to drain, *and* the directory to
   observe all acknowledgments before subsequent operations are
   injected. With 128 cores contending for the directory and the
   interconnect, the latency of a single `FENCE` can grow from
   ~10–30 cycles (small system) to several hundred cycles. This
   directly impacts lock-based critical sections, RCU grace periods,
   and `atomic` operations in the OS.

2. **Queue occupancy:** Every core holds per-message-class queues
   (read requests, write requests, invalidations, data responses,
   writeback data). A fence must account for occupancy of queues
   between the core and the L2 directory *and* between directories
   across the mesh. Without back-pressure, an XH-1 fence could
   complete locally but leave a "hole" in the global order.

3. **Coherence protocol overhead:** Directory-based coherence
   (assumed for XH-1; see the coherence research document) replaces
   broadcast snoop traffic with per-line directory lookups, but
   introduces ack/grant ordering dependencies that the fence must
   not skip past.

4. **Scalable coherence introduces non-atomic transitions.**
   Directory state moves from M→I→S→M through intermediate invalidation
   acknowledgment states. A store-conditional (`SC`) on one core that
   has been racing with a remote `LR` may observe the line in
   intermediate states; the memory model must be specified for these.

5. **Contention of fence-induced drains:** When 128 cores all execute
   `FENCE` in a short window (e.g., a barrier in a parallel workload),
   the interconnect can become congested with drain acknowledgments.
   The microarchitecture must avoid feedback-loop deadlock between
   the queues and the fence-acknowledgment path.

> **Proposed XH-1 stance:** Treat memory ordering as a system-level
> performance constraint, not just a per-core correctness one. Evaluate
> the cost of every `FENCE` site in the OS, runtime, and synchronization
> libraries at the system level, not at single-core simulation.

---

## 5. Microarchitectural Mechanisms

The classic toolkit, applied to XH-1:

### 5.1 Load-Store Unit (LSU) Pipeline

A typical in-order XH-1 core will have:

- A **store buffer (STB)** between the integer pipeline and the L1
  data cache.
- A **load queue (LDQ)** that issues loads past stores (speculatively)
  and validates the result at commit, rolling back on a misspeculation.
- A **memory dependence predictor** (optional; see §9.1) to reduce the
  cost of Ld→St forwarding checks.

The LSU's job relative to ordering is twofold:

1. **Honor PPO locally** — enforce that a load following a store to the
   same address returns the store's value (or a younger value), and
   that the AMO-acquire/release rules produce the required PPO edges.
2. **Provide a clean drain point** for `FENCE` — a place where the LSU
   can mark "all pred-class operations up to this point have completed
   globally."

### 5.2 Store Buffer Drain

A `FENCE` with `pred` including W must drain the store buffer. In a
directory-coherent system the relevant completion is not "store reached
L1" but "store has acquired the directory ownership token (e.g., `M`
response) and the directory has acknowledged writeback/Invalidate
acknowledgments for any invalidated peers."

For XH-1:

- **L1 hit, no coherence traffic:** store can complete in ~1–3 cycles
  after reaching L1.
- **L1 miss, L2 hit, no sharers:** invalidation may be skipped (silent
  upgrade to M); store completes when directory grants M.
- **L1 miss, L2 hit, sharers exist:** directory generates invalidation
  messages to all sharers; the store completes when all invalidation
  acknowledgments are received.
- **L1 miss, L2 miss:** directory forwards to home node; home node
  replies with data and grants M; the store completes when data and
  the M grant are both received.

Only the last three cases impose significant drain latency.

> **Proposed XH-1 stance:** The store buffer must be sized to hold
> stores pending completion, but the fence-drain path must not depend
> on the store buffer draining in order — older stores in the buffer
> that have already received their M response should not block the
> fence on younger stores still in flight, *unless* the fence
> successor set requires it.

### 5.3 Load Issue Rules

Loads are the harder side. RVWMO allows loads to be reordered with
prior stores to different addresses. XH-1's LSU will speculatively
issue loads past older stores; the load is "validated" at commit by
checking that the cache line has not been invalidated and that no
older store wrote to the same address.

A load that needs to "see" the result of an older store on the same
hart uses **store-to-load forwarding** through the store buffer; this
must respect the size and sign-extension rules of the load.

A load that needs to "see" the result of an older store on a *different*
hart (which is what `FENCE`/`FENCE.RW` enforces) must wait for that
remote store to complete at the directory and update the local
view via the coherence protocol. The directory's response is the
"happens-before" edge.

### 5.4 Acquire/Release on AMOs

A standard `AMO*` operation in RVWMO has the ordering semantics of an
acquire (load part) and release (store part) when used in `aq`/`rl`
positions. In RV32A/RV64A, the instructions `LR` and `SC` carry
acquire/release bits; AMOs (e.g., `AMOSWAP.W`, `AMOADD.D`) also carry
these bits.

For XH-1, an `aq` AMO must order prior loads/stores from the issuing
hart before the AMO, and the AMO before subsequent loads/stores.
Microarchitecturally:

- `aq`: drain the load and store queues (or at least establish PPO
  edges) before issuing the AMO to the directory.
- `rl`: drain the load and store queues *after* the AMO completes.
- `aq+rl`: drain on both sides; this is the strong fence.

> **Open:** Whether XH-1 implements `aq`/`rl` purely in the LSU
> (with no global fence message) or with a lightweight fence message
> to the directory. Purely-local enforcement is faster but requires
> that the directory serialize the AMO at the line state machine.

### 5.5 `LR`/`SC`

`LR` loads with reservation; `SC` stores conditionally if the
reservation is still valid. XH-1's reservation representation is
proposed as a small set of reservation-set entries per core, indexed
by physical address, anchored in the L1. Coherence actions that
invalidate the reserved line (a snoop/Invalidate that the L1 sees)
must invalidate the reservation.

> **Proposed XH-1 reservation-granularity choice:** XH-1 reserves at
> cache-line granularity (the natural L1 line size) and treats
> forward-progress as the OS's responsibility. This is the
> conservative default; some designs use sub-line reservation to
> improve SC success rates, but the gain on typical lock code is
> modest.

### 5.6 `FENCE.I` and Self-Modifying Code

`FENCE.I` ensures that stores from the issuing hart that preceded the
`FENCE.I` are visible to subsequent instruction fetches on the same
hart. It does **not** order stores from other harts.

For XH-1, `FENCE.I` typically:

1. Drains the store buffer (in the W sense).
2. Invalidates the L1 I-cache (or the local portion of a unified L1).
3. Optionally issues a `FENCE` internally so that the store is
   visible at the directory before the I-fetch resumes.

A cross-hart self-modifying-code sequence (hart A writes code, hart B
executes it) requires an explicit `FENCE` on hart A and a `FENCE.I` on
hart B (per the RISC-V recommendation), plus coherence to propagate
the new data. This is awkward and XH-1 will document it in the
programmer's manual.

### 5.7 `SFENCE.VMA`

`SFENCE.VMA` orders prior stores to PTEs before subsequent
address-translation results on the same hart. XH-1's TLB refill
engine must observe the store-completion signal; a typical
implementation stalls the pipeline at the `SFENCE.VMA` until the
store buffer drains.

---

## 6. System-Level Ordering Devices

Beyond per-hart mechanisms, XH-1 needs system-level ordering for
I/O and devices. The RISC-V privileged specification provides two
mechanisms:

### 6.1 I/O Ordering Bits in PTE

A PTE can have the **PBMT** field (Page-Based Memory Types), which can
be marked as:

- **Non-cacheable, idempotent, weakly-ordered (NC, IO).**
- **Non-cacheable, idempotent, strongly-ordered (NC, I/O-strong).**

XH-1 must honor these by:

- Routing accesses to the appropriate device port on the interconnect.
- For weakly-ordered I/O, treating the access as device but allowing
  reordering with respect to other weakly-ordered I/O of the same type.
- For strongly-ordered I/O, treating the access as a full-fence —
  the load or store completes in program order with respect to all
  other accesses from the same hart.

### 6.2 Memory-Mapped I/O Fences

XH-1 will provide a `FENCE` variant (the standard `FENCE pred=rw,
succ=rw, fm=0`) that software uses to order I/O accesses. The
microarchitecture must ensure the drain endpoint includes the
device-port queue, not just the L1/L2.

### 6.3 Interrupt Acknowledgment and Ordering

RISC-V interrupt acknowledgment is itself a memory-mapped I/O
write. The microarchitecture must guarantee that the write to the
*Platform-Level Interrupt Controller* (PLIC) is ordered with respect
to the I/O operations it is acknowledging. The standard idiom is a
`FENCE` before the claim/complete operation. XH-1's PLIC integration
must be designed so that the `FENCE` does not require extra
microarchitectural support beyond the standard fence.

> **Open:** Whether XH-1 implements a **"fence fast path"** for
> accesses to specific device memory regions (e.g., the PLIC) that
> bypasses the directory and uses a dedicated sideband signal.
> Such optimizations can reduce interrupt latency by tens of cycles
> but complicate coherence reasoning. Not recommended for the first
> silicon.

---

## 7. Coherence Interaction

XH-1's coherence protocol (described in the coherence research
document) is directory-based, MSI or MESI, with a distributed
directory across the L2 slices. Memory ordering depends on the
coherence protocol's notion of "completion."

For a store to be considered "globally performed":

- The line is held in **Modified** state by the requesting core.
- All prior sharers have sent Invalidate Acknowledgments to the
  directory (or the directory has confirmed there were no sharers).
- The directory has logged the new owner.

For a load to be considered "globally performed":

- The line is held in **Shared** (or **Modified** if exclusive) by
  the requesting core.
- The data response from the directory (or forwarded from a peer) is
  the value the directory currently authorizes.

> **Important subtlety:** A `FENCE W,W` between two stores on the
> same hart must ensure that store 1 has acquired its M grant *and*
> the directory has processed the invalidations *before* store 2
> acquires its M grant. In directory terms, store 2 cannot be issued
> until store 1's M grant has been observed at the directory. XH-1
> must implement this as an ordering dependency at the directory's
> L2 controller, not as a per-line serialization at the directory
> (which would be too coarse and too slow).

### 7.1 Coherence-Induced Ordering Pitfalls

Common pitfalls in directory-based designs:

- **Directory ack starvation under fence pressure:** when many cores
  fence simultaneously, the directory must serialize invalidation
  acknowledgments without deadlock.
- **Coherence downgrade during fence:** a core that downgrades M→S
  while a fence is in progress must not let younger loads proceed
  past the fence.
- **Silent upgrade (no-invalidate transition) on store:** a clean
  line upgraded to M without invalidation must still be ordered
  with respect to other cores' loads that observed the line in S.
  The directory must explicitly synchronize this transition.

### 7.2 Ordering at the Interconnect

XH-1's interconnect is a mesh (proposed; see interconnect research
document). The mesh delivers messages in-order *between a given pair
of routers*, but globally the order is not guaranteed. The directory
controller is the serialization point.

For ordering, XH-1's mesh must:

- Not reorder different message classes on the same virtual channel.
- Support a "tail" or "credit" signal so that a fence-acknowledgment
  message can be generated only when all prior messages on the same
  path have been delivered.

---

## 8. Implementation Approaches Compared

This section lists the realistic implementation strategies for an
in-order XH-1 core. The choice of in-order vs. out-of-order pipeline
is discussed in §9.1; here we focus on ordering-specific mechanisms.

### 8.1 Approach A: Conservative Per-Core Fence (Baseline)

- The LSU drains the relevant queue classes fully on every `FENCE`.
- The store buffer's "drain complete" is defined as "every store has
  received its directory M grant."
- A `FENCE` is implemented as a special instruction that stalls the
  pipeline until drain completes.
- Acquire/release AMOs are implemented as full `FENCE`s around the
  AMO at the LSU.
- `SFENCE.VMA` stalls the pipeline until the store buffer drains.

**Pros:**

- Simple to verify. The fence semantics maps directly to "queue
  empty."
- Predictable performance for small core counts.
- Easy to reason about coherence interactions.

**Cons:**

- Poor scalability. At 128 cores, the worst-case `FENCE` latency
  grows because the drain point is global, not local.
- Overly conservative for `aq`/`rl` AMOs, which do not require a
  full global drain on either side — only ordering relative to
  surrounding ordinary accesses.

### 8.2 Approach B: Token-Based Completion

- The LSU attaches a monotonically increasing **completion token**
  to each outstanding operation.
- A `FENCE` with pred P, succ S becomes: "wait until all operations
  of type P issued before this `FENCE` have produced a token greater
  than or equal to the fence's token."
- The token is generated when the directory acknowledges the
  operation; this is exactly the "global completion" notion.
- Acquire/Release is implemented as a fence of the appropriate
  type around the AMO at the LSU.

**Pros:**

- Fences do not require the entire store buffer to drain; an empty
  store buffer is sufficient, but so is a store buffer where all
  older stores have completed.
- Naturally parallel: many stores can be outstanding, and the
  fence only waits for the *oldest* of them.
- Slightly more complex but a well-understood technique (used in
  academic and commercial cores).

**Cons:**

- Per-message token bookkeeping is more expensive than a simple
  "queue empty" check.
- Verification requires modeling tokens at the testbench, which is
  a non-trivial simulation infrastructure investment.

### 8.3 Approach C: Directory-Assisted Ordering

- A special "fence-ack" message class is added to the coherence
  protocol.
- A `FENCE` instruction sends a fence-ack request to the directory;
  the directory replies only after all prior messages from the
  issuing core have been serialized.
- The directory tracks per-core "last-seen" sequence numbers to
  arbitrate fence-ack requests against ongoing traffic.

**Pros:**

- The directory is the natural serialization point; leveraging it
  avoids per-core reinvention of global ordering.
- Scales well to large core counts because the directory already
  handles 128-core traffic.
- Allows the per-core LSU to continue issuing younger operations
  while the fence is in progress, as long as they are ordered
  correctly.

**Cons:**

- Adds protocol complexity (a new message class, directory state,
  ack/grant logic).
- Risk of fence-ack starvation under heavy traffic.
- Requires careful design to avoid deadlock with normal coherence
  traffic (classic message-class-dependency cycle).

### 8.4 Approach D: Speculative OoO + MOB

This is the out-of-order approach. The LSU becomes a Memory
Ordering Buffer (MOB) that tracks per-operation completion and
issues fences by tracking the "oldest in-flight" of each class.
This is how high-performance OoO cores implement fences; the
trade-off is pipeline depth and complexity.

XH-1 is proposed as **in-order** for power and predictability; see
§9.1. OoO is listed for completeness.

**Pros:**

- Highest performance on workloads with deep memory-level
  parallelism.
- Fences can be "free" if no relevant older operation is in flight.

**Cons:**

- Significantly larger area and power.
- More verification effort, especially for the MOB's speculative
  reordering.
- Diminishing returns on memory-bound workloads at 128 cores (the
  bottleneck is the interconnect, not the per-core issue rate).

### 8.5 Summary of Approaches

| Approach | Perf (single) | Perf (128) | Area | Verif. effort | Risk |
| -------- | ------------- | ---------- | ---- | ------------- | ---- |
| A: Cons. | Low | Lowest | Lowest | Lowest | Low |
| B: Token | Medium | Medium | Low | Medium | Low |
| C: Dir-Assisted | Medium-high | High | Medium | High | Medium |
| D: OoO+MOB | High | High | High | Highest | High |

> **Proposed XH-1 implementation:** **Approach C (Directory-Assisted
> Ordering)** as the primary, with **Approach B (Token-Based
> Completion)** as the per-core implementation strategy. The
> directory handles global serialization, the LSU handles local
> token tracking. This combination is the most natural for a
> directory-coherent 128-core design.

---

## 9. Interactions with the Pipeline and Other Subsystems

### 9.1 Pipeline Choice (In-Order vs. Out-of-Order)

XH-1 is **[Proposed] in-order** for the first silicon. The reasons:

- Power envelope. 128 cores at high frequency is a thermal
  challenge; an in-order core can run cooler.
- Predictable timing. Real-time and safety-critical workloads
  benefit from deterministic `FENCE` and `FENCE.I` latencies.
- Verification cost. An OoO core's MOB is a major verification
  burden; a 128-core OoO design is a multi-year project.

An in-order pipeline interacts with ordering as follows:

- The LSU is non-speculative on the store side (stores are
  committed when the integer pipeline commits them) and lightly
  speculative on the load side (loads may issue before older
  stores, with a structural dependency check at commit).
- `FENCE` stalls the pipeline until drain; the cost is a few tens
  to a few hundred cycles depending on directory state.
- Acquire/release AMOs generate local fence events in the LSU.

An OoO pipeline would:

- Issue stores from the STB to L1, with the MOB tracking
  completion.
- Issue loads speculatively past stores with prediction; the MOB
  validates at commit.
- `FENCE` becomes a "wait for oldest in-flight of pred class to
  complete" check; usually very fast.

> **Open:** Whether a future XH-2 could be OoO. The research
> project is not yet at that question.

### 9.2 Cache Hierarchy

Memory ordering is shaped by the cache hierarchy:

- **L1 D-cache:** the load-side ordering reference. A load that hits
  in L1 sees a value that is at least as recent as the cache's
  coherence state indicates. A load that misses in L1 must go to
  L2, where the directory arbitrates the order.
- **L2 slice (per tile):** home of the directory entry; serializes
  ordering for the addresses it owns.
- **L3 (shared, distributed):** a victim cache and last-level
  coherence point. XH-1's directory is the L2 directory; L3 is
  treated as a backing store. Invalidation ordering for XH-1
  therefore ends at L2.
- **L1 I-cache:** the `FENCE.I` target. The I-cache must be flushed
  or invalidated to ensure stores are visible.

The cache hierarchy directly affects `FENCE` latency because the
"completion" point is the L2 directory. A `FENCE W,W` in a system
with L1-only ordering would be much cheaper (a few cycles) but
incorrect; the L2 directory must be in the loop.

### 9.3 Memory System

The memory system (DRAM controllers, on-chip memory, scratchpads)
must respect the same ordering. For XH-1, accesses to on-chip
scratchpad memory bypass the coherence protocol entirely; they
are ordered by the scratchpad controller.

> **Open:** Whether XH-1 has a per-tile scratchpad or a shared
> on-chip SRAM region. If shared, ordering between scratchpad and
  cache-coherent regions is a research question.

### 9.4 Interconnect

The mesh interconnect has ordering properties (per-link in-order,
globally non-ordered) that determine what the directory and
endpoints must do to maintain memory ordering. The "virtual
channel" abstraction in the mesh is the unit of ordering; XH-1
**[Proposed]** uses:

- **VC0:** Coherence requests (in-order).
- **VC1:** Coherence responses (in-order).
- **VC2:** Data messages (in-order).
- **VC3:** Fence-ack and other ordering-sensitive messages
  (in-order).

This separation avoids deadlock and simplifies ordering reasoning.

### 9.5 Interrupts

RISC-V interrupts are delivered to a hart based on the *global
interrupt-enable* bits and the PLIC. Ordering constraints:

- A hart that disables interrupts and then writes to a shared
  variable does not need a `FENCE` before the interrupt-disable
  write; the write to `mstatus` or `sstatus` is itself a CSR write
  and is ordered architecturally.
- A hart that claims an interrupt at the PLIC and then reads a
  device register must `FENCE` between the claim and the read to
  ensure the claim is globally visible (the claim is a load from
  the PLIC's claim register; the read is a load to the device).
- The interrupt-return path uses `mret`/`sret`, which is
  architecturally ordered with respect to prior CSR writes.
- Trap delivery and `xRET` do not include implicit memory fences;
  the OS must insert them.

XH-1's interrupt controller integration must ensure that the
interrupt-acknowledge signal is recognized only after the
relevant memory accesses are ordered.

### 9.6 Operating System

The OS uses `FENCE` heavily:

- **Spinlocks and mutexes:** the lock acquire is typically a
  `LR`/`SC` sequence; the lock release is an `SC` or a `FENCE`
  followed by a `SW` to the lock variable.
- **Page-table manipulation:** `SFENCE.VMA` after every PTE write.
- **I/O:** `FENCE` before/after device accesses (see §6).
- **Cross-core TLB shootdown:** an IPI sender writes the
  invalidation request to a shared memory location, then sends an
  IPI, and the receiver must see the request after the IPI — this
  requires a `FENCE` on the sender side.
- **RCU grace periods:** RCU relies on the memory model's
  guarantees; a weak model like RVWMO requires explicit
  `FENCE` at grace-period boundaries.
- **Boot and SMP startup:** the boot hart must `FENCE` before
  starting secondary harts to ensure the trampoline code and data
  are visible.

XH-1's OS port (Linux, or a research kernel) must insert `FENCE`
at all these sites. The cost of a `FENCE` in a 128-core system
makes the boot-time and idle-time costs small in absolute terms
but potentially large in *relative* terms; an in-order core that
spends 200 cycles per `FENCE` will spend a noticeable fraction of
its time in fences if it idles on a spinlock.

> **Open:** Whether XH-1 ports a vanilla Linux or a research
> kernel. The memory-ordering analysis in this document assumes
> vanilla Linux semantics but applies in both cases.

### 9.7 Verification

Memory ordering is famously difficult to verify because:

- The bug is typically a very rare, non-deterministic interleaving.
- The legal reorderings are many; enumerating them is hard.
- The coherence protocol and the memory model interact in subtle
  ways.

XH-1's verification strategy **[Proposed]**:

1. **Litmus tests** for the memory model. Run a corpus of litmus
   tests (based on the RISC-V memory-model litmus test suite, plus
   XH-1-specific tests) on RTL simulation. The tests must include
   the published RVWMO tests and tests specific to XH-1's
   coherence protocol.
2. **Coherence protocol model checking.** Use a formal model
   (e.g., Murphi, TLA+) of the directory protocol; prove
   invariant properties.
3. **Pipeline formal verification.** Verify the LSU's local
   reordering rules with bounded model checking.
4. **System-level simulation.** Boot Linux on the RTL or a
   high-level model; run stress workloads (e.g., `perf bench
   futex`, RCU torture test).
5. **Stress with random delay injection.** Vary latency on each
   interconnect path randomly to expose ordering bugs.

The published **herd7** tool from Cambridge and the **diy7**
litmus-test framework are the standard tools for RVWMO
verification. **Operational models** like Zam are more amenable
to formal reasoning than axiomatic models; XH-1 should
**[Proposed]** support both for verification.

> **Open:** Whether XH-1's verification flow includes formal
> proofs of the memory ordering contract end-to-end, or whether
> simulation-based testing is the primary line of defense. A
> research project may choose the latter; a commercial project
> should aim for the former.

### 9.8 Performance

Memory ordering has two performance axes:

1. **Steady-state throughput:** how fast can the system process
   ordinary memory traffic. This is dominated by coherence and
   cache behavior, not by ordering per se.
2. **Fence/atomic latency and throughput:** how fast is a
   synchronization primitive. This is where ordering directly
   impacts performance.

For a 128-core system, the second axis is the more important
contributor to scalability. A workload with frequent
synchronization (e.g., a parallel database, a many-core
machine-learning inference engine) is dominated by `FENCE` and
AMO latency.

> **Quantitative note (rough estimate):** Assuming a 1 GHz in-order
> core, a directory-coherent L2 slice with a hop latency of ~10 ns
> per mesh traversal, and an average of 3–5 hops per coherence
> transaction, a single uncontended `FENCE` on a hot line might
> take ~30–50 ns (30–50 cycles). A contended `FENCE` (multiple
> sharers) requiring invalidation acks could take 100–300 ns
> (100–300 cycles). On a 128-core workload that fences once per
> critical section, this is the dominant cost.

XH-1's performance evaluation **[Proposed]**:

- Measure `FENCE` latency distribution under varying contention
  and across the mesh (corner vs. center tile).
- Measure AMO latency under contention.
- Measure the effect of `aq`/`rl` vs. explicit `FENCE` in the
  OS.
- Compare against an ideal "no ordering cost" baseline.

---

## 10. Specific RISC-V Considerations

### 10.1 The `Ztso` Extension (Optional)

`Ztso` provides TSO (Total Store Order) at every hart. With
`Ztso`, the implementation guarantees that stores are not
reordered with respect to subsequent loads, simplifying the
software model.

> **Proposed XH-1 stance:** Do not advertise `Ztso` for the
> first silicon. The hardware cost of supporting TSO at every
> hart is a tighter store-load ordering in the LSU, and the
> benefit is only felt by software that is willing to be
> incompatible with weak-ordering implementations. The standard
> RISC-V Linux kernel is RVWMO-compatible and does not require
> `Ztso`. Re-evaluate if a major software target requires it.

### 10.2 The `Zam` Extension (If Adopted)

`Zam` is a proposed extension that provides an operational
memory model in addition to the axiomatic RVWMO. It does not
change the contract; it provides operational clarity. XH-1
**[Proposed]** will document its memory model in terms of `Zam`
where possible because operational models are easier to
implement and verify.

### 10.3 The PTE-Based Memory Type (PBMT)

XH-1's MMU must support the PBMT field and route accesses to
the appropriate ordering class. Specifically:

- **PMA = NC (non-cacheable):** accesses go to the device port;
  load/store ordering is enforced by the device-port controller.
- **PMA = IO (weakly ordered I/O):** same as NC, but with
  explicit reordering rules.
- **PMA = WEAK (cacheable but weakly ordered):** accesses are
  cached but ordered weakly; useful for write-combining buffers.
- **PMA = STRONG (cacheable, strongly ordered):** same as
  cacheable but with fence semantics on every access; rarely
  used.

> **Open:** Whether XH-1 supports all PBMT values or a subset.
> A research project may support all four; a commercial project
> may support only NC and cacheable.

### 10.4 The `fence.i` and Instruction-Fetch Coherence

`FENCE.I` orders stores w.r.t. instruction fetches on the same
hart. Cross-hart, the software idiom is:

```c
// hart A:
write_code(addr, code);
__sync_synchronize(); // FENCE
send_ipi(hart_B);

// hart B:
handle_ipi();
__sync_synchronize(); // FENCE
__fence_i();          // FENCE.I
icache_flush();
jump_to(addr);
```

XH-1's I-cache is a VIPT cache; on an aliasing configuration
(small pages, large cache), the OS must use the recommended
sequence (or the larger page size). The architecture team will
document this.

### 10.5 TSO and RVWMO Differences for Software

Although the focus here is hardware, it is worth noting for
completeness:

- Under TSO, ordinary loads and stores have program order
  semantics. Under RVWMO, they do not.
- A program that is correct under TSO may not be correct under
  RVWMO without additional fences. The Linux kernel's
  memory-model code (`tools/memory-model/`) contains the
  required fences for RVWMO.

XH-1's port will use the RVWMO-correct fences; no special
support is required in hardware beyond honoring the standard
`FENCE`.

---

## 11. Unresolved Design Questions

The following questions remain open for the XH-1 architecture:

1. **In-order vs. out-of-order pipeline.** In-order is proposed,
   but the decision is not final. An OoO pipeline changes the
   MOB design and may improve `FENCE` latency.
2. **`Ztso` adoption.** Whether to support TSO at every hart.
3. **`Zam` adoption.** Whether to require the operational memory
   model in the specification.
4. **Coherence protocol.** MSI vs. MESI vs. MOESI. The choice
   affects the number of transitions and therefore the number
   of coherence-induced ordering edges. (See the coherence
   research document.)
5. **Interconnect topology.** Mesh vs. ring vs. hierarchical
   bus+mesh. The choice affects `FENCE` latency and the
   directory's role.
6. **PBMT support scope.** Which PTE memory types XH-1
   implements.
7. **Per-line vs. per-word reservation.** Affects `LR`/`SC`
   success rates.
8. **Fence-ack message class.** Whether to add a dedicated
   fence-ack message (Approach C) or rely entirely on
   per-core token tracking (Approach B).
9. **Verification strategy.** Simulation-based vs. formal;
   end-to-end memory model proofs.
10. **OS port.** Linux vs. a research kernel; this affects
    the set of `FENCE` sites the system must support.

---

## 12. Recommendations

Where evidence supports a recommendation, the following are
XH-1's proposed positions:

1. **Adopt RVWMO as the memory model.** This is the standard
   RISC-V model and is the most widely supported by software.
2. **Implement an in-order pipeline with Approach C
   (Directory-Assisted Ordering) and Approach B (Per-Core
   Token Completion).** This combination scales to 128 cores
   while keeping the per-core pipeline simple.
3. **Use a directory-based MESI coherence protocol at the
   L2 slice.** The directory is the natural serialization
   point for memory ordering. (See the coherence research
   document for the full argument.)
4. **Add a dedicated fence-ack message class to the coherence
   protocol.** This avoids overloading existing messages and
   reduces verification ambiguity.
5. **Honour the standard `FENCE`, `FENCE.I`, `SFENCE.VMA`,
   `LR`/`SC`, and AMO semantics.** No architectural deviations.
6. **Document the cross-hart self-modifying-code sequence in
   the programmer's manual.** Use the standard fence pair.
7. **Implement the PTE-based memory types (PBMT) for at
   least NC and cacheable.** Other types are deferred.
8. **Provide a memory-model litmus-test suite as part of
   the verification deliverables.** This is essential for
   catching ordering bugs.
9. **Measure `FENCE` and AMO latency across the mesh in
   performance evaluation.** Corner tiles have different
   latency from center tiles; the asymmetry is a real
   performance concern.

---

## 13. References

The following sources informed this document. URLs and
publication details are given where known; the reader should
verify currency, as the RISC-V ecosystem is rapidly evolving.

- RISC-V International, *RISC-V Instruction Set Manual,
  Volume I: Unprivileged ISA*. Appendix A, "RVWMO: The RISC-V
  Weak Memory Ordering Model."
- RISC-V International, *RISC-V Instruction Set Manual,
  Volume II: Privileged ISA*. Sections on fence and memory
  types (PBMT).
- Pulte, C., et al. *"Zam: A Complete and Verifiable
  Operational Memory Model for RISC-V."* (Reference to the
  operational memory model paper.)
- Owens, S., Sarkar, S., Sewell, P. *"A Better x86 Memory
  Model: x86-TSO."* (TSO reference; the TSO model is the
  inspiration for `Ztso`.)
- Lamport, L. *"How to Make a Multiprocessor Computer That
  Correctly Executes Multiprocess Programs."* IEEE TC, 1979.
  (The foundational sequential consistency paper.)
- Adve, S. V., Gharachorloo, K. *"Shared Memory Consistency
  Models: A Tutorial."* IEEE Computer, 1996. (Tutorial on
  memory models.)
- Martin, M. M. K., et al. *"Token Coherence."* (Token-based
  ordering reference, applicable to Approach B.)
- ACM SIGOPS Operating Systems Review. *Linux Kernel Memory
  Model Documentation.* (`tools/memory-model/` in the Linux
  kernel source tree.)
- AMD. *AMD64 Architecture Programmer's Manual*, Volume 2.
  (TSO reference for comparison.)
- Intel. *Intel 64 and IA-32 Architectures Software
  Developer's Manual*, Volume 3. (TSO reference for
  comparison.)
- ARM. *ARM Architecture Reference Manual*. (Comparison
  with ARM's weak memory model.)

Specific tool references:

- **herd7**: litmus-test framework for weak memory models,
  by Cambridge. https://github.com/herd/herdtools7
- **diy7**: companion tool for herd7.
- **Murphi**, **TLA+**: protocol model checkers used for
  coherence protocol verification.

The RISC-V Memory Model working group's published
documentation, including the operational model papers and
the litmus-test corpus, is the primary source for the
RVWMO contract. Specifics of XH-1's coherence protocol are
out of scope for this document and are addressed in the
coherence research document.

---

## 14. Document Status

This document is a **research proposal**. It does not represent
a finalized XH-1 architectural commitment. Items marked
**[Proposed]** are engineering recommendations under evaluation;
items marked **[Open]** are unresolved design questions.

The next revisions of this document should:

- Reflect decisions on the open questions in §11.
- Add quantitative `FENCE` latency measurements once RTL or
  high-level models are available.
- Incorporate results from the verification litmus-test
  suite.
- Align with the final coherence protocol choice.
- Document the OS-level `FENCE` insertion policy.

---

*End of document.*