# Memory Ordering

## Status

Stub document. XH-1 memory ordering model has not yet been defined. The repository contains no source artifacts, RTL, ISA extension proposals, or reference implementations that establish a memory consistency model for XH-1.

## Abstract

Memory ordering defines the rules by which loads and stores issued by one or more cores become visible to other cores in the system. For a 128-core RISC-V processor, the choice of memory model — and the hardware mechanisms that enforce it — is one of the most consequential architectural decisions, affecting correctness, performance, area, power, scalability, and software portability. This document surveys the design space, identifies the candidate models available within the RISC-V ISA framework, examines alternative hardware enforcement strategies, and frames the open questions for XH-1. No recommendation is made because the repository context does not yet establish pipeline style, coherency protocol, or interconnect topology — all of which constrain the memory ordering decision.

## Research Question

What memory consistency model and hardware ordering mechanism should XH-1 adopt, given:

1. A 128-core replication target
2. The RISC-V ISA baseline
3. The need to balance software portability against hardware implementation cost
4. Unknown microarchitecture style (in-order vs. out-of-order is not established in the repository)
5. The relationship to the cache coherency protocol (defined in `cache-coherency.md`, currently SOON)

## Background

### Memory Consistency Models — Terminology

PROPOSAL — Use the following working taxonomy in this document:

- **Sequential Consistency (SC)**: All cores observe a single total order of memory operations that respects each core's program order.
- **Total Store Order (TSO)**: Loads may bypass stores; stores from one core are seen in order by all cores. Used by x86.
- **Relaxed / Weak Memory Models**: Many orderings are allowed; fences (barriers) are required to enforce constraints. Used by ARM, RISC-V, and PowerPC.
- **Release Consistency (RC) / RCO / RCpc / RCsc**: A family of models with different fence semantics. Used by academic designs and the basis for much of C11/C++ memory model formalization.

### The RISC-V "RVWMO" Model

FACT — The RISC-V ISA specification defines the **RISC-V Weak Memory Ordering (RVWMO)** model as the baseline. Under RVWMO:

- Loads and stores to the same address from the same hart are ordered.
- A store followed by a load to a *different* address may be reordered.
- The `FENCE` instruction is used to enforce orderings. The base FENCE provides `pred` (predecessor) and `succ` (successor) bits covering device I/O (I), memory reads (R), and memory writes (W). Device ordering follows a stronger PostModel.
- The optional `Ztso` extension provides **RVTSO**, which gives x86-like TSO semantics with a relaxed form of the FENCE.

ASSUMPTION — RVWMO + the Ztso optional extension is the available ISA-level model space that XH-1 must select from. (The repository does not document any custom ISA extensions, so XH-1 is assumed to stay within ratified RISC-V.)

### Why Memory Ordering is Hard at 128 Cores

OPEN QUESTION — The repository does not establish:
- Whether XH-1 is a single die, multi-die package, or chiplet-based system
- The interconnect topology (mesh, ring, crossbar, NoC, etc.)
- Whether coherence is directory-based, broadcast-based, or hybrid
- The private-vs-shared L2/L3 split
- The memory-side directory or snoop filter organization

These decisions have direct bearing on which ordering mechanisms are feasible. Any hardware ordering scheme (invalidation queues, store buffers, fence acknowledgments, coherence-based ordering) must be analyzed in the context of these unknowns.

## Existing Approaches

### Approach A — RISC-V RVWMO (Baseline)

The standard weak memory model provided by the RISC-V ISA. Requires implementation of `FENCE`, `FENCE.I`, and the I/O ordering semantics.

**Mechanism**: Architecture permits aggressive reordering. Hardware typically uses:
- Store buffers with forwarding
- Load-load reordering queues
- Coherence-based ordering: stores become visible at coherence invalidation/upgrade points
- Fence implementation via global acknowledgment counters or by draining reorder queues

### Approach B — RVWMO + Ztso Extension

Adds RVTSO. Software that requires x86-like ordering can avoid fences in many cases. Still RISC-V compliant.

**Mechanism**: Stores from one hart are serialized in the interconnect; loads may still pass earlier stores to different addresses.

### Approach C — Sequential Consistency (SC)

All operations observed in a single global order respecting program order. Simplest software model; most expensive hardware.

**Mechanism**: Typically requires store-buffer draining on loads, broadcast-based coherence, and conservative replay mechanisms. Very costly for 128 cores.

### Approach D — Release Consistency variants (RCsc, RCpc, RCO)

Academic models where release/acquire ordering is cheap but stronger fences are required for full memory ordering. Basis of C11/C++ model.

**Mechanism**: Only release/acquire fences require acknowledgment; acquire-only loads are cheap. Alignment with C11 is a strong software benefit.

## Alternative Designs

### Hardware Ordering Mechanisms (to be evaluated once coherency and interconnect are known)

PROPOSAL — The following mechanisms should be considered for fence/ordering enforcement:

1. **Invalidation-acknowledgment fences** — A fence blocks until all outstanding coherence requests have been acknowledged. Simple, common, but scales poorly with directory hop count and aggregate invalidation traffic at 128 cores.

2. **Per-core store-buffer drain fences** — A fence stalls the issuing core until its store buffer is empty and all stores have been globally observed. Predictable latency; serializes through the coherence fabric.

3. **Tournament / Token-based fences** — Fences acquire a global token (Coherence-Next-Line style) or wait for a "fence epoch" counter. Avoids worst-case drain, but adds global state.

4. **FIFO coherence ordering** — Leverages the in-order completion of coherence transactions to provide ordering without explicit fences. Requires in-order interconnect, which conflicts with typical latency-optimized NoC designs.

5. **Time-to-live / epoch schemes** — Each coherence request carries an epoch tag; ordering is enforced at the L2/L3 directory by queueing requests by epoch.

6. **Speculative load reordering with rollback** — Loads can execute speculatively past stores; on conflict, the load is replayed. Common in high-performance out-of-order cores (e.g., IBM POWER).

ASSUMPTION — XH-1's microarchitecture style is not yet established. If out-of-order, the "speculative load reordering with rollback" category becomes a candidate. If in-order, mechanism choices narrow considerably.

## Comparison

| Property | SC | RVWMO | RVWMO + Ztso | RC variants |
|----------|----|----|----|----|
| Software ease | Highest | Medium | Medium-High | Low-Medium |
| Hardware cost | Highest | Low | Low-Medium | Low |
| Performance headroom | Lowest | Highest | High | High |
| Fence latency | N/A | High | Medium | Variable |
| Verification cost | Lowest | High | High | Highest |
| C11/C++ alignment | Approximate | Approximate | Weak | Direct |
| 128-core scalability | Poor | Good | Good | Good |

Quantitative figures: INSUFFICIENT EVIDENCE. All numbers above are relative qualitative ratings, not benchmarked measurements.

## Advantages

### RVWMO (Baseline)
- **Lowest hardware cost** of all candidates for the base case — most orderings are simply not enforced.
- **Maximum performance** potential — reordering is unconstrained.
- **Familiar** to RISC-V software ecosystem (Linux, RISC-V GCC, LLVM).
- **Standardized** — concrete compliance test suite exists.

### RVWMO + Ztso
- **x86-like TSO** for software that benefits from stronger ordering.
- **Backwards compatible** — software not using Ztso still runs correctly under RVWMO.
- **Modest hardware cost** — primarily requires store-store ordering at the coherence layer.

### Release Consistency Variants
- **Excellent fit for C11/C++** atomics — acquire/release are the most-used fence types in real code.
- **Cheapest common case** — acquire and release can be implemented with lightweight mechanisms.
- **Hardware cost is concentrated on the rare case** of full seq_cst fences.

## Disadvantages

### RVWMO (Baseline)
- **Fence latency is high** at 128 cores — global acknowledgment is expensive.
- **Verification complexity is high** — many legal reorderings; the litmus-test failure surface is large.
- **Software burden** — kernel and runtime code must insert fences correctly.

### RVWMO + Ztso
- **Two consistency models in one chip** — adds documentation, validation, and software education cost.
- **In-flight mixing** of ordering regimes in the same software is hard to reason about.

### Release Consistency Variants
- **Non-standard for RISC-V** — would require ISA extension. Cannot be recommended without a strong software-side driver.
- **Verification cost is the highest** of the candidates because of subtle fence semantics.

### Sequential Consistency
- **Performance cost is severe at 128 cores** — every load may stall on store-buffer drain.
- **No selective escape valve** — software cannot opt out of the strongest model.
- **Does not match RISC-V ecosystem expectations** — surprising to RISC-V software developers.

## XH-1 Considerations

OPEN QUESTION — The following XH-1-internal questions are unresolved by the repository:

1. Is XH-1's pipeline in-order or out-of-order?
2. What is the coherence protocol (MOESI, MESI, directory-based, broadcast)?
3. Is the interconnect a NoC, ring, or crossbar? Does it preserve in-order delivery of coherence responses?
4. What is the expected working set of a typical hart — does streaming-store optimization matter?
5. Does XH-1 target HPC, server, embedded, or mixed workloads? (Workload affects fence frequency.)
6. Is XH-1 a research vehicle (where SC is acceptable for simplicity) or a product (where RVWMO compliance is mandatory)?
7. Does the memory map use non-coherent regions (e.g., DMA, I/O) that interact with the ordering model?

ASSUMPTION — If XH-1 targets RISC-V ecosystem compatibility (Linux, RISC-V GCC, RISC-V LLVM), then RVWMO is mandatory. Ztso may optionally be added for x86-software porting convenience.

## 128-Core Scalability

PROPOSAL — Scalability analysis of each candidate at 128 cores:

### Fence Latency
- **Fence global acknowledgment** scales roughly with the worst-case coherence round-trip time. In a 128-core system, this can be many tens of cycles in a NoC, hundreds in a ring.
- **Store-buffer drain** latency is approximately proportional to the store buffer depth and the time to invalidate all sharers.
- **Token / epoch fences** scale better because they avoid draining the entire coherence fabric — they synchronize at a logical epoch boundary.

### Coherence Traffic
- **RVWMO fences** generate invalidation or acknowledgment traffic that competes with normal coherence traffic.
- **SC at 128 cores** can saturate a directory-based interconnect because every load's coherence transaction may stall pending store ordering.
- **RC variants** concentrate traffic on the fence path only.

### Verification Scalability
- **State-space explosion**: at 128 cores, the number of concurrently observable memory operations grows combinatorially. Memory-ordering verification is a known scaling bottleneck.
- **RC variants** are hardest to verify because of subtle fence interactions.
- **SC is easiest to verify** in principle, but worst in performance.

INSUFFICIENT EVIDENCE — No quantitative fence-latency or coherence-traffic data is available for XH-1. Estimates cannot be made without knowing the interconnect.

## Performance Considerations

PROPOSAL — Performance-impact dimensions to evaluate:

1. **Steady-state IPC** for memory-bound workloads — RVWMO best, SC worst.
2. **Fence latency** — measured as cycles from fence issue to fence completion. Dominant for synchronization-heavy code.
3. **Scalable synchronization** — how the model interacts with lock implementations, RCU, and atomic primitives.
4. **Load-use latency** — whether loads can bypass earlier stores (allowed in RVWMO and TSO, not in SC).
5. **Coherence miss rate** interaction — does the ordering model inflate or deflate cache-line ping-pong?

ASSUMPTION — For HPC workloads with high-fence frequency (e.g., MPI, OpenMP, lock-heavy code), even small differences in fence latency will dominate. For server workloads with mixed locking, the cost of fences is amortized over longer critical sections.

## Area Considerations

PROPOSAL — Area cost of ordering mechanisms:

| Mechanism | Approximate area cost per core |
|-----------|-------------------------------|
| Store buffer with forwarding | Small (few entries × cache-line width) |
| Load-load reordering queue | Small to medium (grows with IPC) |
| Fence-acknowledgment counter | Negligible (per-core counter) |
| Global epoch / token state | Medium (one per chip, not per core) |
| Speculative load reorder + rollback | Large (load-load queue, replay logic) |

ASSUMPTION — All numbers in the table are order-of-magnitude estimates. Precise XH-1 figures: INSUFFICIENT EVIDENCE.

PROPOSAL — At 128 cores, even small per-core area costs multiply by 128. Global state (epoch counters, fence acknowledgers) is O(1) and becomes attractive.

## Power and Energy Considerations

PROPOSAL — Energy implications:

- **Fences** generate global coherence traffic — energy cost scales with the number of cores that must acknowledge.
- **Speculative load reordering** wastes energy on rollback when a violation is detected.
- **Strong ordering models** keep more coherence state in flight (more invalidations, more retries).
- **RVWMO** with lightweight fences (when software permits) is the lowest-energy regime.

OPEN QUESTION — The 128-core replication means the per-fence energy cost is multiplied. A fence that costs 100 pJ per core at 16 cores costs 800× at 128 cores in the worst case (if all cores must acknowledge). No published figure for XH-1 is available — INSUFFICIENT EVIDENCE.

## Implementation Considerations

PROPOSAL — Implementation order of dependencies (the repository does not establish these, but the research area suggests the following):

1. Define coherency protocol → `cache-coherency.md`
2. Define interconnect topology → not yet in repository
3. Define memory hierarchy → `memory-hierarchy.md`
4. Define memory ordering model → this document
5. Define atomic primitives → `atomics.md`

ASSUMPTION — The Ztso extension is a small RTL delta over RVWMO. The dominant implementation cost is verification, not area.

OPEN QUESTION — Does XH-1 have a coherent accelerator fabric or non-coherent IO (e.g., CXL, DMA engines)? If so, the memory ordering model must define how these agents interact.

## Verification Considerations

PROPOSAL — Verification challenges:

- **Litmus tests**: Standard RISC-V litmus test suite must be run on RTL and on model-based environments.
- **State-space explosion**: 128 cores × N outstanding operations = combinatorial explosion. Likely requires state-space reduction or bisimulation techniques.
- **Fence semantics**: Each fence variant must be exhaustively tested at corner cases (interrupts, exceptions, MMIO).
- **Coherence-ordering interaction**: The ordering model must be checked against the chosen coherence protocol for consistency.

INSUFFICIENT EVIDENCE — No XH-1 verification infrastructure, formal spec, or litmus-test set is documented in the repository.

ASSUMPTION — RVWMO is the most documented and tested model for RISC-V. Ztso is a small addition. SC and RC variants would require custom formal infrastructure.

## Software Considerations

PROPOSAL — Software-side impact:

- **Linux kernel**: Designed for weak memory models with appropriate fences. Compatible with RVWMO.
- **C11/C++ atomics**: Map cleanly onto release/acquire semantics. Compatible with RVWMO and with RC variants.
- **x86 software ports**: Benefit from Ztso for fewer fences.
- **OpenSHMEM, MPI shmem**: Often assume TSO-like ordering; Ztso helps.
- **HPC codes with hand-rolled atomics**: Highly sensitive to fence latency.

ASSUMPTION — XH-1 will run RISC-V Linux. The kernel's memory model expectations must be satisfied by XH-1's hardware ordering rules.

## Recommendation

INSUFFICIENT EVIDENCE to make a final recommendation.

The repository does not yet establish:
- Pipeline microarchitecture
- Coherence protocol
- Interconnect topology
- Workload target

A recommendation can be made only after the following documents in the memory research area are filled in: `memory-architecture.md`, `cache-coherency.md`, `memory-hierarchy.md`. Without these, any model selection is premature.

PROPOSAL — Tentative default pending more repository context:

1. **Baseline**: RVWMO (mandatory for RISC-V ecosystem).
2. **Optional add-on**: Ztso extension for x86-software porting convenience and modest hardware cost.
3. **Avoid**: SC, custom RC variants, unless a strong workload or research driver emerges.
4. **Defer to**: `cache-coherency.md` for the coherence-side ordering mechanism.

## Confidence

| Topic | Confidence | Reason |
|-------|------------|--------|
| RVWMO is the RISC-V baseline | High | Ratified ISA spec |
| Ztso is a small optional extension | High | Ratified extension |
| SC is impractical at 128 cores | Medium | General consensus in literature; XH-1-specific number not available |
| Per-fence energy scales with core count | Medium | Logical consequence; no measurement |
| Verification is hardest for RC variants | Medium | General consensus |
| XH-1 should adopt RVWMO | Low | Repository context not yet sufficient |
| XH-1 should adopt RVWMO + Ztso | Low | Same |

## Open Questions

1. What is XH-1's microarchitecture style (in-order vs. out-of-order)?
2. What coherence protocol and directory organization will XH-1 use?
3. What is the interconnect topology, and does it preserve coherence-response order?
4. What workloads are targeted — HPC, server, embedded?
5. Does XH-1 include non-coherent agents (DMA, CXL, accelerators)?
6. Is XH-1 a research vehicle, a prototype, or a product?
7. What is the expected fence frequency in target software?
8. What is the verification methodology — formal, simulation, or hybrid?
9. Will XH-1 use the standard RISC-V litmus-test suite or define a custom one?
10. Does XH-1 require compatibility with x86-software-only ports? (Drives Ztso inclusion.)

## Sources

INSUFFICIENT EVIDENCE — The XH-1 Research repository does not yet contain source material specific to memory ordering. No citations, papers, or measurements can be honestly attributed to XH-1.

External ISA-level facts cited above (RVWMO, Ztso, FENCE instruction) are drawn from the publicly ratified RISC-V ISA specification, which is the source of truth for the ISA-level memory model. No specific paper, version, or page reference is provided because the XH-1 repository does not document which revision is being targeted.

No benchmarks, no measurements, no RTL data, no simulation results are available in the repository at this time.
