TEST: Completed Review #2 | research/05-memory/memory-ordering.md

research/.xh1/runs/20260826T121158Z/attempts

@@ -0,0 +1,6 @@
+2026-08-26T12:12:47Z	research/05-memory/memory-ordering.md	1	research	completed
+2026-08-26T12:15:45Z	research/05-memory/memory-ordering.md	1	review	
+2026-08-26T12:16:25Z	research/05-memory/memory-ordering.md	2	revision	completed
+2026-08-26T12:19:32Z	research/05-memory/memory-ordering.md	2	review	
+2026-08-26T12:20:25Z	research/05-memory/memory-ordering.md	3	revision	completed
+2026-08-26T12:23:32Z	research/05-memory/memory-ordering.md	3	review	

research/.xh1/runs/20260826T121158Z/failed

@@ -0,0 +1 @@
+research/05-memory/memory-ordering.md

research/.xh1/runs/20260826T121158Z/research_05-memory_memory-ordering.md.current.md

@@ -0,0 +1,333 @@
+# Memory Ordering
+
+## Status
+
+Stub document. XH-1 memory ordering model has not yet been defined. The repository contains no source artifacts, RTL, ISA extension proposals, or reference implementations that establish a memory consistency model for XH-1. All quantitative or XH-1-specific claims in this document are flagged INSUFFICIENT EVIDENCE where applicable.
+
+## Abstract
+
+Memory ordering defines the rules by which loads and stores issued by one or more cores become visible to other cores in the system. For a 128-core RISC-V processor, the choice of memory model — and the hardware mechanisms that enforce it — is one of the most consequential architectural decisions, affecting correctness, performance, area, power, scalability, and software portability. This document surveys the design space, identifies the candidate models available within the RISC-V ISA framework, examines alternative hardware enforcement strategies, and frames the open questions for XH-1. No final recommendation is made because the repository does not establish pipeline style, coherency protocol, or interconnect topology, all of which constrain the memory ordering decision. A conditional tentative default is stated, contingent on the assumed constraints being correct.
+
+## Research Question
+
+What memory consistency model and hardware ordering mechanism should XH-1 adopt, given:
+
+1. A 128-core replication target
+2. The RISC-V ISA baseline
+3. The need to balance software portability against hardware implementation cost
+4. Unknown microarchitecture style (in-order vs. out-of-order is not established in the repository)
+5. The relationship to the cache coherency protocol (defined in `cache-coherency.md`, currently SOON)
+
+## Background
+
+### Memory Consistency Models — Terminology
+
+PROPOSAL — Use the following working taxonomy in this document:
+
+- **Sequential Consistency (SC)**: All cores observe a single total order of memory operations that respects each core's program order.
+- **Total Store Order (TSO)**: Loads may bypass stores from the same core; stores from one core are seen in order by all cores. Used by x86.
+- **Relaxed / Weak Memory Models**: Many orderings are allowed; fences (barriers) are required to enforce constraints. Used by ARM, RISC-V, and PowerPC.
+- **Release Consistency (RC) / RCsc / RCpc / RCO**: A family of models with different fence semantics. Used by academic designs and the basis for much of the C11/C++ memory model formalization.
+
+ASSUMPTION — This document uses "RC variants" as an umbrella term for RCsc, RCpc, and RCO. The specific semantics of each variant are not redefined here; readers are referred to the literature.
+
+### The RISC-V "RVWMO" Model
+
+FACT — The RISC-V ISA specification defines the **RISC-V Weak Memory Ordering (RVWMO)** model as the baseline. Under RVWMO:
+
+- Loads and stores to the same address from the same hart are ordered.
+- A store followed by a load to a *different* address may be reordered.
+- The `FENCE` instruction is used to enforce orderings. The base FENCE provides `pred` (predecessor) and `succ` (successor) bits covering device I/O (I), memory reads (R), and memory writes (W). Device ordering follows a stronger "Preserved over Higher Privilege and PMA" model (PostModel).
+- The optional `Ztso` extension provides **RVTSO**, which gives x86-like TSO semantics with a relaxed form of the FENCE.
+
+FACT — RVWMO and the `Ztso` extension are documented in the ratified RISC-V Unprivileged ISA specification. The repository does not state which revision of the specification is the reference for XH-1; this is an open question.
+
+ASSUMPTION — RVWMO + the Ztso optional extension is the available ISA-level model space that XH-1 must select from. (The repository does not document any custom ISA extensions, so XH-1 is assumed to stay within ratified RISC-V.)
+
+### Why Memory Ordering is Hard at 128 Cores
+
+OPEN QUESTION — The repository does not establish:
+- Whether XH-1 is a single die, multi-die package, or chiplet-based system
+- The interconnect topology (mesh, ring, crossbar, NoC, etc.)
+- Whether coherence is directory-based, broadcast-based, or hybrid
+- The private-vs-shared L2/L3 split
+- The memory-side directory or snoop filter organization
+
+These decisions have direct bearing on which ordering mechanisms are feasible. Any hardware ordering scheme (invalidation queues, store buffers, fence acknowledgments, coherence-based ordering) must be analyzed in the context of these unknowns.
+
+## Existing Approaches
+
+### Approach A — RISC-V RVWMO (Baseline)
+
+The standard weak memory model provided by the RISC-V ISA. Requires implementation of `FENCE`, `FENCE.I`, and the I/O ordering semantics.
+
+**Mechanism**: Architecture permits aggressive reordering. Hardware implementations commonly use:
+- Store buffers with forwarding
+- Load-load reordering queues
+- Coherence-based ordering: stores become visible at coherence invalidation/upgrade points
+- Fence implementation via global acknowledgment counters or by draining reorder queues
+
+INSUFFICIENT EVIDENCE — Which of these mechanisms XH-1 will use is not established by the repository.
+
+### Approach B — RVWMO + Ztso Extension
+
+Adds RVTSO. Software that requires x86-like ordering can avoid fences in many cases. Still RISC-V compliant.
+
+**Mechanism**: Stores from one hart are serialized in the interconnect; loads may still pass earlier stores to different addresses.
+
+### Approach C — Sequential Consistency (SC)
+
+All operations observed in a single global order respecting program order. Simplest software model; most expensive hardware in general.
+
+**Mechanism**: Commonly implemented via store-buffer draining on loads, broadcast-based coherence, and conservative replay mechanisms. The qualitative characterization of "very costly" is a general literature claim, not an XH-1 measurement; whether SC is specifically costly at 128 cores is workload- and topology-dependent and is treated here as a hypothesis, not a known result.
+
+### Approach D — Release Consistency variants (RCsc, RCpc, RCO)
+
+Academic models where release/acquire ordering is cheap but stronger fences are required for full memory ordering. Basis of C11/C++ model.
+
+**Mechanism**: Only release/acquire fences require acknowledgment; acquire-only loads are cheap. Alignment with C11 is a strong software benefit. These variants are not part of the ratified RISC-V ISA, so adoption would require a custom ISA extension.
+
+## Alternative Designs
+
+### Hardware Ordering Mechanisms
+
+PROPOSAL — The following mechanisms should be considered for fence/ordering enforcement once coherency and interconnect are known:
+
+1. **Invalidation-acknowledgment fences** — A fence blocks until all outstanding coherence requests have been acknowledged. Simple and common; scaling impact depends on directory hop count, aggregate invalidation traffic, and invalidation filter structure.
+
+2. **Per-core store-buffer drain fences** — A fence stalls the issuing core until its store buffer is empty and all stores have been globally observed. Predictable latency; serializes through the coherence fabric.
+
+3. **Tournament / Token-based fences** — Fences acquire a global token or wait for a "fence epoch" counter. Avoids worst-case drain, but adds global state.
+
+4. **FIFO coherence ordering** — Leverages the in-order completion of coherence transactions to provide ordering without explicit fences. Requires in-order interconnect, which conflicts with typical latency-optimized NoC designs.
+
+5. **Time-to-live / epoch schemes** — Each coherence request carries an epoch tag; ordering is enforced at the L2/L3 directory by queueing requests by epoch.
+
+6. **Speculative load reordering with rollback** — Loads can execute speculatively past stores; on conflict, the load is replayed. Common in high-performance out-of-order cores (e.g., reported in the IBM POWER literature).
+
+ASSUMPTION — XH-1's microarchitecture style is not yet established. If out-of-order, the "speculative load reordering with rollback" category becomes a candidate. If in-order, mechanism choices narrow considerably.
+
+## Comparison
+
+| Property | SC | RVWMO | RVWMO + Ztso | RC variants |
+|----------|----|----|----|----|
+| Software ease | Highest | Medium | Medium-High | Low-Medium |
+| Hardware cost (general) | Highest | Low | Low-Medium | Low |
+| Performance headroom (general) | Lowest | Highest | High | High |
+| Fence latency (general) | N/A | Variable | Variable | Variable |
+| Verification cost (general) | Lowest in principle | High | High | High |
+| C11/C++ alignment | Approximate | Approximate | Weak | Direct |
+| 128-core scalability | Poor (hypothesis) | Unmeasured | Unmeasured | Unmeasured |
+
+INSUFFICIENT EVIDENCE — Quantitative fence-latency, coherence-traffic, and scalability numbers for XH-1 are unavailable. Several rows above contain qualitative judgments rather than measured results, and the "128-core scalability" and "verification cost" rows reflect general literature consensus, not XH-1 measurements.
+
+## Advantages
+
+### RVWMO (Baseline)
+- **Lowest hardware cost** of all candidates for the base case, as a general property of the model — most orderings are simply not enforced.
+- **Maximum performance** potential, as a general property — reordering is unconstrained.
+- **Familiar** to the RISC-V software ecosystem (Linux, RISC-V GCC, LLVM), subject to the open question of whether Linux is the target OS.
+- **Standardized** — concrete compliance test suites exist externally.
+
+### RVWMO + Ztso
+- **x86-like TSO** for software that benefits from stronger ordering.
+- **Backwards compatible** — software not using Ztso still runs correctly under RVWMO.
+- **Modest hardware cost** — primarily requires store-store ordering at the coherence layer (qualitative claim, not measured for XH-1).
+
+### Release Consistency Variants
+- **Excellent fit for C11/C++** atomics — acquire/release are the most-used fence types in real code, per the C/C++ standards.
+- **Cheapest common case** — acquire and release can be implemented with lightweight mechanisms, as a general property.
+- **Hardware cost is concentrated on the rare case** of full seq_cst fences, as a general property.
+
+## Disadvantages
+
+### RVWMO (Baseline)
+- **Fence latency depends on coherence round-trip** — at 128 cores, fence latency is expected to be high because global acknowledgment must traverse the coherence fabric. Specific cycle counts: INSUFFICIENT EVIDENCE.
+- **Verification complexity is high** — many legal reorderings; the litmus-test failure surface is large. This is a general property of weak models, not an XH-1 measurement.
+- **Software burden** — kernel and runtime code must insert fences correctly.
+
+### RVWMO + Ztso
+- **Two consistency models in one chip** — adds documentation, validation, and software education cost.
+- **In-flight mixing** of ordering regimes in the same software is hard to reason about.
+
+### Release Consistency Variants
+- **Non-standard for RISC-V** — would require a custom ISA extension. Cannot be recommended without a strong software-side driver.
+- **Verification cost is high** in general because of subtle fence semantics; whether it is the "highest" among candidates is a claim about the literature, not a measured XH-1 result.
+
+### Sequential Consistency
+- **Performance cost is severe in the general case** — every load may stall on store-buffer drain; whether this is "severe at 128 cores" specifically is workload- and topology-dependent and is not established by the repository.
+- **No selective escape valve** — software cannot opt out of the strongest model.
+- **Does not match RISC-V ecosystem expectations** — surprising to RISC-V software developers, as a general observation.
+
+## XH-1 Considerations
+
+OPEN QUESTION — The following XH-1-internal questions are unresolved by the repository:
+
+1. Is XH-1's pipeline in-order or out-of-order?
+2. What is the coherence protocol (MOESI, MESI, directory-based, broadcast)?
+3. Is the interconnect a NoC, ring, or crossbar? Does it preserve in-order delivery of coherence responses?
+4. What is the expected working set of a typical hart — does streaming-store optimization matter?
+5. Does XH-1 target HPC, server, embedded, or mixed workloads? (Workload affects fence frequency.)
+6. Is XH-1 a research vehicle (where SC is acceptable for simplicity) or a product (where RVWMO compliance is mandatory)?
+7. Does the memory map use non-coherent regions (e.g., DMA, I/O) that interact with the ordering model?
+
+ASSUMPTION — If XH-1 targets RISC-V ecosystem compatibility (Linux, RISC-V GCC, RISC-V LLVM), then RVWMO is mandatory. Ztso may optionally be added for x86-software porting convenience.
+
+## 128-Core Scaling Considerations
+
+INSUFFICIENT EVIDENCE — Quantitative figures for XH-1 at 128 cores are unavailable. The qualitative observations below are hypotheses grounded in general literature, not XH-1 measurements.
+
+### Fence Latency
+- **Fence global acknowledgment** latency is bounded below by the worst-case coherence round-trip time. In a 128-core system, this can plausibly reach many tens of cycles in a NoC and higher in a ring or multi-hop topology; specific XH-1 figures: INSUFFICIENT EVIDENCE.
+- **Store-buffer drain** latency is approximately proportional to the store buffer depth and the time to invalidate all sharers. Specific XH-1 figures: INSUFFICIENT EVIDENCE.
+- **Token / epoch fences** may scale better because they avoid draining the entire coherence fabric — they synchronize at a logical epoch boundary. This is a general property, not an XH-1 measurement.
+
+### Coherence Traffic
+- **RVWMO fences** generate invalidation or acknowledgment traffic that competes with normal coherence traffic. The magnitude depends on directory organization, snoop-filter effectiveness, and share-set size.
+- **SC at 128 cores** can plausibly saturate a directory-based interconnect because every load's coherence transaction may stall pending store ordering; this is a hypothesis, not a measured XH-1 result.
+- **RC variants** concentrate traffic on the fence path only, as a general property.
+
+### Verification Scalability
+- **State-space explosion**: at 128 cores, the number of concurrently observable memory operations grows combinatorially in the unmitigated model. Memory-ordering verification is a known scaling bottleneck in the literature, and the repository does not establish what reduction or abstraction techniques XH-1 will employ.
+- **RC variants** are commonly cited as among the hardest to verify because of subtle fence interactions; whether they are hardest in the absolute sense is a literature claim, not an XH-1 measurement.
+- **SC is easiest to verify in principle** because it admits fewer legal behaviors, but worst in performance.
+
+INSUFFICIENT EVIDENCE — No quantitative fence-latency, coherence-traffic, or verification-scaling data is available for XH-1. Estimates cannot be made without knowing the interconnect, coherence protocol, and verification methodology.
+
+## Performance Considerations
+
+PROPOSAL — Performance-impact dimensions to evaluate once workload and microarchitecture are defined:
+
+1. **Steady-state IPC** for memory-bound workloads — qualitatively, RVWMO permits the most reordering and SC the least; XH-1-specific IPC: INSUFFICIENT EVIDENCE.
+2. **Fence latency** — measured as cycles from fence issue to fence completion. Dominant for synchronization-heavy code. XH-1-specific: INSUFFICIENT EVIDENCE.
+3. **Scalable synchronization** — how the model interacts with lock implementations, RCU, and atomic primitives.
+4. **Load-use latency** — whether loads can bypass earlier stores (allowed in RVWMO and TSO, not in SC).
+5. **Coherence miss rate** interaction — does the ordering model inflate or deflate cache-line ping-pong? Model-dependent; XH-1-specific: INSUFFICIENT EVIDENCE.
+
+ASSUMPTION — For HPC workloads with high-fence frequency (e.g., MPI, OpenMP, lock-heavy code), even small differences in fence latency will dominate. For server workloads with mixed locking, the cost of fences is amortized over longer critical sections. No XH-1 workload profile is documented in the repository.
+
+## Area Considerations
+
+INSUFFICIENT EVIDENCE — All area descriptions in this section are qualitative. Precise XH-1 area figures (in gates, µm², or mm²) are unavailable; no synthesis, layout, or RTL data exists in the repository.
+
+PROPOSAL — Qualitative area cost of ordering mechanisms, as a function of microarchitectural choices:
+
+| Mechanism | Approximate area cost per core (qualitative) |
+|-----------|----------------------------------------------|
+| Store buffer with forwarding | Small (few entries × cache-line width) |
+| Load-load reordering queue | Small to medium (grows with IPC and reorder depth) |
+| Fence-acknowledgment counter | Negligible (per-core counter) |
+| Global epoch / token state | Medium (one per chip, not per core) |
+| Speculative load reorder + rollback | Large (load-load queue, replay logic) |
+
+PROPOSAL — At 128 cores, per-core area costs are replicated 128 times. Global state (epoch counters, fence acknowledgers) is O(1) per chip and therefore amortizes favorably on a per-core basis, but the critical-path and clock-distribution impact of global state is not established by the repository.
+
+## Power and Energy Considerations
+
+INSUFFICIENT EVIDENCE — No per-fence, per-core, or aggregate energy figures for XH-1 are available. The earlier version of this document included a "100 pJ-per-core fence" figure that was not substantiated by any repository artifact or external citation; that figure has been removed.
+
+PROPOSAL — Qualitative energy implications, as general properties of the model classes:
+
+- **Fences** generate global coherence traffic — energy cost scales with the number of cores that must acknowledge. The aggregate chip-level energy per fence is therefore expected to grow with core count. Quantitative XH-1 values: INSUFFICIENT EVIDENCE.
+- **Speculative load reordering** wastes energy on rollback when a violation is detected, as a general property.
+- **Strong ordering models** keep more coherence state in flight (more invalidations, more retries), as a general property.
+- **RVWMO** with lightweight fences (when software permits) is the lowest-energy regime in the general case.
+
+## Implementation Considerations
+
+PROPOSAL — Implementation order of dependencies (the repository does not establish these, but the research area suggests the following):
+
+1. Define coherency protocol → `cache-coherency.md`
+2. Define interconnect topology → not yet in repository
+3. Define memory hierarchy → `memory-hierarchy.md`
+4. Define memory ordering model → this document
+5. Define atomic primitives → `atomics.md`
+
+ASSUMPTION — The Ztso extension is a small RTL delta over RVWMO. The dominant implementation cost is verification, not area. No RTL exists in the repository to substantiate even a relative area claim, so this is treated as a plausible qualitative statement rather than a measured result.
+
+OPEN QUESTION — Does XH-1 have a coherent accelerator fabric or non-coherent IO (e.g., CXL, DMA engines)? If so, the memory ordering model must define how these agents interact, and the repository does not currently do so.
+
+## Verification Considerations
+
+PROPOSAL — Verification challenges:
+
+- **Litmus tests**: A standard RISC-V litmus test suite is available externally; whether XH-1 will adopt it as-is or define a custom suite is OPEN.
+- **State-space explosion**: 128 cores × N outstanding operations produces a combinatorial unmitigated state space. The repository does not establish whether XH-1 will use state-space reduction, abstraction, bisimulation, or formal methods. Any specific XH-1 claim: INSUFFICIENT EVIDENCE.
+- **Fence semantics**: Each fence variant must be exhaustively tested at corner cases (interrupts, exceptions, MMIO).
+- **Coherence-ordering interaction**: The ordering model must be checked against the chosen coherence protocol for consistency. This check cannot be performed until the coherence protocol is defined.
+
+INSUFFICIENT EVIDENCE — No XH-1 verification infrastructure, formal spec, or litmus-test set is documented in the repository.
+
+ASSUMPTION — RVWMO is the most documented and tested model for RISC-V, per the ratified specification and ecosystem practice. Ztso is a small addition. SC and RC variants would require custom formal infrastructure. These are qualitative claims about ecosystem maturity, not XH-1-specific measurements.
+
+## Software Considerations
+
+PROPOSAL — Software-side impact, as a general property of each model:
+
+- **Linux kernel**: Designed for weak memory models with appropriate fences. Compatible with RVWMO.
+- **C11/C++ atomics**: Map cleanly onto release/acquire semantics. Compatible with RVWMO and with RC variants.
+- **x86 software ports**: Benefit from Ztso for fewer fences.
+- **OpenSHMEM, MPI shmem**: Often assume TSO-like ordering; Ztso helps.
+- **HPC codes with hand-rolled atomics**: Highly sensitive to fence latency.
+
+ASSUMPTION — XH-1 will run RISC-V Linux. The kernel's memory model expectations must be satisfied by XH-1's hardware ordering rules. The repository does not document whether Linux is in fact the target.
+
+## Recommendation
+
+INSUFFICIENT EVIDENCE to make a final recommendation.
+
+The repository does not yet establish:
+- Pipeline microarchitecture
+- Coherence protocol
+- Interconnect topology
+- Workload target
+
+A recommendation can be made only after the following documents in the memory research area are filled in: `memory-architecture.md`, `cache-coherency.md`, `memory-hierarchy.md`. Without these, any model selection is premature.
+
+PROPOSAL — Tentative default, contingent on the stated assumptions holding and explicitly subject to revision once repository context is available:
+
+1. **Baseline**: RVWMO (assumed mandatory for RISC-V ecosystem compatibility, if Linux is the target OS).
+2. **Optional add-on**: Ztso extension for x86-software porting convenience and modest hardware cost, contingent on verification cost being acceptable.
+3. **Avoid**: SC, custom RC variants, unless a strong workload or research driver emerges.
+4. **Defer to**: `cache-coherency.md` for the coherence-side ordering mechanism.
+
+ASSUMPTION — The tentative default above depends on (a) Linux being the target OS, (b) ecosystem compatibility being prioritized over x86-port convenience, and (c) the coherence and interconnect unknowns being resolvable without invalidating this default. None of these are established by the repository.
+
+## Confidence
+
+| Topic | Confidence | Reason |
+|-------|------------|--------|
+| RVWMO is the RISC-V baseline | High | Ratified ISA specification |
+| Ztso is a small optional extension | High | Ratified extension in the ISA specification |
+| SC has high implementation cost in general | High | General literature consensus |
+| SC specifically impractical at 128 cores | Low | General claim, not XH-1-measured; workload- and topology-dependent |
+| Per-fence energy scales with acknowledging-core count | Medium | Logical consequence; no measurement |
+| Quantitative per-fence energy at 128 cores | None | INSUFFICIENT EVIDENCE |
+| Verification is hardest for RC variants | Low–Medium | Literature claim, not XH-1-measured |
+| XH-1 should adopt RVWMO | Low | Repository context not yet sufficient |
+| XH-1 should adopt RVWMO + Ztso | Low | Same |
+| Ztso is a small RTL delta over RVWMO | Low–Medium | Plausible qualitative claim; no RTL to substantiate |
+
+## Open Questions
+
+1. What is XH-1's microarchitecture style (in-order vs. out-of-order)?
+2. What coherence protocol and directory organization will XH-1 use?
+3. What is the interconnect topology, and does it preserve coherence-response order?
+4. What workloads are targeted — HPC, server, embedded?
+5. Does XH-1 include non-coherent agents (DMA, CXL, accelerators)?
+6. Is XH-1 a research vehicle, a prototype, or a product?
+7. What is the expected fence frequency in target software?
+8. What is the verification methodology — formal, simulation, or hybrid?
+9. Will XH-1 use the standard RISC-V litmus-test suite or define a custom one?
+10. Does XH-1 require compatibility with x86-software-only ports? (Drives Ztso inclusion.)
+11. What fence-latency, area, and energy budgets are acceptable for the target envelope?
+12. What is the state-space reduction or formal-verification strategy, if any?
+13. Which revision of the RISC-V Unprivileged ISA specification is the reference for XH-1?
+
+## Sources
+
+INSUFFICIENT EVIDENCE — The XH-1 Research repository does not yet contain source material specific to memory ordering. No citations, papers, or measurements can be honestly attributed to XH-1.
+
+External ISA-level facts cited above (RVWMO, Ztso, FENCE instruction semantics) are drawn from the publicly ratified RISC-V Unprivileged ISA specification, which is the source of truth for the ISA-level memory model. The RISC-V ISA specification is the canonical reference; no specific paper, version, or page reference is provided because the XH-1 repository does not document which revision is being targeted.
+
+INSUFFICIENT EVIDENCE — All quantitative claims about fence latency, area, power, energy, and verification cost at the XH-1 level have been removed or flagged. No benchmarks, no measurements, no RTL data, no simulation results are available in the repository at this time.

research/.xh1/runs/20260826T121158Z/research_05-memory_memory-ordering.md.review.md

@@ -0,0 +1 @@
+

research/.xh1/runs/20260826T121158Z/research_05-memory_memory-ordering.md.round1.md

@@ -0,0 +1,319 @@
+# Memory Ordering
+
+## Status
+
+Stub document. XH-1 memory ordering model has not yet been defined. The repository contains no source artifacts, RTL, ISA extension proposals, or reference implementations that establish a memory consistency model for XH-1.
+
+## Abstract
+
+Memory ordering defines the rules by which loads and stores issued by one or more cores become visible to other cores in the system. For a 128-core RISC-V processor, the choice of memory model — and the hardware mechanisms that enforce it — is one of the most consequential architectural decisions, affecting correctness, performance, area, power, scalability, and software portability. This document surveys the design space, identifies the candidate models available within the RISC-V ISA framework, examines alternative hardware enforcement strategies, and frames the open questions for XH-1. No recommendation is made because the repository context does not yet establish pipeline style, coherency protocol, or interconnect topology — all of which constrain the memory ordering decision.
+
+## Research Question
+
+What memory consistency model and hardware ordering mechanism should XH-1 adopt, given:
+
+1. A 128-core replication target
+2. The RISC-V ISA baseline
+3. The need to balance software portability against hardware implementation cost
+4. Unknown microarchitecture style (in-order vs. out-of-order is not established in the repository)
+5. The relationship to the cache coherency protocol (defined in `cache-coherency.md`, currently SOON)
+
+## Background
+
+### Memory Consistency Models — Terminology
+
+PROPOSAL — Use the following working taxonomy in this document:
+
+- **Sequential Consistency (SC)**: All cores observe a single total order of memory operations that respects each core's program order.
+- **Total Store Order (TSO)**: Loads may bypass stores; stores from one core are seen in order by all cores. Used by x86.
+- **Relaxed / Weak Memory Models**: Many orderings are allowed; fences (barriers) are required to enforce constraints. Used by ARM, RISC-V, and PowerPC.
+- **Release Consistency (RC) / RCO / RCpc / RCsc**: A family of models with different fence semantics. Used by academic designs and the basis for much of C11/C++ memory model formalization.
+
+### The RISC-V "RVWMO" Model
+
+FACT — The RISC-V ISA specification defines the **RISC-V Weak Memory Ordering (RVWMO)** model as the baseline. Under RVWMO:
+
+- Loads and stores to the same address from the same hart are ordered.
+- A store followed by a load to a *different* address may be reordered.
+- The `FENCE` instruction is used to enforce orderings. The base FENCE provides `pred` (predecessor) and `succ` (successor) bits covering device I/O (I), memory reads (R), and memory writes (W). Device ordering follows a stronger PostModel.
+- The optional `Ztso` extension provides **RVTSO**, which gives x86-like TSO semantics with a relaxed form of the FENCE.
+
+ASSUMPTION — RVWMO + the Ztso optional extension is the available ISA-level model space that XH-1 must select from. (The repository does not document any custom ISA extensions, so XH-1 is assumed to stay within ratified RISC-V.)
+
+### Why Memory Ordering is Hard at 128 Cores
+
+OPEN QUESTION — The repository does not establish:
+- Whether XH-1 is a single die, multi-die package, or chiplet-based system
+- The interconnect topology (mesh, ring, crossbar, NoC, etc.)
+- Whether coherence is directory-based, broadcast-based, or hybrid
+- The private-vs-shared L2/L3 split
+- The memory-side directory or snoop filter organization
+
+These decisions have direct bearing on which ordering mechanisms are feasible. Any hardware ordering scheme (invalidation queues, store buffers, fence acknowledgments, coherence-based ordering) must be analyzed in the context of these unknowns.
+
+## Existing Approaches
+
+### Approach A — RISC-V RVWMO (Baseline)
+
+The standard weak memory model provided by the RISC-V ISA. Requires implementation of `FENCE`, `FENCE.I`, and the I/O ordering semantics.
+
+**Mechanism**: Architecture permits aggressive reordering. Hardware typically uses:
+- Store buffers with forwarding
+- Load-load reordering queues
+- Coherence-based ordering: stores become visible at coherence invalidation/upgrade points
+- Fence implementation via global acknowledgment counters or by draining reorder queues
+
+### Approach B — RVWMO + Ztso Extension
+
+Adds RVTSO. Software that requires x86-like ordering can avoid fences in many cases. Still RISC-V compliant.
+
+**Mechanism**: Stores from one hart are serialized in the interconnect; loads may still pass earlier stores to different addresses.
+
+### Approach C — Sequential Consistency (SC)
+
+All operations observed in a single global order respecting program order. Simplest software model; most expensive hardware.
+
+**Mechanism**: Typically requires store-buffer draining on loads, broadcast-based coherence, and conservative replay mechanisms. Very costly for 128 cores.
+
+### Approach D — Release Consistency variants (RCsc, RCpc, RCO)
+
+Academic models where release/acquire ordering is cheap but stronger fences are required for full memory ordering. Basis of C11/C++ model.
+
+**Mechanism**: Only release/acquire fences require acknowledgment; acquire-only loads are cheap. Alignment with C11 is a strong software benefit.
+
+## Alternative Designs
+
+### Hardware Ordering Mechanisms (to be evaluated once coherency and interconnect are known)
+
+PROPOSAL — The following mechanisms should be considered for fence/ordering enforcement:
+
+1. **Invalidation-acknowledgment fences** — A fence blocks until all outstanding coherence requests have been acknowledged. Simple, common, but scales poorly with directory hop count and aggregate invalidation traffic at 128 cores.
+
+2. **Per-core store-buffer drain fences** — A fence stalls the issuing core until its store buffer is empty and all stores have been globally observed. Predictable latency; serializes through the coherence fabric.
+
+3. **Tournament / Token-based fences** — Fences acquire a global token (Coherence-Next-Line style) or wait for a "fence epoch" counter. Avoids worst-case drain, but adds global state.
+
+4. **FIFO coherence ordering** — Leverages the in-order completion of coherence transactions to provide ordering without explicit fences. Requires in-order interconnect, which conflicts with typical latency-optimized NoC designs.
+
+5. **Time-to-live / epoch schemes** — Each coherence request carries an epoch tag; ordering is enforced at the L2/L3 directory by queueing requests by epoch.
+
+6. **Speculative load reordering with rollback** — Loads can execute speculatively past stores; on conflict, the load is replayed. Common in high-performance out-of-order cores (e.g., IBM POWER).
+
+ASSUMPTION — XH-1's microarchitecture style is not yet established. If out-of-order, the "speculative load reordering with rollback" category becomes a candidate. If in-order, mechanism choices narrow considerably.
+
+## Comparison
+
+| Property | SC | RVWMO | RVWMO + Ztso | RC variants |
+|----------|----|----|----|----|
+| Software ease | Highest | Medium | Medium-High | Low-Medium |
+| Hardware cost | Highest | Low | Low-Medium | Low |
+| Performance headroom | Lowest | Highest | High | High |
+| Fence latency | N/A | High | Medium | Variable |
+| Verification cost | Lowest | High | High | Highest |
+| C11/C++ alignment | Approximate | Approximate | Weak | Direct |
+| 128-core scalability | Poor | Good | Good | Good |
+
+Quantitative figures: INSUFFICIENT EVIDENCE. All numbers above are relative qualitative ratings, not benchmarked measurements.
+
+## Advantages
+
+### RVWMO (Baseline)
+- **Lowest hardware cost** of all candidates for the base case — most orderings are simply not enforced.
+- **Maximum performance** potential — reordering is unconstrained.
+- **Familiar** to RISC-V software ecosystem (Linux, RISC-V GCC, LLVM).
+- **Standardized** — concrete compliance test suite exists.
+
+### RVWMO + Ztso
+- **x86-like TSO** for software that benefits from stronger ordering.
+- **Backwards compatible** — software not using Ztso still runs correctly under RVWMO.
+- **Modest hardware cost** — primarily requires store-store ordering at the coherence layer.
+
+### Release Consistency Variants
+- **Excellent fit for C11/C++** atomics — acquire/release are the most-used fence types in real code.
+- **Cheapest common case** — acquire and release can be implemented with lightweight mechanisms.
+- **Hardware cost is concentrated on the rare case** of full seq_cst fences.
+
+## Disadvantages
+
+### RVWMO (Baseline)
+- **Fence latency is high** at 128 cores — global acknowledgment is expensive.
+- **Verification complexity is high** — many legal reorderings; the litmus-test failure surface is large.
+- **Software burden** — kernel and runtime code must insert fences correctly.
+
+### RVWMO + Ztso
+- **Two consistency models in one chip** — adds documentation, validation, and software education cost.
+- **In-flight mixing** of ordering regimes in the same software is hard to reason about.
+
+### Release Consistency Variants
+- **Non-standard for RISC-V** — would require ISA extension. Cannot be recommended without a strong software-side driver.
+- **Verification cost is the highest** of the candidates because of subtle fence semantics.
+
+### Sequential Consistency
+- **Performance cost is severe at 128 cores** — every load may stall on store-buffer drain.
+- **No selective escape valve** — software cannot opt out of the strongest model.
+- **Does not match RISC-V ecosystem expectations** — surprising to RISC-V software developers.
+
+## XH-1 Considerations
+
+OPEN QUESTION — The following XH-1-internal questions are unresolved by the repository:
+
+1. Is XH-1's pipeline in-order or out-of-order?
+2. What is the coherence protocol (MOESI, MESI, directory-based, broadcast)?
+3. Is the interconnect a NoC, ring, or crossbar? Does it preserve in-order delivery of coherence responses?
+4. What is the expected working set of a typical hart — does streaming-store optimization matter?
+5. Does XH-1 target HPC, server, embedded, or mixed workloads? (Workload affects fence frequency.)
+6. Is XH-1 a research vehicle (where SC is acceptable for simplicity) or a product (where RVWMO compliance is mandatory)?
+7. Does the memory map use non-coherent regions (e.g., DMA, I/O) that interact with the ordering model?
+
+ASSUMPTION — If XH-1 targets RISC-V ecosystem compatibility (Linux, RISC-V GCC, RISC-V LLVM), then RVWMO is mandatory. Ztso may optionally be added for x86-software porting convenience.
+
+## 128-Core Scalability
+
+PROPOSAL — Scalability analysis of each candidate at 128 cores:
+
+### Fence Latency
+- **Fence global acknowledgment** scales roughly with the worst-case coherence round-trip time. In a 128-core system, this can be many tens of cycles in a NoC, hundreds in a ring.
+- **Store-buffer drain** latency is approximately proportional to the store buffer depth and the time to invalidate all sharers.
+- **Token / epoch fences** scale better because they avoid draining the entire coherence fabric — they synchronize at a logical epoch boundary.
+
+### Coherence Traffic
+- **RVWMO fences** generate invalidation or acknowledgment traffic that competes with normal coherence traffic.
+- **SC at 128 cores** can saturate a directory-based interconnect because every load's coherence transaction may stall pending store ordering.
+- **RC variants** concentrate traffic on the fence path only.
+
+### Verification Scalability
+- **State-space explosion**: at 128 cores, the number of concurrently observable memory operations grows combinatorially. Memory-ordering verification is a known scaling bottleneck.
+- **RC variants** are hardest to verify because of subtle fence interactions.
+- **SC is easiest to verify** in principle, but worst in performance.
+
+INSUFFICIENT EVIDENCE — No quantitative fence-latency or coherence-traffic data is available for XH-1. Estimates cannot be made without knowing the interconnect.
+
+## Performance Considerations
+
+PROPOSAL — Performance-impact dimensions to evaluate:
+
+1. **Steady-state IPC** for memory-bound workloads — RVWMO best, SC worst.
+2. **Fence latency** — measured as cycles from fence issue to fence completion. Dominant for synchronization-heavy code.
+3. **Scalable synchronization** — how the model interacts with lock implementations, RCU, and atomic primitives.
+4. **Load-use latency** — whether loads can bypass earlier stores (allowed in RVWMO and TSO, not in SC).
+5. **Coherence miss rate** interaction — does the ordering model inflate or deflate cache-line ping-pong?
+
+ASSUMPTION — For HPC workloads with high-fence frequency (e.g., MPI, OpenMP, lock-heavy code), even small differences in fence latency will dominate. For server workloads with mixed locking, the cost of fences is amortized over longer critical sections.
+
+## Area Considerations
+
+PROPOSAL — Area cost of ordering mechanisms:
+
+| Mechanism | Approximate area cost per core |
+|-----------|-------------------------------|
+| Store buffer with forwarding | Small (few entries × cache-line width) |
+| Load-load reordering queue | Small to medium (grows with IPC) |
+| Fence-acknowledgment counter | Negligible (per-core counter) |
+| Global epoch / token state | Medium (one per chip, not per core) |
+| Speculative load reorder + rollback | Large (load-load queue, replay logic) |
+
+ASSUMPTION — All numbers in the table are order-of-magnitude estimates. Precise XH-1 figures: INSUFFICIENT EVIDENCE.
+
+PROPOSAL — At 128 cores, even small per-core area costs multiply by 128. Global state (epoch counters, fence acknowledgers) is O(1) and becomes attractive.
+
+## Power and Energy Considerations
+
+PROPOSAL — Energy implications:
+
+- **Fences** generate global coherence traffic — energy cost scales with the number of cores that must acknowledge.
+- **Speculative load reordering** wastes energy on rollback when a violation is detected.
+- **Strong ordering models** keep more coherence state in flight (more invalidations, more retries).
+- **RVWMO** with lightweight fences (when software permits) is the lowest-energy regime.
+
+OPEN QUESTION — The 128-core replication means the per-fence energy cost is multiplied. A fence that costs 100 pJ per core at 16 cores costs 800× at 128 cores in the worst case (if all cores must acknowledge). No published figure for XH-1 is available — INSUFFICIENT EVIDENCE.
+
+## Implementation Considerations
+
+PROPOSAL — Implementation order of dependencies (the repository does not establish these, but the research area suggests the following):
+
+1. Define coherency protocol → `cache-coherency.md`
+2. Define interconnect topology → not yet in repository
+3. Define memory hierarchy → `memory-hierarchy.md`
+4. Define memory ordering model → this document
+5. Define atomic primitives → `atomics.md`
+
+ASSUMPTION — The Ztso extension is a small RTL delta over RVWMO. The dominant implementation cost is verification, not area.
+
+OPEN QUESTION — Does XH-1 have a coherent accelerator fabric or non-coherent IO (e.g., CXL, DMA engines)? If so, the memory ordering model must define how these agents interact.
+
+## Verification Considerations
+
+PROPOSAL — Verification challenges:
+
+- **Litmus tests**: Standard RISC-V litmus test suite must be run on RTL and on model-based environments.
+- **State-space explosion**: 128 cores × N outstanding operations = combinatorial explosion. Likely requires state-space reduction or bisimulation techniques.
+- **Fence semantics**: Each fence variant must be exhaustively tested at corner cases (interrupts, exceptions, MMIO).
+- **Coherence-ordering interaction**: The ordering model must be checked against the chosen coherence protocol for consistency.
+
+INSUFFICIENT EVIDENCE — No XH-1 verification infrastructure, formal spec, or litmus-test set is documented in the repository.
+
+ASSUMPTION — RVWMO is the most documented and tested model for RISC-V. Ztso is a small addition. SC and RC variants would require custom formal infrastructure.
+
+## Software Considerations
+
+PROPOSAL — Software-side impact:
+
+- **Linux kernel**: Designed for weak memory models with appropriate fences. Compatible with RVWMO.
+- **C11/C++ atomics**: Map cleanly onto release/acquire semantics. Compatible with RVWMO and with RC variants.
+- **x86 software ports**: Benefit from Ztso for fewer fences.
+- **OpenSHMEM, MPI shmem**: Often assume TSO-like ordering; Ztso helps.
+- **HPC codes with hand-rolled atomics**: Highly sensitive to fence latency.
+
+ASSUMPTION — XH-1 will run RISC-V Linux. The kernel's memory model expectations must be satisfied by XH-1's hardware ordering rules.
+
+## Recommendation
+
+INSUFFICIENT EVIDENCE to make a final recommendation.
+
+The repository does not yet establish:
+- Pipeline microarchitecture
+- Coherence protocol
+- Interconnect topology
+- Workload target
+
+A recommendation can be made only after the following documents in the memory research area are filled in: `memory-architecture.md`, `cache-coherency.md`, `memory-hierarchy.md`. Without these, any model selection is premature.
+
+PROPOSAL — Tentative default pending more repository context:
+
+1. **Baseline**: RVWMO (mandatory for RISC-V ecosystem).
+2. **Optional add-on**: Ztso extension for x86-software porting convenience and modest hardware cost.
+3. **Avoid**: SC, custom RC variants, unless a strong workload or research driver emerges.
+4. **Defer to**: `cache-coherency.md` for the coherence-side ordering mechanism.
+
+## Confidence
+
+| Topic | Confidence | Reason |
+|-------|------------|--------|
+| RVWMO is the RISC-V baseline | High | Ratified ISA spec |
+| Ztso is a small optional extension | High | Ratified extension |
+| SC is impractical at 128 cores | Medium | General consensus in literature; XH-1-specific number not available |
+| Per-fence energy scales with core count | Medium | Logical consequence; no measurement |
+| Verification is hardest for RC variants | Medium | General consensus |
+| XH-1 should adopt RVWMO | Low | Repository context not yet sufficient |
+| XH-1 should adopt RVWMO + Ztso | Low | Same |
+
+## Open Questions
+
+1. What is XH-1's microarchitecture style (in-order vs. out-of-order)?
+2. What coherence protocol and directory organization will XH-1 use?
+3. What is the interconnect topology, and does it preserve coherence-response order?
+4. What workloads are targeted — HPC, server, embedded?
+5. Does XH-1 include non-coherent agents (DMA, CXL, accelerators)?
+6. Is XH-1 a research vehicle, a prototype, or a product?
+7. What is the expected fence frequency in target software?
+8. What is the verification methodology — formal, simulation, or hybrid?
+9. Will XH-1 use the standard RISC-V litmus-test suite or define a custom one?
+10. Does XH-1 require compatibility with x86-software-only ports? (Drives Ztso inclusion.)
+
+## Sources
+
+INSUFFICIENT EVIDENCE — The XH-1 Research repository does not yet contain source material specific to memory ordering. No citations, papers, or measurements can be honestly attributed to XH-1.
+
+External ISA-level facts cited above (RVWMO, Ztso, FENCE instruction) are drawn from the publicly ratified RISC-V ISA specification, which is the source of truth for the ISA-level memory model. No specific paper, version, or page reference is provided because the XH-1 repository does not document which revision is being targeted.
+
+No benchmarks, no measurements, no RTL data, no simulation results are available in the repository at this time.

research/.xh1/runs/20260826T121158Z/research_05-memory_memory-ordering.md.round2.md

@@ -0,0 +1,323 @@
+# Memory Ordering
+
+## Status
+
+Stub document. XH-1 memory ordering model has not yet been defined. The repository contains no source artifacts, RTL, ISA extension proposals, or reference implementations that establish a memory consistency model for XH-1.
+
+## Abstract
+
+Memory ordering defines the rules by which loads and stores issued by one or more cores become visible to other cores in the system. For a 128-core RISC-V processor, the choice of memory model — and the hardware mechanisms that enforce it — is one of the most consequential architectural decisions, affecting correctness, performance, area, power, scalability, and software portability. This document surveys the design space, identifies the candidate models available within the RISC-V ISA framework, examines alternative hardware enforcement strategies, and frames the open questions for XH-1. No recommendation is made because the repository context does not yet establish pipeline style, coherency protocol, or interconnect topology — all of which constrain the memory ordering decision.
+
+## Research Question
+
+What memory consistency model and hardware ordering mechanism should XH-1 adopt, given:
+
+1. A 128-core replication target
+2. The RISC-V ISA baseline
+3. The need to balance software portability against hardware implementation cost
+4. Unknown microarchitecture style (in-order vs. out-of-order is not established in the repository)
+5. The relationship to the cache coherency protocol (defined in `cache-coherency.md`, currently SOON)
+
+## Background
+
+### Memory Consistency Models — Terminology
+
+PROPOSAL — Use the following working taxonomy in this document:
+
+- **Sequential Consistency (SC)**: All cores observe a single total order of memory operations that respects each core's program order.
+- **Total Store Order (TSO)**: Loads may bypass stores; stores from one core are seen in order by all cores. Used by x86.
+- **Relaxed / Weak Memory Models**: Many orderings are allowed; fences (barriers) are required to enforce constraints. Used by ARM, RISC-V, and PowerPC.
+- **Release Consistency (RC) / RCO / RCpc / RCsc**: A family of models with different fence semantics. Used by academic designs and the basis for much of the C11/C++ memory model formalization.
+
+### The RISC-V "RVWMO" Model
+
+FACT — The RISC-V ISA specification defines the **RISC-V Weak Memory Ordering (RVWMO)** model as the baseline. Under RVWMO:
+
+- Loads and stores to the same address from the same hart are ordered.
+- A store followed by a load to a *different* address may be reordered.
+- The `FENCE` instruction is used to enforce orderings. The base FENCE provides `pred` (predecessor) and `succ` (successor) bits covering device I/O (I), memory reads (R), and memory writes (W). Device ordering follows a stronger "Preserved over Higher Privilege and PMA" model (PostModel).
+- The optional `Ztso` extension provides **RVTSO**, which gives x86-like TSO semantics with a relaxed form of the FENCE.
+
+ASSUMPTION — RVWMO + the Ztso optional extension is the available ISA-level model space that XH-1 must select from. (The repository does not document any custom ISA extensions, so XH-1 is assumed to stay within ratified RISC-V.)
+
+### Why Memory Ordering is Hard at 128 Cores
+
+OPEN QUESTION — The repository does not establish:
+- Whether XH-1 is a single die, multi-die package, or chiplet-based system
+- The interconnect topology (mesh, ring, crossbar, NoC, etc.)
+- Whether coherence is directory-based, broadcast-based, or hybrid
+- The private-vs-shared L2/L3 split
+- The memory-side directory or snoop filter organization
+
+These decisions have direct bearing on which ordering mechanisms are feasible. Any hardware ordering scheme (invalidation queues, store buffers, fence acknowledgments, coherence-based ordering) must be analyzed in the context of these unknowns.
+
+## Existing Approaches
+
+### Approach A — RISC-V RVWMO (Baseline)
+
+The standard weak memory model provided by the RISC-V ISA. Requires implementation of `FENCE`, `FENCE.I`, and the I/O ordering semantics.
+
+**Mechanism**: Architecture permits aggressive reordering. Hardware typically uses:
+- Store buffers with forwarding
+- Load-load reordering queues
+- Coherence-based ordering: stores become visible at coherence invalidation/upgrade points
+- Fence implementation via global acknowledgment counters or by draining reorder queues
+
+### Approach B — RVWMO + Ztso Extension
+
+Adds RVTSO. Software that requires x86-like ordering can avoid fences in many cases. Still RISC-V compliant.
+
+**Mechanism**: Stores from one hart are serialized in the interconnect; loads may still pass earlier stores to different addresses.
+
+### Approach C — Sequential Consistency (SC)
+
+All operations observed in a single global order respecting program order. Simplest software model; most expensive hardware.
+
+**Mechanism**: Typically requires store-buffer draining on loads, broadcast-based coherence, and conservative replay mechanisms. Implementation cost is high in general; whether it is "very costly" specifically at 128 cores is workload- and topology-dependent and is treated here as a hypothesis, not a known result.
+
+### Approach D — Release Consistency variants (RCsc, RCpc, RCO)
+
+Academic models where release/acquire ordering is cheap but stronger fences are required for full memory ordering. Basis of C11/C++ model.
+
+**Mechanism**: Only release/acquire fences require acknowledgment; acquire-only loads are cheap. Alignment with C11 is a strong software benefit. These variants are not part of the ratified RISC-V ISA, so adoption would require a custom ISA extension.
+
+## Alternative Designs
+
+### Hardware Ordering Mechanisms (to be evaluated once coherency and interconnect are known)
+
+PROPOSAL — The following mechanisms should be considered for fence/ordering enforcement:
+
+1. **Invalidation-acknowledgment fences** — A fence blocks until all outstanding coherence requests have been acknowledged. Simple and common; scaling impact depends on directory hop count, aggregate invalidation traffic, and invalidation filter structure.
+
+2. **Per-core store-buffer drain fences** — A fence stalls the issuing core until its store buffer is empty and all stores have been globally observed. Predictable latency; serializes through the coherence fabric.
+
+3. **Tournament / Token-based fences** — Fences acquire a global token or wait for a "fence epoch" counter. Avoids worst-case drain, but adds global state.
+
+4. **FIFO coherence ordering** — Leverages the in-order completion of coherence transactions to provide ordering without explicit fences. Requires in-order interconnect, which conflicts with typical latency-optimized NoC designs.
+
+5. **Time-to-live / epoch schemes** — Each coherence request carries an epoch tag; ordering is enforced at the L2/L3 directory by queueing requests by epoch.
+
+6. **Speculative load reordering with rollback** — Loads can execute speculatively past stores; on conflict, the load is replayed. Common in high-performance out-of-order cores (e.g., IBM POWER).
+
+ASSUMPTION — XH-1's microarchitecture style is not yet established. If out-of-order, the "speculative load reordering with rollback" category becomes a candidate. If in-order, mechanism choices narrow considerably.
+
+## Comparison
+
+| Property | SC | RVWMO | RVWMO + Ztso | RC variants |
+|----------|----|----|----|----|
+| Software ease | Highest | Medium | Medium-High | Low-Medium |
+| Hardware cost | Highest | Low | Low-Medium | Low |
+| Performance headroom | Lowest | Highest | High | High |
+| Fence latency | N/A | Variable | Variable | Variable |
+| Verification cost | Lowest in principle | High | High | High (likely highest) |
+| C11/C++ alignment | Approximate | Approximate | Weak | Direct |
+| 128-core scalability | Poor (hypothesis) | Good (claim) | Good (claim) | Good (claim) |
+
+INSUFFICIENT EVIDENCE — Quantitative fence-latency, coherence-traffic, and scalability numbers for XH-1 are unavailable. Several rows above contain qualitative judgments rather than measured results, and the "128-core scalability" and "verification cost" rows reflect general literature consensus, not XH-1 measurements.
+
+## Advantages
+
+### RVWMO (Baseline)
+- **Lowest hardware cost** of all candidates for the base case — most orderings are simply not enforced.
+- **Maximum performance** potential — reordering is unconstrained.
+- **Familiar** to RISC-V software ecosystem (Linux, RISC-V GCC, LLVM).
+- **Standardized** — concrete compliance test suite exists.
+
+### RVWMO + Ztso
+- **x86-like TSO** for software that benefits from stronger ordering.
+- **Backwards compatible** — software not using Ztso still runs correctly under RVWMO.
+- **Modest hardware cost** — primarily requires store-store ordering at the coherence layer.
+
+### Release Consistency Variants
+- **Excellent fit for C11/C++** atomics — acquire/release are the most-used fence types in real code.
+- **Cheapest common case** — acquire and release can be implemented with lightweight mechanisms.
+- **Hardware cost is concentrated on the rare case** of full seq_cst fences.
+
+## Disadvantages
+
+### RVWMO (Baseline)
+- **Fence latency depends on coherence round-trip** — at 128 cores, fence latency is expected to be high because global acknowledgment must traverse the coherence fabric. Specific cycle counts: INSUFFICIENT EVIDENCE.
+- **Verification complexity is high** — many legal reorderings; the litmus-test failure surface is large.
+- **Software burden** — kernel and runtime code must insert fences correctly.
+
+### RVWMO + Ztso
+- **Two consistency models in one chip** — adds documentation, validation, and software education cost.
+- **In-flight mixing** of ordering regimes in the same software is hard to reason about.
+
+### Release Consistency Variants
+- **Non-standard for RISC-V** — would require a custom ISA extension. Cannot be recommended without a strong software-side driver.
+- **Verification cost is high** in general because of subtle fence semantics; whether it is the "highest" among candidates is a claim about the literature, not a measured XH-1 result.
+
+### Sequential Consistency
+- **Performance cost is severe in the general case** — every load may stall on store-buffer drain; whether this is "severe at 128 cores" specifically is workload- and topology-dependent and is not established by the repository.
+- **No selective escape valve** — software cannot opt out of the strongest model.
+- **Does not match RISC-V ecosystem expectations** — surprising to RISC-V software developers.
+
+## XH-1 Considerations
+
+OPEN QUESTION — The following XH-1-internal questions are unresolved by the repository:
+
+1. Is XH-1's pipeline in-order or out-of-order?
+2. What is the coherence protocol (MOESI, MESI, directory-based, broadcast)?
+3. Is the interconnect a NoC, ring, or crossbar? Does it preserve in-order delivery of coherence responses?
+4. What is the expected working set of a typical hart — does streaming-store optimization matter?
+5. Does XH-1 target HPC, server, embedded, or mixed workloads? (Workload affects fence frequency.)
+6. Is XH-1 a research vehicle (where SC is acceptable for simplicity) or a product (where RVWMO compliance is mandatory)?
+7. Does the memory map use non-coherent regions (e.g., DMA, I/O) that interact with the ordering model?
+
+ASSUMPTION — If XH-1 targets RISC-V ecosystem compatibility (Linux, RISC-V GCC, RISC-V LLVM), then RVWMO is mandatory. Ztso may optionally be added for x86-software porting convenience.
+
+## 128-Core Scalability
+
+PROPOSAL — Scalability analysis of each candidate at 128 cores. Quantitative figures: INSUFFICIENT EVIDENCE. The qualitative observations below are hypotheses grounded in general literature, not XH-1 measurements.
+
+### Fence Latency
+- **Fence global acknowledgment** latency is bounded below by the worst-case coherence round-trip time. In a 128-core system, this can plausibly reach many tens of cycles in a NoC and higher in a ring or multi-hop topology; specific XH-1 figures: INSUFFICIENT EVIDENCE.
+- **Store-buffer drain** latency is approximately proportional to the store buffer depth and the time to invalidate all sharers.
+- **Token / epoch fences** may scale better because they avoid draining the entire coherence fabric — they synchronize at a logical epoch boundary.
+
+### Coherence Traffic
+- **RVWMO fences** generate invalidation or acknowledgment traffic that competes with normal coherence traffic. The magnitude depends on directory organization, snoop-filter effectiveness, and share-set size.
+- **SC at 128 cores** can plausibly saturate a directory-based interconnect because every load's coherence transaction may stall pending store ordering; this is a hypothesis, not a measured XH-1 result.
+- **RC variants** concentrate traffic on the fence path only.
+
+### Verification Scalability
+- **State-space explosion**: at 128 cores, the number of concurrently observable memory operations grows combinatorially in the unmitigated model. Memory-ordering verification is a known scaling bottleneck in the literature, and the repository does not establish what reduction or abstraction techniques XH-1 will employ.
+- **RC variants** are commonly cited as among the hardest to verify because of subtle fence interactions; whether they are hardest in the absolute sense is a literature claim, not an XH-1 measurement.
+- **SC is easiest to verify in principle** because it admits fewer legal behaviors, but worst in performance.
+
+INSUFFICIENT EVIDENCE — No quantitative fence-latency or coherence-traffic data is available for XH-1. Estimates cannot be made without knowing the interconnect.
+
+## Performance Considerations
+
+PROPOSAL — Performance-impact dimensions to evaluate:
+
+1. **Steady-state IPC** for memory-bound workloads — qualitatively, RVWMO permits the most reordering and SC the least; XH-1-specific IPC: INSUFFICIENT EVIDENCE.
+2. **Fence latency** — measured as cycles from fence issue to fence completion. Dominant for synchronization-heavy code. XH-1-specific: INSUFFICIENT EVIDENCE.
+3. **Scalable synchronization** — how the model interacts with lock implementations, RCU, and atomic primitives.
+4. **Load-use latency** — whether loads can bypass earlier stores (allowed in RVWMO and TSO, not in SC).
+5. **Coherence miss rate** interaction — does the ordering model inflate or deflate cache-line ping-pong? Model-dependent; XH-1-specific: INSUFFICIENT EVIDENCE.
+
+ASSUMPTION — For HPC workloads with high-fence frequency (e.g., MPI, OpenMP, lock-heavy code), even small differences in fence latency will dominate. For server workloads with mixed locking, the cost of fences is amortized over longer critical sections. No XH-1 workload profile is documented in the repository.
+
+## Area Considerations
+
+PROPOSAL — Qualitative area cost of ordering mechanisms:
+
+| Mechanism | Approximate area cost per core |
+|-----------|-------------------------------|
+| Store buffer with forwarding | Small (few entries × cache-line width) |
+| Load-load reordering queue | Small to medium (grows with IPC and reorder depth) |
+| Fence-acknowledgment counter | Negligible (per-core counter) |
+| Global epoch / token state | Medium (one per chip, not per core) |
+| Speculative load reorder + rollback | Large (load-load queue, replay logic) |
+
+INSUFFICIENT EVIDENCE — All descriptions above are qualitative. Precise XH-1 area figures (in gates, µm², or mm²) are unavailable.
+
+PROPOSAL — At 128 cores, even small per-core area costs multiply by 128. Global state (epoch counters, fence acknowledgers) is O(1) per chip and therefore becomes attractive on a per-core basis, but requires careful analysis of critical-path and clock-distribution impact, which is not established by the repository.
+
+## Power and Energy Considerations
+
+PROPOSAL — Qualitative energy implications:
+
+- **Fences** generate global coherence traffic — energy cost scales with the number of cores that must acknowledge. The aggregate chip-level energy per fence is therefore expected to grow with core count.
+- **Speculative load reordering** wastes energy on rollback when a violation is detected.
+- **Strong ordering models** keep more coherence state in flight (more invalidations, more retries).
+- **RVWMO** with lightweight fences (when software permits) is the lowest-energy regime in the general case.
+
+INSUFFICIENT EVIDENCE — The earlier text claimed that a 100 pJ-per-core fence at 16 cores costs 800× at 128 cores in the worst case (if all cores must acknowledge). This multiplication assumed 100 pJ is a measured or even a defensible per-core figure, which is not established by the repository or by a cited source. The 100 pJ figure is removed. A defensible claim is only that per-fence chip-level energy is expected to grow with the number of acknowledging cores; quantitative XH-1 values: INSUFFICIENT EVIDENCE.
+
+## Implementation Considerations
+
+PROPOSAL — Implementation order of dependencies (the repository does not establish these, but the research area suggests the following):
+
+1. Define coherency protocol → `cache-coherency.md`
+2. Define interconnect topology → not yet in repository
+3. Define memory hierarchy → `memory-hierarchy.md`
+4. Define memory ordering model → this document
+5. Define atomic primitives → `atomics.md`
+
+ASSUMPTION — The Ztso extension is a small RTL delta over RVWMO. The dominant implementation cost is verification, not area. No RTL exists in the repository to substantiate even a relative area claim, so this is treated as a plausible qualitative statement rather than a measured result.
+
+OPEN QUESTION — Does XH-1 have a coherent accelerator fabric or non-coherent IO (e.g., CXL, DMA engines)? If so, the memory ordering model must define how these agents interact, and the repository does not currently do so.
+
+## Verification Considerations
+
+PROPOSAL — Verification challenges:
+
+- **Litmus tests**: A standard RISC-V litmus test suite is available externally; whether XH-1 will adopt it as-is or define a custom suite is OPEN.
+- **State-space explosion**: 128 cores × N outstanding operations produces a combinatorial unmitigated state space. The repository does not establish whether XH-1 will use state-space reduction, abstraction, bisimulation, or formal methods. Any specific XH-1 claim: INSUFFICIENT EVIDENCE.
+- **Fence semantics**: Each fence variant must be exhaustively tested at corner cases (interrupts, exceptions, MMIO).
+- **Coherence-ordering interaction**: The ordering model must be checked against the chosen coherence protocol for consistency. This check cannot be performed until the coherence protocol is defined.
+
+INSUFFICIENT EVIDENCE — No XH-1 verification infrastructure, formal spec, or litmus-test set is documented in the repository.
+
+ASSUMPTION — RVWMO is the most documented and tested model for RISC-V. Ztso is a small addition. SC and RC variants would require custom formal infrastructure. These are qualitative claims about ecosystem maturity, not XH-1-specific measurements.
+
+## Software Considerations
+
+PROPOSAL — Software-side impact:
+
+- **Linux kernel**: Designed for weak memory models with appropriate fences. Compatible with RVWMO.
+- **C11/C++ atomics**: Map cleanly onto release/acquire semantics. Compatible with RVWMO and with RC variants.
+- **x86 software ports**: Benefit from Ztso for fewer fences.
+- **OpenSHMEM, MPI shmem**: Often assume TSO-like ordering; Ztso helps.
+- **HPC codes with hand-rolled atomics**: Highly sensitive to fence latency.
+
+ASSUMPTION — XH-1 will run RISC-V Linux. The kernel's memory model expectations must be satisfied by XH-1's hardware ordering rules. The repository does not document whether Linux is in fact the target.
+
+## Recommendation
+
+INSUFFICIENT EVIDENCE to make a final recommendation.
+
+The repository does not yet establish:
+- Pipeline microarchitecture
+- Coherence protocol
+- Interconnect topology
+- Workload target
+
+A recommendation can be made only after the following documents in the memory research area are filled in: `memory-architecture.md`, `cache-coherency.md`, `memory-hierarchy.md`. Without these, any model selection is premature.
+
+PROPOSAL — Tentative default pending more repository context:
+
+1. **Baseline**: RVWMO (mandatory for RISC-V ecosystem compatibility, if Linux is the target OS).
+2. **Optional add-on**: Ztso extension for x86-software porting convenience and modest hardware cost, contingent on verification cost being acceptable.
+3. **Avoid**: SC, custom RC variants, unless a strong workload or research driver emerges.
+4. **Defer to**: `cache-coherency.md` for the coherence-side ordering mechanism.
+
+## Confidence
+
+| Topic | Confidence | Reason |
+|-------|------------|--------|
+| RVWMO is the RISC-V baseline | High | Ratified ISA spec |
+| Ztso is a small optional extension | High | Ratified extension |
+| SC has high implementation cost | High | General literature consensus |
+| SC specifically impractical at 128 cores | Low | General claim, not XH-1-measured; workload- and topology-dependent |
+| Per-fence energy scales with acknowledging-core count | Medium | Logical consequence; no measurement |
+| Quantitative per-fence energy at 128 cores | None | INSUFFICIENT EVIDENCE |
+| Verification is hardest for RC variants | Low–Medium | Literature claim, not XH-1-measured |
+| XH-1 should adopt RVWMO | Low | Repository context not yet sufficient |
+| XH-1 should adopt RVWMO + Ztso | Low | Same |
+
+## Open Questions
+
+1. What is XH-1's microarchitecture style (in-order vs. out-of-order)?
+2. What coherence protocol and directory organization will XH-1 use?
+3. What is the interconnect topology, and does it preserve coherence-response order?
+4. What workloads are targeted — HPC, server, embedded?
+5. Does XH-1 include non-coherent agents (DMA, CXL, accelerators)?
+6. Is XH-1 a research vehicle, a prototype, or a product?
+7. What is the expected fence frequency in target software?
+8. What is the verification methodology — formal, simulation, or hybrid?
+9. Will XH-1 use the standard RISC-V litmus-test suite or define a custom one?
+10. Does XH-1 require compatibility with x86-software-only ports? (Drives Ztso inclusion.)
+11. What fence-latency, area, and energy budgets are acceptable for the target envelope?
+12. What is the state-space reduction or formal-verification strategy, if any?
+
+## Sources
+
+INSUFFICIENT EVIDENCE — The XH-1 Research repository does not yet contain source material specific to memory ordering. No citations, papers, or measurements can be honestly attributed to XH-1.
+
+External ISA-level facts cited above (RVWMO, Ztso, FENCE instruction semantics) are drawn from the publicly ratified RISC-V ISA specification, which is the source of truth for the ISA-level memory model. The RISC-V ISA specification is the canonical reference; no specific paper, version, or page reference is provided because the XH-1 repository does not document which revision is being targeted.
+
+INSUFFICIENT EVIDENCE — All quantitative claims about fence latency, area, power, energy, and verification cost at the XH-1 level have been removed or flagged. No benchmarks, no measurements, no RTL data, no simulation results are available in the repository at this time.

research/.xh1/runs/20260826T121158Z/research_05-memory_memory-ordering.md.round3.md

@@ -0,0 +1,333 @@
+# Memory Ordering
+
+## Status
+
+Stub document. XH-1 memory ordering model has not yet been defined. The repository contains no source artifacts, RTL, ISA extension proposals, or reference implementations that establish a memory consistency model for XH-1. All quantitative or XH-1-specific claims in this document are flagged INSUFFICIENT EVIDENCE where applicable.
+
+## Abstract
+
+Memory ordering defines the rules by which loads and stores issued by one or more cores become visible to other cores in the system. For a 128-core RISC-V processor, the choice of memory model — and the hardware mechanisms that enforce it — is one of the most consequential architectural decisions, affecting correctness, performance, area, power, scalability, and software portability. This document surveys the design space, identifies the candidate models available within the RISC-V ISA framework, examines alternative hardware enforcement strategies, and frames the open questions for XH-1. No final recommendation is made because the repository does not establish pipeline style, coherency protocol, or interconnect topology, all of which constrain the memory ordering decision. A conditional tentative default is stated, contingent on the assumed constraints being correct.
+
+## Research Question
+
+What memory consistency model and hardware ordering mechanism should XH-1 adopt, given:
+
+1. A 128-core replication target
+2. The RISC-V ISA baseline
+3. The need to balance software portability against hardware implementation cost
+4. Unknown microarchitecture style (in-order vs. out-of-order is not established in the repository)
+5. The relationship to the cache coherency protocol (defined in `cache-coherency.md`, currently SOON)
+
+## Background
+
+### Memory Consistency Models — Terminology
+
+PROPOSAL — Use the following working taxonomy in this document:
+
+- **Sequential Consistency (SC)**: All cores observe a single total order of memory operations that respects each core's program order.
+- **Total Store Order (TSO)**: Loads may bypass stores from the same core; stores from one core are seen in order by all cores. Used by x86.
+- **Relaxed / Weak Memory Models**: Many orderings are allowed; fences (barriers) are required to enforce constraints. Used by ARM, RISC-V, and PowerPC.
+- **Release Consistency (RC) / RCsc / RCpc / RCO**: A family of models with different fence semantics. Used by academic designs and the basis for much of the C11/C++ memory model formalization.
+
+ASSUMPTION — This document uses "RC variants" as an umbrella term for RCsc, RCpc, and RCO. The specific semantics of each variant are not redefined here; readers are referred to the literature.
+
+### The RISC-V "RVWMO" Model
+
+FACT — The RISC-V ISA specification defines the **RISC-V Weak Memory Ordering (RVWMO)** model as the baseline. Under RVWMO:
+
+- Loads and stores to the same address from the same hart are ordered.
+- A store followed by a load to a *different* address may be reordered.
+- The `FENCE` instruction is used to enforce orderings. The base FENCE provides `pred` (predecessor) and `succ` (successor) bits covering device I/O (I), memory reads (R), and memory writes (W). Device ordering follows a stronger "Preserved over Higher Privilege and PMA" model (PostModel).
+- The optional `Ztso` extension provides **RVTSO**, which gives x86-like TSO semantics with a relaxed form of the FENCE.
+
+FACT — RVWMO and the `Ztso` extension are documented in the ratified RISC-V Unprivileged ISA specification. The repository does not state which revision of the specification is the reference for XH-1; this is an open question.
+
+ASSUMPTION — RVWMO + the Ztso optional extension is the available ISA-level model space that XH-1 must select from. (The repository does not document any custom ISA extensions, so XH-1 is assumed to stay within ratified RISC-V.)
+
+### Why Memory Ordering is Hard at 128 Cores
+
+OPEN QUESTION — The repository does not establish:
+- Whether XH-1 is a single die, multi-die package, or chiplet-based system
+- The interconnect topology (mesh, ring, crossbar, NoC, etc.)
+- Whether coherence is directory-based, broadcast-based, or hybrid
+- The private-vs-shared L2/L3 split
+- The memory-side directory or snoop filter organization
+
+These decisions have direct bearing on which ordering mechanisms are feasible. Any hardware ordering scheme (invalidation queues, store buffers, fence acknowledgments, coherence-based ordering) must be analyzed in the context of these unknowns.
+
+## Existing Approaches
+
+### Approach A — RISC-V RVWMO (Baseline)
+
+The standard weak memory model provided by the RISC-V ISA. Requires implementation of `FENCE`, `FENCE.I`, and the I/O ordering semantics.
+
+**Mechanism**: Architecture permits aggressive reordering. Hardware implementations commonly use:
+- Store buffers with forwarding
+- Load-load reordering queues
+- Coherence-based ordering: stores become visible at coherence invalidation/upgrade points
+- Fence implementation via global acknowledgment counters or by draining reorder queues
+
+INSUFFICIENT EVIDENCE — Which of these mechanisms XH-1 will use is not established by the repository.
+
+### Approach B — RVWMO + Ztso Extension
+
+Adds RVTSO. Software that requires x86-like ordering can avoid fences in many cases. Still RISC-V compliant.
+
+**Mechanism**: Stores from one hart are serialized in the interconnect; loads may still pass earlier stores to different addresses.
+
+### Approach C — Sequential Consistency (SC)
+
+All operations observed in a single global order respecting program order. Simplest software model; most expensive hardware in general.
+
+**Mechanism**: Commonly implemented via store-buffer draining on loads, broadcast-based coherence, and conservative replay mechanisms. The qualitative characterization of "very costly" is a general literature claim, not an XH-1 measurement; whether SC is specifically costly at 128 cores is workload- and topology-dependent and is treated here as a hypothesis, not a known result.
+
+### Approach D — Release Consistency variants (RCsc, RCpc, RCO)
+
+Academic models where release/acquire ordering is cheap but stronger fences are required for full memory ordering. Basis of C11/C++ model.
+
+**Mechanism**: Only release/acquire fences require acknowledgment; acquire-only loads are cheap. Alignment with C11 is a strong software benefit. These variants are not part of the ratified RISC-V ISA, so adoption would require a custom ISA extension.
+
+## Alternative Designs
+
+### Hardware Ordering Mechanisms
+
+PROPOSAL — The following mechanisms should be considered for fence/ordering enforcement once coherency and interconnect are known:
+
+1. **Invalidation-acknowledgment fences** — A fence blocks until all outstanding coherence requests have been acknowledged. Simple and common; scaling impact depends on directory hop count, aggregate invalidation traffic, and invalidation filter structure.
+
+2. **Per-core store-buffer drain fences** — A fence stalls the issuing core until its store buffer is empty and all stores have been globally observed. Predictable latency; serializes through the coherence fabric.
+
+3. **Tournament / Token-based fences** — Fences acquire a global token or wait for a "fence epoch" counter. Avoids worst-case drain, but adds global state.
+
+4. **FIFO coherence ordering** — Leverages the in-order completion of coherence transactions to provide ordering without explicit fences. Requires in-order interconnect, which conflicts with typical latency-optimized NoC designs.
+
+5. **Time-to-live / epoch schemes** — Each coherence request carries an epoch tag; ordering is enforced at the L2/L3 directory by queueing requests by epoch.
+
+6. **Speculative load reordering with rollback** — Loads can execute speculatively past stores; on conflict, the load is replayed. Common in high-performance out-of-order cores (e.g., reported in the IBM POWER literature).
+
+ASSUMPTION — XH-1's microarchitecture style is not yet established. If out-of-order, the "speculative load reordering with rollback" category becomes a candidate. If in-order, mechanism choices narrow considerably.
+
+## Comparison
+
+| Property | SC | RVWMO | RVWMO + Ztso | RC variants |
+|----------|----|----|----|----|
+| Software ease | Highest | Medium | Medium-High | Low-Medium |
+| Hardware cost (general) | Highest | Low | Low-Medium | Low |
+| Performance headroom (general) | Lowest | Highest | High | High |
+| Fence latency (general) | N/A | Variable | Variable | Variable |
+| Verification cost (general) | Lowest in principle | High | High | High |
+| C11/C++ alignment | Approximate | Approximate | Weak | Direct |
+| 128-core scalability | Poor (hypothesis) | Unmeasured | Unmeasured | Unmeasured |
+
+INSUFFICIENT EVIDENCE — Quantitative fence-latency, coherence-traffic, and scalability numbers for XH-1 are unavailable. Several rows above contain qualitative judgments rather than measured results, and the "128-core scalability" and "verification cost" rows reflect general literature consensus, not XH-1 measurements.
+
+## Advantages
+
+### RVWMO (Baseline)
+- **Lowest hardware cost** of all candidates for the base case, as a general property of the model — most orderings are simply not enforced.
+- **Maximum performance** potential, as a general property — reordering is unconstrained.
+- **Familiar** to the RISC-V software ecosystem (Linux, RISC-V GCC, LLVM), subject to the open question of whether Linux is the target OS.
+- **Standardized** — concrete compliance test suites exist externally.
+
+### RVWMO + Ztso
+- **x86-like TSO** for software that benefits from stronger ordering.
+- **Backwards compatible** — software not using Ztso still runs correctly under RVWMO.
+- **Modest hardware cost** — primarily requires store-store ordering at the coherence layer (qualitative claim, not measured for XH-1).
+
+### Release Consistency Variants
+- **Excellent fit for C11/C++** atomics — acquire/release are the most-used fence types in real code, per the C/C++ standards.
+- **Cheapest common case** — acquire and release can be implemented with lightweight mechanisms, as a general property.
+- **Hardware cost is concentrated on the rare case** of full seq_cst fences, as a general property.
+
+## Disadvantages
+
+### RVWMO (Baseline)
+- **Fence latency depends on coherence round-trip** — at 128 cores, fence latency is expected to be high because global acknowledgment must traverse the coherence fabric. Specific cycle counts: INSUFFICIENT EVIDENCE.
+- **Verification complexity is high** — many legal reorderings; the litmus-test failure surface is large. This is a general property of weak models, not an XH-1 measurement.
+- **Software burden** — kernel and runtime code must insert fences correctly.
+
+### RVWMO + Ztso
+- **Two consistency models in one chip** — adds documentation, validation, and software education cost.
+- **In-flight mixing** of ordering regimes in the same software is hard to reason about.
+
+### Release Consistency Variants
+- **Non-standard for RISC-V** — would require a custom ISA extension. Cannot be recommended without a strong software-side driver.
+- **Verification cost is high** in general because of subtle fence semantics; whether it is the "highest" among candidates is a claim about the literature, not a measured XH-1 result.
+
+### Sequential Consistency
+- **Performance cost is severe in the general case** — every load may stall on store-buffer drain; whether this is "severe at 128 cores" specifically is workload- and topology-dependent and is not established by the repository.
+- **No selective escape valve** — software cannot opt out of the strongest model.
+- **Does not match RISC-V ecosystem expectations** — surprising to RISC-V software developers, as a general observation.
+
+## XH-1 Considerations
+
+OPEN QUESTION — The following XH-1-internal questions are unresolved by the repository:
+
+1. Is XH-1's pipeline in-order or out-of-order?
+2. What is the coherence protocol (MOESI, MESI, directory-based, broadcast)?
+3. Is the interconnect a NoC, ring, or crossbar? Does it preserve in-order delivery of coherence responses?
+4. What is the expected working set of a typical hart — does streaming-store optimization matter?
+5. Does XH-1 target HPC, server, embedded, or mixed workloads? (Workload affects fence frequency.)
+6. Is XH-1 a research vehicle (where SC is acceptable for simplicity) or a product (where RVWMO compliance is mandatory)?
+7. Does the memory map use non-coherent regions (e.g., DMA, I/O) that interact with the ordering model?
+
+ASSUMPTION — If XH-1 targets RISC-V ecosystem compatibility (Linux, RISC-V GCC, RISC-V LLVM), then RVWMO is mandatory. Ztso may optionally be added for x86-software porting convenience.
+
+## 128-Core Scaling Considerations
+
+INSUFFICIENT EVIDENCE — Quantitative figures for XH-1 at 128 cores are unavailable. The qualitative observations below are hypotheses grounded in general literature, not XH-1 measurements.
+
+### Fence Latency
+- **Fence global acknowledgment** latency is bounded below by the worst-case coherence round-trip time. In a 128-core system, this can plausibly reach many tens of cycles in a NoC and higher in a ring or multi-hop topology; specific XH-1 figures: INSUFFICIENT EVIDENCE.
+- **Store-buffer drain** latency is approximately proportional to the store buffer depth and the time to invalidate all sharers. Specific XH-1 figures: INSUFFICIENT EVIDENCE.
+- **Token / epoch fences** may scale better because they avoid draining the entire coherence fabric — they synchronize at a logical epoch boundary. This is a general property, not an XH-1 measurement.
+
+### Coherence Traffic
+- **RVWMO fences** generate invalidation or acknowledgment traffic that competes with normal coherence traffic. The magnitude depends on directory organization, snoop-filter effectiveness, and share-set size.
+- **SC at 128 cores** can plausibly saturate a directory-based interconnect because every load's coherence transaction may stall pending store ordering; this is a hypothesis, not a measured XH-1 result.
+- **RC variants** concentrate traffic on the fence path only, as a general property.
+
+### Verification Scalability
+- **State-space explosion**: at 128 cores, the number of concurrently observable memory operations grows combinatorially in the unmitigated model. Memory-ordering verification is a known scaling bottleneck in the literature, and the repository does not establish what reduction or abstraction techniques XH-1 will employ.
+- **RC variants** are commonly cited as among the hardest to verify because of subtle fence interactions; whether they are hardest in the absolute sense is a literature claim, not an XH-1 measurement.
+- **SC is easiest to verify in principle** because it admits fewer legal behaviors, but worst in performance.
+
+INSUFFICIENT EVIDENCE — No quantitative fence-latency, coherence-traffic, or verification-scaling data is available for XH-1. Estimates cannot be made without knowing the interconnect, coherence protocol, and verification methodology.
+
+## Performance Considerations
+
+PROPOSAL — Performance-impact dimensions to evaluate once workload and microarchitecture are defined:
+
+1. **Steady-state IPC** for memory-bound workloads — qualitatively, RVWMO permits the most reordering and SC the least; XH-1-specific IPC: INSUFFICIENT EVIDENCE.
+2. **Fence latency** — measured as cycles from fence issue to fence completion. Dominant for synchronization-heavy code. XH-1-specific: INSUFFICIENT EVIDENCE.
+3. **Scalable synchronization** — how the model interacts with lock implementations, RCU, and atomic primitives.
+4. **Load-use latency** — whether loads can bypass earlier stores (allowed in RVWMO and TSO, not in SC).
+5. **Coherence miss rate** interaction — does the ordering model inflate or deflate cache-line ping-pong? Model-dependent; XH-1-specific: INSUFFICIENT EVIDENCE.
+
+ASSUMPTION — For HPC workloads with high-fence frequency (e.g., MPI, OpenMP, lock-heavy code), even small differences in fence latency will dominate. For server workloads with mixed locking, the cost of fences is amortized over longer critical sections. No XH-1 workload profile is documented in the repository.
+
+## Area Considerations
+
+INSUFFICIENT EVIDENCE — All area descriptions in this section are qualitative. Precise XH-1 area figures (in gates, µm², or mm²) are unavailable; no synthesis, layout, or RTL data exists in the repository.
+
+PROPOSAL — Qualitative area cost of ordering mechanisms, as a function of microarchitectural choices:
+
+| Mechanism | Approximate area cost per core (qualitative) |
+|-----------|----------------------------------------------|
+| Store buffer with forwarding | Small (few entries × cache-line width) |
+| Load-load reordering queue | Small to medium (grows with IPC and reorder depth) |
+| Fence-acknowledgment counter | Negligible (per-core counter) |
+| Global epoch / token state | Medium (one per chip, not per core) |
+| Speculative load reorder + rollback | Large (load-load queue, replay logic) |
+
+PROPOSAL — At 128 cores, per-core area costs are replicated 128 times. Global state (epoch counters, fence acknowledgers) is O(1) per chip and therefore amortizes favorably on a per-core basis, but the critical-path and clock-distribution impact of global state is not established by the repository.
+
+## Power and Energy Considerations
+
+INSUFFICIENT EVIDENCE — No per-fence, per-core, or aggregate energy figures for XH-1 are available. The earlier version of this document included a "100 pJ-per-core fence" figure that was not substantiated by any repository artifact or external citation; that figure has been removed.
+
+PROPOSAL — Qualitative energy implications, as general properties of the model classes:
+
+- **Fences** generate global coherence traffic — energy cost scales with the number of cores that must acknowledge. The aggregate chip-level energy per fence is therefore expected to grow with core count. Quantitative XH-1 values: INSUFFICIENT EVIDENCE.
+- **Speculative load reordering** wastes energy on rollback when a violation is detected, as a general property.
+- **Strong ordering models** keep more coherence state in flight (more invalidations, more retries), as a general property.
+- **RVWMO** with lightweight fences (when software permits) is the lowest-energy regime in the general case.
+
+## Implementation Considerations
+
+PROPOSAL — Implementation order of dependencies (the repository does not establish these, but the research area suggests the following):
+
+1. Define coherency protocol → `cache-coherency.md`
+2. Define interconnect topology → not yet in repository
+3. Define memory hierarchy → `memory-hierarchy.md`
+4. Define memory ordering model → this document
+5. Define atomic primitives → `atomics.md`
+
+ASSUMPTION — The Ztso extension is a small RTL delta over RVWMO. The dominant implementation cost is verification, not area. No RTL exists in the repository to substantiate even a relative area claim, so this is treated as a plausible qualitative statement rather than a measured result.
+
+OPEN QUESTION — Does XH-1 have a coherent accelerator fabric or non-coherent IO (e.g., CXL, DMA engines)? If so, the memory ordering model must define how these agents interact, and the repository does not currently do so.
+
+## Verification Considerations
+
+PROPOSAL — Verification challenges:
+
+- **Litmus tests**: A standard RISC-V litmus test suite is available externally; whether XH-1 will adopt it as-is or define a custom suite is OPEN.
+- **State-space explosion**: 128 cores × N outstanding operations produces a combinatorial unmitigated state space. The repository does not establish whether XH-1 will use state-space reduction, abstraction, bisimulation, or formal methods. Any specific XH-1 claim: INSUFFICIENT EVIDENCE.
+- **Fence semantics**: Each fence variant must be exhaustively tested at corner cases (interrupts, exceptions, MMIO).
+- **Coherence-ordering interaction**: The ordering model must be checked against the chosen coherence protocol for consistency. This check cannot be performed until the coherence protocol is defined.
+
+INSUFFICIENT EVIDENCE — No XH-1 verification infrastructure, formal spec, or litmus-test set is documented in the repository.
+
+ASSUMPTION — RVWMO is the most documented and tested model for RISC-V, per the ratified specification and ecosystem practice. Ztso is a small addition. SC and RC variants would require custom formal infrastructure. These are qualitative claims about ecosystem maturity, not XH-1-specific measurements.
+
+## Software Considerations
+
+PROPOSAL — Software-side impact, as a general property of each model:
+
+- **Linux kernel**: Designed for weak memory models with appropriate fences. Compatible with RVWMO.
+- **C11/C++ atomics**: Map cleanly onto release/acquire semantics. Compatible with RVWMO and with RC variants.
+- **x86 software ports**: Benefit from Ztso for fewer fences.
+- **OpenSHMEM, MPI shmem**: Often assume TSO-like ordering; Ztso helps.
+- **HPC codes with hand-rolled atomics**: Highly sensitive to fence latency.
+
+ASSUMPTION — XH-1 will run RISC-V Linux. The kernel's memory model expectations must be satisfied by XH-1's hardware ordering rules. The repository does not document whether Linux is in fact the target.
+
+## Recommendation
+
+INSUFFICIENT EVIDENCE to make a final recommendation.
+
+The repository does not yet establish:
+- Pipeline microarchitecture
+- Coherence protocol
+- Interconnect topology
+- Workload target
+
+A recommendation can be made only after the following documents in the memory research area are filled in: `memory-architecture.md`, `cache-coherency.md`, `memory-hierarchy.md`. Without these, any model selection is premature.
+
+PROPOSAL — Tentative default, contingent on the stated assumptions holding and explicitly subject to revision once repository context is available:
+
+1. **Baseline**: RVWMO (assumed mandatory for RISC-V ecosystem compatibility, if Linux is the target OS).
+2. **Optional add-on**: Ztso extension for x86-software porting convenience and modest hardware cost, contingent on verification cost being acceptable.
+3. **Avoid**: SC, custom RC variants, unless a strong workload or research driver emerges.
+4. **Defer to**: `cache-coherency.md` for the coherence-side ordering mechanism.
+
+ASSUMPTION — The tentative default above depends on (a) Linux being the target OS, (b) ecosystem compatibility being prioritized over x86-port convenience, and (c) the coherence and interconnect unknowns being resolvable without invalidating this default. None of these are established by the repository.
+
+## Confidence
+
+| Topic | Confidence | Reason |
+|-------|------------|--------|
+| RVWMO is the RISC-V baseline | High | Ratified ISA specification |
+| Ztso is a small optional extension | High | Ratified extension in the ISA specification |
+| SC has high implementation cost in general | High | General literature consensus |
+| SC specifically impractical at 128 cores | Low | General claim, not XH-1-measured; workload- and topology-dependent |
+| Per-fence energy scales with acknowledging-core count | Medium | Logical consequence; no measurement |
+| Quantitative per-fence energy at 128 cores | None | INSUFFICIENT EVIDENCE |
+| Verification is hardest for RC variants | Low–Medium | Literature claim, not XH-1-measured |
+| XH-1 should adopt RVWMO | Low | Repository context not yet sufficient |
+| XH-1 should adopt RVWMO + Ztso | Low | Same |
+| Ztso is a small RTL delta over RVWMO | Low–Medium | Plausible qualitative claim; no RTL to substantiate |
+
+## Open Questions
+
+1. What is XH-1's microarchitecture style (in-order vs. out-of-order)?
+2. What coherence protocol and directory organization will XH-1 use?
+3. What is the interconnect topology, and does it preserve coherence-response order?
+4. What workloads are targeted — HPC, server, embedded?
+5. Does XH-1 include non-coherent agents (DMA, CXL, accelerators)?
+6. Is XH-1 a research vehicle, a prototype, or a product?
+7. What is the expected fence frequency in target software?
+8. What is the verification methodology — formal, simulation, or hybrid?
+9. Will XH-1 use the standard RISC-V litmus-test suite or define a custom one?
+10. Does XH-1 require compatibility with x86-software-only ports? (Drives Ztso inclusion.)
+11. What fence-latency, area, and energy budgets are acceptable for the target envelope?
+12. What is the state-space reduction or formal-verification strategy, if any?
+13. Which revision of the RISC-V Unprivileged ISA specification is the reference for XH-1?
+
+## Sources
+
+INSUFFICIENT EVIDENCE — The XH-1 Research repository does not yet contain source material specific to memory ordering. No citations, papers, or measurements can be honestly attributed to XH-1.
+
+External ISA-level facts cited above (RVWMO, Ztso, FENCE instruction semantics) are drawn from the publicly ratified RISC-V Unprivileged ISA specification, which is the source of truth for the ISA-level memory model. The RISC-V ISA specification is the canonical reference; no specific paper, version, or page reference is provided because the XH-1 repository does not document which revision is being targeted.
+
+INSUFFICIENT EVIDENCE — All quantitative claims about fence latency, area, power, energy, and verification cost at the XH-1 level have been removed or flagged. No benchmarks, no measurements, no RTL data, no simulation results are available in the repository at this time.