
         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         

         
{"id":"gen-1787747590-x3ibYxjUYaHFxu4Pztco","object":"chat.completion","created":1787747590,"model":"minimax/minimax-m3:free","provider":"GMICloud","system_fingerprint":null,"service_tier":null,"choices":[{"index":0,"logprobs":null,"finish_reason":"stop","native_finish_reason":"stop","message":{"role":"assistant","content":"# Memory Ordering\n\n> **Status:** Research proposal for the XH-1 128-core RISC-V processor.\n> Nothing in this document is a finalized architectural commitment unless\n> explicitly stated. Items marked **[Proposed]** are engineering\n> recommendations under evaluation; items marked **[Open]** are unresolved\n> design questions.\n\n---\n\n## 1. Scope and Definitions\n\n**Memory ordering** defines the set of rules that determine when memory\noperations issued by one or more hardware threads become visible to other\nhardware threads, and in what order such visibility is required to occur.\n\nIn a single-core, single-threaded system, \"ordering\" is trivial: program\norder matches execution order matches visibility order. As soon as a\nprocessor exposes:\n\n- out-of-order execution,\n- multiple hardware threads (SMT),\n- multiple cores with private caches,\n- coherent or non-coherent interconnects, or\n- devices that perform DMA,\n\n…memory ordering becomes a first-class architectural and microarchitectural\nconcern. For a 128-core design like XH-1, it is one of the dominant\ncorrectness and performance constraints.\n\nThe RISC-V ISA delegates most ordering policy to a **Memory Consistency\nModel (MCM)** specification rather than baking it into the base ISA. The\ncurrent published MCM is documented in the RISC-V specification volume\n*“RISC-V Instruction Set Manual, Volume I: Unprivileged ISA”*, Appendix A —\n*\"RVWMO: The RISC-V Weak Memory Ordering Model\"*. RVWMO is the canonical\nmodel; \"Zam\" (the \"A\" extension candidate) is a candidate formalization\nof the same weak-ordering contract with additional operational\nclarifications.\n\nThis document covers:\n\n1. The RVWMO contract XH-1 must honor.\n2. Microarchitectural mechanisms needed to enforce that contract.\n3. Scalability concerns specific to 128 cores.\n4. Interactions with coherence, the interconnect, the pipeline, the\n   OS, and verification.\n\n---\n\n## 2. The RVWMO Contract XH-1 Must Honor\n\nRVWMO is defined by:\n\n- **Preserved Program Order (PPO):** a set of (microarchitectural)\n  orderings between pairs of memory operations from the same hart that\n  the implementation must never violate. PPO rules include the obvious\n  ones (a load following a store to the same address, register\n  dependencies, fences such as `FENCE`, `FENCE.I`, etc.) and the\n  subtler \"axiomatic\" PPO rules covering acquire/release semantics on\n  ordinary AMOs.\n- **Memory Model Axioms:** a set of forbidden load–load, load–store,\n  and store–store reorderings when no PPO rule requires otherwise\n  (e.g., \"no load may be reordered with a store to a different\n  location\" except under specific conditions).\n- **Syntactic Dependencies:** register dependencies via the\n  destination register of a load are sufficient to constrain certain\n  reorderings, mirroring the original SPARC RMO/TSO literature.\n\nThe base RVWMO does not mandate TSO; an implementation may run hart-local\nmemory accesses in any order consistent with the PPO rules, with a few\nmandated orderings (e.g., loads after stores to the *same* address are\npreserved).\n\nKey architectural levers available to software:\n\n| Construct                  | Effect (informal)                                       |\n| -------------------------- | ------------------------------------------------------- |\n| `FENCE` (various `pred`/`succ`/fm) | Orders memory operations as specified.       |\n| `FENCE.I`                  | Orders stores to instruction memory w.r.t. subsequent instruction fetches on the same hart. |\n| `FENCE.VMA`                | Orders SFENCE.VMA-style TLB management.                 |\n| `SFENCE.VMA`               | Orders stores w.r.t. subsequent TLB refills on the same hart. |\n| Acquire/release on AMOs    | Provides PPO ordering for paired RMW operations.        |\n| `LR`/`SC` (LR/SC pair)    | Ordering anchored to the reservation.                   |\n| `Zalrsc` / `Zaamo`         | Standard atomic extensions.                             |\n| `Zifencei`                 | Required for `FENCE.I`.                                 |\n| `Zihintpause`              | Hint only; no ordering effect.                          |\n| RV32/64 `fence` on devices | Device ordering via PTE bits (see §10.3).               |\n\n**XH-1 requirement:** the ISA-visible contract of every memory and\nsynchronization instruction listed in the *Unprivileged ISA* and any\nextension XH-1 implements (`A`, `Zifencei`, `Zihintpause`, `Zalrsc`,\n`Zaamo`, `Ztso` if adopted) must be honored by the combined\ncore/LSU/coherence/interconnect subsystem.\n\n> **Open:** Whether XH-1 advertises `Ztso` (an optional RVWMO relaxation\n> that grants TSO at every hart) or only RVWMO. `Ztso` simplifies\n> software reasoning but slightly constrains hardware ordering\n> flexibility. Recommendation deferred until microarchitecture is\n> fixed.\n\n---\n\n## 3. Required Fences and Their Microarchitectural Meaning\n\nThe `FENCE` instruction is parameterized by:\n\n- `pred`: which predecessor operations are ordered (R, W).\n- `succ`: which successor operations are ordered (R, W).\n- `fm` field: indicates \"fence-mode\"; value `0` is the standard fence,\n  value `>=1` is reserved for implementations and is treated as a no-op\n  by standard tools. Custom XH-1 fence-mode encodings are **[Open]**.\n\nXH-1 must ensure that a `FENCE` with predecessor set *P* and successor\nset *S*:\n\n- All memory operations of type *P* issued by this hart prior to the\n  `FENCE` are ordered before all memory operations of type *S* issued by\n  this hart after the `FENCE`, in the global memory order, to the extent\n  that the coherence protocol is willing to serialize them.\n\nMicroarchitecturally, the `FENCE` typically drains in-flight operations\nof the predecessor classes through the load/store unit (LSU), the\ncoherence point, and the write-completion path. On XH-1 with private L1\nand a shared L2 slice, \"the coherence point\" is the directory controller\non the L2 slice; see §7.\n\n---\n\n## 4. Why 128 Cores Changes the Problem\n\nA small-count many-core can often \"get away\" with conservative\nordering: a single shared bus or ring, broadcast snoops, and a\nfull-fence on every store buffer drain. At 128 cores these approaches\ndo not scale:\n\n1. **Bandwidth:** A serialized global fence requires all in-flight\n   traffic from the issuing core to drain, *and* the directory to\n   observe all acknowledgments before subsequent operations are\n   injected. With 128 cores contending for the directory and the\n   interconnect, the latency of a single `FENCE` can grow from\n   ~10–30 cycles (small system) to several hundred cycles. This\n   directly impacts lock-based critical sections, RCU grace periods,\n   and `atomic` operations in the OS.\n\n2. **Queue occupancy:** Every core holds per-message-class queues\n   (read requests, write requests, invalidations, data responses,\n   writeback data). A fence must account for occupancy of queues\n   between the core and the L2 directory *and* between directories\n   across the mesh. Without back-pressure, an XH-1 fence could\n   complete locally but leave a \"hole\" in the global order.\n\n3. **Coherence protocol overhead:** Directory-based coherence\n   (assumed for XH-1; see the coherence research document) replaces\n   broadcast snoop traffic with per-line directory lookups, but\n   introduces ack/grant ordering dependencies that the fence must\n   not skip past.\n\n4. **Scalable coherence introduces non-atomic transitions.**\n   Directory state moves from M→I→S→M through intermediate invalidation\n   acknowledgment states. A store-conditional (`SC`) on one core that\n   has been racing with a remote `LR` may observe the line in\n   intermediate states; the memory model must be specified for these.\n\n5. **Contention of fence-induced drains:** When 128 cores all execute\n   `FENCE` in a short window (e.g., a barrier in a parallel workload),\n   the interconnect can become congested with drain acknowledgments.\n   The microarchitecture must avoid feedback-loop deadlock between\n   the queues and the fence-acknowledgment path.\n\n> **Proposed XH-1 stance:** Treat memory ordering as a system-level\n> performance constraint, not just a per-core correctness one. Evaluate\n> the cost of every `FENCE` site in the OS, runtime, and synchronization\n> libraries at the system level, not at single-core simulation.\n\n---\n\n## 5. Microarchitectural Mechanisms\n\nThe classic toolkit, applied to XH-1:\n\n### 5.1 Load-Store Unit (LSU) Pipeline\n\nA typical in-order XH-1 core will have:\n\n- A **store buffer (STB)** between the integer pipeline and the L1\n  data cache.\n- A **load queue (LDQ)** that issues loads past stores (speculatively)\n  and validates the result at commit, rolling back on a misspeculation.\n- A **memory dependence predictor** (optional; see §9.1) to reduce the\n  cost of Ld→St forwarding checks.\n\nThe LSU's job relative to ordering is twofold:\n\n1. **Honor PPO locally** — enforce that a load following a store to the\n   same address returns the store's value (or a younger value), and\n   that the AMO-acquire/release rules produce the required PPO edges.\n2. **Provide a clean drain point** for `FENCE` — a place where the LSU\n   can mark \"all pred-class operations up to this point have completed\n   globally.\"\n\n### 5.2 Store Buffer Drain\n\nA `FENCE` with `pred` including W must drain the store buffer. In a\ndirectory-coherent system the relevant completion is not \"store reached\nL1\" but \"store has acquired the directory ownership token (e.g., `M`\nresponse) and the directory has acknowledged writeback/Invalidate\nacknowledgments for any invalidated peers.\"\n\nFor XH-1:\n\n- **L1 hit, no coherence traffic:** store can complete in ~1–3 cycles\n  after reaching L1.\n- **L1 miss, L2 hit, no sharers:** invalidation may be skipped (silent\n  upgrade to M); store completes when directory grants M.\n- **L1 miss, L2 hit, sharers exist:** directory generates invalidation\n  messages to all sharers; the store completes when all invalidation\n  acknowledgments are received.\n- **L1 miss, L2 miss:** directory forwards to home node; home node\n  replies with data and grants M; the store completes when data and\n  the M grant are both received.\n\nOnly the last three cases impose significant drain latency.\n\n> **Proposed XH-1 stance:** The store buffer must be sized to hold\n> stores pending completion, but the fence-drain path must not depend\n> on the store buffer draining in order — older stores in the buffer\n> that have already received their M response should not block the\n> fence on younger stores still in flight, *unless* the fence\n> successor set requires it.\n\n### 5.3 Load Issue Rules\n\nLoads are the harder side. RVWMO allows loads to be reordered with\nprior stores to different addresses. XH-1's LSU will speculatively\nissue loads past older stores; the load is \"validated\" at commit by\nchecking that the cache line has not been invalidated and that no\nolder store wrote to the same address.\n\nA load that needs to \"see\" the result of an older store on the same\nhart uses **store-to-load forwarding** through the store buffer; this\nmust respect the size and sign-extension rules of the load.\n\nA load that needs to \"see\" the result of an older store on a *different*\nhart (which is what `FENCE`/`FENCE.RW` enforces) must wait for that\nremote store to complete at the directory and update the local\nview via the coherence protocol. The directory's response is the\n\"happens-before\" edge.\n\n### 5.4 Acquire/Release on AMOs\n\nA standard `AMO*` operation in RVWMO has the ordering semantics of an\nacquire (load part) and release (store part) when used in `aq`/`rl`\npositions. In RV32A/RV64A, the instructions `LR` and `SC` carry\nacquire/release bits; AMOs (e.g., `AMOSWAP.W`, `AMOADD.D`) also carry\nthese bits.\n\nFor XH-1, an `aq` AMO must order prior loads/stores from the issuing\nhart before the AMO, and the AMO before subsequent loads/stores.\nMicroarchitecturally:\n\n- `aq`: drain the load and store queues (or at least establish PPO\n  edges) before issuing the AMO to the directory.\n- `rl`: drain the load and store queues *after* the AMO completes.\n- `aq+rl`: drain on both sides; this is the strong fence.\n\n> **Open:** Whether XH-1 implements `aq`/`rl` purely in the LSU\n> (with no global fence message) or with a lightweight fence message\n> to the directory. Purely-local enforcement is faster but requires\n> that the directory serialize the AMO at the line state machine.\n\n### 5.5 `LR`/`SC`\n\n`LR` loads with reservation; `SC` stores conditionally if the\nreservation is still valid. XH-1's reservation representation is\nproposed as a small set of reservation-set entries per core, indexed\nby physical address, anchored in the L1. Coherence actions that\ninvalidate the reserved line (a snoop/Invalidate that the L1 sees)\nmust invalidate the reservation.\n\n> **Proposed XH-1 reservation-granularity choice:** XH-1 reserves at\n> cache-line granularity (the natural L1 line size) and treats\n> forward-progress as the OS's responsibility. This is the\n> conservative default; some designs use sub-line reservation to\n> improve SC success rates, but the gain on typical lock code is\n> modest.\n\n### 5.6 `FENCE.I` and Self-Modifying Code\n\n`FENCE.I` ensures that stores from the issuing hart that preceded the\n`FENCE.I` are visible to subsequent instruction fetches on the same\nhart. It does **not** order stores from other harts.\n\nFor XH-1, `FENCE.I` typically:\n\n1. Drains the store buffer (in the W sense).\n2. Invalidates the L1 I-cache (or the local portion of a unified L1).\n3. Optionally issues a `FENCE` internally so that the store is\n   visible at the directory before the I-fetch resumes.\n\nA cross-hart self-modifying-code sequence (hart A writes code, hart B\nexecutes it) requires an explicit `FENCE` on hart A and a `FENCE.I` on\nhart B (per the RISC-V recommendation), plus coherence to propagate\nthe new data. This is awkward and XH-1 will document it in the\nprogrammer's manual.\n\n### 5.7 `SFENCE.VMA`\n\n`SFENCE.VMA` orders prior stores to PTEs before subsequent\naddress-translation results on the same hart. XH-1's TLB refill\nengine must observe the store-completion signal; a typical\nimplementation stalls the pipeline at the `SFENCE.VMA` until the\nstore buffer drains.\n\n---\n\n## 6. System-Level Ordering Devices\n\nBeyond per-hart mechanisms, XH-1 needs system-level ordering for\nI/O and devices. The RISC-V privileged specification provides two\nmechanisms:\n\n### 6.1 I/O Ordering Bits in PTE\n\nA PTE can have the **PBMT** field (Page-Based Memory Types), which can\nbe marked as:\n\n- **Non-cacheable, idempotent, weakly-ordered (NC, IO).**\n- **Non-cacheable, idempotent, strongly-ordered (NC, I/O-strong).**\n\nXH-1 must honor these by:\n\n- Routing accesses to the appropriate device port on the interconnect.\n- For weakly-ordered I/O, treating the access as device but allowing\n  reordering with respect to other weakly-ordered I/O of the same type.\n- For strongly-ordered I/O, treating the access as a full-fence —\n  the load or store completes in program order with respect to all\n  other accesses from the same hart.\n\n### 6.2 Memory-Mapped I/O Fences\n\nXH-1 will provide a `FENCE` variant (the standard `FENCE pred=rw,\nsucc=rw, fm=0`) that software uses to order I/O accesses. The\nmicroarchitecture must ensure the drain endpoint includes the\ndevice-port queue, not just the L1/L2.\n\n### 6.3 Interrupt Acknowledgment and Ordering\n\nRISC-V interrupt acknowledgment is itself a memory-mapped I/O\nwrite. The microarchitecture must guarantee that the write to the\n*Platform-Level Interrupt Controller* (PLIC) is ordered with respect\nto the I/O operations it is acknowledging. The standard idiom is a\n`FENCE` before the claim/complete operation. XH-1's PLIC integration\nmust be designed so that the `FENCE` does not require extra\nmicroarchitectural support beyond the standard fence.\n\n> **Open:** Whether XH-1 implements a **\"fence fast path\"** for\n> accesses to specific device memory regions (e.g., the PLIC) that\n> bypasses the directory and uses a dedicated sideband signal.\n> Such optimizations can reduce interrupt latency by tens of cycles\n> but complicate coherence reasoning. Not recommended for the first\n> silicon.\n\n---\n\n## 7. Coherence Interaction\n\nXH-1's coherence protocol (described in the coherence research\ndocument) is directory-based, MSI or MESI, with a distributed\ndirectory across the L2 slices. Memory ordering depends on the\ncoherence protocol's notion of \"completion.\"\n\nFor a store to be considered \"globally performed\":\n\n- The line is held in **Modified** state by the requesting core.\n- All prior sharers have sent Invalidate Acknowledgments to the\n  directory (or the directory has confirmed there were no sharers).\n- The directory has logged the new owner.\n\nFor a load to be considered \"globally performed\":\n\n- The line is held in **Shared** (or **Modified** if exclusive) by\n  the requesting core.\n- The data response from the directory (or forwarded from a peer) is\n  the value the directory currently authorizes.\n\n> **Important subtlety:** A `FENCE W,W` between two stores on the\n> same hart must ensure that store 1 has acquired its M grant *and*\n> the directory has processed the invalidations *before* store 2\n> acquires its M grant. In directory terms, store 2 cannot be issued\n> until store 1's M grant has been observed at the directory. XH-1\n> must implement this as an ordering dependency at the directory's\n> L2 controller, not as a per-line serialization at the directory\n> (which would be too coarse and too slow).\n\n### 7.1 Coherence-Induced Ordering Pitfalls\n\nCommon pitfalls in directory-based designs:\n\n- **Directory ack starvation under fence pressure:** when many cores\n  fence simultaneously, the directory must serialize invalidation\n  acknowledgments without deadlock.\n- **Coherence downgrade during fence:** a core that downgrades M→S\n  while a fence is in progress must not let younger loads proceed\n  past the fence.\n- **Silent upgrade (no-invalidate transition) on store:** a clean\n  line upgraded to M without invalidation must still be ordered\n  with respect to other cores' loads that observed the line in S.\n  The directory must explicitly synchronize this transition.\n\n### 7.2 Ordering at the Interconnect\n\nXH-1's interconnect is a mesh (proposed; see interconnect research\ndocument). The mesh delivers messages in-order *between a given pair\nof routers*, but globally the order is not guaranteed. The directory\ncontroller is the serialization point.\n\nFor ordering, XH-1's mesh must:\n\n- Not reorder different message classes on the same virtual channel.\n- Support a \"tail\" or \"credit\" signal so that a fence-acknowledgment\n  message can be generated only when all prior messages on the same\n  path have been delivered.\n\n---\n\n## 8. Implementation Approaches Compared\n\nThis section lists the realistic implementation strategies for an\nin-order XH-1 core. The choice of in-order vs. out-of-order pipeline\nis discussed in §9.1; here we focus on ordering-specific mechanisms.\n\n### 8.1 Approach A: Conservative Per-Core Fence (Baseline)\n\n- The LSU drains the relevant queue classes fully on every `FENCE`.\n- The store buffer's \"drain complete\" is defined as \"every store has\n  received its directory M grant.\"\n- A `FENCE` is implemented as a special instruction that stalls the\n  pipeline until drain completes.\n- Acquire/release AMOs are implemented as full `FENCE`s around the\n  AMO at the LSU.\n- `SFENCE.VMA` stalls the pipeline until the store buffer drains.\n\n**Pros:**\n\n- Simple to verify. The fence semantics maps directly to \"queue\n  empty.\"\n- Predictable performance for small core counts.\n- Easy to reason about coherence interactions.\n\n**Cons:**\n\n- Poor scalability. At 128 cores, the worst-case `FENCE` latency\n  grows because the drain point is global, not local.\n- Overly conservative for `aq`/`rl` AMOs, which do not require a\n  full global drain on either side — only ordering relative to\n  surrounding ordinary accesses.\n\n### 8.2 Approach B: Token-Based Completion\n\n- The LSU attaches a monotonically increasing **completion token**\n  to each outstanding operation.\n- A `FENCE` with pred P, succ S becomes: \"wait until all operations\n  of type P issued before this `FENCE` have produced a token greater\n  than or equal to the fence's token.\"\n- The token is generated when the directory acknowledges the\n  operation; this is exactly the \"global completion\" notion.\n- Acquire/Release is implemented as a fence of the appropriate\n  type around the AMO at the LSU.\n\n**Pros:**\n\n- Fences do not require the entire store buffer to drain; an empty\n  store buffer is sufficient, but so is a store buffer where all\n  older stores have completed.\n- Naturally parallel: many stores can be outstanding, and the\n  fence only waits for the *oldest* of them.\n- Slightly more complex but a well-understood technique (used in\n  academic and commercial cores).\n\n**Cons:**\n\n- Per-message token bookkeeping is more expensive than a simple\n  \"queue empty\" check.\n- Verification requires modeling tokens at the testbench, which is\n  a non-trivial simulation infrastructure investment.\n\n### 8.3 Approach C: Directory-Assisted Ordering\n\n- A special \"fence-ack\" message class is added to the coherence\n  protocol.\n- A `FENCE` instruction sends a fence-ack request to the directory;\n  the directory replies only after all prior messages from the\n  issuing core have been serialized.\n- The directory tracks per-core \"last-seen\" sequence numbers to\n  arbitrate fence-ack requests against ongoing traffic.\n\n**Pros:**\n\n- The directory is the natural serialization point; leveraging it\n  avoids per-core reinvention of global ordering.\n- Scales well to large core counts because the directory already\n  handles 128-core traffic.\n- Allows the per-core LSU to continue issuing younger operations\n  while the fence is in progress, as long as they are ordered\n  correctly.\n\n**Cons:**\n\n- Adds protocol complexity (a new message class, directory state,\n  ack/grant logic).\n- Risk of fence-ack starvation under heavy traffic.\n- Requires careful design to avoid deadlock with normal coherence\n  traffic (classic message-class-dependency cycle).\n\n### 8.4 Approach D: Speculative OoO + MOB\n\nThis is the out-of-order approach. The LSU becomes a Memory\nOrdering Buffer (MOB) that tracks per-operation completion and\nissues fences by tracking the \"oldest in-flight\" of each class.\nThis is how high-performance OoO cores implement fences; the\ntrade-off is pipeline depth and complexity.\n\nXH-1 is proposed as **in-order** for power and predictability; see\n§9.1. OoO is listed for completeness.\n\n**Pros:**\n\n- Highest performance on workloads with deep memory-level\n  parallelism.\n- Fences can be \"free\" if no relevant older operation is in flight.\n\n**Cons:**\n\n- Significantly larger area and power.\n- More verification effort, especially for the MOB's speculative\n  reordering.\n- Diminishing returns on memory-bound workloads at 128 cores (the\n  bottleneck is the interconnect, not the per-core issue rate).\n\n### 8.5 Summary of Approaches\n\n| Approach | Perf (single) | Perf (128) | Area | Verif. effort | Risk |\n| -------- | ------------- | ---------- | ---- | ------------- | ---- |\n| A: Cons. | Low | Lowest | Lowest | Lowest | Low |\n| B: Token | Medium | Medium | Low | Medium | Low |\n| C: Dir-Assisted | Medium-high | High | Medium | High | Medium |\n| D: OoO+MOB | High | High | High | Highest | High |\n\n> **Proposed XH-1 implementation:** **Approach C (Directory-Assisted\n> Ordering)** as the primary, with **Approach B (Token-Based\n> Completion)** as the per-core implementation strategy. The\n> directory handles global serialization, the LSU handles local\n> token tracking. This combination is the most natural for a\n> directory-coherent 128-core design.\n\n---\n\n## 9. Interactions with the Pipeline and Other Subsystems\n\n### 9.1 Pipeline Choice (In-Order vs. Out-of-Order)\n\nXH-1 is **[Proposed] in-order** for the first silicon. The reasons:\n\n- Power envelope. 128 cores at high frequency is a thermal\n  challenge; an in-order core can run cooler.\n- Predictable timing. Real-time and safety-critical workloads\n  benefit from deterministic `FENCE` and `FENCE.I` latencies.\n- Verification cost. An OoO core's MOB is a major verification\n  burden; a 128-core OoO design is a multi-year project.\n\nAn in-order pipeline interacts with ordering as follows:\n\n- The LSU is non-speculative on the store side (stores are\n  committed when the integer pipeline commits them) and lightly\n  speculative on the load side (loads may issue before older\n  stores, with a structural dependency check at commit).\n- `FENCE` stalls the pipeline until drain; the cost is a few tens\n  to a few hundred cycles depending on directory state.\n- Acquire/release AMOs generate local fence events in the LSU.\n\nAn OoO pipeline would:\n\n- Issue stores from the STB to L1, with the MOB tracking\n  completion.\n- Issue loads speculatively past stores with prediction; the MOB\n  validates at commit.\n- `FENCE` becomes a \"wait for oldest in-flight of pred class to\n  complete\" check; usually very fast.\n\n> **Open:** Whether a future XH-2 could be OoO. The research\n> project is not yet at that question.\n\n### 9.2 Cache Hierarchy\n\nMemory ordering is shaped by the cache hierarchy:\n\n- **L1 D-cache:** the load-side ordering reference. A load that hits\n  in L1 sees a value that is at least as recent as the cache's\n  coherence state indicates. A load that misses in L1 must go to\n  L2, where the directory arbitrates the order.\n- **L2 slice (per tile):** home of the directory entry; serializes\n  ordering for the addresses it owns.\n- **L3 (shared, distributed):** a victim cache and last-level\n  coherence point. XH-1's directory is the L2 directory; L3 is\n  treated as a backing store. Invalidation ordering for XH-1\n  therefore ends at L2.\n- **L1 I-cache:** the `FENCE.I` target. The I-cache must be flushed\n  or invalidated to ensure stores are visible.\n\nThe cache hierarchy directly affects `FENCE` latency because the\n\"completion\" point is the L2 directory. A `FENCE W,W` in a system\nwith L1-only ordering would be much cheaper (a few cycles) but\nincorrect; the L2 directory must be in the loop.\n\n### 9.3 Memory System\n\nThe memory system (DRAM controllers, on-chip memory, scratchpads)\nmust respect the same ordering. For XH-1, accesses to on-chip\nscratchpad memory bypass the coherence protocol entirely; they\nare ordered by the scratchpad controller.\n\n> **Open:** Whether XH-1 has a per-tile scratchpad or a shared\n> on-chip SRAM region. If shared, ordering between scratchpad and\n  cache-coherent regions is a research question.\n\n### 9.4 Interconnect\n\nThe mesh interconnect has ordering properties (per-link in-order,\nglobally non-ordered) that determine what the directory and\nendpoints must do to maintain memory ordering. The \"virtual\nchannel\" abstraction in the mesh is the unit of ordering; XH-1\n**[Proposed]** uses:\n\n- **VC0:** Coherence requests (in-order).\n- **VC1:** Coherence responses (in-order).\n- **VC2:** Data messages (in-order).\n- **VC3:** Fence-ack and other ordering-sensitive messages\n  (in-order).\n\nThis separation avoids deadlock and simplifies ordering reasoning.\n\n### 9.5 Interrupts\n\nRISC-V interrupts are delivered to a hart based on the *global\ninterrupt-enable* bits and the PLIC. Ordering constraints:\n\n- A hart that disables interrupts and then writes to a shared\n  variable does not need a `FENCE` before the interrupt-disable\n  write; the write to `mstatus` or `sstatus` is itself a CSR write\n  and is ordered architecturally.\n- A hart that claims an interrupt at the PLIC and then reads a\n  device register must `FENCE` between the claim and the read to\n  ensure the claim is globally visible (the claim is a load from\n  the PLIC's claim register; the read is a load to the device).\n- The interrupt-return path uses `mret`/`sret`, which is\n  architecturally ordered with respect to prior CSR writes.\n- Trap delivery and `xRET` do not include implicit memory fences;\n  the OS must insert them.\n\nXH-1's interrupt controller integration must ensure that the\ninterrupt-acknowledge signal is recognized only after the\nrelevant memory accesses are ordered.\n\n### 9.6 Operating System\n\nThe OS uses `FENCE` heavily:\n\n- **Spinlocks and mutexes:** the lock acquire is typically a\n  `LR`/`SC` sequence; the lock release is an `SC` or a `FENCE`\n  followed by a `SW` to the lock variable.\n- **Page-table manipulation:** `SFENCE.VMA` after every PTE write.\n- **I/O:** `FENCE` before/after device accesses (see §6).\n- **Cross-core TLB shootdown:** an IPI sender writes the\n  invalidation request to a shared memory location, then sends an\n  IPI, and the receiver must see the request after the IPI — this\n  requires a `FENCE` on the sender side.\n- **RCU grace periods:** RCU relies on the memory model's\n  guarantees; a weak model like RVWMO requires explicit\n  `FENCE` at grace-period boundaries.\n- **Boot and SMP startup:** the boot hart must `FENCE` before\n  starting secondary harts to ensure the trampoline code and data\n  are visible.\n\nXH-1's OS port (Linux, or a research kernel) must insert `FENCE`\nat all these sites. The cost of a `FENCE` in a 128-core system\nmakes the boot-time and idle-time costs small in absolute terms\nbut potentially large in *relative* terms; an in-order core that\nspends 200 cycles per `FENCE` will spend a noticeable fraction of\nits time in fences if it idles on a spinlock.\n\n> **Open:** Whether XH-1 ports a vanilla Linux or a research\n> kernel. The memory-ordering analysis in this document assumes\n> vanilla Linux semantics but applies in both cases.\n\n### 9.7 Verification\n\nMemory ordering is famously difficult to verify because:\n\n- The bug is typically a very rare, non-deterministic interleaving.\n- The legal reorderings are many; enumerating them is hard.\n- The coherence protocol and the memory model interact in subtle\n  ways.\n\nXH-1's verification strategy **[Proposed]**:\n\n1. **Litmus tests** for the memory model. Run a corpus of litmus\n   tests (based on the RISC-V memory-model litmus test suite, plus\n   XH-1-specific tests) on RTL simulation. The tests must include\n   the published RVWMO tests and tests specific to XH-1's\n   coherence protocol.\n2. **Coherence protocol model checking.** Use a formal model\n   (e.g., Murphi, TLA+) of the directory protocol; prove\n   invariant properties.\n3. **Pipeline formal verification.** Verify the LSU's local\n   reordering rules with bounded model checking.\n4. **System-level simulation.** Boot Linux on the RTL or a\n   high-level model; run stress workloads (e.g., `perf bench\n   futex`, RCU torture test).\n5. **Stress with random delay injection.** Vary latency on each\n   interconnect path randomly to expose ordering bugs.\n\nThe published **herd7** tool from Cambridge and the **diy7**\nlitmus-test framework are the standard tools for RVWMO\nverification. **Operational models** like Zam are more amenable\nto formal reasoning than axiomatic models; XH-1 should\n**[Proposed]** support both for verification.\n\n> **Open:** Whether XH-1's verification flow includes formal\n> proofs of the memory ordering contract end-to-end, or whether\n> simulation-based testing is the primary line of defense. A\n> research project may choose the latter; a commercial project\n> should aim for the former.\n\n### 9.8 Performance\n\nMemory ordering has two performance axes:\n\n1. **Steady-state throughput:** how fast can the system process\n   ordinary memory traffic. This is dominated by coherence and\n   cache behavior, not by ordering per se.\n2. **Fence/atomic latency and throughput:** how fast is a\n   synchronization primitive. This is where ordering directly\n   impacts performance.\n\nFor a 128-core system, the second axis is the more important\ncontributor to scalability. A workload with frequent\nsynchronization (e.g., a parallel database, a many-core\nmachine-learning inference engine) is dominated by `FENCE` and\nAMO latency.\n\n> **Quantitative note (rough estimate):** Assuming a 1 GHz in-order\n> core, a directory-coherent L2 slice with a hop latency of ~10 ns\n> per mesh traversal, and an average of 3–5 hops per coherence\n> transaction, a single uncontended `FENCE` on a hot line might\n> take ~30–50 ns (30–50 cycles). A contended `FENCE` (multiple\n> sharers) requiring invalidation acks could take 100–300 ns\n> (100–300 cycles). On a 128-core workload that fences once per\n> critical section, this is the dominant cost.\n\nXH-1's performance evaluation **[Proposed]**:\n\n- Measure `FENCE` latency distribution under varying contention\n  and across the mesh (corner vs. center tile).\n- Measure AMO latency under contention.\n- Measure the effect of `aq`/`rl` vs. explicit `FENCE` in the\n  OS.\n- Compare against an ideal \"no ordering cost\" baseline.\n\n---\n\n## 10. Specific RISC-V Considerations\n\n### 10.1 The `Ztso` Extension (Optional)\n\n`Ztso` provides TSO (Total Store Order) at every hart. With\n`Ztso`, the implementation guarantees that stores are not\nreordered with respect to subsequent loads, simplifying the\nsoftware model.\n\n> **Proposed XH-1 stance:** Do not advertise `Ztso` for the\n> first silicon. The hardware cost of supporting TSO at every\n> hart is a tighter store-load ordering in the LSU, and the\n> benefit is only felt by software that is willing to be\n> incompatible with weak-ordering implementations. The standard\n> RISC-V Linux kernel is RVWMO-compatible and does not require\n> `Ztso`. Re-evaluate if a major software target requires it.\n\n### 10.2 The `Zam` Extension (If Adopted)\n\n`Zam` is a proposed extension that provides an operational\nmemory model in addition to the axiomatic RVWMO. It does not\nchange the contract; it provides operational clarity. XH-1\n**[Proposed]** will document its memory model in terms of `Zam`\nwhere possible because operational models are easier to\nimplement and verify.\n\n### 10.3 The PTE-Based Memory Type (PBMT)\n\nXH-1's MMU must support the PBMT field and route accesses to\nthe appropriate ordering class. Specifically:\n\n- **PMA = NC (non-cacheable):** accesses go to the device port;\n  load/store ordering is enforced by the device-port controller.\n- **PMA = IO (weakly ordered I/O):** same as NC, but with\n  explicit reordering rules.\n- **PMA = WEAK (cacheable but weakly ordered):** accesses are\n  cached but ordered weakly; useful for write-combining buffers.\n- **PMA = STRONG (cacheable, strongly ordered):** same as\n  cacheable but with fence semantics on every access; rarely\n  used.\n\n> **Open:** Whether XH-1 supports all PBMT values or a subset.\n> A research project may support all four; a commercial project\n> may support only NC and cacheable.\n\n### 10.4 The `fence.i` and Instruction-Fetch Coherence\n\n`FENCE.I` orders stores w.r.t. instruction fetches on the same\nhart. Cross-hart, the software idiom is:\n\n```c\n// hart A:\nwrite_code(addr, code);\n__sync_synchronize(); // FENCE\nsend_ipi(hart_B);\n\n// hart B:\nhandle_ipi();\n__sync_synchronize(); // FENCE\n__fence_i();          // FENCE.I\nicache_flush();\njump_to(addr);\n```\n\nXH-1's I-cache is a VIPT cache; on an aliasing configuration\n(small pages, large cache), the OS must use the recommended\nsequence (or the larger page size). The architecture team will\ndocument this.\n\n### 10.5 TSO and RVWMO Differences for Software\n\nAlthough the focus here is hardware, it is worth noting for\ncompleteness:\n\n- Under TSO, ordinary loads and stores have program order\n  semantics. Under RVWMO, they do not.\n- A program that is correct under TSO may not be correct under\n  RVWMO without additional fences. The Linux kernel's\n  memory-model code (`tools/memory-model/`) contains the\n  required fences for RVWMO.\n\nXH-1's port will use the RVWMO-correct fences; no special\nsupport is required in hardware beyond honoring the standard\n`FENCE`.\n\n---\n\n## 11. Unresolved Design Questions\n\nThe following questions remain open for the XH-1 architecture:\n\n1. **In-order vs. out-of-order pipeline.** In-order is proposed,\n   but the decision is not final. An OoO pipeline changes the\n   MOB design and may improve `FENCE` latency.\n2. **`Ztso` adoption.** Whether to support TSO at every hart.\n3. **`Zam` adoption.** Whether to require the operational memory\n   model in the specification.\n4. **Coherence protocol.** MSI vs. MESI vs. MOESI. The choice\n   affects the number of transitions and therefore the number\n   of coherence-induced ordering edges. (See the coherence\n   research document.)\n5. **Interconnect topology.** Mesh vs. ring vs. hierarchical\n   bus+mesh. The choice affects `FENCE` latency and the\n   directory's role.\n6. **PBMT support scope.** Which PTE memory types XH-1\n   implements.\n7. **Per-line vs. per-word reservation.** Affects `LR`/`SC`\n   success rates.\n8. **Fence-ack message class.** Whether to add a dedicated\n   fence-ack message (Approach C) or rely entirely on\n   per-core token tracking (Approach B).\n9. **Verification strategy.** Simulation-based vs. formal;\n   end-to-end memory model proofs.\n10. **OS port.** Linux vs. a research kernel; this affects\n    the set of `FENCE` sites the system must support.\n\n---\n\n## 12. Recommendations\n\nWhere evidence supports a recommendation, the following are\nXH-1's proposed positions:\n\n1. **Adopt RVWMO as the memory model.** This is the standard\n   RISC-V model and is the most widely supported by software.\n2. **Implement an in-order pipeline with Approach C\n   (Directory-Assisted Ordering) and Approach B (Per-Core\n   Token Completion).** This combination scales to 128 cores\n   while keeping the per-core pipeline simple.\n3. **Use a directory-based MESI coherence protocol at the\n   L2 slice.** The directory is the natural serialization\n   point for memory ordering. (See the coherence research\n   document for the full argument.)\n4. **Add a dedicated fence-ack message class to the coherence\n   protocol.** This avoids overloading existing messages and\n   reduces verification ambiguity.\n5. **Honour the standard `FENCE`, `FENCE.I`, `SFENCE.VMA`,\n   `LR`/`SC`, and AMO semantics.** No architectural deviations.\n6. **Document the cross-hart self-modifying-code sequence in\n   the programmer's manual.** Use the standard fence pair.\n7. **Implement the PTE-based memory types (PBMT) for at\n   least NC and cacheable.** Other types are deferred.\n8. **Provide a memory-model litmus-test suite as part of\n   the verification deliverables.** This is essential for\n   catching ordering bugs.\n9. **Measure `FENCE` and AMO latency across the mesh in\n   performance evaluation.** Corner tiles have different\n   latency from center tiles; the asymmetry is a real\n   performance concern.\n\n---\n\n## 13. References\n\nThe following sources informed this document. URLs and\npublication details are given where known; the reader should\nverify currency, as the RISC-V ecosystem is rapidly evolving.\n\n- RISC-V International, *RISC-V Instruction Set Manual,\n  Volume I: Unprivileged ISA*. Appendix A, \"RVWMO: The RISC-V\n  Weak Memory Ordering Model.\"\n- RISC-V International, *RISC-V Instruction Set Manual,\n  Volume II: Privileged ISA*. Sections on fence and memory\n  types (PBMT).\n- Pulte, C., et al. *\"Zam: A Complete and Verifiable\n  Operational Memory Model for RISC-V.\"* (Reference to the\n  operational memory model paper.)\n- Owens, S., Sarkar, S., Sewell, P. *\"A Better x86 Memory\n  Model: x86-TSO.\"* (TSO reference; the TSO model is the\n  inspiration for `Ztso`.)\n- Lamport, L. *\"How to Make a Multiprocessor Computer That\n  Correctly Executes Multiprocess Programs.\"* IEEE TC, 1979.\n  (The foundational sequential consistency paper.)\n- Adve, S. V., Gharachorloo, K. *\"Shared Memory Consistency\n  Models: A Tutorial.\"* IEEE Computer, 1996. (Tutorial on\n  memory models.)\n- Martin, M. M. K., et al. *\"Token Coherence.\"* (Token-based\n  ordering reference, applicable to Approach B.)\n- ACM SIGOPS Operating Systems Review. *Linux Kernel Memory\n  Model Documentation.* (`tools/memory-model/` in the Linux\n  kernel source tree.)\n- AMD. *AMD64 Architecture Programmer's Manual*, Volume 2.\n  (TSO reference for comparison.)\n- Intel. *Intel 64 and IA-32 Architectures Software\n  Developer's Manual*, Volume 3. (TSO reference for\n  comparison.)\n- ARM. *ARM Architecture Reference Manual*. (Comparison\n  with ARM's weak memory model.)\n\nSpecific tool references:\n\n- **herd7**: litmus-test framework for weak memory models,\n  by Cambridge. https://github.com/herd/herdtools7\n- **diy7**: companion tool for herd7.\n- **Murphi**, **TLA+**: protocol model checkers used for\n  coherence protocol verification.\n\nThe RISC-V Memory Model working group's published\ndocumentation, including the operational model papers and\nthe litmus-test corpus, is the primary source for the\nRVWMO contract. Specifics of XH-1's coherence protocol are\nout of scope for this document and are addressed in the\ncoherence research document.\n\n---\n\n## 14. Document Status\n\nThis document is a **research proposal**. It does not represent\na finalized XH-1 architectural commitment. Items marked\n**[Proposed]** are engineering recommendations under evaluation;\nitems marked **[Open]** are unresolved design questions.\n\nThe next revisions of this document should:\n\n- Reflect decisions on the open questions in §11.\n- Add quantitative `FENCE` latency measurements once RTL or\n  high-level models are available.\n- Incorporate results from the verification litmus-test\n  suite.\n- Align with the final coherence protocol choice.\n- Document the OS-level `FENCE` insertion policy.\n\n---\n\n*End of document.*","refusal":null,"reasoning":null}}],"usage":{"prompt_tokens":663,"completion_tokens":9938,"total_tokens":10601,"cost":0,"is_byok":false,"prompt_tokens_details":{"cached_tokens":648,"cache_write_tokens":0,"audio_tokens":0,"video_tokens":0},"cost_details":{"upstream_inference_cost":0,"upstream_inference_prompt_cost":0,"upstream_inference_completions_cost":0},"completion_tokens_details":{"reasoning_tokens":0,"image_tokens":0,"audio_tokens":0}}}