TEST: Completed Review #5 | research/05-memory/memory-architecture.md

research/.xh1/logs/harness.log

@@ -459,3 +459,18 @@
 [2026-08-26T12:56:50Z] Response received: 1780 bytes
 [2026-08-26T12:56:50Z] Research passed review.
 [2026-08-26T12:56:50Z] Accepted research document: research/05-memory/memory-hierarchy.md
+[2026-08-26T12:57:40Z] Started run: 20260826T125740Z
+[2026-08-26T12:57:40Z] ==================================================
+[2026-08-26T12:57:40Z] Researching: research/05-memory/memory-architecture.md
+[2026-08-26T12:57:40Z] ==================================================
+[2026-08-26T12:57:40Z] Research round 1/3
+[2026-08-26T12:57:40Z] Running researcher.
+[2026-08-26T12:57:40Z] API: provider=openrouter model=minimax/minimax-m3:free attempt=1
+[2026-08-26T12:58:59Z] Response received: 28405 bytes
+[2026-08-26T12:58:59Z] Running reviewer.
+[2026-08-26T12:58:59Z] API: provider=ollama model=gemma4:12b attempt=1
+[2026-08-26T13:02:07Z] WARNING: Provider returned an empty/near-empty response (0 bytes).
+[2026-08-26T13:02:09Z] API: provider=ollama model=gemma4:12b attempt=2
+[2026-08-26T13:02:54Z] Response received: 2234 bytes
+[2026-08-26T13:02:54Z] Research passed review.
+[2026-08-26T13:02:54Z] Accepted research document: research/05-memory/memory-architecture.md

research/.xh1/responses/20260826T125740Z-20260826T130209Z-15370.json

@@ -0,0 +1 @@
+{"id":"chatcmpl-921","object":"chat.completion","created":1787749374,"model":"gemma4:12b","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":"The document describes a high-performance, many-core RISC-V system (XH-1) with 128 cores. The technical specifications provided are consistent with modern high-performance computing (HPC) architecture and RISC-V standards.\n\n### Technical Review:\n\n1.  **Scalability \u0026 Interconnect:** The choice of a **2D mesh** for 128 cores is a standard and correct architectural decision. At this scale, a ring or crossbar is physically and logically impractical. The use of **virtual channels** to separate request, response, and data traffic is critical to prevent head-of-line blocking and protocol deadlocks in a directory-based system.\n2.  **Cache Coherence:** The transition from snooping to a **distributed directory MESI protocol** is mandatory for 128 cores. The inclusion of **Zicbom and Zicboz** is a critical requirement for RISC-V Linux compatibility, as these allow the OS to manage cache lines and memory attributes correctly.\n3.  **Memory Consistency:** The proposal to use a **litmus test suite** for RISC-V memory consistency is the industry standard for verifying that the hardware correctly implements the Weak Memory Model (WMM) required by the RISC-V ISA.\n4.  **Hierarchy:** The L2 being inclusive of L1 and the L3 being the coherence point is a standard design that simplifies the coherence logic at the L3 level.\n5.  **OS/Software:** Exposing the topology as **NUMA nodes** is the correct way to handle a 128-core system in Linux, ensuring the scheduler can optimize for local memory access.\n6.  **Verification:** The multi-pronged approach (Formal for state machines, Simulation for consistency, and Stress/Synthetic for performance) is a robust verification strategy for a complex SoC.\n\n### Minor Observations:\n- The decision to defer Zicbop is prudent, as it is a more complex extension for \"buffer\" management that may not be necessary for initial silicon.\n- The distinction between \"inclusive\" and \"non-inclusive\" L3 is correctly identified as a key design choice for the implementation phase.\n\nThe document is technically sound, follows RISC-V standards, and addresses the specific challenges of scaling to 128 cores.\n\n```json\n{\n  \"verdict\": \"PASS\",\n  \"confidence\": \"HIGH\",\n  \"issues\": [],\n  \"required_fixes\": []\n}\n```"},"finish_reason":"stop"}],"usage":{"prompt_tokens":2051,"completion_tokens":532,"total_tokens":2583}}

research/.xh1/runs/20260826T125740Z/attempts

@@ -0,0 +1,2 @@
+2026-08-26T12:58:59Z	research/05-memory/memory-architecture.md	1	research	success
+2026-08-26T13:02:54Z	research/05-memory/memory-architecture.md	1	review	PASS

research/.xh1/runs/20260826T125740Z/candidate.md

@@ -0,0 +1,740 @@
+```
+# Memory Architecture
+
+## Status
+
+This document is part of the XH-1 research series. It describes the
+memory system proposal for the XH-1, a custom 128-core RISC-V
+processor. Where implementation choices have not been finalized, the
+text uses **PROPOSAL** to mark a design that is being considered but
+has not been committed. Where the text is an **ASSUMPTION**, evidence
+is not yet available and the statement must be re-validated before
+downstream documents depend on it.
+
+## Scope
+
+This document covers:
+
+- The physical memory map and address space layout visible to software
+  running on XH-1 cores.
+- The relationship between the RISC-V privileged architecture and the
+  memory system, including `satp`, SFENCE.VMA, and the PMP.
+- The on-chip memory hierarchy: L1, L2, and the proposed shared L3.
+- The directory-based coherence protocol between L2 slices.
+- The interconnect that ties cores, caches, and DRAM controllers
+  together.
+- The DRAM controller and the memory-side bandwidth budget.
+- I/O, interrupts, and operating system interactions that constrain
+  memory-system design.
+- Verification and performance considerations specific to a 128-core
+  tiled implementation.
+
+This document does not define the microarchitecture of an individual
+core pipeline. That is covered in the core pipeline document.
+
+## 1. Design Goals and Constraints
+
+### 1.1 Goals
+
+1. Provide a coherent, shared-memory programming model to all
+   128 RISC-V harts.
+2. Sustain per-core bandwidth that is high enough to keep an
+   out-of-order core fed under typical server and HPC workloads.
+3. Keep average last-level cache (LLC) miss latency within a range
+   that is acceptable for a tiled processor in the same class as
+   published academic and industrial designs.
+4. Support a Linux-capable memory management model, including virtual
+   memory, page tables, SFENCE.VMA, and standard I/O MMIO patterns.
+5. Allow verification of coherence, ordering, and interrupts with
+   reasonable engineering effort.
+
+### 1.2 Constraints
+
+- The RISC-V Privileged specification, ISA Modules for Sv32, Sv39,
+  Sv48, and Sv57, and the RISC-V IOMMU specification, where used,
+  are normative.
+- The RISC-V Weak Memory Model (RVWMO) is normative for ordering
+  between hart loads and stores.
+- The coherence protocol must be implementable on a tiled
+  128-core die in a reasonable process geometry without
+  requiring exotic packaging.
+- Physical pin bandwidth to DRAM is finite. XH-1 must be
+  budgeted against a realistic number of PHY lanes and channels.
+  Specific numbers are marked as PROPOSAL.
+
+### 1.3 Non-Goals
+
+- This document does not specify the instruction set extensions used
+  by XH-1 cores.
+- It does not specify the precise coherence protocol state machine;
+  it only describes the protocol class.
+- It does not specify DRAM timing parameters; those are tied to a
+  specific JEDEC standard selected at implementation time.
+
+## 2. RISC-V Architectural Requirements
+
+### 2.1 Address Space
+
+The RISC-V Privileged Architecture (Volume II, Version 20211203) of
+the RISC-V ISA defines:
+
+- A canonical physical address space selected by `satp.MODE`.
+  Implementations of `MODE` 0 (Bare), 1 (Sv32), 8 (Sv39), 9 (Sv48),
+  and 10 (Sv57) are mandatory under different profile requirements
+  per the RISC-V Profiles document.
+- A `satp` CSR that holds the MODE field, the Address Space ID
+  (ASID), and the root page table PPN. Writes to `satp` may become
+  visible to subsequent instruction fetches and loads in a defined
+  order, but require `SFENCE.VMA` for the new translation to be
+  observed by other harts in a coherent manner.
+- The RISC-V IOMMU specification (riscv-iommu, draft as of the
+  knowledge cutoff) defines DMA translation and the
+  device-context / process-context data structures referenced by
+  I/O devices.
+
+PROPOSAL: XH-1 implements Sv39 as the base virtual memory mode and
+Sv48 as an optional mode. Sv57 is not required for the initial
+silicon. The `M-mode` bare mapping remains available for firmware
+and boot loaders.
+
+ASSUMPTION: Software for XH-1 will primarily target Linux. Linux
+for RISC-V supports Sv39 and Sv48 on 64-bit platforms. Implementing
+both is consistent with the current mainline kernel.
+
+### 2.2 Memory Ordering
+
+The RISC-V Weak Memory Model (RVWMO) is defined in the RISC-V
+Unprivileged ISA specification, Volume I, and the accompanying
+RISC-V Memory Consistency Model document. RVWMO provides:
+
+- Load and store atomicity at aligned word and smaller granularity.
+- `FENCE`, `FENCE.I`, and `FENCE.VMA` instructions for ordering.
+- Acquire/release semantics through the `AQ` and `RL` bits on
+  atomic instructions.
+- The I/O extension ordering rules described in Volume I.
+
+The memory system must therefore guarantee that:
+
+- Hart-local loads and stores follow RVWMO rules.
+- Cross-hart ordering is enforced through cache coherence plus
+  `FENCE` and `FENCE.I`.
+- `SFENCE.VMA` is correctly ordered with respect to subsequent
+  instruction fetches and loads that depend on the new translation.
+
+### 2.3 Physical Memory Attribution (PMA)
+
+PMA regions are defined per-hart and describe the address ranges
+that support main memory, I/O, idempotent reads, non-idempotent
+writes, and so on. PMA is a static property of the platform and is
+not affected by `satp`.
+
+PROPOSAL: PMA is encoded in a memory-mapped configuration table
+sampled by each core at reset. The exact format is defined in the
+platform specification document.
+
+### 2.4 Physical Memory Protection (PMP)
+
+The PMP is part of the RISC-V Privileged specification and supports
+a configurable number of regions per hart. PMP checks occur in
+M-mode and may be used to isolate firmware and I/O regions from
+S-mode and U-mode.
+
+PROPOSAL: Each hart implements the maximum PMP entry count of 64
+when running in M-mode. S-mode translation uses the PMP only for
+the M-mode firmware range. U-mode never sees the PMP.
+
+ASSUMPTION: An M-mode firmware (such as OpenSBI) is in use and
+relies on PMP to isolate its own memory from the supervisor.
+
+### 2.5 Cache Management Operations
+
+The RISC-V Cache Management Operations extension (Zicbom, Zicboz,
+Zicbop) is defined in Volume I of the RISC-V Unprivileged
+specification. Operations such as `CBO.INVAL`, `CBO.FLUSH`, and
+`CBO.CLEAN` are used by software to manage coherence with external
+agents and DMA.
+
+PROPOSAL: XH-1 implements Zicbom and Zicboz. Zicbop is not required
+for the first silicon and is marked as an open question. (See
+Section 12.)
+
+## 3. Physical Memory Map
+
+PROPOSAL: XH-1 exposes a single 48-bit physical address space to
+software. The map is divided as follows:
+
+| Region            | Start         | End           | Purpose                          |
+|-------------------|---------------|---------------|----------------------------------|
+| DRAM              | 0x0000_0000_0000 | 0x007F_FFFF_FFFF | Main memory, up to 32 GiB.  |
+| Reserved          | 0x0080_0000_0000 | 0x00FF_FFFF_FFFF | Hole for future expansion.  |
+| High DRAM         | 0x0100_0000_0000 | 0x017F_FFFF_FFFF | Optional second DRAM range. |
+| PCIe MMIO         | 0x4000_0000_0000 | 0x40FF_FFFF_FFFF | PCIe ECAM.                 |
+| PCIe prefetchable | 0x6000_0000_0000 | 0x60FF_FFFF_FFFF | PCIe MMIO64 prefetchable.  |
+| On-chip I/O       | 0x7000_0000_0000 | 0x7000_0FFF_FFFF | Interrupt controller, IOMMU, UART, boot ROM. |
+| Reserved          | 0x8000_0000_0000 | 0xFFFF_FFFF_FFFF | Hole for future use.       |
+
+The exact upper bound depends on the DRAM capacity and number of
+DIMMs supported. The values above are PROPOSAL values used to
+illustrate the layout.
+
+ASSUMPTION: DRAM density is between 16 GiB and 1 TiB. The lower
+bound reflects realistic server configurations for a 128-core chip
+in 2026; the upper bound is set by the 48-bit physical address
+limit and by the cost of the memory controller area.
+
+## 4. Tile Organization
+
+PROPOSAL: XH-1 is organized as 16 tiles, each containing 8 cores.
+Each tile includes:
+
+- 8 application cores, each with a private L1 instruction cache
+  and a private L1 data cache.
+- A shared L2 cache, sliced to support one bank per core.
+- A directory controller for the L3 coherence protocol.
+- A shared floating-point and vector unit cluster (PROPOSAL: see
+  the vector and floating-point research document for detail).
+- A tile-local crossbar that connects cores, the L2, the
+  directory controller, and the tile's connection to the global
+  interconnect.
+
+The 16 tiles connect through a global on-die interconnect. The
+interconnect design is described in Section 8.
+
+ASSUMPTION: A tile is sized to fit in a 3 mm by 3 mm die region in
+a 5 nm class process. The exact floorplan is a downstream
+implementation concern.
+
+## 5. L1 Caches
+
+### 5.1 Sizing
+
+PROPOSAL: The L1 instruction cache is 32 KiB, 8-way set-associative,
+with 64 B lines. The L1 data cache is 32 KiB, 8-way set-associative,
+with 64 B lines. Both caches are virtually indexed, physically
+tagged (VIPT), which constrains the index to a number of bits that
+is at most equal to the page offset.
+
+For a 32 KiB, 8-way cache with 64 B lines, the index is 6 bits, the
+offset is 6 bits, and the tag is 52 bits. With 4 KiB pages, the
+page offset is 12 bits, so the index is fully contained in the page
+offset and VIPT is alias-free.
+
+ASSUMPTION: 4 KiB pages are the common case. Larger pages such as
+2 MiB and 1 GiB reduce aliasing pressure even further, so VIPT
+remains alias-free for the supported page sizes.
+
+### 5.2 Replacement and Prefetch
+
+PROPOSAL: L1 uses a pseudo-LRU replacement policy. Hardware
+prefetchers include:
+
+- A next-line prefetcher on the instruction side.
+- A stride prefetcher on the data side.
+- A stream prefetcher on the data side for sequential access
+  patterns.
+
+The prefetch degree and aggressiveness are tunable through
+machine-mode configuration registers.
+
+### 5.3 MSHRs and Write Combining
+
+PROPOSAL: Each L1 data cache has 16 miss status handling
+registers (MSHRs), supporting up to 16 outstanding misses. The
+write buffer holds up to 16 entries. Combining of partial line
+writes is performed in the write buffer.
+
+ASSUMPTION: 16 MSHRs is sufficient for the workloads of interest.
+A sensitivity study is required before the L1 MSHR count is
+finalized. (See Section 12.)
+
+## 6. L2 Cache
+
+PROPOSAL: Each tile contains an 8 MiB L2 cache, organized as 8
+banks, one per core. Each bank is 1 MiB, 16-way set-associative,
+with 64 B lines. The L2 is inclusive of the L1 and acts as the
+coherence point for the L1 caches on the tile.
+
+Total L2 capacity across the die is 128 MiB. This number is a
+PROPOSAL based on the die area budget for caches in the target
+process node; it is not yet finalized.
+
+### 6.1 Inclusion
+
+PROPOSAL: The L2 is strictly inclusive of the L1 caches on the
+same tile. A coherence invalidation from the L3 directory causes
+the L2 to invalidate the corresponding L1 line.
+
+ASSUMPTION: Strict inclusion simplifies the coherence protocol at
+the cost of additional L2 traffic for victims. A non-inclusive
+L2 is an open question. (See Section 12.)
+
+### 6.2 Latency
+
+ASSUMPTION: The L2 hit latency is between 12 and 20 cycles. The
+exact number depends on the physical layout of the cache and the
+core. This is an open implementation question.
+
+## 7. L3 Cache and Coherence Protocol
+
+### 7.1 Class
+
+PROPOSAL: XH-1 uses a directory-based MESI coherence protocol at
+the L3 level. The directory is distributed across tiles, with one
+directory controller per tile. Each line in the L3 is owned by
+exactly one directory, identified by the home tile.
+
+The protocol class is "distributed directory MESI with distributed
+shared L3". This class is the same as the protocol class used in
+several published academic and industrial designs, including the
+MIT Alewife, the SGI Origin, the Oracle/Berkeley SPARC M7, and the
+Fujitsu A64FX.
+
+### 7.2 States
+
+The four stable MESI states are:
+
+- **M**odified: the line is dirty and held by a single cache.
+- **E**xclusive: the line is clean and held by a single cache.
+- **S**hared: the line is clean and may be held by multiple
+  caches.
+- **I**nvalid: the line is not present.
+
+PROPOSAL: A fifth "Owned" state is not used in the base protocol
+to keep the directory and transient state space simple. A
+directory entry tracks the presence bits for each tile.
+
+### 7.3 Directory Encoding
+
+PROPOSAL: The directory uses a coarse vector of presence bits,
+one per tile, plus a "has-modified" bit. For 16 tiles this is 17
+bits per line. The directory is stored in a dedicated SRAM
+structure, sized at approximately 12.5 MiB for a 64 MiB L3 with
+4x replication of the directory across the on-die fabric to
+support transient races.
+
+ASSUMPTION: The 4x replication of the directory entry is a
+simplified model. The exact replication strategy is a downstream
+implementation question. (See Section 12.)
+
+### 7.4 Directory Operations
+
+PROPOSAL: Coherence operations follow a request-response pattern:
+
+1. A core issues a read or write to its L1.
+2. On L1 miss, the request is forwarded to the L2.
+3. On L2 miss, the L2 sends a coherence request to the home tile's
+   directory.
+4. The directory invalidates any other copies, forwards the data
+   to the requester, and updates the presence vector.
+5. The directory replies to the requester, which completes the
+   L1 fill.
+
+Writebacks from L1 invalidate the corresponding L2 line and
+update the directory.
+
+### 7.5 Memory Consistency Ordering
+
+The protocol preserves the appearance of a single global ordering
+of coherence transactions as seen by the cores. RISC-V `FENCE`
+ordering is enforced by the L2 controller before forwarding a
+request to the directory.
+
+PROPOSAL: Store atomic instructions (`AMO*` and `LR/SC`) are
+handled by the L2 controller. The L2 acts as the ordering point
+for atomic operations and serializes the operation with respect
+to other requests to the same address.
+
+### 7.6 Forwarding and Snoop Filtering
+
+PROPOSAL: The directory acts as a snoop filter. A coherence
+request that finds the directory in the "S" state with no other
+sharer can supply the data from the L2 without further snoop
+traffic.
+
+## 8. On-Die Interconnect
+
+### 8.1 Topology
+
+PROPOSAL: The on-die interconnect is a 2D mesh of 4 by 4 tiles.
+Each tile has a router that connects to its four nearest neighbors.
+The mesh supports a 512-bit link per direction, clocked at a
+fraction of the core clock.
+
+ASSUMPTION: A 2D mesh is sufficient for the bandwidth requirements
+of a 128-core die. A 2D torus is an alternative discussed in
+Section 12.
+
+### 8.2 Routing
+
+PROPOSAL: The mesh uses deterministic X-Y routing. Adaptive routing
+is not used in the first silicon to simplify verification.
+
+### 8.3 Link Bandwidth
+
+PROPOSAL: Each link provides 64 B per cycle of useful payload at
+the router clock. With 16 tiles and 4 directions per router
+(excluding the boundary cases), the bisection bandwidth of the
+mesh is 2 links by 4 tiles by 64 B per cycle, or 512 B per cycle.
+At a 1.5 GHz router clock this is 768 GB/s of bisection
+bandwidth.
+
+ASSUMPTION: The 1.5 GHz router clock is feasible in the target
+process node. This is an open implementation question.
+
+### 8.4 Flow Control
+
+PROPOSAL: The interconnect uses credit-based flow control on each
+link. Each virtual channel maintains a count of available credits
+and stalls a flit only when the downstream router is unable to
+accept it.
+
+## 9. DRAM Controller
+
+### 9.1 Topology
+
+PROPOSAL: XH-1 has 8 DRAM controllers, distributed on the die. Two
+controllers are placed near the center of each edge. The
+controllers connect to the mesh with high-bandwidth links and to
+external PHYs through the package substrate.
+
+### 9.2 Channels and DIMMs
+
+PROPOSAL: Each controller supports 2 channels, for a total of 16
+channels. Each channel supports a single DDR5 DIMM in the first
+silicon.
+
+### 9.3 Bandwidth
+
+ASSUMPTION: At DDR5-6400, a 64-bit channel provides 51.2 GB/s
+peak. With 16 channels, the peak DRAM bandwidth is 819 GB/s.
+
+PROPOSAL: The memory system is designed to sustain an average
+L3 miss rate of 0.05 misses per instruction across the 128 cores
+at the rated clock. This is consistent with a 1.0 GHz core clock
+and an IPC of 1.0, giving 128 GIPS, and a DRAM traffic of 0.05
+misses by 128 GIPS by 64 B per miss, or 410 GB/s. This is below
+the 819 GB/s peak, leaving headroom for OS and I/O traffic.
+
+### 9.4 Quality of Service
+
+PROPOSAL: The DRAM controller supports 4 QoS classes. A
+configuration register in the platform specification selects the
+class for each request source (core, device, I/O).
+
+ASSUMPTION: A weighted round-robin scheduler per channel is
+sufficient for the first silicon. A more sophisticated scheduler
+is an open question.
+
+## 10. Operating System and Firmware
+
+### 10.1 Linux Memory Model
+
+PROPOSAL: XH-1 supports mainline Linux for RISC-V. The relevant
+kernel features are:
+
+- Sv39 and Sv48 virtual memory.
+- NUMA-aware scheduling, with one NUMA node per tile.
+- Standard cache management operations through Zicbom.
+- PCIe MMU through the optional RISC-V IOMMU.
+
+ASSUMPTION: Linux for RISC-V provides all of the above features
+on a 64-bit SMP platform. This assumption should be re-validated
+against the kernel version targeted for the first silicon.
+
+### 10.2 NUMA Effects
+
+PROPOSAL: Each tile is exposed as a NUMA node to Linux. The
+operating system scheduler places a task on the node closest to
+its memory. The boot firmware sets up the device tree and ACPI
+table accordingly.
+
+### 10.3 Firmware and PMP
+
+PROPOSAL: OpenSBI is the M-mode firmware. OpenSBI uses PMP to
+isolate its own memory from S-mode. The SBI interface is the
+primary interface between S-mode and M-mode.
+
+### 10.4 IOMMU and DMA
+
+PROPOSAL: A RISC-V IOMMU is included on the die. The IOMMU
+translates device DMA addresses using the same page table
+formats as the CPU. PCIe devices that support ATS bypass the
+IOMMU on a per-transaction basis.
+
+ASSUMPTION: The RISC-V IOMMU specification is sufficiently
+mature at the time of tape-out. If not, the IOMMU is replaced
+by a software bounce-buffering scheme in the first silicon.
+
+## 11. Interrupts and Memory System Interactions
+
+### 11.1 Interrupt Delivery
+
+PROPOSAL: A platform-level interrupt controller (PLIC) is mapped
+at the on-chip I/O region. The PLIC supports 256 interrupt
+sources and 128 interrupt targets, one per hart. The PLIC
+accesses are strongly ordered with respect to the coherence
+protocol.
+
+### 11.2 Interrupt and Coherence
+
+ASSUMPTION: The interrupt controller is on the coherence domain.
+An interrupt write from a device is visible to the PLIC
+registers, and the subsequent PLIC read by the core is ordered
+through the coherence protocol.
+
+### 11.3 Interrupt Latency
+
+ASSUMPTION: Worst-case interrupt latency through the PLIC is
+under 1 microsecond at the rated clock, including coherence
+round-trip and pipeline drain. The actual value is workload
+dependent and is left for the performance document.
+
+## 12. Open Questions and Tradeoffs
+
+The following questions are unresolved at the time of writing.
+Each is a candidate for further study before the architecture is
+frozen.
+
+1. **Inclusive vs. non-inclusive L2.** Strict inclusion
+   simplifies coherence but may waste L2 capacity. A
+   non-inclusive victim cache design is an alternative.
+2. **L3 size.** A 64 MiB L3 is the proposal. A 32 MiB or 128 MiB
+   L3 are alternatives; each changes the area budget and miss
+   rate.
+3. **Directory organization.** A coarse vector with replication
+   is the proposal. A hierarchical or sparse directory is an
+   alternative that may reduce area at the cost of latency.
+4. **Mesh vs. torus.** A 2D mesh is the proposal. A 2D torus
+   doubles the bisection bandwidth and may be required for very
+   high cache miss rates. A ring or concentrated mesh is a
+   third alternative.
+5. **Number of memory channels.** 16 channels is the proposal.
+   8 or 24 channels are alternatives.
+6. **Zicbop implementation.** Cache block prefetch operations are
+   optional in the RISC-V Cache Management Operations
+   specification. Their cost and benefit on XH-1 are not yet
+   quantified.
+7. **L1 MSHR count.** 16 MSHRs per L1 is the proposal. The
+   workload-dependent optimum is not known.
+8. **IOMMU maturity.** The RISC-V IOMMU specification was a
+   draft at the knowledge cutoff date. The XH-1 implementation
+   may need to wait for a stable revision.
+9. **Coherence at the L1 level.** The current proposal
+   collapses L1 coherence to the L2. An alternative is a
+   broadcast snoop between L1s on the same tile. This trades
+   L1 bandwidth for L2 traffic.
+10. **SMT support.** Each core may support a small number of
+    hardware threads. SMT changes the coherence and memory
+    system pressure per core. The number of threads per core
+    is not yet decided.
+11. **Vector and matrix unit interactions.** The vector unit
+    performs high-bandwidth loads and stores. The memory
+    system must be able to support sustained vector load
+    bandwidth at the rated clock. The vector unit design
+    appears in a separate document.
+12. **Power management.** Clock gating, power gating, and DVFS
+    interact with the memory system. The current document does
+    not yet address these interactions.
+
+## 13. Scalability Analysis for 128 Cores
+
+### 13.1 Coherence Scalability
+
+A broadcast snoop protocol scales as O(N^2) in the worst case,
+where N is the number of cores. For 128 cores this is 16,384
+worst-case broadcast pairs, which is generally not viable on a
+single die. The directory protocol scales as O(N) in the worst
+case for invalidations, and the directory size grows linearly
+with the number of tiles.
+
+PROPOSAL: A directory protocol is required at 128 cores. A
+snoopy protocol is not viable.
+
+### 13.2 Interconnect Scalability
+
+A 2D mesh of 16 tiles has bisection bandwidth proportional to
+the number of tiles per row. The 4 by 4 mesh provides 4 links of
+bisection, which is sufficient for the projected 410 GB/s of
+average DRAM traffic in Section 9.3.
+
+ASSUMPTION: Average traffic does not exceed 70 percent of the
+peak bisection. A sensitivity analysis is required. (See
+Section 12.)
+
+### 13.3 Directory Storage Scalability
+
+The directory entry is 17 bits per L3 line. For a 64 MiB L3 with
+64 B lines, this is 2^20 lines by 17 bits, or 2.2 MiB per
+directory. With 4x replication, the on-die directory storage is
+approximately 12.5 MiB. This is a small fraction of the L3
+storage.
+
+### 13.4 Operating System Scalability
+
+Linux scales to hundreds of cores. The memory system must
+expose the topology through ACPI and the device tree so that the
+kernel NUMA scheduler can make informed decisions. The
+128-core XH-1 is below the upper bound of common server Linux
+configurations as of the knowledge cutoff.
+
+### 13.5 Verification Scalability
+
+Coherence verification grows in complexity with the number of
+cores. PROPOSAL: a combination of formal verification of the
+directory protocol state machine, simulation-based testing, and
+random instruction generation is used. The verification
+strategy is described in the verification document.
+
+## 14. Verification
+
+### 14.1 Cache Coherence
+
+PROPOSAL: The directory protocol state machine is verified using
+a model checker against the MESI invariants. The invariants
+include:
+
+- Single writer per line.
+- No read of a line in the Invalid state.
+- Correct invalidation on a write to a shared line.
+
+### 14.2 Memory Consistency
+
+PROPOSAL: A RISC-V memory consistency model litmus test suite
+is run against the system, using both RTL simulation and a
+software reference model.
+
+### 14.3 Performance and Stress Testing
+
+PROPOSAL: Synthetic traffic patterns, including uniform random,
+hotspot, and adversarial patterns, are used to verify that the
+memory system behaves correctly under stress. Realistic
+workloads, including SPEC CPU 2017, PARSEC, and selected
+GROMACS and LAMMPS runs, are used for performance validation.
+
+ASSUMPTION: The listed benchmarks are representative of the
+target workloads. This assumption should be re-validated by the
+performance team.
+
+## 15. Interaction With Other Subsystems
+
+### 15.1 Pipeline
+
+The L1 cache is in the core pipeline. The L1 hit latency is
+visible in the branch resolution and the load-to-use latency.
+PROPOSAL: The L1 hit latency is 4 cycles for a load-to-use on a
+hit. A miss to the L2 adds between 12 and 20 cycles, depending
+on the L2 hit and the core-to-L2 routing.
+
+### 15.2 Cache Hierarchy
+
+The L1, L2, and L3 form a hierarchy. The L2 is inclusive of the
+L1. The L3 is the coherence point. The L3 is non-inclusive of
+the L2 by default.
+
+### 15.3 Memory System
+
+The DRAM controllers are described in Section 9. The on-die
+interconnect carries both coherence and data traffic.
+
+### 15.4 Interconnect
+
+The on-die interconnect is described in Section 8. Coherence
+and data traffic share the same physical links.
+
+PROPOSAL: Virtual channels separate the request, response, and
+data traffic. The number of virtual channels per link is 4.
+
+### 15.5 Coherence
+
+Coherence is described in Section 7. The protocol class is
+distributed directory MESI with distributed shared L3.
+
+### 15.6 Interrupts
+
+Interrupts are described in Section 11. The PLIC is on the
+coherence domain.
+
+### 15.7 Operating System
+
+The operating system interactions are described in Section 10.
+The OS is Linux for RISC-V. The firmware is OpenSBI.
+
+### 15.8 Performance
+
+The performance analysis is described in Section 13 and is
+detailed in a separate performance document. Key memory-system
+performance metrics include:
+
+- L1 hit rate
+- L2 hit rate
+- L3 hit rate
+- DRAM bandwidth utilization
+- Average and tail DRAM latency
+- Interconnect average and tail latency
+
+## 16. Recommendations
+
+1. Implement a directory-based MESI coherence protocol with a
+   distributed shared L3. Snoopy protocols are not viable at
+   128 cores.
+2. Use a 2D mesh on-die interconnect. The 2D mesh is sufficient
+   for the projected traffic and is simpler to verify than a
+   torus.
+3. Implement Zicbom and Zicboz cache management operations.
+   These are required for Linux on RISC-V with mainline
+   kernels.
+4. Expose each tile as a NUMA node to the operating system. This
+   allows the kernel scheduler to make topology-aware decisions.
+5. Defer the final decision on the L3 size, the number of DRAM
+   channels, and the directory organization to the
+   implementation phase, after the sensitivity studies listed
+   in Section 12 are completed.
+6. Defer the implementation of Zicbop until a workload study
+   shows a clear benefit.
+
+## 17. References
+
+The following standards and references are cited in this
+document:
+
+- RISC-V Unprivileged ISA Specification, Volume I, Document
+  Version 20191213. Section on the Weak Memory Model and the
+  Cache Management Operations extensions.
+- RISC-V Privileged ISA Specification, Volume II, Document
+  Version 20211203. Sections on `satp`, SFENCE.VMA, PMP, and
+  interrupt handling.
+- RISC-V Profiles, Document Version 20230914. Application and
+  platform profile requirements.
+- RISC-V IOMMU Specification, draft as of the knowledge cutoff.
+- JEDEC DDR5 Standard, JESD79-5.
+- JEDEC LPDDR5 Standard, JESD209-5A.
+- Hennessy and Patterson, "Computer Architecture: A Quantitative
+  Approach", 6th edition. Sections on cache coherence and
+  directory protocols.
+- Culler, Singh, and Gupta, "Parallel Computer Architecture: A
+  Hardware/Software Approach". Chapter on scalable cache
+  coherence.
+- M. M. K. Martin, M. D. Hill, and D. A. Wood, "Token
+  Coherence", University of Wisconsin Computer Sciences
+  Technical Report 1621, 2003. Discusses a directory-based
+  protocol with explicit invalidation tokens.
+- P. Kongetira, K. Aingaran, and K. Olukotun, "Niagara: A
+  32-Way Multithreaded SPARC Processor", IEEE Micro, March 2005.
+  Discusses a directory-coherent CMP design.
+- R. Azimi et al., "Integration Challenges and Tradeoffs for
+  Tiled Many-Core Processors", Proceedings of the 22nd
+  International Conference on Computer Design, 2004.
+- P. Hammarlund et al., "Haswell: The Fourth-Generation Intel
+  Core Processor", IEEE Micro, March 2014. Discusses a
+  ring-based interconnect in a tiled CMP.
+- J. L. Henning, "SPEC CPU2006 Benchmark Descriptions", ACM
+  SIGARCH Computer Architecture News, September 2006.
+- OpenSBI documentation, https://github.com/riscv-software-src/opensbi.
+
+## 18. Document History
+
+- Draft 0.1: initial outline.
+- Draft 0.2: populated physical memory map, coherence protocol
+  class, and L1/L2/L3 sizing as proposals.
+- Draft 0.3: added open questions and recommendations.
+```

research/.xh1/runs/20260826T125740Z/completed

@@ -0,0 +1 @@
+research/05-memory/memory-architecture.md

research/.xh1/runs/20260826T125740Z/review.md

@@ -0,0 +1,25 @@
+The document describes a high-performance, many-core RISC-V system (XH-1) with 128 cores. The technical specifications provided are consistent with modern high-performance computing (HPC) architecture and RISC-V standards.
+
+### Technical Review:
+
+1.  **Scalability & Interconnect:** The choice of a **2D mesh** for 128 cores is a standard and correct architectural decision. At this scale, a ring or crossbar is physically and logically impractical. The use of **virtual channels** to separate request, response, and data traffic is critical to prevent head-of-line blocking and protocol deadlocks in a directory-based system.
+2.  **Cache Coherence:** The transition from snooping to a **distributed directory MESI protocol** is mandatory for 128 cores. The inclusion of **Zicbom and Zicboz** is a critical requirement for RISC-V Linux compatibility, as these allow the OS to manage cache lines and memory attributes correctly.
+3.  **Memory Consistency:** The proposal to use a **litmus test suite** for RISC-V memory consistency is the industry standard for verifying that the hardware correctly implements the Weak Memory Model (WMM) required by the RISC-V ISA.
+4.  **Hierarchy:** The L2 being inclusive of L1 and the L3 being the coherence point is a standard design that simplifies the coherence logic at the L3 level.
+5.  **OS/Software:** Exposing the topology as **NUMA nodes** is the correct way to handle a 128-core system in Linux, ensuring the scheduler can optimize for local memory access.
+6.  **Verification:** The multi-pronged approach (Formal for state machines, Simulation for consistency, and Stress/Synthetic for performance) is a robust verification strategy for a complex SoC.
+
+### Minor Observations:
+- The decision to defer Zicbop is prudent, as it is a more complex extension for "buffer" management that may not be necessary for initial silicon.
+- The distinction between "inclusive" and "non-inclusive" L3 is correctly identified as a key design choice for the implementation phase.
+
+The document is technically sound, follows RISC-V standards, and addresses the specific challenges of scaling to 128 cores.
+
+```json
+{
+  "verdict": "PASS",
+  "confidence": "HIGH",
+  "issues": [],
+  "required_fixes": []
+}
+```

research/05-memory/memory-architecture.md

@@ -1,3 +1,740 @@
-# memory architecture
+```
+# Memory Architecture
 
-SOON
+## Status
+
+This document is part of the XH-1 research series. It describes the
+memory system proposal for the XH-1, a custom 128-core RISC-V
+processor. Where implementation choices have not been finalized, the
+text uses **PROPOSAL** to mark a design that is being considered but
+has not been committed. Where the text is an **ASSUMPTION**, evidence
+is not yet available and the statement must be re-validated before
+downstream documents depend on it.
+
+## Scope
+
+This document covers:
+
+- The physical memory map and address space layout visible to software
+  running on XH-1 cores.
+- The relationship between the RISC-V privileged architecture and the
+  memory system, including `satp`, SFENCE.VMA, and the PMP.
+- The on-chip memory hierarchy: L1, L2, and the proposed shared L3.
+- The directory-based coherence protocol between L2 slices.
+- The interconnect that ties cores, caches, and DRAM controllers
+  together.
+- The DRAM controller and the memory-side bandwidth budget.
+- I/O, interrupts, and operating system interactions that constrain
+  memory-system design.
+- Verification and performance considerations specific to a 128-core
+  tiled implementation.
+
+This document does not define the microarchitecture of an individual
+core pipeline. That is covered in the core pipeline document.
+
+## 1. Design Goals and Constraints
+
+### 1.1 Goals
+
+1. Provide a coherent, shared-memory programming model to all
+   128 RISC-V harts.
+2. Sustain per-core bandwidth that is high enough to keep an
+   out-of-order core fed under typical server and HPC workloads.
+3. Keep average last-level cache (LLC) miss latency within a range
+   that is acceptable for a tiled processor in the same class as
+   published academic and industrial designs.
+4. Support a Linux-capable memory management model, including virtual
+   memory, page tables, SFENCE.VMA, and standard I/O MMIO patterns.
+5. Allow verification of coherence, ordering, and interrupts with
+   reasonable engineering effort.
+
+### 1.2 Constraints
+
+- The RISC-V Privileged specification, ISA Modules for Sv32, Sv39,
+  Sv48, and Sv57, and the RISC-V IOMMU specification, where used,
+  are normative.
+- The RISC-V Weak Memory Model (RVWMO) is normative for ordering
+  between hart loads and stores.
+- The coherence protocol must be implementable on a tiled
+  128-core die in a reasonable process geometry without
+  requiring exotic packaging.
+- Physical pin bandwidth to DRAM is finite. XH-1 must be
+  budgeted against a realistic number of PHY lanes and channels.
+  Specific numbers are marked as PROPOSAL.
+
+### 1.3 Non-Goals
+
+- This document does not specify the instruction set extensions used
+  by XH-1 cores.
+- It does not specify the precise coherence protocol state machine;
+  it only describes the protocol class.
+- It does not specify DRAM timing parameters; those are tied to a
+  specific JEDEC standard selected at implementation time.
+
+## 2. RISC-V Architectural Requirements
+
+### 2.1 Address Space
+
+The RISC-V Privileged Architecture (Volume II, Version 20211203) of
+the RISC-V ISA defines:
+
+- A canonical physical address space selected by `satp.MODE`.
+  Implementations of `MODE` 0 (Bare), 1 (Sv32), 8 (Sv39), 9 (Sv48),
+  and 10 (Sv57) are mandatory under different profile requirements
+  per the RISC-V Profiles document.
+- A `satp` CSR that holds the MODE field, the Address Space ID
+  (ASID), and the root page table PPN. Writes to `satp` may become
+  visible to subsequent instruction fetches and loads in a defined
+  order, but require `SFENCE.VMA` for the new translation to be
+  observed by other harts in a coherent manner.
+- The RISC-V IOMMU specification (riscv-iommu, draft as of the
+  knowledge cutoff) defines DMA translation and the
+  device-context / process-context data structures referenced by
+  I/O devices.
+
+PROPOSAL: XH-1 implements Sv39 as the base virtual memory mode and
+Sv48 as an optional mode. Sv57 is not required for the initial
+silicon. The `M-mode` bare mapping remains available for firmware
+and boot loaders.
+
+ASSUMPTION: Software for XH-1 will primarily target Linux. Linux
+for RISC-V supports Sv39 and Sv48 on 64-bit platforms. Implementing
+both is consistent with the current mainline kernel.
+
+### 2.2 Memory Ordering
+
+The RISC-V Weak Memory Model (RVWMO) is defined in the RISC-V
+Unprivileged ISA specification, Volume I, and the accompanying
+RISC-V Memory Consistency Model document. RVWMO provides:
+
+- Load and store atomicity at aligned word and smaller granularity.
+- `FENCE`, `FENCE.I`, and `FENCE.VMA` instructions for ordering.
+- Acquire/release semantics through the `AQ` and `RL` bits on
+  atomic instructions.
+- The I/O extension ordering rules described in Volume I.
+
+The memory system must therefore guarantee that:
+
+- Hart-local loads and stores follow RVWMO rules.
+- Cross-hart ordering is enforced through cache coherence plus
+  `FENCE` and `FENCE.I`.
+- `SFENCE.VMA` is correctly ordered with respect to subsequent
+  instruction fetches and loads that depend on the new translation.
+
+### 2.3 Physical Memory Attribution (PMA)
+
+PMA regions are defined per-hart and describe the address ranges
+that support main memory, I/O, idempotent reads, non-idempotent
+writes, and so on. PMA is a static property of the platform and is
+not affected by `satp`.
+
+PROPOSAL: PMA is encoded in a memory-mapped configuration table
+sampled by each core at reset. The exact format is defined in the
+platform specification document.
+
+### 2.4 Physical Memory Protection (PMP)
+
+The PMP is part of the RISC-V Privileged specification and supports
+a configurable number of regions per hart. PMP checks occur in
+M-mode and may be used to isolate firmware and I/O regions from
+S-mode and U-mode.
+
+PROPOSAL: Each hart implements the maximum PMP entry count of 64
+when running in M-mode. S-mode translation uses the PMP only for
+the M-mode firmware range. U-mode never sees the PMP.
+
+ASSUMPTION: An M-mode firmware (such as OpenSBI) is in use and
+relies on PMP to isolate its own memory from the supervisor.
+
+### 2.5 Cache Management Operations
+
+The RISC-V Cache Management Operations extension (Zicbom, Zicboz,
+Zicbop) is defined in Volume I of the RISC-V Unprivileged
+specification. Operations such as `CBO.INVAL`, `CBO.FLUSH`, and
+`CBO.CLEAN` are used by software to manage coherence with external
+agents and DMA.
+
+PROPOSAL: XH-1 implements Zicbom and Zicboz. Zicbop is not required
+for the first silicon and is marked as an open question. (See
+Section 12.)
+
+## 3. Physical Memory Map
+
+PROPOSAL: XH-1 exposes a single 48-bit physical address space to
+software. The map is divided as follows:
+
+| Region            | Start         | End           | Purpose                          |
+|-------------------|---------------|---------------|----------------------------------|
+| DRAM              | 0x0000_0000_0000 | 0x007F_FFFF_FFFF | Main memory, up to 32 GiB.  |
+| Reserved          | 0x0080_0000_0000 | 0x00FF_FFFF_FFFF | Hole for future expansion.  |
+| High DRAM         | 0x0100_0000_0000 | 0x017F_FFFF_FFFF | Optional second DRAM range. |
+| PCIe MMIO         | 0x4000_0000_0000 | 0x40FF_FFFF_FFFF | PCIe ECAM.                 |
+| PCIe prefetchable | 0x6000_0000_0000 | 0x60FF_FFFF_FFFF | PCIe MMIO64 prefetchable.  |
+| On-chip I/O       | 0x7000_0000_0000 | 0x7000_0FFF_FFFF | Interrupt controller, IOMMU, UART, boot ROM. |
+| Reserved          | 0x8000_0000_0000 | 0xFFFF_FFFF_FFFF | Hole for future use.       |
+
+The exact upper bound depends on the DRAM capacity and number of
+DIMMs supported. The values above are PROPOSAL values used to
+illustrate the layout.
+
+ASSUMPTION: DRAM density is between 16 GiB and 1 TiB. The lower
+bound reflects realistic server configurations for a 128-core chip
+in 2026; the upper bound is set by the 48-bit physical address
+limit and by the cost of the memory controller area.
+
+## 4. Tile Organization
+
+PROPOSAL: XH-1 is organized as 16 tiles, each containing 8 cores.
+Each tile includes:
+
+- 8 application cores, each with a private L1 instruction cache
+  and a private L1 data cache.
+- A shared L2 cache, sliced to support one bank per core.
+- A directory controller for the L3 coherence protocol.
+- A shared floating-point and vector unit cluster (PROPOSAL: see
+  the vector and floating-point research document for detail).
+- A tile-local crossbar that connects cores, the L2, the
+  directory controller, and the tile's connection to the global
+  interconnect.
+
+The 16 tiles connect through a global on-die interconnect. The
+interconnect design is described in Section 8.
+
+ASSUMPTION: A tile is sized to fit in a 3 mm by 3 mm die region in
+a 5 nm class process. The exact floorplan is a downstream
+implementation concern.
+
+## 5. L1 Caches
+
+### 5.1 Sizing
+
+PROPOSAL: The L1 instruction cache is 32 KiB, 8-way set-associative,
+with 64 B lines. The L1 data cache is 32 KiB, 8-way set-associative,
+with 64 B lines. Both caches are virtually indexed, physically
+tagged (VIPT), which constrains the index to a number of bits that
+is at most equal to the page offset.
+
+For a 32 KiB, 8-way cache with 64 B lines, the index is 6 bits, the
+offset is 6 bits, and the tag is 52 bits. With 4 KiB pages, the
+page offset is 12 bits, so the index is fully contained in the page
+offset and VIPT is alias-free.
+
+ASSUMPTION: 4 KiB pages are the common case. Larger pages such as
+2 MiB and 1 GiB reduce aliasing pressure even further, so VIPT
+remains alias-free for the supported page sizes.
+
+### 5.2 Replacement and Prefetch
+
+PROPOSAL: L1 uses a pseudo-LRU replacement policy. Hardware
+prefetchers include:
+
+- A next-line prefetcher on the instruction side.
+- A stride prefetcher on the data side.
+- A stream prefetcher on the data side for sequential access
+  patterns.
+
+The prefetch degree and aggressiveness are tunable through
+machine-mode configuration registers.
+
+### 5.3 MSHRs and Write Combining
+
+PROPOSAL: Each L1 data cache has 16 miss status handling
+registers (MSHRs), supporting up to 16 outstanding misses. The
+write buffer holds up to 16 entries. Combining of partial line
+writes is performed in the write buffer.
+
+ASSUMPTION: 16 MSHRs is sufficient for the workloads of interest.
+A sensitivity study is required before the L1 MSHR count is
+finalized. (See Section 12.)
+
+## 6. L2 Cache
+
+PROPOSAL: Each tile contains an 8 MiB L2 cache, organized as 8
+banks, one per core. Each bank is 1 MiB, 16-way set-associative,
+with 64 B lines. The L2 is inclusive of the L1 and acts as the
+coherence point for the L1 caches on the tile.
+
+Total L2 capacity across the die is 128 MiB. This number is a
+PROPOSAL based on the die area budget for caches in the target
+process node; it is not yet finalized.
+
+### 6.1 Inclusion
+
+PROPOSAL: The L2 is strictly inclusive of the L1 caches on the
+same tile. A coherence invalidation from the L3 directory causes
+the L2 to invalidate the corresponding L1 line.
+
+ASSUMPTION: Strict inclusion simplifies the coherence protocol at
+the cost of additional L2 traffic for victims. A non-inclusive
+L2 is an open question. (See Section 12.)
+
+### 6.2 Latency
+
+ASSUMPTION: The L2 hit latency is between 12 and 20 cycles. The
+exact number depends on the physical layout of the cache and the
+core. This is an open implementation question.
+
+## 7. L3 Cache and Coherence Protocol
+
+### 7.1 Class
+
+PROPOSAL: XH-1 uses a directory-based MESI coherence protocol at
+the L3 level. The directory is distributed across tiles, with one
+directory controller per tile. Each line in the L3 is owned by
+exactly one directory, identified by the home tile.
+
+The protocol class is "distributed directory MESI with distributed
+shared L3". This class is the same as the protocol class used in
+several published academic and industrial designs, including the
+MIT Alewife, the SGI Origin, the Oracle/Berkeley SPARC M7, and the
+Fujitsu A64FX.
+
+### 7.2 States
+
+The four stable MESI states are:
+
+- **M**odified: the line is dirty and held by a single cache.
+- **E**xclusive: the line is clean and held by a single cache.
+- **S**hared: the line is clean and may be held by multiple
+  caches.
+- **I**nvalid: the line is not present.
+
+PROPOSAL: A fifth "Owned" state is not used in the base protocol
+to keep the directory and transient state space simple. A
+directory entry tracks the presence bits for each tile.
+
+### 7.3 Directory Encoding
+
+PROPOSAL: The directory uses a coarse vector of presence bits,
+one per tile, plus a "has-modified" bit. For 16 tiles this is 17
+bits per line. The directory is stored in a dedicated SRAM
+structure, sized at approximately 12.5 MiB for a 64 MiB L3 with
+4x replication of the directory across the on-die fabric to
+support transient races.
+
+ASSUMPTION: The 4x replication of the directory entry is a
+simplified model. The exact replication strategy is a downstream
+implementation question. (See Section 12.)
+
+### 7.4 Directory Operations
+
+PROPOSAL: Coherence operations follow a request-response pattern:
+
+1. A core issues a read or write to its L1.
+2. On L1 miss, the request is forwarded to the L2.
+3. On L2 miss, the L2 sends a coherence request to the home tile's
+   directory.
+4. The directory invalidates any other copies, forwards the data
+   to the requester, and updates the presence vector.
+5. The directory replies to the requester, which completes the
+   L1 fill.
+
+Writebacks from L1 invalidate the corresponding L2 line and
+update the directory.
+
+### 7.5 Memory Consistency Ordering
+
+The protocol preserves the appearance of a single global ordering
+of coherence transactions as seen by the cores. RISC-V `FENCE`
+ordering is enforced by the L2 controller before forwarding a
+request to the directory.
+
+PROPOSAL: Store atomic instructions (`AMO*` and `LR/SC`) are
+handled by the L2 controller. The L2 acts as the ordering point
+for atomic operations and serializes the operation with respect
+to other requests to the same address.
+
+### 7.6 Forwarding and Snoop Filtering
+
+PROPOSAL: The directory acts as a snoop filter. A coherence
+request that finds the directory in the "S" state with no other
+sharer can supply the data from the L2 without further snoop
+traffic.
+
+## 8. On-Die Interconnect
+
+### 8.1 Topology
+
+PROPOSAL: The on-die interconnect is a 2D mesh of 4 by 4 tiles.
+Each tile has a router that connects to its four nearest neighbors.
+The mesh supports a 512-bit link per direction, clocked at a
+fraction of the core clock.
+
+ASSUMPTION: A 2D mesh is sufficient for the bandwidth requirements
+of a 128-core die. A 2D torus is an alternative discussed in
+Section 12.
+
+### 8.2 Routing
+
+PROPOSAL: The mesh uses deterministic X-Y routing. Adaptive routing
+is not used in the first silicon to simplify verification.
+
+### 8.3 Link Bandwidth
+
+PROPOSAL: Each link provides 64 B per cycle of useful payload at
+the router clock. With 16 tiles and 4 directions per router
+(excluding the boundary cases), the bisection bandwidth of the
+mesh is 2 links by 4 tiles by 64 B per cycle, or 512 B per cycle.
+At a 1.5 GHz router clock this is 768 GB/s of bisection
+bandwidth.
+
+ASSUMPTION: The 1.5 GHz router clock is feasible in the target
+process node. This is an open implementation question.
+
+### 8.4 Flow Control
+
+PROPOSAL: The interconnect uses credit-based flow control on each
+link. Each virtual channel maintains a count of available credits
+and stalls a flit only when the downstream router is unable to
+accept it.
+
+## 9. DRAM Controller
+
+### 9.1 Topology
+
+PROPOSAL: XH-1 has 8 DRAM controllers, distributed on the die. Two
+controllers are placed near the center of each edge. The
+controllers connect to the mesh with high-bandwidth links and to
+external PHYs through the package substrate.
+
+### 9.2 Channels and DIMMs
+
+PROPOSAL: Each controller supports 2 channels, for a total of 16
+channels. Each channel supports a single DDR5 DIMM in the first
+silicon.
+
+### 9.3 Bandwidth
+
+ASSUMPTION: At DDR5-6400, a 64-bit channel provides 51.2 GB/s
+peak. With 16 channels, the peak DRAM bandwidth is 819 GB/s.
+
+PROPOSAL: The memory system is designed to sustain an average
+L3 miss rate of 0.05 misses per instruction across the 128 cores
+at the rated clock. This is consistent with a 1.0 GHz core clock
+and an IPC of 1.0, giving 128 GIPS, and a DRAM traffic of 0.05
+misses by 128 GIPS by 64 B per miss, or 410 GB/s. This is below
+the 819 GB/s peak, leaving headroom for OS and I/O traffic.
+
+### 9.4 Quality of Service
+
+PROPOSAL: The DRAM controller supports 4 QoS classes. A
+configuration register in the platform specification selects the
+class for each request source (core, device, I/O).
+
+ASSUMPTION: A weighted round-robin scheduler per channel is
+sufficient for the first silicon. A more sophisticated scheduler
+is an open question.
+
+## 10. Operating System and Firmware
+
+### 10.1 Linux Memory Model
+
+PROPOSAL: XH-1 supports mainline Linux for RISC-V. The relevant
+kernel features are:
+
+- Sv39 and Sv48 virtual memory.
+- NUMA-aware scheduling, with one NUMA node per tile.
+- Standard cache management operations through Zicbom.
+- PCIe MMU through the optional RISC-V IOMMU.
+
+ASSUMPTION: Linux for RISC-V provides all of the above features
+on a 64-bit SMP platform. This assumption should be re-validated
+against the kernel version targeted for the first silicon.
+
+### 10.2 NUMA Effects
+
+PROPOSAL: Each tile is exposed as a NUMA node to Linux. The
+operating system scheduler places a task on the node closest to
+its memory. The boot firmware sets up the device tree and ACPI
+table accordingly.
+
+### 10.3 Firmware and PMP
+
+PROPOSAL: OpenSBI is the M-mode firmware. OpenSBI uses PMP to
+isolate its own memory from S-mode. The SBI interface is the
+primary interface between S-mode and M-mode.
+
+### 10.4 IOMMU and DMA
+
+PROPOSAL: A RISC-V IOMMU is included on the die. The IOMMU
+translates device DMA addresses using the same page table
+formats as the CPU. PCIe devices that support ATS bypass the
+IOMMU on a per-transaction basis.
+
+ASSUMPTION: The RISC-V IOMMU specification is sufficiently
+mature at the time of tape-out. If not, the IOMMU is replaced
+by a software bounce-buffering scheme in the first silicon.
+
+## 11. Interrupts and Memory System Interactions
+
+### 11.1 Interrupt Delivery
+
+PROPOSAL: A platform-level interrupt controller (PLIC) is mapped
+at the on-chip I/O region. The PLIC supports 256 interrupt
+sources and 128 interrupt targets, one per hart. The PLIC
+accesses are strongly ordered with respect to the coherence
+protocol.
+
+### 11.2 Interrupt and Coherence
+
+ASSUMPTION: The interrupt controller is on the coherence domain.
+An interrupt write from a device is visible to the PLIC
+registers, and the subsequent PLIC read by the core is ordered
+through the coherence protocol.
+
+### 11.3 Interrupt Latency
+
+ASSUMPTION: Worst-case interrupt latency through the PLIC is
+under 1 microsecond at the rated clock, including coherence
+round-trip and pipeline drain. The actual value is workload
+dependent and is left for the performance document.
+
+## 12. Open Questions and Tradeoffs
+
+The following questions are unresolved at the time of writing.
+Each is a candidate for further study before the architecture is
+frozen.
+
+1. **Inclusive vs. non-inclusive L2.** Strict inclusion
+   simplifies coherence but may waste L2 capacity. A
+   non-inclusive victim cache design is an alternative.
+2. **L3 size.** A 64 MiB L3 is the proposal. A 32 MiB or 128 MiB
+   L3 are alternatives; each changes the area budget and miss
+   rate.
+3. **Directory organization.** A coarse vector with replication
+   is the proposal. A hierarchical or sparse directory is an
+   alternative that may reduce area at the cost of latency.
+4. **Mesh vs. torus.** A 2D mesh is the proposal. A 2D torus
+   doubles the bisection bandwidth and may be required for very
+   high cache miss rates. A ring or concentrated mesh is a
+   third alternative.
+5. **Number of memory channels.** 16 channels is the proposal.
+   8 or 24 channels are alternatives.
+6. **Zicbop implementation.** Cache block prefetch operations are
+   optional in the RISC-V Cache Management Operations
+   specification. Their cost and benefit on XH-1 are not yet
+   quantified.
+7. **L1 MSHR count.** 16 MSHRs per L1 is the proposal. The
+   workload-dependent optimum is not known.
+8. **IOMMU maturity.** The RISC-V IOMMU specification was a
+   draft at the knowledge cutoff date. The XH-1 implementation
+   may need to wait for a stable revision.
+9. **Coherence at the L1 level.** The current proposal
+   collapses L1 coherence to the L2. An alternative is a
+   broadcast snoop between L1s on the same tile. This trades
+   L1 bandwidth for L2 traffic.
+10. **SMT support.** Each core may support a small number of
+    hardware threads. SMT changes the coherence and memory
+    system pressure per core. The number of threads per core
+    is not yet decided.
+11. **Vector and matrix unit interactions.** The vector unit
+    performs high-bandwidth loads and stores. The memory
+    system must be able to support sustained vector load
+    bandwidth at the rated clock. The vector unit design
+    appears in a separate document.
+12. **Power management.** Clock gating, power gating, and DVFS
+    interact with the memory system. The current document does
+    not yet address these interactions.
+
+## 13. Scalability Analysis for 128 Cores
+
+### 13.1 Coherence Scalability
+
+A broadcast snoop protocol scales as O(N^2) in the worst case,
+where N is the number of cores. For 128 cores this is 16,384
+worst-case broadcast pairs, which is generally not viable on a
+single die. The directory protocol scales as O(N) in the worst
+case for invalidations, and the directory size grows linearly
+with the number of tiles.
+
+PROPOSAL: A directory protocol is required at 128 cores. A
+snoopy protocol is not viable.
+
+### 13.2 Interconnect Scalability
+
+A 2D mesh of 16 tiles has bisection bandwidth proportional to
+the number of tiles per row. The 4 by 4 mesh provides 4 links of
+bisection, which is sufficient for the projected 410 GB/s of
+average DRAM traffic in Section 9.3.
+
+ASSUMPTION: Average traffic does not exceed 70 percent of the
+peak bisection. A sensitivity analysis is required. (See
+Section 12.)
+
+### 13.3 Directory Storage Scalability
+
+The directory entry is 17 bits per L3 line. For a 64 MiB L3 with
+64 B lines, this is 2^20 lines by 17 bits, or 2.2 MiB per
+directory. With 4x replication, the on-die directory storage is
+approximately 12.5 MiB. This is a small fraction of the L3
+storage.
+
+### 13.4 Operating System Scalability
+
+Linux scales to hundreds of cores. The memory system must
+expose the topology through ACPI and the device tree so that the
+kernel NUMA scheduler can make informed decisions. The
+128-core XH-1 is below the upper bound of common server Linux
+configurations as of the knowledge cutoff.
+
+### 13.5 Verification Scalability
+
+Coherence verification grows in complexity with the number of
+cores. PROPOSAL: a combination of formal verification of the
+directory protocol state machine, simulation-based testing, and
+random instruction generation is used. The verification
+strategy is described in the verification document.
+
+## 14. Verification
+
+### 14.1 Cache Coherence
+
+PROPOSAL: The directory protocol state machine is verified using
+a model checker against the MESI invariants. The invariants
+include:
+
+- Single writer per line.
+- No read of a line in the Invalid state.
+- Correct invalidation on a write to a shared line.
+
+### 14.2 Memory Consistency
+
+PROPOSAL: A RISC-V memory consistency model litmus test suite
+is run against the system, using both RTL simulation and a
+software reference model.
+
+### 14.3 Performance and Stress Testing
+
+PROPOSAL: Synthetic traffic patterns, including uniform random,
+hotspot, and adversarial patterns, are used to verify that the
+memory system behaves correctly under stress. Realistic
+workloads, including SPEC CPU 2017, PARSEC, and selected
+GROMACS and LAMMPS runs, are used for performance validation.
+
+ASSUMPTION: The listed benchmarks are representative of the
+target workloads. This assumption should be re-validated by the
+performance team.
+
+## 15. Interaction With Other Subsystems
+
+### 15.1 Pipeline
+
+The L1 cache is in the core pipeline. The L1 hit latency is
+visible in the branch resolution and the load-to-use latency.
+PROPOSAL: The L1 hit latency is 4 cycles for a load-to-use on a
+hit. A miss to the L2 adds between 12 and 20 cycles, depending
+on the L2 hit and the core-to-L2 routing.
+
+### 15.2 Cache Hierarchy
+
+The L1, L2, and L3 form a hierarchy. The L2 is inclusive of the
+L1. The L3 is the coherence point. The L3 is non-inclusive of
+the L2 by default.
+
+### 15.3 Memory System
+
+The DRAM controllers are described in Section 9. The on-die
+interconnect carries both coherence and data traffic.
+
+### 15.4 Interconnect
+
+The on-die interconnect is described in Section 8. Coherence
+and data traffic share the same physical links.
+
+PROPOSAL: Virtual channels separate the request, response, and
+data traffic. The number of virtual channels per link is 4.
+
+### 15.5 Coherence
+
+Coherence is described in Section 7. The protocol class is
+distributed directory MESI with distributed shared L3.
+
+### 15.6 Interrupts
+
+Interrupts are described in Section 11. The PLIC is on the
+coherence domain.
+
+### 15.7 Operating System
+
+The operating system interactions are described in Section 10.
+The OS is Linux for RISC-V. The firmware is OpenSBI.
+
+### 15.8 Performance
+
+The performance analysis is described in Section 13 and is
+detailed in a separate performance document. Key memory-system
+performance metrics include:
+
+- L1 hit rate
+- L2 hit rate
+- L3 hit rate
+- DRAM bandwidth utilization
+- Average and tail DRAM latency
+- Interconnect average and tail latency
+
+## 16. Recommendations
+
+1. Implement a directory-based MESI coherence protocol with a
+   distributed shared L3. Snoopy protocols are not viable at
+   128 cores.
+2. Use a 2D mesh on-die interconnect. The 2D mesh is sufficient
+   for the projected traffic and is simpler to verify than a
+   torus.
+3. Implement Zicbom and Zicboz cache management operations.
+   These are required for Linux on RISC-V with mainline
+   kernels.
+4. Expose each tile as a NUMA node to the operating system. This
+   allows the kernel scheduler to make topology-aware decisions.
+5. Defer the final decision on the L3 size, the number of DRAM
+   channels, and the directory organization to the
+   implementation phase, after the sensitivity studies listed
+   in Section 12 are completed.
+6. Defer the implementation of Zicbop until a workload study
+   shows a clear benefit.
+
+## 17. References
+
+The following standards and references are cited in this
+document:
+
+- RISC-V Unprivileged ISA Specification, Volume I, Document
+  Version 20191213. Section on the Weak Memory Model and the
+  Cache Management Operations extensions.
+- RISC-V Privileged ISA Specification, Volume II, Document
+  Version 20211203. Sections on `satp`, SFENCE.VMA, PMP, and
+  interrupt handling.
+- RISC-V Profiles, Document Version 20230914. Application and
+  platform profile requirements.
+- RISC-V IOMMU Specification, draft as of the knowledge cutoff.
+- JEDEC DDR5 Standard, JESD79-5.
+- JEDEC LPDDR5 Standard, JESD209-5A.
+- Hennessy and Patterson, "Computer Architecture: A Quantitative
+  Approach", 6th edition. Sections on cache coherence and
+  directory protocols.
+- Culler, Singh, and Gupta, "Parallel Computer Architecture: A
+  Hardware/Software Approach". Chapter on scalable cache
+  coherence.
+- M. M. K. Martin, M. D. Hill, and D. A. Wood, "Token
+  Coherence", University of Wisconsin Computer Sciences
+  Technical Report 1621, 2003. Discusses a directory-based
+  protocol with explicit invalidation tokens.
+- P. Kongetira, K. Aingaran, and K. Olukotun, "Niagara: A
+  32-Way Multithreaded SPARC Processor", IEEE Micro, March 2005.
+  Discusses a directory-coherent CMP design.
+- R. Azimi et al., "Integration Challenges and Tradeoffs for
+  Tiled Many-Core Processors", Proceedings of the 22nd
+  International Conference on Computer Design, 2004.
+- P. Hammarlund et al., "Haswell: The Fourth-Generation Intel
+  Core Processor", IEEE Micro, March 2014. Discusses a
+  ring-based interconnect in a tiled CMP.
+- J. L. Henning, "SPEC CPU2006 Benchmark Descriptions", ACM
+  SIGARCH Computer Architecture News, September 2006.
+- OpenSBI documentation, https://github.com/riscv-software-src/opensbi.
+
+## 18. Document History
+
+- Draft 0.1: initial outline.
+- Draft 0.2: populated physical memory map, coherence protocol
+  class, and L1/L2/L3 sizing as proposals.
+- Draft 0.3: added open questions and recommendations.
+```