``` # Memory Architecture ## Status This document is part of the XH-1 research series. It describes the memory system proposal for the XH-1, a custom 128-core RISC-V processor. Where implementation choices have not been finalized, the text uses **PROPOSAL** to mark a design that is being considered but has not been committed. Where the text is an **ASSUMPTION**, evidence is not yet available and the statement must be re-validated before downstream documents depend on it. ## Scope This document covers: - The physical memory map and address space layout visible to software running on XH-1 cores. - The relationship between the RISC-V privileged architecture and the memory system, including `satp`, SFENCE.VMA, and the PMP. - The on-chip memory hierarchy: L1, L2, and the proposed shared L3. - The directory-based coherence protocol between L2 slices. - The interconnect that ties cores, caches, and DRAM controllers together. - The DRAM controller and the memory-side bandwidth budget. - I/O, interrupts, and operating system interactions that constrain memory-system design. - Verification and performance considerations specific to a 128-core tiled implementation. This document does not define the microarchitecture of an individual core pipeline. That is covered in the core pipeline document. ## 1. Design Goals and Constraints ### 1.1 Goals 1. Provide a coherent, shared-memory programming model to all 128 RISC-V harts. 2. Sustain per-core bandwidth that is high enough to keep an out-of-order core fed under typical server and HPC workloads. 3. Keep average last-level cache (LLC) miss latency within a range that is acceptable for a tiled processor in the same class as published academic and industrial designs. 4. Support a Linux-capable memory management model, including virtual memory, page tables, SFENCE.VMA, and standard I/O MMIO patterns. 5. Allow verification of coherence, ordering, and interrupts with reasonable engineering effort. ### 1.2 Constraints - The RISC-V Privileged specification, ISA Modules for Sv32, Sv39, Sv48, and Sv57, and the RISC-V IOMMU specification, where used, are normative. - The RISC-V Weak Memory Model (RVWMO) is normative for ordering between hart loads and stores. - The coherence protocol must be implementable on a tiled 128-core die in a reasonable process geometry without requiring exotic packaging. - Physical pin bandwidth to DRAM is finite. XH-1 must be budgeted against a realistic number of PHY lanes and channels. Specific numbers are marked as PROPOSAL. ### 1.3 Non-Goals - This document does not specify the instruction set extensions used by XH-1 cores. - It does not specify the precise coherence protocol state machine; it only describes the protocol class. - It does not specify DRAM timing parameters; those are tied to a specific JEDEC standard selected at implementation time. ## 2. RISC-V Architectural Requirements ### 2.1 Address Space The RISC-V Privileged Architecture (Volume II, Version 20211203) of the RISC-V ISA defines: - A canonical physical address space selected by `satp.MODE`. Implementations of `MODE` 0 (Bare), 1 (Sv32), 8 (Sv39), 9 (Sv48), and 10 (Sv57) are mandatory under different profile requirements per the RISC-V Profiles document. - A `satp` CSR that holds the MODE field, the Address Space ID (ASID), and the root page table PPN. Writes to `satp` may become visible to subsequent instruction fetches and loads in a defined order, but require `SFENCE.VMA` for the new translation to be observed by other harts in a coherent manner. - The RISC-V IOMMU specification (riscv-iommu, draft as of the knowledge cutoff) defines DMA translation and the device-context / process-context data structures referenced by I/O devices. PROPOSAL: XH-1 implements Sv39 as the base virtual memory mode and Sv48 as an optional mode. Sv57 is not required for the initial silicon. The `M-mode` bare mapping remains available for firmware and boot loaders. ASSUMPTION: Software for XH-1 will primarily target Linux. Linux for RISC-V supports Sv39 and Sv48 on 64-bit platforms. Implementing both is consistent with the current mainline kernel. ### 2.2 Memory Ordering The RISC-V Weak Memory Model (RVWMO) is defined in the RISC-V Unprivileged ISA specification, Volume I, and the accompanying RISC-V Memory Consistency Model document. RVWMO provides: - Load and store atomicity at aligned word and smaller granularity. - `FENCE`, `FENCE.I`, and `FENCE.VMA` instructions for ordering. - Acquire/release semantics through the `AQ` and `RL` bits on atomic instructions. - The I/O extension ordering rules described in Volume I. The memory system must therefore guarantee that: - Hart-local loads and stores follow RVWMO rules. - Cross-hart ordering is enforced through cache coherence plus `FENCE` and `FENCE.I`. - `SFENCE.VMA` is correctly ordered with respect to subsequent instruction fetches and loads that depend on the new translation. ### 2.3 Physical Memory Attribution (PMA) PMA regions are defined per-hart and describe the address ranges that support main memory, I/O, idempotent reads, non-idempotent writes, and so on. PMA is a static property of the platform and is not affected by `satp`. PROPOSAL: PMA is encoded in a memory-mapped configuration table sampled by each core at reset. The exact format is defined in the platform specification document. ### 2.4 Physical Memory Protection (PMP) The PMP is part of the RISC-V Privileged specification and supports a configurable number of regions per hart. PMP checks occur in M-mode and may be used to isolate firmware and I/O regions from S-mode and U-mode. PROPOSAL: Each hart implements the maximum PMP entry count of 64 when running in M-mode. S-mode translation uses the PMP only for the M-mode firmware range. U-mode never sees the PMP. ASSUMPTION: An M-mode firmware (such as OpenSBI) is in use and relies on PMP to isolate its own memory from the supervisor. ### 2.5 Cache Management Operations The RISC-V Cache Management Operations extension (Zicbom, Zicboz, Zicbop) is defined in Volume I of the RISC-V Unprivileged specification. Operations such as `CBO.INVAL`, `CBO.FLUSH`, and `CBO.CLEAN` are used by software to manage coherence with external agents and DMA. PROPOSAL: XH-1 implements Zicbom and Zicboz. Zicbop is not required for the first silicon and is marked as an open question. (See Section 12.) ## 3. Physical Memory Map PROPOSAL: XH-1 exposes a single 48-bit physical address space to software. The map is divided as follows: | Region | Start | End | Purpose | |-------------------|---------------|---------------|----------------------------------| | DRAM | 0x0000_0000_0000 | 0x007F_FFFF_FFFF | Main memory, up to 32 GiB. | | Reserved | 0x0080_0000_0000 | 0x00FF_FFFF_FFFF | Hole for future expansion. | | High DRAM | 0x0100_0000_0000 | 0x017F_FFFF_FFFF | Optional second DRAM range. | | PCIe MMIO | 0x4000_0000_0000 | 0x40FF_FFFF_FFFF | PCIe ECAM. | | PCIe prefetchable | 0x6000_0000_0000 | 0x60FF_FFFF_FFFF | PCIe MMIO64 prefetchable. | | On-chip I/O | 0x7000_0000_0000 | 0x7000_0FFF_FFFF | Interrupt controller, IOMMU, UART, boot ROM. | | Reserved | 0x8000_0000_0000 | 0xFFFF_FFFF_FFFF | Hole for future use. | The exact upper bound depends on the DRAM capacity and number of DIMMs supported. The values above are PROPOSAL values used to illustrate the layout. ASSUMPTION: DRAM density is between 16 GiB and 1 TiB. The lower bound reflects realistic server configurations for a 128-core chip in 2026; the upper bound is set by the 48-bit physical address limit and by the cost of the memory controller area. ## 4. Tile Organization PROPOSAL: XH-1 is organized as 16 tiles, each containing 8 cores. Each tile includes: - 8 application cores, each with a private L1 instruction cache and a private L1 data cache. - A shared L2 cache, sliced to support one bank per core. - A directory controller for the L3 coherence protocol. - A shared floating-point and vector unit cluster (PROPOSAL: see the vector and floating-point research document for detail). - A tile-local crossbar that connects cores, the L2, the directory controller, and the tile's connection to the global interconnect. The 16 tiles connect through a global on-die interconnect. The interconnect design is described in Section 8. ASSUMPTION: A tile is sized to fit in a 3 mm by 3 mm die region in a 5 nm class process. The exact floorplan is a downstream implementation concern. ## 5. L1 Caches ### 5.1 Sizing PROPOSAL: The L1 instruction cache is 32 KiB, 8-way set-associative, with 64 B lines. The L1 data cache is 32 KiB, 8-way set-associative, with 64 B lines. Both caches are virtually indexed, physically tagged (VIPT), which constrains the index to a number of bits that is at most equal to the page offset. For a 32 KiB, 8-way cache with 64 B lines, the index is 6 bits, the offset is 6 bits, and the tag is 52 bits. With 4 KiB pages, the page offset is 12 bits, so the index is fully contained in the page offset and VIPT is alias-free. ASSUMPTION: 4 KiB pages are the common case. Larger pages such as 2 MiB and 1 GiB reduce aliasing pressure even further, so VIPT remains alias-free for the supported page sizes. ### 5.2 Replacement and Prefetch PROPOSAL: L1 uses a pseudo-LRU replacement policy. Hardware prefetchers include: - A next-line prefetcher on the instruction side. - A stride prefetcher on the data side. - A stream prefetcher on the data side for sequential access patterns. The prefetch degree and aggressiveness are tunable through machine-mode configuration registers. ### 5.3 MSHRs and Write Combining PROPOSAL: Each L1 data cache has 16 miss status handling registers (MSHRs), supporting up to 16 outstanding misses. The write buffer holds up to 16 entries. Combining of partial line writes is performed in the write buffer. ASSUMPTION: 16 MSHRs is sufficient for the workloads of interest. A sensitivity study is required before the L1 MSHR count is finalized. (See Section 12.) ## 6. L2 Cache PROPOSAL: Each tile contains an 8 MiB L2 cache, organized as 8 banks, one per core. Each bank is 1 MiB, 16-way set-associative, with 64 B lines. The L2 is inclusive of the L1 and acts as the coherence point for the L1 caches on the tile. Total L2 capacity across the die is 128 MiB. This number is a PROPOSAL based on the die area budget for caches in the target process node; it is not yet finalized. ### 6.1 Inclusion PROPOSAL: The L2 is strictly inclusive of the L1 caches on the same tile. A coherence invalidation from the L3 directory causes the L2 to invalidate the corresponding L1 line. ASSUMPTION: Strict inclusion simplifies the coherence protocol at the cost of additional L2 traffic for victims. A non-inclusive L2 is an open question. (See Section 12.) ### 6.2 Latency ASSUMPTION: The L2 hit latency is between 12 and 20 cycles. The exact number depends on the physical layout of the cache and the core. This is an open implementation question. ## 7. L3 Cache and Coherence Protocol ### 7.1 Class PROPOSAL: XH-1 uses a directory-based MESI coherence protocol at the L3 level. The directory is distributed across tiles, with one directory controller per tile. Each line in the L3 is owned by exactly one directory, identified by the home tile. The protocol class is "distributed directory MESI with distributed shared L3". This class is the same as the protocol class used in several published academic and industrial designs, including the MIT Alewife, the SGI Origin, the Oracle/Berkeley SPARC M7, and the Fujitsu A64FX. ### 7.2 States The four stable MESI states are: - **M**odified: the line is dirty and held by a single cache. - **E**xclusive: the line is clean and held by a single cache. - **S**hared: the line is clean and may be held by multiple caches. - **I**nvalid: the line is not present. PROPOSAL: A fifth "Owned" state is not used in the base protocol to keep the directory and transient state space simple. A directory entry tracks the presence bits for each tile. ### 7.3 Directory Encoding PROPOSAL: The directory uses a coarse vector of presence bits, one per tile, plus a "has-modified" bit. For 16 tiles this is 17 bits per line. The directory is stored in a dedicated SRAM structure, sized at approximately 12.5 MiB for a 64 MiB L3 with 4x replication of the directory across the on-die fabric to support transient races. ASSUMPTION: The 4x replication of the directory entry is a simplified model. The exact replication strategy is a downstream implementation question. (See Section 12.) ### 7.4 Directory Operations PROPOSAL: Coherence operations follow a request-response pattern: 1. A core issues a read or write to its L1. 2. On L1 miss, the request is forwarded to the L2. 3. On L2 miss, the L2 sends a coherence request to the home tile's directory. 4. The directory invalidates any other copies, forwards the data to the requester, and updates the presence vector. 5. The directory replies to the requester, which completes the L1 fill. Writebacks from L1 invalidate the corresponding L2 line and update the directory. ### 7.5 Memory Consistency Ordering The protocol preserves the appearance of a single global ordering of coherence transactions as seen by the cores. RISC-V `FENCE` ordering is enforced by the L2 controller before forwarding a request to the directory. PROPOSAL: Store atomic instructions (`AMO*` and `LR/SC`) are handled by the L2 controller. The L2 acts as the ordering point for atomic operations and serializes the operation with respect to other requests to the same address. ### 7.6 Forwarding and Snoop Filtering PROPOSAL: The directory acts as a snoop filter. A coherence request that finds the directory in the "S" state with no other sharer can supply the data from the L2 without further snoop traffic. ## 8. On-Die Interconnect ### 8.1 Topology PROPOSAL: The on-die interconnect is a 2D mesh of 4 by 4 tiles. Each tile has a router that connects to its four nearest neighbors. The mesh supports a 512-bit link per direction, clocked at a fraction of the core clock. ASSUMPTION: A 2D mesh is sufficient for the bandwidth requirements of a 128-core die. A 2D torus is an alternative discussed in Section 12. ### 8.2 Routing PROPOSAL: The mesh uses deterministic X-Y routing. Adaptive routing is not used in the first silicon to simplify verification. ### 8.3 Link Bandwidth PROPOSAL: Each link provides 64 B per cycle of useful payload at the router clock. With 16 tiles and 4 directions per router (excluding the boundary cases), the bisection bandwidth of the mesh is 2 links by 4 tiles by 64 B per cycle, or 512 B per cycle. At a 1.5 GHz router clock this is 768 GB/s of bisection bandwidth. ASSUMPTION: The 1.5 GHz router clock is feasible in the target process node. This is an open implementation question. ### 8.4 Flow Control PROPOSAL: The interconnect uses credit-based flow control on each link. Each virtual channel maintains a count of available credits and stalls a flit only when the downstream router is unable to accept it. ## 9. DRAM Controller ### 9.1 Topology PROPOSAL: XH-1 has 8 DRAM controllers, distributed on the die. Two controllers are placed near the center of each edge. The controllers connect to the mesh with high-bandwidth links and to external PHYs through the package substrate. ### 9.2 Channels and DIMMs PROPOSAL: Each controller supports 2 channels, for a total of 16 channels. Each channel supports a single DDR5 DIMM in the first silicon. ### 9.3 Bandwidth ASSUMPTION: At DDR5-6400, a 64-bit channel provides 51.2 GB/s peak. With 16 channels, the peak DRAM bandwidth is 819 GB/s. PROPOSAL: The memory system is designed to sustain an average L3 miss rate of 0.05 misses per instruction across the 128 cores at the rated clock. This is consistent with a 1.0 GHz core clock and an IPC of 1.0, giving 128 GIPS, and a DRAM traffic of 0.05 misses by 128 GIPS by 64 B per miss, or 410 GB/s. This is below the 819 GB/s peak, leaving headroom for OS and I/O traffic. ### 9.4 Quality of Service PROPOSAL: The DRAM controller supports 4 QoS classes. A configuration register in the platform specification selects the class for each request source (core, device, I/O). ASSUMPTION: A weighted round-robin scheduler per channel is sufficient for the first silicon. A more sophisticated scheduler is an open question. ## 10. Operating System and Firmware ### 10.1 Linux Memory Model PROPOSAL: XH-1 supports mainline Linux for RISC-V. The relevant kernel features are: - Sv39 and Sv48 virtual memory. - NUMA-aware scheduling, with one NUMA node per tile. - Standard cache management operations through Zicbom. - PCIe MMU through the optional RISC-V IOMMU. ASSUMPTION: Linux for RISC-V provides all of the above features on a 64-bit SMP platform. This assumption should be re-validated against the kernel version targeted for the first silicon. ### 10.2 NUMA Effects PROPOSAL: Each tile is exposed as a NUMA node to Linux. The operating system scheduler places a task on the node closest to its memory. The boot firmware sets up the device tree and ACPI table accordingly. ### 10.3 Firmware and PMP PROPOSAL: OpenSBI is the M-mode firmware. OpenSBI uses PMP to isolate its own memory from S-mode. The SBI interface is the primary interface between S-mode and M-mode. ### 10.4 IOMMU and DMA PROPOSAL: A RISC-V IOMMU is included on the die. The IOMMU translates device DMA addresses using the same page table formats as the CPU. PCIe devices that support ATS bypass the IOMMU on a per-transaction basis. ASSUMPTION: The RISC-V IOMMU specification is sufficiently mature at the time of tape-out. If not, the IOMMU is replaced by a software bounce-buffering scheme in the first silicon. ## 11. Interrupts and Memory System Interactions ### 11.1 Interrupt Delivery PROPOSAL: A platform-level interrupt controller (PLIC) is mapped at the on-chip I/O region. The PLIC supports 256 interrupt sources and 128 interrupt targets, one per hart. The PLIC accesses are strongly ordered with respect to the coherence protocol. ### 11.2 Interrupt and Coherence ASSUMPTION: The interrupt controller is on the coherence domain. An interrupt write from a device is visible to the PLIC registers, and the subsequent PLIC read by the core is ordered through the coherence protocol. ### 11.3 Interrupt Latency ASSUMPTION: Worst-case interrupt latency through the PLIC is under 1 microsecond at the rated clock, including coherence round-trip and pipeline drain. The actual value is workload dependent and is left for the performance document. ## 12. Open Questions and Tradeoffs The following questions are unresolved at the time of writing. Each is a candidate for further study before the architecture is frozen. 1. **Inclusive vs. non-inclusive L2.** Strict inclusion simplifies coherence but may waste L2 capacity. A non-inclusive victim cache design is an alternative. 2. **L3 size.** A 64 MiB L3 is the proposal. A 32 MiB or 128 MiB L3 are alternatives; each changes the area budget and miss rate. 3. **Directory organization.** A coarse vector with replication is the proposal. A hierarchical or sparse directory is an alternative that may reduce area at the cost of latency. 4. **Mesh vs. torus.** A 2D mesh is the proposal. A 2D torus doubles the bisection bandwidth and may be required for very high cache miss rates. A ring or concentrated mesh is a third alternative. 5. **Number of memory channels.** 16 channels is the proposal. 8 or 24 channels are alternatives. 6. **Zicbop implementation.** Cache block prefetch operations are optional in the RISC-V Cache Management Operations specification. Their cost and benefit on XH-1 are not yet quantified. 7. **L1 MSHR count.** 16 MSHRs per L1 is the proposal. The workload-dependent optimum is not known. 8. **IOMMU maturity.** The RISC-V IOMMU specification was a draft at the knowledge cutoff date. The XH-1 implementation may need to wait for a stable revision. 9. **Coherence at the L1 level.** The current proposal collapses L1 coherence to the L2. An alternative is a broadcast snoop between L1s on the same tile. This trades L1 bandwidth for L2 traffic. 10. **SMT support.** Each core may support a small number of hardware threads. SMT changes the coherence and memory system pressure per core. The number of threads per core is not yet decided. 11. **Vector and matrix unit interactions.** The vector unit performs high-bandwidth loads and stores. The memory system must be able to support sustained vector load bandwidth at the rated clock. The vector unit design appears in a separate document. 12. **Power management.** Clock gating, power gating, and DVFS interact with the memory system. The current document does not yet address these interactions. ## 13. Scalability Analysis for 128 Cores ### 13.1 Coherence Scalability A broadcast snoop protocol scales as O(N^2) in the worst case, where N is the number of cores. For 128 cores this is 16,384 worst-case broadcast pairs, which is generally not viable on a single die. The directory protocol scales as O(N) in the worst case for invalidations, and the directory size grows linearly with the number of tiles. PROPOSAL: A directory protocol is required at 128 cores. A snoopy protocol is not viable. ### 13.2 Interconnect Scalability A 2D mesh of 16 tiles has bisection bandwidth proportional to the number of tiles per row. The 4 by 4 mesh provides 4 links of bisection, which is sufficient for the projected 410 GB/s of average DRAM traffic in Section 9.3. ASSUMPTION: Average traffic does not exceed 70 percent of the peak bisection. A sensitivity analysis is required. (See Section 12.) ### 13.3 Directory Storage Scalability The directory entry is 17 bits per L3 line. For a 64 MiB L3 with 64 B lines, this is 2^20 lines by 17 bits, or 2.2 MiB per directory. With 4x replication, the on-die directory storage is approximately 12.5 MiB. This is a small fraction of the L3 storage. ### 13.4 Operating System Scalability Linux scales to hundreds of cores. The memory system must expose the topology through ACPI and the device tree so that the kernel NUMA scheduler can make informed decisions. The 128-core XH-1 is below the upper bound of common server Linux configurations as of the knowledge cutoff. ### 13.5 Verification Scalability Coherence verification grows in complexity with the number of cores. PROPOSAL: a combination of formal verification of the directory protocol state machine, simulation-based testing, and random instruction generation is used. The verification strategy is described in the verification document. ## 14. Verification ### 14.1 Cache Coherence PROPOSAL: The directory protocol state machine is verified using a model checker against the MESI invariants. The invariants include: - Single writer per line. - No read of a line in the Invalid state. - Correct invalidation on a write to a shared line. ### 14.2 Memory Consistency PROPOSAL: A RISC-V memory consistency model litmus test suite is run against the system, using both RTL simulation and a software reference model. ### 14.3 Performance and Stress Testing PROPOSAL: Synthetic traffic patterns, including uniform random, hotspot, and adversarial patterns, are used to verify that the memory system behaves correctly under stress. Realistic workloads, including SPEC CPU 2017, PARSEC, and selected GROMACS and LAMMPS runs, are used for performance validation. ASSUMPTION: The listed benchmarks are representative of the target workloads. This assumption should be re-validated by the performance team. ## 15. Interaction With Other Subsystems ### 15.1 Pipeline The L1 cache is in the core pipeline. The L1 hit latency is visible in the branch resolution and the load-to-use latency. PROPOSAL: The L1 hit latency is 4 cycles for a load-to-use on a hit. A miss to the L2 adds between 12 and 20 cycles, depending on the L2 hit and the core-to-L2 routing. ### 15.2 Cache Hierarchy The L1, L2, and L3 form a hierarchy. The L2 is inclusive of the L1. The L3 is the coherence point. The L3 is non-inclusive of the L2 by default. ### 15.3 Memory System The DRAM controllers are described in Section 9. The on-die interconnect carries both coherence and data traffic. ### 15.4 Interconnect The on-die interconnect is described in Section 8. Coherence and data traffic share the same physical links. PROPOSAL: Virtual channels separate the request, response, and data traffic. The number of virtual channels per link is 4. ### 15.5 Coherence Coherence is described in Section 7. The protocol class is distributed directory MESI with distributed shared L3. ### 15.6 Interrupts Interrupts are described in Section 11. The PLIC is on the coherence domain. ### 15.7 Operating System The operating system interactions are described in Section 10. The OS is Linux for RISC-V. The firmware is OpenSBI. ### 15.8 Performance The performance analysis is described in Section 13 and is detailed in a separate performance document. Key memory-system performance metrics include: - L1 hit rate - L2 hit rate - L3 hit rate - DRAM bandwidth utilization - Average and tail DRAM latency - Interconnect average and tail latency ## 16. Recommendations 1. Implement a directory-based MESI coherence protocol with a distributed shared L3. Snoopy protocols are not viable at 128 cores. 2. Use a 2D mesh on-die interconnect. The 2D mesh is sufficient for the projected traffic and is simpler to verify than a torus. 3. Implement Zicbom and Zicboz cache management operations. These are required for Linux on RISC-V with mainline kernels. 4. Expose each tile as a NUMA node to the operating system. This allows the kernel scheduler to make topology-aware decisions. 5. Defer the final decision on the L3 size, the number of DRAM channels, and the directory organization to the implementation phase, after the sensitivity studies listed in Section 12 are completed. 6. Defer the implementation of Zicbop until a workload study shows a clear benefit. ## 17. References The following standards and references are cited in this document: - RISC-V Unprivileged ISA Specification, Volume I, Document Version 20191213. Section on the Weak Memory Model and the Cache Management Operations extensions. - RISC-V Privileged ISA Specification, Volume II, Document Version 20211203. Sections on `satp`, SFENCE.VMA, PMP, and interrupt handling. - RISC-V Profiles, Document Version 20230914. Application and platform profile requirements. - RISC-V IOMMU Specification, draft as of the knowledge cutoff. - JEDEC DDR5 Standard, JESD79-5. - JEDEC LPDDR5 Standard, JESD209-5A. - Hennessy and Patterson, "Computer Architecture: A Quantitative Approach", 6th edition. Sections on cache coherence and directory protocols. - Culler, Singh, and Gupta, "Parallel Computer Architecture: A Hardware/Software Approach". Chapter on scalable cache coherence. - M. M. K. Martin, M. D. Hill, and D. A. Wood, "Token Coherence", University of Wisconsin Computer Sciences Technical Report 1621, 2003. Discusses a directory-based protocol with explicit invalidation tokens. - P. Kongetira, K. Aingaran, and K. Olukotun, "Niagara: A 32-Way Multithreaded SPARC Processor", IEEE Micro, March 2005. Discusses a directory-coherent CMP design. - R. Azimi et al., "Integration Challenges and Tradeoffs for Tiled Many-Core Processors", Proceedings of the 22nd International Conference on Computer Design, 2004. - P. Hammarlund et al., "Haswell: The Fourth-Generation Intel Core Processor", IEEE Micro, March 2014. Discusses a ring-based interconnect in a tiled CMP. - J. L. Henning, "SPEC CPU2006 Benchmark Descriptions", ACM SIGARCH Computer Architecture News, September 2006. - OpenSBI documentation, https://github.com/riscv-software-src/opensbi. ## 18. Document History - Draft 0.1: initial outline. - Draft 0.2: populated physical memory map, coherence protocol class, and L1/L2/L3 sizing as proposals. - Draft 0.3: added open questions and recommendations. ```