xh1-research-fork/research/05-memory/memory-architecture.md

# Memory Architecture

## Status

This document is part of the XH-1 research series. It describes the
memory system proposal for the XH-1, a custom 128-core RISC-V
processor. Where implementation choices have not been finalized, the
text uses **PROPOSAL** to mark a design that is being considered but
has not been committed. Where the text is an **ASSUMPTION**, evidence
is not yet available and the statement must be re-validated before
downstream documents depend on it.

## Scope

This document covers:

- The physical memory map and address space layout visible to software
  running on XH-1 cores.
- The relationship between the RISC-V privileged architecture and the
  memory system, including `satp`, SFENCE.VMA, and the PMP.
- The on-chip memory hierarchy: L1, L2, and the proposed shared L3.
- The directory-based coherence protocol between L2 slices.
- The interconnect that ties cores, caches, and DRAM controllers
  together.
- The DRAM controller and the memory-side bandwidth budget.
- I/O, interrupts, and operating system interactions that constrain
  memory-system design.
- Verification and performance considerations specific to a 128-core
  tiled implementation.

This document does not define the microarchitecture of an individual
core pipeline. That is covered in the core pipeline document.

## 1. Design Goals and Constraints

### 1.1 Goals

1. Provide a coherent, shared-memory programming model to all
   128 RISC-V harts.
2. Sustain per-core bandwidth that is high enough to keep an
   out-of-order core fed under typical server and HPC workloads.
3. Keep average last-level cache (LLC) miss latency within a range
   that is acceptable for a tiled processor in the same class as
   published academic and industrial designs.
4. Support a Linux-capable memory management model, including virtual
   memory, page tables, SFENCE.VMA, and standard I/O MMIO patterns.
5. Allow verification of coherence, ordering, and interrupts with
   reasonable engineering effort.

### 1.2 Constraints

- The RISC-V Privileged specification, ISA Modules for Sv32, Sv39,
  Sv48, and Sv57, and the RISC-V IOMMU specification, where used,
  are normative.
- The RISC-V Weak Memory Model (RVWMO) is normative for ordering
  between hart loads and stores.
- The coherence protocol must be implementable on a tiled
  128-core die in a reasonable process geometry without
  requiring exotic packaging.
- Physical pin bandwidth to DRAM is finite. XH-1 must be
  budgeted against a realistic number of PHY lanes and channels.
  Specific numbers are marked as PROPOSAL.

### 1.3 Non-Goals

- This document does not specify the instruction set extensions used
  by XH-1 cores.
- It does not specify the precise coherence protocol state machine;
  it only describes the protocol class.
- It does not specify DRAM timing parameters; those are tied to a
  specific JEDEC standard selected at implementation time.

## 2. RISC-V Architectural Requirements

### 2.1 Address Space

The RISC-V Privileged Architecture (Volume II, Version 20211203) of
the RISC-V ISA defines:

- A canonical physical address space selected by `satp.MODE`.
  Implementations of `MODE` 0 (Bare), 1 (Sv32), 8 (Sv39), 9 (Sv48),
  and 10 (Sv57) are mandatory under different profile requirements
  per the RISC-V Profiles document.
- A `satp` CSR that holds the MODE field, the Address Space ID
  (ASID), and the root page table PPN. Writes to `satp` may become
  visible to subsequent instruction fetches and loads in a defined
  order, but require `SFENCE.VMA` for the new translation to be
  observed by other harts in a coherent manner.
- The RISC-V IOMMU specification (riscv-iommu, draft as of the
  knowledge cutoff) defines DMA translation and the
  device-context / process-context data structures referenced by
  I/O devices.

PROPOSAL: XH-1 implements Sv39 as the base virtual memory mode and
Sv48 as an optional mode. Sv57 is not required for the initial
silicon. The `M-mode` bare mapping remains available for firmware
and boot loaders.

ASSUMPTION: Software for XH-1 will primarily target Linux. Linux
for RISC-V supports Sv39 and Sv48 on 64-bit platforms. Implementing
both is consistent with the current mainline kernel.

### 2.2 Memory Ordering

The RISC-V Weak Memory Model (RVWMO) is defined in the RISC-V
Unprivileged ISA specification, Volume I, and the accompanying
RISC-V Memory Consistency Model document. RVWMO provides:

- Load and store atomicity at aligned word and smaller granularity.
- `FENCE`, `FENCE.I`, and `FENCE.VMA` instructions for ordering.
- Acquire/release semantics through the `AQ` and `RL` bits on
  atomic instructions.
- The I/O extension ordering rules described in Volume I.

The memory system must therefore guarantee that:

- Hart-local loads and stores follow RVWMO rules.
- Cross-hart ordering is enforced through cache coherence plus
  `FENCE` and `FENCE.I`.
- `SFENCE.VMA` is correctly ordered with respect to subsequent
  instruction fetches and loads that depend on the new translation.

### 2.3 Physical Memory Attribution (PMA)

PMA regions are defined per-hart and describe the address ranges
that support main memory, I/O, idempotent reads, non-idempotent
writes, and so on. PMA is a static property of the platform and is
not affected by `satp`.

PROPOSAL: PMA is encoded in a memory-mapped configuration table
sampled by each core at reset. The exact format is defined in the
platform specification document.

### 2.4 Physical Memory Protection (PMP)

The PMP is part of the RISC-V Privileged specification and supports
a configurable number of regions per hart. PMP checks occur in
M-mode and may be used to isolate firmware and I/O regions from
S-mode and U-mode.

PROPOSAL: Each hart implements the maximum PMP entry count of 64
when running in M-mode. S-mode translation uses the PMP only for
the M-mode firmware range. U-mode never sees the PMP.

ASSUMPTION: An M-mode firmware (such as OpenSBI) is in use and
relies on PMP to isolate its own memory from the supervisor.

### 2.5 Cache Management Operations

The RISC-V Cache Management Operations extension (Zicbom, Zicboz,
Zicbop) is defined in Volume I of the RISC-V Unprivileged
specification. Operations such as `CBO.INVAL`, `CBO.FLUSH`, and
`CBO.CLEAN` are used by software to manage coherence with external
agents and DMA.

PROPOSAL: XH-1 implements Zicbom and Zicboz. Zicbop is not required
for the first silicon and is marked as an open question. (See
Section 12.)

## 3. Physical Memory Map

PROPOSAL: XH-1 exposes a single 48-bit physical address space to
software. The map is divided as follows:

| Region            | Start         | End           | Purpose                          |
|-------------------|---------------|---------------|----------------------------------|
| DRAM              | 0x0000_0000_0000 | 0x007F_FFFF_FFFF | Main memory, up to 32 GiB.  |
| Reserved          | 0x0080_0000_0000 | 0x00FF_FFFF_FFFF | Hole for future expansion.  |
| High DRAM         | 0x0100_0000_0000 | 0x017F_FFFF_FFFF | Optional second DRAM range. |
| PCIe MMIO         | 0x4000_0000_0000 | 0x40FF_FFFF_FFFF | PCIe ECAM.                 |
| PCIe prefetchable | 0x6000_0000_0000 | 0x60FF_FFFF_FFFF | PCIe MMIO64 prefetchable.  |
| On-chip I/O       | 0x7000_0000_0000 | 0x7000_0FFF_FFFF | Interrupt controller, IOMMU, UART, boot ROM. |
| Reserved          | 0x8000_0000_0000 | 0xFFFF_FFFF_FFFF | Hole for future use.       |

The exact upper bound depends on the DRAM capacity and number of
DIMMs supported. The values above are PROPOSAL values used to
illustrate the layout.

ASSUMPTION: DRAM density is between 16 GiB and 1 TiB. The lower
bound reflects realistic server configurations for a 128-core chip
in 2026; the upper bound is set by the 48-bit physical address
limit and by the cost of the memory controller area.

## 4. Tile Organization

PROPOSAL: XH-1 is organized as 16 tiles, each containing 8 cores.
Each tile includes:

- 8 application cores, each with a private L1 instruction cache
  and a private L1 data cache.
- A shared L2 cache, sliced to support one bank per core.
- A directory controller for the L3 coherence protocol.
- A shared floating-point and vector unit cluster (PROPOSAL: see
  the vector and floating-point research document for detail).
- A tile-local crossbar that connects cores, the L2, the
  directory controller, and the tile's connection to the global
  interconnect.

The 16 tiles connect through a global on-die interconnect. The
interconnect design is described in Section 8.

ASSUMPTION: A tile is sized to fit in a 3 mm by 3 mm die region in
a 5 nm class process. The exact floorplan is a downstream
implementation concern.

## 5. L1 Caches

### 5.1 Sizing

PROPOSAL: The L1 instruction cache is 32 KiB, 8-way set-associative,
with 64 B lines. The L1 data cache is 32 KiB, 8-way set-associative,
with 64 B lines. Both caches are virtually indexed, physically
tagged (VIPT), which constrains the index to a number of bits that
is at most equal to the page offset.

For a 32 KiB, 8-way cache with 64 B lines, the index is 6 bits, the
offset is 6 bits, and the tag is 52 bits. With 4 KiB pages, the
page offset is 12 bits, so the index is fully contained in the page
offset and VIPT is alias-free.

ASSUMPTION: 4 KiB pages are the common case. Larger pages such as
2 MiB and 1 GiB reduce aliasing pressure even further, so VIPT
remains alias-free for the supported page sizes.

### 5.2 Replacement and Prefetch

PROPOSAL: L1 uses a pseudo-LRU replacement policy. Hardware
prefetchers include:

- A next-line prefetcher on the instruction side.
- A stride prefetcher on the data side.
- A stream prefetcher on the data side for sequential access
  patterns.

The prefetch degree and aggressiveness are tunable through
machine-mode configuration registers.

### 5.3 MSHRs and Write Combining

PROPOSAL: Each L1 data cache has 16 miss status handling
registers (MSHRs), supporting up to 16 outstanding misses. The
write buffer holds up to 16 entries. Combining of partial line
writes is performed in the write buffer.

ASSUMPTION: 16 MSHRs is sufficient for the workloads of interest.
A sensitivity study is required before the L1 MSHR count is
finalized. (See Section 12.)

## 6. L2 Cache

PROPOSAL: Each tile contains an 8 MiB L2 cache, organized as 8
banks, one per core. Each bank is 1 MiB, 16-way set-associative,
with 64 B lines. The L2 is inclusive of the L1 and acts as the
coherence point for the L1 caches on the tile.

Total L2 capacity across the die is 128 MiB. This number is a
PROPOSAL based on the die area budget for caches in the target
process node; it is not yet finalized.

### 6.1 Inclusion

PROPOSAL: The L2 is strictly inclusive of the L1 caches on the
same tile. A coherence invalidation from the L3 directory causes
the L2 to invalidate the corresponding L1 line.

ASSUMPTION: Strict inclusion simplifies the coherence protocol at
the cost of additional L2 traffic for victims. A non-inclusive
L2 is an open question. (See Section 12.)

### 6.2 Latency

ASSUMPTION: The L2 hit latency is between 12 and 20 cycles. The
exact number depends on the physical layout of the cache and the
core. This is an open implementation question.

## 7. L3 Cache and Coherence Protocol

### 7.1 Class

PROPOSAL: XH-1 uses a directory-based MESI coherence protocol at
the L3 level. The directory is distributed across tiles, with one
directory controller per tile. Each line in the L3 is owned by
exactly one directory, identified by the home tile.

The protocol class is "distributed directory MESI with distributed
shared L3". This class is the same as the protocol class used in
several published academic and industrial designs, including the
MIT Alewife, the SGI Origin, the Oracle/Berkeley SPARC M7, and the
Fujitsu A64FX.

### 7.2 States

The four stable MESI states are:

- **M**odified: the line is dirty and held by a single cache.
- **E**xclusive: the line is clean and held by a single cache.
- **S**hared: the line is clean and may be held by multiple
  caches.
- **I**nvalid: the line is not present.

PROPOSAL: A fifth "Owned" state is not used in the base protocol
to keep the directory and transient state space simple. A
directory entry tracks the presence bits for each tile.

### 7.3 Directory Encoding

PROPOSAL: The directory uses a coarse vector of presence bits,
one per tile, plus a "has-modified" bit. For 16 tiles this is 17
bits per line. The directory is stored in a dedicated SRAM
structure, sized at approximately 12.5 MiB for a 64 MiB L3 with
4x replication of the directory across the on-die fabric to
support transient races.

ASSUMPTION: The 4x replication of the directory entry is a
simplified model. The exact replication strategy is a downstream
implementation question. (See Section 12.)

### 7.4 Directory Operations

PROPOSAL: Coherence operations follow a request-response pattern:

1. A core issues a read or write to its L1.
2. On L1 miss, the request is forwarded to the L2.
3. On L2 miss, the L2 sends a coherence request to the home tile's
   directory.
4. The directory invalidates any other copies, forwards the data
   to the requester, and updates the presence vector.
5. The directory replies to the requester, which completes the
   L1 fill.

Writebacks from L1 invalidate the corresponding L2 line and
update the directory.

### 7.5 Memory Consistency Ordering

The protocol preserves the appearance of a single global ordering
of coherence transactions as seen by the cores. RISC-V `FENCE`
ordering is enforced by the L2 controller before forwarding a
request to the directory.

PROPOSAL: Store atomic instructions (`AMO*` and `LR/SC`) are
handled by the L2 controller. The L2 acts as the ordering point
for atomic operations and serializes the operation with respect
to other requests to the same address.

### 7.6 Forwarding and Snoop Filtering

PROPOSAL: The directory acts as a snoop filter. A coherence
request that finds the directory in the "S" state with no other
sharer can supply the data from the L2 without further snoop
traffic.

## 8. On-Die Interconnect

### 8.1 Topology

PROPOSAL: The on-die interconnect is a 2D mesh of 4 by 4 tiles.
Each tile has a router that connects to its four nearest neighbors.
The mesh supports a 512-bit link per direction, clocked at a
fraction of the core clock.

ASSUMPTION: A 2D mesh is sufficient for the bandwidth requirements
of a 128-core die. A 2D torus is an alternative discussed in
Section 12.

### 8.2 Routing

PROPOSAL: The mesh uses deterministic X-Y routing. Adaptive routing
is not used in the first silicon to simplify verification.

### 8.3 Link Bandwidth

PROPOSAL: Each link provides 64 B per cycle of useful payload at
the router clock. With 16 tiles and 4 directions per router
(excluding the boundary cases), the bisection bandwidth of the
mesh is 2 links by 4 tiles by 64 B per cycle, or 512 B per cycle.
At a 1.5 GHz router clock this is 768 GB/s of bisection
bandwidth.

ASSUMPTION: The 1.5 GHz router clock is feasible in the target
process node. This is an open implementation question.

### 8.4 Flow Control

PROPOSAL: The interconnect uses credit-based flow control on each
link. Each virtual channel maintains a count of available credits
and stalls a flit only when the downstream router is unable to
accept it.

## 9. DRAM Controller

### 9.1 Topology

PROPOSAL: XH-1 has 8 DRAM controllers, distributed on the die. Two
controllers are placed near the center of each edge. The
controllers connect to the mesh with high-bandwidth links and to
external PHYs through the package substrate.

### 9.2 Channels and DIMMs

PROPOSAL: Each controller supports 2 channels, for a total of 16
channels. Each channel supports a single DDR5 DIMM in the first
silicon.

### 9.3 Bandwidth

ASSUMPTION: At DDR5-6400, a 64-bit channel provides 51.2 GB/s
peak. With 16 channels, the peak DRAM bandwidth is 819 GB/s.

PROPOSAL: The memory system is designed to sustain an average
L3 miss rate of 0.05 misses per instruction across the 128 cores
at the rated clock. This is consistent with a 1.0 GHz core clock
and an IPC of 1.0, giving 128 GIPS, and a DRAM traffic of 0.05
misses by 128 GIPS by 64 B per miss, or 410 GB/s. This is below
the 819 GB/s peak, leaving headroom for OS and I/O traffic.

### 9.4 Quality of Service

PROPOSAL: The DRAM controller supports 4 QoS classes. A
configuration register in the platform specification selects the
class for each request source (core, device, I/O).

ASSUMPTION: A weighted round-robin scheduler per channel is
sufficient for the first silicon. A more sophisticated scheduler
is an open question.

## 10. Operating System and Firmware

### 10.1 Linux Memory Model

PROPOSAL: XH-1 supports mainline Linux for RISC-V. The relevant
kernel features are:

- Sv39 and Sv48 virtual memory.
- NUMA-aware scheduling, with one NUMA node per tile.
- Standard cache management operations through Zicbom.
- PCIe MMU through the optional RISC-V IOMMU.

ASSUMPTION: Linux for RISC-V provides all of the above features
on a 64-bit SMP platform. This assumption should be re-validated
against the kernel version targeted for the first silicon.

### 10.2 NUMA Effects

PROPOSAL: Each tile is exposed as a NUMA node to Linux. The
operating system scheduler places a task on the node closest to
its memory. The boot firmware sets up the device tree and ACPI
table accordingly.

### 10.3 Firmware and PMP

PROPOSAL: OpenSBI is the M-mode firmware. OpenSBI uses PMP to
isolate its own memory from S-mode. The SBI interface is the
primary interface between S-mode and M-mode.

### 10.4 IOMMU and DMA

PROPOSAL: A RISC-V IOMMU is included on the die. The IOMMU
translates device DMA addresses using the same page table
formats as the CPU. PCIe devices that support ATS bypass the
IOMMU on a per-transaction basis.

ASSUMPTION: The RISC-V IOMMU specification is sufficiently
mature at the time of tape-out. If not, the IOMMU is replaced
by a software bounce-buffering scheme in the first silicon.

## 11. Interrupts and Memory System Interactions

### 11.1 Interrupt Delivery

PROPOSAL: A platform-level interrupt controller (PLIC) is mapped
at the on-chip I/O region. The PLIC supports 256 interrupt
sources and 128 interrupt targets, one per hart. The PLIC
accesses are strongly ordered with respect to the coherence
protocol.

### 11.2 Interrupt and Coherence

ASSUMPTION: The interrupt controller is on the coherence domain.
An interrupt write from a device is visible to the PLIC
registers, and the subsequent PLIC read by the core is ordered
through the coherence protocol.

### 11.3 Interrupt Latency

ASSUMPTION: Worst-case interrupt latency through the PLIC is
under 1 microsecond at the rated clock, including coherence
round-trip and pipeline drain. The actual value is workload
dependent and is left for the performance document.

## 12. Open Questions and Tradeoffs

The following questions are unresolved at the time of writing.
Each is a candidate for further study before the architecture is
frozen.

1. **Inclusive vs. non-inclusive L2.** Strict inclusion
   simplifies coherence but may waste L2 capacity. A
   non-inclusive victim cache design is an alternative.
2. **L3 size.** A 64 MiB L3 is the proposal. A 32 MiB or 128 MiB
   L3 are alternatives; each changes the area budget and miss
   rate.
3. **Directory organization.** A coarse vector with replication
   is the proposal. A hierarchical or sparse directory is an
   alternative that may reduce area at the cost of latency.
4. **Mesh vs. torus.** A 2D mesh is the proposal. A 2D torus
   doubles the bisection bandwidth and may be required for very
   high cache miss rates. A ring or concentrated mesh is a
   third alternative.
5. **Number of memory channels.** 16 channels is the proposal.
   8 or 24 channels are alternatives.
6. **Zicbop implementation.** Cache block prefetch operations are
   optional in the RISC-V Cache Management Operations
   specification. Their cost and benefit on XH-1 are not yet
   quantified.
7. **L1 MSHR count.** 16 MSHRs per L1 is the proposal. The
   workload-dependent optimum is not known.
8. **IOMMU maturity.** The RISC-V IOMMU specification was a
   draft at the knowledge cutoff date. The XH-1 implementation
   may need to wait for a stable revision.
9. **Coherence at the L1 level.** The current proposal
   collapses L1 coherence to the L2. An alternative is a
   broadcast snoop between L1s on the same tile. This trades
   L1 bandwidth for L2 traffic.
10. **SMT support.** Each core may support a small number of
    hardware threads. SMT changes the coherence and memory
    system pressure per core. The number of threads per core
    is not yet decided.
11. **Vector and matrix unit interactions.** The vector unit
    performs high-bandwidth loads and stores. The memory
    system must be able to support sustained vector load
    bandwidth at the rated clock. The vector unit design
    appears in a separate document.
12. **Power management.** Clock gating, power gating, and DVFS
    interact with the memory system. The current document does
    not yet address these interactions.

## 13. Scalability Analysis for 128 Cores

### 13.1 Coherence Scalability

A broadcast snoop protocol scales as O(N^2) in the worst case,
where N is the number of cores. For 128 cores this is 16,384
worst-case broadcast pairs, which is generally not viable on a
single die. The directory protocol scales as O(N) in the worst
case for invalidations, and the directory size grows linearly
with the number of tiles.

PROPOSAL: A directory protocol is required at 128 cores. A
snoopy protocol is not viable.

### 13.2 Interconnect Scalability

A 2D mesh of 16 tiles has bisection bandwidth proportional to
the number of tiles per row. The 4 by 4 mesh provides 4 links of
bisection, which is sufficient for the projected 410 GB/s of
average DRAM traffic in Section 9.3.

ASSUMPTION: Average traffic does not exceed 70 percent of the
peak bisection. A sensitivity analysis is required. (See
Section 12.)

### 13.3 Directory Storage Scalability

The directory entry is 17 bits per L3 line. For a 64 MiB L3 with
64 B lines, this is 2^20 lines by 17 bits, or 2.2 MiB per
directory. With 4x replication, the on-die directory storage is
approximately 12.5 MiB. This is a small fraction of the L3
storage.

### 13.4 Operating System Scalability

Linux scales to hundreds of cores. The memory system must
expose the topology through ACPI and the device tree so that the
kernel NUMA scheduler can make informed decisions. The
128-core XH-1 is below the upper bound of common server Linux
configurations as of the knowledge cutoff.

### 13.5 Verification Scalability

Coherence verification grows in complexity with the number of
cores. PROPOSAL: a combination of formal verification of the
directory protocol state machine, simulation-based testing, and
random instruction generation is used. The verification
strategy is described in the verification document.

## 14. Verification

### 14.1 Cache Coherence

PROPOSAL: The directory protocol state machine is verified using
a model checker against the MESI invariants. The invariants
include:

- Single writer per line.
- No read of a line in the Invalid state.
- Correct invalidation on a write to a shared line.

### 14.2 Memory Consistency

PROPOSAL: A RISC-V memory consistency model litmus test suite
is run against the system, using both RTL simulation and a
software reference model.

### 14.3 Performance and Stress Testing

PROPOSAL: Synthetic traffic patterns, including uniform random,
hotspot, and adversarial patterns, are used to verify that the
memory system behaves correctly under stress. Realistic
workloads, including SPEC CPU 2017, PARSEC, and selected
GROMACS and LAMMPS runs, are used for performance validation.

ASSUMPTION: The listed benchmarks are representative of the
target workloads. This assumption should be re-validated by the
performance team.

## 15. Interaction With Other Subsystems

### 15.1 Pipeline

The L1 cache is in the core pipeline. The L1 hit latency is
visible in the branch resolution and the load-to-use latency.
PROPOSAL: The L1 hit latency is 4 cycles for a load-to-use on a
hit. A miss to the L2 adds between 12 and 20 cycles, depending
on the L2 hit and the core-to-L2 routing.

### 15.2 Cache Hierarchy

The L1, L2, and L3 form a hierarchy. The L2 is inclusive of the
L1. The L3 is the coherence point. The L3 is non-inclusive of
the L2 by default.

### 15.3 Memory System

The DRAM controllers are described in Section 9. The on-die
interconnect carries both coherence and data traffic.

### 15.4 Interconnect

The on-die interconnect is described in Section 8. Coherence
and data traffic share the same physical links.

PROPOSAL: Virtual channels separate the request, response, and
data traffic. The number of virtual channels per link is 4.

### 15.5 Coherence

Coherence is described in Section 7. The protocol class is
distributed directory MESI with distributed shared L3.

### 15.6 Interrupts

Interrupts are described in Section 11. The PLIC is on the
coherence domain.

### 15.7 Operating System

The operating system interactions are described in Section 10.
The OS is Linux for RISC-V. The firmware is OpenSBI.

### 15.8 Performance

The performance analysis is described in Section 13 and is
detailed in a separate performance document. Key memory-system
performance metrics include:

- L1 hit rate
- L2 hit rate
- L3 hit rate
- DRAM bandwidth utilization
- Average and tail DRAM latency
- Interconnect average and tail latency

## 16. Recommendations

1. Implement a directory-based MESI coherence protocol with a
   distributed shared L3. Snoopy protocols are not viable at
   128 cores.
2. Use a 2D mesh on-die interconnect. The 2D mesh is sufficient
   for the projected traffic and is simpler to verify than a
   torus.
3. Implement Zicbom and Zicboz cache management operations.
   These are required for Linux on RISC-V with mainline
   kernels.
4. Expose each tile as a NUMA node to the operating system. This
   allows the kernel scheduler to make topology-aware decisions.
5. Defer the final decision on the L3 size, the number of DRAM
   channels, and the directory organization to the
   implementation phase, after the sensitivity studies listed
   in Section 12 are completed.
6. Defer the implementation of Zicbop until a workload study
   shows a clear benefit.

## 17. References

The following standards and references are cited in this
document:

- RISC-V Unprivileged ISA Specification, Volume I, Document
  Version 20191213. Section on the Weak Memory Model and the
  Cache Management Operations extensions.
- RISC-V Privileged ISA Specification, Volume II, Document
  Version 20211203. Sections on `satp`, SFENCE.VMA, PMP, and
  interrupt handling.
- RISC-V Profiles, Document Version 20230914. Application and
  platform profile requirements.
- RISC-V IOMMU Specification, draft as of the knowledge cutoff.
- JEDEC DDR5 Standard, JESD79-5.
- JEDEC LPDDR5 Standard, JESD209-5A.
- Hennessy and Patterson, "Computer Architecture: A Quantitative
  Approach", 6th edition. Sections on cache coherence and
  directory protocols.
- Culler, Singh, and Gupta, "Parallel Computer Architecture: A
  Hardware/Software Approach". Chapter on scalable cache
  coherence.
- M. M. K. Martin, M. D. Hill, and D. A. Wood, "Token
  Coherence", University of Wisconsin Computer Sciences
  Technical Report 1621, 2003. Discusses a directory-based
  protocol with explicit invalidation tokens.
- P. Kongetira, K. Aingaran, and K. Olukotun, "Niagara: A
  32-Way Multithreaded SPARC Processor", IEEE Micro, March 2005.
  Discusses a directory-coherent CMP design.
- R. Azimi et al., "Integration Challenges and Tradeoffs for
  Tiled Many-Core Processors", Proceedings of the 22nd
  International Conference on Computer Design, 2004.
- P. Hammarlund et al., "Haswell: The Fourth-Generation Intel
  Core Processor", IEEE Micro, March 2014. Discusses a
  ring-based interconnect in a tiled CMP.
- J. L. Henning, "SPEC CPU2006 Benchmark Descriptions", ACM
  SIGARCH Computer Architecture News, September 2006.
- OpenSBI documentation, https://github.com/riscv-software-src/opensbi.

## 18. Document History

- Draft 0.1: initial outline.
- Draft 0.2: populated physical memory map, coherence protocol
  class, and L1/L2/L3 sizing as proposals.
- Draft 0.3: added open questions and recommendations.