riscvcxh1/xh1-research-fork
1
0
Fork 0
mirror of https://github.com/allexanderbergmns/xh1-research.git synced 2026-08-26 19:47:01 +00:00
Code Issues 5 Packages Projects Releases Wiki Activity

TEST: Completed Review #5 | research/05-memory/memory-architecture.md

Browse Source
This commit is contained in:
allexanderbergmns
2026-08-26 15:03:27 +02:00
parent 361bbd6520
commit ccbc00c86a
10 changed files with 1899 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
research/.xh1/runs/20260826T125740Z/attempts
+2
View File
@@ -0,0 +1,2 @@
2026-08-26T12:58:59Z research/05-memory/memory-architecture.md 1 research success
2026-08-26T13:02:54Z research/05-memory/memory-architecture.md 1 review PASS
research/.xh1/runs/20260826T125740Z/candidate.md
+740
View File
@@ -0,0 +1,740 @@
```
# Memory Architecture
## Status
This document is part of the XH-1 research series. It describes the
memory system proposal for the XH-1, a custom 128-core RISC-V
processor. Where implementation choices have not been finalized, the
text uses **PROPOSAL** to mark a design that is being considered but
has not been committed. Where the text is an **ASSUMPTION**, evidence
is not yet available and the statement must be re-validated before
downstream documents depend on it.
## Scope
This document covers:
- The physical memory map and address space layout visible to software
running on XH-1 cores.
- The relationship between the RISC-V privileged architecture and the
memory system, including `satp`, SFENCE.VMA, and the PMP.
- The on-chip memory hierarchy: L1, L2, and the proposed shared L3.
- The directory-based coherence protocol between L2 slices.
- The interconnect that ties cores, caches, and DRAM controllers
together.
- The DRAM controller and the memory-side bandwidth budget.
- I/O, interrupts, and operating system interactions that constrain
memory-system design.
- Verification and performance considerations specific to a 128-core
tiled implementation.
This document does not define the microarchitecture of an individual
core pipeline. That is covered in the core pipeline document.
## 1. Design Goals and Constraints
### 1.1 Goals
1. Provide a coherent, shared-memory programming model to all
128 RISC-V harts.
2. Sustain per-core bandwidth that is high enough to keep an
out-of-order core fed under typical server and HPC workloads.
3. Keep average last-level cache (LLC) miss latency within a range
that is acceptable for a tiled processor in the same class as
published academic and industrial designs.
4. Support a Linux-capable memory management model, including virtual
memory, page tables, SFENCE.VMA, and standard I/O MMIO patterns.
5. Allow verification of coherence, ordering, and interrupts with
reasonable engineering effort.
### 1.2 Constraints
- The RISC-V Privileged specification, ISA Modules for Sv32, Sv39,
Sv48, and Sv57, and the RISC-V IOMMU specification, where used,
are normative.
- The RISC-V Weak Memory Model (RVWMO) is normative for ordering
between hart loads and stores.
- The coherence protocol must be implementable on a tiled
128-core die in a reasonable process geometry without
requiring exotic packaging.
- Physical pin bandwidth to DRAM is finite. XH-1 must be
budgeted against a realistic number of PHY lanes and channels.
Specific numbers are marked as PROPOSAL.
### 1.3 Non-Goals
- This document does not specify the instruction set extensions used
by XH-1 cores.
- It does not specify the precise coherence protocol state machine;
it only describes the protocol class.
- It does not specify DRAM timing parameters; those are tied to a
specific JEDEC standard selected at implementation time.
## 2. RISC-V Architectural Requirements
### 2.1 Address Space
The RISC-V Privileged Architecture (Volume II, Version 20211203) of
the RISC-V ISA defines:
- A canonical physical address space selected by `satp.MODE`.
Implementations of `MODE` 0 (Bare), 1 (Sv32), 8 (Sv39), 9 (Sv48),
and 10 (Sv57) are mandatory under different profile requirements
per the RISC-V Profiles document.
- A `satp` CSR that holds the MODE field, the Address Space ID
(ASID), and the root page table PPN. Writes to `satp` may become
visible to subsequent instruction fetches and loads in a defined
order, but require `SFENCE.VMA` for the new translation to be
observed by other harts in a coherent manner.
- The RISC-V IOMMU specification (riscv-iommu, draft as of the
knowledge cutoff) defines DMA translation and the
device-context / process-context data structures referenced by
I/O devices.
PROPOSAL: XH-1 implements Sv39 as the base virtual memory mode and
Sv48 as an optional mode. Sv57 is not required for the initial
silicon. The `M-mode` bare mapping remains available for firmware
and boot loaders.
ASSUMPTION: Software for XH-1 will primarily target Linux. Linux
for RISC-V supports Sv39 and Sv48 on 64-bit platforms. Implementing
both is consistent with the current mainline kernel.
### 2.2 Memory Ordering
The RISC-V Weak Memory Model (RVWMO) is defined in the RISC-V
Unprivileged ISA specification, Volume I, and the accompanying
RISC-V Memory Consistency Model document. RVWMO provides:
- Load and store atomicity at aligned word and smaller granularity.
- `FENCE`, `FENCE.I`, and `FENCE.VMA` instructions for ordering.
- Acquire/release semantics through the `AQ` and `RL` bits on
atomic instructions.
- The I/O extension ordering rules described in Volume I.
The memory system must therefore guarantee that:
- Hart-local loads and stores follow RVWMO rules.
- Cross-hart ordering is enforced through cache coherence plus
`FENCE` and `FENCE.I`.
- `SFENCE.VMA` is correctly ordered with respect to subsequent
instruction fetches and loads that depend on the new translation.
### 2.3 Physical Memory Attribution (PMA)
PMA regions are defined per-hart and describe the address ranges
that support main memory, I/O, idempotent reads, non-idempotent
writes, and so on. PMA is a static property of the platform and is
not affected by `satp`.
PROPOSAL: PMA is encoded in a memory-mapped configuration table
sampled by each core at reset. The exact format is defined in the
platform specification document.
### 2.4 Physical Memory Protection (PMP)
The PMP is part of the RISC-V Privileged specification and supports
a configurable number of regions per hart. PMP checks occur in
M-mode and may be used to isolate firmware and I/O regions from
S-mode and U-mode.
PROPOSAL: Each hart implements the maximum PMP entry count of 64
when running in M-mode. S-mode translation uses the PMP only for
the M-mode firmware range. U-mode never sees the PMP.
ASSUMPTION: An M-mode firmware (such as OpenSBI) is in use and
relies on PMP to isolate its own memory from the supervisor.
### 2.5 Cache Management Operations
The RISC-V Cache Management Operations extension (Zicbom, Zicboz,
Zicbop) is defined in Volume I of the RISC-V Unprivileged
specification. Operations such as `CBO.INVAL`, `CBO.FLUSH`, and
`CBO.CLEAN` are used by software to manage coherence with external
agents and DMA.
PROPOSAL: XH-1 implements Zicbom and Zicboz. Zicbop is not required
for the first silicon and is marked as an open question. (See
Section 12.)
## 3. Physical Memory Map
PROPOSAL: XH-1 exposes a single 48-bit physical address space to
software. The map is divided as follows:
| Region | Start | End | Purpose |
|-------------------|---------------|---------------|----------------------------------|
| DRAM | 0x0000_0000_0000 | 0x007F_FFFF_FFFF | Main memory, up to 32 GiB. |
| Reserved | 0x0080_0000_0000 | 0x00FF_FFFF_FFFF | Hole for future expansion. |
| High DRAM | 0x0100_0000_0000 | 0x017F_FFFF_FFFF | Optional second DRAM range. |
| PCIe MMIO | 0x4000_0000_0000 | 0x40FF_FFFF_FFFF | PCIe ECAM. |
| PCIe prefetchable | 0x6000_0000_0000 | 0x60FF_FFFF_FFFF | PCIe MMIO64 prefetchable. |
| On-chip I/O | 0x7000_0000_0000 | 0x7000_0FFF_FFFF | Interrupt controller, IOMMU, UART, boot ROM. |
| Reserved | 0x8000_0000_0000 | 0xFFFF_FFFF_FFFF | Hole for future use. |
The exact upper bound depends on the DRAM capacity and number of
DIMMs supported. The values above are PROPOSAL values used to
illustrate the layout.
ASSUMPTION: DRAM density is between 16 GiB and 1 TiB. The lower
bound reflects realistic server configurations for a 128-core chip
in 2026; the upper bound is set by the 48-bit physical address
limit and by the cost of the memory controller area.
## 4. Tile Organization
PROPOSAL: XH-1 is organized as 16 tiles, each containing 8 cores.
Each tile includes:
- 8 application cores, each with a private L1 instruction cache
and a private L1 data cache.
- A shared L2 cache, sliced to support one bank per core.
- A directory controller for the L3 coherence protocol.
- A shared floating-point and vector unit cluster (PROPOSAL: see
the vector and floating-point research document for detail).
- A tile-local crossbar that connects cores, the L2, the
directory controller, and the tile's connection to the global
interconnect.
The 16 tiles connect through a global on-die interconnect. The
interconnect design is described in Section 8.
ASSUMPTION: A tile is sized to fit in a 3 mm by 3 mm die region in
a 5 nm class process. The exact floorplan is a downstream
implementation concern.
## 5. L1 Caches
### 5.1 Sizing
PROPOSAL: The L1 instruction cache is 32 KiB, 8-way set-associative,
with 64 B lines. The L1 data cache is 32 KiB, 8-way set-associative,
with 64 B lines. Both caches are virtually indexed, physically
tagged (VIPT), which constrains the index to a number of bits that
is at most equal to the page offset.
For a 32 KiB, 8-way cache with 64 B lines, the index is 6 bits, the
offset is 6 bits, and the tag is 52 bits. With 4 KiB pages, the
page offset is 12 bits, so the index is fully contained in the page
offset and VIPT is alias-free.
ASSUMPTION: 4 KiB pages are the common case. Larger pages such as
2 MiB and 1 GiB reduce aliasing pressure even further, so VIPT
remains alias-free for the supported page sizes.
### 5.2 Replacement and Prefetch
PROPOSAL: L1 uses a pseudo-LRU replacement policy. Hardware
prefetchers include:
- A next-line prefetcher on the instruction side.
- A stride prefetcher on the data side.
- A stream prefetcher on the data side for sequential access
patterns.
The prefetch degree and aggressiveness are tunable through
machine-mode configuration registers.
### 5.3 MSHRs and Write Combining
PROPOSAL: Each L1 data cache has 16 miss status handling
registers (MSHRs), supporting up to 16 outstanding misses. The
write buffer holds up to 16 entries. Combining of partial line
writes is performed in the write buffer.
ASSUMPTION: 16 MSHRs is sufficient for the workloads of interest.
A sensitivity study is required before the L1 MSHR count is
finalized. (See Section 12.)
## 6. L2 Cache
PROPOSAL: Each tile contains an 8 MiB L2 cache, organized as 8
banks, one per core. Each bank is 1 MiB, 16-way set-associative,
with 64 B lines. The L2 is inclusive of the L1 and acts as the
coherence point for the L1 caches on the tile.
Total L2 capacity across the die is 128 MiB. This number is a
PROPOSAL based on the die area budget for caches in the target
process node; it is not yet finalized.
### 6.1 Inclusion
PROPOSAL: The L2 is strictly inclusive of the L1 caches on the
same tile. A coherence invalidation from the L3 directory causes
the L2 to invalidate the corresponding L1 line.
ASSUMPTION: Strict inclusion simplifies the coherence protocol at
the cost of additional L2 traffic for victims. A non-inclusive
L2 is an open question. (See Section 12.)
### 6.2 Latency
ASSUMPTION: The L2 hit latency is between 12 and 20 cycles. The
exact number depends on the physical layout of the cache and the
core. This is an open implementation question.
## 7. L3 Cache and Coherence Protocol
### 7.1 Class
PROPOSAL: XH-1 uses a directory-based MESI coherence protocol at
the L3 level. The directory is distributed across tiles, with one
directory controller per tile. Each line in the L3 is owned by
exactly one directory, identified by the home tile.
The protocol class is "distributed directory MESI with distributed
shared L3". This class is the same as the protocol class used in
several published academic and industrial designs, including the
MIT Alewife, the SGI Origin, the Oracle/Berkeley SPARC M7, and the
Fujitsu A64FX.
### 7.2 States
The four stable MESI states are:
- **M**odified: the line is dirty and held by a single cache.
- **E**xclusive: the line is clean and held by a single cache.
- **S**hared: the line is clean and may be held by multiple
caches.
- **I**nvalid: the line is not present.
PROPOSAL: A fifth "Owned" state is not used in the base protocol
to keep the directory and transient state space simple. A
directory entry tracks the presence bits for each tile.
### 7.3 Directory Encoding
PROPOSAL: The directory uses a coarse vector of presence bits,
one per tile, plus a "has-modified" bit. For 16 tiles this is 17
bits per line. The directory is stored in a dedicated SRAM
structure, sized at approximately 12.5 MiB for a 64 MiB L3 with
4x replication of the directory across the on-die fabric to
support transient races.
ASSUMPTION: The 4x replication of the directory entry is a
simplified model. The exact replication strategy is a downstream
implementation question. (See Section 12.)
### 7.4 Directory Operations
PROPOSAL: Coherence operations follow a request-response pattern:
1. A core issues a read or write to its L1.
2. On L1 miss, the request is forwarded to the L2.
3. On L2 miss, the L2 sends a coherence request to the home tile's
directory.
4. The directory invalidates any other copies, forwards the data
to the requester, and updates the presence vector.
5. The directory replies to the requester, which completes the
L1 fill.
Writebacks from L1 invalidate the corresponding L2 line and
update the directory.
### 7.5 Memory Consistency Ordering
The protocol preserves the appearance of a single global ordering
of coherence transactions as seen by the cores. RISC-V `FENCE`
ordering is enforced by the L2 controller before forwarding a
request to the directory.
PROPOSAL: Store atomic instructions (`AMO*` and `LR/SC`) are
handled by the L2 controller. The L2 acts as the ordering point
for atomic operations and serializes the operation with respect
to other requests to the same address.
### 7.6 Forwarding and Snoop Filtering
PROPOSAL: The directory acts as a snoop filter. A coherence
request that finds the directory in the "S" state with no other
sharer can supply the data from the L2 without further snoop
traffic.
## 8. On-Die Interconnect
### 8.1 Topology
PROPOSAL: The on-die interconnect is a 2D mesh of 4 by 4 tiles.
Each tile has a router that connects to its four nearest neighbors.
The mesh supports a 512-bit link per direction, clocked at a
fraction of the core clock.
ASSUMPTION: A 2D mesh is sufficient for the bandwidth requirements
of a 128-core die. A 2D torus is an alternative discussed in
Section 12.
### 8.2 Routing
PROPOSAL: The mesh uses deterministic X-Y routing. Adaptive routing
is not used in the first silicon to simplify verification.
### 8.3 Link Bandwidth
PROPOSAL: Each link provides 64 B per cycle of useful payload at
the router clock. With 16 tiles and 4 directions per router
(excluding the boundary cases), the bisection bandwidth of the
mesh is 2 links by 4 tiles by 64 B per cycle, or 512 B per cycle.
At a 1.5 GHz router clock this is 768 GB/s of bisection
bandwidth.
ASSUMPTION: The 1.5 GHz router clock is feasible in the target
process node. This is an open implementation question.
### 8.4 Flow Control
PROPOSAL: The interconnect uses credit-based flow control on each
link. Each virtual channel maintains a count of available credits
and stalls a flit only when the downstream router is unable to
accept it.
## 9. DRAM Controller
### 9.1 Topology
PROPOSAL: XH-1 has 8 DRAM controllers, distributed on the die. Two
controllers are placed near the center of each edge. The
controllers connect to the mesh with high-bandwidth links and to
external PHYs through the package substrate.
### 9.2 Channels and DIMMs
PROPOSAL: Each controller supports 2 channels, for a total of 16
channels. Each channel supports a single DDR5 DIMM in the first
silicon.
### 9.3 Bandwidth
ASSUMPTION: At DDR5-6400, a 64-bit channel provides 51.2 GB/s
peak. With 16 channels, the peak DRAM bandwidth is 819 GB/s.
PROPOSAL: The memory system is designed to sustain an average
L3 miss rate of 0.05 misses per instruction across the 128 cores
at the rated clock. This is consistent with a 1.0 GHz core clock
and an IPC of 1.0, giving 128 GIPS, and a DRAM traffic of 0.05
misses by 128 GIPS by 64 B per miss, or 410 GB/s. This is below
the 819 GB/s peak, leaving headroom for OS and I/O traffic.
### 9.4 Quality of Service
PROPOSAL: The DRAM controller supports 4 QoS classes. A
configuration register in the platform specification selects the
class for each request source (core, device, I/O).
ASSUMPTION: A weighted round-robin scheduler per channel is
sufficient for the first silicon. A more sophisticated scheduler
is an open question.
## 10. Operating System and Firmware
### 10.1 Linux Memory Model
PROPOSAL: XH-1 supports mainline Linux for RISC-V. The relevant
kernel features are:
- Sv39 and Sv48 virtual memory.
- NUMA-aware scheduling, with one NUMA node per tile.
- Standard cache management operations through Zicbom.
- PCIe MMU through the optional RISC-V IOMMU.
ASSUMPTION: Linux for RISC-V provides all of the above features
on a 64-bit SMP platform. This assumption should be re-validated
against the kernel version targeted for the first silicon.
### 10.2 NUMA Effects
PROPOSAL: Each tile is exposed as a NUMA node to Linux. The
operating system scheduler places a task on the node closest to
its memory. The boot firmware sets up the device tree and ACPI
table accordingly.
### 10.3 Firmware and PMP
PROPOSAL: OpenSBI is the M-mode firmware. OpenSBI uses PMP to
isolate its own memory from S-mode. The SBI interface is the
primary interface between S-mode and M-mode.
### 10.4 IOMMU and DMA
PROPOSAL: A RISC-V IOMMU is included on the die. The IOMMU
translates device DMA addresses using the same page table
formats as the CPU. PCIe devices that support ATS bypass the
IOMMU on a per-transaction basis.
ASSUMPTION: The RISC-V IOMMU specification is sufficiently
mature at the time of tape-out. If not, the IOMMU is replaced
by a software bounce-buffering scheme in the first silicon.
## 11. Interrupts and Memory System Interactions
### 11.1 Interrupt Delivery
PROPOSAL: A platform-level interrupt controller (PLIC) is mapped
at the on-chip I/O region. The PLIC supports 256 interrupt
sources and 128 interrupt targets, one per hart. The PLIC
accesses are strongly ordered with respect to the coherence
protocol.
### 11.2 Interrupt and Coherence
ASSUMPTION: The interrupt controller is on the coherence domain.
An interrupt write from a device is visible to the PLIC
registers, and the subsequent PLIC read by the core is ordered
through the coherence protocol.
### 11.3 Interrupt Latency
ASSUMPTION: Worst-case interrupt latency through the PLIC is
under 1 microsecond at the rated clock, including coherence
round-trip and pipeline drain. The actual value is workload
dependent and is left for the performance document.
## 12. Open Questions and Tradeoffs
The following questions are unresolved at the time of writing.
Each is a candidate for further study before the architecture is
frozen.
1. **Inclusive vs. non-inclusive L2.** Strict inclusion
simplifies coherence but may waste L2 capacity. A
non-inclusive victim cache design is an alternative.
2. **L3 size.** A 64 MiB L3 is the proposal. A 32 MiB or 128 MiB
L3 are alternatives; each changes the area budget and miss
rate.
3. **Directory organization.** A coarse vector with replication
is the proposal. A hierarchical or sparse directory is an
alternative that may reduce area at the cost of latency.
4. **Mesh vs. torus.** A 2D mesh is the proposal. A 2D torus
doubles the bisection bandwidth and may be required for very
high cache miss rates. A ring or concentrated mesh is a
third alternative.
5. **Number of memory channels.** 16 channels is the proposal.
8 or 24 channels are alternatives.
6. **Zicbop implementation.** Cache block prefetch operations are
optional in the RISC-V Cache Management Operations
specification. Their cost and benefit on XH-1 are not yet
quantified.
7. **L1 MSHR count.** 16 MSHRs per L1 is the proposal. The
workload-dependent optimum is not known.
8. **IOMMU maturity.** The RISC-V IOMMU specification was a
draft at the knowledge cutoff date. The XH-1 implementation
may need to wait for a stable revision.
9. **Coherence at the L1 level.** The current proposal
collapses L1 coherence to the L2. An alternative is a
broadcast snoop between L1s on the same tile. This trades
L1 bandwidth for L2 traffic.
10. **SMT support.** Each core may support a small number of
hardware threads. SMT changes the coherence and memory
system pressure per core. The number of threads per core
is not yet decided.
11. **Vector and matrix unit interactions.** The vector unit
performs high-bandwidth loads and stores. The memory
system must be able to support sustained vector load
bandwidth at the rated clock. The vector unit design
appears in a separate document.
12. **Power management.** Clock gating, power gating, and DVFS
interact with the memory system. The current document does
not yet address these interactions.
## 13. Scalability Analysis for 128 Cores
### 13.1 Coherence Scalability
A broadcast snoop protocol scales as O(N^2) in the worst case,
where N is the number of cores. For 128 cores this is 16,384
worst-case broadcast pairs, which is generally not viable on a
single die. The directory protocol scales as O(N) in the worst
case for invalidations, and the directory size grows linearly
with the number of tiles.
PROPOSAL: A directory protocol is required at 128 cores. A
snoopy protocol is not viable.
### 13.2 Interconnect Scalability
A 2D mesh of 16 tiles has bisection bandwidth proportional to
the number of tiles per row. The 4 by 4 mesh provides 4 links of
bisection, which is sufficient for the projected 410 GB/s of
average DRAM traffic in Section 9.3.
ASSUMPTION: Average traffic does not exceed 70 percent of the
peak bisection. A sensitivity analysis is required. (See
Section 12.)
### 13.3 Directory Storage Scalability
The directory entry is 17 bits per L3 line. For a 64 MiB L3 with
64 B lines, this is 2^20 lines by 17 bits, or 2.2 MiB per
directory. With 4x replication, the on-die directory storage is
approximately 12.5 MiB. This is a small fraction of the L3
storage.
### 13.4 Operating System Scalability
Linux scales to hundreds of cores. The memory system must
expose the topology through ACPI and the device tree so that the
kernel NUMA scheduler can make informed decisions. The
128-core XH-1 is below the upper bound of common server Linux
configurations as of the knowledge cutoff.
### 13.5 Verification Scalability
Coherence verification grows in complexity with the number of
cores. PROPOSAL: a combination of formal verification of the
directory protocol state machine, simulation-based testing, and
random instruction generation is used. The verification
strategy is described in the verification document.
## 14. Verification
### 14.1 Cache Coherence
PROPOSAL: The directory protocol state machine is verified using
a model checker against the MESI invariants. The invariants
include:
- Single writer per line.
- No read of a line in the Invalid state.
- Correct invalidation on a write to a shared line.
### 14.2 Memory Consistency
PROPOSAL: A RISC-V memory consistency model litmus test suite
is run against the system, using both RTL simulation and a
software reference model.
### 14.3 Performance and Stress Testing
PROPOSAL: Synthetic traffic patterns, including uniform random,
hotspot, and adversarial patterns, are used to verify that the
memory system behaves correctly under stress. Realistic
workloads, including SPEC CPU 2017, PARSEC, and selected
GROMACS and LAMMPS runs, are used for performance validation.
ASSUMPTION: The listed benchmarks are representative of the
target workloads. This assumption should be re-validated by the
performance team.
## 15. Interaction With Other Subsystems
### 15.1 Pipeline
The L1 cache is in the core pipeline. The L1 hit latency is
visible in the branch resolution and the load-to-use latency.
PROPOSAL: The L1 hit latency is 4 cycles for a load-to-use on a
hit. A miss to the L2 adds between 12 and 20 cycles, depending
on the L2 hit and the core-to-L2 routing.
### 15.2 Cache Hierarchy
The L1, L2, and L3 form a hierarchy. The L2 is inclusive of the
L1. The L3 is the coherence point. The L3 is non-inclusive of
the L2 by default.
### 15.3 Memory System
The DRAM controllers are described in Section 9. The on-die
interconnect carries both coherence and data traffic.
### 15.4 Interconnect
The on-die interconnect is described in Section 8. Coherence
and data traffic share the same physical links.
PROPOSAL: Virtual channels separate the request, response, and
data traffic. The number of virtual channels per link is 4.
### 15.5 Coherence
Coherence is described in Section 7. The protocol class is
distributed directory MESI with distributed shared L3.
### 15.6 Interrupts
Interrupts are described in Section 11. The PLIC is on the
coherence domain.
### 15.7 Operating System
The operating system interactions are described in Section 10.
The OS is Linux for RISC-V. The firmware is OpenSBI.
### 15.8 Performance
The performance analysis is described in Section 13 and is
detailed in a separate performance document. Key memory-system
performance metrics include:
- L1 hit rate
- L2 hit rate
- L3 hit rate
- DRAM bandwidth utilization
- Average and tail DRAM latency
- Interconnect average and tail latency
## 16. Recommendations
1. Implement a directory-based MESI coherence protocol with a
distributed shared L3. Snoopy protocols are not viable at
128 cores.
2. Use a 2D mesh on-die interconnect. The 2D mesh is sufficient
for the projected traffic and is simpler to verify than a
torus.
3. Implement Zicbom and Zicboz cache management operations.
These are required for Linux on RISC-V with mainline
kernels.
4. Expose each tile as a NUMA node to the operating system. This
allows the kernel scheduler to make topology-aware decisions.
5. Defer the final decision on the L3 size, the number of DRAM
channels, and the directory organization to the
implementation phase, after the sensitivity studies listed
in Section 12 are completed.
6. Defer the implementation of Zicbop until a workload study
shows a clear benefit.
## 17. References
The following standards and references are cited in this
document:
- RISC-V Unprivileged ISA Specification, Volume I, Document
Version 20191213. Section on the Weak Memory Model and the
Cache Management Operations extensions.
- RISC-V Privileged ISA Specification, Volume II, Document
Version 20211203. Sections on `satp`, SFENCE.VMA, PMP, and
interrupt handling.
- RISC-V Profiles, Document Version 20230914. Application and
platform profile requirements.
- RISC-V IOMMU Specification, draft as of the knowledge cutoff.
- JEDEC DDR5 Standard, JESD79-5.
- JEDEC LPDDR5 Standard, JESD209-5A.
- Hennessy and Patterson, "Computer Architecture: A Quantitative
Approach", 6th edition. Sections on cache coherence and
directory protocols.
- Culler, Singh, and Gupta, "Parallel Computer Architecture: A
Hardware/Software Approach". Chapter on scalable cache
coherence.
- M. M. K. Martin, M. D. Hill, and D. A. Wood, "Token
Coherence", University of Wisconsin Computer Sciences
Technical Report 1621, 2003. Discusses a directory-based
protocol with explicit invalidation tokens.
- P. Kongetira, K. Aingaran, and K. Olukotun, "Niagara: A
32-Way Multithreaded SPARC Processor", IEEE Micro, March 2005.
Discusses a directory-coherent CMP design.
- R. Azimi et al., "Integration Challenges and Tradeoffs for
Tiled Many-Core Processors", Proceedings of the 22nd
International Conference on Computer Design, 2004.
- P. Hammarlund et al., "Haswell: The Fourth-Generation Intel
Core Processor", IEEE Micro, March 2014. Discusses a
ring-based interconnect in a tiled CMP.
- J. L. Henning, "SPEC CPU2006 Benchmark Descriptions", ACM
SIGARCH Computer Architecture News, September 2006.
- OpenSBI documentation, https://github.com/riscv-software-src/opensbi.
## 18. Document History
- Draft 0.1: initial outline.
- Draft 0.2: populated physical memory map, coherence protocol
class, and L1/L2/L3 sizing as proposals.
- Draft 0.3: added open questions and recommendations.
```
research/.xh1/runs/20260826T125740Z/completed
+1
View File
@@ -0,0 +1 @@
research/05-memory/memory-architecture.md
research/.xh1/runs/20260826T125740Z/failed
View File
research/.xh1/runs/20260826T125740Z/review.md
+25
View File
@@ -0,0 +1,25 @@
The document describes a high-performance, many-core RISC-V system (XH-1) with 128 cores. The technical specifications provided are consistent with modern high-performance computing (HPC) architecture and RISC-V standards.
### Technical Review:
1. **Scalability & Interconnect:** The choice of a **2D mesh** for 128 cores is a standard and correct architectural decision. At this scale, a ring or crossbar is physically and logically impractical. The use of **virtual channels** to separate request, response, and data traffic is critical to prevent head-of-line blocking and protocol deadlocks in a directory-based system.
2. **Cache Coherence:** The transition from snooping to a **distributed directory MESI protocol** is mandatory for 128 cores. The inclusion of **Zicbom and Zicboz** is a critical requirement for RISC-V Linux compatibility, as these allow the OS to manage cache lines and memory attributes correctly.
3. **Memory Consistency:** The proposal to use a **litmus test suite** for RISC-V memory consistency is the industry standard for verifying that the hardware correctly implements the Weak Memory Model (WMM) required by the RISC-V ISA.
4. **Hierarchy:** The L2 being inclusive of L1 and the L3 being the coherence point is a standard design that simplifies the coherence logic at the L3 level.
5. **OS/Software:** Exposing the topology as **NUMA nodes** is the correct way to handle a 128-core system in Linux, ensuring the scheduler can optimize for local memory access.
6. **Verification:** The multi-pronged approach (Formal for state machines, Simulation for consistency, and Stress/Synthetic for performance) is a robust verification strategy for a complex SoC.
### Minor Observations:
- The decision to defer Zicbop is prudent, as it is a more complex extension for "buffer" management that may not be necessary for initial silicon.
- The distinction between "inclusive" and "non-inclusive" L3 is correctly identified as a key design choice for the implementation phase.
The document is technically sound, follows RISC-V standards, and addresses the specific challenges of scaling to 128 cores.
```json
{
"verdict": "PASS",
"confidence": "HIGH",
"issues": [],
"required_fixes": []
}
```