mirror of
https://github.com/allexanderbergmns/xh1-research.git
synced 2026-08-26 17:47:02 +00:00
121 lines
13 KiB
Markdown
121 lines
13 KiB
Markdown
# Atomic Memory Operations
|
|
|
|
## 1. Introduction and RISC-V Architectural Requirements
|
|
|
|
Atomic memory operations are fundamental to multiprocessor synchronization, ensuring that read-modify-write (RMW) sequences execute indivisibly with respect to other cores and I/O devices. In the RISC-V architecture, atomic operations are defined by the **'A' Standard Extension for Atomic Instructions**.
|
|
|
|
The RISC-V 'A' extension provides two primary mechanisms for atomicity:
|
|
1. **Load-Reserved / Store-Conditional (LR/SC)**: A two-instruction sequence where `LR` loads a value and registers a reservation on a memory address, and `SC` stores a value only if the reservation is still valid.
|
|
2. **Atomic Memory Operations (AMOs)**: Single instructions (e.g., `amoadd`, `amoswap`, `amoand`) that atomically load a value, apply a binary operation, and store the result back, returning the original value.
|
|
|
|
Additionally, the 'A' extension introduces the **Acquire (`aq`)** and **Release (`rl`)** bits, which are critical for implementing the **RISC-V Weak Memory Ordering (RVWMO)** memory consistency model. These bits enforce ordering constraints on memory operations without requiring the heavy hardware overhead of a strictly sequentially consistent model.
|
|
|
|
For the XH-1 128-core processor, compliance with the RVA22 (or later) profile is assumed, mandating support for both LR/SC and AMOs, including 32-bit and 64-bit widths (A64), and the `aq`/`rl` ordering bits.
|
|
|
|
## 2. Implementation Approaches and Alternatives
|
|
|
|
Implementing atomics in a many-core processor requires deciding *where* in the memory hierarchy the atomic operation is physically executed.
|
|
|
|
### Approach A: L1 Cache Execution
|
|
The atomic operation is executed in the core's private L1 data cache.
|
|
* **Mechanism**: For an AMO, the L1 cache must acquire the cache line in an Exclusive/Modified state via the coherence protocol, perform the RMW locally, and mark the line dirty. For LR/SC, the L1 cache maintains a local reservation table.
|
|
* **Industry Practice**: Common in lower-core-count processors (e.g., early ARM Cortex-A, some MIPS implementations) where L1 hit rates are high and coherence traffic is manageable.
|
|
|
|
### Approach B: Shared Last-Level Cache (LLC) / L3 Execution
|
|
The atomic operation is forwarded to the shared LLC.
|
|
* **Mechanism**: The core sends an atomic request to the LLC. The LLC bank that holds the cache line performs the RMW. The result is returned to the core. The line remains in the LLC.
|
|
* **Industry Practice**: Standard in high-core-count server processors (e.g., AMD EPYC, Intel Xeon) to prevent cache line bouncing and reduce L1 coherence traffic.
|
|
|
|
### Approach C: Dedicated Atomic Execution Units / Memory Controller
|
|
Atomic operations are handled by a dedicated hardware unit near the memory controller or within the coherence directory.
|
|
* **Mechanism**: Bypasses the cache hierarchy entirely for the execution phase, operating directly on the directory state or DRAM.
|
|
* **Industry Practice**: Rare in general-purpose CPUs; more common in specialized accelerators or GPUs (e.g., NVIDIA's atomic units in L2/DRAM controllers).
|
|
|
|
## 3. Scalability Problems in a 128-Core Architecture
|
|
|
|
Scaling atomic operations to 128 cores introduces severe non-linear bottlenecks.
|
|
|
|
### 3.1 LR/SC Livelock and Contention
|
|
The RISC-V specification explicitly states that hardware is *not* required to guarantee forward progress for LR/SC sequences. In a 128-core system, if $N$ cores are spinning on a single lock, the probability of an `SC` succeeding drops precipitously.
|
|
* **Quantitative Impact**: If 128 cores execute a tight LR/SC loop on the same cache line, every successful `SC` (and many failed ones) triggers a coherence invalidation to all other 127 cores. If the interconnect can sustain $I$ invalidations per cycle, and each core generates an invalidation every $C$ cycles, the interconnect utilization is $U = (128 \times I) / C$. At 128 cores, $U$ easily exceeds 1.0, leading to network saturation, exponential latency increases, and severe livelock.
|
|
|
|
### 3.2 AMO Serialization
|
|
AMOs are inherently serializing. If 128 cores issue AMOs to the same memory region, they must be serialized.
|
|
* **Quantitative Impact**: If AMOs are executed in a single LLC bank, the throughput is limited by the bank's RMW pipeline (typically 1 operation per clock cycle). 128 cores attempting 1 AMO per cycle will result in a queue depth of 127, adding >120 cycles of latency per operation. This creates a massive performance cliff for highly contended data structures (e.g., global spinlocks, reference counters).
|
|
|
|
### 3.3 False Sharing and Reservation Granularity
|
|
If the reservation set for LR/SC is defined at the cache line granularity (typically 64 bytes), unrelated atomic operations to the same cache line will cause spurious `SC` failures. In a 128-core system, the probability of false sharing approaches 1.0 for densely packed data structures.
|
|
|
|
## 4. Microarchitectural Interactions
|
|
|
|
### 4.1 Pipeline
|
|
* **LSU Blocking**: AMOs require a read-modify-write cycle. The Load-Store Unit (LSU) must block the pipeline for the issuing thread until the atomic operation completes.
|
|
* **Reservation State**: The pipeline must maintain the reservation state (address and validity) for LR/SC across context switches and exceptions.
|
|
|
|
### 4.2 Cache Hierarchy and Coherence
|
|
* **Coherence Protocol**: The directory-based coherence protocol must handle atomic requests. If an AMO is executed in the LLC, the directory must transition the line to a state that prevents other cores from reading stale data (e.g., an 'Atomic' or 'Exclusive' state in MOESI/MESIF).
|
|
* **Reservation Tracking**: For LR/SC, the coherence directory must track which cores hold reservations for a given cache line. When an `SC` or a standard store occurs, the directory must invalidate all other reservations for that line.
|
|
|
|
### 4.3 Interconnect
|
|
* **Ordering Guarantees**: The Network-on-Chip (NoC) must guarantee that atomic requests and their responses are strictly ordered. An AMO request must not bypass a preceding store to the same address.
|
|
* **Deadlock Avoidance**: Atomic requests often require virtual channels to prevent deadlock, as they consume buffer space while waiting for the coherence protocol to resolve.
|
|
|
|
### 4.4 Memory System
|
|
* **DRAM Atomics**: If an atomic operation misses all cache levels, it must be executed at the memory controller. The memory controller must support atomic RMW cycles at the DRAM interface, which typically requires locking the DRAM bank or using specialized DRAM commands (e.g., DDR4/DDR5 Read-Modify-Write features, though these are rarely exposed to CPUs).
|
|
|
|
### 4.5 Interrupts and Exceptions
|
|
* **Reservation Invalidation**: The RISC-V specification permits (and industry practice dictates) that taking a trap, executing a context switch, or writing to certain CSRs (like `mstatus`) invalidates the LR reservation. The XH-1 pipeline must flush the reservation register upon interrupt entry.
|
|
|
|
### 4.6 Operating System
|
|
* **Synchronization Primitives**: The OS relies on atomics for futexes, spinlocks, and rwlocks.
|
|
* **Livelock Mitigation**: Because hardware does not guarantee LR/SC forward progress, the XH-1 OS *must* implement exponential backoff or randomized delays in spinlock routines to prevent 128-core livelock.
|
|
|
|
### 4.7 Verification
|
|
* **RVWMO Compliance**: Verifying weak memory ordering with atomics is highly complex. XH-1 requires formal verification using tools like `herd7` or `isla`, and extensive randomized testing using `riscv-dv` with litmus tests to ensure `aq`/`rl` bits correctly constrain memory ordering.
|
|
|
|
### 4.8 Performance
|
|
* **Throughput vs. Latency**: Atomics optimize for correctness, not throughput. High contention will drastically reduce the Instructions Per Cycle (IPC) of the cores involved. Performance monitoring must include hardware performance counters (HPCs) for `SC` failures and AMO queue depths to allow software profiling.
|
|
|
|
## 5. Advantages and Disadvantages of Approaches
|
|
|
|
| Approach | Advantages | Disadvantages |
|
|
| :--- | :--- | :--- |
|
|
| **L1 Execution** | Lowest latency for L1 hits. Simple pipeline integration. | Causes severe cache line bouncing in 128-core systems. High coherence traffic. High area cost for 128 L1 reservation tables. |
|
|
| **LLC Execution** | Eliminates L1 cache line bouncing. Centralizes reservation tracking. Scales better to 128 cores. | Higher latency (NoC round trip). LLC bank serialization bottleneck. |
|
|
| **Memory Controller** | Bypasses cache hierarchy entirely for misses. | Extremely high latency. Complex DRAM interface modifications. |
|
|
|
|
## 6. Unresolved Design Questions
|
|
|
|
1. **Reservation Set Granularity**: Should XH-1 implement sub-cache-line reservation sets (e.g., 8 bytes or 16 bytes) to mitigate false sharing, or stick to 64-byte cache-line reservations to save directory state area?
|
|
2. **AMO Execution Location**: Should AMOs be executed in the L1 (requiring exclusive coherence state) or forwarded to the LLC?
|
|
3. **Hardware Backoff**: Should XH-1 include a hardware-based randomized backoff mechanism for `SC` failures to assist the OS, or strictly rely on software backoff as permitted by the RISC-V spec?
|
|
4. **Vector Atomics**: Will XH-1 support the proposed Vector Atomic extensions, or restrict atomics to scalar (XLEN) widths?
|
|
|
|
## 7. Proposals for XH-1
|
|
|
|
Based on the scalability constraints of a 128-core architecture, the following implementations are proposed for the XH-1 microarchitecture. *Note: These are proposals pending final architectural sign-off.*
|
|
|
|
### Proposal 1: LLC-Executed AMOs and LR/SC
|
|
**Proposal**: Execute all AMOs and manage LR/SC reservations in the shared Last-Level Cache (LLC), not in the private L1 caches.
|
|
**Rationale**: In a 128-core system, L1-executed atomics will cause catastrophic cache line bouncing and coherence network saturation. By executing atomics in the LLC, the cache line remains resident in the shared hierarchy, eliminating unnecessary invalidations to the L1 caches of the 127 non-participating cores. The LLC directory will natively track reservation states.
|
|
|
|
### Proposal 2: Cache-Line Granularity Reservations with Hardware Failure Counters
|
|
**Proposal**: Implement LR/SC reservation sets at the standard 64-byte cache line granularity to minimize directory state overhead. However, implement a dedicated Hardware Performance Counter (HPC) per core to count `SC` failures.
|
|
**Rationale**: Sub-cache-line reservations require complex byte-enable logic in the coherence directory, increasing area and access latency. Cache-line granularity is sufficient if the OS implements proper padding for lock variables. The HPC for `SC` failures is critical for OS developers to tune backoff algorithms in a 128-core environment.
|
|
|
|
### Proposal 3: Distributed LLC Atomic Execution Units
|
|
**Proposal**: Equip each LLC bank with a dedicated, pipelined Atomic Execution Unit capable of processing one AMO or `SC` per cycle.
|
|
**Rationale**: To prevent the LLC banks from becoming a serialized bottleneck, atomic operations must be decoupled from standard load/store pipelines. Distributing atomic units across all LLC banks allows 128 cores to issue atomics to different memory regions in parallel, maximizing aggregate atomic throughput.
|
|
|
|
### Proposal 4: Strict NoC Ordering for Atomic Transactions
|
|
**Proposal**: Utilize dedicated virtual channels (VCs) in the XH-1 Network-on-Chip specifically for atomic requests and coherence invalidations.
|
|
**Rationale**: Atomic operations require strict request-response ordering. Mixing atomic requests with standard cacheable loads/stores in the same VCs can lead to head-of-line blocking and deadlock in a 128-core mesh. Dedicated VCs guarantee forward progress and simplify the verification of RVWMO compliance.
|
|
|
|
## 8. References
|
|
|
|
1. Waterman, A., & Asanović, K. (Eds.). (2019). *The RISC-V Instruction Set Manual, Volume I: Unprivileged ISA*. RISC-V International. (Specifically Chapter 8: "A" Standard Extension for Atomic Instructions, and Chapter 17: "RVWMO Memory Consistency Model").
|
|
2. Asanović, K., & Patterson, D. (2014). *The RISC-V Reader: An Open Architecture Primer*. Strawman Publishing.
|
|
3. Lustig, D., & Martonosi, M. (2019). "Decoupled Vectorized Atomic Memory Operations". *Proceedings of the ACM SIGARCH International Symposium on Computer Architecture (ISCA)*. (Provides context on scaling atomics in many-core systems).
|
|
4. Alglave, J., et al. (2014). "Herding Cats: Modelling, Simulation, Testing, and Data-Mining for Weak Memory". *ACM Transactions on Programming Languages and Systems (TOPLAS)*. (Reference for RVWMO verification methodologies using `herd7`).
|
|
5. RISC-V International. (2021). *RISC-V Architecture Profiles (RVA22)*. RISC-V International. (Defines the mandatory atomic requirements for application processors).
|