# Atomics

## 1. Overview

Atomic operations are fundamental to synchronization, lockless data structures, and concurrent memory management in multi-core systems. In the context of the XH-1, a custom 128-core RISC-V processor, atomics present significant microarchitectural challenges. While a 4-core or 8-core system can tolerate cache-line ping-ponging and moderate interconnect traffic, a 128-core system amplifies contention, false sharing, and network saturation by orders of magnitude. 

This document details the architectural requirements, implementation strategies, scalability bottlenecks, and cross-subsystem interactions for atomic operations in the XH-1 processor.

## 2. RISC-V Architectural Requirements

The XH-1 must comply with the RISC-V Atomic specification. Historically defined as the standard 'A' extension, the RISC-V ISA has recently modularized these features into two distinct extensions for finer granularity:

1.  **Zaamo (Atomic Memory Operations)**: Defines the AMO (Atomic Memory Operation) instructions (e.g., `amoadd`, `amoswap`, `amoand`). These are read-modify-write operations that execute atomically with respect to other memory operations to the same address.
2.  **Zalrsc (Load-Reserved/Store-Conditional)**: Defines the `lr` (Load-Reserved) and `sc` (Store-Conditional) instructions, which together provide a mechanism for atomic read-modify-write sequences.

### Memory Ordering and RVWMO
Atomics in RISC-V operate within the **RVWMO** (RISC-V Weak Memory Ordering) memory model. 
*   **Acquire and Release Semantics**: AMO and `sc` instructions support `.aq` (acquire) and `.rl` (release) bits. These enforce Preserved Program Order (PPO) rules, ensuring that subsequent memory operations do not reorder before an acquire, and preceding operations do not reorder after a release.
*   **Sequential Consistency**: An AMO or `sc` instruction with both `.aq` and `.rl` set provides sequentially consistent ordering for that specific memory location.

### Reservation Semantics (Zalrsc)
The RISC-V specification mandates strict rules for `lr`/`sc` reservations:
*   A successful `lr` establishes a reservation set (typically a cache line).
*   An `sc` will fail (return non-zero) if any other hart successfully stores to the reservation set between the `lr` and `sc`.
*   **Mandatory Invalidation**: The architecture *requires* that a reservation be invalidated upon a context switch, or if the hart takes an interrupt or exception. 
*   **Spurious Failures**: The architecture *permits* `sc` to fail spuriously, even if no other hart wrote to the address, though excessive spurious failures degrade performance.

## 3. Implementation Approaches and Alternatives

Implementing atomics efficiently requires deciding *where* the atomic operation is physically executed within the memory hierarchy.

### Approach A: Core-Side Execution (L1 Cache)
The AMO or `sc` is executed in the core's execution pipeline, interacting directly with the L1 data cache.
*   **Mechanism**: The core requests exclusive ownership of the cache line. The ALU performs the read-modify-write locally. The modified line is written back.
*   **Advantages**: Low latency for uncontended cases; simple pipeline integration.
*   **Disadvantages**: Generates massive invalidation traffic. In a 128-core system, if multiple cores attempt an AMO on the same address, the cache line will ping-pong across the L1 caches, saturating the interconnect.

### Approach B: Mid-Level Cache Execution (L2/L3)
The AMO is forwarded to a shared L2 or private L3 slice.
*   **Mechanism**: The L1 forwards the AMO request to the L3. The L3 controller performs the read-modify-write and returns the result.
*   **Advantages**: Reduces L1 invalidation traffic; keeps the cache line resident in the shared cache.
*   **Disadvantages**: Higher latency than L1 execution; requires complex state machines in the L3 controller to handle partial writes and data type conversions.

### Approach C: Directory/Home Node Execution
The AMO is routed to the directory controller or the "home node" responsible for the physical address.
*   **Mechanism**: The interconnect routes the AMO directly to the memory controller or directory node. The operation is performed in the directory's scratchpad or the main memory interface.
*   **Advantages**: Eliminates cache-line bouncing entirely. Highly scalable for 128 cores.
*   **Disadvantages**: Highest latency for uncontended accesses; requires the directory protocol to natively support AMO payloads.

## 4. Scalability Challenges in a 128-Core Architecture

Scaling atomics from a few cores to 128 cores introduces severe non-linear performance degradation if not carefully managed.

### 4.1. Contention and Interconnect Saturation
Consider a highly contended 64-byte cache line (e.g., a global spinlock or a shared counter). 
*   **Quantitative Impact**: In an 8x16 mesh interconnect, the maximum hop count is 22. Assuming 1 ns per hop, the network round-trip time (RTT) is ~44 ns. If AMOs are executed at the L1 (Approach A), every AMO requires acquiring exclusive ownership, invalidating the line in up to 127 other L1 caches. The invalidation/acknowledgment traffic for a single AMO could take >100 ns. 
*   **Throughput Collapse**: Serialized execution at the home node limits throughput to $1 / (44\text{ns} + \text{memory latency}) \approx 10\text{M}$ ops/sec. If L1 ping-ponging occurs, effective throughput could drop below $2\text{M}$ ops/sec, leaving 126 cores idle.

### 4.2. LR/SC Livelock
In a 128-core system, the probability of `sc` failure increases drastically due to high contention and interrupt rates.
*   If 128 harts attempt a compare-and-swap (CAS) loop using `lr`/`sc` on the same address, the probability of a successful `sc` for any given hart approaches $1/128$. 
*   Furthermore, if the OS uses timer interrupts frequently, the mandatory reservation clearing on interrupt entry will cause `sc` to fail even in the absence of memory contention, leading to livelock.

### 4.3. False Sharing
With 128 cores, the likelihood of independent variables sharing a 64-byte cache line is high. An atomic update to one variable will invalidate the cache line for all other variables in the same block, causing unnecessary coherence traffic and stalling unrelated cores.

## 5. Architectural Interactions

### 5.1. Pipeline
*   **Execution Unit**: AMOs require a dedicated read-modify-write execution unit or multiplexing of the existing ALU. 
*   **Stalls**: An `sc` failure must be handled without architectural exception. The pipeline must squash the `sc` and allow the core to retry. AMOs with `.aq`/`.rl` bits may block subsequent memory operations, requiring the pipeline to track memory ordering buffers (MOBs) or store queues carefully.

### 5.2. Cache Hierarchy and Coherence
*   **State Transitions**: An AMO requires the cache line to transition to the Modified (M) or Exclusive (E) state in the MESI/MOESI protocol. 
*   **Directory Protocol**: The coherence directory must process AMO requests. If an AMO arrives for a line in the Shared (S) state, the directory must issue invalidations to all sharing cores before granting the AMO, or perform the AMO locally if the protocol supports "Shared-Modify" transitions.

### 5.3. Memory System and Interconnect
*   **Payload Size**: The interconnect must support the payload size of AMOs (up to 64 bits for `amoadd.d`, or 128 bits if the Zicbom/Zve extensions are considered, though standard Zaamo is up to 64-bit). 
*   **Ordering**: The interconnect must preserve the ordering of AMO requests to the same address to prevent race conditions at the home node.

### 5.4. Interrupts and Exceptions
*   **Reservation Clearing**: The hart's local reservation register must be cleared synchronously upon taking an interrupt, exception, or executing an `xret` (return from trap). 
*   **Implementation**: This requires a hardware signal from the trap handler to the load/store unit to invalidate the `lr` state.

### 5.5. Operating System
*   **Futexes and Spinlocks**: The Linux kernel relies heavily on `lr`/`sc` for `cmpxchg` and futex operations. The OS expects `sc` to fail only under contention. Excessive spurious failures will cause the OS scheduler to consume excessive CPU cycles in retry loops.
*   **Context Switching**: The OS does not need to save/restore the `lr` reservation state across context switches, as the architecture mandates its invalidation.

### 5.6. Verification
*   **RVWMO Compliance**: Verifying RVWMO with atomics is notoriously difficult. The XH-1 verification environment must utilize formal verification tools (e.g., RISC-V Formal Verification SIG tools) and execute extensive litmus tests (e.g., `mp`, `iriw`, `sb` with atomic variants) to ensure `.aq` and `.rl` bits correctly constrain memory reordering.

### 5.7. Performance
*   **IPC Impact**: Uncontended atomics should ideally execute in 1-2 cycles in the pipeline. Contended atomics will stall the pipeline. The performance monitor (PMU) must include events to track `sc` failures, AMO latency, and cache-line bouncing to allow software profiling.

## 6. Unresolved Design Questions

1.  **AMO Execution Location**: Should XH-1 execute AMOs in the L1 cache (lower latency, high traffic) or at the L3/Home node (higher latency, high scalability)? *See Section 7 for proposal.*
2.  **LR/SC Livelock Mitigation**: Should the microarchitecture implement hardware-level backoff or priority mechanisms for `sc` retries, or should this be left entirely to software (compiler/OS)?
3.  **Reservation Set Granularity**: The RISC-V spec allows the reservation set to be larger than the requested address. Should XH-1 strictly limit the reservation set to the exact 64-byte cache line, or allow it to cover a larger physical page to simplify hardware?
4.  **Zaamo/Zalrsc Modularity**: Will XH-1 implement both Zaamo and Zalrsc, or only one? (Linux requires both for standard operation).

## 7. Proposals and Recommendations

Based on the scalability requirements of a 128-core architecture, the following proposals are submitted for the XH-1 design:

### Proposal 1: Home-Node AMO Execution
**XH-1 Proposal**: Execute all Zaamo (AMO) instructions at the directory home node (or L3 slice acting as the home node) rather than in the L1 cache.
*   **Rationale**: In a 128-core system, L1-based AMOs will cause catastrophic invalidation storms. By routing AMOs to the home node, the cache line remains in the shared L3, and only the directory state is updated. This trades a small amount of uncontended latency (approx. 20-30 ns) for massive gains in contended throughput and interconnect bandwidth preservation.

### Proposal 2: Strict Cache-Line Reservation Tracking
**XH-1 Proposal**: Implement `lr`/`sc` reservations strictly at the 64-byte cache line granularity.
*   **Rationale**: While the RISC-V spec permits larger reservation sets, a larger set (e.g., page-level) would cause `sc` to fail spuriously when unrelated data in the same page is modified by other cores. In a 128-core system, page-level reservations would render `lr`/`sc` practically unusable due to constant spurious failures.

### Proposal 3: Hardware-Assisted LR/SC Backoff (Under Investigation)
**XH-1 Proposal**: Investigate the addition of a microarchitectural, hardware-assisted exponential backoff for `sc` instructions.
*   **Rationale**: To prevent livelock and interconnect saturation when multiple harts spin on the same `lr`/`sc` loop, the load/store unit could introduce a small, randomized stall before allowing the `sc` to attempt the store. *Note: This must be carefully designed to ensure it does not violate the architectural visibility of memory operations or introduce deadlocks.*

### Recommendation: PMU Extensions for Atomics
It is strongly recommended to include specific Performance Monitoring Unit (PMU) counters for:
*   `sc` instruction failures (distinguishing between contention-induced and spurious/interrupt-induced if possible).
*   AMO execution latency.
*   Cache line invalidations triggered specifically by atomic operations.
These counters are critical for OS developers and application programmers to optimize lockless algorithms on a 128-core system.
