Tuning DDR3+ DRAM Interfaces at Boot
Understanding the critical calibration processes between FPGA controllers and DRAM hardware interfaces.
1DRAM Interface Architecture
A DRAM interface consists of two distinct halves: the command-address bus and the bidirectional data lanes. The command-address path is shared across all chips in a rank, while each byte lane operates independently. This separation requires precise, per-lane tuning to ensure data integrity across the physical board topology during high-speed operations.
2Source-Synchronous Data Transfer
DDR memories transfer two words per clock cycle, making controller-sent clocks unreliable for sampling due to board delays. Data travels with a source-synchronous strobe, DQS, which acts as the sampling clock. Because both signals traverse the same path, their relative delays cancel out, allowing the receiver to accurately capture incoming data bursts.
3Fly-By Topology Challenges
DDR3 and DDR4 utilize fly-by routing for clocks and addresses to maintain signal integrity at high rates. This topology causes each chip to receive the clock slightly later than the previous one. Consequently, the skew between the first and last chip can exceed a full clock period, necessitating individual lane delays.
4Write Leveling Mechanism
Write leveling aligns the DQS strobe with the DRAM clock at each chip. The controller enters a specific mode, sending DQS pulses while the DRAM samples its clock. By adjusting DQS step-by-step until the sampled value transitions, the controller ensures that DQS is perfectly aligned with the clock at every individual chip.
5Write Latency Calibration
While write leveling aligns signals within one clock period, a lane might still be shifted by whole clock cycles. Write latency calibration involves writing known patterns and shifting lanes by whole clocks until the data is correctly aligned. This process ensures the burst arrives at the DRAM at the expected timing.
6DQ-DQS Training Process
DQ-DQS training centers the data (DQ) against the strobe (DQS) during write operations. The controller sweeps the output delay of DQ pins across a range, writing and reading back patterns to identify the optimal window. This relies on a previously established, functional read side to ensure the timing margins are robust.
7Read Leveling and Bitslips
On reads, the controller must determine when data arrives after the CAS latency. This requires two settings: a coarse bitslip to align the deserializer output to the correct clock cycle, and a fine input delay to sample the bit accurately. A wide data eye indicates a robust interface, while narrow windows signal issues.
8On-Die Termination Strategies
To prevent signal reflections, DDR3 and DDR4 use on-die termination (ODT). Chips switch internal resistors based on whether they are idle or being written to. LiteDRAM manages these settings via mode registers, sometimes employing dynamic ODT to disable terminations during reads, ensuring the signal remains clean and properly driven at high speeds.
9ZQ Calibration for Drift
Driver and termination resistance varies significantly with temperature, voltage, and manufacturing process. Chips calibrate these values against a precision 240-ohm resistor on the ZQ pin. ZQCL runs once at initialization, while ZQCS periodically corrects for drift during operation, ensuring consistent signal quality despite changing environmental conditions within the system hardware.
10Reference Voltage Management
Receivers compare inputs to a reference voltage, VREF, set between high and low levels. DDR3 uses dedicated board pins for VREF, whereas DDR4 generates it internally. FPGA-side receivers may also require VREF tuning, which is often managed via specific registers that the BIOS must scan to ensure reliable data sampling.
11Frequency Ratios and DFI
FPGA fabrics cannot match DRAM clock speeds, so controllers operate at a fraction of the DRAM clock, such as 1:4. The DFI interface bridges the controller and PHY, carrying command slots and data phases. This architecture allows the controller to handle multiple data words per cycle despite the slower internal clock.
12System Initialization Conclusion
Successful DRAM operation requires a complex sequence of calibration steps, including write leveling, read training, and impedance matching. These procedures, managed by the BIOS and LiteDRAM, compensate for physical board delays and signal integrity challenges. Proper tuning at boot is essential for stable, high-performance memory access in modern FPGA-based computing systems.
Original article: Hacker News: Front Page — Tuning DDR3+ DRAM Interfaces at Boot