From CUDA to MLX: Evolutionary Kernel Optimization for Apple Silicon
K-Search transfers decades of NVIDIA's kernel expertise to Apple chips automatically.
1Hardware Diversity Challenges
With the rise of specialized AI chips, kernel optimization has become critical. While CUDA has 20+ years of optimization knowledge, new platforms like Apple Silicon lack these. K-Search bridges this gap via automated transfer. Traditional porting required rediscovering optimizations from scratch. The framework preserves architectural wisdom across hardware generations.
2Why MLX Matters
Apple's MLX framework has seen rapid adoption since late 2023, enabling local AI inference on 100M+ M-series devices. However, key CUDA optimizations like paged attention are missing in MLX, causing up to 30% performance loss. K-Search's translation layer adapts CUDA optimizations to MLX, closing this gap. Unified memory architecture makes MLX ideal for 7B-70B parameter models.
3K-Search Mechanics
This evolutionary search framework from Berkeley Sky Lab automates kernel optimization. LLMs generate candidates based on hardware specs, tested on real devices. A 'world model' decision tree explores optimization paths, scoring each by 8 metrics. The system prioritizes promising branches (e.g., SIMD width optimizations for Apple chips) until performance plateaus.
4Structured CUDA-to-MLX Translation
The novel translation layer converts CUDA kernels into architecture-native MLX strategies rather than direct copies. For example, it replaces threadgroup memory accesses with register operations suited for Apple's SIMD. This achieved 20x prefill speedup on Mamba SSM kernels versus community mlx-lm implementations. Attention kernels reached 0.97x expert-level performance.
5World Model Architecture
An LLM acting as GPU engineer classifies kernel types and maps data access patterns. Each optimization candidate is scored (0-10) across memory bandwidth, register pressure, etc. The tree structure persists across iterations - successful changes spawn child nodes. After stagnation windows, search explores alternative branches. This mirrors human expert debugging workflows.
6Attention Kernel Breakthrough
Achieved 0.97x speedup on MLX Attention kernels - near expert human level. The translation layer replaced softmax reductions with SIMD shuffle operations, eliminating threadgroup barriers. This single change improved memory bandwidth by 8% and register usage by 4%, scoring 9/10 hardware fit. The approach demonstrates transfer learning's potential in low-level optimization.
7Spec Document's Critical Role
Domain-specific Spec documents prevent invalid code generation by encoding hardware rules (e.g., M2's 32-bit SIMD width) and mathematical constraints. Without this, LLMs hallucinate incompatible primitives. The document also includes optimization patterns like loop tiling strategies for Apple's memory hierarchy. This grounding enables reliable compilation across search iterations.
8Vendor-Agnostic Potential
The method isn't MLX-specific - applicable to any ecosystem where CUDA knowledge transfers. Could optimize for custom AI accelerators or RISC-V chips, reducing hardware lock-in. Published in IEEE Micro 2026, the technique represents a shift toward portable performance. The team plans extensions for PyTorch/TensorFlow backends, democratizing cross-platform optimization.
9Quantifiable Results
20x prefill speedup on Mamba SSM kernels versus mlx-lm, with 45% latency reduction. The translation layer contributed 68% of gains, while evolutionary search provided the rest. Tests used 70B parameter models on Mac Studio M2 Ultra. Real-world impact includes making local LLM inference feasible on consumer Apple devices without cloud dependencies.
10Generative AI Integration
A single model (Gemini 3.5 Pro) handles both reasoning and code generation. First, it analyzes kernels in canonical form, mapping data access patterns. Then it identifies bottlenecks (bandwidth/latency) before proposing optimizations. Each iteration implements one atomic change. This mimics expert debugging while scaling beyond human throughput.
11Industry Implications
Enables performant AI on Apple Silicon without cloud costs, especially for 7B-70B models. MLX's unified memory reduces data transfer overhead. Challenges NVIDIA's CUDA dominance in AI acceleration. Reduces need for scarce kernel engineering expertise. Early adopters include local LLM developers and on-device AI applications seeking vendor diversity.
12Future Directions
K-Search signals a future where AI automates hardware optimization. Adaptable to RISC-V and open architectures, it may revolutionize compiler design within 5 years. Berkeley team plans PyTorch/TensorFlow backend extensions. Broader adoption could democratize cross-platform performance, making specialized optimization knowledge accessible beyond chip vendors.
Original article: Berkeley AI Research Blog — Apple Silicon için CUDA'dan MLX'e Kernel Optimizasyonu