DirectGeMM: Eliminating the GeMV Bottleneck in Analog In-Memory Computing
General matrix-matrix multiplication (GeMM) dominates 40-95% of AI workload execution time, but existing in-memory computing (IMC) architectures suffer from a fundamental cycle bottleneck, decomposing GeMM into sequential general matrix-vector multiplications (GeMV) requiring O(M) cycles. In this paper, we propose DirectGeMM, a direct GeMM architecture for analog crossbar arrays that eliminates this bottleneck for the prevalent K ≪ M, N regime through FP8 pre-alignment (shared-exponent extraction for analog-compatible floating-point) and outer-product accumulation (in-situ conductance updates across K phases). Experimental results show that our architecture reduces compute cycles from O(M) to O(K) for AI workloads where K ≪ M, N (K: inner dimension, M: output rows, N: output columns), achieving 58-847× cycle and 12-47× energy reduction across Transformer and CNN models, with 90% classification accuracy on MNIST.
