AI inference is running into a memory bottleneck. Huawei wants to tackle it with OceanStor M900
Processing increasingly large amounts of context places significant pressure on memory capacity, while keeping accelerators supplied with data becomes just as important as raw computing power. One particular challenge is the key-value (KV) cache, which stores information generated during inference…