Almost all the chatter surrounding AI infrastructure revolves around GPUs and data-centers. Training an AI model is a monumental task, but the job doesn’t end there. A lot also goes into processing every prompt an AI model receives.

When a large language model (LLM) is running, the GPU has to load billions of model parameters from memory, process them through its compute cores, and then move on to the next chunk, all during a single inference request. Inference, and inference at scale, is what all leading AI companies have to deliver, and doing so has its own challenges.

These challenges are of great import and pressing in nature, so much so that the world finds itself in a bottleneck of sorts, and leading corporations are rushing to secure the resources required to avoid getting stuck in it. This problem is the memory problem, or bottleneck.

Role of Memory in AI

When an AI model is deployed and is at inference, it requires these same billions of parameters to be constantly shifted in and out of memory. To process this sheer volume of information requires significant memory bandwidth (the maximum theoretical volume of data that can be moved across the memory bus per second, measured in Gigabytes per second or Terabytes per second) and capacity.

AI processors (GPUs) today have become so fast that standard Double Data Rate (DDR) and Graphics Double Data Rate (GDDR) memory cannot keep up with them and feed them data quickly enough. Traditional memory architectures have reached their limits, a state known as being memory bandwidth bound, or more commonly, the ‘Memory Wall”.

The memory wall is essentially a performance issue which results in processors often sitting idle as they wait for the data to arrive from memory, leading to wasted energy and reduced computing efficiency. For AI, idle compute cores cost real money, so memory bandwidth is not a nice-to-have feature, but a necessity.

High Bandwidth Memory (HBM)

HBM boasts vastly superior bandwidth speeds, with HBM3 delivering over 819 GB/s and HBM3E reaching over 1.2 TB/s. HBM achieves blazing speeds through a combination of two design features. It has a 3D architecture where, instead of placing memory chips flat on a motherboard, the DRAM (Dynamic Random Access Memory) chips are stacked vertically on top of each other and connected directly to the GPU with microscopic vertical wires called Through-Silicon Vias (TSVs). DRAM is a type of memory that temporarily stores data that needs to be accessed quickly in a computing system.

The second feature is the massive bus width, the number of parallel wires, or lanes, available to move data simultaneously. HBM3 utilizes a 1024-bit interface compared to the 64-bit bus DDR4/DDR5 uses.

HBM solves the memory wall issue to a greater extent because its massive parallel connection to the processor keeps the GPU cores continuously fed. NVIDIA’s Blackwell B300 GPU carries 288 GB of HBM3E memory built using 12-high stacks, delivering 8 TB per second, allowing the GPU to stay productive rather than stall.

Brief History of HBM

High Bandwidth Memory reached the market in June 2015 when AMD launched its Radeon R9 Fury X graphics card, built on the “Fiji” GPU. AMD’s document describes HBM as “pioneered by a consortium led by AMD,” delivering 60% more memory bandwidth than GDDR5 through a 4096-bit interface.

HBM itself is a formal industry standard developed by the JEDEC Solid State Technology Association, a semiconductor industry standards organization headquartered in Virginia, United States. SK hynix says it first developed HBM in 2013, and the company has since led on nearly every subsequent generation of the standard, mass-producing the first 12-layer HBM3E in 2024 and completing the first HBM4 development in 2025.

Broader Industry Trends

That race for HBM leadership has only intensified. HBM is essential to AI, and consequently, leading AI giants are all racing to ensure they have as much of it as they need. The leading manufacturers of HBM, including Samsung, SK Hynix, and Micron, are all making elaborate plans and taking decisive steps to meet rapidly growing demand.

Samsung and SK Hynix’s announcement of investing $518 billion, with support from the government of South Korea, to build four new memory chip fabrication plants drew significant attention across the semiconductor and AI industry. These new fabrication plants will be built for the purpose of manufacturing HBM memory chips.

In a clear move to curb its dependence on NVIDIA and to own a part of the silicon infrastructure, OpenAI unveiled ‘Jalapeño’ last month, its custom-built ASIC (Application Specific Integrated Circuit) designed specifically for running large language models. Designed in a record period of nine months, it features a structure that mounts eight HBM modules.

SK Hynix’s Nasdaq debut happened on July 10, raising $26.5 billion, the largest U.S. share sale ever completed by a foreign company, surpassing the $25 billion Alibaba raised in its 2014 U.S. market entry. Speaking on the day of the listing, SK Hynix CEO Kwak Noh-jung said 2027 will be the worst year in the industry’s history from a supply perspective, with demand expected to outstrip supply even beyond 2030.

Earlier, the company disclosed that it plans to use these net proceeds primarily for new semiconductor facilities in South Korea and advanced manufacturing equipment, including  extreme ultraviolet (EUV) lithography systems.

Also Read: South Korean Chipmaker SK hynix Launches $28B U.S. Listing as Global AI Chip Demand Surges

Conclusion

It is now increasingly clear that memory is no longer background infrastructure but a strategic resource and part of the critical foundation of AI systems.

As models grow larger and inference scales, the race for AI computing power will increasingly be shaped not just by access to GPUs, but by access to the memory required to keep them running. For an industry obsessed with computing power, memory may prove to be the constraint that defines its next phase.

Also Read: Why Taiwan is the Most Critical Node in AI’s Global Supply Chain