Emerging Monolithic 3D (M3D) architectures stack memory and logic with sub-micron vertical vias but lacked unified evaluation tools; LLMET delivers cross-layer architectural simulation connecting device physics to AI application benchmarks.
Traditional 2.5D chip packaging relies on micro-bumps to connect dies, introducing parasitic capacitance, physical spacing limits, and high manufacturing thermal resistance.
Monolithic 3D (M3D) technology overcomes packaging limits by building sequential vertical layers of nanoscale transistors and memory cells with ultra-dense sub-micron inter-layer vias, but chip designers lacked tools to evaluate how M3D device physics affects real-world software performance.
LLMET provides a comprehensive, cross-layer evaluation framework that links lower-level material properties and thermal dissipation models directly to AI workload performance and energy consumption across M3D memory systems.
By providing chip architects with a unified simulation platform, LLMET accelerates the design of ultra-dense, energy-efficient microchips for on-device generative AI, edge robotics, and hyperscale data centers.
LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving
The energy consumption of Large Language Model (LLM) serving is becoming a major system challenge as deployment scales, driven by hardware power and thermal constraints and rising electricity costs. A key contributor to chip energy dissipation is data movement between limited on-chip cache and off-chip High Bandwidth Memory (HBM). Meanwhile, emerging memory technologies such as monolithic 3D (M3D) integration of cache memories at the Back-End-Of-Line (BEOL) of logic chips enable larger and denser on-chip memories, creating new opportunities to reduce costly off-chip traffic. However, it remains unclear whether continuously scaling on-chip memory using emerging technologies can effectively improve the energy efficiency of LLM serving. To address this gap, we develop LLMET (LLM with Emerging Technology), a validated cross-layer simulation framework, and conduct a comprehensive study on the impact of large-capacity on-chip memory technologies across a broad range of models, applications and platforms. Utilizing M3D technology to expand the L2 cache from 40MB to 1GB yields a 44% reduction in chip energy during the Llama3.1-70B prefill phase with a 16K context window, based on LLMET simulation on a dual NVIDIA A100 GPU setup. On the 8x NVIDIA B200-like platform, extending the L2 cache from 128MB to 4GB saves the prefill energy by up to 24%. For the edge platform and workloads, the decode energy saving reaches 30% when increasing the 8MB cache size to 256MB. These results highlight the promise of ultra-large on-chip memories for energy-efficient LLM serving systems.
Telecommunication engineers believed that static noise permanently corrupted electrical signals; Claude Shannon proved that information can be transmitted with zero errors across noisy channels by converting meaning into binary bits. Published in 1948, Shannon’s masterwork founded information theory and created the digital language of the modern world.
Physical sensors in rockets and satellites are constantly corrupted by noise, vibration, and sensor drift; Rudolf Kalman developed an optimal mathematical filter that predicts true position by fusing imperfect sensor readings with equations of motion in real time. Published in 1960, the Kalman Filter guided the Apollo spacecraft to the Moon and is the mathematical brain powering every GPS receiver, autonomous drone, and commercial jetliner on Earth.
Computer scientists believed that mastering the ancient game of Go required human intuition that machines would not achieve for decades; DeepMind’s AlphaGo paired deep neural networks with Monte Carlo tree search to defeat the world champion. Published in 2016, AlphaGo proved that deep reinforcement learning could exhibit genuine creative intuition, sparking the modern Artificial General Intelligence revolution.
For thousands of years, secret communication required sending a physical key via a trusted courier before a message could be sent; Whitfield Diffie and Martin Hellman proved that two total strangers can agree on a shared secret over a compromised public channel. Published in 1976, this mathematical breakthrough invented public-key cryptography and made secure global e-commerce possible.
Multiplying two massive prime numbers takes a fraction of a second on a computer; factoring that product back into its original primes takes millions of years. By anchoring asymmetric encryption to the asymmetry of prime numbers, Ron Rivest, Adi Shamir, and Leonard Adleman created RSA—the cryptographic algorithm that encrypts every browser connection, banking transfer, and software update on Earth.
Albert Einstein spent his final decades insisting that quantum entanglement was an incomplete illusion masking hidden common-sense variables; John Stewart Bell devised a single mathematical inequality that allowed laboratory experiments to test whether Einstein or quantum mechanics was right. Published in an obscure, short-lived journal in 1964, Bell’s Theorem proved that the physical universe is fundamentally non-local—founding modern quantum computing, quantum cryptography, and quantum teleportation.
Physical clocks drift out of sync across distributed computer networks because electrical signals cannot travel faster than the speed of light; Leslie Lamport defined time through cause-and-effect causality rather than ticking seconds. Published in 1978, Lamport’s logical clock paper is the most cited work in distributed systems, establishing the foundational physics of cloud computing.
Storing millions of search queries in computer memory wastes vast amounts of expensive hardware; Bloom filters verify whether a record exists using a tiny fraction of a single bit per item. Published in 1970 to speed up hyphenation in typesetting machines, Burton Bloom’s compact data structure now speeds up every Google search query and Bitcoin transaction on Earth.
Ask this paper your own questions, or keep browsing the verified research catalogue.
