Memory Hierarchy

No single memory technology is simultaneously fast, large, and cheap. The memory hierarchy solves this by layering small-fast-expensive memory close to the CPU and large-slow-cheap memory further away. Caches exploit locality of reference to make the hierarchy transparent, creating the illusion of a large, fast memory system.

Objectives

• Describe the memory hierarchy pyramid (registers → cache → RAM → storage)
• Explain temporal and spatial locality of reference
• Understand cache concepts: hits, misses, hit rate, and miss penalty
• Compare direct-mapped, set-associative, and fully-associative caches
• Calculate effective memory access time

Key Takeaways

• Memory hierarchy: speed decreases and capacity increases at each level
• Locality of reference makes caching effective (90-99% hit rates)
• Effective access time ≈ cache hit time (when hit rate is high)
• Cache organization: direct-mapped, set-associative, fully-associative
• The hierarchy creates the illusion of large, fast memory

Applications

• CPU Design: L1/L2/L3 cache hierarchy is critical for processor performance.
• Virtual Memory: OS uses DRAM as cache for disk/SSD storage.
• Database Systems: Buffer pools cache frequently accessed disk pages in RAM.
• Web Caching: CDNs and browser caches follow the same hierarchical principle.

Practice Problems

Problem 1: L1 cache hit rate = 95%, L1 access time = 2 ns, L2 access time = 10 ns. What is the effective access time?

Problem 2: A program loops over a 1000-element array. What type of locality does this exhibit?

Problem 3: A 4-way set-associative cache has 256 sets. Each line is 64 bytes. What is the total cache size?

Problem 4: DRAM latency is 100 ns, SSD latency is 100 μs. How many times faster is DRAM?