Memory Hierarchy In Programming Explained

Key Points

• Memory hierarchy is a fundamental concept in programming that improves performance by reorganizing the way components respond to tasks.
• Memory units store information important for immediate use in a computer and optimal software performance.
• Memory hierarchy assigns tasks to different components based on the principle of locality, which refers to accessing the same type of memory location repeatedly.
• Memory devices are divided into two groups: internal memory (primary memory) and external memory (secondary memory).
• The top of the memory hierarchy includes CPU registers and cache memory, while external memory units like hard disk drives are lower in the hierarchy.

Memory hierarchy is a fundamental concept in programming. We are seeing advances and discoveries leading to more efficient and peak-performance software every day, and much of it is thanks to memory hierarchy.

The apparition of computers truly changed the game. Computers, other than fire or the wheel, are probably the invention that has single-handedly helped mold our current world.

There are some basic computer concepts that developers keep working on since day one. Everything we programmers study is built upon them. Memory is one of those concepts, a fundamental pillar of the body that forms a modern computer.

Today we will learn about what memory is, and more specifically, what having a memory hierarchy means for a computer. Without further delay, let’s get started!

What Is Memory?

Computers may appear complex to unpack for the untrained user, especially with the great variety of available components. But they really are a combination of basic elements, that depending on their own characteristics can lead to an infinite number of computer designs, each one special on its own.

Along with motherboards, CPUs, GPUs, and storage units, we have memory units. They, in basic terms, store information that is important for immediate use in the computer and for optimal software performance.

Memory works at a very high speed, compared to other storage units (like hard disks), but retains less information. Essentially, memory stores whatever the computer needs to be stored at the moment to execute a certain program.

What Is Memory Hierarchy in Programming?

In computer science, a superb way to improve performance is by reorganizing the way that components respond to the different tasks we need to execute.

Having said that, memory hierarchy in programming refers to the organization of different levels of computer memory. These range from fast and expensive caches to slower and larger main memory and storage, to optimize data access and performance. 

An optimized flow of data through storage units leads to better access time between components. Essentially, this means that by knowing what things it should remember first, our computers can remember those things faster. Depending on many considerations, the hierarchy assigns each component one or more tasks.

Memory hierarchy directly affects how a computer works, and also the power of algorithm predictions and other basic-level programs that every computer nowadays runs. It is based upon the principle of locality (also known as locality of reference). This principle refers to the tendency of computer systems to access the same type of memory location over a short period, for a task that is executed repeatedly.

Memory Devices

Memory hierarchy establishes an order that has achieved an exceptionally high optimization level ever since its first implementation.

The design of the hierarchy divides storage or memory units into two groups. Each group contains a variety of devices that get assigned tasks depending on their storage quality.

The first group, called Internal Memory or Primary Memory, are devices that directly communicate with the CPU. This is crucial, as they are at the vanguard of our system. If they fail, most stuff fails. The main memory, cache memory, and CPU storage are primary memory devices. 

The second group is called External Memory or Secondary Memory. It is composed of any kind of device that communicates with the processor using an I/O Module. Pretty much every hard drive ever falls into this category. Magnetic disks, optical disks, and magnetic tape are examples of external memory devices.

Hierarchy Levels and Design

At the top of the memory hierarchy, we find the CPU registers. These are small sets of data holding places that are part of a computer’s processor.

Because they are so small, registers can only store very high-priority information. Here, we are talking about 16 to 64 bits of space. 

While this might sound like a ridiculously small amount of space, you have to keep in mind that these registers store vital information. Imagine you couldn’t remember your name or address and had to keep an ID card with you at all times: this is basically what’s happening here.

Next, we have the well-known cache memory. The cache also stores frequently used information that connects to the type of tasks that the users execute. We won’t get too much into those, but you should know that a healthy cache is essential for most modern operating systems.

Cache memory is less powerful than CPU Registers but can store a larger amount of data. Usernames, passwords, and other essential values are often held in cache memory as a sort of safe. 

The secondary storage group refers to external disks, like hard disk drives (or HDDs). These are slower but cheaper and with more storage space. It also includes solid-state drives, or SSDs, which are faster, but expensive and with typically smaller storage capacity.

HDD and SSD units can store a large amount of data that is typically not always in active use by the computer but can be accessed at any time. Things like games, pictures, and text documents get stored here.

Interestingly, they don’t even need electric power to retain the information. You can unplug them and nothing will happen to your data. However, they lack the high-speed data transition that other types of memory have.

Random Access Memory (RAM)

If you know anything about computers, you’re probably wondering why we haven’t mentioned ram yet. Well, here it is! Main memory, better known as Random Access Memory (RAM), is the most powerful memory device of the group.

A computer equipped with a great RAM system will perform better in most tasks required by users, like gaming, design, and programming. The reason for this is complicated, so make sure to pay attention!

We have two types of main memory devices: Static, and Dynamic.

Dynamic RAM is the one that we as users can easily expand and upgrade to improve our computers’ performance. It works by storing binary data on a large number of capacitors that need constant power to keep the information alive. It’s slower but can contain more information than static RAM, and it’s cheaper to manufacture. 

Static RAM doesn’t work like dynamic RAM but it still needs constant power to work. It often works in conjunction with cache memory. Moreover, static RAM has a smaller capacity and needs fewer resources to function. It is also much quicker and generally closer to the CPU. This means that it often serves as the cache memory for our CPU, making it the spinal cord of our PC.

Memory Hierarchy’s Main Attributes

So far we’ve learned about the wide variety of devices that can act as memory units in our computer. We also know what memory hierarchy entails. But we haven’t talked about how we can know which memory device is best to store which kind of data. 

That’s where these attributes come into play. We use a series of attributes that help us discern which device is the best option to execute any required tasks.

Capacity is the easiest way to know how much information memory can store. As we saw before, the storage capacity generally correlates with how quickly that information can be stored, deleted, and replaced. 

Typically, the larger the capacity a drive has, the slower its speed will be. Additionally, high-storage and slower devices are cheaper to manufacture. 

Similarly, the access time tends to be faster in devices with smaller storage space. We define it as the time that it takes for the memory and CPU to communicate with each other. This communication is the process by which our CPU writes and rewrites the information stored in all our memory devices.

We can calculate the overall performance of our memory units by using these attributes, and then define how efficient the design of our memory hierarchy is. 

In short, smaller and faster memory cells will be higher on the hierarchy as they will be closer to the CPU in our PC’s overall scheme. Naturally, the bigger and slower storage devices will be lower and be accessed less often.

Wrapping Up

So far we have covered what memory is, and what memory hierarchy means. We also have learned about the different kinds of storage devices that make up the framework of any modern computer system.

CPU registers and Cache memory are at the top of the memory hierarchy. This is because they need to be accessed constantly for the computer to work. As such, because of the attributes that we discussed, they are the smallest and quickest of all memory devices.

In contrast, hard disk drives and other external memory units are the slowest and most expedient of memory units. These devices are lower in the memory hierarchy, and their data is less frequently accessed.

The image featured at the top of this post is ©nikkytok/Shutterstock.com.