Memory Architecture

❗️

This is a legacy Apache Ignite documentation

The new documentation is hosted here: https://ignite.apache.org/docs/latest/

Ignite Durable Memory is a page-based memory architecture that is split into pages of fixed size. The pages are stored in managed off-heap regions in RAM (outside of Java heap) and are organized in a special hierarchy on disk as explained in the Ignite persistence documentation.

📘

Data Format

Ignite maintains the same binary data representation both in memory and on disk. This removes the need for costly serialization whenever moving data between memory and disk.

The picture below represents the structure of the Ignite durable memory architecture.

• Data Regions
• Operational vs Historical Data
• Memory Segment
• Data Pages
• B+ Trees and Index Pages
• Free Lists

Data Regions

A data region is a logical expandable area. Durable memory can consist of one or more data regions that can vary in size and eviction policies, be persisted on disk, and have other differences as explained in the regions configuration section.

By default, Ignite allocates a single data region occupying up to 20% of the RAM available on a local cluster node.

Operational vs Historical Data

For better performance, you should always try to keep operational data in memory. To achieve this, you will need to configure multiple memory regions.

For example, let us assume that we have Person, Purchases, and Records entities stored in PersonsCache, PurchasesCache and RecordsCache respectively. Here, the Person and Purchases data is operational, i.e. the data we have to access frequently, and the Records data contains historical data that is accessed less often.

Now, let us assume that we only have 200GB of RAM available. In this case, we may wish to share the physical memory as follows:

• 190 GB region will be created for our operational or frequently accessed data such as Persons and Purchases. So both PersonsCache and PurchasesCache will get the maximum performance out of our cluster.
• 10 GB region will be allocated for historical or rarely accessed data sets like RecordsCache, whose data will be mostly located on disk.

Memory Segment

Every data region starts with an initial size and has a maximum size it can grow to. The region expands to its maximum boundary allocating continuous memory segments. By default, the max size of a region is set to 20% of the RAM available on the system.

A memory segment is a continuous byte array of physical memory allocated from the operating system. The array is divided into pages of fixed size. There are several types of pages that can reside in the segment, as shown in the picture below.

See the Memory Configuration documentation on how to configure data regions.

Data Pages

A data page stores entries you put into Apache Ignite caches from an application side (data pages are colored in green in the picture above).

Usually, a single data page holds multiple key-value entries in order to use the memory as efficiently as possible and avoid memory fragmentation. When a new entry is being added to a cache, Ignite will look for an optimal page that can fit the whole key-value entry.

However, if an entry's total size exceeds the page size configured via the DataStorageConfiguration.setPageSize(..) parameter, then the entry will occupy more than one data page.

🚧

If you have many cache entries that do not fit in a single page, then it makes sense to increase the page size configuration parameter.

During an update if an entry size expands beyond the free space available in its data page, then Ignite will search for a new data page that has enough room to take the updated entry and will move the entry there.

📘

Memory/Disk Use and Deleting Data from Caches

If you delete data from Ignite caches/tables, the cache size and memory use will not decrease because the memory is assigned to Ignite by the OS (and is reserved for later reuse). As for disk/physical memory, typically the Write Ahead Log (WAL) and the contents of Ignite database files on disk only grow in size. That disk space is only freed when caches are deleted. Therefore, if you want to release memory to the OS or delete data to save disk space, you should delete the cache.

📘

Memory Defragmentation

More on data pages and overall memory defragmentation is discussed in the memory-defragmentation documentation.

B+ Trees and Index Pages

All SQL indexes defined and used in an application are maintained in a B+ tree data structure. For every unique index declared in a SQL schema, Ignite instantiates and manages a dedicated B+ tree instance.

📘

Hash Index

The cache keys are also stored in B+ trees and are ordered by their hash code values.

As shown in the picture above, the purpose of a B+ tree is to link and order the index pages that are allocated and stored within the durable memory. Internally, an index page contains all the information required to locate the indexed value, entry offset in a data page, and links to other index pages in order to traverse the tree (index pages are colored in purple in the picture above).

B+ tree Meta Page is needed to get to the root of a specific B+ tree and to its layers for efficient execution of range queries. For instance, when myCache.get(keyA) is executed, it will trigger the following execution flow:

1. Ignite looks for a memory region to which myCache belongs.
2. The meta page pointing to the hash index B+ tree of myCache is located.
3. Based on the keyA hash code, the index page the key belongs to is located in the B+ tree.
4. If the corresponding index page is not found in the memory or on disk, then Ignite concludes that the key does not exist and will return null.
5. If the index page exists, then it will contain all the information needed to find the data page of the cache entry keyA refers to.
6. Ignite locates the data page for keyA and returns the value to the user.

Free Lists

The execution flow above explains how a cache entry is looked up in the page memory. Now let us go over how Ignite stores a new cache entry if an operation like myCache.put(keyA, valueA) is called.

In this scenario, the durable memory relies on the free list data structure. A free list is a doubly linked list that stores references to memory pages of approximately equal free space. For example, the image below shows a free list that stores all the data pages that have up to 75% free space, and a list that keeps track of the index pages with 25% free space. Data and index pages are tracked in separate free lists.

Here is the execution flow of the myCache.put(keyA, valueA) operation:

1. Ignite looks for a memory region to which myCache belongs.
2. The meta page pointing to the hash index B+ Tree of myCache is located.
3. Based on the keyA hash code, the index page the key belongs to is located in the B+ tree.
4. If the corresponding index page is not found in the memory or on disk, then a new page will be requested from one of the free lists. Once the index page is provided, it will be added to the B+ tree.
5. If the index page is empty (i.e. does not refer to any data page), then the data page will be provided by one of the free lists, depending on the total cache entry size. A reference to the data page will be added to the index page.
6. The cache entry is added to the data page.