Linux Huge Pages: Optimizing Performance for Your System

Understanding Huge Pages in Linux

Huge Pages in Linux provide an optimized way to manage large memory allocations by reducing Translation Lookaside Buffer (TLB) misses and improving performance. By default, Linux uses small 4KB pages, but Huge Pages allow applications to use larger 2MB or 1GB pages, significantly improving memory efficiency.

1. Types of Huge Pages in Linux

There are two main types of Huge Pages:

• Explicit Huge Pages (Static Huge Pages) – Manually allocated and managed by system administrators explicitly by vm.nr_hugepages sysctl parameter

• Transparent Huge Pages (THP) – Automatically managed by the kernel without application modifications.

---

2. Explicit Huge Pages

How Explicit Huge Pages Work

Explicit Huge Pages require manual configuration and allocation. Applications must explicitly request Huge Pages, and the system administrator must reserve a predefined number of Huge Pages at boot time or runtime. These pages are stored in a special pool and cannot be used for regular memory allocation.

Benefits

• Reduces page table overhead.

• Improves CPU cache efficiency.

• Useful for databases and memory-intensive applications.

Enabling Explicit Huge Pages

Explicit HugePages DISABLED:

• If the value of HugePages_Total is "0" it means HugePages is disabled on the system.

# grep -i HugePages_Total /proc/meminfo 
HugePages_Total:       0

• Similarly, if the value in /proc/sys/vm/nr_hugepages file or vm.nr_hugepages sysctl parameter is "0" it means HugePages is disabled on the system:

# cat /proc/sys/vm/nr_hugepages 
0
# sysctl vm.nr_hugepages
vm.nr_hugepages = 0

Explicit HugePages ENABLED:

• If the value of HugePages_Total is greater than "0", it means HugePages is enabled on the system:

# grep -i HugePages_Total /proc/meminfo 
HugePages_Total:       1024

• Similarly if the value in /proc/sys/vm/nr_hugepages file or vm.nr_hugepages sysctl parameter is greater than "0", it means HugePages is enabled on the system:

# cat /proc/sys/vm/nr_hugepages 
1024
# sysctl vm.nr_hugepages
vm.nr_hugepages = 1024

Checking Explicit Huge Page Usage

grep -i HugePages_Total /proc/meminfo

If HugePages_Total is greater than 0, explicit Huge Pages are enabled.

Disadvantages of Explicit Huge Pages

• Requires manual allocation and cannot be dynamically resized.

• Needs application modifications.

• Unused Huge Pages remain reserved, leading to potential memory wastage.

  Disadvantages of using the explicit hugepages (libhugetlbfs): Using hugetlbfs requires significant work from both application developers and system administrators; explicit hugepages must be set aside at boot time, and applications must map them explicitly.

  The process is fiddly enough that use of hugetlbfs is restricted to those who really care and who have the time to mess with it. Hugetlbfs is often seen as a feature for large, proprietary database management systems and little else.

---

3. Transparent Huge Pages (THP)

Transparent Huge Pages (THP) allow automatic allocation of Huge Pages without application modifications. This can improve performance by reducing memory management overhead and TLB lookups.

Linux's Transparent Huge Pages (THP) feature automatically merges small pages into huge pages to improve system performance. THP is a memory management feature that's enabled by default in most Linux distributions.

How THP works

• THP monitors memory usage and promotes smaller pages to huge pages when it's beneficial. The Linux kernel dynamically promotes and demotes memory pages between 4KB and 2MB based on system activity.

• THP demotes pages back to smaller pages when it's not beneficial

• THP works with anonymous memory mappings and tmpfs/shmem

Benefits

• Provides automatic Huge Page allocation.

• No application changes required. It provide the benefits of huge pages without requiring changes to application code

• Improves performance by reducing TLB misses.

Why Databases Disable THP

• Databases like MySQL have their own memory management systems, optimized for performance.

• THP can interfere with memory allocation, leading to performance degradation.

• Memory fragmentation issues may cause unexpected latency spikes.

Checking THP Status

cat /sys/kernel/mm/transparent_hugepage/enabled

Output:

[always] madvise never

where,

• [always] → THP is enabled for all applications by default.

• madvise → THP is enabled only for applications that explicitly request it using madvise().

• never → THP is disabled, and only manually allocated Huge Pages will be used.

Disabling Transparent Huge Pages (THP)

Dynamically (Without Reboot)

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

Permanently Using GRUB

Edit GRUB configuration:

vim /etc/default/grub

Add the following:

GRUB_CMDLINE_LINUX="transparent_hugepage=never"

Update GRUB and reboot:

update-grub
reboot

Disabling THP Using systemd

Create a systemd service:

vim /etc/systemd/system/disable-thp.service

Add:

[Unit]
Description=Disable Transparent Huge Pages
DefaultDependencies=no
After=sysinit.target local-fs.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo never | tee /sys/kernel/mm/transparent_hugepage/enabled > /dev/null'

[Install]
WantedBy=basic.target

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl start disable-thp
sudo systemctl enable disable-thp

Why Some Applications Disable THP

• THP can cause performance issues in databases (MySQL, PostgreSQL, Oracle) due to memory fragmentation.

• Applications with their own memory management systems may perform better with explicit Huge Pages.

---

4. Checking Huge Page Usage

To inspect Huge Page memory usage:

# grep AnonHugePages /proc/meminfo for THP
cat /proc/meminfo | grep -i huge

AnonHugePages:    123904 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB

Explanation of Each Field:

• AnonHugePages → Amount of memory used by Transparent Huge Pages (THP).

• ShmemHugePages → Shared memory pages that are backed by Huge Pages.

• ShmemPmdMapped → Shared memory segments mapped with Huge Pages at the PMD (Page Middle Directory) level.

• HugePages_Total → Total preallocated Huge Pages.

• HugePages_Free → Number of free Huge Pages available.

• HugePages_Rsvd → Reserved Huge Pages (allocated but not yet used).

• HugePages_Surp → Extra Huge Pages beyond the preallocated number.

• Hugepagesize → The size of each Huge Page (typically 2048 kB (2MB)).

• Hugetlb → Total memory allocated to Huge Pages.

If Huge Pages are not configured, you will see:

HugePages_Total:       0
HugePages_Free:        0
Hugetlb:               0 kB

---

5. Example: Running a Process with Huge Pages

Users can use the huge page support in Linux kernel by either using the mmap system call or standard SYSV shared memory system calls (shmget, shmat).

To demonstrate a process using Huge Pages, we allocate memory using mmap with Huge Page support in C.

#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>

#define HUGEPAGE_SIZE (2 * 1024 * 1024)  // 2MB Huge Page

int main() {
    int fd = open("/dev/hugepages/myhugepage", O_CREAT | O_RDWR, 0755);
    if (fd < 0) {
        perror("open");
        return 1;
    }

    void *ptr = mmap(NULL, HUGEPAGE_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    if (ptr == MAP_FAILED) {
        perror("mmap");
        close(fd);
        return 1;
    }

    strcpy((char *)ptr, "Hello, Huge Pages!");
    printf("Stored message: %s\n", (char *)ptr);
    
    munmap(ptr, HUGEPAGE_SIZE);
    close(fd);
    return 0;
}

This example maps a Huge Page, writes a message, and then unmaps it.

libhugetblfs is a library in Linux that helps applications use huge pages (large memory pages, typically 2 MB or 1 GB in size) for improved performance. It provides an easy interface to allocate and manage huge pages, reducing overhead and memory fragmentation. It can be used to map files into huge pages, which is especially useful for memory-intensive applications like databases and scientific computing. The library helps optimize memory usage and access speed by leveraging the benefits of huge pages.

---

6. Why Databases Disable THP

• Databases like MySQL have their own memory management systems, optimized for performance.

• THP can interfere with memory allocation, leading to performance degradation.

• Memory fragmentation issues may cause unexpected latency spikes.

7. Conclusion

• Explicit Huge Pages are manually managed and provide better control over memory allocation but require additional configuration.

• Transparent Huge Pages (THP) allow automatic allocation but may cause performance issues in certain workloads.

• Databases and performance-sensitive applications often disable THP and use Explicit Huge Pages instead.

• Proper configuration and testing are essential before applying Huge Pages in production environments.

Further reading:

1. Transparent Huge Pages Refresher by percona

2. How to use, monitor, and disable transparent hugepages in Red Hat Enterprise Linux 6 ,7, 8, 9, and 10 Beta?

3. Low Latency Optimization: Understanding Huge Pages and Part-2 by HRT