Performance analysis is a critical part of software development and system optimization. It helps developers understand where CPU time is spent, how efficiently cache memory is used, and how code changes influence performance. Without such insight, optimization becomes guesswork.

Linux provides a rich ecosystem of performance analysis tools. Among these, Cachegrind, Callgrind, and perf stat stand out as powerful utilities that enable developers to understand their program’s behaviour from both simulated and real-hardware perspectives.

Before diving into these tools, let’s briefly discuss CPU caching.

Modern CPUs use multi-level caches (L1, L2, L3) to bridge the speed gap between the processor and main memory. Efficient cache usage can dramatically improve performance, while cache misses lead to costly main-memory accesses.

Two core principles govern cache efficiency:

• Temporal Locality: If a program accesses a memory location, it is likely to access the same location again soon.
• Spatial Locality: If a program accesses a memory location, it is likely to access nearby locations soon.

Optimizing code to exploit these properties — for example, by accessing arrays sequentially, can make a dramatic difference. Profiling tools such as Cachegrind and perf stat allow us to observe and quantify these effects.

Matrix Multiplication Example

To demonstrate these tools, we will analyze three versions of a matrix multiplication program that multiplies two 512×512 floating-point matrices. The example set are part of Linux Debug Training course.

Example link: https://github.com/SpecialistLinuxTraining/linux-debug-training/tree/main/Examples/matrix_multiplication

1. matrix_traditional — Implements the classic triple nested loop algorithm:

for (i = 0; i < SIZE; i++)
 for (j = 0; j < SIZE; j++)
 for (k = 0; k < SIZE; k++)
 C[i][j] += A[i][k] * B[k][j];

This version is simple but cache-inefficient, as it repeatedly accesses non-contiguous elements of matrix B.

1. matrix_avx2_basic — Uses AVX2 SIMD instructions to process eight floating-point values simultaneously. This reduces loop iterations and improves throughput by vectorizing operations.

2. matrix_avx2_optimized — Builds upon the AVX2 version with optimization techniques like blocking (tiling) and loop unrolling:

   • Blocking enhances cache locality by dividing matrices into smaller blocks that fit into L1/L2 cache.
   • Loop unrolling reduces loop overhead and increases instruction-level parallelism.

We’ll use these three programs to compare performance using Cachegrind and perf stat.

Results with Cachegrind: Simulated Cache Behavior

Cachegrind simulates how a program interacts with the CPU’s cache hierarchy. It reports metrics such as instruction counts, cache hits/misses, and memory access patterns — helping identify inefficient memory access.

$ valgrind --tool=cachegrind ./program

• The detailed profiling data is written to a file named cachegrind.out.{pid}
• cg_annotate is a command line tool which summarizes cache-related information per function and annotates source code with color-coded markers for cache events.
• kcachegrind can provide the cachegrind report in GUI form.

Running Cachegrind on our matrix multiplication programs gave:

Results from Cachegrind

Key observations:

• The traditional version suffered from poor spatial locality, as elements of matrix B were fetched from distant memory addresses repeatedly.
• The AVX2 basic version improved performance by vectorizing loads, enhancing both temporal and spatial locality.
• The AVX2 optimized version, with blocking, ensured that working sets fit within cache boundaries, minimizing evictions and L1 data misses.

Cachegrind provides reproducible, deterministic results since it simulates an idealized cache model rather than depending on hardware variability.

Results with perf stat: Real hardware events

While Cachegrind and Callgrind rely on simulation, perf stat measures real hardware events via CPU performance counters. It reports true execution behavior — including speculation, out-of-order execution, branch prediction, and actual cache misses.

We collected the following events:

perf stat -e instructions,cycles,cache-misses,\
L1-dcache-load-misses,L1-dcache-loads,\
L1-dcache-stores,LLC-load-misses,LLC-loads,\
branch-misses,fp_arith_inst_retired.256b_packed_single\
 ./matrix_variant

matrix_traditional:

perf stats results for matrix_traditional

matrix_avx2_basics:

perf stat results with matrix_avx2_basics

matrix_avx2_optimized:

perf stats results for matrix_avx2_optimized

In summary:

• Cachegrind is ideal for micro-analysis of algorithms and memory access behavior.
• perf stat is indispensable for measuring true runtime performance and hardware utilization.

Conclusion

Performance analysis in Linux is a systematic process of measuring, comparing, and interpreting how programs execute at both the instruction and cache levels.

Tools like Callgrind and Cachegrind help you understand how efficiently your code executes in theory, while perf stat helps you measure how efficiently it runs in practice.

Through our matrix multiplication case study, we demonstrated how:

• AVX2 vectorization reduces instruction count by ~7×.
• Blocking and loop unrolling improve cache utilization, reducing L1 cache misses by ~10×.
• perf stat confirms real performance gains, showing higher IPC and lower CPU cycles.

In essence, combining Valgrind-based analysis with perf-based hardware profiling provides a complete picture of software performance — bridging the gap between code behavior and hardware efficiency.

LINUX DEBUG TRAINING

If you want to dive deeper, then you can also check out Part One and Part Two of the Linux Debugging Training Course. This course is a collaboration between my colleague John OSullivan and myself. We have combined our many years of experience in developing and debugging Linux systems to create a comprehensive technical guide that consolidates our knowledge.

This course offers a structured approach to Linux debugging, covering the entire stack — from user-space to kernel-space.

Join the course today! Access it via these links: Part 1, Part 2

Memory management is vital for efficient resource use, system stability, and performance. In an unmanaged languages like C, poor memory management can cause performance degradation, application crashes, and security vulnerabilities. Addressing these issues is essential to ensure reliable and secure software.

There are powerful tools available to catch memory issues, one of which is Sanitizers. Sanitizers are runtime analysis tools that instrument code during compilation to detect programming errors such as memory issues, undefined behaviours, and race conditions. They rely on compiler instrumentation to insert runtime checks around memory accesses and reports violations as they occur. Techniques like shadow memory are also employed to track the validity of memory states.

A wide range of sanitizers is available, including AddressSanitizer, MemorySanitizer, ThreadSanitizer, and UndefinedBehaviorSanitizer, applicable to both user-space and kernel code. The sanitizers are supported by major OS platforms (Linux, OS X, iOS Simulator, FreeBSD, Android) and compilers like GCC and Clang, enabling developers to address issues effectively. Refer to the compiler documentation for specific sanitizer options.

AddressSanitizer (ASan)

AddressSanitizer (ASan) is a runtime memory error detector for C/C++ programs. It uses compiler instrumentation and shadow memory to detect and report memory issues effectively. ASan helps identify common memory-related problems such as:

• Use after free (dangling pointer de-reference)
• Heap buffer overflow
• Stack buffer overflow
• Global buffer overflow
• Use after return
• Use after scope
• Initialization order bugs
• Memory leaks

Installation and Usage

To use AddressSanitizer, the libasan library must be installed on your system. On Ubuntu, it can be installed using the command: sudo apt-get install libasan8. Note that the number in libasan (e.g., 8) corresponds to the compatible version of the library with your GCC version. For example, libasan8 is compatible with GCC version 12.

You can verify this compatibility by running apt-cache show libasan8. In the “Depends” section of the output, you will find the GCC version it supports—in this case, GCC 12.

ubuntu@sandbox:~$ apt-cache show libasan8
Package: libasan8
...
Depends: gcc-12-base (= 12.3.0-1ubuntu1~22.04), libc6 (>= 2.34), libgcc-s1 (>= 3.3)
...

To check gcc version you can use command gcc --version.

ubuntu@sandbox:~$ gcc --version
gcc (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
...

To enable AddressSanitizer (ASan), compile your code with the -fsanitize=address compiler option. Additionally, it is recommended to include the -fno-omit-frame-pointer option to ensure a reliable stack trace when an issue is detected.

gcc -fsanitize=address -fno-omit-frame-pointer -g -O1 -o program program.c

ASan Demonstration

We will demonstrate AddressSanitizer (ASan) using a sample program intentionally written with three memory-related issues. This will help illustrate how ASan detects and reports such errors during runtime.

Out-of-bound memory issue code:

void out_of_bounds_access() {
    int arr[5];
    // Condition: Accessing array out of bounds (i=5).
#ifdef CAUSE_OUT_BOUND
    for (int i = 0; i <= 5; i++) {
#else
    for (int i = 0; i < 5; i++) {
#endif
        arr[i] = i * 10;
    }
}

The sample program includes an integer array of size 5. While initializing the array, it writes to the 6th element, which exceeds the array’s upper bound. This out-of-bounds write can be enabled or disabled by defining or undefining the CAUSE_OUT_BOUND macro.

Use-after-free memory issue code:

void use_after_free() {
    int *ptr = malloc(sizeof(int));
    *ptr = 42;
    free(ptr);

    // Re-allocate memory to overwrite the freed block
    int *new_ptr = malloc(sizeof(int));
    *new_ptr = 99;

    // Condition: use-after-free, ptr was freed before
#ifdef CAUSE_USE_FREE
    printf("Use-after-free value: %d\n", *ptr);
#endif
    free(new_ptr);
}

In the program, we dynamically allocate memory for an integer, assign its address to a pointer ptr, initialize it with a value, and then free the memory. Subsequently, we reallocate memory, overwriting the previously freed block, and repeat the operation. However, in a print statement, we attempt to access ptr after it has been freed, resulting in a use-after-free memory issue. This issue can be enabled or disabled by defining or undefining the CAUSE_USE_FREE macro.

Memory-leak code:

void memory_leak() {
    int *leaked_ptr = malloc(10 * sizeof(int)); 
    for (int i = 0; i < 10; i++) {
        leaked_ptr[i] = i * i;
    }
    printf("Leaked memory initialized.\n");
    // Condition: Memory leak, Intentionally no free here.
#ifndef CAUSE_MEM_LEAK
    free(leaked_ptr);
#endif
}

The program allocates memory for 10 integers and initializes it. However, the allocated memory is not freed before the program exits, causing a memory leak. This memory leak issue can be enabled or disabled by defining or undefining the CAUSE_MEM_LEAK macro.

In the CMake file, the -fsanitize=address compiler option is used to enable AddressSanitizer, along with the -fno-omit-frame-pointer option to ensure a reliable stack trace for effective debugging.

Next, we will build the program using the provided helper build script.

ubuntu@sandbox:~/work/Examples/Sanitizers/Asan$ ./build.sh 
-- The C compiler identification is GNU 12.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ubuntu/work/Examples/Sanitizers/Asan
[ 50%] Building C object CMakeFiles/asan_demo.dir/main.c.o
[100%] Linking C executable asan_demo
[100%] Built target asan_demo

The application binary is named asan_demo. You can use the ldd command to verify that libasan is linked to the binary and ensure AddressSanitizer is properly integrated.

ubuntu@sandbox:~/work/Examples/Sanitizers/Asan$ ldd asan_demo 
 linux-vdso.so.1 (0x00007ffc417ed000)
 libasan.so.8 => /lib/x86_64-linux-gnu/libasan.so.8 (0x00007553be800000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007553be400000)
 libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007553beeff000)
 libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007553beedf000)
 /lib64/ld-linux-x86-64.so.2 (0x00007553bf00a000)

Run the program as usual. If AddressSanitizer (ASan) detects a memory error during runtime, it will generate an error message along with a detailed stack trace, pinpointing the source of the issue for easier debugging.

Out-of-bound memory issue report:

Now, we will execute the program. The first error we encounter is an out-of-bounds error, which occurs when the program tries to write outside the allocated memory for the array.

ubuntu@sandbox:~/work/Examples/Sanitizers/Asan$ ./asan_demo 
=================================================================
==8709==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffd02937fe4 at pc 0x5a828e8fa4a9 bp 0x7ffd02937fa0 sp 0x7ffd02937f90
WRITE of size 4 at 0x7ffd02937fe4 thread T0
    #0 0x5a828e8fa4a8 in out_of_bounds_access /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:16
    #1 0x5a828e8fa5ae in main /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:49
    #2 0x7f7038029d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #3 0x7f7038029e3f in __libc_start_main_impl ../csu/libc-start.c:392
    #4 0x5a828e8fa1e4 in _start (/home/ubuntu/work/Examples/Sanitizers/Asan/asan_demo+0x11e4)

Address 0x7ffd02937fe4 is located in stack of thread T0 at offset 52 in frame
    #0 0x5a828e8fa2b8 in out_of_bounds_access /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:8

  This frame has 1 object(s):
    [32, 52) 'arr' (line 9) <== Memory access at offset 52 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:16 in out_of_bounds_access
Shadow bytes around the buggy address:
  0x10002051efa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002051efb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002051efc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002051efd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002051efe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x10002051eff0: 00 00 00 00 00 00 f1 f1 f1 f1 00 00[04]f3 f3 f3
  0x10002051f000: f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002051f010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002051f020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002051f030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002051f040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==8709==ABORTING

Let’s look into the key details from the report.

=================================================================
==8709==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffd02937fe4 at pc 0x5a828e8fa4a9 bp 0x7ffd02937fa0 sp 0x7ffd02937f90
WRITE of size 4 at 0x7ffd02937fe4 thread T0

The heading of the error message indicates the type of memory issue detected, along with the address, program counter, and other relevant CPU registers. In this case, the issue is a stack-buffer-overflow, caused by a write of size 4 at the specified address.

#0 0x5a828e8fa4a8 in out_of_bounds_access /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:16
#1 0x5a828e8fa5ae in main /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:49
#2 0x7f7038029d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#3 0x7f7038029e3f in __libc_start_main_impl ../csu/libc-start.c:392
#4 0x5a828e8fa1e4 in _start (/home/ubuntu/work/Examples/Sanitizers/Asan/asan_demo+0x11e4)

Address 0x7ffd02937fe4 is located in stack of thread T0 at offset 52 in frame
    #0 0x5a828e8fa2b8 in out_of_bounds_access /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:8

The next block is the stack trace. Stack trace is pointing to the function call where the issue occured.

This frame has 1 object(s):
    [32, 52) 'arr' (line 9) <== Memory access at offset 52 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:16 in out_of_bounds_access

The buffer overflow is caused by accessing an associated object, in this case, the array arr. The array is allocated between offsets 32 and 52, and the access occurs at offset 52, which overflows the last byte of the array.

Shadow bytes around the buggy address:
  0x10002051efa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002051efb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002051efc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002051efd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002051efe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x10002051eff0: 00 00 00 00 00 00 f1 f1 f1 f1 00 00[04]f3 f3 f3
  0x10002051f000: f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002051f010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002051f020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002051f030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002051f040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Next, we have shadow memory mappings, which provide metadata about the application memory. Each shadow byte corresponds to 8 bytes of application memory. When an issue occurs, the pointer in the error report points to the shadow memory mapping related to the buggy address.

The addressable memory of the array is surrounded by inaccessible memory zones for protection. In the shadow memory, f1 represents the left guard zone of the stack, and f3 represents the right guard zone. The 0s indicate the addressable memory, while numbers like [04] represent partially addressable ranges. In this case, 04 means that 4 out of 8 bytes are valid, helping to visualize the precise memory boundaries and detect overflows.

Let’s fix the out of bound memory issue and try to find other issues present in our code.

Use-after-free memory issue report:

This time, we encounter a use-after-free memory issue. The program attempts to access memory that was freed earlier, which results in undefined behavior.

ubuntu@sandbox:~/work/Examples/Sanitizers/Asan$ ./asan_demo 
=================================================================
==8844==ERROR: AddressSanitizer: heap-use-after-free on address 0x602000000010 at pc 0x5f75065c0502 bp 0x7ffee6601c20 sp 0x7ffee6601c10
READ of size 4 at 0x602000000010 thread T0
    #0 0x5f75065c0501 in use_after_free /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:31
    #1 0x5f75065c0583 in main /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:50
    #2 0x743c3f429d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #3 0x743c3f429e3f in __libc_start_main_impl ../csu/libc-start.c:392
    #4 0x5f75065c01e4 in _start (/home/ubuntu/work/Examples/Sanitizers/Asan/asan_demo+0x11e4)

0x602000000010 is located 0 bytes inside of 4-byte region [0x602000000010,0x602000000014)
freed by thread T0 here:
    #0 0x743c3f8be470 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52
    #1 0x5f75065c04c5 in use_after_free /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:23
    #2 0x5f75065c0583 in main /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:50
    #3 0x743c3f429d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

previously allocated by thread T0 here:
    #0 0x743c3f8bf91f in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x5f75065c04ba in use_after_free /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:21
    #2 0x5f75065c0583 in main /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:50
    #3 0x743c3f429d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

SUMMARY: AddressSanitizer: heap-use-after-free /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:31 in use_after_free
Shadow bytes around the buggy address:
  0x0c047fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c047fff8000: fa fa[fd]fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==8844==ABORTING

Looking into the error report, the issue is caused by a use-after-free error. The program attempts to read 4 bytes of memory at a given address that has already been freed.

#0 0x5f75065c0501 in use_after_free /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:31
    #1 0x5f75065c0583 in main /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:50
    #2 0x743c3f429d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #3 0x743c3f429e3f in __libc_start_main_impl ../csu/libc-start.c:392
    #4 0x5f75065c01e4 in _start (/home/ubuntu/work/Examples/Sanitizers/Asan/asan_demo+0x11e4)

The stack trace provides detailed information about the sequence of function calls leading to the issue. It points to the exact location where the use-after-free error occurred.

0x602000000010 is located 0 bytes inside of 4-byte region [0x602000000010,0x602000000014)

The next line in the error report provides details about the memory block involved in the issue. The allocated memory block was 4 bytes in size, starting at address 0x602000000010 and ending at 0x602000000014. The problematic access occurred exactly at the start of this region, at address 0x602000000010 (0 bytes offset).

freed by thread T0 here:
    #0 0x743c3f8be470 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52
    #1 0x5f75065c04c5 in use_after_free /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:23
    #2 0x5f75065c0583 in main /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:50
    #3 0x743c3f429d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

previously allocated by thread T0 here:
    #0 0x743c3f8bf91f in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x5f75065c04ba in use_after_free /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:21
    #2 0x5f75065c0583 in main /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:50
    #3 0x743c3f429d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

These two traces provide a history of the object’s lifecycle. The first trace indicates where the memory was freed, pointing to the exact location in the code where the free operation occurred. The second trace shows where the memory was initially allocated, providing the original location where the memory block was created. Together, these traces help to understand the sequence of events that led to the use-after-free issue.

Shadow bytes around the buggy address:
  0x0c047fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c047fff8000: fa fa[fd]fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa

In the shadow memory section, we analyze the pointed buggy address. FA denotes a Heap redzone, which is a guard region used to detect buffer overflows. The fd entry indicates a Freed heap region, meaning the memory was previously allocated but has now been freed. The shadow memory points to the exact location that has been freed, but the program attempts to reference it again, leading to the use-after-free error.

Let’s fix the use-after-free and see the remaining issue in our program.

Memory leak error report

ubuntu@sandbox:~/work/Examples/Sanitizers/Asan$ ./asan_demo 
Leaked memory initialized.

=================================================================
==8906==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x7541cdebf91f in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x5ccf6207745a in memory_leak /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:37
    #2 0x5ccf620774ca in main /home/ubuntu/work/Examples/Sanitizers/Asan/main.c:51
    #3 0x7541cda29d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

SUMMARY: AddressSanitizer: 40 byte(s) leaked in 1 allocation(s).

AddressSanitizer (ASan) is reporting a memory leak. The stack trace points to the location where the leaked memory was allocated, providing context on where the memory was not properly freed. In the summary, ASan also reports the total number of leaked bytes, helping to quantify the memory that was allocated but not deallocated, thus causing the leak.

The added instrumentation significantly slows down the program and it is recommended to use ASan in development and testing environment only. You can refer to various ASAN options here.

I hope this demonstration was helpful for you to understand the address sanitizer and its usage.

LINUX DEBUG TRAINING

If you want to dive deeper, then you can also check out Part One and Part Two of the Linux Debugging Training Course. This course is a collaboration between my colleague John OSullivan and myself. We have combined our many years of experience in developing and debugging Linux systems to create a comprehensive technical guide that consolidates our knowledge.

This course offers a structured approach to Linux debugging, covering the entire stack — from user-space to kernel-space.

Join the course today! Access it via these links: Part 1, Part 2

Kexec and Kdump are mechanisms in the Linux kernel used to capture and recover the kernel memory snapshot (/proc/vmcore) after a kernel panic, without requiring a full system reboot. It enables the preservation and retrieval of the system kernel’s memory image, which can later be used for detailed post-mortem analysis to diagnose the cause of the crash.

Image Credit: Wikipedia

• On panic, the kernel’s kexec support allows the execution of a “dump-capture” kernel directly from the crashed kernel.
• The “dump-capture” or Kdump kernel includes only the essential device drivers and features required to capture the crash dump.
• Kexec reserves a small section of memory in the RAM for the dump-capture kernel.
• Use crashkernel kernel boot parameter to specify the memory region for dump-capture kernel.
• The kexec -p command loads the dump-capture kernel into the reserved memory.

/sbin/kexec -p - command-line="BOOT_IMAGE=/boot/vmlinuz-6.8.0–40-generic root=UUID=3e78aa08-eb2d-4019-b50c-ade108081f72 ro quiet splash vt.handoff=7 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb" - initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

• After booting from the “dump-capture” kernel, the kernel coredump found at /proc/vmcore can be stored as a file on the local filesystem or sent remotely over NFS or SSH.

Kexec & Kdump configuration on Ubuntu 22.04 environment

The Linux kernel documentation explains the kexec and kdump mechanisms and their configuration in detail. This post discusses how to configure kexec and kdump in an Ubuntu 22.04 environment.

Installation and Configuration

linux-crashdump package provides all necessary tools to configure Kexec and Kdump.

sudo apt install linux-crashdump

During the package installation, when prompted, select ‘Yes’ to allow kexec-tools to handle reboots.

┌────────────────────────┤ Configuring kexec-tools ├────────────────────────┐
 │                                                                           │
 │ If you choose this option, a system reboot will trigger a restart into a  │
 │ kernel loaded by kexec instead of going through the full system boot      │
 │ loader process.                                                           │
 │                                                                           │
 │ Should kexec-tools handle reboots?                                        │
 │                                                                           │
 │                                                                  │
 │                                                                           │
 └───────────────────────────────────────────────────────────────────────────┘

Select ‘Yes’ to enable Kdump.

┌────────────────────────┤ Configuring kdump-tools ├────────────────────────┐
 │                                                                           │
 │ If you choose this option, the kdump-tools mechanism will be enabled. A   │
 │ reboot is still required in order to enable the crashkernel kernel        │
 │ parameter.                                                                │
 │                                                                           │
 │ Should kdump-tools be enabled by default?                                 │
 │                                                                           │
 │                                                                  │
 │                                                                           │
 └───────────────────────────────────────────────────────────────────────────┘

After installation, you can confirm the status of kexec and kdump by checking the respective configuration files.

# Invoke root user using 'sudo su' command.

root@sandbox:~# cat /etc/default/kexec | grep LOAD_KEXEC
LOAD_KEXEC=true

root@sandbox:~# cat /etc/default/kdump-tools | grep USE_KDUMP
# USE_KDUMP - controls kdump will be configured
USE_KDUMP=1

You can also manually enable the Kexec and Kdump functionality using dpkg-reconfigure kexec-tools and dpkg-reconfigure kdump-tools.

Increase the reserved memory size for dump-capture kernel

“/etc/default/grub.d/kdump-tools.cfg” defines the crashkernel boot parameter, which is appended to the kernel command line. I found that the default size was insufficient for my needs, so I increased it. Depending on the size of your RAM, you may need to adjust the crashkernel boot parameter accordingly.

root@sandbox:~# cat /etc/default/grub.d/kdump-tools.cfg 
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT crashkernel=2G-:512M"

root@sandbox:~# update-grub

Install Linux debug symbols on Ubuntu 22.04

Linux debug symbols or the vmlinux file is required to analyze the core dump. Follow the below given instructions to install vmlinux on Ubuntu 22.04.

• Import the signing key

sudo apt install ubuntu-dbgsym-keyring

• Create a ddebs.list file

echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list

• Install Linux debug symbols

sudo apt-get update
sudo apt-get install linux-image-`uname -r`-dbgsym

root@sandbox:/home/ubuntu# ls /lib/debug/boot/
vmlinux-6.8.0–40-generic

Reboot the system to enable the kdump-tools.

At any time, if you choose to update the Kdump and Kexec configuration, you can reload the Kdump using the command kdump-config reload.

root@sandbox:~# kdump-config reload
 * unloaded kdump kernel
 * Creating symlink /var/lib/kdump/vmlinuz
 * Creating symlink /var/lib/kdump/initrd.img
 * loaded kdump kernel

Verify Kdump configuration after reboot

After reboot, verify the status of kdump using kdump-config show command.

root@sandbox:~# kdump-config show
DUMP_MODE:              kdump
USE_KDUMP:              1
KDUMP_COREDIR:          /var/crash
crashkernel addr: 0xb000000
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-6.8.0-40-generic
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-6.8.0-40-generic
current state:    ready to kdump

kexec command:
  /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-6.8.0-40-generic root=UUID=3e78aa08-eb2d-4019-b50c-ade108081f72 ro quiet splash vt.handoff=7 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

“Ready to kdump” indicates that your kdump configuration is correct.

Check the crashkernel parameter in kernel command line.

root@sandbox:/home/ubuntu# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.0-40-generic root=UUID=3e78aa08-eb2d-4019-b50c-ade108081f72 ro quiet splash crashkernel=2G-:512M vt.handoff=7

Check the kernel boot logs to verify the memory reservation for the kdump kernel.

root@sandbox:/home/ubuntu# dmesg | grep -i crash
...
[    0.004218] crashkernel reserved: 0x000000000b000000 - 0x000000002b000000 (512 MB)

Trigger the crash kernel (the moment of truth!)

There are various ways to trigger a panic in the Linux kernel. For example, using a custom kernel module to induce a panic. However, we will use the ‘SysRq’ to trigger a panic and crash the kernel. You can read more about SysRq here.

# Invoke root user using 'sudo su' command.
# Crash the kernel using SysRq trigger
root@sandbox:~# echo c > /proc/sysrq-trigger

Upon triggering a kernel crash, the system memory is preserved, and the system boots directly into the kdump kernel, bypassing the usual BIOS initialisation process. This transition happens quickly, and the dump-capture process begins immediately.

On Ubuntu, it’s important to note that the system does not boot into the normal desktop mode at this point. To observe the kdump kernel capturing the memory dump, switch to a virtual terminal (press Ctrl+Alt+F3 after the crash) to see the dump-capture logs in action. During this phase, the kdump kernel uses the makedumpfile utility to capture /proc/vmcore and dmesg into the /var/crash directory. Refer the snapshot showing logs from kdump kernel.

After the memory dump is saved, the kdump kernel reboots, and the system then boots into the normal desktop mode. Because this process happens quickly, it may seem like the system has booted normally right after the crash, but the crash dump is completed in the background beforehand.

Kdump kernel dumping vmcore file in /var/crash

Core dump analysis

The core dumps are collected in /var/crash within a directory named with a date timestamp. The /proc/vmcore is saved in a file named dump.<timestamp>, and the dmesg output is saved in a file named dmesg.<timestamp>.

root@sandbox:/var/crash# tree 
.
├── 202409061559
│   ├── dmesg.202409061559
│   └── dump.202409061559
├── kdump_lock
├── kexec_cmd
└── linux-image-6.8.0-40-generic-202409061210.crash

1 directory, 5 files

Core dump post-mortem analysis using crash utility

We can use the crash utility to perform a post-mortem analysis of the core dump. Load the vmlinux file and the core dump into the crash utility to analyze the cause of the crash. For example, using the bt (backtrace) command within crash, you can determine that the panic was triggered by sysrq_handle_crash

root@sandbox:/var/crash/202409061559# crash /usr/lib/debug/boot/vmlinux-6.8.0-40-generic dump.202409061559

WARNING: kernel version inconsistency between vmlinux and dumpfile

      KERNEL: /usr/lib/debug/boot/vmlinux-6.8.0-40-generic
    DUMPFILE: dump.202409061559  [PARTIAL DUMP]
        CPUS: 32
        DATE: Fri Sep  6 15:59:05 IST 2024
      UPTIME: 00:34:52
LOAD AVERAGE: 0.20, 0.43, 0.36
       TASKS: 947
    NODENAME: sandbox
     RELEASE: 6.8.0-40-generic
     VERSION: #40~22.04.3-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 30 17:30:19 UTC 2
     MACHINE: x86_64  (2000 Mhz)
      MEMORY: 15.7 GB
       PANIC: "Kernel panic - not syncing: sysrq triggered crash"
         PID: 2711
     COMMAND: "bash"
        TASK: ffff952455685200  [THREAD_INFO: ffff952455685200]
         CPU: 14
       STATE: TASK_RUNNING (PANIC)

crash> bt
PID: 2711   TASK: ffff952455685200  CPU: 14  COMMAND: "bash"
 #0 [ffffbab508593980] machine_kexec at ffffffffadab753b
 #1 [ffffbab5085939e0] __crash_kexec at ffffffffadc2a4f3
 #2 [ffffbab508593aa8] panic at ffffffffadb00704
 #3 [ffffbab508593b28] sysrq_handle_crash at ffffffffae4d5dda
 #4 [ffffbab508593b38] __handle_sysrq at ffffffffae4d659e
 #5 [ffffbab508593b80] write_sysrq_trigger at ffffffffae4d6e32
 #6 [ffffbab508593bb0] proc_reg_write at ffffffffadfa1069
 #7 [ffffbab508593bd0] vfs_write at ffffffffadee08ed
 #8 [ffffbab508593c68] ksys_write at ffffffffadee1033
 #9 [ffffbab508593ca8] __x64_sys_write at ffffffffadee10e9
#10 [ffffbab508593cb8] x64_sys_call at ffffffffada07891
#11 [ffffbab508593cc8] do_syscall_64 at ffffffffaebfccc1
#12 [ffffbab508593f50] entry_SYSCALL_64_after_hwframe at ffffffffaee00130
    RIP: 0000737a90d14887  RSP: 00007ffd21746498  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: 0000000000000002  RCX: 0000737a90d14887
    RDX: 0000000000000002  RSI: 000060d472c243a0  RDI: 0000000000000001
    RBP: 000060d472c243a0   R8: 0000000000000000   R9: 000060d472c243a0
    R10: 000060d472c24240  R11: 0000000000000246  R12: 0000000000000002
    R13: 0000737a90e1b780  R14: 0000737a90e17600  R15: 0000737a90e16a00
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b
crash>

We can also investigate various aspects of the system at the time of the crash, such as the process table, virtual memory, and more. To gain a deeper understanding of the crash utility and its capabilities, you can refer to the documentation available on the crash utility documentation page.

Notes on GDB

Kdump uses the makedump utility to dump /proc/vmcore into a dump file. By default, makedump compresses the dump file in a format that only the crash utility can read. To generate an ELF format that gdb can understand, you can modify the MAKEDUMP_ARGS configuration option in /etc/default/kdump-tools .

MAKEDUMP_ARGS="-E -d 31"

References

• Debug symbol packages
• Kernel crash dump
• The Linux Kernel
• White Paper: Crash Utility

LINUX DEBUG TRAINING

If you want to dive deeper, then you can also check out Part One and Part Two of the Linux Debugging Training Course. This course is a collaboration between my colleague John OSullivan and myself. We have combined our many years of experience in developing and debugging Linux systems to create a comprehensive technical guide that consolidates our knowledge.

This course offers a structured approach to Linux debugging, covering the entire stack — from user-space to kernel-space.

Join the course today! Access it via these links: Part 1, Part 2

The perf tool is a part of the Linux kernel code base. The standard Linux distribution package for the perf tool includes only basic support, lacking the full array of options. This necessitates building the perf tool from scratch with all options enabled. Here are the steps to build perf from scratch:

Step 1: Download the Kernel source code

curl -o linux-source.{VERSION}.tar.gz https://mirrors.edge.kernel.org/pub/linux/kernel/v6.x/linux-{VERSION}.tar.gz

Note: Replace the kernel version. I am using Version 6.2

Step 2: Extract the Kernel source package

tar -xvf linux-source.{VERSION}.tar.gz

Step 3: Install build essential packages.

sudo apt install make gcc flex bison pkg-config

Step 4: Install the required libraries to enable all the feature in perf

libzstd1
libdwarf-dev
libdw-dev
binutils-dev
libcap-dev
libelf-dev
libnuma-dev
python3
python3-dev
python-setuptools
libssl-dev
libunwind-dev
libdwarf-dev 
zlib1g-dev 
liblzma-dev 
libaio-dev
libtraceevent-dev
debuginfod
libpfm4-dev
libslang2-dev
systemtap-sdt-dev
libperl-dev
binutils-dev
libbabeltrace-dev
libiberty-dev
libzstd-dev

sudo apt install libzstd1 libdwarf-dev libdw-dev binutils-dev libcap-dev libelf-dev libnuma-dev python3 python3-dev python-setuptools libssl-dev libunwind-dev libdwarf-dev zlib1g-dev liblzma-dev libaio-dev libtraceevent-dev debuginfod libpfm4-dev libslang2-dev systemtap-sdt-dev libperl-dev binutils-dev libbabeltrace-dev libiberty-dev libzstd-dev

Step 5: Step into the perf directory and make

manas@sandbox: ~/kernel cd linux-6.2/tools/perf
manas@sandbox: ~/kernel/linux-6.2/tools/perf make

The build configuration will display the supported features. In this case, all the required libraries are present. The build configuration will also indicate any missing libraries if they are not present in the system

Auto-detecting system features:
…                               dwarf: [ on ]
…                  dwarf_getlocations: [ on ]
…                               glibc: [ on ]
…                              libbfd: [ on ]
…                      libbfd-buildid: [ on ]
…                              libcap: [ on ]
…                              libelf: [ on ]
…                             libnuma: [ on ]
…              numa_num_possible_cpus: [ on ]
…                             libperl: [ on ]
…                           libpython: [ on ]
…                           libcrypto: [ on ]
…                           libunwind: [ on ]
…                  libdw-dwarf-unwind: [ on ]
…                                zlib: [ on ]
…                                lzma: [ on ]
…                           get_cpuid: [ on ]
…                                 bpf: [ on ]
…                              libaio: [ on ]
…                             libzstd: [ on ]

UPDATE: If building for kernel version 6.5. Suppress the warning options

make -C tools/perf/ WERROR=0

Step 6: Check perf version

manas@sandbox: ~/kernel/linux-6.2/tools/perf$ ./perf --version
perf version 6.2.0

This article discusses installing two different versions of Ubuntu (or two copies of the same version of Ubuntu side by side) in a “Dual boot” configuration. We are using a UEFI-based system as opposed to a legacy BIOS-based system to achieve our goal. The goal is to create a system which allows us to switch between the two instances of the operating system by switching the EFI boot order rather than rely on having to make manual selections from a GRUB menu.

Note: Although we are using the Ubuntu distribution here, the process is applicable to other Linux distributions. For example Fedora alongside Fedora, etc.

Step 1: Create a bootable Ubuntu USB drive. Refer to the following tutorial for instructions.

https://ubuntu.com/tutorials/create-a-usb-stick-on-ubuntu#1-overview

Step 2: Boot your system from the USB key by selecting the USB option from the boot menu (or selecting that option in your bios boot order configuration). These options are normally accessible by hitting designated keys during the early boot phase. For example on some systems hitting F7 will bring up a boot menu and hitting the DEL or ESC key will bring up the Bios Configuration Menu.

Step 3: After booting from USB key and starting the installation process, the installer will prompt for a disk location to install the Ubuntu. Select the “Something else” option.

Ubuntu installer screen

Step 4: Create the partitions as defined in the table below. For this example we are using a 250 GB disk and we have sized the partitions accordingly. Given the size of the disk you are using you can change the size or scheme of the partition but the partitions marked in **BOLD **are essential for dual boot.

Partition Table

Ubuntu installer showing partitions

Step 5: Select partition #5 to install the Ubuntu OS. Define the mount point as “/” and format the partition using ext4 formatting. You can refer to the following link for more detailed Ubuntu install directions.

https://ubuntu.com/tutorials/install-ubuntu-desktop-1804#7-begin-installation

Step 6: The installer will install the bootloader and configuration files into EFI A partition #1 and OS in partition #5. Once the installation is complete, the system will boot up with EFI A on partition #1 and the GRUB bootloader will load the OS installed in partition #5.

Step 7: After successful login into Ubuntu OS B, we need to move the EFI bootloader from EFI A partition #1 to EFI B partition #4. For this, perform the below given operations.

A). Mount EFI B partition temporarily: sudo mount /dev/nvme0n1p4 /mnt

B). Copy EFI files from EFI A to EFI B: sudo cp -R /boot/efi/EFI /mnt/

C). Unmount EFI B partition: sudo umount /mnt

D). Get the UUID for EFI B partition /dev/nvme0n1p4: sudo blkid -s UUID -o value /dev/nvme0n1p4

E). Replace the UUID of **/dev/nvme0n1p4 with /dev/nvme0n1p1 **in /etc/fstab (/boot/efi) entry.

root@tester:~# sudo blkid -s UUID -o value /dev/nvme0n1p4
43F5-E57Y

root@tester:~# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#                
# / was on /dev/nvme0n1p2 during installation
UUID=6916e733-647a-4ce8-8b0c-61c4197000ef /               ext4    errors=remount-ro 0       1
# /boot/efi was on /dev/nvme0n1p1 during installation
UUID=45D1-4A56  /boot/efi       vfat    umask=0077      0       1
# swap was on /dev/nvme0n1p3 during installation
UUID=0811a10d-3312-48a7-a850-0c113d8bfdb8 none            swap    sw              0       0

Step 8: Now install Ubuntu A to the Ubuntu OS A partition. For that repeat steps 1 to 3. At the partitioning screen, select partition #2 for Ubuntu OS A. Define the mount point as “/” and format the partition using ext4 formatting. Please refer to the following link for more detailed Ubuntu install directions.

https://ubuntu.com/tutorials/install-ubuntu-desktop-1804#7-begin-installation

After finishing step #8, you have UBUNTU OS A and UBUNTU OS B installed on partition #2 and partition #5 respectively. The EFI configuration installed on partition #1 will point to Ubuntu OS A. The EFI configuration installed on partition #4 will points to Ubuntu OS B. The next step is to use the EFI boot variables to switch between two Ubuntu instances. For this, we will use efibootmgr command. This allows us to alter the boot options from userspace.

efibootmgr -v will dump the current boot options. Ideally, we want to keep only OS A and OS B boot options and delete all the others, but depending on your requirement, you can alter the boot options.

root@tester:~# efibootmgr -v
BootCurrent: 0003
Timeout: 1 seconds
BootOrder: 0003,0000,0001,0002,0004,0005
Boot0000* Ubuntu-OS-A  HD(1,GPT,d92d26b7-1251-49f7-819e-742475c27581,0x800,0x30000)/File(\EFI\UBUNTU\SHIMX64.EFI)
Boot0001* Ubunut-OS-B  HD(4,GPT,9c85e16a-f97a-4b9b-bd25-64217d1b1884,0x6ff3000,0x30000)/File(\EFI\UBUNTU\SHIMX64.EFI)
Boot0002* UEFI:CD/DVD Drive     BBS(129,,0x0)
Boot0003* ubuntu        HD(1,GPT,d92d26b7-1251-49f7-819e-742475c27581,0x800,0x30000)/File(\EFI\UBUNTU\SHIMX64.EFI)
Boot0004* UEFI:Removable Device BBS(130,,0x0)
Boot0005* UEFI:Network Device   BBS(131,,0x0)

Delete all of the boot options below using the following command.

sudo efibootmgr -b  -B

Example: sudo efibootmgr -b 0000 -B to delete the boot entry at 0000, and similarly for the rest of the boot entries by referencing the corresponding boot entry number.

Add two new boot entries for UBUNTU OS A and OS A

sudo efibootmgr -c -d /dev/nvme0n1 -p 1 -L "UBUNTUOSA" -l '\EFI\UBUNTU\SHIMX64.EFI'
sudo efibootmgr -c -d /dev/nvme0n1 -p 4 -L "UBUNTUOSB" -l '\EFI\UBUNTU\SHIMX64.EFI'

Switch the boot order from user space to boot from other OS after the next reboot.

sudo efibootmgr -o 0000,0001 -> To boot from OS A
sudo efibootmgr -o 0001,0000 -> To boot from OS B