Estimated reading time: 10 minutes

As part of my research, I spend time learning about eBPF. Recently, I have been busy exploring Arena — a new eBPF API that enables programs to allocate memory pages; similar functionality as mmap and munmap [12].

The eBPF community is doing a great job documenting the system and writing tutorials. I felt I could contribute to this effort by writing about Arena.

1 Introduction

The Arena is a new MAP type (BPF_MAP_TYPE_ARENA) available to eBPF programs since kernel version 6.9. This map is semantically different than previous ones. Unlike data-structure MAPs (e.g., hash, array, bloom, stack, …) Arena provides a direct access to the kernel memory instead of abstracting it away. It increases the programs expressivity and enables them to implement their own data-structures on-demand (e.g, a specific type of tree).

Three use-cases have been named for the Arena in the introduction message [12] sent along its patch to the Linux mailing list. I summarize them below:

• User-space can use Arena’s memory to implement its own data-structures (it is a normal memory region), while the memory is visible to eBPF programs which can implement a fast-path for operations like packet processing.
• Arena can be used as a communication channel between an eBPF and a user-space program.
• Arena can be used as a heap of memory for the eBPF program

My understanding from these use-cases is that Arena provides access to raw pages of memory shareable among eBPF programs and user-space. Each can decide to use it as they see fit with their own responsibility. It is unlike other types of data-structure MAPs that enforce certain constraints such as fixed key and value sizes, or limiting the operations to lookup and update.

In this tutorial, I explore Arena’s API, and provide some examples using it. I hope this document helps others explore Arena and facilitate its development. The example codes in this text are shared with you at this GitHub repository. I’ve only shared excerpts from the codes — The parts I thought was important. You may find and read the full examples in the repository.

2 Arena’s API

2.1 Declare a MAP

A program can declare the Arena MAP similar to other MAPs. See Listing 1 for an example. Here are some notes to keep in mind. The key size and value sizes must be zero. The “max_entries” field defines the maximum number of pages for the map, and must be non-zero. As any other map, the maximum size is limited to 4 GB. At the time of writing this text, Arena map declaration supports three flags [5]:

• BPF_F_MMAPABLE: Always required. Indicates the map should support memory mapping.
• BPF_F_SEGV_ON_FAULT: The user-space program will receive a SEGFAULT signal if the memory is not mapped by the eBPF program (e.g., out of range access on a list created by the eBPF program).
• BPF_F_NO_USER_CONV: The memory region is private to the eBPF program and user-space program will not access it. This allows the JIT to perform optimizations avoiding address space casting (more about it later).

note: The BPF_F_MMAPABLE must always be present.

struct { 
    __uint(type, BPF_MAP_TYPE_ARENA); 
    __uint(map_flags, BPF_F_MMAPABLE); 
    __uint(max_entries, 2); /* number of pages */ 
} arena SEC(".maps");

2.2 Helper functions

Normal eBPF MAP helpers such as “bpf_map_lookup_elem” are not defined for Arena [5]. Instead the following pair of functions are available:

• bpf_arena_alloc_pages: Allocates a memory page
• bpf_arena_free_pages: Frees the memory page

These functions are defined using kfunc [4, 2]. As required by kfunc subsystem, you should declare the signature of these helpers in your eBPF program to use these functions.

void __arena* bpf_arena_alloc_pages(void *map, void __arena *addr, 
    __u32 page_cnt, int node_id, __u64 flags) __ksym; 
void bpf_arena_free_pages(void *map, void __arena *ptr, 
    __u32 page_cnt) __ksym;

Calling these functions may put the thread to sleep. For this reason the functions are marked with KF_SLEEPABLE [5], and the verifier only allows sleepable eBPF programs to use them. An eBPF program has sleeping privilege if the BPF_F_SLEEPABLE flag was set when loading it (see Listing 4 for an example). Not all eBPF hooks support this flag. A list of sleepable hooks are provided in libbpf documentations [9].

note: The memory pages allocated use the following flags: GFP_KERNEL, __GFP_ZERO, __GFP_ACCOUNT.

2.3 Managing two address spaces

Clang compiler, the verifier, the JIT, and the runtime all work together to make sure the eBPF program accesses the correct memory address. When using Arena, there is a translation between the user address space and kernel address space (unless the map is declared with “BPF_F_NO_USER_CONV” flag present). The eBPF program must mark the pointers with “__attribute__((address_space(1)))” to let the Clang know about it and cause the generation of “bpf_arena_cast_user/kern” instructions. The “__arena” tag used in the examples (and the accompanying repository), especially when declaring variables and parameters, serves this purpose.

3 Examples

I try to demonstrate how these APIs can be used to implement the use-cases named above.

3.1 Memory shared among eBPF and user-space

As a first step, let us demonstrate how to share a counter between an eBPF and an user-space program. For the sake of example, I used an eBPF program of type BPF_PROT_TYPE_SYSCALL [11] which does not need to be attached to a hook and can simply be invoked using a system call (for more information search for BPF_PROG_TEST_RUN [10]).

Listing 3 shows the code for the example program. In this program, a global variable (“flag_initialized”) tracks whether it is the first time the program is executed or not. In the first run, a page is allocated (using “bpf_arena_alloc_pages”) and its address is kept in “mem” global variable. The allocated memory will hold a counter keeping the number of invocations.

note: The global variable support is available since kernel version 5.2 [1].

/* Declaration of Arena MAP as in Listing 1 ... */ 
static bool flag_initialized = false; 
__arena void *mem = NULL; 
SEC("syscall") 
int mogu(void *_) 
{ 
    bpf_printk("mogu: hello\n"); 
    if (!flag_initialized) { 
        mem = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); 
        if (mem == NULL) { 
            bpf_printk("Failed to allocate memory"); 
            return 1; 
        } 
        flag_initialized = true; 
    } 
    if (mem == NULL) { 
        /* this branch must never happen! */ 
        return 1; 
    } 
    __arena entry_t *e = mem; 
    e->counter += 1; 
    bpf_printk("counter: %lld\n", e->counter); 
    return 0; 
}

Next, in Listing 4, we see how user-space program loads the eBPF program and Listing 5 shows it accessing the counter on the Arena memory page (The user-space relies on libbpf and its skeleton framework [6]).

/* Some global vars */ 
static volatile int running = 0; 
static int ebpf_prog_fd = -1; 
static struct mogu *skel = NULL; 
 
int main(int argc, char *argv[]) 
{ 
skel = mogu__open(); 
if (!skel) { 
    fprintf(stderr, "Failed to open the skeleton\n"); 
    return EXIT_FAILURE; 
} 
/* Set sleepable flag */ 
bpf_program__set_flags(skel->progs.mogu, BPF_F_SLEEPABLE); 
if (mogu__load(skel)) { 
    fprintf(stderr, "Failed to load eBPF program\n"); 
    return EXIT_FAILURE; 
} 
 
ebpf_prog_fd = bpf_program__fd(skel->progs.mogu); 
/* It will invoke the eBPF program for the first time */ 
handle_invoke_signal(0); 
 
/* Keep running and handle signals */ 
running = 1; 
signal(SIGINT, handle_signal); 
signal(SIGHUP, handle_signal); 
signal(SIGUSR1, handle_invoke_signal); 
printf("Hit Ctrl+C to terminate ...\n"); 
printf("Invoke eBPF program:\n"); 
printf("\tMogu: pkill -SIGUSR1 mogu_loader\n"); 
 
while (running) { pause(); } 
 
mogu__detach(skel); 
mogu__destroy(skel); 
printf("Done!\n"); 
return 0; 
}

entry_t *e = skel->bss->mem; 
if (e == NULL) { 
    printf("NOTE: the initialization was not successful!\n"); 
    return; 
} 
printf("user: counter=%lld\n", e->counter);

3.2 Using Arena in XDP (or other hooks that can not sleep)

As was mentioned, Arena memory allocation helper functions may put the calling thread to sleep. This is not acceptable for programs attached to hooks such as XDP. But, this limitation does not mean that XDP programs can not access the Arena memory. They just can not allocate or free the pages.

For the next step, Let us share the memory page allocated by the program from the previous example with another eBPF program. This allows the second program to use Arena backed memory pages without allocating memory itself. Listing 6 shows the second eBPF program.

Here are some details that I like to bring to your attention:

1. Notice that both eBPF programs declare an Arena (as shown in Listing 1). The loader (will be discussed next) will make sure both are using the same MAP using the libbpf’s bpf_map__reuse_fd helper.
2. The second program expects the loader to pass the address of allocated page through its global variable called “mem”. This address could have been shared among the two program in other ways, e.g. through a shared array MAP.
3. The second program is using a non-standard helper named “my_kfunc_reg_arena”. This function (a kfunc defined by me) does not perform any operation. Its sole role is to let the verifier recognize that the program is using the Arena. See §3.2.1 for more detail.

/* Declaration of Arena MAP as in Listing 1 ... */ 
__arena void *mem = NULL; 
 
/* Load the kernel module for adding this kfunc */ 
long my_kfunc_reg_arena(void *p__map) __ksym; 
 
SEC("syscall") 
int aloe_main(void *_) 
{ 
    my_kfunc_reg_arena(&arena_map); 
    if (mem == NULL) { 
        bpf_printk("aloe: not seeing the memory!\n"); 
        return 1; 
    } 
    __arena entry_t *e = mem; 
    bpf_printk("aloe: counter=%lld\n", e->counter); 
    e->counter += 100; 
    return 0; 
}

To share the Arena from the first eBPF program (Listing 3) with the second one (Listing 6), the loader (user-space) program has to mediate and pass the reference from first program to the second. Listing 7 shows an example of this procedure.

int arena_fd = bpf_map__fd(skel1->maps.arena); 
bpf_map__reuse_fd(skel2->maps.arena_map, arena_fd); 
/* Pass the pointer to the  second program */ 
skel2->bss->mem = skel1->bss->mem;

3.2.1 Why do we need a custom Arena function?

If we remove the call to the non-standard my_kfunc_reg_arena in Listing 6, although the program would compile, the verifier will complain that the program does not have any associated Arena map, but is trying to use Arena memory addressing. The error message from verifier is as follows:

addr_space_cast insn can only be used in a program that has an associated arena

Looking at the verifiers source code to decipher the error message, we see that the error is raised when the program is not associated with an Arena while the program has an address space cast instruction [7] (exactly as written in the error message).

The association of an Arena MAP to a program happens when the it is referenced (e.g., by using a helper function). Unfortunately, in the second program we deliberately do not want to use Arena specific helpers. Other helper functions such as bpf_map_lookup_elem also cause other complains from the verifier (in my test it complained due to attempting a zero byte read). Defining a new helper that does nothing but still satisfies the verifier is a smart hack around the situation. It showcases the flexibility that kfunc has brought to eBPF ecosystem.

note: The idea of having a dummy helper was suggested by my lab-mate Marco Mole.

3.3 User-space managed data-structures

In the third example, I demonstrate how user-space program can allocate pages of Arena map (useful to implement its own custom data-structure). Since these data-structures are on Arena pages, they will be accessible to eBPF programs sharing the same Arena.

We can use libbpf to get access to the Arena memory. The library has an API named bpf_map__initial_value (not well documented at the moment). As input, this function receive a pointer to a MAP and a pointer to a 64-bit variable. The return value is the address for the beginning of Arena memory region (mapped to the user-space program memory address), also the 64-bit variable will be set to the size of the currently allocated memory [8].

To allocate memory pages, the user-space can cause page-faults by accessing memory addresses in Arena. The page fault will notify the kernel to allocate the memory page. For example, in our previous examples the Arena has two pages (see Listing 1). Assuming size of each page is 4 KB, the program can access the Arena memory (the address we found using bpf_map__initial_value) at offset 0 and 4096 to allocate the two pages. Accessing the memory outside of the Arena region will cause a SIGSEGV (Address boundary error).

The user-space program can use these memory pages to implement its own data-structures in the Arena which is will be visible to eBPF programs.

4 Final Notes

The Arena feature is very new. It will probably change and improve. I am sure by the time it is mature, eBPF community has written good documentation and tutorials on it but meanwhile I hope this post fill the gap. It is important for me to say that although I tried my best to share correct information, I can not claim that I understand every aspect of Arena’s implementation. There may be misunderstandings on my side. In such a case please let me know and accept my apology.

I want to conclude by listing some of the limitations of Arena that I observed looking at the verifier and x86 JIT code.

1. The atomic reads/writes are not supported in Arena memory region. At the moment, in Arena memory region, limited atomic operations are support from eBPF side [14].
2. Arena pointer arithmetics are converted to 32 bit operations.
3. There can be only one Arena map per program of maximum size of 4 GB.
4. Program must be JIT compiled (it is almost always the case)
5. BPF_CAP and CAP_PERFMON is needed for loading a program with Arena
6. Sign extending load instruction is not supported for Arena memory region
7. The cast to address-space(1) will generate a “dst = (u32)src”
8. The cast lets the verifier know that the pointer is to arena the cast to address-space (0) turns the value into a scalar.
9. x86 JIT uses the R12 register to track the start of Arena virtual address (it was unused before).

4.1 Updates

• 1^st March, 2025:

  1. Correction: Some atomic instructions available to eBPF programs to access Arena [14] and others can be supported in the future by the JIT
  2. New information: There are on going work to support memory allocation in non-sleepable programs (e.g., XDP) [3, 13]. Thanks to Kumar Kartikeya Dwivedi

References