1 What is /dev/mem?
“/dev/mem” is a character device file, image of the main memory of system. It allows to directly access any phys address.
2 How to use
2.1 Requisites
To use /dev/mem, your kernel must be configured with “CONFIG_STRICT_DEVMEM=n”, or it prevent access from even privileged user.
- “CONFIG_STRICT_DEVMEM=y”, the default kernel configuration in general, disallows to access RAM area via /dev/mem or only allows first 1MB size of RAM
FYI: “CONFIG_IO_STRICT_DEVMEM=y”, disallows to access register via dev/mem
2.2 Let’s code
To start with, open dev/mem device file and map the phys address we are interested in. Note that the phys address should be page-aligned. Depending on request, read or write value to the mapped address. To avoid a code sequence optimization from compiler, use volatile when you read or write.
int mem_fd = open("/dev/mem", O_RDWR | O_SYNC);
void *map_base = mmap(NULL,
PAGE_SIZE,
PROT_READ | PROT_WRITE,
MAP_SHARED,
mem_fd,
phys_addr); // phys_addr should be page-aligned.
void *virt_addr = (char *)map_base + offset_in_page;
if (is_read) {
read_result = *(volatile uint64_t*)virt_addr;
} else { // write
*(volatile uint64_t*)virt_addr = write_value;
}
Useful tool: busybox.
Example of devmem utility from busybox: it tries to directly access MMIO region of Mali GPU and achieves its GPU ID.
3 Problem: impact of cache
If you directly use RAM, the memory might be cached by the CPU which possibly incurs cache coherence problem.
- When you write, the exact value could not be written to the memory yet but CPU cache.
- We don’t know when it will be flushed.
Scenario: I implemented kernel module that allocates pages by alloc_pages (low-level page request mechanism). When user-space application requested, the module allocates the page and passes its phys address to the user-space. The application starts to write something into the allocated page by using /dev/mem.
Observation: When writing some values via /dev/mem, it seems the phys memory is corrupted after the write is done as follows.
Memory is being corrupted as time goes on
Guess: It seems that phys memory where we try to write was cached before. Then we write it directly using dev/mem which bypasses the CPU cache and later, the cache is flushed which corrupts the data we wrote. This may be affected by other cores as well when the device has multiple cores.
Troubleshooting:
i) turn off all the other cores except for the main core (cpu0) which seems work (no data corruption; no flush from other cores)
echo 0 | sudo tee /sys/devices/system/cpu/cpu1/online
ii) map the allocated phys memory as DMA memory, which prevents cached access.
struct page *p = alloc_pages(gfp, pool->order);
dma_addr_t dma_addr = dma_map_page(dev, p, 0, (PAGE_SIZE << pool->order), DMA_BIDIRECTIONAL);
Note: Depending on the memory region the phys address you try to access, the mapping is done with either “pgprot_noncached” or “pgprot_writecombined”. For instance, MMIO region is mapped as noncached while normal memory region as writecombined. It also depends on the your kernel arch, so look into how it is implemented in your kernel source code, “driver/char/ mem.c”.
]]>
