Process Memory: Virtual Address Space, RSS, and the OOM Killer
ps -o vsz,rss and top's memory columns show two numbers per process, and they can differ by an order of magnitude for the exact same process without anything being wrong. A process can report a 2 GB virtual size and still be running perfectly fine on a machine with 512 MB of RAM, because virtual size and actual memory consumption answer two different questions. Confusing them is one of the most common causes of a wrong "we're out of memory" diagnosis — and, at the other extreme, of a genuine leak going unnoticed until the kernel's OOM killer starts terminating processes.
This article builds on Process Monitoring, where top's summary memory line was mentioned but not explained, and gives you the vocabulary and commands to read process memory correctly: virtual versus resident size, shared versus private pages, swap behavior, and how the Out-Of-Memory killer decides which process to sacrifice when the system genuinely runs out.
Virtual Memory: Every Process Gets Its Own Address Space
Every process on Linux runs inside its own virtual address space — a private range of addresses the process's code uses, translated by the kernel and the CPU's memory management unit into actual physical RAM (or, for swapped-out pages, disk) behind the scenes. Two unrelated processes can both reference address 0x400000 without any conflict, because each process's 0x400000 maps to entirely separate physical memory.
This abstraction is why a single process can safely be given a large virtual size (VSZ) without that number reflecting real memory pressure. VSZ includes:
- the process's own code and data;
- every shared library it has mapped, even though the physical pages backing those libraries are shared across every other process using the same library;
- memory the process has reserved (via
malloc, for example) but never actually touched — Linux does not back a page with physical RAM until it is first written to, a behavior called demand paging; - memory-mapped files.
A process can therefore show VSZ of several gigabytes purely from mapping large shared libraries and reserving address space it never fully uses.
Resident Set Size: What's Actually in RAM
RSS (Resident Set Size) is the portion of a process's virtual memory that currently has real physical pages backing it — the number that actually corresponds to RAM consumption at this instant.
VSZ and RSS are both reported in kibibytes by default. postgres here reserves roughly 400 MB of address space but is currently resident in about 183 MB of actual RAM — the gap is unused reservation, shared library mappings counted once per process even though physically shared, and pages that were touched once and later reclaimed by the kernel.
Warning
Summing RSS across all processes to estimate total memory usage systematically overcounts real usage whenever processes share memory — most commonly, multiple worker processes of the same server (several nginx or PHP-FPM workers) mapping the same shared libraries and, in some architectures, shared memory segments. Each worker's RSS includes those shared pages in full, so ten workers each showing 40 MB of RSS do not necessarily mean 400 MB of unique physical memory is in use.
PSS: A Fairer Per-Process Number
PSS (Proportional Set Size) solves exactly this overcounting problem by dividing each shared page's cost proportionally among every process mapping it. smem (install with sudo apt install smem on Debian/Ubuntu) reports it directly:
PID Command RSS PSS USS
2201 nginx: worker process 9.8M 5.1M 3.2M
2202 nginx: worker process 9.7M 5.0M 3.1M
1842 nginx: master process 8.4M 6.9M 6.4M
-------------------------------------------------------
Total 27.9M 17.0M 12.7M
USS (Unique Set Size) is the memory that only this one process holds — nothing shared — and is the most accurate figure for "how much memory would actually be freed if this exact process alone were killed." PSS is the fairest number to sum across processes to estimate a service's true total footprint, since shared pages are counted only once in total, split fractionally per process.
Where /proc Fits In
Every number ps and top display for a process ultimately comes from that process's own directory under /proc, covered in depth in /proc Filesystem. For memory specifically:
VmSwap is worth checking specifically — a nonzero value here means part of this individual process has been pushed out to swap, which can cause noticeably slower response on its next access even while overall system memory looks fine in a summary tool.
Reading free -h Correctly
total used free shared buff/cache available
Mem: 15Gi 9.4Gi 411Mi 412Mi 5.6Gi 4.0Gi
Swap: 2.0Gi 1.8Gi 157Mi
The single most misread line on any Linux system. free under free looks alarmingly low, but it only counts memory holding nothing at all — it does not count buff/cache, which is the kernel using otherwise-idle RAM to cache disk blocks and recently-used files. That cache is not wasted memory; it is memory doing useful work (avoiding a disk read next time the same data is needed) that the kernel will instantly reclaim and hand to an application the moment it actually needs more.
The column that matters for "can a new process get memory right now" is available, which is the kernel's own estimate of memory that could be freed for new allocations without swapping, accounting for reclaimable cache. 4.0Gi available here, despite free showing only 411Mi, is the accurate figure.
Interview angle
"Your server shows 400 MB free out of 16 GB — is it about to run out of memory?" is a common trick question. The correct answer starts by pointing at buff/cache and available, not free. Only sustained pressure on available memory, combined with rising swap usage (Swap: used climbing) and, eventually, kernel log messages about reclaim, indicates genuine memory pressure. A large buff/cache with a healthy available figure is a normally functioning, well-utilized system, not a warning sign.
Swap: A Safety Net, Not Extra RAM
Swap space lets the kernel move rarely-used pages out of RAM onto disk, freeing physical memory for more active use. It is not a substitute for RAM in terms of speed — a page fault that must be satisfied from swap is orders of magnitude slower than one satisfied from RAM, because it requires a disk (or SSD) I/O operation.
NAME TYPE SIZE USED PRIO
/swap.img file 2G 1.8G -2
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 2 1843200 421200 812400 5872000 4 12 180 420 2210 3890 18 6 68 8 0
Nonzero, sustained si/so (swap in/swap out) in vmstat while the system is under load is the clearest sign of active swap thrashing — pages being moved to and from disk repeatedly, which degrades performance far more than the raw "used" swap percentage alone would suggest. A system with 90% of its swap used but si/so both consistently at 0 is not actively thrashing; it simply moved some cold pages out once and left them there.
The OOM Killer
When the kernel genuinely cannot satisfy a memory allocation — no free RAM, swap exhausted or disabled, and reclaiming cache is not enough — it invokes the OOM (Out-Of-Memory) killer rather than letting the whole system deadlock. The OOM killer selects a process to terminate based on an internal oom_score, computed per process and readable directly:
cat /proc/$(pgrep -x postgres | head -1)/oom_score
cat /proc/$(pgrep -x postgres | head -1)/oom_score_adj
A higher oom_score makes a process a more likely target. The score is influenced heavily by how much memory the process is using, and can be manually biased with oom_score_adj, ranging from -1000 (never kill this process for being an OOM target) to 1000 (kill this one first). Setting -1000 on a critical system process (commonly done automatically by systemd for its own core units) protects it from ever being chosen, but it does not protect the system from memory exhaustion — something else will still be killed instead.
Jul 29 03:14:02 web01 kernel: Out of memory: Killed process 18422 (java) total-vm:4213212kB, anon-rss:3921004kB, ...
This kernel log entry is the definitive record of an OOM kill — the total-vm and anon-rss figures show exactly how much memory the killed process was using at the moment of termination, which is the starting point for figuring out whether it was a genuine leak or simply an undersized memory limit for legitimate workload growth.
Danger
Do not "fix" recurring OOM kills by disabling swap or setting vm.overcommit_memory to a permissive value without understanding the consequence. Disabling swap removes the safety margin the kernel uses before reaching for the OOM killer, which usually makes the kernel invoke it sooner, not less often. Address the actual cause — a leaking process, an undersized container memory limit, or genuinely insufficient RAM for the workload — rather than the symptom.
A Realistic Scenario: Distinguishing a Leak from Normal Cache Growth
Problem. free -h shows memory usage climbing steadily over several days on an application server, and a teammate wants to restart the service "just in case." Before doing that, you need evidence: is this an actual leak in the application, or is it the kernel simply using more RAM for cache, which is expected and harmless?
Check the overall picture first:
If available stays healthy and mostly buff/cache is growing while used (the non-cache figure) is flat, this points toward normal cache growth, not a leak — proceed no further with the "just restart it" plan.
If used (not cache) is genuinely climbing, isolate the specific process:
Run this same command every hour (or capture it in a small script writing to a log) over several hours:
PID ELAPSED RSS VSZ
22110 18000 412300 1820400
PID ELAPSED RSS VSZ
22110 21600 498100 1820400
PID ELAPSED RSS VSZ
22110 25200 583900 1820400
Expected result if healthy. RSS should plateau once the application reaches a steady working set — some initial growth as caches warm up is normal, but it should stop increasing under a stable, repeating workload.
Conclusion. In this example, RSS is climbing roughly linearly with elapsed time under what should be a steady workload, while VSZ stays flat — a strong signal of an actual memory leak inside the application, not cache behavior (VSZ staying flat also rules out the process simply reserving more address space without using it). The fix belongs in the application (a profiler such as a language-specific heap analyzer, or a memory-limited restart policy as a stopgap) — restarting the process only delays the same growth from recurring, and does not, by itself, constitute a diagnosis.
Common Mistakes
Reading VSZ as Actual Memory Usage
VSZ includes reserved-but-untouched address space and every shared library mapping. Use RSS, or better, PSS/USS from smem, for anything resembling a real memory usage claim.
Treating free's "free" Column as Available Memory
free under free -h excludes reclaimable cache. Use the available column for any decision about whether the system is actually short on memory.
Summing RSS Across Worker Processes
Multiple workers of the same service share library and, often, shared-memory-segment pages. Their RSS values double-count that shared memory; use PSS from smem for an accurate total.
Assuming High Swap Usage Alone Means Trouble
A large amount of used swap with si/so both near zero in vmstat means cold pages were moved out once and are simply sitting there — not actively harming performance. Watch si/so, not the raw swap percentage, for real thrashing.
Disabling Swap to "Solve" OOM Kills
Removing swap removes a buffer the kernel uses before invoking the OOM killer, typically making out-of-memory kills happen sooner under pressure, not later. Diagnose the actual memory consumer first.
Exercises
- Run
ps -eo pid,comm,vsz,rss --sort=-rss | headon your own system and explain, for the top process, roughly why itsVSZis likely larger than itsRSS. - Install
smemand compareRSSversusPSSfor a service with multiple worker processes (ornginx/PHP-FPM if available; otherwise start several copies of a small script that share a common interpreter). Explain the direction and rough size of the difference. - Run
free -hand identify which figure represents genuinely available memory for a new process, and explain why the "free" column alone would have given a misleading answer. - Without running anything destructive, explain what
oom_score_adjset to-1000on a process does and does not protect against, using the kernel log format shown above as your reference for what an actual OOM kill event looks like.
Verification criterion: any claim about "how much memory a process uses" in your answers must cite RSS, PSS, or USS explicitly and explain which one is appropriate for the specific question being asked — VSZ alone is not an acceptable answer to any of these exercises.
References
- Linux man-pages:
proc(5)—/proc/[pid]/status,/proc/meminfo: https://man7.org/linux/man-pages/man5/proc.5.html - Linux kernel documentation: overcommit accounting: https://www.kernel.org/doc/html/latest/mm/overcommit-accounting.html
- Linux kernel documentation: the OOM killer: https://www.kernel.org/doc/gorman/html/understand/understand016.html
- Linux man-pages:
free(1): https://man7.org/linux/man-pages/man1/free.1.html - Linux man-pages:
vmstat(8): https://man7.org/linux/man-pages/man8/vmstat.8.html - Linux man-pages:
smem(8): https://man7.org/linux/man-pages/man8/smem.8.html - Red Hat Enterprise Linux 9: understanding memory and swap: https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/monitoring_and_managing_system_status_and_performance/index
- Linux kernel documentation:
oom_score,oom_score_adj: https://www.kernel.org/doc/html/latest/filesystems/proc.html