The /proc Filesystem: Talking to the Kernel Directly

Nearly every tool covered so far in this module — ps, top, nice, renice, even kill's existence check — ultimately reads from or writes to the same underlying source. Where does RSS in ps aux actually come from? What computes the NI column in top? How does the kernel expose a process's open file descriptors so lsof can list them? The answer to all of these is the same: /proc, a virtual filesystem that exposes live kernel and process state as ordinary files and directories, generated on demand rather than read from disk.

This article is the capstone of the module: it shows where the numbers from every earlier article actually live, how to read them directly without any intermediate tool, and why understanding /proc turns you from someone who runs commands into someone who can verify, script against, and debug situations no single existing tool covers.

/proc Is Not a Real Filesystem

/proc looks like an ordinary directory tree, but nothing under it exists on disk. It is a pseudo-filesystem — the kernel generates each file's content at the moment it is read, reflecting the system's live state at that exact instant. Writing to certain files under /proc similarly changes live kernel behavior immediately, without a reboot or service restart.

mount | grep '^proc'

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

ls -l on a file under /proc typically reports a size of 0 for files whose content is computed on read, since there is no static byte count to report in advance — a small, telling clue that you are not looking at a conventional file.

Per-Process Directories: /proc/[pid]

Every running process has its own directory named after its PID:

ls /proc/$$

$$ expands to the current shell's own PID. A representative (trimmed) listing:

cmdline  cwd  environ  exe  fd  limits  maps  mounts  net  root  smaps  stat  statm  status  task

status: The Human-Readable Summary

cat /proc/$$/status | head -15

Name:   bash
State:  S (sleeping)
Tgid:   3990
Pid:    3990
PPid:   3821
Uid:    1000    1000    1000    1000
Gid:    1000    1000    1000    1000
VmSize:    23412 kB
VmRSS:      5120 kB
Threads:    1

Uid/Gid each show four values — real, effective, saved-set, and filesystem — the same distinctions covered under process credentials; a mismatch between the real and effective values here (for example, during a sudo-elevated command) is exactly what id and ps -o ruser,user summarize into a friendlier form. VmRSS is the exact figure Process Memory covered as RSS in ps output — ps simply reads this same field for you.

stat and statm: The Machine-Readable Form

cat /proc/$$/stat

3990 (bash) S 3821 3990 3990 34816 4021 4194560 3421 891 0 0 12 4 0 0 20 0 1 0 8821 23977984 1280 ...

This single space-separated line is what ps and top actually parse to build their columns — field 3 is the state character (S here, matching STAT in ps), field 4 is the PPID, and field 18 (20 in this excerpt) is the raw priority value underlying nice. Reading this file directly is rarely necessary day to day, but knowing it exists demystifies exactly how every process-monitoring tool gets its numbers — none of them have privileged access you do not; they all read the same public interface.

cmdline, cwd, exe: Identifying What a Process Actually Is

cat /proc/$$/cmdline | tr '\0' ' '; echo
readlink /proc/$$/cwd
readlink /proc/$$/exe

-bash 
/home/ali
/usr/bin/bash

cmdline separates its arguments with null bytes rather than spaces (translated here with tr for readability) — this is why a command containing a literal space in one argument still shows up correctly in tools that parse this file properly, unlike naively splitting ps's formatted text output on whitespace. exe is a symlink to the actual binary currently backing this process — useful for confirming exactly which file, and from which path, a running process was launched from, especially after a package upgrade replaced the file on disk but the old process (per the note in Programs and Processes) is still running the previous version.

fd: Every Open File Descriptor

sudo ls -l /proc/$(pgrep -x nginx | head -1)/fd

lrwx------ 1 www-data www-data 64 Jul 29 09:12 0 -> /dev/null
lrwx------ 1 www-data www-data 64 Jul 29 09:12 3 -> socket:[48212]
lrwx------ 1 www-data www-data 64 Jul 29 09:12 4 -> /var/log/nginx/access.log

This is exactly what lsof -p <pid> reads and formats for you — every open file, socket, and pipe a process currently holds, as a symlink under its fd directory. This is the definitive way to answer "which process still has this deleted file open, keeping its disk space from actually being freed" — a deleted-but-still-open file shows up here with (deleted) appended to its target path.

sudo ls -l /proc/*/fd 2>/dev/null | grep '(deleted)'

limits: Resource Limits in Effect for This Process

cat /proc/$$/limits | head -5

Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max open files             1024                 4096                files

This is the authoritative, live view of the limits a process is actually running under, as configured by ulimit, PAM's limits.conf, or a systemd unit's LimitNOFILE= and similar directives — reading this file directly settles arguments about whether a configuration change actually took effect for a specific already-running process, rather than only for new ones started after the change.

System-Wide Files: Not Tied to Any One Process

/proc also exposes machine-wide state, independent of any particular process.

cat /proc/loadavg

4.15 2.03 0.98 2/187 22884

This is the exact source uptime and top's header read for the load average figures covered in Process Monitoring — the three familiar numbers, followed by "currently runnable / total processes" and the most recently created PID.

cat /proc/meminfo | head -6

MemTotal:       16310924 kB
MemFree:          421356 kB
MemAvailable:    4198212 kB
Buffers:          198420 kB
Cached:          5488216 kB
SwapTotal:        2097148 kB

MemAvailable here is precisely the available column free -h derives its own figure from — reading it directly confirms free's summary rather than trusting it blindly, useful when scripting a monitoring check that needs the raw kilobyte figure rather than a human-formatted table.

cat /proc/cpuinfo | grep -c ^processor

Counting processor entries in /proc/cpuinfo is the same figure nproc reports, just derived directly from the source nproc itself reads.

Tuning the Kernel Live: /proc/sys

A subset of /proc, under /proc/sys, is writable and changes kernel behavior immediately — no reboot, no service restart:

cat /proc/sys/vm/swappiness

60

vm.swappiness (readable both as /proc/sys/vm/swappiness and via the sysctl command, which is the preferred, safer interface for the exact same value) controls how aggressively the kernel prefers reclaiming page cache versus swapping out anonymous memory — a value covered in more depth in later system tuning material, mentioned here specifically as an example of a live, writable kernel parameter exposed through this same filesystem.

sudo sysctl vm.swappiness
sudo sysctl -w vm.swappiness=10

A Realistic Scenario: Finding the Process Holding a Deleted File Open

Problem. df -h shows a filesystem at 98% capacity, but du -sh across the visible directory tree only accounts for a fraction of that space, and no obviously large file can be found by browsing normally.

Confirm the discrepancy first, without guessing:

df -h /var
du -sh /var/* 2>/dev/null | sort -rh | head

If the sum from du falls well short of what df reports as used, the missing space is very likely held by a deleted-but-still-open file.

Search /proc directly for exactly this pattern:

sudo find /proc/*/fd -maxdepth 1 -lname '*(deleted)*' 2>/dev/null -exec ls -l {} \;

lrwx------ 1 www-data www-data 64 Jul 29 09:12 /proc/2214/fd/7 -> /var/log/app/debug.log (deleted)

Confirm which process this actually belongs to:

ps -o pid,ppid,user,cmd -p 2214

Conclusion. The debug.log file was deleted (perhaps by a naive rm instead of a proper log rotation), but process 2214 still has it open and is presumably still writing to it, so its disk blocks remain allocated and invisible to any directory listing. The correct fix is to signal the owning process to reopen its log file — many services do this on SIGHUP or via a dedicated reload command — rather than searching further for a file that genuinely no longer has a name anywhere in the filesystem. If the process cannot cleanly reopen its log, a controlled restart releases the file descriptor and, with it, the disk space; either way, the actual underlying fix for next time is proper log rotation (logrotate, covered in a later module) instead of manually deleting active log files.

Common Mistakes

Editing /etc/sysctl.conf Directly for a One-Off Test

A distribution or package update can silently overwrite this file. Use sysctl -w for a temporary, immediate test, and a dedicated file under /etc/sysctl.d/ only once you are certain the change should persist.

Assuming a Deleted File's Space Is Immediately Freed

The directory entry disappears immediately; the underlying disk blocks stay allocated until every process with the file open closes it. /proc/*/fd scanning is the direct way to find the responsible process, not repeated du runs on the visible tree.

Parsing ps or top Output Instead of Reading /proc in a Script

ps and top's human-formatted columns can shift subtly between distributions and versions. A script that needs a specific, stable numeric field (VmRSS, load average, open file descriptor count) is often more robust reading the corresponding /proc file directly, in the exact format documented in proc(5).

Treating Every /proc/sys Value as Safe to Change

Most tunables are safe to adjust and revert; a small number can destabilize the system immediately if set incorrectly. Always check the kernel documentation for a specific parameter before writing to it, especially anything under /proc/sys/kernel/ or /proc/sys/vm/ you have not used before.

Forgetting That /proc/[pid] Disappears the Instant the Process Exits

A script reading /proc/<pid>/status in a loop must handle the directory vanishing mid-read if the process exits between checks — treat a missing directory as "process no longer exists," not as an error condition to retry indefinitely.

Exercises

1. Find your current shell's PID with $$, then read /proc/$$/status and identify the VmRSS value; compare it against what ps -o rss -p $$ reports for the same process, and confirm they match.
2. Start a background process, open a file for writing inside it (for example tail -f against a file you create), delete that file with rm while tail -f is still running, and use /proc/<pid>/fd to prove the file's data is still accessible through the still-open descriptor even though ls no longer shows the file by name.
3. Read /proc/loadavg directly and compare its first three fields against uptime's reported load average on the same system at roughly the same moment.
4. Without changing anything, look up (via sysctl -a or the kernel documentation) what /proc/sys/vm/overcommit_memory controls, and explain in your own words why setting it incorrectly is more risky than adjusting vm.swappiness.

Verification criterion: exercise 2's proof must show the file's content still readable through /proc/<pid>/fd/<n> after deletion — merely observing that the process kept running is not sufficient evidence.

Where This Module Leaves You

Across this module, Programs and Processes established what a process actually is and how the kernel creates one; Process Monitoring covered watching that state continuously with top and htop; Process Memory covered reading memory consumption correctly; Process Priority covered biasing how CPU time is shared; Signals and Process Control covered terminating, pausing, and communicating with a running process; Foreground and Background Jobs covered the shell's own layer of control over where a job runs and whether it survives a disconnect. This article showed that every one of those tools is, underneath, reading from or writing to the same public, kernel-exposed filesystem — which means nothing covered in this module was ever hidden behind a special privilege you do not also have direct access to.

The next module turns from processes themselves to how software gets onto the system in the first place: package management, dependency resolution, and keeping installed software up to date safely.

References

• Linux man-pages: proc(5): https://man7.org/linux/man-pages/man5/proc.5.html
• Linux kernel documentation: filesystems/proc.rst: https://www.kernel.org/doc/html/latest/filesystems/proc.html
• Linux man-pages: sysctl(8), sysctl.conf(5): https://man7.org/linux/man-pages/man8/sysctl.8.html
• Linux kernel documentation: sysctl documentation for vm.*: https://www.kernel.org/doc/html/latest/admin-guide/sysctl/vm.html
• Linux man-pages: lsof(8): https://man7.org/linux/man-pages/man8/lsof.8.html
• Ubuntu Server documentation: kernel documentation and sysctl: https://ubuntu.com/server/docs