[Tracking] procfs scan performance improvements

[ekoops]

Procfs scan in linux platform introduces overhead. Fortunately, in the current codebase, there is room for improvements. This issue aims at documenting the effort of improving information gathering from procfs.

---

• perf(userspace/libscap): rewrite /proc/<pid>/comm parsing logic #2785
• perf(userspace/libscap): rewrite /proc/<pid>/environ parsing logic #2786
• perf(userspace/libscap): rewrite /proc/<pid>/cmdline parsing logic #2787
• refactor(userspace/libscap): rename helper reading /proc/<pid>/cwd #2788
• perf(userspace/libscap): rewrite /proc/<pid>/status parsing logic #2789
• perf(userspace/libscap): rewrite /proc/<pid>/stat parsing logic #2793
  • fix(userspace/libscap) fix /proc/pid/stat read operation #2794
• perf(userspace/libscap): rewrite scap_linux_getpid_global() logic #2795
• perf(userspace/libscap): rewrite /proc/<pid>/loginuid parsing logic #2797
• perf(userspace/libscap): rewrite scap_linux_is_thread_alive() logic #2798
• refactor(userspace/libscap): rename helper reading /proc/<pid>/root #2799
• fix(userspace/libscap): re-add interruption resiliency to reads #2800
• perf(userspace/libscap): improve scap_proc_add_from_proc() logic #2801
• perf(userspace/libscap): rewrite scap_linux_get_device_by_mount_id() #2809
• refactor(userspace/libscap/linux): extract string helpers #2810
• perf(userspace/libscap): rewrite /proc/pid/fdinfo/fd parsing logic #2813
• perf(userspace/libscap): rewrite /proc/pid/net/{tcp,udp,raw} parsing logic #2815
• perf(userspace/libscap): rewrite /proc/pid/net/{tcp6,udp6,raw6} parsing logic #2816
• perf(userspace/libscap): rewrite /proc/pid/net/unix parsing logic #2823
• fix(userspace/libscap): rewrite /proc/pid/net/netlink parsing logic #2831
• perf(userspace/libscap/linux): rewrite scap_fd_handle_regular_file #2928

[ekoops]

Regarding collecting info for sockets, I explored an alternative approach. Currently:

1. we scan the /proc/<pid>/fd and only consider those files pointing to sockets (readlink() resolves to socket[<inode_num>])
2. we scan the content of /proc/<pid>/net/{tcp,udp,raw,tcp6,udp6,raw6,unix,netlink} files to collect sockets information
3. we store these information in an intermediate hash table, mapping the inode number of each socket to the corresponding info
4. correlate the inode number extracted in the first point with the ones in the hash table, to associate each file descriptor to the corresponding socket info.

The alternative approach leverages netlink diag sockets for directly asking the kernel to provide the same information provided by /proc/<pid>/net/<file> interfaces. The support for this netlink protocol is also available in old kernels (in 3.10 for sure). I made a prototype in rust, and the only information I couldn't extract is the source and destination pointers we are used to extract from /proc/<pid>/net/unix for unix sockets. Netlink doesn't expose these legacy information as the modern and more secure way of uniquely identifying a socket is by using its cookie. The quirk here is that it exposes the cookie for the current socket (the source) but the inode number for the peer (destination) socket. There is no easy way of retrieving the peer cookie, especially if we take into account the fact that sockets can live in a different network namespace with respect to the one we switch to seek the task sockets. Nevertheless, we can iterate two times over the returned sockets and fill the peer cookie field for each with the one extracted from the other retrieved sockets.

The question now is: do we really want to continue to use the legacy pointer-based mechanism, or do we want to switch to socket cookies? Also please consider that, from my preliminary analysis, it appears that we are not using much the source and destination pointers, if we exclude printing.

[ekoops]

Just want to give a quick update regarding my tests. Unexpectedly, turned out that retrieving sockets information using netlink is slower than using using our current approach. This is due to the fact that, in order to get information from netlink for a single AF_* socket family, we must first create a socket (via socket()) and connect it to the kernel via connect()) before sending the request (sendmsg()) and starting receiving responses (recvmsg()s): this is heavier than just opening a file like /proc/<pid>/net/tcp via open() and reading it via read()s. I run benchmarks by considering up to 10k sockets, and by constructing sockets in such a way that an increasing fraction of them belonged to the task's network namespace: still, the advantage of parsing binary data doesn't overcome the advantage of issuing fewer system calls.
Given this, it is now clear that switching to the netlink-based approach is not beneficial.