5

Prologue:

On a number of machines, which happen to act as NFS clients, netstat reports two ports that are open with no PID listed for an associated daemon. Ordinarily this might be a bit concerning.

# netstat -lnp | egrep -- '- +$'
tcp        0      0 0.0.0.0:57448           0.0.0.0:*               LISTEN     -
udp        0      0 0.0.0.0:48933           0.0.0.0:*                          -

Additionally netcat confirms that the TCP port really is open.

# nc -v localhost 57448
localhost [127.0.0.1] 57448 (?) open
^C

Yet lsof reports nothing for these two ports. Intrigue grows.

# lsof -i TCP:57448 -i UDP:48933

However rpcinfo finally points us in the right direction. It's being held open by nlockmgr, aka lockd for NFS. Call off the search.

# rpcinfo -p | egrep '57448|48933'
    100021    1   udp  48933  nlockmgr
    100021    3   udp  48933  nlockmgr
    100021    4   udp  48933  nlockmgr
    100021    1   tcp  57448  nlockmgr
    100021    3   tcp  57448  nlockmgr
    100021    4   tcp  57448  nlockmgr

It becomes clear that lockd/rpc.lockd is called when mounting an NFS export. This is a kernel thread (is it always?) which binds itself to one TCP and one UDP port in the ephemeral range. The ports are typically reconfigurable with the fs.nfs.nlm_tcpport and fs.nfs.nlm_udpport sysctls.

Questions:

I am intrigued though. Would love some kernel-internals insight as to..

  1. Why isn't the kernel thread's PID visible from netstat?

  2. Why aren't the bound ports visible from lsof?

Dan Carley
  • 25,617
  • 5
  • 53
  • 70

1 Answers1

7

netstat and lsof both crawl /proc/<pid>/fd to associate processes to ports/sockets, and /proc/<pid>/fd doesn't get populated for kernel threads AFAIK.

lockd is always a kernel thread - at least on modern (newer than say 2.2) kernels.

James
  • 7,643
  • 2
  • 24
  • 33