0

Running Ubuntu 22.04.2 with latest patches. This host is an NFS client running kernel Linux adam408 5.19.0-35-generic #36~22.04.1-Ubuntu

Connectivity is over 25gig Ethernet to Cisco Nexus fabric.

NFS mounts mount fine but performance is very slow and very variable. A user may run

/usr/bin/time -p /usr/bin/ls -l /nfsdata/dir/ | wc -l and get the following

real 9.36 user 0.03 sys 0.57 7127

Real times vary any where from 1.5seconds to 10seconds or higher. The command always completes but takes a very long time. There are only 7127 files in the directory.

Here are the mount options: fsserver:/nfsdata/dir on /nfsdata/dir type nfs4 (rw,relatime,vers=4.2,rsize=65536,wsize=65536,namlen=255,hard,nosharecache,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.1.1.140,local_lock=none,addr=10.1.1.250)

Further context and history:

This is happening across all nodes in a small cluster. All running the same OS kernel and patch level. Prior to an update of the OS (from ubuntu 20.04 to 22.04.2) this was not an issue.

We have tried adjusting various settings in fstab like nordirplus, mountvers, sync as well as many others with no effect. We have also investigated networking both the host and switch interfaces show now errors. Pinging between the Netapp NFS server and the host is sub-milisecond with no packet loss. No other applications seem to be affected. The NetApp is brand new fully patched. We do see this performance problem whether we are accessing a NetApp NFS server or another Ubuntu NFS server. The hosts and the NFS servers are all on the same subnet so there is no routing involved.

We are running out of ideas.

Anyone seen this before?

jds
  • 1
  • 2
  • After more testing I took another machine running Ubuntu 22.04.2 with same kernel and patch level and mounted the storage on it over a 1gig adapter and it works fine. no performance issues at all. – jds Mar 24 '23 at 20:58
  • Otherwise problem still exists on the server hardware. – jds Mar 24 '23 at 20:59
  • ok this issue us getting weirder. I run my test from my newly built test machine and performance looks good ls -l on the mounted file system takes less than a second. but then i log out and log back in and run the exact same test and it takes 10 to 15 seconds. – jds Mar 24 '23 at 22:00
  • Does this answer your question? [Surprising slowdown with Apache/NFS on Ubuntu 22.04.2](https://askubuntu.com/questions/1459450/surprising-slowdown-with-apache-nfs-on-ubuntu-22-04-2) – karel Mar 25 '23 at 14:02

1 Answers1

1

Sounds like what happened to us. Surprising slowdown with Apache/NFS on Ubuntu 22.04.2

Issue was related to the latest kernel https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2009325

Param Dhaliwal
  • 11
  • 2