How do I ensure hv_utils is updating the system clock in Azure

Question

We have Ubuntu 16.04 VMs running in Azure, and have noticed the clock has drifted ~20 seconds. The system is using a generic Ubuntu Image (Managed by our IT department, unsure of any changes/modifications that were made) and does NOT have access to the Internet or any public NTP pools. Also there are no NTP servers in our Azure region, but I see it is trying to reach the default configuration of ntp.ubuntu.com:

$ cat /etc/systemd/timesyncd.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See timesyncd.conf(5) for details.

[Time]
#NTP=
#FallbackNTP=ntp.ubuntu.com

$ sudo systemctl status systemd-timesyncd
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
           └─disable-with-time-daemon.conf
   Active: active (running) since Tue 2017-12-05 15:59:12 UTC; 1 weeks 6 days ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 957 (systemd-timesyn)
   Status: "Idle."
    Tasks: 2
   Memory: 560.0K
      CPU: 17.173s
   CGroup: /system.slice/systemd-timesyncd.service
           └─957 /lib/systemd/systemd-timesyncd

Dec 18 23:52:54 VMNAME systemd-timesyncd[957]: Timed out waiting for reply from 91.189.91.157:123 (ntp.ubuntu.com).
Dec 18 23:53:04 VMNAME systemd-timesyncd[957]: Timed out waiting for reply from 91.189.89.199:123 (ntp.ubuntu.com).

It is my understanding that recent Ubuntu images natively support the built in Time synchronization of Hyper-V. Indeed I can see that the hv_utils kernel module has been loaded, but I don't see any messages for it in syslog

$ lsmod | grep hv
hv_balloon             24576  0
hv_storvsc             20480  2
hv_netvsc              40960  0
scsi_transport_fc      61440  1 hv_storvsc
hv_utils               24576  2
hv_vmbus               73728  7 hv_balloon,hyperv_keyboard,hv_netvsc,hid_hyperv,hv_utils,hyperv_fb,hv_storvsc

$ dmesg
[    2.706739] clocksource: hyperv_clocksource_tsc_page: mask: 0xffffffffffffffff max_cycles: 0x24e6a1710, max_idle_ns: 440795202120 ns
[    2.709052] Floppy drive(s): fd0 is 1.44M
[    2.735624] FDC 0 is an 82078.
[    2.737857] clocksource: Switched to clocksource hyperv_clocksource_tsc_page
[    2.748059] hv_vmbus: Hyper-V Host Build:14393-10.0-0-0.230; Vmbus version:4.0
[    2.757269] hv_vmbus: Unknown GUID: f8e65716-3cb3-4a06-9a60-1889c5cccab5
[    2.761412] hv_vmbus: Unknown GUID: 3375baf4-9e15-4b30-b765-67acb10d607b
[    2.766081] hv_vmbus: Unknown GUID: 276aacf4-ac15-426c-98dd-7521ad3f01fe
[    2.775497] hv_vmbus: registering driver hyperv_fb
[    2.779440] hyperv_fb: Screen resolution: 1152x864, Color depth: 32
[    2.786286] Console: switching to colour frame buffer device 144x54
[    2.807248] hv_vmbus: registering driver hyperv_keyboard
[    2.812765] hv_utils: Registering HyperV Utility Driver
[    2.815236] hv_vmbus: registering driver hv_util
[    2.819533] hidraw: raw HID events driver (C) Jiri Kosina
[    2.825316] AVX2 version of gcm_enc/dec engaged.
[    2.827788] AES CTR mode by8 optimization enabled
[    2.831632] hv_vmbus: registering driver hid_hyperv
[    2.834734] input: Microsoft Vmbus HID-compliant Mouse as /devices/0006:045E:0621.0001/input/input4
[    2.839602] hid 0006:045E:0621.0001: input: <UNKNOWN> HID v0.01 Mouse [Microsoft Vmbus HID-compliant Mouse] on
[    2.845660] hv_vmbus: registering driver hv_storvsc
[    2.849843] scsi host2: storvsc_host_t
[    2.853512] scsi 2:0:0:0: Direct-Access     Msft     Virtual Disk     1.0  PQ: 0 ANSI: 5
[    2.857949] hv_vmbus: registering driver hv_netvsc

Is there anything else I need to do to enable the Hyper-V Utilities Driver time keeping/syncronization?

Answer 1

So it looks like there was a recent known issue with the Hyper-V hv_util time source. Also when we restarted the VM, the 20 second drift in time went away, so it is likely that the hardware clock was working but having issues updating the system clock.

Our VM was using kernel 4.4.0-47, and the above issue was patched in 4.4.0-87.110. After running sudo apt-get update && sudo apt-get upgrade and then restarting the VM we now have kernel 4.4.0-104. I'll keep monitoring for drift, but am still curious if there is any way to confirm that hv_utils is updating the system clock.

Update: So far this looks like it fixed the issue. Previously we saw about ~1ms drift ever 24 seconds (just over 3.5s drift every 24hr), and after updating the kernel we are seeing the delta stay below 200ms.

$ sudo apt-get install iputils-clockdiff
$ clockdiff TESTNTPSERVER
.
host=VMNAME rtt=750(187)ms/0ms delta=-182ms/-182ms Tue Dec 19 03:46:43 2017

Update: Over the next 24 hours we saw it continue to drift from -200ms to +2904ms. It appears the above patch was to fix contention between the in-kernel time adjustments and a user space ntpd/systemd-timesyncd process that caused the server to keep drifting. In our case systemd-timesyncd has never successfully updated the clock since it can't reach the default ntp server (confirmed by the journalctl -u systemd-timesyncd error messages and the empty /var/lib/systemd/clock file). So what we are seeing is the new behavior in the 4.4.0-87.110 kernel, where instead of in-kernel time adjustments the hyper-v clock is now exposed as a PTP Hardware Clock (PHC). Using phc_ctl we can see this clock on the VM (note this also enables the ptp4l service):

$ clockdiff TESTNTPSERVER
............
host=VMNAME rtt=33(93)ms/0ms delta=2904ms/2904ms Tue Dec 19 20:59:48 2017

$ ls -la /var/lib/systemd/clock
-rw-r--r-- 1 systemd-timesync systemd-timesync 0 Feb 11  2016 /var/lib/systemd/clock

$ sudo apt-get install linuxptp
$ sudo systemctl disable ptp4l
$ sudo phc_ctl /dev/ptp0 get;echo "              date: clock time is $(date +%s.%N) or $(date)"
phc_ctl[65577.406]: clock time is 1513721576.691973400 or Tue Dec 19 22:12:56 2017
              date: clock time is 1513721574.065892000 or Tue Dec 19 22:12:54 UTC 2017

So if we want to use the built in Hyper-V time synchronization on a 4.4.0-87.110 or newer kernel we need to use a time sync process that supports a PTP Hardware Clock.