127

I am running Windows 10 Pro 64 bit with Hyper-V and Intel VT-x virtualization technology enabled. When I try to run VirtualBox 64bit, Windows goes into a BSOD. When I run VMware it shows an error.

Why can't VirtualBox and VMware run with Hyper-V enabled? Please explain with all details you have including hardware and software. I want to know the internal cause of this error.

Here are some findings of mine. Most sites suggest adding a boot entry with BCDedit or to disable Hyper-V with BCDedit. e.g. Creating a "no hypervisor" boot entry, Run Hyper-V and VirtualBox on the same machine. But I can run QEMU with Hyper-V. Qemu does not show any error with Hyper-V and runs smoothly.

Stevoisiak
  • 13,555
  • 39
  • 101
  • 154
Biswapriyo
  • 10,831
  • 10
  • 47
  • 78
  • 2
    Hyper-V does not support nested virtualization (with hardware acceleration). However, it will not crash under normal circumstances. VirtualBox would complain that it cannot run x64 guests and that’s it. So something else is wrong like a malfunctioning device driver or whatever. – Daniel B May 13 '17 at 15:37
  • I see, it does indeed crash. However, again: This is not normal. A crash is *never* normal. It appears this is a bug in Hyper-V. You should probably get in touch with Microsoft about it. – Daniel B May 13 '17 at 17:51
  • 1
    Worth pointing out that QEMU isn't a hypervisor. Hyper-V [does](https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/nested-virtualization) support nested virtualization. – Ramhound May 13 '17 at 19:09

4 Answers4

142

VirtualBox and VMware Workstation (and VMware Player) are "level 2 hypervisors." Hyper-V and VMware ESXi are "level 1 hypervisors."

The main difference is that a level 2 hypervisor is an application running inside an existing OS, while a level 1 hypervisor is the OS itself.

This means that when you enable Hyper-V, your Windows 10 "host" becomes a virtual machine. A special one, but nonetheless a virtual machine.

So your question would more aptly be: "Why don't VirtualBox and VMware Workstation work inside a Hyper-V virtual machine?" One can answer because as a VM, the Intel VT-X instruction are no longer accessible from your virtual machine, only the host has access to it.

QEMU works because it does not do virtualization but emulation, which is completely different and explains why QEMU is painfully slow. Virtualization is the process of running a complete isolated machine inside another, but with the help of the processor. This requires the virtual machine and the host to be instruction compatible.

Emulation is the process of running any machine inside a running OS, there is no platform restriction, and is why QEMU can run an ARM machine on an amd64 platform.

Note: QEMU has 2 operating modes:

  • it can work as an emulator, this is the mode explained above
  • it can work as virtualization software with the help of KVM if the guest architecture is compatible with the host's and if the VT instruction is present of course.
Dave Mackey
  • 437
  • 3
  • 10
  • 22
Veovis
  • 1,590
  • 1
  • 10
  • 7
  • 2
    The virtualization creates a whole "fake" system, except for the processor where the hypervisor will only limits the amount of processor time the vm can use. So you can virtualize arm only on an arm host, x86 on an x86/amd64 host, amd64 on an amd64 host etc... An emulator will recompile each instruction for a binary compiled for a specific platform to run on another platform. QEMU belongs to the same family as console emulators for example (psx, dolphin, virtualboy, project64, MAME...) – Veovis May 13 '17 at 16:50
  • Virtualization (yes, *not* emulation) has been available long before hardware-assisted virtualization became available. It can be nested as desired. – Daniel B May 13 '17 at 17:53
  • 4
    Why can VirtualBox not take advantage of "Nested VT-x/AMD-V"? Is this a limitation of the hardware? Of Hyper-V? – user643011 Aug 11 '19 at 21:43
  • 6
    @user643011 - This answer describes the problem well, but one part "as a VM, the Intel VT-X instruction is no longer accessible from your virtual machine, only the host has access to it" isn't completely accurate. Microsoft's Hyper-V chose not to expose these instruction to the guest systems, but nested accelerated virtualization is completely possible with the right software. VMWare for example has an option to expose AMD-V/Intel-vtx to guest called "virtualize Intel-vtx/AMD-V". This incompatibility is caused by Microsoft Hyper-V, not nested virtualization in general. – 1283822 Dec 12 '19 at 19:49
  • @1283822 Thanks! Is this useful? `Set-VMProcessor -ExposeVirtualizationExtensions $true` ("Specifies whether the hypervisor should expose the presence of virtualization extensions to the virtual machine, which enables support for nested virtualization.") Source: https://docs.microsoft.com/en-us/powershell/module/hyper-v/set-vmprocessor?view=win10-ps – user643011 Dec 13 '19 at 04:36
  • It's worth noting that, years on, Microsoft still only supports nested virtualisation (enabled with `ExposeVirtualizationExtensions`) for Intel CPUs - support for AMD CPUs is still inexplicably only available in Insiders builds. – Cocowalla Apr 09 '21 at 20:21
  • Nice explanation. – Naveen Kumar V Nov 19 '21 at 10:54
41

Starting from Windows Redstone 4 build, QEMU will be able to run under Hyper-V by employing the Windows Hypervisor Platform API.

The Windows Hypervisor Platform accelerator (WHPX) is being integrated to QEMU patches are submitted for merging.

The Windows Hypervisor Platform API will be included in Redstone 4 build

Experimental support for WHPX is included in QEMU 2.12

Update: Virtualbox 6.0 added support for WHPX Virtualbox 6.0 Changelog.

Update 2: VMWare will add support for WHPX in upcoming version of VMWare Workstation and fusion. VMware Workstation and Hyper-V – Working Together

Update 3: Initial VMWare Workstation release with WHPX support. Software requirements are Windows Insiders Minimum build number 19041. Hardwre requirements are Intel Haswell or newer CPU, AMD Bulldozer or newer CPU. VMware Workstation Tech Preview 20H1 — Hyper-V/Host VBS Support

nkef
  • 553
  • 4
  • 5
  • 4
    This API in theory could be used by VMWare and VirtualBox also. – Ramhound Feb 04 '18 at 13:45
  • @Ramhound Can it be done with VirtualBox or VMware? That article only mention qemu. – Biswapriyo Mar 28 '18 at 10:08
  • @Biswapriyo I determined that VMWare and VirtualBox was possible directly from the documentation. However, VMWare and VirtualBox would have to make the necessary changes to their application – Ramhound Mar 28 '18 at 11:25
  • Can you explain how this is different from Nested VT-x/AMD-V? – user643011 Aug 11 '19 at 21:21
  • 2
    Virtualbox 6.0' WHPX support is very alpha quality. It barely works right now. – user643011 Aug 11 '19 at 21:41
  • 1
    Tried Virturalbox 6.0 with the vms created before, and changed the setttings to use Hyper-V, none work as expected. – Hantsy Nov 11 '19 at 09:35
  • Note, in my understanding of the available documentation, Windows Hypervisor Platform API allows host software like QEmu, VirtualBox, VMWare to manage Hyper-V guests. That means while it may appear these program work while Hyper-V is enabled, they are actually using Microsoft's hypervisor implementation. Host software can still define their own protocol meaning file-sharing, drag&drop still work. But certain hypervisor level feature like graphics acceleration depend on Hyper-V. That is a regression for certain features where VMWare, VirtualBox hypervisor implementated but Hyper-V didn't. – 1283822 Dec 12 '19 at 19:59
  • @user643011: How does it work at *all*? I get `VERR_NEM_MISSING_KERNEL_API` with `VERR_VMX_NO_VMX`. – user541686 Dec 15 '19 at 18:17
  • @Mehrdad Force-Upgrade to Windows 10 November 2019 Update (1909) and try again https://en.wikipedia.org/wiki/Windows_10#Updates_and_support You can get the update here: https://www.microsoft.com/en-us/software-download/windows10 – user643011 Dec 17 '19 at 05:31
  • @user643011: I already have actually; I wrote that comment afterward. – user541686 Dec 17 '19 at 05:34
  • @Mehrdad: Ok sorry I don't know either. Support is very hit and miss for me. I have complained in the Virtualbox forums before, but their devs just work as fast as they work and I can't really complain on an open source software. I wish Microsoft would have just allowed us to enable https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/nested-virtualization globally so that other VMs can run on top, rather than force VMware and Virtualbox to adopt to Microsoft's hypervisor backend via Windows Hypervisor Platform API. It's a mess. – user643011 Dec 17 '19 at 05:38
2

Vmware 15.5+ will work even if Hyper-v is enabled

With the release of VMware Workstation/Player 15.5.5, we are very excited and proud to announce support for Windows hosts with Hyper-V mode enabled! As you may know, this is a joint project from both Microsoft and VMware.

Abdelouahab
  • 121
  • 3
1

Here's what worked for me.

You need to check to make sure that Hyper-V is not running.

The telltale test is running "systeminfo" from the command prompt and if you see ‘A hypervisor has been detected. Features required for Hyper-V will not be displayed’ - then it's still not disabled.

Steps to disable

  1. Get into your machine's BIOS, typically UEFI today, and disable TPM ! Otherwise none of the steps below will work!!

Boot computer using F2 into the BIOS setup mode ( alternate keys include 'Enter' or 'Delete'). Alternately use the Windows method to get there using the Settings app ( reference : https://www.maketecheasier.com/boot-uefi-from-windows10/ )

Once inside the UEFI BIOS : Locate the “Security” option and expand Locate the “TPM” option nested under the “Security” setting To clear the TPM you must check the box saying: “Clear” to clear the TPM , or "Disable" depending on your hardware manufacturer

Save these settings in the UEFI interface, the BIOS will likely ask you to reboot ( follow these steps )

Boot normally ... you probably won't see any difference.

Check : systeminfo It should still complain about Hyper-V

  1. Open Powershell in administrator mode, enter: Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-Hypervisor -All

wait until it completes ... this will tell you the action is complete, not that the action was successful.

  1. In powershell also enter: DISM /Online /Disable-Feature:Microsoft-Hyper-V

wait until it completes ... this will tell you the action is complete, not that the action was successful.

check systeminfo again -- Hyper-V is proably still in the way .. keep going.

  1. Disable Windows Credential Guard This appears to start Hyper-V regardless of settings.

The steps to do this are in the link provided, but I'll pull the salient ones:

https://docs.microsoft.com/en-us/windows/security/identity-protection/credential-guard/credential-guard-manage

Open the app "Local Group Policy Editor" from Microsoft, it's already part of the OS.

From the Group Policy Management Console, go to Computer Configuration -> Administrative Templates -> System -> Device Guard.

Change "Turn On Virtualization Based Security" and set this to "Disabled".

  1. Now, restart by going through the typical windows shutdown process.

  2. Once logged in again, run systeminfo , it SHOULD show you

Something like this now :

Hyper-V Requirements: VM Monitor Mode Extensions: Yes Virtualization Enabled In Firmware: Yes Second Level Address Translation: Yes Data Execution Prevention Available: Yes

This means that the VT-x CPU extensions are not stolen by Hyper-V, and Type-2 Virtualization engines like VirtualBox can use VT-x directly to dramatically speed things up.

I anticipate that other VM technologies will benefit from this as well, Hyper-V stealing the VT-x interface from other VM technologies almost guarantees that Windows platforms will NOT be used as a host OS as time progresses.

This is incredibly difficult for the average user to disable and unfortunately really discourages anyone using VM technology on windows because any virtualization runs quite slowly.

Good luck. Add instructions if you find this needs some more detail.

beadon
  • 11
  • 1