Hyper-V #0x0 - Research setup

There has been some interest and questions about starting Hyper-V research, so I thought of writing little about it (also, it's been a long time since the last blog post). I will probably do it in quite small pieces.

Also, I will hopefully be covering the topic at INFILTRATE (https://infiltratecon.com/) in October, so I will not go deep into my research for now. But I hope to cover a lot of things needed to get going for anyone starting.

I will start with the most straightforward topic - my research environment and how to do the most simple stuff when starting. This information is available in MULTIPLE other places, but since my goal is to at some point have entire series on Hyper-V, then I think it makes sense to start from the very very basics.

My setup

My main machine is running Linux, so all my Windows-based tools and targets are running on virtual machines. I use VMWare Workstation and I'm quite happy with that. In the past, I used Virtualbox, but things really seem to work better and faster with VMWare + I can easily use the same tool if I need to connect to some ESX or ESXi servers. It's also effortless to set up connections between VMs for kernel/hypervisor debugging purposes via emulated serial ports or virtual networks (it's not related to VMWare - all other main virtualization tools also make it easy).

For Hyper-V research, I use 3 VMs:

1. Debugging machine that contains Windbg, IDA, and development tools. 
2. Target machine with the latest Windows build and everything set up so it could be debugged.
3. Target machine with the latest Windows Slow ring build and everything set up so it could be debugged. "Why?" you might ask. Mainly because Microsoft has such sentence in Hyper-V bug bounty page "If you are submitting a vulnerability for Hyper-V on Windows 10, then the vulnerability must reproduce on the recent WIP slow builds to qualify for a bounty" [https://www.microsoft.com/en-us/msrc/bounty-hyper-v]

Setting everything up

I'm going to explain all based on multiple VMs & VMWare Workstation point of view. But almost the same approach works when using some other virtualization tool. Also, when your main OS in Windows, then you don't need separate debugging VM but can use host OS for that (quite a bit faster too).

First things

1. Install Windows on debugging VM and Windbg
2. Install Windows on target VM (I would start with only one VM)
3. Update all that is needed & install any other thing you like
4. In the virtualization tool, enable Intel VT-x/EPT (or AMD-V/RVI when running on AMD machine) and IOMMU virtualization on the target machine. This is needed to actually run Hyper-V. On VMWare, it's on VM settings Hardware tab under "Processors" config.
5. Since you almost certainly want to debug Hypervisor AND Kernel, then you need two different channels for it, so add serial port between VMs and virtual network.
6. Activate Hyper-V on target VM via "Windows features"

Target VM conf

Because I use a lot of snapshotting on my target machine, I prefer to use serial-based communication. When using NET-based, then it's a lot faster, but there is some session management in NET-based connections. Because of that, I have had a lot of problems when Windbg and target can't talk after the VM revert. Also, the NET-based link has had stability problems.
But in the case of the hypervisor, the serial-based communication seems not to work. So I usually use serial for kernel debugging and NET for the hypervisor.

Setting debugger for the kernel (over serial COM 1):

 bcdedit /dbgsettings serial debugport:1 baudrate:115200
 bcdedit /debug on

Setting debugger for the hypervisor (over the network - write up the code generated by the first command):

 bcdedit /hypervisorsettings NET HOSTIP:???.???.???.??? PORT:50000
 bcdedit /set hypervisordebug on
 bcdedit /set hypervisorlaunchtype auto

Debuggers running

When now setting 2 windbg sessions for each connection, you should see something like this:

If hypervisor connection does not happen then some things to check:

• The firewall does not block windbg from opening the port
• Both VMs can actually reach each other over the network (maybe the virtual network is configured in a way that VMs can't access other VMs, etc.)
• That Hyper-V is actually installed. If you installed Hyper-V in "Windows features" BEFORE enabling VMs virtualization features, then Windows only installs the Hyper-V client but not the hypervisor.

Setup works

If both debuggers get the connection, then you have everything running. You can run some simple commands on the hypervisor debugger to test it. For example, ?hv to get the base address for the hypervisor.

Now from that moment on, things get a bit trickier with the hypervisor. Microsoft has not made hypervisor debug symbols public, and also, the hvexts.dll extension for windbg is available only to MS partners. So from now on, all stuff in hypervisor you have to find by reversing the code. Luckily this is not such a big piece of code as the kernel itself, and a lot of kernel side components (used to talk to the hypervisor or via the hypervisor) do have symbol information.

I will quickly show one example with the hypercalls that I will cover more deeply in the next blog post. 

Hypercall from the kernel side

Hypercalls are made by the kernel via location defined by the pointer at nt!HvcallCodeVa. This is likely because opcodes for switching to hypervisor are different between Intel and AMD. Now if you take a look of the code referenced by that location, the result is such:

1: kd> u poi(nt!HvcallCodeVa) L18

fffff804`0cdf0000 0f01c1          vmcall

fffff804`0cdf0003 c3              ret

fffff804`0cdf0004 8bc8            mov     ecx,eax

fffff804`0cdf0006 b811000000      mov     eax,11h

fffff804`0cdf000b 0f01c1          vmcall

fffff804`0cdf000e c3              ret

fffff804`0cdf000f 488bc1          mov     rax,rcx

fffff804`0cdf0012 48c7c111000000  mov     rcx,11h

fffff804`0cdf0019 0f01c1          vmcall

fffff804`0cdf001c c3              ret

fffff804`0cdf001d 8bc8            mov     ecx,eax

fffff804`0cdf001f b812000000      mov     eax,12h

fffff804`0cdf0024 0f01c1          vmcall

fffff804`0cdf0027 c3              ret

fffff804`0cdf0028 488bc1          mov     rax,rcx

fffff804`0cdf002b 48c7c112000000  mov     rcx,12h

fffff804`0cdf0032 0f01c1          vmcall

fffff804`0cdf0035 c3              ret

In my case, the CPU is Intel one, so the operation for switching is vmcall. You can see multiple different vmcalls here and this will be covered in the future. What is important is that when some piece of code wants to make a hypercall, it almost always does it via such a trampoline. Also, the nt!HvcallInitiateHypercall function is often used (not always).

To intercept such calls, you want to stop execution right at the vmcall operation, so the first reaction is to use bp poi(nt!HvcallCodeVa) command on kernel debugger. But this would be a mistake because the kernel debugger also runs under the control of the hypervisor and this area of code is protected by the hypervisor. So you can't write there anything, not even 0xCC byte :)

The solution is luckily easy - the hardware breakpoints are still allowed

ba e 1 poi(nt!HvcallCodeVa)
or (depending on where you want to break)

ba r 8 nt!HvcallCodeVa

Now you can break and study how and from where such calls are made. For the overall structure and an excellent description of how to do hypercalls with a custom driver, I would highly recommend Alex Ionescu's blog posts about the Hyper-V bridge. But as of now, it seems that the blog is down. I do hope it gets back up soon and will contain that post :)

If not, I will add that information to my own hypercall post, but I don't think that I could explain it as well.

Hypercall from hypervisor side

Hypercalls are number-based, as are the syscalls. This number, in turn, will end up used as an index for an array of 0x18 byte sized elements located at hv+0xC00000. The address of the handler functions is at the first 8 bytes of each element. So, for example, to break in the hypercall 0xB handling function you could run such command in the hypervisor debugger: bp poi(hv+0xC00000+0xb*0x18)
This hypercall happens very often and should be good for testing.

This was very basic info on how to track hypercalls from both kernel and hypervisor sides. I hope it helps anyone starting on this topic.

This is all for now, and I will try to follow this post soon with the post where I cover hypercalls a lot more deeply.