Unable to use HPA >= 4GB when vTLB is active

[udosteinberg]

This is not really an NRE bug, but something that NRE needs to be aware of, since we have seen 64-bit NRE use high addresses for VMs, which has exposed the problem...

For 32-bit guests, the vTLB uses a 2-level shadow page table with 4-byte PTEs. Even though a 64-bit VMM can install a GPA-to-HPA mapping where HPA >= 4GB, the vTLB cannot store HPA wider than 32-bit in its shadow PTEs. The most recent version of the microhypervisor catches such cases and terminates the vCPU.

To be able to make use of HPA beyond 4GB when the vTLB is active, the microhypervisor would have to use a 3-level PAE shadow page table with 8-byte PTEs, which has performance implications.

For the time being, NRE should avoid using HPA >= 4GB for VMs that use the vTLB.

The vTLB is used when "vtlb" is specified on the hypervisor command line. Otherwise it is used...

• always, for CPUs before Nehalem
• when the guest runs without paging, for CPUs before Westmere
• never, for Westmere and newer

[hrniels]

Ok, thanks for the notice and the explanation. I'll take a look at that if I have time.

[udosteinberg]

A patch for this is in the vtlb branch of my tree. @Nils-TUD @parthy feel free to give this a spin. You should boot the hypervisor with "vtlb" parameter to force vTLB operation and map some host-physical region beyond 4GB into the guest (obviously you'd need a 64bit host system for that). Note that this causes 3-level PAE paging to be used instead of 2-level paging, so I would expect some performance impact. It would be good to run some benchmark with vTLB enabled, once with and once without the patch to see how much overhead we add for this.

[parthy]

I ran a single-core kernel compile today with vanilla NRE (without the hypervisor changes) and the two vTLB versions. I got 522.4s for the old version and 562.6s for the one. That would be around 7.6% overhead introduced by the patch. It would be interesting to see some other machine's results, too.

[blitz]

What machine are you using? If you configure a boot entry on erwin, I can test it on Ivy Bridge.

[parthy]

I'm on a SandyBridge. I'm not sure I can provide such a boot entry easily.. I just build NRE with HAVE_KERNEL_EXTENSIONS set to 0 and change the pulsar config it generates when building boot/vancouver-kernelbuild.wv to use the official NOVA hypervisor binary. In NOVA, I change the aux field to be an mword in include/hip.hpp and build it as 64bit binary, once in master, once in the vtlb branch. I'm sure you will get this running a lot faster than me trying to figure out how to set up this boot entry and copying all the stuff over (which I can do only from home, in the evening).

[udosteinberg]

@blitz - it would be more interesting to run this on an old machine, because for the new ones with EPT, you wouldn't want to use vTLB anyway. I'd be more interested in performance numbers for P4, Yonah, Merom, Penryn machines. AFAIR Carsten had a P4-based Presler machine (CPUID F:6:2), where vTLB would be mandatory. Nonetheless, performance numbers for newer machines are welcome as well, to get a wider picture.

[blitz]

@udosteinberg I'll see what I can find. @parthy Can you mail me the pulsar config and all the binaries? (Yes, I am lazy.)

[Nils-TUD]

I've just tested it with my i5:

 [ 0] CORE:0:0:0 6:25:5:1 [2] Intel(R) Core(TM) i5 CPU         650  @ 3.20GHz

The old version needs 615s and the new version 656s. That's an overhead of 6.67%.

[blitz]

So far I only got it to work on my laptop (i7 L640: 770s -> 819s 6%). For vtlb-only boxes, I could only find a Pentium D, but I couldn't get the benchmark to run right away. Working on it.

[udosteinberg]

Based on the reported numbers so far, overhead seems to be in the 6-8 percent range. Because that is significant, I would rather not enable PAE for the vTLB by default. We could make it a compile-time option. What do others think?

[blitz]

Is it possible to make it a runtime option?

[udosteinberg]

Sure, but it would inflate the binary because the code would have to include both vTLB versions. We could then even go as far as using the 2-level vTLB for VMs with HPA below 4GB and the 3-level vTLB for those with HPA above 4GB. I'm reluctant to do that because it results in non-deterministic VM performance for all sorts of benchmarks.

[blitz]

On 08/03/2013 01:51 AM, Udo Steinberg wrote:

Sure, but it would inflate the binary because the code would have to
include both vTLB versions.

It would be nice, if NOVA Just Works™ even on old boxes. Maybe have the
slow (but working) variant be the default and print a big warning that
you can switch to a faster version with restrictions with a compiler switch?

Julian

[parthy]

It's not that you can't use NOVA on those machines ;) You just cannot assign memory beyond 4GiB (physical) to the guest. It is a limitation that you can circumvent, but not a killer. So to me it would make sense to do it the other way round: Only have a special case if you want to compile NOVA for the use on such an older machine with >4GiB RAM. Maybe you could have a compile switch saying "build and use both both vTLB versions"? With that, the default version advocated by the README would still be deterministic in this sense, but if you choose to overcome the limitation, you get to use higher addresses and only pay the performance overhead where necessary, but have to compile the kernel differently.

[blitz]

It only works if a) the box has no memory beyond 4G or b) the userspace is aware of this limitation and will not use memory beyond 4G for VMs.