Grundlagenartikel Threadripper/X399 ESXi GPU-Passthrough

Besterino

Urgestein
Mitglied seit
Jul 22, 2018
Beiträge
1.470
Punkte
82
Alright, I finally figured out how to get create a virtuale machine (VM) on ESXi to play games with as if you had installed windows "bare metal" on the machine, i.e. without a hypervisor between your gaming OS and the hardware on my Threadripper system [and with ESXi 6.7U3]. In comparison, Intel is a piece of cake. On intel everything basically "just works" with ESXi 6.5U2. AMD, however, is something else... and ESXi 6.7U3 as well...

Hardware used:
Mainboard: ASRock Rack X399D8A-2T
CPU: Threadripper 1920X
RAM: 64GB non-ECC (to be replaced with ECC RAM if the machine becomes a 24/7 server)

Preparation:
Check Device ID's of all your Nvidia-Devices (vendor ID: 10de) related to the GPU. They appear in your hardware list grouped together and starting with the same PCIe-address. We are looking for the 4-digit-code behind "0x" like this:
1511
Write them down for each nvidia-Device.

BIOS Settings:
[to come]

Basic ESXi Host Settings:
1. Enable SSH

2. Connect with SSH with your favorite SSH-Client (e.g. putty)

3. type:
Code :
 vi /etc/vmware/passthru.map
4. navigate to the bottom to the #Nvidia section, comment-out the existing entry ("10de ffff...") and enter new ones for all devices EXCEPT the GPU (I have included the GPU-lines only for demonstration purposes). Also add a new section for AMD for the USB-Controller you actually want to use for keyboard, mouse etc. To edit the file press "i"):
Code :
# NVIDIA
#10de  ffff  bridge   false
#1 GPU - CAN BE IGNORED, FOR DEMONSTRATION ONLY
#10de  1e07  link   false
#2 Audio - not sure yet, whether really necessary
10de  10f7  d3d0   false
#3 Serial Bus - not sure yet, whether really necessary
10de  1ad7  d3d0   false
#4 USB - THIS IS A MUST HAVE (change device ID - second column - depending on your own ID)
10de  1ad6  d3d0   false
# AMD - THIS IS A MUST HAVE
1022  ffff  d3d0   false
(if you don't do this, you will get yellow exclamation marks on the respective device after reboot of the VM)

5. after you successfully made the changes press the escape-key followed by ":w" --> ENTER and ":q" --> ENTER

6. if something goes wrong in vi, press escape a couple of times and then ":q!", which will close vi discarding all changes (do not type ":w" if you're not sure your changes are correct!)

ESXi VM-Settings
1. Start options: BIOS or EFI - it doesn't matter, works both.

2. attach all Nvidia devices coming with your GPU to your Windows-VM.
1509

3. SAVE the VM-Settings and open the VM-Settings again.

4. Edit advanced settings and add parameters:
4.1 hypervisor.cpuid.v0 = FALSE
(if you don't add this, you will always get a yellow exclamation mark / "error 43" for the GPU in device manager)

4.2 pcipassthrou[X].msiEnabled = FALSE for ALL Nvidia devices coming with your GPU EXCEPT (!) the USB-Controller, replace [X] by the number for the respective device (note: they are numbered in the same order as you added the devices to the VM
1510


Everything else is business as usual. Install vmware tools, drivers etc. I suggest, you add the Nvidia-devices to the VM only after you did everthing else.

For the sake of completeness:

vmware tools
After you have installed a fresh Windows 10 in the VM, do install the vmware tools. They are necessary to get the vmxnet3 network adapter working and provide some other helpful functionalities.

1. On the "VM Window" you click on "Actions"
2. select guest system (in German: Gastbetriebssystem) --> "install vmware tools"
3. navigate to the new CD/DVD drive recognized by Windows
4. doubleclick or open it and click on "setup64.exe".
5. select "complete"
6. Reboot

In the meantime we can also download the nvidia driver for our GPU. Side note: for a 2080Ti we need to select the DCH type, not standard.

1512


Cosmetics
If you are sure everything works (i.e. you see the windows desktop on the monitor directly connected to the GPU and the core USB-Devices work as well) & still works after rebooting the VM, feel free to disable the vmware virtual gpu in the advanced settings, i.e. svga.present = FALSE. In case something goes wrong with your passthrough-GPU, you need to set the parameter to "TRUE" again.
 
Zuletzt bearbeitet :
Oben Unten