Linux with UP2715K (or T221 or any other composite monitor), Xorg, Xinera, XRandR

It is most unfortunate that the best monitors of any era are always a little quirky. I used an IBM T221 for over a decade as my main work and play monitor with it’s, for the time (they came out in 2003!), staggering 3840×2400 (4K+) resolution. But the fact that it had to pretend to be two separate monitors to shift enough pixels to achieve such high resolutions at a decent refresh rate always caused problems, especially on Linux.

The much younger Dell UP2715K also requires two separate DisplayPort inputs to achieve the full 5K (5120×2880) resolution.

The problem is that this kind of a setup is very unusual. I am aware of only the two mentioned monitors that use this kind of input arrangement. Consequently, most developers of desktop environments and GPU drivers seemingly couldn’t care less about supporting it. I covered how to deal with this with a T221 monitor in the past using both Nvidia drivers (which have built in option to lie to the desktop environment about the monitor geometry) and AMD drivers (which require use of the fakexinerama LD_PRELOAD library).

The problem is that both of these workarounds require Xinerama support, and this is mostly being deprecated and removed. Modern Xorg, for example, by default has Xinerama disabled and XRandR (as a more modern replacement) enabled. Unfortunately, Nvidia drivers don’t seem to have the option to fake the RandR geometry data.

So, to make this work, we need to pass Xorg options that are not passable purely via xorg.conf. The trick we used years ago with T221 on Linux no longer work. How do we do this? Modern EL7/EL8/EL9 ship with a shell script for /usr/bin/Xorg:

#!/usr/bin/sh
#
# Execute Xorg.wrap if it exists otherwise execute Xorg directly.
# This allows distros to put the suid wrapper in a separate package.

basedir=/usr/libexec
if [ -x "$basedir"/Xorg.wrap ]; then
	exec "$basedir"/Xorg.wrap "$@"
else
	exec "$basedir"/Xorg "$@"
fi

We can modify this to additionally pass additional parameters: -extension RANDR +xinerama

But it turns out that as of EL9, this breaks gdm login. So we have to run gdm without these extra Xorg options, but for the real user session we have to add those options in so that both gdm and KDE plasma work. Thankfully, there is an easy way to detect whether Xorg is running for gdm login or a real user because gdm login runs as the gdm user, but a user session runs as that user. So to make it all work, we modify the Xorg shell script wrapper as follows:

#!/bin/sh
#
# Execute Xorg.wrap if it exists otherwise execute Xorg directly.
# This allows distros to put the suid wrapper in a separate package.

basedir=/usr/libexec

USER=$(whoami)
EXTRA=""

if [ "$USER" != "gdm" ]; then
	EXTRA="-extension RANDR +xinerama"
fi

if [ -x "$basedir"/Xorg.wrap ]; then
	exec "$basedir"/Xorg.wrap $EXTRA "$@"
else
	exec "$basedir"/Xorg $EXTRA "$@"
fi

Combine that with a xorg.conf fragment similar to what we used before with the T221, and we have a fully working GDM login with KDE plasma desktop. All the benefits of fantastic image quality, and none of the compromises.

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "GeForce GTX 1080Ti"
    Option         "NoLogo" "1"
    Option         "TripleBuffer" "True"
EndSection

Section "Screen"
	Identifier     "Screen0"
	Device         "Device0"
	Monitor        "Monitor0"
	DefaultDepth    24
	Option	"Coolbits" "28"
	Option	"UseEdidDpi" "False"
	Option	"DPI" "96 x 96"
	Option	"TwinView" "True"
	Option	"ConnectedMonitor" "DP-0, DP-2"
	Option	"NoTwinViewXineramaInfo" "True"
	Option	"nvidiaXineramaInfo" "False"
	Option	"MetaModes" "DP-2:2560x2880 +0+0, DP-0:2560x2880 +2560+0"
	SubSection     "Display"
		Depth	24
	EndSubSection
EndSection

Obviously these options require an Nvidia GPU driver. For AMD GPUs you will have to apply a different workaround.

Virtually Gaming, Part 1: In the Beginning – Hardware and Xen

For about two years now I have managed to stick to the “No Windows on bare metal.” policy. This was instated for many reasons, including security and ease of backups (it is difficult to beat ZFS snapshots and send/receive functionality). The key reason for using Windows at all has been gaming, and both myself and my wife do play various games, mostly of the co-op FPS genre. While native Linux support has increased dramatically in that time, the availability of native Linux games still hasn’t quite reached parity with availability on the Windows platform.

Combining the “No Windows on bare metal.” policy with the requirement for high performance gaming capability meant that the only solution that fits is PCI passthrough of a high end GPU to the virtual machine. In this article I will describe the journey to the solution over the past two years, including (often unfortunate) choices of hardware, software, working around hardware, firmware, driver and software bugs, crippling and limitations, and other bumps on the road to virtualized gaming.

Hardware

When I first embarked on this project, it was an off-shoot of the project to upgrade my workstation. While there was nothing wrong with my Quad Core 2 in terms of performance, I needed to get a second machine up and running for my wife. So, somewhat optimistically, I thought this would be an ideal opportunity to solve three problems at the same time:

  1. Get a gaming grade workstation up and running for my wife
  2. Virtualize the Windows part of my dual-boot setup so I never have to reboot for the sake of joining a game when my friends invite me
  3. Implement the “No Windows on bare metal.” policy

The motherboard that caught my eye was the EVGA SR-2. It seemed to fit all of the necessary requirements:

  1. Plenty of CPU power (dual socket, capable of taking up to two 6-core Xeons)
  2. Full support for plenty of ECC memory (after the last build I have vowed to never build another machine without ECC RAM, having spent days troubleshooting a stock-setting stability issue that turned out to be marginal memory)
  3. Plenty of PCIe slots (7 x16 slots, with 64 usable PCIe lanes between them)
  4. VT-d support (originally listed in the spec, and confirmed with EVGA tech support prior to purchase – a claim that turned out to be rather stretching the truth)

A sizable investment into the motherboard, a pair of 6-core X5650 Xeons, and 48GB (6x8GB) of registered ECC RAM later, problems began.

Hardware Problems

The first motherboard I got turned out to have a faulty PCIe slot #1. The retailer I bought the motherboard from went bust a few weeks after my purchase, but EVGA generally have excellent RMA service, and I registered the motherboard as soon as I had received it to qualify for the full 10 year manufacturer’s warranty that was offered on this motherboard.

In order to not put my build on hold, before I RMA-ed the faulty SR-2, I bought another, second hand SR-2 on eBay. I thoroughly tested it, and to this day, this is the SR-2 that has been completely fault free in the main workstation that was the product of this project. It turns out, I was quite lucky to have bought a second motherboard – because the replacement that was sent was also faulty, and failed to reliably finish POST-ing with either of my CPUs in either socket. That got RMA-ed as well, and the replacement is currently in use as a prototyping rig for the next incarnation of this workstation, but that motherboard also has problems which cause it to fail to boot on a hot reboot (I am putting off RMA-ing it until the prototyping stage of the project is completed and being without a working prototyping machine for a week won’t be a problem.

In conclusion: Beware EVGA warranty replacement motherboards – they are all refurbished items that were sent back as faulty, and either repaired or the fault was never reproducible by their testing team so they got recycled as is. Always test any refurbished replacements extremely thoroughly (all slots, sockets, ports and features) when you receive them – if you get a faulty replacement, EVGA will pay for the shipping costs back to them for another replacement, but only within the first month after you receive the replacement, so acting quickly and thoroughly is of vital importance to avoid courier costs that can quickly add up to a lot.

More RAM

At this time I looked into using 96GB of RAM on the SR-2. This turned out to be very difficult as the machine would generally refuse to POST, except after a fresh CMOS reset. This was particularly annoying because the CPUs themselves (which contain the MCH) officially support 192GB of RAM each. After a lot of trial and error, I found a way to make the machine reliably post with 96GB of RAM:

  1. Use dual-ranked (this is important, single ranked won’t work for 96GB!) x4 registered 1600MHz 1.35V DIMMs
  2. Boot the machine with only 6 DIMMs. Go to the memory settings, and manually set all of the memory timings to what they defaulted to. Make sure you set the command rate to 2T (defaults to 1T).
  3. If you are overclocking, make sure you set the MCH strap to 1600MHz.

Do this and your SR-2 should POST with 96GB. It may require a few attempts where the motherboard re-sets itself and re-attempts the POST, but both of mine successfully POST within 30 seconds.

All of the symptoms indicate that there is a BIOS bug in timeouts at various stages of the POST that cause some initializations to fail and time out when more than 48GB of RAM is used. Officially, EVGA only claim the SR-2 supports up to 48GB of RAM, and it is unlikely they will be fixing this BIOS bug.

Hypervisor

Back when I began this project (late 2012), the only hypervisor with notable reports of GPU passthrough success without requiring a lot of manually applied experimental patches was Xen, so this was what I chose for the project. Additionally, my previous tests indicated that the performance overheads of using Xen were among the lowest of all the available hypervisors, so it seemed like a win-win situation.

The primary GPU in the machine was an ageing but perfectly adequate GeForce 8800GT that came from my previous workstation. Then I had to select a suitable GPU for passthrough to a virtual machine. Nvidia passthrough only worked on expensive Quadro (and not all Quadros, only the expensive ones), Tesla and Grid cards which they refer to as “MultiOS compatible”. The cost of most of those made them not an option worth considering. That meant trying an ATI card, so I got a cheap passively cooled single-slot Radeon HD6450. This is where a whole array of real problems began:

  1. EVGA SR-2 motherboard uses a pair of NF200 PCIe bridges to multiplex 32 PCIe lanes available on the upstream Intel 5520 PCIe hub into 64 PCIe lanes available for GPUs. NF200 bridges have severe bugs and limitations when it comes to compatibility with VT-d. They bypass IOMMU for DMA transfers, so when the VM tries to access RAM within it’s virtual address space that overlaps the physical address of a PCI BAR (aperture) that belongs to a hardware device, the memory writes will hit the BAR, which will crash the machine (and maybe corrupt your disks, if the BAR being trampled belongs to a disk controller). The solution to this was to write a hvmloader patch that marked all of the IOMEM areas from the host as reserved. This was an ugly bodge that resulted in a fair amount of memory in the domU (what Xen calls guest VMs) becoming unusable, but it worked (and with enough RAM it wasn’t a major problem).
  2. More than likely related to point 1, this motherboard appears to have broken (or non-existent) support for interrupt remapping, which means that any devices passed to a VM have to have dedicated, unshared interrupts. If you pass a device sharing an interrupt to the VM, the VM will most likely crash the entire host. Problems 1 and 2 are very similar in symptoms (host crash), which made them quite difficult to troubleshoot and get to the bottom of because no one change to the configuration made the problem go away. It took some help from the Xen developers and a fair amount of guesswork to figure it all out. The only solution is to move cards around to different slots until all of the hardware you intend to pass through to virtual machines has dedicated interrupts that aren’t shared with other hardware. This can be fiddly, but it is generally achievable – in my final configuration, I am successfully passing two GPUs and three USB controllers to VMs.
  3. ATI cards suffer from terrible drivers that fail to re-initialize the card without full BIOS level re-POST-ing (and said re-POST-ing doesn’t happen when the VM is rebooted, only when the entire physical machine is rebooted). The consequence is that they work OK when the VM is first booted up after a host reboot, but subsequent VM reboots result in massive performance degradation, glitches, and sometimes complete host crashes. While some of this is being worked on (e.g. functionality to reset the GPU via a bus reset from Xen dom0), it is still not available in the current released version. This particular problem turned out to not be easily solvable (having already written a patch for Xen’s hvmloader, I was very keen to avoid having to write any more to implement PCI bus resetting functionality for the Xen pci-stub driver. To at least be able to prove the concept, I bought the cheapest Nvidia Quadro that is supported for GPU passthrough (Quadro 2000), and this worked absolutely fine. Having finally found a solution that works perfectly, I went on to find ways of making GeForce cards work with PCI passthrough through fooling the Nvidia driver into initializing them even though they weren’t expensive enough, by modifying the cards’ ID number into an equivalent Quadro card. As discussed in previous articles, Nvidia cards up to and including the Fermi generation can be modified into equivalent Quadro cards by changing the appropriate ID strap bits in the cards’ BIOS using nvflash. Kepler cards require a small hardware modification. The easiest modifications are GTX 680 to Tesla K10 (remove one resistor) and GTX780Ti to Quadro K6000 (add one large, easy to solder resistor across appropriate pins on the EEPROM). I am currently running a pair of GTX 780Ti cards.

Issues 1 and 2 listed above are why I said that claiming the SR-2 supports VT-d was seriously stretching the truth. On a well designed workstation motherboard, the above problems should never have arisen. After all that, and many, many man-days invested in it working around the various bugs mentioned above, I have Xen working on the system, with EL6 (CentOS) dom0, and two domUs, one running XP x64 and one running Windows 7 x64. The hardware passed through on PCIe level is:

XP x64:

  • Intel ICH10 HD Audio
  • 2x ICH10 USB
  • GeForce GTX 780Ti

Windows 7 x64:

  • NEC USB 3 controller
  • GeForce GTX 780Ti

GRUB options:

 kernel /xen.gz noreboot unrestricted_guest=1 msi=1
 module <kernel and options> intel_iommu=on pcie_ports=compat

Note that unrestricted_guest=1 and pcie_ports=compat are required on the SR-2, but may not be required if you hardware behaves better. If your IOMMU implementation is good and includes ACS functionality, you shouldn’t need unrestricted_guest=1.

pcie_ports=compat is required because without it the SR-2 makes the PCI hotplug driver flap very quickly on one of the PCI devices built into the south bridge chipset, which causes an interrupt flood that makes the machine grind to a halt. (Have I mention enough times yet that the SR-2 is extremely buggy?)

Xen domU config:

name="mydomu"
description="None"
uuid="a57e6840-e9f5-4a14-a822-abcdef012345"
memory=16384
maxmem=16384
vcpus=6
on_poweroff="destroy"
on_reboot="restart"
on_crash="destroy"
localtime=1
keymap="en-gb"

builder="hvm"
device_model_override="/usr/lib/xen/bin/qemu-dm"
device_model_version="qemu-xen-traditional"
boot="c"
disk=[ '/dev/zvol/ssd/mydomu,raw,hda,rw', '/dev/sr0,raw,hdc:cdrom,rw' ]
vif=[ 'mac=00:11:22:33:44:55,bridge=br0,model=e1000', ]
stdvga=1
usb=1
acpi=1
apic=1
pae=1
gfx_passthru=0
pci = [ '07:00.0', '07:00.1', '00:1b.0', '00:1a.1' ]
xen_platform_pci=1
pci_msitranslate=0
pci_power_mgmt=1

Obviously you will need to change things like PCI addresses, MAC addresses, block device paths, and suchlike to suit your own system.

/etc/modprobe.d/xen-pciback.conf:

options xen-pciback permissive=1 hide=(07:00.0)(07:00.1)(00:1b.0)(00:1a.1)

Note the PCI IDs in the xen-pciback module options correspond to the PCI IDs in the Xen domU configuration. You may not need permissive=1 if you have better hardware than I do.

And Another Thing

One thing I feel I have to mention is that I have had extremely bad experience with every SAS card I have tried to use in the SR-2 with virtualization. This includes two different LSI cards, an Adaptec card and a 3ware card. They all work fine in a normal bare metal setup, and cause all kinds of crash inducing problems, some more difficult to debug than others when IOMMU is enabled and VMs are running with PCI devices passed through to them. SATA cards (I tried Silicon Image and Marvell), OTOH, seem to always work just fine, with no problems whatsoever, including when using 1:5 SATA port multipliers. In some cases this is caused by the SAS controller being a native PCI-X chip and using a phantom PCI-X to PCIe bridge. In other cases it seems to be caused by the SAS card’s driver trying to do some interesting DMA accesses that crash the entire host when virtual machines are running with PCI devices passed through to them. In short – avoid using SAS cards and stick with SATA – but then again I find that to be good advice to follow regardless.

This setup has worked without any significant problems for the past two years. But things have changed in that time. There is now a native Linux version of Steam, and many games have native Linux ports. It is time that this long term reliable system is updated accordingly. More on that in the next article.

Chromebook Pixel – Long Term Review

I have been using my Chromebook Pixel for nearly a year now, so I feel it has been long enough to form a reasonably objective view, which may be useful to others who are considering buying one.

When I bought it, I was looking for a worthy successor to my venerable ThinkPad T60. The ThinkPad had been upgraded as far as it would go, with a 2.33GHz Core 2 Duo, 3GB of RAM, and most importantly, a 2048×1536 screen. It is still quite a usable machine, but the main reasons why I was looking for a replacement were battery life (90 minutes with the extended capacity battery on a good day), and weight (I haven’t weighed it, but when carrying it around for any length of time it feels like it weighs a tonne. All in all, barely livable with for the commute to work.

The Pixel was promising to address all the issues I had with the ThinkPad – it weighs a fraction as much, the battery life is about 6 hours, depending on the load, the other features are no worse, with the screen being a significant improvement on the ThinkPad. Since I use Linux (EL6), I needed to make sure all of the hardware is fully supported, which was the main reason why I didn’t choose a Macbook Pro Retina – the only other contender at the time.

Needless to say, ChromeOS only lasted for long enough to enable developer mode to facilitate installing a proper Linux distribution.

How did this very promising spec on paper work out in reality? Well, my experience is very mixed. The performance is more than sufficient, even for light gaming loads (e.g. Left 4 Dead 2 with maxed out settings at 1280×800, quarter of native resolution). The screen is nothing short of amazing. The touchpad is reliable. The battery life is good. But that is where the good things I can say about it end.

There are two things that let it down quite badly. The keyboard is less than perfect – it lacks a number of keys: PgUp, PgDn, Home, End, Delete, Insert, F11 and F12. While inconvenient, this is reasonably workable around using a custom keyboard map and key combinations using AltGr.

The fundamental thing that makes the Chromebook Pixel nearly unusuable is the amount of heat it produces. Under any load above idle, the aluminium casing gets too hot to touch for any length of time. Under a gaming load, even the plastic keys on the keyboard get so hot they are painful to touch. The CPU itself doesn’t overheat (it tops out at about 85C), but the outer casing gets past 47C within a few minutes of Left4Dead 2.

47C may not sound like a lot, but given the high thermal conductivity of aluminium, 47C is actually very uncomfortable to touch. This problem isn’t unique to the Chromebook, either – I had a similar problem with the Macbook Pro Retina I was using at work previously. Consequently, I can only strongly recommend against getting the Chromebook Pixel.

Due to these issues, I am still using my old ThinkPad more frequently than the Pixel. My commute to work machine is now an ARM based Chromebook (XE303C12), which stays stone cold even under a heavy load, the battery lasts 6-8 hours, and is even lighter than the Pixel. It’s touchpad is quite terrible, but I can live with that in return for it not burning me as soon as I ask it to compile something for me.

All I can say is – beware the marketing hype and sexy looks. A laptop that looks fantastic on paper can easily turn out to be nearly useless due to how hot it gets.