PDF guide and patches for making Linux v5.3 kernel to work with Thunderbolt 3 add-in card.
Please follow this guide at your own risk. If you do not accept the risk, then please do not use this guide, and instead buy an Intel system with Thunderbolt 3 support.
It is not perfect and I have been procrastinating to get it out for months. But the zip archive contains a guide on applying my Linux patches and creating a kernel with working Thunderbolt hot-plug with a Thunderbolt 3 add-in card (sorry, Titan Ridge only for now) on any computer. Sorry, Windows will only work if the devices are attached at boot - there is nothing I can do about that one without fancy chainloaders, which are well outside of my experience.
I removed the last patch for enabling Alpine Ridge Thunderbolt 3 add-in cards for now, as it is finicky and not up to standards. Plus, I am away from my test rig where I can make sure it works. I will add it later.
If I do not do this now, I will never get around to it.
Please let me know if there are any mistakes or problems.
See the guide for known problems and notes. The biggest issue is no suspend / sleep - it will almost certainly crash the computer if you suspend, so make sure you disable sleep.
Sorry it took so long.
Oops, forgot to add pin numbering image in PDF. I used to have to jump GC-TITAN RIDGE to be detectable on startup - but strangely, I have recently found it is no longer needed. I am not sure what changed.
___ __/ \__ |o o o o o| |_________| 1 2 3 4 5
Progress on getting accepted into Linux mainline kernel:
1. Got the new kernel parameters accepted to specify mmio and mmio_pref separately: https://lkml.org/lkml/2019/10/23/726
2. Got the bug fix for double allocation accepted: https://lkml.org/lkml/2019/11/14/627
These will appear in Linux v5.5 in a few weeks. If you are willing to run a release candidate kernel (could have problems) then Ubuntu will compile it and put it here (link broken until they do):
These changes are enough to hot-add any single Thunderbolt 3 device, including an eGPU. Daisy chain will not work, yet. If you want daisy chaining, then you will have to compile Linux v5.3 kernel with the zip.
Bjorn Helgaas (Linux PCI subsystem maintainer) says he wants to wait for early in the next cycle to accept the rest, because that way they will get more testing before making it into a Linux v5.6 release candidates.
Assuming none of the patches cause problems with other people's different hardware and do not get pulled (I am confident they will not, I have been using these on my computers for many months) then Linux v5.6 will have all of the fixes needed.
@karatekid430 Congratulations and thank you for sharing this project!
@karatekid430, i'm sorta curious why this isn't, like, in big flashing letters across all the linux news sites. this is a big deal isn't it?
I think i read you say somewhere else that you've never touched a thunderbolt2 card. would you like to have one? i'll happily buy and send you a thunderbolt 2 card if you think you can find time to investigate and possibly make them work. reason being, as more and more of the older apple devices drop off of apple's supported lists, more of that hardware is going to become available to tinkerers for cheap. even the 2011-era machines like the old imacs and mac minis, and laptops, which are no longer able to run macos, can run linux very nicely - but they only have firewire800, usb2.0, or thunderbolt1/2 for direct connectivity. being able to install a used $50 card in a real modern computer and instantly have a 10g interconnect could be pretty damn useful imo - such as using an old mac mini to legitimately virtualize macos for testing, and being able to use a fast freenas box for an iscsi or NFS backing store for VMs over the 10g link with just a cheap used tbolt2 card and a tbolt2 cable, which are a dime a dozen nowadays. there's also a slew of the older tbolt2 kit that it would be nice to be able to use without an adapter cable, which is more expensive than the thunderbolt2 cards themselves nowadays.
just a thought, PM me if that sounds useful.
@micah_roth, thanks. I thought it would receive slightly more attention than it did, but it was probably never going to make as much attention as people putting those Titan Ridge cards in Threadripper and them partially working.
As for Thunderbolt 2, I could do it, but the pcie2tbt hack I have in Linux is not the most reliable thing, and I cannot be sure that the handshake is not completely different for Thunderbolt 2. Even if I get the card awake, then I need to test with a device and I would need to borrow the Apple adapter from a friend. Also, when using a card designed for BIOS enumeration with native enumeration, there are some annoyances.
For instance, the USB controller sits on a hotplug bridge, because the USB controller disappears when unused. That means the OS will assign many bus numbers and waste them to the USB controller unless I hardcode a workaround into the kernel. Oops, I am talking out of my butt. Thunderbolt 2 does not contain a USB controller.
I could do it if you wish, but since I have never, ever laid my hands on a system with Thunderbolt 2, I do not know what to expect / cannot guarantee anything. Oh, if you are expecting this to work in anything but Linux (you mentioned running / virtualising MacOS), I can tell you it will not. The only way for that to happen is for somebody with way more skills than myself (perhaps people who write the GRUB bootloader or BIOS images) could make a chain loader which enables the Thunderbolt controller and then passes onto Windows or MacOS. But hotplug would likely be broken unless they overrode the ACPI tables as well. Also, even if I do get this working for Linux, if you have any PCIe devices attached to the Thunderbolt card and sleep the computer, it *will* crash in my experience upon wake. I tried and failed to make power management work.
Thinking of Thunderbolt Networking as you mentioned. They say it is 10 Gb/s. With Thunderbolt 3, I have benchmarked upto 16-17 Gb/s, which implies it uses one 20 Gb/s link of Thunderbolt 3. In Linux v5.6 (either that or v5.5) there are new sysfs entries in the Thunderbolt driver rx_lanes, tx_lanes, rx_speed, tx_speed. For now, rx and tx always show the same amount - it is hard coded in the kernel that they read data from the same place. In the future, asymmetric links may be possible, though. Unfortunately, these do not show for Thunderbolt Networking, and cannot show unless using software connection manager in driver (information unavailable if the Thunderbolt firmware in the controller is setting the connections up for us). Software connection manager is some Apple stuff, and USB4. Even Ice Lake Thunderbolt 3 does not use the driver. Also, another driver patch may be required, and Mika Westerberg at Intel is busy with bigger fish to fry. However, I have connected to a Thunderbolt 3 Apple, and the Mac reported 20 Gb/s x1. It means it is only using the single lane. I do not know why the firmware does not bond two lanes for Thunderbolt Networking. Nevertheless, there is a good chance that USB4 will bond two lanes together and give 40 Gb/s for Networking. Reduce that to 27 Gb/s for PCIe after 128B max payload size tanks the performance. But that is still a lot of performance for something marketed as 10 Gb/s.
Most people will not realise even 10 Gb/s, though. Assuming most people use Windows and not Linux (unfortunately), I find that the out of the performance iperf is 2-3 Gb/s because of the firewall or something. I see a thread maxed out, mostly in kernel mode. I have gotten about 10 Gb/s but only after disabling a lot of stuff. Linux flies right out of the box at about 10 Gb/s and with tuning can get easily to 14 Gb/s without too much effort. But that is just iperf. If you use scp on Linux to copy files, you get 2-3 Gb/s because of CPU bottleneck. Here we are with AES instructions baked into the CPU capable of tens of GB/s and yet the software developers cannot reach that potential. Frustrating. I am not sure where the bottleneck is, but I hope it is not in the SSL library. Before I was born, people hand assembled code and had to be efficient because the computers were slow and had little RAM. I feel that the observation "as computers get more powerful, developers get lazier" is true.
That spiel about Thunderbolt Networking is irrelevant for Thunderbolt 2, but I thought somebody on the forum would find it interesting.
Why can a 2011 Mac not run MacOS?
I looked on Ebay and cannot find a Gigabyte GC-THUNDERBOLT 2 card. Do you know where they can be found?
Another option is for me to polish up the kernel patch to enable old Thunderbolt controllers in Linux and give it to you for testing. Have you ever compiled a kernel before? The problem is you should never run a pre-compiled kernel from a stranger because you never know what they put in it.
Cheers for your reply.
Oh, I forgot to mention - all of my kernel patches have been accepted into mainline with Linux v5.6, with the first RC released about one month ago. For now, Linux v5.6 is still a release candidate and not stable. But if you are comfortable using a RC, then install it with Gigabyte GC-TITAN RIDGE and do in kernel command line:
You can adjust the 128M and 512M to suit. These quantities are per Thunderbolt port. Use "lspci -vvvv" to see information. The hpmmiosize comes from 32-bit address space so theoretically 4G max but in reality you will likely only get 1G in total (less per Thunderbolt port) because other things will have consumed parts of it. I routinely put 64G for hpmmioprefsize but that is complete and utter overkill unless you want to attach Xeon Phi Coprocessors in a Thunderbolt eGPU. The theoretical max for this is your number of address lines on CPU. Mainstream platforms (last I heard) and laptops (such as i7-8650U) have 39-bits = 512G which is used by RAM, MMIO, MMIO_PREF and other things. Server CPUs might have 48-57 (57 is Intel 5-level paging, IIRC) which would allow for insane amounts. To this day, nothing has ever implemented all 64 address lines, as nobody can afford that much RAM, nor can they make it compact enough to fit in a single system.
Ideally try without nocrs but nocrs tends to open up a lot more space, and may be needed.
Sleep will still not work. If you sleep with Thunderbolt devices attached, do not expect the computer to wake successfully. If you want things to work perfectly, you will have to buy the Gigabyte TRX40 board that is bundled with GC-TITAN RIDGE.
You may need to short out the jumper still to make GC-TITAN RIDGE stay awake - or not. I used to have to, and now I have tried and it no longer needs it, and I have absolutely no idea why.
I know how to turn MSI Thunderbolt 3 cards from BIOS mode to native mode (make them visible like GC-TITAN RIDGE) and possibly work on Asrock (unconfirmed as I damaged the ROM chip on my card). If anybody is interested I can give source code for the program which uses /dev/mem to hit up pcie2tbt and change enumeration mode. Will not post if no interest. Never owned Asus one as overpriced and single port so no idea there.
Unfortunately, these do not show for Thunderbolt Networking, and cannot show unless using software connection manager in driver (information unavailable if the Thunderbolt firmware in the controller is setting the connections up for us). Software connection manager is some Apple stuff, and USB4. Even Ice Lake Thunderbolt 3 does not use the driver.
Is software connection manager only usable on Thunderbolt controllers containing Apple firmware? Is there no way to switch from firmware connection manager to software connection manager for other controllers? I suppose there is a method to determine what connection manager is being used?
Apple does something in their software connection manager or somewhere to enable dual HBR3 links for the XDR display (6K dual HBR3 tile mode instead of 6K HBR2 with DSC mode). Normally Titan Ridge can only have one HBR3 link but each tile of the 6K resolution doesn't require the entire HBR3 link and Thunderbolt doesn't transfer DisplayPort stuffing symbols so dual HBR3 can work.
However, I have connected to a Thunderbolt 3 Apple, and the Mac reported 20 Gb/s x1. It means it is only using the single lane. I do not know why the firmware does not bond two lanes for Thunderbolt Networking. Nevertheless, there is a good chance that USB4 will bond two lanes together and give 40 Gb/s for Networking. Reduce that to 27 Gb/s for PCIe after 128B max payload size tanks the performance.
Does Thunderbolt Networking uses PCIe packets over Thunderbolt? I was guessing some kind of DMA was used which I guess could mean PCIe.
Why would bonding depend on the traffic being sent? Thunderbolt can send different kinds of traffic down the same cable - DisplayPort, PCIe, etc - so the connection should be the best possible.
I'm not sure the 128B max payload size (MPS) is the complete reason for the performance drop of Thunderbolt. Have you compared with a device connected without Thunderbolt that has a 128B max payload size?
The USB4 spec has a note about performance tradeoff with PCIe (and USB4) traffic which depends on the number of buffers or something.
If anybody is interested I can give source code for the program which uses /dev/mem to hit up pcie2tbt and change enumeration mode.
I am interested in this. Is enumeration mode (BIOS/native) the same as connection manager (software/firmware) or something else? I think not?
I use mm in EFI Shell to enable PCIe 2.0 for PCIe 3.0 devices in my MacPro3,1, and to enable a GC-ALPINE RIDGE (USB only). I suppose MSI Thunderbolt 3 is similar to GC-ALPINE RIDGE (uses Alpine Ridge) since GC-TITAN RIDGE is the only Thunderbolt add-in card that uses Titan Ridge.
An EFI driver would be a better implementation of this because the PCIe address would not need to be hardcoded (requires the cards to be in the same slots and PCIe enumeration to be consistent between boots). I have two GC-TITAN RIDGE and two GC-ALPINE RIDGE in a PCIe expansion box. When I boot into Windows, it can see devices connected to GC-TITAN RIDGE but not GC-ALPINE RIDGE. I can then warm boot into MacOS and use PCIe devices of Thunderbolt devices connected to GC-TITAN RIDGE (and USB devices connected to GC-ALPINE RIDGE and GC-TITAN RIDGE).
echo "Enabling PCIe 2.0 for slot 2..." mm 0000010009C 42 -w 1 -PCIE -n mm 0000010007C 20 -w 1 -PCIE -n mm 0000010009C -w 1 -PCIE -n echo "Enabling Alpine Ridge (bus 0f and 10)..." mm 000f000054C 00 -w 1 -PCIE -n mm 000f000054C 0D -w 1 -PCIE -n mm 0010000054C 00 -w 1 -PCIE -n mm 0010000054C 0D -w 1 -PCIE -n mm 000f0000548 -w 1 -PCIE -n mm 00100000548 -w 1 -PCIE -n
There are recent developments in GC-TITAN RIDGE firmware flashing to get full MacOS support (requires SSDT patching as well via Open Core EFI boot loader) but I don't want to go down that route if possible. I think @itsage is planning on testing this.