Ice Lake CPU on-die Thunderbolt 3 Controller Bandwidth
@mac_editor, I did the test with Mac mini 2018. 23 Gbps RAID0 from two ports - one controller. The limit goes away when using two different controllers (each controller is connected to a separate CPU x4 port).
@itsage, that 2019 13-in MacBook Pro results in a 23Gbps cap on opposing sides means both controllers are limited by PCH? I think this was discussed before?
I mentioned a test using eGPUs and CL!ing Memory Bandwidth test. I tried running two instances of CL!ing against the same eGPU. Both reported 2600 MB/s which means it's not a valid test for multiple bandwidth testing.
I had linked a post that shows IO Registry Explorer of the Ice Lake Thunderbolt stuff:
It shows the integrated Thunderbolt layout like HWiNFO does - all the Thunderbolt ports are root ports. There's no Thunderbolt upstream/downstream bridge layout that you get with a discrete Thunderbolt controller.
You can try my pcitree.sh script at https://gist.github.com/joevt/e3cd4ff08aae06279134969c98ca3ab7
Use it with pciutils (either mine or the original https://github.com/joevt/pciutils )
Install pciutils. run "sudo update-pciids". Then run the pcitree.sh script. It will list the PCIe devices like lspci does, but it includes info about link width and link speed (capabilities and status).
@itsage, I noticed in your screenshot that the integrated Thunderbolt (iTBT) root ports are showing 2.5 GT/s x4 even though your tests are showing much more bandwidth. This is the same as discrete Thunderbolt (dTBT) (the downstream PCIe bridge devices of dTBT report 2.5 GT/s even though they have higher performance). In either case (the iTBT root ports or the dTBT downstream bridge devices) the reported link rate doesn't matter because they are integrated devices - not real PCIe - they have PCIe registers but they probably don't need to implement all of PCIe (I guess link training to discover link rate and link width is unnecessary for some integrated PCIe devices like these).
Next question is what happens when there's a RAID0 with 4x NVMe drives on the Ice Lake system? Unfortunately I don't have anymore M2-TB3 enclosures to test. Although I can use a regular TB3 enclosure with an M.2 PCIe adapter.
Intel have certainly improved the TB3 controller in Ice Lake CPUs. Though do we get the full burrito?
Yes, please use the regular TB3 enclosure with M.2 adapter to wired up 4x NVME drives. We'll then see if there is a full FAT 4x TB3 pipe or if DMI applies some brakes.
I added a third NVM3 drive through an M.2 PCIe adapter in Akitio Node Pro. Unfortunately that did not improve shared bandwidth over the two drive RAID0 set. So it seems there are at most two iTBT controllers which can handle any of the four PCI Express root ports concurrently.
@joevt I tried your pcitree.sh script but there was not output. Can you provide some instructions? Thank you!
@itsage, thanks for testing. Could you try with larger test file size in AJA?
This is a somewhat confusing result. When you say:
So it seems there are at most two iTBT controllers which can handle any of the four PCI Express root ports concurrently.
Do you mean one controller can handle any 2 of the 4 PCIe express root ports (it can pick any 2)? I was expecting that we have 2 TB ports per controller and the shared bandwidth was improved (but doesn’t explain the results if we consider a traditional TBT layout) or straight up more overall lanes. But this instead seems dynamic. What results do you get when you test each drive individually (but all 3 are connected)?
Your 3 port NVMe raid indicates a ~5000 MB/s cap rather than a 2750x4 MB/s cap. That makes it about as good as a two controller setup with x4 CPU lanes each (like the Mac mini 2018), except that you can choose any two of the four ports instead of having to select a port from each controller. The total is similar to PCIe 2.0 x16 or PCIe 3.0 x8. We were hoping for something closer to PCIe 3.0 x16 but maybe that's too much for a mobile CPU (which needs bandwidth for GPU and PCH, etc.).
Hopefully all 3 NVMe drives have similar performance (~2400 MB/s read) when used individually.
@mac_editor, Sorry I should have phrased my sentence better. My guess is there are two integrated Thunderbolt 3 controllers on the Ice Lake CPU. Each can dynamically handle up to two TB3 ports. I tried a larger test file size with no improvement to total shared bandwidth.
@joevt I will try that again tomorrow. The NVMe-Thunderbolt 3 drives can reach the max 22Gbps individually when I had all threes connected.
Each can dynamically handle up to two TB3 ports.
Which I presume is the new feature here.