Ice Lake CPU on-die Thunderbolt 3 Controller Bandwidth
 
Notifications
Clear all

Ice Lake CPU on-die Thunderbolt 3 Controller Bandwidth  

 of  2
  RSS

joevt
(@joevt)
Noble Member
Joined: 3 years ago
 

@mac_editor, I did the test with Mac mini 2018. 23 Gbps RAID0 from two ports - one controller. The limit goes away when using two different controllers (each controller is connected to a separate CPU x4 port).
https://forums.macrumors.com/threads/testing-tb3-aic-with-mp-5-1.2143042/post-27371729

@itsage, that 2019 13-in MacBook Pro results in a 23Gbps cap on opposing sides means both controllers are limited by PCH? I think this was discussed before?
https://egpu.io/forums/thunderbolt-enclosures/blackbeast-pro-thunderbolt-egpu/paged/6/#post-70326
I mentioned a test using eGPUs and CL!ing Memory Bandwidth test. I tried running two instances of CL!ing against the same eGPU. Both reported 2600 MB/s which means it's not a valid test for multiple bandwidth testing.

The 2019 16-in MacBook Pro is similar to the Mac mini 2018 result so it probably has two CPU x4 ports instead of PCH.

I had linked a post that shows IO Registry Explorer of the Ice Lake Thunderbolt stuff: 
https://www.tonymacx86.com/threads/success-gigabyte-designare-z390-thunderbolt-3-i7-9700k-amd-rx-580.267551/page-2132#post-2124650
https://egpu.io/forums/builds/2020-13-macbook-pro-10th4cg-rtx-2080-ti-32gbps-tb3-gaming-box-win10-2004-itsage/#post-81158

It shows the integrated Thunderbolt layout like HWiNFO does - all the Thunderbolt ports are root ports. There's no Thunderbolt upstream/downstream bridge layout that you get with a discrete Thunderbolt controller.

You can try my pcitree.sh script at https://gist.github.com/joevt/e3cd4ff08aae06279134969c98ca3ab7
Use it with pciutils (either mine or the original https://github.com/joevt/pciutils )
Install pciutils. run "sudo update-pciids". Then run the pcitree.sh script. It will list the PCIe devices like lspci does, but it includes info about link width and link speed (capabilities and status).

 

Mac mini (2018), Mac Pro (Early 2008), MacBook Pro (Retina, 15-inch, Mid 2015), GA-Z170X-Gaming 7, Sapphire Pulse Radeon RX 580 8GB GDDR5, Sonnet Echo Express III-D, Trebleet Thunderbolt 3 to NVMe M.2 case


ReplyQuote
joevt
(@joevt)
Noble Member
Joined: 3 years ago
 

@itsage, I noticed in your screenshot that the integrated Thunderbolt (iTBT) root ports are showing 2.5 GT/s x4 even though your tests are showing much more bandwidth. This is the same as discrete Thunderbolt (dTBT) (the downstream PCIe bridge devices of dTBT report 2.5 GT/s even though they have higher performance). In either case (the iTBT root ports or the dTBT downstream bridge devices) the reported link rate doesn't matter because they are integrated devices - not real PCIe - they have PCIe registers but they probably don't need to implement all of PCIe (I guess link training to discover link rate and link width is unnecessary for some integrated PCIe devices like these).

 

 

Mac mini (2018), Mac Pro (Early 2008), MacBook Pro (Retina, 15-inch, Mid 2015), GA-Z170X-Gaming 7, Sapphire Pulse Radeon RX 580 8GB GDDR5, Sonnet Echo Express III-D, Trebleet Thunderbolt 3 to NVMe M.2 case


ReplyQuote
itsage
(@itsage)
Illustrious Member Admin
Joined: 4 years ago
 
Posted by: @nando4
Posted by: @itsage

Next question is what happens when there's a RAID0 with 4x NVMe drives on the Ice Lake system? Unfortunately I don't have anymore M2-TB3 enclosures to test. Although I can use a regular TB3 enclosure with an M.2 PCIe adapter.

Intel have certainly improved the TB3 controller in Ice Lake CPUs. Though do we get the full burrito?

Yes, please use the regular TB3 enclosure with M.2 adapter to wired up 4x NVME drives. We'll then see if there is a full FAT 4x TB3 pipe or if DMI applies some brakes.

I added a third NVM3 drive through an M.2 PCIe adapter in Akitio Node Pro. Unfortunately that did not improve shared bandwidth over the two drive RAID0 set. So it seems there are at most two iTBT controllers which can handle any of the four PCI Express root ports concurrently.

@joevt I tried your pcitree.sh script but there was not output. Can you provide some instructions? Thank you!

external graphics card builds
best laptops for external GPU
eGPU enclosure buyer's guide

 
2019 13" MacBook Pro [8th,4C,U] + RX 5600 XT @ 32Gbps-TB3 (VisionTek mini eGFX) + macOS 11 & Win10 2004 [build link]  


ReplyQuote
mac_editor
(@mac_editor)
Famed Member Moderator
Joined: 3 years ago
 

@itsage, thanks for testing. Could you try with larger test file size in AJA?

This is a somewhat confusing result. When you say:

Posted by: @itsage

So it seems there are at most two iTBT controllers which can handle any of the four PCI Express root ports concurrently.

Do you mean one controller can handle any 2 of the 4 PCIe express root ports (it can pick any 2)? I was expecting that we have 2 TB ports per controller and the shared bandwidth was improved (but doesn’t explain the results if we consider a traditional TBT layout) or straight up more overall lanes. But this instead seems dynamic. What results do you get when you test each drive individually (but all 3 are connected)?

 

purge-wranglertbt-flashpurge-nvdaset-eGPU
Insights Into macOS Video Editing Performance

Master Threads:
2014 15-inch MacBook Pro 750M
2018 15-inch MacBook Pro

 
2019 13" MacBook Pro [8th,4C,U] + RX Vega 64 @ 32Gbps-TB3 (Mantiz Venus) + macOS 10.14.6 & Win10 [build link]  


ReplyQuote
joevt
(@joevt)
Noble Member
Joined: 3 years ago
 

@itsage, "sudo pcitree.sh". It requires that you have pciutils installed (lspci, setpci commands).  #474 

Your 3 port NVMe raid indicates a ~5000 MB/s cap rather than a 2750x4 MB/s cap. That makes it about as good as a two controller setup with x4 CPU lanes each (like the Mac mini 2018), except that you can choose any two of the four ports instead of having to select a port from each controller. The total is similar to PCIe 2.0 x16 or PCIe 3.0 x8. We were hoping for something closer to PCIe 3.0 x16 but maybe that's too much for a mobile CPU (which needs bandwidth for GPU and PCH, etc.).

Hopefully all 3 NVMe drives have similar performance (~2400 MB/s read) when used individually.

 

Mac mini (2018), Mac Pro (Early 2008), MacBook Pro (Retina, 15-inch, Mid 2015), GA-Z170X-Gaming 7, Sapphire Pulse Radeon RX 580 8GB GDDR5, Sonnet Echo Express III-D, Trebleet Thunderbolt 3 to NVMe M.2 case


itsage liked
ReplyQuote
itsage
(@itsage)
Illustrious Member Admin
Joined: 4 years ago
 

@mac_editor, Sorry I should have phrased my sentence better. My guess is there are two integrated Thunderbolt 3 controllers on the Ice Lake CPU. Each can dynamically handle up to two TB3 ports. I tried a larger test file size with no improvement to total shared bandwidth.

@joevt I will try that again tomorrow. The NVMe-Thunderbolt 3 drives can reach the max 22Gbps individually when I had all threes connected.

 

external graphics card builds
best laptops for external GPU
eGPU enclosure buyer's guide

 
2019 13" MacBook Pro [8th,4C,U] + RX 5600 XT @ 32Gbps-TB3 (VisionTek mini eGFX) + macOS 11 & Win10 2004 [build link]  


ReplyQuote
mac_editor
(@mac_editor)
Famed Member Moderator
Joined: 3 years ago
 
Posted by: @itsage

Each can dynamically handle up to two TB3 ports.

Which I presume is the new feature here.

purge-wranglertbt-flashpurge-nvdaset-eGPU
Insights Into macOS Video Editing Performance

Master Threads:
2014 15-inch MacBook Pro 750M
2018 15-inch MacBook Pro

 
2019 13" MacBook Pro [8th,4C,U] + RX Vega 64 @ 32Gbps-TB3 (Mantiz Venus) + macOS 10.14.6 & Win10 [build link]  


itsage liked
ReplyQuote
 of  2