2012 Mac Mini [3rd,4C,Q] + RX 480 @ 10Gbps-TB1 (Atto Thunderlink) + macOS 10.15....
 
Notifications
Clear all

2012 Mac Mini [3rd,4C,Q] + RX 480 @ 10Gbps-TB1 (Atto Thunderlink) + macOS 10.15.7  

 of  2
  RSS

Yukikaze
(@yukikaze)
Noble Member Moderator
Joined: 5 years ago
Builds: 24
 

@joevt, these are good ideas, but to be honest, I don't have the time to devote to doing that, so I am not likely to bother. At least as long as this keeps on holding together Smile

 

Want to output 4K@60Hz out of an old system on the cheap? Read here.
Give your Node Pro a second Thunderbolt3 controller for reliable peripherals by re-using a TB3 dock (~50$).

"Always listen to experts. They'll tell you what can't be done, and why. Then do it."- Robert A. Heinlein, "Time Enough for Love."

 
2012 Mac Mini [3rd,4C,Q] + RX 480 @ 10Gbps-TB1 (Atto Thunderlink) + macOS 10.15.7 [build link]  


ReplyQuote
Yukikaze
(@yukikaze)
Noble Member Moderator
Joined: 5 years ago
Builds: 24
 

Can Thunderbolt1 actually run two 10Gbps devices at once? Let's see.

I have an old Sonnet Echo Express Pro, and I decided to test whether I can trick out this Mac even further by giving it a NVMe SSD alongside the eGPU, as well as test whether a FebSmart USB-C 3.1 Gen2 expansion card would also work on this setup.

With respect to the SSD, the question of bandwidth arose, but Thunderbolt1 is supposed to have two separate 10Gbps channels available (in Thunderbolt2 these channels are aggregated, which is how we get to the 20Gbps theoretical bandwidth), so in theory, two PCIe devices could each achieve full individual bandwidth over the same link.

In theory.

Spoiler alert: That doesn't happen perfectly in practice.

My NVMe SSD for this test is an el-cheapo SSSTC CL1 128Gb NVMe drive. It is a postage stamp sized 2230 NVMe SSD that costs 15$ on Amazon. It is connected to a generic PCIe x4 to NVMe riser, and then placed into the Sonnet Echo Express Pro.

Here's the NVMe device bandwidth courtesy of ATTO Disk Benchmark. The Sonnet box is daisy-chained to my Thunderlink-based eGPU.

image

Reads top out at 870 MiB/s, while writes reach 595 MiB/s. 4KB block sizes are read at 61 MiB/s and written at 204 MiB/s.

Next is the MX500 drive in the Mac mini itself:

image

Reads top out at 487 MiB/s, while writes reach 474 MiB/s. 4KB blocks are read at 28 MiB/s and written at 196 MiB/s.

So yes, the TB1 NVMe el cheapo drive is faster than a top of the line SATA SSD.

Now let's see what it does to eGPU bandwidth.

Here we see CL!ng of the eGPU's H2D and D2H bandwidth with the NVMe drive idle. The bandwidth is pretty much as expected for the TB1 link, and is stable:

image

Now let's fire up the ATTO benchmark on the NVMe drive and see what happens to both results:

image

The eGPU bandwidth is taking a beating, dropping to about 5Gbps. What does the NVMe drive perform like?

image

The 4KB results are not affected, matching the idle eGPU case, but the top of the graph suffers and tops out at around 720 MiB/s for reads (a 17% decrease) and 520 MiB/s for writes (a 13% drop). The total bandwidth we're seeing from both eGPU and NVMe drive is about 11-12Gbps. Far lower than the expected 15-16Gbps if the dual channel arrangement was working perfectly, but more than what a single channel can achieve on its own. So the dual-channel architecture does work, but it is far from perfect.

For good measure, let's benchmark the internal SSD and see what the eGPU sees, to make sure we're not just seeing a system-wide effect:

image

You might notice some jitter around the middle of the graph. There's a tiny performance impact in the cases where the SATA disk is getting slammed with 4KB and 8KB requests, likely due to overall system resource contention, but this has no real effect on eGPU performance. Disk performance is also maintained, matching the eGPU idle case within the margin of error.

Long story short: Stick with a SATA SSD in these old systems. The NVMe drive in the chain will negatively impact your gaming performance and is not recommended. However, if you're just trying to get the fastest drive in an old TB1 system, and not using the TB1 port for anything else mission-critical, then an NVMe drive over TB1 is superior to internal SATA.

As a side note, the FebSmart USB-C add-on card did not work in this arrangement. It gets power, and it can charge devices, but Catalina doesn't see it and devices connected to it are not available in Mac OS.

That's it, thanks for sticking around for yet another episode of Yukikaze benchmarks old shit. Ciao!

 

Want to output 4K@60Hz out of an old system on the cheap? Read here.
Give your Node Pro a second Thunderbolt3 controller for reliable peripherals by re-using a TB3 dock (~50$).

"Always listen to experts. They'll tell you what can't be done, and why. Then do it."- Robert A. Heinlein, "Time Enough for Love."

 
2012 Mac Mini [3rd,4C,Q] + RX 480 @ 10Gbps-TB1 (Atto Thunderlink) + macOS 10.15.7 [build link]  


joevt and nando4 liked
ReplyQuote
joevt
(@joevt)
Noble Member
Joined: 4 years ago
 

@yukikaze

I don't think using two separate benchmarks simultaneously is a good way to measure total bandwidth because you can't know that the timing intervals are continuous in both benchmarks.

For the ATTO Disk Benchmark, there is a pause between each block size being tested. Also, you should change the units to MB/s to make it easier to convert to Gb/s.

For the CL!ing benchmark, there may be a pause between testing Host to Device and testing Device to Host. Also, the 7 Gbit/s number is an average.

So you can't really add them up. Maybe you can take the minimum from CLIng (3.7 Gbps) and add it the ATTO minimum (600 MiB = 6.29 Gbps) which is < 10 Gbps.

I did some Thunderbolt 1 tests with 2x NVMe's and two displays at  #44 . ATTO Disk Benchmark is a benchmark that can test multiple devices at the same time (but the Windows version can't?)

I did not find evidence that two channels from the same Thunderbolt port can be used for PCIe tunnelling. The second channel seems to be only usable with a display. Maybe I need Thunderbolt 1 devices to connect the Thunderbolt 3 devices to. I would like a utility to read Thunderbolt registers (like Linux can) and then output the Thunderbolt paths, channel info, adapters (DisplayPort In/Out, PCIe Up/Down, link adapters, etc.), etc. by interpreting those registers.

 

 

Mac mini (2018), Mac Pro (Early 2008), GA-Z170X-Gaming 7, Sapphire Pulse Radeon RX 580 8GB GDDR5, Radeon Pro W5700, Sonnet Echo Express III-D, Trebleet Thunderbolt 3 to NVMe M.2 case


ReplyQuote
Yukikaze
(@yukikaze)
Noble Member Moderator
Joined: 5 years ago
Builds: 24
 

@joevt, you're probably right. Something to keep in mind, my overall goal was to see whether using an NVMe drive down the chain would impact eGPU bandwidth of not. The oscillations on the H2D/D2H measurements tell me that it is the case, at which point I effectively gave up on the idea.

 

Want to output 4K@60Hz out of an old system on the cheap? Read here.
Give your Node Pro a second Thunderbolt3 controller for reliable peripherals by re-using a TB3 dock (~50$).

"Always listen to experts. They'll tell you what can't be done, and why. Then do it."- Robert A. Heinlein, "Time Enough for Love."

 
2012 Mac Mini [3rd,4C,Q] + RX 480 @ 10Gbps-TB1 (Atto Thunderlink) + macOS 10.15.7 [build link]  


ReplyQuote
joevt
(@joevt)
Noble Member
Joined: 4 years ago
 

@yukikaze, but what situation would be hammering the eGPU and NVMe at the same time? Games don't do that. What you need to know is if having the NVMe down the chain impacts eGPU bandwidth when nothing is being transferred to/from the NVMe. I think your CL!ng screenshot during the ATTO test shows that (at the beginning before ATTO started and at the end when ATTO stopped).

 

Mac mini (2018), Mac Pro (Early 2008), GA-Z170X-Gaming 7, Sapphire Pulse Radeon RX 580 8GB GDDR5, Radeon Pro W5700, Sonnet Echo Express III-D, Trebleet Thunderbolt 3 to NVMe M.2 case


ReplyQuote
 of  2