Enclosure & Hardware Discussions
16x to 16x external PCIe riser?
 

16x to 16x external PCIe riser?  

  RSS

DrEGPU
(@dregpu)
Trusted Member
Joined: 1 year ago
 

So, I've been toying with the idea of getting an external PCIe expander for multiple GPU use. Not many (if any) real TB3 enclosures let you do this. Most of the PCIe expanders have mlutiple 16x slots in the external enclosure, which is good. However, the PCI card that you must use to connect TO the the external case fits into a 1x PCI slot. Are there any that can plug into a PCIe 3.0 16x slot to maximize the available bandwith to the external expander case. 

Some examples:
https://www.cubix.com/xpander-2/
https://www.amazon.com/StarTech-Express-Slot-Expansion-System/dp/B000UZL1GC/
https://www.amazon.com/SEDNA-express-slots-Riser-Multiplier/dp/B075DF9L5V/

MBP 2018 15 inch + 2X Aorus gaming box 1080 in BootCamp and MacOS 10.13.6
2011 13-inch MacBook Pro + Razer Core X@10Gbps-TB1 (Zotac RTX 2080 Ti Amp) + Ubuntu 18.04


ReplyQuote
omardesu
(@omardesu)
Trusted Member
Joined: 1 year ago
 

What host computer are you planning to use?  And are you planning to run multiple GPU's at the same time?

Looking at the 3rd link you provided, USB3.0 to PCIe x16 wouldn't even make sense as the USB3.0 would be the bottleneck.  As from your title, I only think of the PCIe riser that I will soon get with my new SFF case which just lets me relocate my GPU to another part of the case since it won't fit if it were directly connected to the ITX motherboard (see the link below).

https://www.sfflab.com/collections/parts/products/pcie-extender-adt

1070 Ti: 2014 Mac mini / 2016 MBP 15"
Mini ITX Build in Dan A4 Case


ReplyQuote
OliverB
(@oliverb)
Noble Member
Joined: 1 year ago
 

@dregpu
Recently I have seen a video where they put 16 GPUs in a server with Riser cards. Would this help you?

You could put 16 Tesla V100 into this rig.

This post was modified 7 months ago

2018 15" MBP & 2015 13" MBP connected to RTX2080Ti GTX1080Ti GTX1080 Vega56 RX580 R9-290 GTX680


ReplyQuote
DrEGPU
(@dregpu)
Trusted Member
Joined: 1 year ago
 

Right now, I'm just researching options. I was thinking of buying a regular workstation rig and trying either stuff multiple GPU's into the box or an external case. It would be nice to maximize bandwidth and throughput to minimize the bottleneck. It seems weird to try and connect 4 slots that 16x to a single dinky 1x PCI slot. That makes total sense for cryptocurrency mining, but I'm trying to move massive amounts of data (100's of TBs) from the SSD to the CPU to the GPU and back again. At the very least, you could split the 16x PCIe 3.0 slot (15.7GB/s) 4 ways instead if the 1x slot (250MB/s). 

@ omardesu Yes, I've seen what you've linked. As you point out, it's good for relocating a GPU, but I want to add many GPUs. As for the USB, that's just a convenient and cheap socket and cabling. It really isn't USB. You'll most likely fry something if you try to plug multiple GPUs into a regular USB port!

@OliverB For image classification, RTX 2080 Ti appears to be about 80% the performance at 10% of the cost. The increased VRAM in the V100's are nice, in that they allow larger batch sizes though. The riser boards in server rackmounts are rather expensive and I'm trying to be a bit judicious with the dollars!

MBP 2018 15 inch + 2X Aorus gaming box 1080 in BootCamp and MacOS 10.13.6
2011 13-inch MacBook Pro + Razer Core X@10Gbps-TB1 (Zotac RTX 2080 Ti Amp) + Ubuntu 18.04


ReplyQuote
OliverB
(@oliverb)
Noble Member
Joined: 1 year ago
 
Posted by: DrEGPU

@OliverB For image classification, RTX 2080 Ti appears to be about 80% the performance at 10% of the cost. The increased VRAM in the V100's are nice, in that they allow larger batch sizes though. The riser boards in server rackmounts are rather expensive and I'm trying to be a bit judicious with the dollars!

Depending a little bit on the application. For exact scientific calculations you would need FP64 performance and ECC, for both the RTX2080Ti cannot help a lot. It's eventually a gaming gpu. Games don't need that stuff.

2018 15" MBP & 2015 13" MBP connected to RTX2080Ti GTX1080Ti GTX1080 Vega56 RX580 R9-290 GTX680


ReplyQuote
DrEGPU
(@dregpu)
Trusted Member
Joined: 1 year ago
 
Posted by: OliverB
Posted by: DrEGPU

@OliverB For image classification, RTX 2080 Ti appears to be about 80% the performance at 10% of the cost. The increased VRAM in the V100's are nice, in that they allow larger batch sizes though. The riser boards in server rackmounts are rather expensive and I'm trying to be a bit judicious with the dollars!

Depending a little bit on the application. For exact scientific calculations you would need FP64 performance and ECC, for both the RTX2080Ti cannot help a lot. It's eventually a gaming gpu. Games don't need that stuff.

Yes, I know. That's why I mentioned image classification specifically, as it works just fine on FP32 and just as well with FP16 (although you have to be a bit more careful and specific). For MY needs, the RTX 2080 Ti's represent an extremely cost-efficient cards. See the below reference. I actually use many of those models and fine-tune them for my use. 

https://lambdalabs.com/blog/titan-rtx-tensorflow-benchmarks/

MBP 2018 15 inch + 2X Aorus gaming box 1080 in BootCamp and MacOS 10.13.6
2011 13-inch MacBook Pro + Razer Core X@10Gbps-TB1 (Zotac RTX 2080 Ti Amp) + Ubuntu 18.04


OliverB liked
ReplyQuote
OliverB
(@oliverb)
Noble Member
Joined: 1 year ago
 

@dregpu
very good and well educated answer.

2018 15" MBP & 2015 13" MBP connected to RTX2080Ti GTX1080Ti GTX1080 Vega56 RX580 R9-290 GTX680


ReplyQuote