Four PCIe slots into the Razer Core v2???  

@lefuturdelavenir Amazing findings! Please post updates as you go.

@lefuturdelavenir

What is the original I/O board's PCIe arrangement? The review shows 3 ASM1142 controllers. Does one of them use two lanes? Or are they all single lane? Two lanes gives 8 Gbps instead of 7.877 Gbps. If they are all single lane, then you might have a 4th usable lane. Making the lanes usable by individual devices would require a custom PCB or some creative wiring - a PCIe slot has holes on the top where you might be able to connect wires to (4 wires for each lane - the number of super speed wires in a USB 3.x cable - I'm not sure if you need to connect the 3 or 4 ground lines per lane - check the wiring of the PCIe x1 to USB adapter that you have for the 3 slot daughter board - Is PRSNT#1 connected to PRSNT#2 at pin 31 on the I/O board?).
https://en.wikipedia.org/wiki/PCI_Express#Pinout

Maybe you can find x4 extenders and just use those wires, attach USB connectors, and connect to risers. I don't know if all risers are wired the same.
https://www.microsatacables.com/pcie-express-4x-riser-card-with-flexible-cable-12-inches-pci-e4x-rs-12
https://www.microsatacables.com/pcie-1x-to-pcie-16x-card-with-usb-30-cable-pci-e1x-16x-usb
https://www.microsatacables.com/pcie-express-1x-to-16x-riser-extender-card-pci-e1x-16x-ext
https://www.microsatacables.com/pcie-express-3-port-1x-multiplier-riser-card-pci-e1x-3pswt

Are the lanes divided by connecting individual lanes to different devices, or are they set that way by the firmware of the the Thunderbolt controller or the hardware on the main board? In other words, if you connected an x2 or x4 device, would it work at x2 or x4? Switches that have an x4 upstream link are much more expensive than the single lane upstream link that you have with the 3 slot daughter board so you might not want to go that way anyway.

@joevt

I was under the impression that each ASM1142 is using a single PCIe lane. Using a 2x lanes on one of the USB controller would not have made a lot of sense to me. Having 8GT/s for 2 USB3.0 ports is way enough. I think we have to remember that the overall bandwidth left for the daughter board is limited. Even if the eGPU does not consume 32Gbps all the time I think it is safe to assume that going over 8Gbps on the daughter board is not « safe » since it could lead to congestion. In that case, the dual thunderbolt configuration is supposed to prioritize the thunderbolt controller that is closer to the MacBook, so the GPU, but I would not rely on that.

That’s being said, I would love to try to play with the PCIe bifurcation of the Razer Core.

About bandwidth, here are some thoughts:

• Using USB3.0 cables to transmit PCIe is possible even if it’s not designed for that. My test bench uses this method that is wildly use by GPU miners. It’s pretty reliable with PCIe Gen2 since it’s 5GT/s when the cable is designed to transmit 5Gb/s. Regarding PCIe Gen3, that’s 8GT/s so… let’s say the cable is not designed for that.
• All the 3 PCIe cards I used in my test are PCIe Gen2 anyway. When a Gen2 card is plugged in a Gen3 slot, the controller automatically use Gen2 and slow down the signaling.
• By my understanding of PCIe, it’s not possible to use two 1x lanes together to form a 2x slot. I’ll have to study the DSL6540 datasheet but I‘m pretty sure the integrated PCIe switch is configurable in two modes : 4x and 1x1x1x1x (1 big link vs 4 small links). I’m not sure about 1x1x2x. I guess this PCIe configuration can be set by changing a value in a registry (in a ROM somewhere near the controller) or maybe by changing a pinout wiring (so soldered on the board). I’m pretty sure that configuration cannot be changed dynamically.
• Quick recap on the bandwidth available on PCIe Gen2 and Gen3. Those don’t use the same encoding and that's why the ratio between signaling speed and real data speed are different. PCIe Gen2 use 8b/10b encoding meaning you have to multiply the transfer rate by 80% to get the real throughput. The PCIe Gen3 (and above) use 128b/130b so it’s around 98%. Because it's so close to 100% a lot of people intentionally (or not) use transmission rate and throughput indifferently. So that’s 4Gb/s and 7,88Gb/s real throughput respectively per lane for PCIe Gen2 and Gen3.

My idea is to finalize my testing in order to definitely take out the original daughter board. That way, and because I use my setup every day, I'll have more time with the study of the board without having to tear down my Razer Core and make my setup unusable. Then I'll work on bifurcation. I plan to make a special PCIe riser with PCIe 4x on one end and tree PCIe 1x in a form of tree USB3.0 connector on the other end. I'll use a PCIe 4x extension ribbon and tree USB3.0 PCB modules.

I highly recommend this awesome tread forum on PCIe bifurcation and this video from Intel on that topic.

@lefuturdelavenir

ASM1142 is a USB 3.1 gen 2 controller. If it's only operating at gen 1 speed (5 Gbps -> 4 Gbps), then you can update the firmware to allow gen 2 speed (10 Gbps -> 9.697 Gbps). Note that the controller usually has a bug where it reports gen 1 speed even when it's communicating at gen 2 speed (you need a device capable of more than 500 MB/s to show this). The ASM1142 is limited to PCIe 3.0 x1 (7.877 Gbps) or PCIe 2.0 x2 (8 Gbps) so you can't get full gen 2 performance. Using 2 lanes is slightly better than using one line (by 123 Mbps) or they can choose to waste a lane.

The eGPU is limited to 22 Gbps (can't get more than 2750 MB/s). I guess the USB controllers also share that limit. I don't know about congestion. Usually I would not be doing a lot of accessing a USB drive while playing a game.

There is nothing stopping the USB cables from being able to transmit PCIe 3.0 (8 GT/s) except bad quality that causes errors. If a device allows it, the Thunderbolt controller will try to negotiate 8 GT/s and fall back to 5 GT/s if link training fails. Worst case, it could drop to 2.5 GT/s. Like you said, all your devices (and the The PI7C9X2G404SL of the 3 slot daughter board) are limited to 5 GT/s.

There are many ASM1142 based USB cards that are PCIe 2.0 x2 electrically, and PCIe x4 physically (the StartTech product pages are bad at explaining this).
https://www.startech.com/Cards-Adapters/USB-3.0/Cards/2-port-usb-3-1-card~PEXUSB311EI

The Sunix UPD2018 uses an ASM1142 at PCIe 3.0 x1.
http://www.sunix.com/en/product_detail.php?cid=1&kid=2&gid=11&pid=1909

Some USB controllers (or mixed USB/SATA) use a PCIe 2.0 x4 switch with x4 upstream and two downstream x2 links where one or both are connected to an ASM1142.
https://www.sonnettech.com/product/allegro-pro-usb31-pcie.html

The ASM1142 is slow and limited but ok if you only want to give each one a single lane. If you have 2 or more lanes of PCIe 3.0, then the ASM2142 or ASM3142 or Thunderbolt add-in card are better (especially if you use more than one USB port).

The Node Duo is an example where the Thunderbolt controller is setup as PCIe 3.0 x2x2. I'm not sure if x2x1x1 is an option.
https://egpu.io/akitio-node-duo-review-two-slots-endless-possibilities/

I see in the bifurcation thread a pinout picture that includes REFCLK. Would that be transmitted through the USB 3.x cable using the USB 2.0 lines (D+, D-)? It's possible since REFCLK is 100 MHz and D+D- can do 480 Mbps. Maybe each lane requires a REFCLK. Or maybe it's not needed. Have to read the PCIe spec to be sure. Maybe you can't just wire the same refclk to multiple outputs - you need a clock buffer?

Looks like you have all the pieces you need to move ahead. I think you should try the fourth lane as well. Good luck on your testing.

@joevt

I did not realize the ASM1142 was capable of PCIe Gen3! Mine is correctly working in Gen2 mode according to the device manager (5GT/s).

About the 22Gb/s it's still obscure to me why Intel has set that limitation on a data only link (with no display link). Clearly it's a limitation set to the total data bandwidth, not specifically the GPU data. That's why I'm concern about congestion. I'm using the USB controller to plug my keyboard and mouse, and I don't want them to be lagged. I'll check if I experience this kind of behavior and I'll let you know.

If I've understand the specs correctly, the REFCLK signal needs to be multiplied using a dedicated chip before sending thru the USB cable (I think you're right about the use of the USB2.0 line used for that matter). That may be a problem because my plan was to do wiring only with no active parts involved. But I did not notice that kind of chip on the original daughter board! Is it possible that Razer got rid of that part? Maybe they use some kind of pinout tricks to avoid having to use it? Or maybe that's the reason why they use USB controller only on the daughter board by finding a way of making the tree controllers relying on a single REFCLK signal?

Anyway, thank you so much for your comment. I didn't know about the ASM1142 situation.

Do you think it is possible to use a PCIe switch to merge two upstream links? I guess the PCIe switch has to be configured that way beforehand?

            PRSNT1#     (tied to PRSNT2#, not USB?)

WAKE#       PERST#

            REFCLK+	
HSOp(0)     REFCLK−
HSOn(0)
            HSIp(0)
PRSNT2#     HSIn(0)     (tied to PRSNT1#, not USB?)

Correction: you did mention the ASM1061 SATA controller but not the FL1100 or ethernet controllers. I assume the FL1100 belongs to the Inatek KT4006 USB controller in your photo and the third slot shows the ethernet controller (but PCIe information doesn't show vendor/product ID - it's a Broadcom 10G controller?).