I really love my Razer Core v2… except for one or two little things that I just hate. The most obvious one is noise. Using a flexATX form factor has to be the worse choice possible for the PSU. So I've take the original PSU out and replace it by a standard ATX outside the case. Problem solved, it's now dead silent.
The other thing I dislike is the USB controller board. Razer chose to implement two daisy chained DSL6540 thunderbolt controllers to take a better advantage of the 40 Gbps bandwidth. Sounds great, except the second thunderbolt controller is only used by 3 PCIe to USB chips. One of them is tied to a USB to Ethernet controller than does not get along with macOS in gigabit mode. Why not use a PCIe to Ethernet directly like a RTL8111? I don't know.
The Ethernet problem putted aside, I've found that USB ports are not ideal for storage use case. I've tried all the USB to SATA controller I could fond at home (ASM235CM, JMB509, INIC3619), plus a brand new USB to NVMe (ASM2362) only to realize that USB will always be USB: unreliable and not so fast for storage. I've try for some time to boot my MacBook on Windows 10 stored on an external SSD using one of the USB ports of the razer core. I've kind of succeeded, with the windows-to-go feature. But it was not reliable (blue screens everywhere), and it was slow. One day I've ended up trying a Kanex Thunderbolt to eSATA adapter directly plugged to the macbook with light an day results. Even if the Kanex SATA controller (ASM1061) is bottlenecked by the PCIe 5GT/s (around 4GB/s) connectivity, it is both more reliable and way faster than USB3.0 (and does not need the windows-to-go feature to boot Windows).
I'm not saying USB is cannot handle Ethernet or storage. It's just not as reliable and performant than a dedicated chip specifically design for that. What is frustrating to me is that the DSL6540 present in the razer core can manage up to 4 PCIe links that could have been paired with those dedicated chips. Instead of 3 USB, I would rather have a single USB alongside SATA and Ethernet controllers. Is it just me?
So I've looked at this daughter board and noticed that, just like the new razer core X chroma, it look like a 4x PCIe card. Close inspection reveal the head of the PCIe connector match the PCIe specification in term of voltage. I supposed they used some kind of PCI bifurcation so I tried using the first lane with a PCIe 1x card and it worked!
I guess it is possible to use all 3 PCIe lanes with 3 different types of attached controllers. But it will require some time to get it to work because of the non-standard nature of this configuration. So for now I will only use the first lane and use a PCIe switch in order to multiply the PCIe slots available.
I tried two options: the first option is a daughter board that use a Pericom PI7C9X2G404SL and offer 3 PCIe 1x slots. The second option is a single PCIe card that use a ASM1182 PCIe switch and an ASM1042a USB controller and finally an ASM1061 SATA controller. So basically everything packed onto a single board with the same manufacturer for every integrated circuit mounted on it. This had the most chance of success… and it did not worked at all. No idea why. So back to plan A.
Of course all 3 PCIe slots will use a single PCIe Gen2 lane so basically 4Gbps. I've tried a simple sequential benchmark on the SSD thru a ASM1061 SATA controller that topped at 380MBps in read speed. It's about 75% of the theorical maximum speed: not so bad. I've also plugged a USB controller and an Ethernet card (that has no driver in macOS, great). All are correctly discovered by the OS.
That's all for now. I wanted to verify the feasibility of this concept before continuing. Now I'm thinking about how I'm going to fit it all into the case. This is going to be fun!
I'll keep you posted. Here are some pics:
MacBook Pro 2017 no-touchbar; Razer Core V2; RX 560
What is the original I/O board's PCIe arrangement? The review shows 3 ASM1142 controllers. Does one of them use two lanes? Or are they all single lane? Two lanes gives 8 Gbps instead of 7.877 Gbps. If they are all single lane, then you might have a 4th usable lane. Making the lanes usable by individual devices would require a custom PCB or some creative wiring - a PCIe slot has holes on the top where you might be able to connect wires to (4 wires for each lane - the number of super speed wires in a USB 3.x cable - I'm not sure if you need to connect the 3 or 4 ground lines per lane - check the wiring of the PCIe x1 to USB adapter that you have for the 3 slot daughter board - Is PRSNT#1 connected to PRSNT#2 at pin 31 on the I/O board?).
https://en.wikipedia.org/wiki/PCI_Express#Pinout
Maybe you can find x4 extenders and just use those wires, attach USB connectors, and connect to risers. I don't know if all risers are wired the same.
https://www.microsatacables.com/pcie-express-4x-riser-card-with-flexible-cable-12-inches-pci-e4x-rs-12
https://www.microsatacables.com/pcie-1x-to-pcie-16x-card-with-usb-30-cable-pci-e1x-16x-usb
https://www.microsatacables.com/pcie-express-1x-to-16x-riser-extender-card-pci-e1x-16x-ext
https://www.microsatacables.com/pcie-express-3-port-1x-multiplier-riser-card-pci-e1x-3pswt
Are the lanes divided by connecting individual lanes to different devices, or are they set that way by the firmware of the the Thunderbolt controller or the hardware on the main board? In other words, if you connected an x2 or x4 device, would it work at x2 or x4? Switches that have an x4 upstream link are much more expensive than the single lane upstream link that you have with the 3 slot daughter board so you might not want to go that way anyway.
Pending: Add my system information and expected eGPU configuration to my signature to give context to my posts
I was under the impression that each ASM1142 is using a single PCIe lane. Using a 2x lanes on one of the USB controller would not have made a lot of sense to me. Having 8GT/s for 2 USB3.0 ports is way enough. I think we have to remember that the overall bandwidth left for the daughter board is limited. Even if the eGPU does not consume 32Gbps all the time I think it is safe to assume that going over 8Gbps on the daughter board is not « safe » since it could lead to congestion. In that case, the dual thunderbolt configuration is supposed to prioritize the thunderbolt controller that is closer to the MacBook, so the GPU, but I would not rely on that.
That’s being said, I would love to try to play with the PCIe bifurcation of the Razer Core.
About bandwidth, here are some thoughts:
- Using USB3.0 cables to transmit PCIe is possible even if it’s not designed for that. My test bench uses this method that is wildly use by GPU miners. It’s pretty reliable with PCIe Gen2 since it’s 5GT/s when the cable is designed to transmit 5Gb/s. Regarding PCIe Gen3, that’s 8GT/s so… let’s say the cable is not designed for that.
- All the 3 PCIe cards I used in my test are PCIe Gen2 anyway. When a Gen2 card is plugged in a Gen3 slot, the controller automatically use Gen2 and slow down the signaling.
- By my understanding of PCIe, it’s not possible to use two 1x lanes together to form a 2x slot. I’ll have to study the DSL6540 datasheet but I‘m pretty sure the integrated PCIe switch is configurable in two modes : 4x and 1x1x1x1x (1 big link vs 4 small links). I’m not sure about 1x1x2x. I guess this PCIe configuration can be set by changing a value in a registry (in a ROM somewhere near the controller) or maybe by changing a pinout wiring (so soldered on the board). I’m pretty sure that configuration cannot be changed dynamically.
- Quick recap on the bandwidth available on PCIe Gen2 and Gen3. Those don’t use the same encoding and that's why the ratio between signaling speed and real data speed are different. PCIe Gen2 use 8b/10b encoding meaning you have to multiply the transfer rate by 80% to get the real throughput. The PCIe Gen3 (and above) use 128b/130b so it’s around 98%. Because it's so close to 100% a lot of people intentionally (or not) use transmission rate and throughput indifferently. So that’s 4Gb/s and 7,88Gb/s real throughput respectively per lane for PCIe Gen2 and Gen3.
My idea is to finalize my testing in order to definitely take out the original daughter board. That way, and because I use my setup every day, I'll have more time with the study of the board without having to tear down my Razer Core and make my setup unusable. Then I'll work on bifurcation. I plan to make a special PCIe riser with PCIe 4x on one end and tree PCIe 1x in a form of tree USB3.0 connector on the other end. I'll use a PCIe 4x extension ribbon and tree USB3.0 PCB modules.
I highly recommend this awesome tread forum on PCIe bifurcation and this video from Intel on that topic.
MacBook Pro 2017 no-touchbar; Razer Core V2; RX 560
ASM1142 is a USB 3.1 gen 2 controller. If it's only operating at gen 1 speed (5 Gbps -> 4 Gbps), then you can update the firmware to allow gen 2 speed (10 Gbps -> 9.697 Gbps). Note that the controller usually has a bug where it reports gen 1 speed even when it's communicating at gen 2 speed (you need a device capable of more than 500 MB/s to show this). The ASM1142 is limited to PCIe 3.0 x1 (7.877 Gbps) or PCIe 2.0 x2 (8 Gbps) so you can't get full gen 2 performance. Using 2 lanes is slightly better than using one line (by 123 Mbps) or they can choose to waste a lane.
The eGPU is limited to 22 Gbps (can't get more than 2750 MB/s). I guess the USB controllers also share that limit. I don't know about congestion. Usually I would not be doing a lot of accessing a USB drive while playing a game.
There is nothing stopping the USB cables from being able to transmit PCIe 3.0 (8 GT/s) except bad quality that causes errors. If a device allows it, the Thunderbolt controller will try to negotiate 8 GT/s and fall back to 5 GT/s if link training fails. Worst case, it could drop to 2.5 GT/s. Like you said, all your devices (and the The PI7C9X2G404SL of the 3 slot daughter board) are limited to 5 GT/s.
There are many ASM1142 based USB cards that are PCIe 2.0 x2 electrically, and PCIe x4 physically (the StartTech product pages are bad at explaining this).
https://www.startech.com/Cards-Adapters/USB-3.0/Cards/2-port-usb-3-1-card~PEXUSB311EI
The Sunix UPD2018 uses an ASM1142 at PCIe 3.0 x1.
http://www.sunix.com/en/product_detail.php?cid=1&kid=2&gid=11&pid=1909
Some USB controllers (or mixed USB/SATA) use a PCIe 2.0 x4 switch with x4 upstream and two downstream x2 links where one or both are connected to an ASM1142.
https://www.sonnettech.com/product/allegro-pro-usb31-pcie.html
The ASM1142 is slow and limited but ok if you only want to give each one a single lane. If you have 2 or more lanes of PCIe 3.0, then the ASM2142 or ASM3142 or Thunderbolt add-in card are better (especially if you use more than one USB port).
The Node Duo is an example where the Thunderbolt controller is setup as PCIe 3.0 x2x2. I'm not sure if x2x1x1 is an option.
https://egpu.io/akitio-node-duo-review-two-slots-endless-possibilities/
I see in the bifurcation thread a pinout picture that includes REFCLK. Would that be transmitted through the USB 3.x cable using the USB 2.0 lines (D+, D-)? It's possible since REFCLK is 100 MHz and D+D- can do 480 Mbps. Maybe each lane requires a REFCLK. Or maybe it's not needed. Have to read the PCIe spec to be sure. Maybe you can't just wire the same refclk to multiple outputs - you need a clock buffer?
Looks like you have all the pieces you need to move ahead. I think you should try the fourth lane as well. Good luck on your testing.
Pending: Add my system information and expected eGPU configuration to my signature to give context to my posts
I did not realize the ASM1142 was capable of PCIe Gen3! Mine is correctly working in Gen2 mode according to the device manager (5GT/s).
About the 22Gb/s it's still obscure to me why Intel has set that limitation on a data only link (with no display link). Clearly it's a limitation set to the total data bandwidth, not specifically the GPU data. That's why I'm concern about congestion. I'm using the USB controller to plug my keyboard and mouse, and I don't want them to be lagged. I'll check if I experience this kind of behavior and I'll let you know.
If I've understand the specs correctly, the REFCLK signal needs to be multiplied using a dedicated chip before sending thru the USB cable (I think you're right about the use of the USB2.0 line used for that matter). That may be a problem because my plan was to do wiring only with no active parts involved. But I did not notice that kind of chip on the original daughter board! Is it possible that Razer got rid of that part? Maybe they use some kind of pinout tricks to avoid having to use it? Or maybe that's the reason why they use USB controller only on the daughter board by finding a way of making the tree controllers relying on a single REFCLK signal?
Anyway, thank you so much for your comment. I didn't know about the ASM1142 situation.
Do you think it is possible to use a PCIe switch to merge two upstream links? I guess the PCIe switch has to be configured that way beforehand?
MacBook Pro 2017 no-touchbar; Razer Core V2; RX 560
PRSNT1# (tied to PRSNT2#, not USB?)
WAKE# PERST#
REFCLK+
HSOp(0) REFCLK−
HSOn(0)
HSIp(0)
PRSNT2# HSIn(0) (tied to PRSNT1#, not USB?)
Correction: you did mention the ASM1061 SATA controller but not the FL1100 or ethernet controllers. I assume the FL1100 belongs to the Inatek KT4006 USB controller in your photo and the third slot shows the ethernet controller (but PCIe information doesn't show vendor/product ID - it's a Broadcom 10G controller?).
Pending: Add my system information and expected eGPU configuration to my signature to give context to my posts