Enclosure & Hardware Discussions
Four PCIe slots into the Razer Core v2???
 

Four PCIe slots into the Razer Core v2???  

  RSS

lefuturdelavenir
(@lefuturdelavenir)
Active Member
Joined: 1 year ago
 

I really love my Razer Core v2… except for one or two little things that I just hate. The most obvious one is noise. Using a flexATX form factor has to be the worse choice possible for the PSU. So I've take the original PSU out and replace it by a standard ATX outside the case. Problem solved, it's now dead silent.

The other thing I dislike is the USB controller board. Razer chose to implement two daisy chained DSL6540 thunderbolt controllers to take a better advantage of the 40 Gbps bandwidth. Sounds great, except the second thunderbolt controller is only used by 3 PCIe to USB chips. One of them is tied to a USB to Ethernet controller than does not get along with macOS in gigabit mode. Why not use a PCIe to Ethernet directly like a RTL8111? I don't know.

The Ethernet problem putted aside, I've found that USB ports are not ideal for storage use case. I've tried all the USB to SATA controller I could fond at home (ASM235CM, JMB509, INIC3619), plus a brand new USB to NVMe (ASM2362) only to realize that USB will always be USB: unreliable and not so fast for storage. I've try for some time to boot my MacBook on Windows 10 stored on an external SSD using one of the USB ports of the razer core. I've kind of succeeded, with the windows-to-go feature. But it was not reliable (blue screens everywhere), and it was slow. One day I've ended up trying a Kanex Thunderbolt to eSATA adapter directly plugged to the macbook with light an day results. Even if the Kanex SATA controller (ASM1061) is bottlenecked by the PCIe 5GT/s (around 4GB/s) connectivity, it is both more reliable and way faster than USB3.0 (and does not need the windows-to-go feature to boot Windows).

I'm not saying USB is cannot handle Ethernet or storage. It's just not as reliable and performant than a dedicated chip specifically design for that. What is frustrating to me is that the DSL6540 present in the razer core can manage up to 4 PCIe links that could have been paired with those dedicated chips. Instead of 3 USB, I would rather have a single USB alongside SATA and Ethernet controllers. Is it just me?

So I've looked at this daughter board and noticed that, just like the new razer core X chroma, it look like a 4x PCIe card. Close inspection reveal the head of the PCIe connector match the PCIe specification in term of voltage. I supposed they used some kind of PCI bifurcation so I tried using the first lane with a PCIe 1x card and it worked!

I guess it is possible to use all 3 PCIe lanes with 3 different types of attached controllers. But it will require some time to get it to work because of the non-standard nature of this configuration. So for now I will only use the first lane and use a PCIe switch in order to multiply the PCIe slots available.

I tried two options: the first option is a daughter board that use a Pericom PI7C9X2G404SL and offer 3 PCIe 1x slots. The second option is a single PCIe card that use a ASM1182 PCIe switch and an ASM1042a USB controller and finally an ASM1061 SATA controller. So basically everything packed onto a single board with the same manufacturer for every integrated circuit mounted on it. This had the most chance of success… and it did not worked at all. No idea why. So back to plan A.

Of course all 3 PCIe slots will use a single PCIe Gen2 lane so basically 4Gbps. I've tried a simple sequential benchmark on the SSD thru a ASM1061 SATA controller that topped at 380MBps in read speed. It's about 75% of the theorical maximum speed: not so bad. I've also plugged a USB controller and an Ethernet card (that has no driver in macOS, great). All are correctly discovered by the OS.

That's all for now. I wanted to verify the feasibility of this concept before continuing. Now I'm thinking about how I'm going to fit it all into the case. This is going to be fun!

I'll keep you posted. Here are some pics:

Screenshot 2019 09 08 at 15.03.56
New 19 09 08 15 42 42 0079.heic
New 19 09 08 15 42 29 0078.heic
New 19 09 08 15 42 03 0077.heic
New 19 09 08 15 41 21 0076.heic

This topic was modified 1 month ago

MacBook Pro 2017 no-touchbar; Razer Core V2; RX 560


itsage and mac_editor liked
ReplyQuote
itsage
(@itsage)
Famed Member Admin
Joined: 3 years ago
 

@lefuturdelavenir Amazing findings! Please post updates as you go.

Best ultrabooks for eGPU use

eGPU enclosure buying guide


ReplyQuote
joevt
(@joevt)
Honorable Member
Joined: 2 years ago
 

@lefuturdelavenir

What is the original I/O board's PCIe arrangement? The review shows 3 ASM1142 controllers. Does one of them use two lanes? Or are they all single lane? Two lanes gives 8 Gbps instead of 7.877 Gbps. If they are all single lane, then you might have a 4th usable lane. Making the lanes usable by individual devices would require a custom PCB or some creative wiring - a PCIe slot has holes on the top where you might be able to connect wires to (4 wires for each lane - the number of super speed wires in a USB 3.x cable - I'm not sure if you need to connect the 3 or 4 ground lines per lane - check the wiring of the PCIe x1 to USB adapter that you have for the 3 slot daughter board - Is PRSNT#1 connected to PRSNT#2 at pin 31 on the I/O board?).
https://en.wikipedia.org/wiki/PCI_Express#Pinout

Maybe you can find x4 extenders and just use those wires, attach USB connectors, and connect to risers. I don't know if all risers are wired the same.
https://www.microsatacables.com/pcie-express-4x-riser-card-with-flexible-cable-12-inches-pci-e4x-rs-12
https://www.microsatacables.com/pcie-1x-to-pcie-16x-card-with-usb-30-cable-pci-e1x-16x-usb
https://www.microsatacables.com/pcie-express-1x-to-16x-riser-extender-card-pci-e1x-16x-ext
https://www.microsatacables.com/pcie-express-3-port-1x-multiplier-riser-card-pci-e1x-3pswt

Are the lanes divided by connecting individual lanes to different devices, or are they set that way by the firmware of the the Thunderbolt controller or the hardware on the main board? In other words, if you connected an x2 or x4 device, would it work at x2 or x4? Switches that have an x4 upstream link are much more expensive than the single lane upstream link that you have with the 3 slot daughter board so you might not want to go that way anyway.

Pending: Add my system information and expected eGPU configuration to my signature to give context to my posts


ReplyQuote
lefuturdelavenir
(@lefuturdelavenir)
Active Member
Joined: 1 year ago
 

@joevt

I was under the impression that each ASM1142 is using a single PCIe lane. Using a 2x lanes on one of the USB controller would not have made a lot of sense to me. Having 8GT/s for 2 USB3.0 ports is way enough. I think we have to remember that the overall bandwidth left for the daughter board is limited. Even if the eGPU does not consume 32Gbps all the time I think it is safe to assume that going over 8Gbps on the daughter board is not « safe » since it could lead to congestion. In that case, the dual thunderbolt configuration is supposed to prioritize the thunderbolt controller that is closer to the MacBook, so the GPU, but I would not rely on that.

That’s being said, I would love to try to play with the PCIe bifurcation of the Razer Core.

About bandwidth, here are some thoughts:

  • Using USB3.0 cables to transmit PCIe is possible even if it’s not designed for that. My test bench uses this method that is wildly use by GPU miners. It’s pretty reliable with PCIe Gen2 since it’s 5GT/s when the cable is designed to transmit 5Gb/s. Regarding PCIe Gen3, that’s 8GT/s so… let’s say the cable is not designed for that.
  • All the 3 PCIe cards I used in my test are PCIe Gen2 anyway. When a Gen2 card is plugged in a Gen3 slot, the controller automatically use Gen2 and slow down the signaling.
  • By my understanding of PCIe, it’s not possible to use two 1x lanes together to form a 2x slot. I’ll have to study the DSL6540 datasheet but I‘m pretty sure the integrated PCIe switch is configurable in two modes : 4x and 1x1x1x1x (1 big link vs 4 small links). I’m not sure about 1x1x2x. I guess this PCIe configuration can be set by changing a value in a registry (in a ROM somewhere near the controller) or maybe by changing a pinout wiring (so soldered on the board). I’m pretty sure that configuration cannot be changed dynamically.
  • Quick recap on the bandwidth available on PCIe Gen2 and Gen3. Those don’t use the same encoding and that's why the ratio between signaling speed and real data speed are different. PCIe Gen2 use 8b/10b encoding meaning you have to multiply the transfer rate by 80% to get the real throughput. The PCIe Gen3 (and above) use 128b/130b so it’s around 98%. Because it's so close to 100% a lot of people intentionally (or not) use transmission rate and throughput indifferently. So that’s 4Gb/s and 7,88Gb/s real throughput respectively per lane for PCIe Gen2 and Gen3.

My idea is to finalize my testing in order to definitely take out the original daughter board. That way, and because I use my setup every day, I'll have more time with the study of the board without having to tear down my Razer Core and make my setup unusable. Then I'll work on bifurcation. I plan to make a special PCIe riser with PCIe 4x on one end and tree PCIe 1x in a form of tree USB3.0 connector on the other end. I'll use a PCIe 4x extension ribbon and tree USB3.0 PCB modules.

New 19 09 14 12 49 31 0082.heic

I highly recommend this awesome tread forum on PCIe bifurcation and this video from Intel on that topic.

MacBook Pro 2017 no-touchbar; Razer Core V2; RX 560


ReplyQuote
joevt
(@joevt)
Honorable Member
Joined: 2 years ago
 

@lefuturdelavenir

ASM1142 is a USB 3.1 gen 2 controller. If it's only operating at gen 1 speed (5 Gbps -> 4 Gbps), then you can update the firmware to allow gen 2 speed (10 Gbps -> 9.697 Gbps). Note that the controller usually has a bug where it reports gen 1 speed even when it's communicating at gen 2 speed (you need a device capable of more than 500 MB/s to show this). The ASM1142 is limited to PCIe 3.0 x1 (7.877 Gbps) or PCIe 2.0 x2 (8 Gbps) so you can't get full gen 2 performance. Using 2 lanes is slightly better than using one line (by 123 Mbps) or they can choose to waste a lane.

The eGPU is limited to 22 Gbps (can't get more than 2750 MB/s). I guess the USB controllers also share that limit. I don't know about congestion. Usually I would not be doing a lot of accessing a USB drive while playing a game.

There is nothing stopping the USB cables from being able to transmit PCIe 3.0 (8 GT/s) except bad quality that causes errors. If a device allows it, the Thunderbolt controller will try to negotiate 8 GT/s and fall back to 5 GT/s if link training fails. Worst case, it could drop to 2.5 GT/s. Like you said, all your devices (and the The PI7C9X2G404SL of the 3 slot daughter board) are limited to 5 GT/s.

There are many ASM1142 based USB cards that are PCIe 2.0 x2 electrically, and PCIe x4 physically (the StartTech product pages are bad at explaining this).
https://www.startech.com/Cards-Adapters/USB-3.0/Cards/2-port-usb-3-1-card~PEXUSB311EI

The Sunix UPD2018 uses an ASM1142 at PCIe 3.0 x1.
http://www.sunix.com/en/product_detail.php?cid=1&kid=2&gid=11&pid=1909

Some USB controllers (or mixed USB/SATA) use a PCIe 2.0 x4 switch with x4 upstream and two downstream x2 links where one or both are connected to an ASM1142.
https://www.sonnettech.com/product/allegro-pro-usb31-pcie.html

The ASM1142 is slow and limited but ok if you only want to give each one a single lane. If you have 2 or more lanes of PCIe 3.0, then the ASM2142 or ASM3142 or Thunderbolt add-in card are better (especially if you use more than one USB port).

The Node Duo is an example where the Thunderbolt controller is setup as PCIe 3.0 x2x2. I'm not sure if x2x1x1 is an option.
https://egpu.io/akitio-node-duo-review-two-slots-endless-possibilities/

I see in the bifurcation thread a pinout picture that includes REFCLK. Would that be transmitted through the USB 3.x cable using the USB 2.0 lines (D+, D-)? It's possible since REFCLK is 100 MHz and D+D- can do 480 Mbps. Maybe each lane requires a REFCLK. Or maybe it's not needed. Have to read the PCIe spec to be sure. Maybe you can't just wire the same refclk to multiple outputs - you need a clock buffer?

Looks like you have all the pieces you need to move ahead. I think you should try the fourth lane as well. Good luck on your testing.

Pending: Add my system information and expected eGPU configuration to my signature to give context to my posts


ReplyQuote
lefuturdelavenir
(@lefuturdelavenir)
Active Member
Joined: 1 year ago
 

@joevt

I did not realize the ASM1142 was capable of PCIe Gen3! Mine is correctly working in Gen2 mode according to the device manager (5GT/s).

Screenshot 2019 09 14 at 17.25.09

About the 22Gb/s it's still obscure to me why Intel has set that limitation on a data only link (with no display link). Clearly it's a limitation set to the total data bandwidth, not specifically the GPU data. That's why I'm concern about congestion. I'm using the USB controller to plug my keyboard and mouse, and I don't want them to be lagged. I'll check if I experience this kind of behavior and I'll let you know.

If I've understand the specs correctly, the REFCLK signal needs to be multiplied using a dedicated chip before sending thru the USB cable (I think you're right about the use of the USB2.0 line used for that matter). That may be a problem because my plan was to do wiring only with no active parts involved. But I did not notice that kind of chip on the original daughter board! Is it possible that Razer got rid of that part? Maybe they use some kind of pinout tricks to avoid having to use it? Or maybe that's the reason why they use USB controller only on the daughter board by finding a way of making the tree controllers relying on a single REFCLK signal?

Anyway, thank you so much for your comment. I didn't know about the ASM1142 situation.

Do you think it is possible to use a PCIe switch to merge two upstream links? I guess the PCIe switch has to be configured that way beforehand?

MacBook Pro 2017 no-touchbar; Razer Core V2; RX 560


ReplyQuote
joevt
(@joevt)
Honorable Member
Joined: 2 years ago
 
 
Your screenshot shows FL1100 USB controller and ASM1061 SATA controller (not a card that you mentioned before?) instead of the ASM1142 USB controller.
 
About the 22 Gbps limit: I tried looking at the USB4 spec for a clue about that. It's very long and I didn't read the entire thing but they did mention balancing latency and performance for PCIe (and USB4 in the case of a USB4 device) when choosing a number of buffers.
GPU data is PCIe data. I don't think there's a distinction. We have seen different eGPU boxes having different max bandwidths just by changing their firmware. I think the firmwares affects both GPU and non GPU PCIe devices.
 
I think the refclk, if it's used, would be multiplied after the USB cable by the PCIe device or something on the PCIe card. Anyway, check continuity between PCIe connector pins and USB connector pins. With a single slot riser, you could do that at either end of the riser. With your 3 slot riser, just check the PCIe 1x side.
 
I can see by looking at a single slot riser the following signals are used (maybe 8 are transmitted over USB, and maybe grounds are all connected to shell of USB).
            PRSNT1#     (tied to PRSNT2#, not USB?)

WAKE#       PERST#

            REFCLK+	
HSOp(0)     REFCLK−
HSOn(0)
            HSIp(0)
PRSNT2#     HSIn(0)     (tied to PRSNT1#, not USB?)
I wonder how the I/O card handles PRSNT1#, WAKE#, PERST#, REFCLK+, REFCLK−, PRSNT2# for multiple ASM1142 controllers...
 
I think a PCIe switch cannot make a single link from two links.
 

Correction: you did mention the ASM1061 SATA controller but not the FL1100 or ethernet controllers. I assume the FL1100 belongs to the Inatek KT4006 USB controller in your photo and the third slot shows the ethernet controller (but PCIe information doesn't show vendor/product ID - it's a Broadcom 10G controller?).

This post was modified 1 month ago

Pending: Add my system information and expected eGPU configuration to my signature to give context to my posts


ReplyQuote