Schwarzgerät III

(8 intermediate revisions by the same user not shown)

Line 191:

* [https://www.elegoo.com/products/elegoo-mega-2560-r3-board ELEGOO MEGA 2560] Revision 3, connected to NXZT internal USB hub, mounted to back of PSU chamber

As I detailed regarding [[Schwarzgerat II]], the 3990X is an amazing achievement in chip design and fabrication, but I believe it to be severely starved for many tasks by its memory bandwidth; with its four memory channels populated, the ThreadRipper 3990X can hit about 90GB/s from fast DDR4; its ~~Epyc~~ brother can pull down ~190 through its eight channels. For my tasks, it's rare enough that I can drive all my 32 cores; with the 3990X, I'd be paying twice as much to hit full utilization less often, and be unable to bring full bandwidth to bear when I did.

As I detailed regarding [[Schwarzgerat II]], the 3990X is an amazing achievement in chip design and fabrication, but I believe it to be severely starved for many tasks by its memory bandwidth; with its four memory channels populated, the ThreadRipper 3990X can hit about 90GB/s from fast DDR4; its EPYC brother can pull down ~190 through its eight channels. For my tasks, it's rare enough that I can drive all my 32 cores; with the 3990X, I'd be paying twice as much to hit full utilization less often, and be unable to bring full bandwidth to bear when I did.

I absolutely 🖤 my 3970X, though. Bitch screams. Anyone overclocking on Linux should be aware of [https://www.linux.org/docs/man8/turbostat.html turbostat]. The 3970X supports 88 lanes of PCIe 4.0, of which the TRX40 chipset consumes 24, leaving 64 for expansion devices. I've got 16 (GPU) + 16 (Hyper X) + 2x8 (LSI cards) + 12 (M.2 onboards) for 60 total, coming in just under saturation.

Line 224:

{| class="wikitable"

! Name !! Setup !! Availability !! MaxTol !! MinFail !! Max Rebuild ~~!! Perf~~

! Name !! Setup !! Availability !! MaxTol !! MinFail !! Max Rebuild

|-

| raid3z

Line 232:

| 4

| 100% + fail

|

|-

| raid2z

Line 240:

Line 239:

| 3

| 85.7% + fail

|

|-

| striped raid3z

Line 248:

Line 246:

| 4

| 50% + fail

|

|-

| striped raid2z

Line 256:

Line 253:

| 3

| 50% + fail

|

|-

| striped raidz

Line 264:

Line 260:

| 2

| 50% + fail

|

|-

| mirrored raid3z

Line 272:

Line 267:

| 8

| 50% + fail

|

|-

| mirrored raid2z

Line 280:

Line 274:

| 6

| 50% + fail

|

|-

| mirrored raidz

Line 288:

Line 281:

| 4

| 50% + fail

|

|-

| raidz3 of stripes

Line 303:

Line 295:

| 3

| 85.7% + fail

|

|}

Line 331:

Line 322:

==Distributing power==

[[File:Tobsun.jpg|200px|thumb|A 12V->5V buck transformer~~. By upping the output amperage~~, ~~it is~~ more efficient than a voltage regulator.]]

[[File:Tobsun.jpg|200px|thumb|A 12V->5V buck transformer, more efficient than a mere voltage regulator.]]

I began to run into some serious power issues on this build, originating in the Exos X18 drives (of which, you might remember, there are 14). It will be worth your time to consult the [https://www.seagate.com/www-content/product-content/enterprise-hdd-fam/exos-x18/_shared/en-us/docs/100865854a.pdf Exos 18 manual]. Remember, 12V is for the motor, and 5V is for the logic.

Line 511:

Line 502:

| Perif (Molex) || 0 || 55 || 132 || Can bridge to SATA or PCIe. Wiring might not be safe for the full pin capacity.

|-

| PCIe x4|| 9 || 0 || 66 || Can bridge to (underpowered) 12V-only (2-pin) Molex. Can probably bridge (with buck transformer) to underpowered Molex.

| PCIe x4|| 0 || 0 || 75 || Can bridge to (underpowered) 12V-only (2-pin) Molex. Can probably bridge (with buck transformer) to underpowered Molex.

|-

|}

Line 574:

Line 565:

==Ongoing issues==

* My flow rate is lower than I would like, assuming this flowmeter to be reliable. Maxing out the PWM pump hits desirable flow rates, but with more noise than I want. I think there might be some flow reduction in the tubing between the flowmeter and Quantum Kinetic; I'm considering replacing that with hardline. For all I know, there's still a big air pocket in some radiator.

* <s>My flow rate is lower than I would like, assuming this flowmeter to be reliable. Maxing out the PWM pump hits desirable flow rates, but with more noise than I want. I think there might be some flow reduction in the tubing between the flowmeter and Quantum Kinetic; I'm considering replacing that with hardline. For all I know, there's still a big air pocket in some radiator.</s> there was indeed a big air pocket

* <s>My onboard Ethernet stopped working, and I'm still not sure why.</s> the ethernet cable wasn't plugged into the outlet all the way, lol, you dumb idiot

* <s>I'm getting a kernel oops in the igb driver on startup, not sure why.</s>

* I have to unplug my keyboard and plug it back in on each boot =/ (this was also happening before). No, this is not fixed by changing the state of XHCI Handoff in my firmware.

* <s>I have to unplug my keyboard and plug it back in on each boot =/ (this was also happening before). No, this is not fixed by changing the state of XHCI Handoff in my firmware.</s> <tt>powertop</tt> was being invoked on each boot, turning on autosuspend.

* I'm not sure this 4x140mm structure of fans in the roof is really doing me any good, and need to test with and without them. What really happened here is I saw [[Noctua]] had a chromax.black 140mm and creamed my jeans and was like "gotta order that!" Then I realized I had no 140mm mounts, and rather than do something sensible, I built up an autistic rage and was like OH FUCK YOU GOD, YOU TRY TO FUCK ME, NAH, I'LL JUST BUY THREE MORE FANS AND THREE-DEE PRINT MYSELF A BARUD-DÛR AND IT'LL INTERLOCK BECAUSE I TOO AM A MASTER OF REALITY, YOU OLD SHITTER, THEN I'LL GO BACK IN TIME AND KILL MY PARENTS BEFORE I CAN BE BORN AND UNDO ALL YOUR WORKS BITCH and then it was like, well, better use these fans I guess.

* <s>I'm not sure this 4x140mm structure of fans in the roof is really doing me any good, and need to test with and without them. What really happened here is I saw [[Noctua]] had a chromax.black 140mm and creamed my jeans and was like "gotta order that!" Then I realized I had no 140mm mounts, and rather than do something sensible, I built up an autistic rage and was like OH FUCK YOU GOD, YOU TRY TO FUCK ME, NAH, I'LL JUST BUY THREE MORE FANS AND THREE-DEE PRINT MYSELF A BARUD-DÛR AND IT'LL INTERLOCK BECAUSE I TOO AM A MASTER OF REALITY, YOU OLD SHITTER, THEN I'LL GO BACK IN TIME AND KILL MY PARENTS BEFORE I CAN BE BORN AND UNDO ALL YOUR WORKS BITCH and then it was like, well, better use these fans I guess.</s> they were indeed doing me no good, and have been replaced with a [[inaMORAta|MoRa-3]].

* Meanwhile, I don't think there's enough inflow. Need to test with the door open.

* <s>Meanwhile, I don't think there's enough inflow. Need to test with the door open.</s> remedied.

* Apparently Type-C USB Power Delivery runs at 20 and 48 volts. No idea if I'm supplying this correctly, or if it's even in play.

Line 598:

Line 589:

[[File:Ohyeah.jpg|center]]

That'll do, ~~machine~~. That'll do.

That'll do, pig. That'll do.

==Future directions==

Line 610:

Line 601:

<li>LoRa. [[LoRa]] is a long-range, low-bandwidth radio protocol. I could bring an antenna out, and use the Arduino together with a LoRa chip.</li>

<li>Battery for the CCFL. It would be nice to have some light when I'm working inside the machine. If I could provide selectable battery-based backup for these rods, that would be useful.</li>

<li>PID control for fans/pumps. The Proportional-Integral-Derivative controller is a simple feedback mechanism that I suspect would work well with fans and pumps. I don't care how many RPM my fans are spinning at; what I care about is how warm my coolant and components are (and noise). I'd like to set up target ΔTs (as a function of ambient temp) and a target noise ceiling, and use an inline sensor, an ambient sensor, and an acoustic sensor in combination to manage my loop's active components.</li>

<li>PID control for fans/pumps. The Proportional-Integral-Derivative controller is a simple feedback mechanism that I suspect would work well with fans and pumps. I don't care how many RPM my fans are spinning at; what I care about is how warm my coolant and components are (and noise). I'd like to set up target ΔTs (as a function of ambient temp) and a target noise ceiling, and use an inline sensor, an ambient sensor, and an acoustic sensor in combination to manage my loop's active components. update: see my [[Counterforce]] project, which does all this and much more!</li>

</ul>

Schwarzgerät III: Difference between revisions

Dank (talk | contribs)

@@ Line 191: / Line 191: @@
 * [https://www.elegoo.com/products/elegoo-mega-2560-r3-board ELEGOO MEGA 2560] Revision 3, connected to NXZT internal USB hub, mounted to back of PSU chamber
-As I detailed regarding [[Schwarzgerat II]], the 3990X is an amazing achievement in chip design and fabrication, but I believe it to be severely starved for many tasks by its memory bandwidth; with its four memory channels populated, the ThreadRipper 3990X can hit about 90GB/s from fast DDR4; its Epyc brother can pull down ~190 through its eight channels. For my tasks, it's rare enough that I can drive all my 32 cores; with the 3990X, I'd be paying twice as much to hit full utilization less often, and be unable to bring full bandwidth to bear when I did.
+As I detailed regarding [[Schwarzgerat II]], the 3990X is an amazing achievement in chip design and fabrication, but I believe it to be severely starved for many tasks by its memory bandwidth; with its four memory channels populated, the ThreadRipper 3990X can hit about 90GB/s from fast DDR4; its EPYC brother can pull down ~190 through its eight channels. For my tasks, it's rare enough that I can drive all my 32 cores; with the 3990X, I'd be paying twice as much to hit full utilization less often, and be unable to bring full bandwidth to bear when I did.
 I absolutely 🖤 my 3970X, though. Bitch screams. Anyone overclocking on Linux should be aware of [https://www.linux.org/docs/man8/turbostat.html turbostat]. The 3970X supports 88 lanes of PCIe 4.0, of which the TRX40 chipset consumes 24, leaving 64 for expansion devices. I've got 16 (GPU) + 16 (Hyper X) + 2x8 (LSI cards) + 12 (M.2 onboards) for 60 total, coming in just under saturation.
@@ Line 224: / Line 224: @@
 {| class="wikitable"
-! Name !! Setup !! Availability !! MaxTol !! MinFail !! Max Rebuild !! Perf
+! Name !! Setup !! Availability !! MaxTol !! MinFail !! Max Rebuild
 |-
 | raid3z
@@ Line 232: / Line 232: @@
 | 4
 | 100% + fail
-|
 |-
 | raid2z
@@ Line 240: / Line 239: @@
 | 3
 | 85.7% + fail
-|
 |-
 | striped raid3z
@@ Line 248: / Line 246: @@
 | 4
 | 50% + fail
-|
 |-
 | striped raid2z
@@ Line 256: / Line 253: @@
 | 3
 | 50% + fail
-|
 |-
 | striped raidz
@@ Line 264: / Line 260: @@
 | 2
 | 50% + fail
-|
 |-
 | mirrored raid3z
@@ Line 272: / Line 267: @@
 | 8
 | 50% + fail
-|
 |-
 | mirrored raid2z
@@ Line 280: / Line 274: @@
 | 6
 | 50% + fail
-|
 |-
 | mirrored raidz
@@ Line 288: / Line 281: @@
 | 4
 | 50% + fail
-|
 |-
 | raidz3 of stripes
@@ Line 303: / Line 295: @@
 | 3
 | 85.7% + fail
-|
 |}
@@ Line 331: / Line 322: @@
 ==Distributing power==
-[[File:Tobsun.jpg|200px|thumb|A 12V->5V buck transformer. By upping the output amperage, it is more efficient than a voltage regulator.]]
+[[File:Tobsun.jpg|200px|thumb|A 12V->5V buck transformer, more efficient than a mere voltage regulator.]]
 I began to run into some serious power issues on this build, originating in the Exos X18 drives (of which, you might remember, there are 14). It will be worth your time to consult the [https://www.seagate.com/www-content/product-content/enterprise-hdd-fam/exos-x18/_shared/en-us/docs/100865854a.pdf Exos 18 manual]. Remember, 12V is for the motor, and 5V is for the logic.
@@ Line 511: / Line 502: @@
 | Perif (Molex) || 0 || 55 || 132 || Can bridge to SATA or PCIe. Wiring might not be safe for the full pin capacity.
 |-
-| PCIe x4|| 9 || 0 || 66 || Can bridge to (underpowered) 12V-only (2-pin) Molex. Can probably bridge (with buck transformer) to underpowered Molex.
+| PCIe x4|| 0 || 0 || 75 || Can bridge to (underpowered) 12V-only (2-pin) Molex. Can probably bridge (with buck transformer) to underpowered Molex.
 |-
 |}
@@ Line 574: / Line 565: @@
 ==Ongoing issues==
-* My flow rate is lower than I would like, assuming this flowmeter to be reliable. Maxing out the PWM pump hits desirable flow rates, but with more noise than I want. I think there might be some flow reduction in the tubing between the flowmeter and Quantum Kinetic; I'm considering replacing that with hardline. For all I know, there's still a big air pocket in some radiator.
+* <s>My flow rate is lower than I would like, assuming this flowmeter to be reliable. Maxing out the PWM pump hits desirable flow rates, but with more noise than I want. I think there might be some flow reduction in the tubing between the flowmeter and Quantum Kinetic; I'm considering replacing that with hardline. For all I know, there's still a big air pocket in some radiator.</s> <b>there was indeed a big air pocket</b>
 * <s>My onboard Ethernet stopped working, and I'm still not sure why.</s> <b>the ethernet cable wasn't plugged into the outlet all the way, lol, you dumb idiot</b>
 * <s>I'm getting a kernel oops in the igb driver on startup, not sure why.</s>
-* I have to unplug my keyboard and plug it back in on each boot =/ (this was also happening before). No, this is not fixed by changing the state of XHCI Handoff in my firmware.
+* <s>I have to unplug my keyboard and plug it back in on each boot =/ (this was also happening before). No, this is not fixed by changing the state of XHCI Handoff in my firmware.</s> <tt>powertop</tt> was being invoked on each boot, turning on autosuspend.
-* I'm not sure this 4x140mm structure of fans in the roof is really doing me any good, and need to test with and without them. What really happened here is I saw [[Noctua]] had a chromax.black 140mm and creamed my jeans and was like "gotta order that!" Then I realized I had no 140mm mounts, and rather than do something sensible, I built up an autistic rage and was like OH FUCK YOU GOD, YOU TRY TO FUCK ME, NAH, I'LL JUST BUY THREE MORE FANS AND THREE-DEE PRINT MYSELF A BARUD-DÛR AND IT'LL INTERLOCK BECAUSE I TOO AM A MASTER OF REALITY, YOU OLD SHITTER, THEN I'LL GO BACK IN TIME AND KILL MY PARENTS BEFORE I CAN BE BORN AND UNDO ALL YOUR WORKS BITCH and then it was like, well, better use these fans I guess.
+* <s>I'm not sure this 4x140mm structure of fans in the roof is really doing me any good, and need to test with and without them. What really happened here is I saw [[Noctua]] had a chromax.black 140mm and creamed my jeans and was like "gotta order that!" Then I realized I had no 140mm mounts, and rather than do something sensible, I built up an autistic rage and was like OH FUCK YOU GOD, YOU TRY TO FUCK ME, NAH, I'LL JUST BUY THREE MORE FANS AND THREE-DEE PRINT MYSELF A BARUD-DÛR AND IT'LL INTERLOCK BECAUSE I TOO AM A MASTER OF REALITY, YOU OLD SHITTER, THEN I'LL GO BACK IN TIME AND KILL MY PARENTS BEFORE I CAN BE BORN AND UNDO ALL YOUR WORKS BITCH and then it was like, well, better use these fans I guess.</s> they were indeed doing me no good, and have been replaced with a [[inaMORAta|MoRa-3]].
-* Meanwhile, I don't think there's enough inflow. Need to test with the door open.
+* <s>Meanwhile, I don't think there's enough inflow. Need to test with the door open.</s> remedied.
 * Apparently Type-C USB Power Delivery runs at 20 and 48 volts. No idea if I'm supplying this correctly, or if it's even in play.
@@ Line 598: / Line 589: @@
 [[File:Ohyeah.jpg|center]]
-That'll do, machine. That'll do.
+That'll do, pig. That'll do.
 ==Future directions==
@@ Line 610: / Line 601: @@
 <li><b>LoRa.</b> [[LoRa]] is a long-range, low-bandwidth radio protocol. I could bring an antenna out, and use the Arduino together with a LoRa chip.</li>
 <li><b>Battery for the CCFL.</b> It would be nice to have some light when I'm working inside the machine. If I could provide selectable battery-based backup for these rods, that would be useful.</li>
-<li><b>PID control for fans/pumps.</b> The Proportional-Integral-Derivative controller is a simple feedback mechanism that I suspect would work well with fans and pumps. I don't care how many RPM my fans are spinning at; what I care about is how warm my coolant and components are (and noise). I'd like to set up target ΔTs (as a function of ambient temp) and a target noise ceiling, and use an inline sensor, an ambient sensor, and an acoustic sensor in combination to manage my loop's active components.</li>
+<li><b>PID control for fans/pumps.</b> The Proportional-Integral-Derivative controller is a simple feedback mechanism that I suspect would work well with fans and pumps. I don't care how many RPM my fans are spinning at; what I care about is how warm my coolant and components are (and noise). I'd like to set up target ΔTs (as a function of ambient temp) and a target noise ceiling, and use an inline sensor, an ambient sensor, and an acoustic sensor in combination to manage my loop's active components. <b>update: see my [[Counterforce]] project, which does all this and much more!</b></li>
 </ul>