Schwarzgerät III: Difference between revisions

Tags: Mobile edit Mobile web edit
 
(22 intermediate revisions by the same user not shown)
Line 4: Line 4:


My 2020 rebuild, [[Schwarzgerät_II|Schwarzgerät II]], was a beast of a machine. Like the Hubble or LHC, however, I had to give it more powah. The 2022 upgrade, Schwarzgerät III, does just that despite scrotumtightening supply chain madness. This rebuild focused on cooling, power, and aesthetics. I require that the machine be no louder than ambient noise (I live in the middle of [[Atlanta|Midtown Atlanta]]) when weakly loaded—I ought not hear the machine unless seriously engaged in active computation. At the same time, I want to overclock my 3970X as far as she can (reasonably) go.
My 2020 rebuild, [[Schwarzgerät_II|Schwarzgerät II]], was a beast of a machine. Like the Hubble or LHC, however, I had to give it more powah. The 2022 upgrade, Schwarzgerät III, does just that despite scrotumtightening supply chain madness. This rebuild focused on cooling, power, and aesthetics. I require that the machine be no louder than ambient noise (I live in the middle of [[Atlanta|Midtown Atlanta]]) when weakly loaded—I ought not hear the machine unless seriously engaged in active computation. At the same time, I want to overclock my 3970X as far as she can (reasonably) go.
'''(further [[Schwarzgerät_III_upgrade|upgraded]] 2022-04, hopefully the last bit for a minute)'''


<blockquote>“But out here, down here among the people, the truer currencies come into being.”―<i>''Gravity's Rainbow''</i> (1973)</blockquote>
<blockquote>“But out here, down here among the people, the truer currencies come into being.”―<i>''Gravity's Rainbow''</i> (1973)</blockquote>
Line 31: Line 33:
* Self-designed and -printed case for RHElectronics Geiger counter ([https://github.com/dankamongmen/openscad-models/blob/master/geiger.scad source])
* Self-designed and -printed case for RHElectronics Geiger counter ([https://github.com/dankamongmen/openscad-models/blob/master/geiger.scad source])
* Self-designed and -printed covering case for EKWB Quantum Kinetic FLT 240 mounting kit ([https://github.com/dankamongmen/openscad-models/blob/master/reservoir-part.scad source])
* Self-designed and -printed covering case for EKWB Quantum Kinetic FLT 240 mounting kit ([https://github.com/dankamongmen/openscad-models/blob/master/reservoir-part.scad source])
* Self-designed and -printed cable shroud for bottom of Gigabyte Aorus Master TRX40
* Self-designed and -printed false floor for bottom of Caselabs Magnum T10 PSU side
* Self-designed and -printed PCI brackets with cable channels ([https://github.com/dankamongmen/openscad-models/blob/master/bracketpci.scad source])
* Self-designed and -printed PCI brackets with cable channels ([https://github.com/dankamongmen/openscad-models/blob/master/bracketpci.scad source])
* Self-designed and -printed 4x140mm fan mount for roof ([https://github.com/dankamongmen/openscad-models/blob/master/140-interlocking.scad source])
* Self-designed and -printed 4x140mm fan mount for roof ([https://github.com/dankamongmen/openscad-models/blob/master/140-interlocking.scad source])
Line 63: Line 63:
* 2x [http://hardwarelabs.com/nemesis/gtsxflow/240gts-xflow/ Hardware Labs Black Ice Nemesis GTS240 XFLOW] 16 FPI 29.6mm crossflow radiators, mounted to bottoms (292x133mm).
* 2x [http://hardwarelabs.com/nemesis/gtsxflow/240gts-xflow/ Hardware Labs Black Ice Nemesis GTS240 XFLOW] 16 FPI 29.6mm crossflow radiators, mounted to bottoms (292x133mm).
* Fancasee 1-to-4 PWM splitter, power from header
* Fancasee 1-to-4 PWM splitter, power from header
* Silverstone 8-way PWM splitter, SATA power
* [[SilverStone_fan_hub|Silverstone 8-way]] PWM splitter, SATA power
* 12.5mm ID EKWB [https://www.ekwb.com/shop/ek-tube-zmt-matte-black-19-4-12-5mm ZMT], black
* 12.5mm ID EKWB [https://www.ekwb.com/shop/ek-tube-zmt-matte-black-19-4-12-5mm ZMT], black


Most calculations will require knowing the total amount of coolant in our system. Fluid volumes of liquid cooling components are irritatingly difficult to establish (once the hardware is installed, anyway). With an inner diameter of 12.5mm, the ZMT has a cross-section of 122.72mm² (0.015m²). I got the EKWB fluid volumes directly from their support channel. G¼ has a diameter of 13.157mm, but that doesn't tell you anything about the inner diameter, which is not standardized =\.
Most calculations will require knowing the total amount of coolant in our system. Fluid volumes of liquid cooling components are irritatingly difficult to establish (once the hardware is installed, anyway). With an inner diameter of 12.5mm, the ZMT has a cross-section of 122.72mm² (1.2272cm²). I got the EKWB fluid volumes directly from their support channel. G¼ has a diameter of 13.157mm, but that doesn't tell you anything about the inner diameter, which is not standardized =\. I used a graduated cylinder to measure the Monsoon and the radiators. Attachments can be considered 1--2mL each (a drain structure including a ball valve attachment, a T attachment, a 90° adapter and several extenders contained a total of 10mL).


<b>TODO: get lenghts of PSU-side tubing, volumes of HWL radiators+Monsoon reservoir. I acquired a graduated cylinder to perform these tests, and will do them next time I tear down the loop. get more PSI numbers. PSI drop is a function of flow rate...</b>
{| class="wikitable"
{| class="wikitable"
! Component !! Volume (mL) !! Climb (m) !! θ (rad) !! PSI drop
! Component !! Volume (mL) !! Climb (m) !! θ (rad)
|-
|-
| HWL XFLOW 240 || ? || 0 || 0 ||
| HWL XFLOW 240 || 90 || 0 || 0  
|-
|-
| Tubing || ? || 0 || 0 ||
| Tubing || ? || 0 || 0
|-
|-
| HWL GTR360 || ? || 0 || 0 || 0.25
| HWL GTR360 || 320 || 0 || 0
|-
|-
| Tubing || ? || 0 || 0 ||
| Tubing || ? || 0 || 0
|-
|-
| HWL GTS360 || ? || 0 || 0 || 0.93
| HWL GTS360 || 130 || 0 || 0
|-
|-
| Tubing || 44 || || ||
| Tubing || 44 || ||
|-
|-
| Aorus Master Monoblock || 45 || || π/2 ||
| Aorus Master monoblock || 45 || || π/2
|-
|-
| Tubing || 11 || || ||
| Tubing || 11 || ||
|-
|-
| EK-Vector waterblock || 50 || || π/2 ||
| EK-Vector waterblock || 50 || || π/2
|-
|-
| Tubing || 22 || || ||
| Tubing || 22 || ||
|-
|-
| Quantum Kinetic FLT 240 || 265 || 0 || 0 ||
| Quantum Kinetic FLT 240 || 265 || 0 || 0
|-
|-
| Tubing || 28 || || ||
| Tubing || 28 || ||
|-
|-
| Flowmeter || ? || || π/4 ||
| DiyHZ flowmeter || 4 || || π/4
|-
|-
| HWL XFLOW 240 || ? || 0 || 0 ||
| HWL XFLOW 240 || 90 || 0 || 0
|-
|-
| Tubing || 11 || 0 || 0 ||
| Tubing || 11 || 0 || 0
|-
|-
| Monsoon Series Two D5 || ? || 0 || 0 ||
| Monsoon Series Two D5 || 300 || 0 || 0
|-
|-
| Tubing || ? || 0 || 0 ||
| Tubing || ? || 0 || 0
|-
|-
|}
|}
Line 192: Line 191:
* [https://www.elegoo.com/products/elegoo-mega-2560-r3-board ELEGOO MEGA 2560] Revision 3, connected to NXZT internal USB hub, mounted to back of PSU chamber
* [https://www.elegoo.com/products/elegoo-mega-2560-r3-board ELEGOO MEGA 2560] Revision 3, connected to NXZT internal USB hub, mounted to back of PSU chamber


As I detailed regarding [[Schwarzgerat II]], the 3990X is an amazing achievement in chip design and fabrication, but I believe it to be severely starved for many tasks by its memory bandwidth; with its four memory channels populated, the ThreadRipper 3990X can hit about 90GB/s from fast DDR4; its Epyc brother can pull down ~190 through its eight channels. For my tasks, it's rare enough that I can drive all my 32 cores; with the 3990X, I'd be paying twice as much to hit full utilization less often, and be unable to bring full bandwidth to bear when I did.
As I detailed regarding [[Schwarzgerat II]], the 3990X is an amazing achievement in chip design and fabrication, but I believe it to be severely starved for many tasks by its memory bandwidth; with its four memory channels populated, the ThreadRipper 3990X can hit about 90GB/s from fast DDR4; its EPYC brother can pull down ~190 through its eight channels. For my tasks, it's rare enough that I can drive all my 32 cores; with the 3990X, I'd be paying twice as much to hit full utilization less often, and be unable to bring full bandwidth to bear when I did.


I absolutely 🖤 my 3970X, though. Bitch screams. Anyone overclocking on Linux should be aware of [https://www.linux.org/docs/man8/turbostat.html turbostat]. The 3970X supports 88 lanes of PCIe 4.0, of which the TRX40 chipset consumes 24, leaving 64 for expansion devices. I've got 16 (GPU) + 16 (Hyper X) + 2x8 (LSI cards) + 12 (M.2 onboards) for 60 total, coming in just under saturation.
I absolutely 🖤 my 3970X, though. Bitch screams. Anyone overclocking on Linux should be aware of [https://www.linux.org/docs/man8/turbostat.html turbostat]. The 3970X supports 88 lanes of PCIe 4.0, of which the TRX40 chipset consumes 24, leaving 64 for expansion devices. I've got 16 (GPU) + 16 (Hyper X) + 2x8 (LSI cards) + 12 (M.2 onboards) for 60 total, coming in just under saturation.
Line 225: Line 224:


{| class="wikitable"
{| class="wikitable"
! Name !! Setup !! Availability !! MaxTol !! MinFail !! Max Rebuild !! Perf
! Name !! Setup !! Availability !! MaxTol !! MinFail !! Max Rebuild
|-
|-
| raid3z
| raid3z
Line 233: Line 232:
| 4
| 4
| 100% + fail
| 100% + fail
|
|-
|-
| raid2z
| raid2z
Line 241: Line 239:
| 3
| 3
| 85.7% + fail
| 85.7% + fail
|
|-
|-
| striped raid3z
| striped raid3z
Line 249: Line 246:
| 4
| 4
| 50% + fail
| 50% + fail
|
|-
|-
| striped raid2z
| striped raid2z
Line 257: Line 253:
| 3
| 3
| 50% + fail
| 50% + fail
|
|-
|-
| striped raidz
| striped raidz
Line 265: Line 260:
| 2
| 2
| 50% + fail
| 50% + fail
|
|-
|-
| mirrored raid3z
| mirrored raid3z
Line 273: Line 267:
| 8
| 8
| 50% + fail
| 50% + fail
|
|-
|-
| mirrored raid2z
| mirrored raid2z
Line 281: Line 274:
| 6
| 6
| 50% + fail
| 50% + fail
|
|-
|-
| mirrored raidz
| mirrored raidz
Line 289: Line 281:
| 4
| 4
| 50% + fail
| 50% + fail
|
|-
|-
| raidz3 of stripes
| raidz3 of stripes
| 7x(2) 4 data, 3 parity
| 7x(2) 4 data, 3 parity
| 57.1%
| 57.1% (8)
| 2 + 2 + 2
| 2 + 2 + 2
| 7
| 4
| 100% + fail
| 100% + fail
|-
|-
Line 302: Line 293:
| 71.4% (10)
| 71.4% (10)
| 2 + 2
| 2 + 2
| 5
| 3
| 85.7% + fail
| 85.7% + fail
|
|}
|}


Line 332: Line 322:


==Distributing power==
==Distributing power==
[[File:Tobsun.jpg|200px|thumb|A 12V->5V buck transformer. By upping the output amperage, it is more efficient than a voltage regulator.]]
[[File:Tobsun.jpg|200px|thumb|A 12V->5V buck transformer, more efficient than a mere voltage regulator.]]
I began to run into some serious power issues on this build, originating in the Exos X18 drives (of which, you might remember, there are 14). It will be worth your time to consult the [https://www.seagate.com/www-content/product-content/enterprise-hdd-fam/exos-x18/_shared/en-us/docs/100865854a.pdf Exos 18 manual]. Remember, 12V is for the motor, and 5V is for the logic.
I began to run into some serious power issues on this build, originating in the Exos X18 drives (of which, you might remember, there are 14). It will be worth your time to consult the [https://www.seagate.com/www-content/product-content/enterprise-hdd-fam/exos-x18/_shared/en-us/docs/100865854a.pdf Exos 18 manual]. Remember, 12V is for the motor, and 5V is for the logic.


Line 457: Line 447:
|-
|-
|}
|}
The Silverstone supports draws up to 4.5A (54W). The Corsair supports a 12V fan draw of 4.5A (54W) and a 5V RGB draw of 4.5A (22.5W).


The PSU claims support of up to 100W as 5V (20A), and up to 850W as 12V (71A). The 5V line is dominated by hard drives, which will consume 65W when in active use (and 70W when spinning up). Beyond them, there are very few 5V draws; the ARGB light strips are going through the motherboard's ARGB headers, the internal USB hub and downstream devices are getting 5V from Molex, and the Corsair is pulling over SATA. Even if the motherboard ARGB headers are fed by the ATX12V's 5V lines, we're talking 20W max, and probably less than that.
The PSU claims support of up to 100W as 5V (20A), and up to 850W as 12V (71A). The 5V line is dominated by hard drives, which will consume 65W when in active use (and 70W when spinning up). Beyond them, there are very few 5V draws; the ARGB light strips are going through the motherboard's ARGB headers, the internal USB hub and downstream devices are getting 5V from Molex, and the Corsair is pulling over SATA. Even if the motherboard ARGB headers are fed by the ATX12V's 5V lines, we're talking 20W max, and probably less than that.
Line 510: Line 502:
| Perif (Molex) || 0 || 55 || 132 || Can bridge to SATA or PCIe. Wiring might not be safe for the full pin capacity.
| Perif (Molex) || 0 || 55 || 132 || Can bridge to SATA or PCIe. Wiring might not be safe for the full pin capacity.
|-
|-
| PCIe x4|| 9 || 0 || 66 || Can bridge to (underpowered) 12V-only (2-pin) Molex. Can probably bridge (with buck transformer) to underpowered Molex.
| PCIe x4|| 0 || 0 || 75 || Can bridge to (underpowered) 12V-only (2-pin) Molex. Can probably bridge (with buck transformer) to underpowered Molex.
|-
|-
|}
|}
Line 573: Line 565:


==Ongoing issues==
==Ongoing issues==
* My flow rate is lower than I would like, assuming this flowmeter to be reliable. Maxing out the PWM pump hits desirable flow rates, but with more noise than I want. I think there might be some flow reduction in the tubing between the flowmeter and Quantum Kinetic; I'm considering replacing that with hardline. For all I know, there's still a big air pocket in some radiator.
* <s>My flow rate is lower than I would like, assuming this flowmeter to be reliable. Maxing out the PWM pump hits desirable flow rates, but with more noise than I want. I think there might be some flow reduction in the tubing between the flowmeter and Quantum Kinetic; I'm considering replacing that with hardline. For all I know, there's still a big air pocket in some radiator.</s> <b>there was indeed a big air pocket</b>
* <s>My onboard Ethernet stopped working, and I'm still not sure why.</s> <b>the ethernet cable wasn't plugged into the outlet all the way, lol, you dumb idiot</b>
* <s>My onboard Ethernet stopped working, and I'm still not sure why.</s> <b>the ethernet cable wasn't plugged into the outlet all the way, lol, you dumb idiot</b>
* <s>I'm getting a kernel oops in the igb driver on startup, not sure why.</s>
* <s>I'm getting a kernel oops in the igb driver on startup, not sure why.</s>
* I have to unplug my keyboard and plug it back in on each boot =/ (this was also happening before). No, this is not fixed by changing the state of XHCI Handoff in my firmware.
* <s>I have to unplug my keyboard and plug it back in on each boot =/ (this was also happening before). No, this is not fixed by changing the state of XHCI Handoff in my firmware.</s> <tt>powertop</tt> was being invoked on each boot, turning on autosuspend.
* I'm not sure this 4x140mm structure of fans in the roof is really doing me any good, and need to test with and without them. What really happened here is I saw [[Noctua]] had a chromax.black 140mm and creamed my jeans and was like "gotta order that!" Then I realized I had no 140mm mounts, and rather than do something sensible, I built up an autistic rage and was like OH FUCK YOU GOD, YOU TRY TO FUCK ME, NAH, I'LL JUST BUY THREE MORE FANS AND THREE-DEE PRINT MYSELF A BARUD-DÛR AND IT'LL INTERLOCK BECAUSE I TOO AM A MASTER OF REALITY, YOU OLD SHITTER, THEN I'LL GO BACK IN TIME AND KILL MY PARENTS BEFORE I CAN BE BORN AND UNDO ALL YOUR WORKS BITCH and then it was like, well, better use these fans I guess.
* <s>I'm not sure this 4x140mm structure of fans in the roof is really doing me any good, and need to test with and without them. What really happened here is I saw [[Noctua]] had a chromax.black 140mm and creamed my jeans and was like "gotta order that!" Then I realized I had no 140mm mounts, and rather than do something sensible, I built up an autistic rage and was like OH FUCK YOU GOD, YOU TRY TO FUCK ME, NAH, I'LL JUST BUY THREE MORE FANS AND THREE-DEE PRINT MYSELF A BARUD-DÛR AND IT'LL INTERLOCK BECAUSE I TOO AM A MASTER OF REALITY, YOU OLD SHITTER, THEN I'LL GO BACK IN TIME AND KILL MY PARENTS BEFORE I CAN BE BORN AND UNDO ALL YOUR WORKS BITCH and then it was like, well, better use these fans I guess.</s> they were indeed doing me no good, and have been replaced with a [[inaMORAta|MoRa-3]].
* Meanwhile, I don't think there's enough inflow. Need to test with the door open.
* <s>Meanwhile, I don't think there's enough inflow. Need to test with the door open.</s> remedied.
* Apparently Type-C USB Power Delivery runs at 20 and 48 volts. No idea if I'm supplying this correctly, or if it's even in play.
* Apparently Type-C USB Power Delivery runs at 20 and 48 volts. No idea if I'm supplying this correctly, or if it's even in play.


Line 597: Line 589:
[[File:Ohyeah.jpg|center]]
[[File:Ohyeah.jpg|center]]


That'll do, machine. That'll do.
That'll do, pig. That'll do.


==Future directions==
==Future directions==
Line 609: Line 601:
<li><b>LoRa.</b> [[LoRa]] is a long-range, low-bandwidth radio protocol. I could bring an antenna out, and use the Arduino together with a LoRa chip.</li>
<li><b>LoRa.</b> [[LoRa]] is a long-range, low-bandwidth radio protocol. I could bring an antenna out, and use the Arduino together with a LoRa chip.</li>
<li><b>Battery for the CCFL.</b> It would be nice to have some light when I'm working inside the machine. If I could provide selectable battery-based backup for these rods, that would be useful.</li>
<li><b>Battery for the CCFL.</b> It would be nice to have some light when I'm working inside the machine. If I could provide selectable battery-based backup for these rods, that would be useful.</li>
<li><b>PID control for fans/pumps.</b> The Proportional-Integral-Derivative controller is a simple feedback mechanism that I suspect would work well with fans and pumps. I don't care how many RPM my fans are spinning at; what I care about is how warm my coolant and components are (and noise). I'd like to set up target ΔTs (as a function of ambient temp) and a target noise ceiling, and use an inline sensor, an ambient sensor, and an acoustic sensor in combination to manage my loop's active components.</li>
<li><b>PID control for fans/pumps.</b> The Proportional-Integral-Derivative controller is a simple feedback mechanism that I suspect would work well with fans and pumps. I don't care how many RPM my fans are spinning at; what I care about is how warm my coolant and components are (and noise). I'd like to set up target ΔTs (as a function of ambient temp) and a target noise ceiling, and use an inline sensor, an ambient sensor, and an acoustic sensor in combination to manage my loop's active components. <b>update: see my [[Counterforce]] project, which does all this and much more!</b></li>
</ul>
</ul>