A Rack of One's Own: Difference between revisions
No edit summary |
No edit summary |
||
| (2 intermediate revisions by the same user not shown) | |||
| Line 15: | Line 15: | ||
i'd hoped not to require watercooling, but with that dream dashed i set to work. the idea of in-case radiators was laughable, and anyway why have a rack if you're not gonna stuff it with shit? for a moment i considered immersion cooling, but due to other projects i already had two [[MO-RA3]] 420x420mm external radiators sitting around unused, whereas i had neither knowledge of nor hardware for immersion. there aren't very many LGA2011-compatible waterblocks; i was hoping for HEATKILLER IVs, but neither PerformancePCs nor TitanRig seem to be actually stocking parts as of late, so i settled for Alphacool Eisblock XPX Pro Aurora waterblocks at $50 a throw. as it turned out, i hardly needed the capability of the HEATKILLERS (my processors are rated at a mere 105W), so this was a win cashwise, and the blocks boast lovely ARGB. i went to install the XPXs, only to discover that the squarish LGA2011 socket has a rectangular cousin, the "LGA2011-Narrow", which despite its physical incompatibility apparently didn't rate a new socket designation. i raged skyward for a few moments, and set to designing [https://github.com/dankamongmen/openscad-models/blob/master/lga2011-narrow.scad custom LGA2011-Narrow mounts] in OpenSCAD. i printed them up on my SLA Elegoo Saturn, and—miracle of miracles!—they worked on the first try. that's the power of actually measuring things instead of just trying to eyeball millimeter distances, i guess! i then suffered the first of three major leaks, all of them due to shoddy manufacturing on some no-name G¼ compression fittings. no matter where in my loop i employed them, no matter how solid the connections looked to the eye, these pieces of shit lived to spray neon green coolant all over my electronics, my rugs, and myself. i was stupidly ready to give them one more try (see above regarding cheap bastardhood), when i noticed they didn't even have fucking o-rings. hell, even Challenger had goddamn o-rings...until it didn't, anyway. into the trash they went, and a week later i had a phat sack of chungal (chungusy? chungy? chungesque perhaps?) Koolance Blacks, which have served without fail since installation. | i'd hoped not to require watercooling, but with that dream dashed i set to work. the idea of in-case radiators was laughable, and anyway why have a rack if you're not gonna stuff it with shit? for a moment i considered immersion cooling, but due to other projects i already had two [[MO-RA3]] 420x420mm external radiators sitting around unused, whereas i had neither knowledge of nor hardware for immersion. there aren't very many LGA2011-compatible waterblocks; i was hoping for HEATKILLER IVs, but neither PerformancePCs nor TitanRig seem to be actually stocking parts as of late, so i settled for Alphacool Eisblock XPX Pro Aurora waterblocks at $50 a throw. as it turned out, i hardly needed the capability of the HEATKILLERS (my processors are rated at a mere 105W), so this was a win cashwise, and the blocks boast lovely ARGB. i went to install the XPXs, only to discover that the squarish LGA2011 socket has a rectangular cousin, the "LGA2011-Narrow", which despite its physical incompatibility apparently didn't rate a new socket designation. i raged skyward for a few moments, and set to designing [https://github.com/dankamongmen/openscad-models/blob/master/lga2011-narrow.scad custom LGA2011-Narrow mounts] in OpenSCAD. i printed them up on my SLA Elegoo Saturn, and—miracle of miracles!—they worked on the first try. that's the power of actually measuring things instead of just trying to eyeball millimeter distances, i guess! i then suffered the first of three major leaks, all of them due to shoddy manufacturing on some no-name G¼ compression fittings. no matter where in my loop i employed them, no matter how solid the connections looked to the eye, these pieces of shit lived to spray neon green coolant all over my electronics, my rugs, and myself. i was stupidly ready to give them one more try (see above regarding cheap bastardhood), when i noticed they didn't even have fucking o-rings. hell, even Challenger had goddamn o-rings...until it didn't, anyway. into the trash they went, and a week later i had a phat sack of chungal (chungusy? chungy? chungesque perhaps?) Koolance Blacks, which have served without fail since installation. | ||
[[File:Penguinrack.jpg|every rack needs a penguin!]] | [[File:Penguinrack.jpg|thumb|every rack needs a penguin!]] | ||
9x Arctic P14s on the MO-RA at 1200rpm plus an EKWB Dual XTOP with 2x D5s keep my ~3L of coolant barely suprambient, moving fluid at about 2.2Lpm at 4Krpm of pumping, with the four Xeons below 30 at idle, all of it silent. w00t! i'm using an ESP8266-based control, similar to my existing [[InaMORAta|inaMORAta]] solution (but slightly improved). three sets of Koolance Black QD4 quick disconnects, an XCPC filter, and an EKWB Quantum Torque drainage fitting ensure loop maintainability across almost six meters of EKWB ZMT soft tubing. distilled water plus three bottles of EKWB Acid Green Cyrofuel concentrate and a Singularity Computing Protium reservoir complete the successful cooling story. i've drilled two ¾" (technically 20mm) holes through the roof through which the ZMT protrudes, though i'm likely to replace that steel with glass or Lexan so that the internals are visible (i've got a CODI6 lighting up the Aurora LEDs, but it's currently disabled, which is unfortunate as they look fucking awesome). | 9x Arctic P14s on the MO-RA at 1200rpm plus an EKWB Dual XTOP with 2x D5s keep my ~3L of coolant barely suprambient, moving fluid at about 2.2Lpm at 4Krpm of pumping, with the four Xeons below 30 at idle, all of it silent. w00t! i'm using an ESP8266-based control, similar to my existing [[InaMORAta|inaMORAta]] solution (but slightly improved). three sets of Koolance Black QD4 quick disconnects, an XCPC filter, and an EKWB Quantum Torque drainage fitting ensure loop maintainability across almost six meters of EKWB ZMT soft tubing. distilled water plus three bottles of EKWB Acid Green Cyrofuel concentrate and a Singularity Computing Protium reservoir complete the successful cooling story. i've drilled two ¾" (technically 20mm) holes through the roof through which the ZMT protrudes, though i'm likely to replace that steel with glass or Lexan so that the internals are visible (i've got a CODI6 lighting up the Aurora LEDs, but it's currently disabled, which is unfortunate as they look fucking awesome). | ||
| Line 24: | Line 24: | ||
the beast at this point will still consume less than a kilowatt at full juice. i have no idea why the chassis ships with 1620W supplies, unless perhaps SAS drives consume ridiculous amounts of power? which, speaking of storage... | the beast at this point will still consume less than a kilowatt at full juice. i have no idea why the chassis ships with 1620W supplies, unless perhaps SAS drives consume ridiculous amounts of power? which, speaking of storage... | ||
[[File:Rackpower.png|thumb|Grafana time series for various power draws]] | |||
the original point of this hoss was to get hard drives out of my workstation, and it was time they went. sixteen 18TB exosen were removed from use, and joined eight more virgin 18TBs. the result is 3x octadrive raid6zs each boasting 144 total, 108 usable, for a total of 324TB accessible at any given time. at full jam, this is about another 200W of mixed 5V+12V (see "[[Schwarzger%C3%A4t_III#The_drive_problem|The Drive Problem]]" for more details), and generates the system's only noise (though that'll change once the K80s arrive). my office is now several degrees cooler, a happy change i thoroughly welcome. | the original point of this hoss was to get hard drives out of my workstation, and it was time they went. sixteen 18TB exosen were removed from use, and joined eight more virgin 18TBs. the result is 3x octadrive raid6zs each boasting 144 total, 108 usable, for a total of 324TB accessible at any given time. at full jam, this is about another 200W of mixed 5V+12V (see "[[Schwarzger%C3%A4t_III#The_drive_problem|The Drive Problem]]" for more details), and generates the system's only noise (though that'll change once the K80s arrive). my office is now several degrees cooler, a happy change i thoroughly welcome. | ||
| Line 30: | Line 32: | ||
so, that's the story of my [[Strangelet|living room supercomputer]]. 96 watercooled Xeon cores, 9984 CUDA cores, 2.3TiB DDR3, 2.8TB NVMe SSD, 324TB SATA3 rust, 280Gb/s networking on a mix of fiber and copper, and 4x 80PLUS Platinum 1280W PSUs, drawing a maximum of ~1200W. the entire rack (machine, lighting, cooling, and all externals) consumes under 600W most of the time: | so, that's the story of my [[Strangelet|living room supercomputer]]. 96 watercooled Xeon cores, 9984 CUDA cores, 2.3TiB DDR3, 2.8TB NVMe SSD, 324TB SATA3 rust, 280Gb/s networking on a mix of fiber and copper, and 4x 80PLUS Platinum 1280W PSUs, drawing a maximum of ~1200W. the entire rack (machine, lighting, cooling, and all externals) consumes under 600W most of the time: | ||
do you think this is some kind of game, son? | do you think this is some kind of game, son? | ||