A Rack of One's Own

Line 19:

9x Arctic P14s on the MO-RA at 1200rpm plus an EKWB Dual XTOP with 2x D5s keep my ~3L of coolant barely suprambient, moving fluid at about 2.2Lpm at 4Krpm of pumping, with the four Xeons below 30 at idle, all of it silent. w00t! i'm using an ESP8266-based control, similar to my existing [[InaMORAta|inaMORAta]] solution (but slightly improved). three sets of Koolance Black QD4 quick disconnects, an XCPC filter, and an EKWB Quantum Torque drainage fitting ensure loop maintainability across almost six meters of EKWB ZMT soft tubing. distilled water plus three bottles of EKWB Acid Green Cyrofuel concentrate and a Singularity Computing Protium reservoir complete the successful cooling story. i've drilled two ¾" (technically 20mm) holes through the roof through which the ZMT protrudes, though i'm likely to replace that steel with glass or Lexan so that the internals are visible (i've got a CODI6 lighting up the Aurora LEDs, but it's currently disabled, which is unfortunate as they look fucking awesome).

with that worked out, 'twas time to stuff the chassis like a motherfucking Caravan (as lil' wayne did before me). the motherboard offers eight slots for 8x X10QBi-MEM1 memory boards supporting 12 DDR3 DIMMs each, 4 PCIe 3.0x16 slots, and 7 PCIe 3.0x8 slots, but three of the latter slots prevent 2 memory boards from being used. two memory boards map to each NUMA zone, and any given socket requires locally-connected memory to function. the two boards contraindicated by the three x8 slots map to Zone 2 and 3 of 4. six boards of 12x 32GiB DIMMs each mean a maximum of 2304GiB RAM if all PCIe slots are to be used (though the real hit comes in terms of bandwidth, not capacity). well, 2.3TiB is still almost 10x as much memory as i've ever had in a machine, so it seemed fine for now. 72x 32GiB ECC DIMMs can be had on ebay for less than a kilodollar, and boom, after learning the difference between LRDIMMs and LDIMMs and what 4x4 means in a DIMM context i was live with over two terabytes of DDR3, and eleven PCIe slots supporting a total of 120 3.0 lanes (the dual-port Intel X540 and BPN-SAS-846A SAS3 controller used dedicated slots). how to fill them?

with that worked out, 'twas time to stuff the chassis like a motherfucking Caravan (as [https://www.youtube.com/watch?v=_hDPUoUi6_s lil' wayne did before me]). the motherboard offers eight slots for 8x X10QBi-MEM1 memory boards supporting 12 DDR3 DIMMs each, 4 PCIe 3.0x16 slots, and 7 PCIe 3.0x8 slots, but three of the latter slots prevent 2 memory boards from being used. two memory boards map to each NUMA zone, and any given socket requires locally-connected memory to function. the two boards contraindicated by the three x8 slots map to Zone 2 and 3 of 4. six boards of 12x 32GiB DIMMs each mean a maximum of 2304GiB RAM if all PCIe slots are to be used (though the real hit comes in terms of bandwidth, not capacity). well, 2.3TiB is still almost 10x as much memory as i've ever had in a machine, so it seemed fine for now. 72x 32GiB ECC DIMMs can be had on ebay for less than a kilodollar, and boom, after learning the difference between LRDIMMs and LDIMMs and what 4x4 means in a DIMM context i was live with over two terabytes of DDR3, and eleven PCIe slots supporting a total of 120 3.0 lanes (the dual-port Intel X540 and BPN-SAS-846A SAS3 controller used dedicated slots). how to fill them?

there are disappointingly no m.2 slots on the X10Qbi, so four x8 slots were paired up with chinesium PCBs to support 2x Intel 750 U.2 SSDs + 2x Samsung 980 M.2 SSDs, all of them NVMe (the 980s do not appear to be afflicted by the same firmware flaw which has proved so fatal for the 980 PROs, huzzah). a similarly nonamed USB 3.2 controller and an Intel AX200 WiFi6 provided valuable HID and wireless expansion capability (though i'm likely to remove the latter, read on). in went a Mellanox ConnectX-4 single-port 100Gbps (most CX4s are 50Gbps or below, but several models boast the full QSFP28 goodness, though only in a single port), directly linked with armored 850nm OM4 fiber to another CX4 in my gateway. a Chelsio T320 added another 2x 10Gbps ports, linking SFP+ to my workstation and gateway with fiber (the existing X540 connects to same using Cat6E). yes, before you ask, this actually <i>is</i> all necessary for running some of my code and experiments. but it was incomplete...$160 later, i have two Tesla K80s headed my way from ebay for a total of 9984 Kepler CUDA cores, 48GiB of GDDR5, and 960GB/s of bandwidth to same. the enhanced chonk of the Teslas mean i'll have to give up one x8 link; i intend to sacrifice the AX200, as i can easily throw a USB wifi adapter onto the hub i've affixed with Command strips to the inside of my rack.

A Rack of One's Own: Difference between revisions

Dank (talk | contribs)

@@ Line 19: / Line 19: @@
 x Arctic P14s on the MO-RA at 1200rpm plus an EKWB Dual XTOP with 2x D5s keep my ~3L of coolant barely suprambient, moving fluid at about 2.2Lpm at 4Krpm of pumping, with the four Xeons below 30 at idle, all of it silent. w00t! i'm using an ESP8266-based control, similar to my existing [[InaMORAta|inaMORAta]] solution (but slightly improved). three sets of Koolance Black QD4 quick disconnects, an XCPC filter, and an EKWB Quantum Torque drainage fitting ensure loop maintainability across almost six meters of EKWB ZMT soft tubing. distilled water plus three bottles of EKWB Acid Green Cyrofuel concentrate and a Singularity Computing Protium reservoir complete the successful cooling story. i've drilled two ¾" (technically 20mm) holes through the roof through which the ZMT protrudes, though i'm likely to replace that steel with glass or Lexan so that the internals are visible (i've got a CODI6 lighting up the Aurora LEDs, but it's currently disabled, which is unfortunate as they look fucking awesome).
-with that worked out, 'twas time to stuff the chassis like a motherfucking Caravan (as lil' wayne did before me). the motherboard offers eight slots for 8x X10QBi-MEM1 memory boards supporting 12 DDR3 DIMMs each, 4 PCIe 3.0x16 slots, and 7 PCIe 3.0x8 slots, but three of the latter slots prevent 2 memory boards from being used. two memory boards map to each NUMA zone, and any given socket requires locally-connected memory to function. the two boards contraindicated by the three x8 slots map to Zone 2 and 3 of 4. six boards of 12x 32GiB DIMMs each mean a maximum of 2304GiB RAM if all PCIe slots are to be used (though the real hit comes in terms of bandwidth, not capacity). well, 2.3TiB is still almost 10x as much memory as i've ever had in a machine, so it seemed fine for now. 72x 32GiB ECC DIMMs can be had on ebay for less than a kilodollar, and boom, after learning the difference between LRDIMMs and LDIMMs and what 4x4 means in a DIMM context i was live with over two terabytes of DDR3, and eleven PCIe slots supporting a total of 120 3.0 lanes (the dual-port Intel X540 and BPN-SAS-846A SAS3 controller used dedicated slots). how to fill them?
+with that worked out, 'twas time to stuff the chassis like a motherfucking Caravan (as [https://www.youtube.com/watch?v=_hDPUoUi6_s lil' wayne did before me]). the motherboard offers eight slots for 8x X10QBi-MEM1 memory boards supporting 12 DDR3 DIMMs each, 4 PCIe 3.0x16 slots, and 7 PCIe 3.0x8 slots, but three of the latter slots prevent 2 memory boards from being used. two memory boards map to each NUMA zone, and any given socket requires locally-connected memory to function. the two boards contraindicated by the three x8 slots map to Zone 2 and 3 of 4. six boards of 12x 32GiB DIMMs each mean a maximum of 2304GiB RAM if all PCIe slots are to be used (though the real hit comes in terms of bandwidth, not capacity). well, 2.3TiB is still almost 10x as much memory as i've ever had in a machine, so it seemed fine for now. 72x 32GiB ECC DIMMs can be had on ebay for less than a kilodollar, and boom, after learning the difference between LRDIMMs and LDIMMs and what 4x4 means in a DIMM context i was live with over two terabytes of DDR3, and eleven PCIe slots supporting a total of 120 3.0 lanes (the dual-port Intel X540 and BPN-SAS-846A SAS3 controller used dedicated slots). how to fill them?
 there are disappointingly no m.2 slots on the X10Qbi, so four x8 slots were paired up with chinesium PCBs to support 2x Intel 750 U.2 SSDs + 2x Samsung 980 M.2 SSDs, all of them NVMe (the 980s do not appear to be afflicted by the same firmware flaw which has proved so fatal for the 980 PROs, huzzah). a similarly nonamed USB 3.2 controller and an Intel AX200 WiFi6 provided valuable HID and wireless expansion capability (though i'm likely to remove the latter, read on). in went a Mellanox ConnectX-4 single-port 100Gbps (most CX4s are 50Gbps or below, but several models boast the full QSFP28 goodness, though only in a single port), directly linked with armored 850nm OM4 fiber to another CX4 in my gateway. a Chelsio T320 added another 2x 10Gbps ports, linking SFP+ to my workstation and gateway with fiber (the existing X540 connects to same using Cat6E). yes, before you ask, this actually <i>is</i> all necessary for running some of my code and experiments. but it was incomplete...$160 later, i have two Tesla K80s headed my way from ebay for a total of 9984 Kepler CUDA cores, 48GiB of GDDR5, and 960GB/s of bandwidth to same. the enhanced chonk of the Teslas mean i'll have to give up one x8 link; i intend to sacrifice the AX200, as i can easily throw a USB wifi adapter onto the hub i've affixed with Command strips to the inside of my rack.