Check out my first novel, midnight's simulacra!

Strangelet: Difference between revisions

From dankwiki
No edit summary
 
(52 intermediate revisions by the same user not shown)
Line 1: Line 1:
A 24x 3.5" bay 4U machine I bought off ebay in 2022 for $999.
As a longtime [https://www.reddit.com/r/DataHoarder/ data hoarder], I hope to have a petabyte in my living room by 2025 or so. Until we have 30TB disks, that'll likely require 2x 24-bay servers (a single 36-bay can do it with 28TB disks). I'll thus likely acquire an additional 24- or 36-bay disk shelf to go along with this server, especially as I now own a rack. Assuming $250 each, every 24 drives represent a $6000 outlay. 48 bays with 21TB disks represent just over a petabyte.
 
I want it watercooled. I want it silent. I want it to consume less than a kilowatt. I want redundancy on everything. I want a P-Bomb, and I want to spend less than twenty thousand dollars on it.
 
All prices listed include any tax and shipping. The first sixteen disks will be coming out of [[Schwarzger%C3%A4t_III|my workstation]]. There's a semidecent writeup on [[A_Rack_of_One%27s_Own|DANKBLOG]].


==As delivered==
==As delivered==
A diskless but otherwise complete [https://www.supermicro.com/en/products/system/4U/8048/SYS-8048B-TRFT.cfm 8048B-TRFT] system:
A diskless but otherwise complete [https://www.supermicro.com/en/products/system/4U/8048/SYS-8048B-TRFT.cfm 8048B-TRFT] 24x3.5" 4U system was purchased off ebay in 2022 for $1457 from [https://unixsurplus.com/ UNIXSurplusCom], who were overall pretty straight.


* Supermicro SC848XTS-R3240BP chassis
* Supermicro SC848XTS-R3240BP chassis
* Supermicro [https://www.supermicro.com/en/products/motherboard/X10QBI X10QBi] motherboard with Intel 602J chipset, 4x PCIe 3.0x16, 7x PCIe 3.0x4
* Supermicro [https://www.supermicro.com/en/products/motherboard/X10QBI X10QBi] motherboard with Intel 602J chipset, 4x PCIe 3.0x16, 7x PCIe 3.0x4
* 4x Supermicro [https://store.supermicro.com/1620w-1u-pws-1k62p-1r.html PWS-1K62P-1R] 80+ Platinum PSUs in 2+2 configuration
* 4x Supermicro [https://store.supermicro.com/1620w-1u-pws-1k62p-1r.html PWS-1K62P-1R] 80+ Platinum 1620W PSUs in 2+2 configuration
* Supermicro AOM-X10QBi-A I/O card (dual 10Gbps + VGA)
* Supermicro AOM-X10QBi-A I/O card (dual 10Gbps + VGA)
* 4x [https://ark.intel.com/content/www/us/en/ark/products/75248/intel-xeon-processor-e74850-v2-24m-cache-2-30-ghz.html Intel® Xeon® E7-4850 v2] 2.3GHz CPUs
* 4x [https://ark.intel.com/content/www/us/en/ark/products/75248/intel-xeon-processor-e74850-v2-24m-cache-2-30-ghz.html Intel® Xeon® E7-4850 v2] 2.3GHz CPUs
* 3x 80mm FAN-0148L4 rear exhaust fans
* 3x 80mm [https://store.supermicro.com/80mm-fan-0148l4.html FAN-0148L4] rear exhaust fans
* 4x 92mm FAN-0146L4 fans
** Nidec 80x80x38mm 62.5 dBA, 116.5 CFM, 1.95A
* 24x 3.5" MCP-220-00075-0B trays for BPN-SAS-846A backplane
* 4x 92mm FAN-0146L4 San Ace 92 fans
** [https://products.sanyodenki.com/en/sanace/dc/counter-rotating-fan/9CRA0912P0G001/ San Ace 92] 92x92x38mm 63 dBA, 141 CFM, 2.1A
* 24x 3.5" MCP-220-00075-0B trays (one with dead LED)
* BPN-SAS3-846EL1 12Gbs SAS3 backplane
** upgraded by vendor from [https://www.supermicro.org.cn/manuals/other/BPN-SAS-846A.pdf BPN-SAS-846A] 12Gbps SAS2 backplane
* 8x X10QBi-MEM1 memory boards
* 8x X10QBi-MEM1 memory boards
* 16x 16GB DDR3 DIMMs
* 16x 16GB DDR3 DIMMs
* MCP-290-00057-0N rail set
* MCP-290-00057-0N rail set
==Changes==
The most important thing was to reduce the quite intolerable 81dBA acoustic footprint of the server as received. I also wanted to hit 1TB of DRAM, up from the original 256GB. I needed fast NVMe for <tt>/</tt> and <tt>/home</tt> (though nothing more than PCIe 3.0, as that's all the motherboard supports). Finally, the entire purpose of this server was to move my hard drives out of my workstation. With 24 bays, I'd need (unavailable) 42TB disks to hit a full p-bomb, but a half-P is doable with 21TB disks.
===Replacements (newly purchased)===
* Replaced PSUs with 4x Supermicro [https://store.supermicro.com/1280w-1u-pws-1k28p-sq.html PWS-1K28P-SQ] "SuperQuiet" 80+ Platinum 1280W PSUs ($197)
* Replaced dead tray with a new one ($15)
* Replaced 3x Nidec Ultra Flow rear fans with Noctua NF-A8 chromax.Black ($63)
* Replaced 4x San Ace 92 fans with Noctua NF-A9 chromax.Black ($86)
* Replaced 4x 4U SuperMicro heatsinks with 4x Alphacool Eisblock XPX Pro Aurora waterblocks ($215)
* Replaced 4x E7-4850 v2 Xeons with 4x E7-8890 v3 Xeons ($60)
* Sold 16x 16GB DDR3-1066 ECC 4x4 RDIMMs (-$74)
We lose a good deal of airflow and static pressure with the fan change; the new fans are thinner (25mm), and rotate at significantly lower speeds. The NF-A8 is 2.37 mmH₂O and 55.5 m³/h, and the NF-A9 2.28 mmH₂O and 78.9 m³/h, compared to 197.9 and 239.6 m³/h for the builtin equivalents. We're thus down to about 25% the original airflow, but in exchange, we've almost completely eliminated noise.
It turns out that these mounts are LGA 2011 "Narrow", as opposed to the LGA 2011 with which we're all familiar. I couldn't find any brackets for these 90mm x 54mm mounts, so I designed and printed my own. They work just fine. You can find the OpenSCAD source [https://github.com/dankamongmen/openscad-models/blob/master/lga2011-narrow.scad here].
The E7-8890s v3s are big upgrades on the E7-4850 v2s, increasing my core count by 50% and moving to the far superior [[Ivy Bridge]] microarchitecture.
Replacements total: $197 + $15 + $63 + $86 + $215 + $60 - $74: $562
===Additions (newly purchased)===
* 32x 32GB Samsung PC3L-12800L DDR3 ($679)
* 32x 32GB Samsung PC3L-12800L DDR3 ($326)
* 2x [https://www.amazon.com/dp/B09JM5FVC7?psc=1&ref=ppx_yo2ov_dt_b_product_details Bejavr M.2 PCIe cards] ($23)
* 2x [https://www.amazon.com/dp/B099185SSV?psc=1&ref=ppx_yo2ov_dt_b_product_details GLOTRENDS U.2 PCIe card] ($46)
* Connect-X CX455A [[100GbE|100Gbps]] QSFP28 PCIe ($150 including BiDi transceiver)
* 2x Samsung 980 1TB M.2 2280 PCIe 3.0 NVMe SSDs ($175)
* SuperMicro SSD-DM016-SMCMVN1 16GB 6G SATA-DOM ($35)
* Chelsio N320E-CR 10Gb/s dual-port ($66)
* 2x NVIDIA K80 Tesla ($183)
* 2x USB 3.0 powered hub
* USB 3.2 PCIe card
Additions total: $679 + $326 + $23 + $46 + $150 + $175 + $35 + $66 + $183 = $1683
===Additions (from stores on hand)===
* 2x Intel 750 400GB U.2 PCIe 3.0 NVMe SSD
* 18x Seagate Exos 18TB 3.5" SATA3
* [[MO-RA3]] external radiator
* 9x Arctic P14 RGB 140mm fans
* EKWB Dual-XTOP D5 pumps
==TCO==
I bought an [https://www.server-rack-online.com/apc-netshelter-ar2105blk.html APC NetShelter VX] secondhand to house this monster, though that is not included in the TCO. Similarly, there are several hundred dollars of watercooling expenses (a [[inaMORAta|MoRa-3]], an EKWB Dual XTOP D5...) that I've not bothered to include. Altogether, they would add about another thousand to the bill.
* Original: $1457
* Replacements: $562
* Additions: $1683
Total: $3702

Latest revision as of 09:26, 26 March 2023

As a longtime data hoarder, I hope to have a petabyte in my living room by 2025 or so. Until we have 30TB disks, that'll likely require 2x 24-bay servers (a single 36-bay can do it with 28TB disks). I'll thus likely acquire an additional 24- or 36-bay disk shelf to go along with this server, especially as I now own a rack. Assuming $250 each, every 24 drives represent a $6000 outlay. 48 bays with 21TB disks represent just over a petabyte.

I want it watercooled. I want it silent. I want it to consume less than a kilowatt. I want redundancy on everything. I want a P-Bomb, and I want to spend less than twenty thousand dollars on it.

All prices listed include any tax and shipping. The first sixteen disks will be coming out of my workstation. There's a semidecent writeup on DANKBLOG.

As delivered

A diskless but otherwise complete 8048B-TRFT 24x3.5" 4U system was purchased off ebay in 2022 for $1457 from UNIXSurplusCom, who were overall pretty straight.

  • Supermicro SC848XTS-R3240BP chassis
  • Supermicro X10QBi motherboard with Intel 602J chipset, 4x PCIe 3.0x16, 7x PCIe 3.0x4
  • 4x Supermicro PWS-1K62P-1R 80+ Platinum 1620W PSUs in 2+2 configuration
  • Supermicro AOM-X10QBi-A I/O card (dual 10Gbps + VGA)
  • 4x Intel® Xeon® E7-4850 v2 2.3GHz CPUs
  • 3x 80mm FAN-0148L4 rear exhaust fans
    • Nidec 80x80x38mm 62.5 dBA, 116.5 CFM, 1.95A
  • 4x 92mm FAN-0146L4 San Ace 92 fans
  • 24x 3.5" MCP-220-00075-0B trays (one with dead LED)
  • BPN-SAS3-846EL1 12Gbs SAS3 backplane
  • 8x X10QBi-MEM1 memory boards
  • 16x 16GB DDR3 DIMMs
  • MCP-290-00057-0N rail set

Changes

The most important thing was to reduce the quite intolerable 81dBA acoustic footprint of the server as received. I also wanted to hit 1TB of DRAM, up from the original 256GB. I needed fast NVMe for / and /home (though nothing more than PCIe 3.0, as that's all the motherboard supports). Finally, the entire purpose of this server was to move my hard drives out of my workstation. With 24 bays, I'd need (unavailable) 42TB disks to hit a full p-bomb, but a half-P is doable with 21TB disks.

Replacements (newly purchased)

  • Replaced PSUs with 4x Supermicro PWS-1K28P-SQ "SuperQuiet" 80+ Platinum 1280W PSUs ($197)
  • Replaced dead tray with a new one ($15)
  • Replaced 3x Nidec Ultra Flow rear fans with Noctua NF-A8 chromax.Black ($63)
  • Replaced 4x San Ace 92 fans with Noctua NF-A9 chromax.Black ($86)
  • Replaced 4x 4U SuperMicro heatsinks with 4x Alphacool Eisblock XPX Pro Aurora waterblocks ($215)
  • Replaced 4x E7-4850 v2 Xeons with 4x E7-8890 v3 Xeons ($60)
  • Sold 16x 16GB DDR3-1066 ECC 4x4 RDIMMs (-$74)

We lose a good deal of airflow and static pressure with the fan change; the new fans are thinner (25mm), and rotate at significantly lower speeds. The NF-A8 is 2.37 mmH₂O and 55.5 m³/h, and the NF-A9 2.28 mmH₂O and 78.9 m³/h, compared to 197.9 and 239.6 m³/h for the builtin equivalents. We're thus down to about 25% the original airflow, but in exchange, we've almost completely eliminated noise.

It turns out that these mounts are LGA 2011 "Narrow", as opposed to the LGA 2011 with which we're all familiar. I couldn't find any brackets for these 90mm x 54mm mounts, so I designed and printed my own. They work just fine. You can find the OpenSCAD source here.

The E7-8890s v3s are big upgrades on the E7-4850 v2s, increasing my core count by 50% and moving to the far superior Ivy Bridge microarchitecture.

Replacements total: $197 + $15 + $63 + $86 + $215 + $60 - $74: $562

Additions (newly purchased)

  • 32x 32GB Samsung PC3L-12800L DDR3 ($679)
  • 32x 32GB Samsung PC3L-12800L DDR3 ($326)
  • 2x Bejavr M.2 PCIe cards ($23)
  • 2x GLOTRENDS U.2 PCIe card ($46)
  • Connect-X CX455A 100Gbps QSFP28 PCIe ($150 including BiDi transceiver)
  • 2x Samsung 980 1TB M.2 2280 PCIe 3.0 NVMe SSDs ($175)
  • SuperMicro SSD-DM016-SMCMVN1 16GB 6G SATA-DOM ($35)
  • Chelsio N320E-CR 10Gb/s dual-port ($66)
  • 2x NVIDIA K80 Tesla ($183)
  • 2x USB 3.0 powered hub
  • USB 3.2 PCIe card

Additions total: $679 + $326 + $23 + $46 + $150 + $175 + $35 + $66 + $183 = $1683

Additions (from stores on hand)

  • 2x Intel 750 400GB U.2 PCIe 3.0 NVMe SSD
  • 18x Seagate Exos 18TB 3.5" SATA3
  • MO-RA3 external radiator
  • 9x Arctic P14 RGB 140mm fans
  • EKWB Dual-XTOP D5 pumps

TCO

I bought an APC NetShelter VX secondhand to house this monster, though that is not included in the TCO. Similarly, there are several hundred dollars of watercooling expenses (a MoRa-3, an EKWB Dual XTOP D5...) that I've not bothered to include. Altogether, they would add about another thousand to the bill.

  • Original: $1457
  • Replacements: $562
  • Additions: $1683

Total: $3702

Retrieved from "https://nick-black.com/dankwiki/index.php?title=Strangelet&oldid=9702"