Strangelet

(33 intermediate revisions by the same user not shown)

Line 1:

~~A 24x 3.5" bay 4U machine I bought off ebay in 2022 for $1457 from~~ [https://~~unixsurplus~~.com/ ~~UNIXSurplusCom~~], ~~who were overall pretty straight~~. All prices listed include any tax and shipping.

As a longtime [https://www.reddit.com/r/DataHoarder/ data hoarder], I hope to have a petabyte in my living room by 2025 or so. Until we have 30TB disks, that'll likely require 2x 24-bay servers (a single 36-bay can do it with 28TB disks). I'll thus likely acquire an additional 24- or 36-bay disk shelf to go along with this server, especially as I now own a rack. Assuming $250 each, every 24 drives represent a $6000 outlay. 48 bays with 21TB disks represent just over a petabyte.

I want it watercooled. I want it silent. I want it to consume less than a kilowatt. I want redundancy on everything. I want a P-Bomb, and I want to spend less than twenty thousand dollars on it.

All prices listed include any tax and shipping. The first sixteen disks will be coming out of [[Schwarzger%C3%A4t_III|my workstation]]. There's a semidecent writeup on [[A_Rack_of_One%27s_Own|DANKBLOG]].

==As delivered==

A diskless but otherwise complete [https://www.supermicro.com/en/products/system/4U/8048/SYS-8048B-TRFT.cfm 8048B-TRFT] system:

A diskless but otherwise complete [https://www.supermicro.com/en/products/system/4U/8048/SYS-8048B-TRFT.cfm 8048B-TRFT] 24x3.5" 4U system was purchased off ebay in 2022 for $1457 from [https://unixsurplus.com/ UNIXSurplusCom], who were overall pretty straight.

* Supermicro SC848XTS-R3240BP chassis

Line 11:

Line 15:

* 3x 80mm [https://store.supermicro.com/80mm-fan-0148l4.html FAN-0148L4] rear exhaust fans

** Nidec 80x80x38mm 62.5 dBA, 116.5 CFM, 1.95A

* 4x 92mm FAN-0146L4 fans

* 4x 92mm FAN-0146L4 San Ace 92 fans

** [https://products.sanyodenki.com/en/sanace/dc/counter-rotating-fan/9CRA0912P0G001/ San Ace 92] 92x92x38mm 63 dBA, 141 CFM, 2.1A

* 24x 3.5" MCP-220-00075-0B trays (one with dead LED)

* BPN-SAS3-846EL1 12Gbs SAS3 backplane

Line 26:

Line 31:

* Replaced dead tray with a new one ($15)

* Replaced 3x Nidec Ultra Flow rear fans with Noctua NF-A8 chromax.Black ($63)

* Replaced 4x with Noctua NF-A9 chromax.Black ($86)

* Replaced 4x San Ace 92 fans with Noctua NF-A9 chromax.Black ($86)

* Replaced 4x 4U SuperMicro heatsinks with 4x Alphacool Eisblock XPX Pro Aurora waterblocks ($215)

* Replaced 4x E7-4850 v2 Xeons with 4x E7-8890 v3 Xeons ($60)

* Sold 16x 16GB DDR3-1066 ECC 4x4 RDIMMs (-$74)

We lose a good deal of airflow and static pressure with the fan change; the new fans are thinner (25mm), and rotate at significantly lower speeds. The NF-A8 is 2.37 mmH₂O and 55.5 m³/h, and the NF-A9 2.28 mmH₂O and 78.9 m³/h, compared to 197.9 and 239.6 m³/h for the builtin equivalents. We're thus down to about 25% the original airflow, but in exchange, we've almost completely eliminated noise.

It turns out that these mounts are LGA 2011 "Narrow", as opposed to the LGA 2011 with which we're all familiar. I couldn't find any brackets for these 90mm x 54mm mounts, so I designed and printed my own. They work just fine. You can find the OpenSCAD source [https://github.com/dankamongmen/openscad-models/blob/master/lga2011-narrow.scad here].

The E7-8890s v3s are big upgrades on the E7-4850 v2s, increasing my core count by 50% and moving to the far superior [[Ivy Bridge]] microarchitecture.

Replacements total: $197 + $15 + $63 + $86 + $215 + $60 - $74: $562

===Additions (newly purchased)===

* 8x 32GB Samsung PC3L-~~10600L~~ DDR3 ($~~218~~)

* 32x 32GB Samsung PC3L-12800L DDR3 ($679)

* 4x 32GB Samsung PC3L-~~14900L~~ DDR3 ($~~108)~~

* 32x 32GB Samsung PC3L-12800L DDR3 ($326)

* 4x 32GB Samsung PC3L-10600R DDR3 ($125)

* 2x [https://www.amazon.com/dp/B09JM5FVC7?psc=1&ref=ppx_yo2ov_dt_b_product_details Bejavr M.2 PCIe cards] ($23)

* [https://www.amazon.com/dp/B099185SSV?psc=1&ref=ppx_yo2ov_dt_b_product_details GLOTRENDS U.2 PCIe card] ($23)

* 2x [https://www.amazon.com/dp/B099185SSV?psc=1&ref=ppx_yo2ov_dt_b_product_details GLOTRENDS U.2 PCIe card] ($46)

* Connect-X CX455A [[100GbE|100Gbps]] QSFP28 PCIe ($150 including BiDi transceiver)

* 2x Samsung 980 1TB M.2 2280 PCIe 3.0 NVMe SSDs ($175)

* SuperMicro SSD-DM016-SMCMVN1 16GB 6G SATA-DOM ($35)

* Chelsio N320E-CR 10Gb/s dual-port ($66)

* 2x NVIDIA K80 Tesla ($183)

* 2x USB 3.0 powered hub

* USB 3.2 PCIe card

* USB 3.0 powered hub

Additions total: $679 + $326 + $23 + $46 + $150 + $175 + $35 + $66 + $183 = $1683

===Additions (from stores on hand)===

* Intel 750 400GB U.2 PCIe 3.0 NVMe SSD

* 2x Intel 750 400GB U.2 PCIe 3.0 NVMe SSD

* ~~16x~~ Seagate Exos 18TB 3.5" SATA3

* 18x Seagate Exos 18TB 3.5" SATA3

* [[MO-RA3]] external radiator

* 9x Arctic P14 RGB 140mm fans

* EKWB Dual-XTOP D5 pumps

==TCO==

I bought an [https://www.server-rack-online.com/apc-netshelter-ar2105blk.html APC NetShelter VX] secondhand to house this monster, though that is not included in the TCO. Similarly, there are several hundred dollars of watercooling expenses (~~4x waterblocks,~~ a [[inaMORAta|MoRa-3]], an EKWB Dual XTOP D5...) that I've not bothered to include. Altogether, they would add about another thousand to the bill.

I bought an [https://www.server-rack-online.com/apc-netshelter-ar2105blk.html APC NetShelter VX] secondhand to house this monster, though that is not included in the TCO. Similarly, there are several hundred dollars of watercooling expenses (a [[inaMORAta|MoRa-3]], an EKWB Dual XTOP D5...) that I've not bothered to include. Altogether, they would add about another thousand to the bill.

* Original: $1457

* Replacements: $~~197 + $15 + $63 + $86 = $361~~

* Replacements: $562

* Additions: $~~218 + $108 + $125 + $23 + $23 + $175 = $672~~

* Additions: $1683

Total: $3702

Total: $~~2490~~

Strangelet: Difference between revisions

Dank (talk | contribs)

@@ Line 1: / Line 1: @@
-A 24x 3.5" bay 4U machine I bought off ebay in 2022 for $1457 from [https://unixsurplus.com/ UNIXSurplusCom], who were overall pretty straight. All prices listed include any tax and shipping.
+As a longtime [https://www.reddit.com/r/DataHoarder/ data hoarder], I hope to have a petabyte in my living room by 2025 or so. Until we have 30TB disks, that'll likely require 2x 24-bay servers (a single 36-bay can do it with 28TB disks). I'll thus likely acquire an additional 24- or 36-bay disk shelf to go along with this server, especially as I now own a rack. Assuming $250 each, every 24 drives represent a $6000 outlay. 48 bays with 21TB disks represent just over a petabyte.
+I want it watercooled. I want it silent. I want it to consume less than a kilowatt. I want redundancy on everything. I want a P-Bomb, and I want to spend less than twenty thousand dollars on it.
+All prices listed include any tax and shipping. The first sixteen disks will be coming out of [[Schwarzger%C3%A4t_III|my workstation]]. There's a semidecent writeup on [[A_Rack_of_One%27s_Own|DANKBLOG]].
 ==As delivered==
-A diskless but otherwise complete [https://www.supermicro.com/en/products/system/4U/8048/SYS-8048B-TRFT.cfm 8048B-TRFT] system:
+A diskless but otherwise complete [https://www.supermicro.com/en/products/system/4U/8048/SYS-8048B-TRFT.cfm 8048B-TRFT] 24x3.5" 4U system was purchased off ebay in 2022 for $1457 from [https://unixsurplus.com/ UNIXSurplusCom], who were overall pretty straight.
 * Supermicro SC848XTS-R3240BP chassis
@@ Line 11: / Line 15: @@
 * 3x 80mm [https://store.supermicro.com/80mm-fan-0148l4.html FAN-0148L4] rear exhaust fans
 ** Nidec 80x80x38mm 62.5 dBA, 116.5 CFM, 1.95A
-* 4x 92mm FAN-0146L4 fans
+* 4x 92mm FAN-0146L4 San Ace 92 fans
+** [https://products.sanyodenki.com/en/sanace/dc/counter-rotating-fan/9CRA0912P0G001/ San Ace 92] 92x92x38mm 63 dBA, 141 CFM, 2.1A
 * 24x 3.5" MCP-220-00075-0B trays (one with dead LED)
 * BPN-SAS3-846EL1 12Gbs SAS3 backplane
@@ Line 26: / Line 31: @@
 * Replaced dead tray with a new one ($15)
 * Replaced 3x Nidec Ultra Flow rear fans with Noctua NF-A8 chromax.Black ($63)
-* Replaced 4x with Noctua NF-A9 chromax.Black ($86)
+* Replaced 4x San Ace 92 fans with Noctua NF-A9 chromax.Black ($86)
+* Replaced 4x 4U SuperMicro heatsinks with 4x Alphacool Eisblock XPX Pro Aurora waterblocks ($215)
+* Replaced 4x E7-4850 v2 Xeons with 4x E7-8890 v3 Xeons ($60)
+* Sold 16x 16GB DDR3-1066 ECC 4x4 RDIMMs (-$74)
+We lose a good deal of airflow and static pressure with the fan change; the new fans are thinner (25mm), and rotate at significantly lower speeds. The NF-A8 is 2.37 mmH₂O and 55.5 m³/h, and the NF-A9 2.28 mmH₂O and 78.9 m³/h, compared to 197.9 and 239.6 m³/h for the builtin equivalents. We're thus down to about 25% the original airflow, but in exchange, we've almost completely eliminated noise.
+It turns out that these mounts are LGA 2011 "Narrow", as opposed to the LGA 2011 with which we're all familiar. I couldn't find any brackets for these 90mm x 54mm mounts, so I designed and printed my own. They work just fine. You can find the OpenSCAD source [https://github.com/dankamongmen/openscad-models/blob/master/lga2011-narrow.scad here].
+The E7-8890s v3s are big upgrades on the E7-4850 v2s, increasing my core count by 50% and moving to the far superior [[Ivy Bridge]] microarchitecture.
+Replacements total: $197 + $15 + $63 + $86 + $215 + $60 - $74: $562
 ===Additions (newly purchased)===
-* 8x 32GB Samsung PC3L-10600L DDR3 ($218)
+* 32x 32GB Samsung PC3L-12800L DDR3 ($679)
-* 4x 32GB Samsung PC3L-14900L DDR3 ($108)
+* 32x 32GB Samsung PC3L-12800L DDR3 ($326)
-* 4x 32GB Samsung PC3L-10600R DDR3 ($125)
 * 2x [https://www.amazon.com/dp/B09JM5FVC7?psc=1&ref=ppx_yo2ov_dt_b_product_details Bejavr M.2 PCIe cards] ($23)
-* [https://www.amazon.com/dp/B099185SSV?psc=1&ref=ppx_yo2ov_dt_b_product_details GLOTRENDS U.2 PCIe card] ($23)
+* 2x [https://www.amazon.com/dp/B099185SSV?psc=1&ref=ppx_yo2ov_dt_b_product_details GLOTRENDS U.2 PCIe card] ($46)
+* Connect-X CX455A [[100GbE|100Gbps]] QSFP28 PCIe ($150 including BiDi transceiver)
 * 2x Samsung 980 1TB M.2 2280 PCIe 3.0 NVMe SSDs ($175)
+* SuperMicro SSD-DM016-SMCMVN1 16GB 6G SATA-DOM ($35)
+* Chelsio N320E-CR 10Gb/s dual-port ($66)
+* 2x NVIDIA K80 Tesla ($183)
+* 2x USB 3.0 powered hub
 * USB 3.2 PCIe card
-* USB 3.0 powered hub
+Additions total: $679 + $326 + $23 + $46 + $150 + $175 + $35 + $66 + $183 = $1683
 ===Additions (from stores on hand)===
-* Intel 750 400GB U.2 PCIe 3.0 NVMe SSD
+* 2x Intel 750 400GB U.2 PCIe 3.0 NVMe SSD
-* 16x Seagate Exos 18TB 3.5" SATA3
+* 18x Seagate Exos 18TB 3.5" SATA3
+* [[MO-RA3]] external radiator
+* 9x Arctic P14 RGB 140mm fans
+* EKWB Dual-XTOP D5 pumps
 ==TCO==
-I bought an [https://www.server-rack-online.com/apc-netshelter-ar2105blk.html APC NetShelter VX] secondhand to house this monster, though that is not included in the TCO. Similarly, there are several hundred dollars of watercooling expenses (4x waterblocks, a [[inaMORAta|MoRa-3]], an EKWB Dual XTOP D5...) that I've not bothered to include. Altogether, they would add about another thousand to the bill.
+I bought an [https://www.server-rack-online.com/apc-netshelter-ar2105blk.html APC NetShelter VX] secondhand to house this monster, though that is not included in the TCO. Similarly, there are several hundred dollars of watercooling expenses (a [[inaMORAta|MoRa-3]], an EKWB Dual XTOP D5...) that I've not bothered to include. Altogether, they would add about another thousand to the bill.
 * Original: $1457
-* Replacements: $197 + $15 + $63 + $86 = $361
+* Replacements: $562
-* Additions: $218 + $108 + $125 + $23 + $23 + $175 = $672
+* Additions: $1683
+Total: $3702
-Total: $2490