Schwarzgerät

(14 intermediate revisions by the same user not shown)

Line 1:

[[File:I76950X.jpg|thumb|Roughly one year's wages in Kyrgyzstan]]

'''Update: this has been rebuilt as [[Schwarzgerät_II|Schwarzgerät II]].'''

<blockquote>“There is no way to tell his story without telling my own. And if his story really is a confession, then so is mine.”

―''Apocalypse Now'' (1979)</blockquote>

Line 5:

Line 8:

'''ABSTRACT:''' In August of 2016, I pulled the trigger on a long-planned workstation build. That same month, Intel and NVIDIA dropped new product. Hot, salivation-provoking product: densely packed marvels bursting with FLOPS, rough dank beasts woven up from high-κ 16nm and 14nm strained-silicon FinFETs. Both companies, utterly dominant at their markets' high ends, announced pricing that shoved atom bombs up the ass of every computing enthusiast in the free world.

This is the rambling, poorly-edited, relentlessly technical story of that build. Also: epigraphs. You can skip to the [[#~~final bill of materials~~|bill of materials]], if you just want specs.

This is the rambling, poorly-edited, relentlessly technical story of that build. Also: epigraphs. You can skip to the [[#Final_Bill_of_Materials|bill of materials]], if you just want specs.

==Intro: 10 cores of garbage==

Line 19:

Line 22:

==Build Goals==

[[File:Teragross.png|thumb|12x 12TB EXOS drives]]

<blockquote>“Money has only a different value in the eyes of each.”

―William Makepeace Thackeray, ''Vanity Fair'' (1847), Chapter XLIV</blockquote>

Line 43:

Line 47:

―Ludwig Wittgenstein, ''Tractatus Logico-Philosophicus'' (1921)</blockquote>

I'd purchased a [~~http~~://~~www.caselabs~~-~~store~~.com/~~magnum-th10a~~/ CaseLabs Magnum T10] in 2013 (the Magnum T10 has been discontinued; the linked Magnum TH10A is similar), and was determined to finally make fitting use of it. The T10 is a gorgeous, brilliantly engineered monster of a double-wide reconfigurable case, weighing in at 24 pounds of aluminum. At 15 inches wide, 25.06 inches tall, and 20.06 inches deep (381mm x 637mm x 510mm), it'll easily fit EATX, SSI-CEB or even XL-ATX motherboards, dozens of hard drives, dual PSUs, and the powerful radiators/fans necessary to quietly cool it all. With casters attached, it could transport a small child in comfort and style.

I'd purchased a [https://nick-black.com/tabpower/MAGNUM%20Case%20Owner's%20Manual.pdf CaseLabs Magnum T10] in 2013 (the Magnum T10 has been discontinued; the linked Magnum TH10A is similar), and was determined to finally make fitting use of it. The T10 is a gorgeous, brilliantly engineered monster of a double-wide reconfigurable case, weighing in at 24 pounds of aluminum. At 15 inches wide, 25.06 inches tall, and 20.06 inches deep (381mm x 637mm x 510mm), it'll easily fit EATX, SSI-CEB or even XL-ATX motherboards, dozens of hard drives, dual PSUs, and the powerful radiators/fans necessary to quietly cool it all. With casters attached, it could transport a small child in comfort and style.

Multisocket and custom liquid-cooled solutions are thus in play. Look upon my works, ye Mighty, and despair!

Line 266:

Line 270:

―Thomas Pynchon, ''Gravity's Rainbow'' (1973)</blockquote>

You can't go multisocket without C612, though, and C612 got me down. It doesn't bring certain workstationish elements I want, seems USB-starved, and even large SSI motherboards don't seem to put as many I/O slots on there as one could. Aesthetics are brutish and fan support is limited. IPMI scares the hell out of me. So far as I could tell, no multisocket enthusiast/DIY scene exists. Shit shit shit. Meanwhile, there were any number of simply lovely "Generation 2" X99 boards, loaded with valuable capabilities that would consume PCIe slots on the SSI boards, PCIe slots I needed for GPUs and fast storage (with that said, I did ~~appreciated~~ the 10GigE built onto many C612 boards, and [https://www.sprezzatech.com/blog/0009-a-disquisition-into-the-sadly-slovenly-takeup-of-10gbase-t.html lament its absence] on all save one X99). And anything but the 2x2640v4 was going to be ''awfully'' expensive in absolute terms. Shit shit shit. And it's not like $1880 is exactly cheap. See, this is what multi-socket thinking does to you. Shit. Alright. Single, boring old unisocket on a phat X99, FML.

You can't go multisocket without C612, though, and C612 got me down. It doesn't bring certain workstationish elements I want, seems USB-starved, and even large SSI motherboards don't seem to put as many I/O slots on there as one could. Aesthetics are brutish and fan support is limited. IPMI scares the hell out of me. So far as I could tell, no multisocket enthusiast/DIY scene exists. Shit shit shit. Meanwhile, there were any number of simply lovely "Generation 2" X99 boards, loaded with valuable capabilities that would consume PCIe slots on the SSI boards, PCIe slots I needed for GPUs and fast storage (with that said, I did appreciate the 10GigE built onto many C612 boards, and [https://www.sprezzatech.com/blog/0009-a-disquisition-into-the-sadly-slovenly-takeup-of-10gbase-t.html lament its absence] on all save one X99). And anything but the 2x2640v4 was going to be ''awfully'' expensive in absolute terms. Shit shit shit. And it's not like $1880 is exactly cheap. See, this is what multi-socket thinking does to you. Shit. Alright. Single, boring old unisocket on a phat X99, FML.

===The processor===

Line 288:

Line 292:

* The CaseLabs Magnum T10's PSU side could really use some fan cutaways.

* When both U.2 connectors are populated with NVMe SSDs, Linux notes corrected PCIe errors at a rate proportional to load.

** Update: this ~~no longer seems~~ to ~~happen with kernels since at least 4~~.14, ~~probably earlier~~

** Update: this was due to [https://en.wikipedia.org/wiki/Active_State_Power_Management ASPM], which I have disabled. Problem gone.

* The ASUS X99 Deluxe II's hardware sensors do not appear to have Linux support.

** Update: with the kernel command line option <tt>acpi_enforce_resources=lax</tt>, the nct6775 chip can be found.

Line 294:

Line 298:

==Final Bill of Materials==

===Chassis===

* [https://~~www.caselabs~~-~~store~~.com/~~the-magnum-double-wide-case-line~~/ Caselabs Magnum T10] chassis with [http://www.caselabs-store.com/magnum-th10-a-and-mh10-top-covers-pricing-varies/ 78mm ventilated top] plus...

* [https://nick-black.com/tabpower/MAGNUM%20Case%20Owner's%20Manual.pdf Caselabs Magnum T10] chassis with [http://www.caselabs-store.com/magnum-th10-a-and-mh10-top-covers-pricing-varies/ 78mm ventilated top] plus...

** 2x [http://www.caselabs-store.com/double-wide-magnum-standard-hdd-cage/ Caselabs MAC-101] HDA+fan cages

** 2x [http://www.caselabs-store.com/2-5-hdd-ssd-mounting-kit/ Caselabs 2.5" MAC-322] SSD mounts

Line 310:

Line 314:

* [https://www.asus.com/us/Motherboards/X99-DELUXE-II/ ASUS X99 Deluxe II] motherboard

* [http://ark.intel.com/products/94456/Intel-Core-i7-6950X-Processor-Extreme-Edition-25M-Cache-up-to-3_50-GHz Intel 6950X] Broadwell-E CPU

* 4x [http://www.crucial.com/usa/en/bls4k16g4d240fsc Crucial Ballistix Sport LT] 16GB DDR4 DIMMs (64GB total)

* 4x [http://www.crucial.com/usa/en/bls4k16g4d240fsc Crucial Ballistix Sport LT] 16GB DDR4-2400 DIMMs (64GB total)

* [https://www.asus.com/Graphics-Cards/ROG-STRIX-GTX1080-A8G-GAMING/ ASUS GTX1080 A8G] video card (NVidia GP104 GPU)

* [https://www.amazon.com/gp/product/B01A5DGJQA E-SDS USB/HD-Audio] 3.5" front panel

* [https://www.amazon.com/gp/product/B008KEPWMY NZXT AA-APMU3-B1] 5.25" USB/media card reader

* [https://www.amazon.com/gp/product/B01DQQLH74/ ASUS R2.0] 14-pin TPM

===Storage===

* 2x [http://www.intel.com/content/www/us/en/solid-state-drives/solid-state-drives-750-series.html Intel 750] 400GB NVMe U.2 SSDs

* 12x [https://www.amazon.com/gp/product/B0759Q9FXZ Seagate EXOS] 12TB SATA3 HDDs

* 2x [https://www.amazon.com/gp/product/B01COOQ3MS Funtin U.2 to Mini-SAS] cables

* 2x [https://www.westerndigital.com/products/internal-drives/pc-sn720-ssd Western Digital SN720] 1TB NVMe SSDs

* 4x [http://www.wdc.com/global/products/specs/?driveID=1324 WD40EFRX Western Digital Red] 4TB SATA3 HDDs

* 2x [http://www.intel.com/content/www/us/en/solid-state-drives/solid-state-drives-750-series.html Intel 750] 400GB NVMe U.2 SSDs (removed)

* 4x ST4000VN000 Seagate NAS 4TB SATA3 HDDs

* 2x [https://www.amazon.com/gp/product/B01COOQ3MS Funtin U.2 to Mini-SAS] cables (removed)

* Samsung 840 Pro 256GB SATA3 SSD

* 4x [http://www.wdc.com/global/products/specs/?driveID=1324 WD40EFRX Western Digital Red] 4TB SATA3 HDDs (removed, sold for $144)

* 6x ST4000VN000 Seagate NAS 4TB SATA3 HDDs (removed, sold for $216)

* Samsung 840 Pro 256GB SATA3 SSD (removed)

* Lite-on Bluray burner

Line 327:

Line 334:

* [http://edgeup.asus.com/2016/06/17/broadwell-e-overclocking-guide/ The Broadwell-E Overclocking Guide], from ASUS's EdgeUP site 2016-06-17

* ASUS X99 Deluxe II [http://www.overclockers.com/asus-x99-deluxe-ii-motherboard-review/ review] at overclockers.com 2016-08-08

[[CATEGORY: Hardware]]

Schwarzgerät: Difference between revisions

Dank (talk | contribs)

@@ Line 1: / Line 1: @@
 [[File:I76950X.jpg|thumb|Roughly one year's wages in Kyrgyzstan]]
+'''Update: this has been rebuilt as [[Schwarzgerät_II|Schwarzgerät II]].'''
 <blockquote>“There is no way to tell his story without telling my own. And if his story really is a confession, then so is mine.”
 ―''Apocalypse Now'' (1979)</blockquote>
@@ Line 5: / Line 8: @@
 '''ABSTRACT:''' In August of 2016, I pulled the trigger on a long-planned workstation build. That same month, Intel and NVIDIA dropped new product. Hot, salivation-provoking product: densely packed marvels bursting with FLOPS, rough dank beasts woven up from high-κ 16nm and 14nm strained-silicon FinFETs. Both companies, utterly dominant at their markets' high ends, announced pricing that shoved atom bombs up the ass of every computing enthusiast in the free world.
-This is the rambling, poorly-edited, relentlessly technical story of that build. Also: epigraphs. You can skip to the [[#final bill of materials|bill of materials]], if you just want specs.
+This is the rambling, poorly-edited, relentlessly technical story of that build. Also: epigraphs. You can skip to the [[#Final_Bill_of_Materials|bill of materials]], if you just want specs.
 ==Intro: 10 cores of garbage==
@@ Line 19: / Line 22: @@
 ==Build Goals==
+[[File:Teragross.png|thumb|12x 12TB EXOS drives]]
 <blockquote>“Money has only a different value in the eyes of each.”
 ―William Makepeace Thackeray, ''Vanity Fair'' (1847), Chapter XLIV</blockquote>
@@ Line 43: / Line 47: @@
 ―Ludwig Wittgenstein, ''Tractatus Logico-Philosophicus'' (1921)</blockquote>
-I'd purchased a [http://www.caselabs-store.com/magnum-th10a/ CaseLabs Magnum T10] in 2013 (the Magnum T10 has been discontinued; the linked Magnum TH10A is similar), and was determined to finally make fitting use of it. The T10 is a gorgeous, brilliantly engineered monster of a double-wide reconfigurable case, weighing in at 24 pounds of aluminum. At 15 inches wide, 25.06 inches tall, and 20.06 inches deep (381mm x 637mm x 510mm), it'll easily fit EATX, SSI-CEB or even XL-ATX motherboards, dozens of hard drives, dual PSUs, and the powerful radiators/fans necessary to quietly cool it all. With casters attached, it could transport a small child in comfort and style.
+I'd purchased a [https://nick-black.com/tabpower/MAGNUM%20Case%20Owner's%20Manual.pdf CaseLabs Magnum T10] in 2013 (the Magnum T10 has been discontinued; the linked Magnum TH10A is similar), and was determined to finally make fitting use of it. The T10 is a gorgeous, brilliantly engineered monster of a double-wide reconfigurable case, weighing in at 24 pounds of aluminum. At 15 inches wide, 25.06 inches tall, and 20.06 inches deep (381mm x 637mm x 510mm), it'll easily fit EATX, SSI-CEB or even XL-ATX motherboards, dozens of hard drives, dual PSUs, and the powerful radiators/fans necessary to quietly cool it all. With casters attached, it could transport a small child in comfort and style.
 Multisocket and custom liquid-cooled solutions are thus in play. Look upon my works, ye Mighty, and despair!
@@ Line 266: / Line 270: @@
 ―Thomas Pynchon, ''Gravity's Rainbow'' (1973)</blockquote>
-You can't go multisocket without C612, though, and C612 got me down. It doesn't bring certain workstationish elements I want, seems USB-starved, and even large SSI motherboards don't seem to put as many I/O slots on there as one could. Aesthetics are brutish and fan support is limited. IPMI scares the hell out of me. So far as I could tell, no multisocket enthusiast/DIY scene exists. Shit shit shit. Meanwhile, there were any number of simply lovely "Generation 2" X99 boards, loaded with valuable capabilities that would consume PCIe slots on the SSI boards, PCIe slots I needed for GPUs and fast storage (with that said, I did appreciated the 10GigE built onto many C612 boards, and [https://www.sprezzatech.com/blog/0009-a-disquisition-into-the-sadly-slovenly-takeup-of-10gbase-t.html lament its absence] on all save one X99). And anything but the 2x2640v4 was going to be ''awfully'' expensive in absolute terms. Shit shit shit. And it's not like $1880 is exactly cheap. See, this is what multi-socket thinking does to you. Shit. Alright. Single, boring old unisocket on a phat X99, FML.
+You can't go multisocket without C612, though, and C612 got me down. It doesn't bring certain workstationish elements I want, seems USB-starved, and even large SSI motherboards don't seem to put as many I/O slots on there as one could. Aesthetics are brutish and fan support is limited. IPMI scares the hell out of me. So far as I could tell, no multisocket enthusiast/DIY scene exists. Shit shit shit. Meanwhile, there were any number of simply lovely "Generation 2" X99 boards, loaded with valuable capabilities that would consume PCIe slots on the SSI boards, PCIe slots I needed for GPUs and fast storage (with that said, I did appreciate the 10GigE built onto many C612 boards, and [https://www.sprezzatech.com/blog/0009-a-disquisition-into-the-sadly-slovenly-takeup-of-10gbase-t.html lament its absence] on all save one X99). And anything but the 2x2640v4 was going to be ''awfully'' expensive in absolute terms. Shit shit shit. And it's not like $1880 is exactly cheap. See, this is what multi-socket thinking does to you. Shit. Alright. Single, boring old unisocket on a phat X99, FML.
 ===The processor===
@@ Line 288: / Line 292: @@
 * The CaseLabs Magnum T10's PSU side could really use some fan cutaways.
 * When both U.2 connectors are populated with NVMe SSDs, Linux notes corrected PCIe errors at a rate proportional to load.
-** Update: this no longer seems to happen with kernels since at least 4.14, probably earlier
+** Update: this was due to [https://en.wikipedia.org/wiki/Active_State_Power_Management ASPM], which I have disabled. Problem gone.
 * The ASUS X99 Deluxe II's hardware sensors do not appear to have Linux support.
 ** Update: with the kernel command line option <tt>acpi_enforce_resources=lax</tt>, the nct6775 chip can be found.
@@ Line 294: / Line 298: @@
 ==Final Bill of Materials==
 ===Chassis===
-* [https://www.caselabs-store.com/the-magnum-double-wide-case-line/ Caselabs Magnum T10] chassis with [http://www.caselabs-store.com/magnum-th10-a-and-mh10-top-covers-pricing-varies/ 78mm ventilated top] plus...
+* [https://nick-black.com/tabpower/MAGNUM%20Case%20Owner's%20Manual.pdf Caselabs Magnum T10] chassis with [http://www.caselabs-store.com/magnum-th10-a-and-mh10-top-covers-pricing-varies/ 78mm ventilated top] plus...
 ** 2x [http://www.caselabs-store.com/double-wide-magnum-standard-hdd-cage/ Caselabs MAC-101] HDA+fan cages
 ** 2x [http://www.caselabs-store.com/2-5-hdd-ssd-mounting-kit/ Caselabs 2.5" MAC-322] SSD mounts
@@ Line 310: / Line 314: @@
 * [https://www.asus.com/us/Motherboards/X99-DELUXE-II/ ASUS X99 Deluxe II] motherboard
 * [http://ark.intel.com/products/94456/Intel-Core-i7-6950X-Processor-Extreme-Edition-25M-Cache-up-to-3_50-GHz Intel 6950X] Broadwell-E CPU
-* 4x [http://www.crucial.com/usa/en/bls4k16g4d240fsc Crucial Ballistix Sport LT] 16GB DDR4 DIMMs (64GB total)
+* 4x [http://www.crucial.com/usa/en/bls4k16g4d240fsc Crucial Ballistix Sport LT] 16GB DDR4-2400 DIMMs (64GB total)
 * [https://www.asus.com/Graphics-Cards/ROG-STRIX-GTX1080-A8G-GAMING/ ASUS GTX1080 A8G] video card (NVidia GP104 GPU)
 * [https://www.amazon.com/gp/product/B01A5DGJQA E-SDS USB/HD-Audio] 3.5" front panel
 * [https://www.amazon.com/gp/product/B008KEPWMY NZXT AA-APMU3-B1] 5.25" USB/media card reader
 * [https://www.amazon.com/gp/product/B01DQQLH74/ ASUS R2.0] 14-pin TPM
 ===Storage===
-* 2x [http://www.intel.com/content/www/us/en/solid-state-drives/solid-state-drives-750-series.html Intel 750] 400GB NVMe U.2 SSDs
+* 12x [https://www.amazon.com/gp/product/B0759Q9FXZ Seagate EXOS] 12TB SATA3 HDDs
-* 2x [https://www.amazon.com/gp/product/B01COOQ3MS Funtin U.2 to Mini-SAS] cables
+* 2x [https://www.westerndigital.com/products/internal-drives/pc-sn720-ssd Western Digital SN720] 1TB NVMe SSDs
-* 4x [http://www.wdc.com/global/products/specs/?driveID=1324 WD40EFRX Western Digital Red] 4TB SATA3 HDDs
+* 2x [http://www.intel.com/content/www/us/en/solid-state-drives/solid-state-drives-750-series.html Intel 750] 400GB NVMe U.2 SSDs (removed)
-* 4x ST4000VN000 Seagate NAS 4TB SATA3 HDDs
+* 2x [https://www.amazon.com/gp/product/B01COOQ3MS Funtin U.2 to Mini-SAS] cables (removed)
-* Samsung 840 Pro 256GB SATA3 SSD
+* 4x [http://www.wdc.com/global/products/specs/?driveID=1324 WD40EFRX Western Digital Red] 4TB SATA3 HDDs (removed, sold for $144)
+* 6x ST4000VN000 Seagate NAS 4TB SATA3 HDDs (removed, sold for $216)
+* Samsung 840 Pro 256GB SATA3 SSD (removed)
 * Lite-on Bluray burner
@@ Line 327: / Line 334: @@
 * [http://edgeup.asus.com/2016/06/17/broadwell-e-overclocking-guide/ The Broadwell-E Overclocking Guide], from ASUS's EdgeUP site 2016-06-17
 * ASUS X99 Deluxe II [http://www.overclockers.com/asus-x99-deluxe-ii-motherboard-review/ review] at overclockers.com 2016-08-08
+[[CATEGORY: Hardware]]