Reflections on watercooling

dankblog! 2022-04-17, 0836 EDT, at the danktower

I've been watercooling my primary personal machine since 2020. It started off simple—a waterblock on my Threadripper 3970X, an EKWB reservoir+pump combo, and a Hardware Labs GTR360 radiator. A total of about $480 (not including fans), less than my motherboard cost. It is no longer so simple: the same machine in its current incarnation boasts a monoblock, a GPU block, a custom GPU backplane, four radiators, two reservoirs, four D5 pumps, two temperature sensors, two discrete inline sensors, and about twenty attachments of all kinds (note that this is an almost laughably overpowered setup). There's at least $1000 worth of watercooling crap in there, and another $500 of fans. I guess you could say things are getting serious.

I've put together some thoughts having watercooled for over two years.

Generic cooling notes

It might be useful to read section six from this essay of mine, but you can probably get by without it. It is very useful to have a working knowledge of the Laws of Thermodynamics and Fourier's Law (Pascal's Law and some basic hydrostatics wouldn't be bad, either).

• We don't use Fahrenheit, but it's worth knowing that each ℃ is 1.8℉, with an offset of 32. We don't really use Kelvins either, but Kelvins are equivalent to Celsius with an offset of -273.15. Rankines can fuck right off. Liquids boil at lower temperatures at higher altitudes (boiling occurs when internal vapor pressure equals atmospheric pressure). Water expands both above and below 4℃. A quick table:

℉	℃	Notes
212	100	Water boils at sea level. Very undesirable in computers.
194	90	Thermal throttling begins around here.
140	60	Upper limit for pump + some tubing.
75	23.9
60	15.6
39.2	4	Water's volume is minimized (density maximum).
32	0	STP. Water freezes at sea level, expanding by 9%.

• Neither air- nor water-based cooling can get your temperatures below ambient (this can be achieved with phase cooling, thermoelectric aka Peltier, and cyrogenics). If it's 30 degrees in your room, your coolant and component temps are going to be at least 30 degrees. It is useless to report temperatures without reporting ambient temps. The effectiveness of your cooling system is measured by how close it gets to ambient, ΔT. Likewise, your heat-generating components are going to be hotter than your coolant.
• The goal is to move heat from the heat-generating components, and ultimately from the case. If you can't remove as much heat as you generate, temperatures will rise with time. If you can remove all the heat you generate, temperatures will fall towards ambient. Effectively moving heat requires (a) thermal conductivity, (b) contact area and (c) a temperature gradient. The thermal conductivity of air is about two orders of magnitude less than water, which is about two orders of magnitude less than aluminum or copper.
  • A processor without a heatsink is depositing heat directly to stagnant air, which will heat up (reducing the temperature gradient) and can't take much heat anyway.
  • Adding a heatsink means the heat is deposited into a good conductor with more surface area...which in turn deposits into stagnant air. This air will heat up, and eventually the heatsink will, as well.
  • Adding a waterblock to the heatsink means that heat is deposited into water as opposed to air. That water has ~4 times the heat capacity of air, meaning it takes 4x heat to raise a unit mass of water by one degree. Furthermore, that water is ~800 times denser than air, meaning there's much more mass per unit volume. As noted above, the water is ~100 times as thermally conductive. It thus takes heat better, and the temperature gradient is lessened much less quickly. With that said, it's still depositing its heat into stagnant air, and will reach equilibrium with the heatsink given sufficient time.
  • Adding fans moves the hot air (ideally out of the case), to be replaced by ambient air. The ejected hot air will diffuse into the ambient air, which is (ideally) a large convecting volume.
  • Adding more water (via tubing and reservoirs) is like using a larger case: there is now more water to heat up. The heat moves through the water slowly, however; the water closest to the heat-producing components is still likely to overheat.
  • Adding water flow (via pumps) is like using internal fans, without expelling air from the case: we now make use of the full volume of the conductor, because it's moving around, and the heat is deposited throughout (much more quickly than it would move through the material itself). Without heat being expelled, however, we're still going to heat the entire volume given enough time.
  • Adding radiators at the surface of the case means the water is now depositing its heat to the ambient air. By using powerful fans colocated with the radiators, we can help the hot air diffuse more quickly into the ambient air.

We thus see the necessary components of an effective watercooling system: waterblocks (to conduct heat into the water), tubing (to contain the water), pumps (to move the water), radiators (to conduct heat into the ambient air), and fans (to move the air). Immersion cooling eliminates the need for the first two elements, at the cost of some madness.

Thermal interface material

TIM is thermally conductive viscous material applied between heat contacts. There's some between your dies and the IHS, and then some between the IHS and your waterblock/heatsink. Its purpose is to smooth out the conducting surface, which is dotted with microscopic pits that would otherwise preclude contact. TIM is one or two orders of magnitude less thermally conductive than metals making direct contact, but much more conductive than the air which would otherwise fill these regions. Thermal conductance is measured in W/Km (watts per kelvin-meter).

• I don't delid my processors, but it's an intriguing idea. Delidding is the removal of the processor's integrated heat spreader. The IHS is the metal cover atop your processor. The actual heat-generating die elements tend to be only a small portion of this total area (the remainder is mostly devoted to pins). Delidding is a bet that you have better thermal interface material than what was used between the die(s) and the IHS. This was absolutely true in the past, but modern processors often use an indium soldering solution that is just about as good, IMHO, as anything you're going to do.
• Liquid metal is electrically conductive thermal paste. It is more thermally conductive than non-metal pastes, which is Good. It will short out any electronics it touches, which is Bad. I don't mess with it. The place to use liquid metal IMHO is between the die and IHS when delidding. In this case it's contained, and you're usually replacing a crap TIM. Liquid metal's thermal conductance is no better than a good indium solder, so there's just no point with a good die-to-IHS TIM. Atop the IHS, it's unlikely that liquid metal will beat quality non-conducting paste by more than one or two degrees.
• No matter your cooling strategy, you'll be fucked without properly-applied high-quality thermal paste. It is easy to apply too little or too much. Application of the waterblock/heatsink will spread the paste for you; you want a very thin layer across the entirety of the integrated heat spreader. A thick layer will absolutely cause problems. Thermal paste ought be replaced every few years, theoretically.

It's not unwise to keep track of your ambient, coolant, and component temperatures over time. If you see the latter rising while the former is constant, your thermal material might be breaking down, or you might have dust in the machine, or you might have a contaminated waterblock.

Air cooling

Modern tower CPU coolers are pretty damn good, and can handle most processors without getting very loud or allowing the chip to get too hot. A top-of-the-line bequiet! or Noctua tower cooler can be had for $100, and will serve most people just fine. Get one, slap on some case fans, and you're done. GPU air coolers are not generally as capable (just look at the power draw of modern GPUs vs CPUs); if you intend to make heavy use of a powerful GPU, you'll be making a lot of noise. Air coolers are not generally well-suited for overclocking, but it depends entirely on how much power the die is drawing.

Your computer can almost certainly be effectively cooled with a tower cooler and some case fans.

Water cooling

Watercooling, then, has four advantages over air cooling:

• Fans are the primary producers of noise within most computers. With a good watercooling setup, fans can be run at lower speeds for the same temperatures, reducing noise. It might alternatively be possible to use fewer fans (there are no CPU or GPU fans, for instance, when these components are watercooled). With that said, I have many more fans in my current waterbuild than I've ever used in a computer, but I'm also paying much more attention to temperatures and noise. By controlling fans based off coolant temperatures rather than component temperatures, there are far fewer distracting transitions between noise regimes.
• Water can typically move more total heat than air, allowing higher performance envelopes on heat-generating components. Even without overclocking, it's easy to trigger thermal throttling on an air-cooled CPU or GPU. With overclocking in play, especially if higher voltages are used, watercooling can become necessary for proper function. Recall that the dynamic CMOS power equation is linear with regards to frequency and quadratic with regards to voltage.
• Aesthetics. If you are friends with the kind of people who appreciate a classy computer, they're sure to be wowed by a good watercooling setup, before they go behind your back and talk about how dumb you are to be watercooling.
• It sucks money out of your wallet, preventing you from spending it on cocaine.

I skipped over AIOs (all-in-ones, aka "closed loops") entirely, opting for a custom loop out of the gate. An AIO will consist of a waterblock with integrated pump, a radiator, tubing to connect the two, and fans on said radiator. AIOs are significantly cheaper, easier to install, simpler to work with, and don't end up requiring a bunch of small expensive attachments. You buy it, install it, and you're done. They have limited lifetimes due to coolant loss, though I wouldn't be at all surprised if someone brought out a refillable AIO. They're comparable to the best air coolers with regards to cooling capability, though they'll likely run quieter.

• This is an expensive hobby. I'm lucky enough to have essentially unlimited funds, as I'm a software engineer of more than twenty years with no dependents. If you can't comfortably make a $100 purchase without thinking about it, you probably shouldn't be spending money on watercooling.
• It's less risky than you think. I rejected watercooling for many years because the entire premise seemed ludicrous: one might as well fill one's box with powerful magnets, or fragmentation grenades, or alligator snapping turtles. Water doesn't belong in machines. It's why we have them in a case rather than just sitting around in the front yard. So first off, your watercooling components needn't come anywhere near your storage, which is where the irreplaceable stuff lives. My M.2 drives are all sealed away, and my spinning disks are located far away from any water. Any leak is almost certainly going to be your fault, and will be detectable before turning on any other components: this is the leak test. Hook up your power supply and watercooling components, but leave everything else unpowered. Fill and run your loop, ideally for several hours at maximum levels. Put paper towels underneath all loop components. If the paper towels are dry, go ahead and turn everything else on, hurrah. If they're wet, you have a problem. Getting water on components while they're powered down is no big deal.
• Avoid particulate coolants. They gunkify components over time, making them look like shit and reducing flow. Distilled water ought be treated with biocides and anticorrosive agents (easily available as coolant concentrates), which will be included in any premixed cooler.
• Order of components doesn't matter assuming sufficient flow. If your flow rate is very low, you can get significant localized heating, at which point heat-generating components without intermediary heat-expelling components can be suboptimal. With sufficient flow rate, your loop will reach effective equilibrium. This question comes up because if you're watercooling both a CPU and GPU (or multiple GPUs): they're generally close together, and it's unwieldy to separate them with a radiator. It's certainly not worth running distinct loops (which would require multiple pumps, which ought be able to achieve the necessary flow rate when placed in series).
• Increasing flow rate is pretty much only useful up to rough temperature equilibrium. Running your pumps higher will add more heat to the loop (assuming they're immersed, as they usually are), and reduce their lifespan.
• Tubing kinks reduce flow, as does transport (especially vertically), as do attachments (especially angled ones), as do radiators and waterblocks.
• Don't mix metals. Doing so invites galvanic corrosion, which can only be slowed down by anticorrosives. This is most easily achieved by eschewing aluminum entirely. Don't worry about bronze vis-à-vis copper.
• SSDs are meant to run hotter than hard drives. You don't need to watercool them, nor RAM. Please don't try to watercool your hard drives.

Every pump I'm aware of runs off 12V, with a maximum draw no greater than 30W. You can thus power them with any combination of Molex, SATA, or even PCIe cables, so long as you match the final form factor. I use PCIe cables with Molex converters; PerformancePCs sell a nice sleeved one. Almost all pumps are either D5 or DDC; the former is quieter and more powerful, but slightly larger. I roll exclusively with Laing D5s. PWM models can be controlled on the fly, while "Vario" models have a physical potentiometer which almost always ends up being a bitch and a half to get to. I recommend going with PWM. Varios will typically have a 3-pin hookup reporting tach, while PWM will have a 4-pin hookup.

As for soft vs hard tubing, I use soft tubing, and don't intend to ever do otherwise. Soft tubing is trivial to cut, cheap, has plenty of give, bends without a heating gun, doesn't shatter, and is already quite enough of a pain in the ass. Old types of soft tubing were prone to plasticizer leeching, but newer types don't have them. I recommend EKWB ZMT (zero-maintenance tubing), so long as you don't mind it being opaque. Cheap anti-kinking coil can wrap your soft tubing to give it more resistance, and also provide some color. (Some) hard tubing can survive higher temperatures than soft tubing, but such high temperatures are bad for your other components, especially pumps, and ought be avoided.

Reservoirs are primarily useful to keep pumps wet. Adding water volume doesn't solve any real problems in one's loop. Pumps will quickly deteriorate if run dry. It's doubtful that you'll have more than a liter of coolant throughout your loop. I've provided some component volumes in this table.

Draining your loop will be a pain in the ass no matter what. The process is simplified with drains, which ought be put as low in your loop as possible (gravitational low). With that said, no drain is going to completely clear your loop by itself; there will be some shaking and leaning involved. The use of quick-disconnect valves can help; when engaged, they partition the loop. Draining is typically accomplished with either ball valves (they have a lever which controls flow) or drain valves (they are airtight unless depressed, at which point they allow flow). Drain valves typically come with two tops: a short closed one which is typically used, and a longer, open one which is used for draining (it presses the flow control, and allows a compression fitting to be screwed in). If you don't have a free port in which you might add a drain, a T-valve can be useful.

Hard and soft tubing use different connectors. Soft tubing is best combined with G¼ compression fittings. Hard tubing seems to use barb fittings, but I don't really know much about it. Compression fittings are virtually leak-proof when used correctly. First, ensure that there's no dust on the o-ring(s). Screw the base securely into the port, being careful not to overtighten (overtightening reduces the effectiveness of o-rings. Do you want to blow up Challenger?). Insert the tube into the ring (run the tube under warm water if you encounter difficulties), until a bit of tube comes out the other side. Connect the tube to the base, pushing it as far down as it can go. Rotate the ring until it is tight and touching the base. Ideally, you should see neither threading nor tubing between the base and the ring. Go ahead and yank on the tube. It's not going anywhere.

Once everything's connected, fill your system. This will usually be done through a fill port on a reservoir. You will almost certainly need to fill the reservoir, run the pump for a second or two (only until the reservoir is emptied; do not run the pump dry!), and repeat this process several times. Eventually, running the pump will not lower the reservoir level; at this point, the system is full. Water above 4℃ expands as it heats, so you might want to leave a few milliliters empty, especially if you're using inflexible hard tubing. There might be some small air bubbles in your loop; they'll go away with time.

Theoretically, the loop ought be drained every six months or so, but I've never bothered, and suffered not at all for it. Like I said earlier, keep a record of your temps vs ambient, and if you have a problem, deal with it.

Congratulations! You're now a watercooling pimp, aka a huge dork.

previously: "bogon kernel command line options" 2022-03-12