Check out my first novel, midnight's simulacra!
X10QBi: Difference between revisions
| (13 intermediate revisions by the same user not shown) | |||
| Line 5: | Line 5: | ||
==Memory boards== | ==Memory boards== | ||
There are 8 slots for SuperMicro memory boards, of which there are three variants (different boards cannot be mixed): | There are 8 slots for SuperMicro memory boards, of which there are three variants (different boards cannot be mixed): | ||
* 10QBi-MEM1 1.01: DDR3 | * 10QBi-MEM1 1.01: 1333MHz DDR3 (Jordan Creek 1) | ||
* 10QBi-MEM1 2.0: DDR3 | * 10QBi-MEM1 2.0: 1600MHz DDR3 (Jordan Creek 2) | ||
* 10QBi-MEM2 1.01: 1866MHz DDR4 | * 10QBi-MEM2 1.01: 1866MHz DDR4 (Jordan Creek 2) | ||
The X10QBi supports ECC memory, correcting single-bit errors and detecting double-bit errors. Jordan Creek is a "Scalable Memory Buffer" capable of running in "lockstep" or "performance" mode. In Lockstep, the SMB runs at the same speed as the DIMMs, you can recover from two faults, less power is consumed, and bandwidth is diminished. Performance mode uses two DIMMs per memory channel, consumes more power (approximately 9W vs 7W), and can recover from a single fault. Each processor supports four controllers, and each controller supports 6 DIMMs. With all four sockets populated, this yields 96 DIMMs on sixteen controllers. 32GB DIMMs in all 96 slots yield 3 terabytes of RAM. Each socket is wired to two memory boards, two controllers per board, and thus all eight boards | The X10QBi supports ECC memory, correcting single-bit errors and detecting double-bit errors. Jordan Creek is a "Scalable Memory Buffer" capable of running in "lockstep" or "performance" mode. In Lockstep, the SMB runs at the same speed as the DIMMs, you can recover from two faults, less power is consumed, and bandwidth is diminished. Performance mode uses two DIMMs per memory channel, consumes more power (approximately 9W vs 7W), and can recover from a single fault. Each processor supports four controllers, and each controller supports 6 DIMMs. With all four sockets populated, this yields 96 DIMMs on sixteen controllers. 32GB DIMMs in all 96 slots yield 3 terabytes of RAM. Each socket is wired to two memory boards, two controllers per board, and thus maximum bandwidth requires all eight boards, with at least 4 DIMMs per board, in "Performance" mode. | ||
The manual claims that MEM2 boards are required to use v3/v4 Xeons, but my experience is that these newer processors work fine with MEM1, so long as you have the most recent firmware. | The manual claims that MEM2 boards are required to use v3/v4 Xeons, but my experience is that these newer processors work fine with MEM1, so long as you have the most recent firmware. | ||
== | Different cards probably cannot be mixed, though I've never tried it. | ||
==SDOM== | |||
The X10QBi has a port for a "[https://www.supermicro.com/products/nfo/SATADOM.cfm SATA Disk-On-Module]", basically a small amount of flash on a SATA3 interface. They come in 16, 32, 64, and 128GB versions. While the SuperMicro page would seem to suggest that the X10QBi does not require a 5V power connector for the SDOM, my experience suggests otherwise (also, the SDOM connector on the X10QBi is a white one, similar to the X9 interface on that page, rather than the yellow X10 interface). This connector is just a 28AWG 5V+Ground 2-pin to 5V+Ground+Ground 3-pin, model number CBL-PWEX-0625. | |||
==Failure modes and boot codes== | |||
If the machine immediately goes to state FF and doesn't leave, the CPU output ports are not being written to. You have either a bad motherboard, at least one bad processor, and/or at least one badly-seated processor. | If the machine immediately goes to state FF and doesn't leave, the CPU output ports are not being written to. You have either a bad motherboard, at least one bad processor, and/or at least one badly-seated processor. | ||
Sometimes the IPMI subsystem gets fucked, manifesting in missing sensor values. I've managed to repair this with <tt>ipmitool mc reset cold</tt>. Wait for a few minutes, and see if running <tt>ipmi-sensors</tt> doesn't now show all expected samples. | |||
==Updating BIOS via BMC== | ==Updating BIOS via BMC== | ||
The most recent BIOS is 3.2 from 2019-08-08, <tt>X10QBi9.808</tt>. | The most recent BIOS is [https://www.supermicro.com/en/support/resources/downloadcenter/firmware/MBD-X10QBi/BIOS 3.2] from 2019-08-08, <tt>X10QBi9.808</tt>. The most recent BMC firmware is 03.90 from 2020-07-17. You will need the newest BIOS in order to boot from NVMe. | ||
<pre>[schwarzgerat](0) $ ./SMCIPMITool BMCADDRESS USERNAME PASSWORD bios update | <pre>[schwarzgerat](0) $ ./SMCIPMITool BMCADDRESS USERNAME PASSWORD bios update -N -R -MER -FORCEREBOOT BIOSFILE | ||
Option: Reboot | |||
Option: NVRAM overwrite | |||
Option: ME region | |||
Option: SMBIOS preserve | Option: SMBIOS preserve | ||
=============== | =============== | ||
| Line 45: | Line 55: | ||
Done | Done | ||
[schwarzgerat](0) $ </pre> | [schwarzgerat](0) $ </pre> | ||
==Virtual boot media== | |||
The BMC allows you to supply an ISO as virtual boot media...but it requires a SAMBA share. You'll want "server min protocol = NT1" in your <tt>smb.conf</tt> if you're seeing errors along the lines of "Requested protocol [NT LM 0.12]" in your samba logs. | |||
==See Also== | ==See Also== | ||
* My machine [[strangelet]] is based on the X10QBi platform | * My machine [[strangelet]] is based on the X10QBi platform | ||
==External Links== | |||
* ServeTheHome's [https://forums.servethehome.com/index.php?threads/x10qbi-and-v3-v4-cpus-e-g-supermicro-sys-4048b-trft.27636/ "X10QBI and v3/v4 cpus (e.g. supermicro sys-4048b-trft)"] thread | |||
* [https://hackaday.io/project/177892-supermicro-to-atx-power-supply-adapter Supermicro to ATX PSU] adapter on hackaday.io | |||
* [https://www.unixgr.com/pinout-for-supermicro-fp836-front-panel-connector/ Pinout for Supermicro] front panel connector | |||
[[CATEGORY: Hardware]] | [[CATEGORY: Hardware]] | ||
Latest revision as of 08:00, 1 April 2024
A SuperMicro motherboard supporting four LGA2011 CPUs and eight SuperMicro memory boards. The physical board comes in at least versions 1.01, 1.2a, and 1.21. It requires a SuperMicro AOM-X10QBi-A/L I/O board to boot (the card uses a proprietary slot). This card implements BNC/IPMI, offers two 10GigE ports via an Intel controller, and supports VGA. The chipset is Intel C602J.
Contrary to some information online, it boots just fine off a single CPU in Socket 1. It requires about five minutes to reach UEFI handoff, and goes through several dozen boot codes (some of them repeated) before doing so. Similarly, you can boot off any subset of CPU sockets, so long as there's one in socket 1, and there is working memory on board 1.
Memory boards
There are 8 slots for SuperMicro memory boards, of which there are three variants (different boards cannot be mixed):
- 10QBi-MEM1 1.01: 1333MHz DDR3 (Jordan Creek 1)
- 10QBi-MEM1 2.0: 1600MHz DDR3 (Jordan Creek 2)
- 10QBi-MEM2 1.01: 1866MHz DDR4 (Jordan Creek 2)
The X10QBi supports ECC memory, correcting single-bit errors and detecting double-bit errors. Jordan Creek is a "Scalable Memory Buffer" capable of running in "lockstep" or "performance" mode. In Lockstep, the SMB runs at the same speed as the DIMMs, you can recover from two faults, less power is consumed, and bandwidth is diminished. Performance mode uses two DIMMs per memory channel, consumes more power (approximately 9W vs 7W), and can recover from a single fault. Each processor supports four controllers, and each controller supports 6 DIMMs. With all four sockets populated, this yields 96 DIMMs on sixteen controllers. 32GB DIMMs in all 96 slots yield 3 terabytes of RAM. Each socket is wired to two memory boards, two controllers per board, and thus maximum bandwidth requires all eight boards, with at least 4 DIMMs per board, in "Performance" mode.
The manual claims that MEM2 boards are required to use v3/v4 Xeons, but my experience is that these newer processors work fine with MEM1, so long as you have the most recent firmware.
Different cards probably cannot be mixed, though I've never tried it.
SDOM
The X10QBi has a port for a "SATA Disk-On-Module", basically a small amount of flash on a SATA3 interface. They come in 16, 32, 64, and 128GB versions. While the SuperMicro page would seem to suggest that the X10QBi does not require a 5V power connector for the SDOM, my experience suggests otherwise (also, the SDOM connector on the X10QBi is a white one, similar to the X9 interface on that page, rather than the yellow X10 interface). This connector is just a 28AWG 5V+Ground 2-pin to 5V+Ground+Ground 3-pin, model number CBL-PWEX-0625.
Failure modes and boot codes
If the machine immediately goes to state FF and doesn't leave, the CPU output ports are not being written to. You have either a bad motherboard, at least one bad processor, and/or at least one badly-seated processor.
Sometimes the IPMI subsystem gets fucked, manifesting in missing sensor values. I've managed to repair this with ipmitool mc reset cold. Wait for a few minutes, and see if running ipmi-sensors doesn't now show all expected samples.
Updating BIOS via BMC
The most recent BIOS is 3.2 from 2019-08-08, X10QBi9.808. The most recent BMC firmware is 03.90 from 2020-07-17. You will need the newest BIOS in order to boot from NVMe.
[schwarzgerat](0) $ ./SMCIPMITool BMCADDRESS USERNAME PASSWORD bios update -N -R -MER -FORCEREBOOT BIOSFILE Option: Reboot Option: NVRAM overwrite Option: ME region Option: SMBIOS preserve =============== BIOS Image info =============== Date = 08/08/2019 MB Type = X10QBi Size = 16 MB ============= BIOS ROM info ============= 0726 BIOS Date: 6/12/2018 ==================== Uploading BIOS image ==================== Progress:|>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>| 100% Upload Time: 0 min 48 sec(s) ============= Updating BIOS ============= Progress:|>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>| 100% Update Time: 5 min 33 sec(s) Total Elapse Time: 6 min 38 sec(s) Done [schwarzgerat](0) $
Virtual boot media
The BMC allows you to supply an ISO as virtual boot media...but it requires a SAMBA share. You'll want "server min protocol = NT1" in your smb.conf if you're seeing errors along the lines of "Requested protocol [NT LM 0.12]" in your samba logs.
See Also
- My machine strangelet is based on the X10QBi platform
External Links
- ServeTheHome's "X10QBI and v3/v4 cpus (e.g. supermicro sys-4048b-trft)" thread
- Supermicro to ATX PSU adapter on hackaday.io
- Pinout for Supermicro front panel connector
