Visualization

(4 intermediate revisions by the same user not shown)

Line 1:

My current home metrics + visualization stack is self-hosted [https://grafana.com/ Grafana OSS] atop a [https://github.com/prometheus Prometheus] time series database. Prometheus is fed by [https://github.com/hikhvar/mqtt2prometheus mqtt2prometheus], which reads [https://mosquitto.org/ Mosquitto]-brokered metrics from my [https://github.com/dankamongmen/counterforce bespoke scripts] and various IoT devices. It also ingests [[SNMP]] via its own snmp_exporter. I was originally using Prometheus's node_exporter, but thought it horribly heavyweight.

My current home metrics + visualization stack is self-hosted [https://grafana.com/ Grafana OSS] atop a [https://github.com/prometheus Prometheus] time series database. Prometheus is fed by [https://github.com/hikhvar/mqtt2prometheus mqtt2prometheus], which reads [https://mosquitto.org/ Mosquitto]-brokered metrics from my [https://github.com/dankamongmen/counterforce bespoke scripts] and various IoT devices. Grafana also ingests [[SNMP]] from my switches via its own snmp_exporter. I was originally using Prometheus's node_exporter, but thought it horribly heavyweight and kicked that bitch to the curb.

At Microsoft, I use Grafana atop an otherwise custom stack. At Google, I used an entirely custom stack because hey, SWEs gotta get promoted, and the authors of Monarch weren't gonna demonstrate complexity by adapting standard open source tooling.

==Data sources==

I've got an [[ESP8266]]-powered [https://sonoff.tech/product/smart-plugs/s31-s31lite/ Sonoff S31] smart plug on pretty much every outlet plate. They can be found for less than $20 on sale and consume negligible parasitic power. The vendor firmware is cursed, and ~~they~~ ought immediately ~~be flashed~~ with [https://tasmota.github.io/docs/ Tasmota]. Once this is done, enable MQTT. From these, I sample wattage, AC voltage, and wireless quality.

I've got an [[ESP8266]]-powered [https://sonoff.tech/product/smart-plugs/s31-s31lite/ Sonoff S31] smart plug on pretty much every outlet plate. They can be found for less than $20 on sale and consume negligible parasitic power. The vendor firmware is cursed, and ought be immediately replaced with open source [https://tasmota.github.io/docs/ Tasmota] (this takes about ten minutes per device). Once this is done, enable MQTT. From these, I sample wattage, AC voltage, and wireless quality.

[[File:Voltage-grafana.png|24 hour history of AC voltages in my condo]]

On each machine, I run a script (from [[systemd]], of course) that periodically samples a bunch of crap, marshals it into JSON with <tt>jq</tt>, and then sends it off as a unit with <tt>mosquitto_pub</tt>. Try to get everything for a single machine into a single block so as not to make more than one TLS connection per sampling period (note that this means all metrics will arrive under one topic, which might not jive with your schema). I run with a 15s quantua across all my devices. I recommend being liberal but loud in the face of sampling errors; ideally, errors in sampling will themselves be graphed, and brought to your immediate attention without breaking the device's reporting entirely.

For different machines, supply a different <tt>--id</tt> to <tt>mosquitto_pub</tt>. This will allow you to differentiate them via the <tt>sensor</tt> label in Grafana.

==Prometheus==

Line 12:

Line 16:

By default, Prometheus provides a data explorer UI and API on port 9090. Exporters (e.g. snmp_exporter) usually run their own UIs/APIs on their own ports (e.g. 9116 for snmp_exporter).

===mqtt2prometheus===

===[[SNMP]]===

Visualization: Difference between revisions

Dank (talk | contribs)

@@ Line 1: / Line 1: @@
-My current home metrics + visualization stack is self-hosted [https://grafana.com/ Grafana OSS] atop a [https://github.com/prometheus Prometheus] time series database. Prometheus is fed by [https://github.com/hikhvar/mqtt2prometheus mqtt2prometheus], which reads [https://mosquitto.org/ Mosquitto]-brokered metrics from my [https://github.com/dankamongmen/counterforce bespoke scripts] and various IoT devices. It also ingests [[SNMP]] via its own snmp_exporter. I was originally using Prometheus's node_exporter, but thought it horribly heavyweight.
+My current home metrics + visualization stack is self-hosted [https://grafana.com/ Grafana OSS] atop a [https://github.com/prometheus Prometheus] time series database. Prometheus is fed by [https://github.com/hikhvar/mqtt2prometheus mqtt2prometheus], which reads [https://mosquitto.org/ Mosquitto]-brokered metrics from my [https://github.com/dankamongmen/counterforce bespoke scripts] and various IoT devices. Grafana also ingests [[SNMP]] from my switches via its own snmp_exporter. I was originally using Prometheus's node_exporter, but thought it horribly heavyweight and kicked that bitch to the curb.
 At Microsoft, I use Grafana atop an otherwise custom stack. At Google, I used an entirely custom stack because hey, SWEs gotta get promoted, and the authors of Monarch weren't gonna demonstrate complexity by adapting standard open source tooling.
 ==Data sources==
-I've got an [[ESP8266]]-powered [https://sonoff.tech/product/smart-plugs/s31-s31lite/ Sonoff S31] smart plug on pretty much every outlet plate. They can be found for less than $20 on sale and consume negligible parasitic power. The vendor firmware is cursed, and they ought immediately be flashed with [https://tasmota.github.io/docs/ Tasmota]. Once this is done, enable MQTT. From these, I sample wattage, AC voltage, and wireless quality.
+I've got an [[ESP8266]]-powered [https://sonoff.tech/product/smart-plugs/s31-s31lite/ Sonoff S31] smart plug on pretty much every outlet plate. They can be found for less than $20 on sale and consume negligible parasitic power. The vendor firmware is cursed, and ought be immediately replaced with open source [https://tasmota.github.io/docs/ Tasmota] (this takes about ten minutes per device). Once this is done, enable MQTT. From these, I sample wattage, AC voltage, and wireless quality.
 [[File:Voltage-grafana.png|24 hour history of AC voltages in my condo]]
+On each machine, I run a script (from [[systemd]], of course) that periodically samples a bunch of crap, marshals it into JSON with <tt>jq</tt>, and then sends it off as a unit with <tt>mosquitto_pub</tt>. Try to get everything for a single machine into a single block so as not to make more than one TLS connection per sampling period (note that this means all metrics will arrive under one topic, which might not jive with your schema). I run with a 15s quantua across all my devices. I recommend being liberal but loud in the face of sampling errors; ideally, errors in sampling will themselves be graphed, and brought to your immediate attention without breaking the device's reporting entirely.
+For different machines, supply a different <tt>--id</tt> to <tt>mosquitto_pub</tt>. This will allow you to differentiate them via the <tt>sensor</tt> label in Grafana.
 ==Prometheus==
@@ Line 12: / Line 16: @@
 By default, Prometheus provides a data explorer UI and API on port 9090. Exporters (e.g. snmp_exporter) usually run their own UIs/APIs on their own ports (e.g. 9116 for snmp_exporter).
+===mqtt2prometheus===
 ===[[SNMP]]===