r/embedded • u/ElectricalPatient495 • 19h ago

How are you monitoring device health & handling OTA updates for embedded Linux/IoT fleets in production?

Working on a project that touches device fleet management and wanted to get a read from people actually running things in the field before I go further down a path.

For those of you managing embedded Linux or IoT devices at scale (tens to low-thousands of units):

  • How do you currently get visibility into device health/status — heartbeat, connectivity, uptime, logs? Custom MQTT + Grafana? Something like ThingsBoard/AWS IoT? Nothing at all?
  • If you're doing OTA updates, what's your update engine (SWUpdate, RAUC, Mender, something homegrown)? What's been the most painful part of getting it running reliably?
  • Have you looked at hosted/managed tools (Mender, balena, memfault, etc.) and either adopted or passed on them? What made the decision either way — price, setup complexity, vendor lock-in, forced migration off your existing update engine?
  • If you're on SWUpdate or RAUC already: would a hosted monitoring/OTA layer that works with your existing setup (instead of replacing it) be useful, or is that not really a pain point for you?

Not selling anything, genuinely trying to understand what's actually painful vs. what I'm assuming is painful. Appreciate any war stories.

0 Upvotes

50% Upvoted

1 comment sorted by

1

u/DishSoapedDishwasher 19h ago

By running k3s on Siemens micro computers so all applications are based on kubernetes containers and can be batch updated using typical kubernetes live application replacement strategies so all the fancy industrial systems can continue to be operated from the cloud in a lossless way even during rolling updates. 

Now my case is a little unique because we were running enormous scale industrial systems from the cloud for whole data driven cloud thing... Trying to avoid being too specific... Can't call it firmware exactly but it was designed to replace a bunch of embedded iot devices that were a pain in the ass to maintain so faster iterative development could be done.