Logs from Fleets for good image

Anthias Balena
1

I have an issue I’m trying to chase down using the Fleets for good image.

After an uptime of just over two months without issue, I noticed today that only our static image asset was displaying and not the webpage it alternates between. As well I could not access the local web server on the device when I tried to connect to it. I think this has something to do with a soft spot in Anthias when the network disconnects and reconnects but I’m trying to get to the Anthias logs to know for sure. After a device reboot, everything works normally again.

How do I get to the logs if I deploy using Fleets for good? I believe SSH authentication with Fleets for good is done using public-private keys. I won’t have that, as it’s managed by the fleet based on my understanding.

Is the best way to get logs in this case to deploy using the standard Raspberry Pi image that would give me full SSH access or is there something else I’m missing?

2

You have read it correctly, and I am afraid the short answer is that you are not missing anything.

With a Fleets for Good image you cannot SSH in. The device is provisioned into the fleet we run rather than one of your own, so there is no key for you to hold and no dashboard for you to open a terminal from. Getting logs off one of these is genuinely awkward today, and that is a real weakness of the balena setup rather than something you have configured wrong. We need to look properly at giving people a way to pull logs without shell access, because at the moment the honest answer to “how do I see what happened” is “you mostly cannot”, and that is not good enough.

What we can do from our side

Because the fleet is ours, we can pull a device’s logs for you. The catch with your case is timing. The retention window is short and you have already rebooted, so the logs from the actual event are almost certainly gone.

If it happens again, get in touch before rebooting and we can look while the evidence is still there. That is worth far more than anything we can reconstruct afterwards.

On crash reports

Anthias reports crashes back to us automatically, so a hard crash would normally show up on our side without you doing anything. I went and looked, and there is nothing from around your incident.

That absence is itself a small clue. It points more towards something seizing up or running out of a resource than towards a process dying outright, which also fits what you describe: the screen kept showing the image it already had, the web interface stopped answering, and a reboot cleared it.

A likely candidate, already fixed

There is a bug that matches your symptoms quite closely. Nothing limited how many connections something could open against the device at once. A burst, for example from a network scanner, could use up all the file handles the server is allowed. Once that happens it cannot even open its own database, so every request fails and the web interface stops responding, and it stays that way until a restart.

That is fixed here. The fix landed after 2026.10.0 was released, so it is not on your device yet. It will be in the next release, along with a handful of other stability fixes from the past couple of weeks. I would not call it confirmed as your problem, but it is the first thing I would want to rule out, and the next release may simply make it go away.

If you want logs in your own hands now

Installing on Raspberry Pi OS Lite and running the installer on top gives you full SSH. It also gives you this:

~/anthias/bin/collect_debug.sh

That gathers the container logs, system state, storage health, and power and throttling readings into a single archive, and scrubs addresses, hostnames and credentials out of it first, so it is safe to attach to a forum post or a GitHub issue. It is by far the best way to hand us a full picture of a misbehaving device.

The trade is that you give up the automatic updates you get with Fleets for Good. For a device you are actively debugging, that is usually a trade worth making.

3

Thanks for this! I’ll keep an eye out if it happens again and try and send you a private message as soon as possible to pull the logs before anyone gets a chance to reboot.

I do try and keep ad hoc tabs on the resource utilization of these devices and I wasn’t seeing the memory was getting close to 100% saturated but as you said, burst of network traffic of some kind of might also cause it.

If the PR 3394 does not seem to fix it after the next release then I’ll probably set up another pie on the same network using the pi image so I can also grab logs as well.

4

Quick follow-up, because this moved faster than I expected.

The gap I described — no way to pull logs off a device you cannot SSH into — now has a fix in review: #3423. It adds GET /api/v2/system/logs to the device’s own API, with the same behaviour on a Fleets for Good device as on a self-installed one. It returns a snapshot of the system journal (which on our images also carries the container logs), filterable by count, unit, container, since and until, and it blanks WiFi keys, tokens and auth headers on the way out, so the output is safe to paste straight into a thread like this one.

The part that matters for your case is a side effect of it: the journal is now made to persist across reboots. So the next time this happens, rebooting to get the screen back no longer costs you the evidence — you can bring the device back up and then ask it for the window around the incident.

One honest caveat. It is served by the same web server that stopped answering on you, so while the device is wedged you still will not reach it. It is a tool for reading what happened afterwards, not for watching it live.

It is not merged yet and so not in a release. Until then, your plan stands: if it happens again, get in touch before rebooting.