Small thing but it got me last week. I relay my stream to twitch through nginx-rtmp on a cheap VPS, and one morning unattended-upgrades pulled a new systemd, needrestart restarted nginx (and postgres, and everything else) and the stream dropped. OBS reconnected fast but twitch counted it as two streams.
default timer is 06:00 UTC + up to 1h random, which is exactly when I stream lol. moved it with a drop-in:
# systemctl edit apt-daily-upgrade.timer
[Timer]
OnCalendar=
OnCalendar=*-*-* 13:45:00 UTC
RandomizedDelaySec=15m
the empty OnCalendar= matters, without it you just add a second time and keep the old one.
anyone else got bitten by this?
Unattended upgrade?
Why? Specially on critical systems?
If you do such upgrades, schedule for a time you don’t use the system at very least …
I’ve been running it on a Debian VPS for years as a test. So far not a single problem, although I don’t let it reboot on its own (I do monthly reboots, that I am thinking of changing to fortnightly).
same here on the reboots, I never let it reboot on its own either. for me the reboot wasn’t the problem, it was needrestart bouncing nginx and postgres right after the upgrade, which you only notice if something long lived is running at that moment, like a stream lol
I mean, unattended upgrades has it’s own config file where you can specify when to schedule a reboot (when needed), which you’d usually do in off-hours like 2 AM your local time, so changing the Systemd unit activation time seems like the wrong fix.
Edit: Nevermind, your issue was that the daemons were being rebooted, not the whole system, if I understood that correctly
yeah exactly, no reboot, it was needrestart restarting nginx and postgres after the systemd update. you’re right that the timer is kind of the blunt fix, the cleaner one is probably telling needrestart to only list instead of restart ($nrconf{restart} = ‘l’ in needrestart.conf), or excluding nginx from it. I went with the timer because I still want things restarted, just not while I’m live
I’ve been burnt by this, my nginx lxc on proxmox was crashing at about 6:30 every morning for a week. Turns out auto update and debsecure was running on it and using all the 512mb of memory and the lxc would get killed by the host, so removed both as I update Tuesday evenings anyway
512mb and getting killed every morning at the same time sounds like a fun one to track down lol. a fixed weekly slot like your tuesday evening is probably the sanest setup honestly, I just know I’d forget.
I have some of my works colleagues join and we all do system updates and talk crap for the evening, its a great way to help others sort things
that’s actually a nice idea, updating with people around makes it way less of a chore, and someone always knows the weird fix. I’d probably steal it if I had more than one box to look after.
What timezone are you?
I’m in the UK, kick off it about 7:45
