nixos, mostly. It has a very steep learning curve but if it breaks I’m just “whatever, rollback now, fix properly later”.

I actually don’t remember off the top of my head, huh. The output chain is not the one I use often. I’d think nft add rule tracing filter output ip protocol icmp icmp type echo-reply meta nftrace set 1 would do it. Just make sure the priority is low enough again.

systemd will try to save any kernel panics in pstore via systemd-pstore, if it’s enabled in your kernel. I’d check /var/lib/systemd/pstore and see if anything in there on the next boot.

Can you elaborate on ping “running”? Do you get actual icmp replies coming back? Because there’s no code path I can imagine where a ping would cause hdd activity (on a normally running system).

If ssh times out (and you don’t do anything fancy with the firewall), then it’s not sshd dead, it’s sshd not being able to respond. Grab a tcpdump for dst port 22 from your local machine while RPi is stuck and see if you get any replies whatsoever or it’s just retransmits going into the void.

My first rough suspicion would be ram abuse. Something eats up all ram and the system locks up and semi-dies. Pstore would have OOMs. You could run a local script for telemetry recording too to see if ram use spikes up before the system gets unresponsive.

It is accepted just fine then. You might want to check the reverse, start in top of the output chain and trace the echo reply.

nftables offers a very decent debugging interface. First, you add a rule to trace the packet (a new chain with high priority works best). Usually I’d suggest to add those rules by hand instead of relying on declarative configs:

nft add table ip ping_trace
nft 'add chain ip ping_trace prerouting { type filter hook prerouting priority -301; policy accept; }'

Then you add the actual tracing rule there with the shape to match the incoming traffic:

nft 'add rule ip ping_trace prerouting icmp type { echo-request, echo-reply } meta nftrace set 1'

now you can run nft monitor trace and see the decisions made for the matching packets. Remember to delete the ping_trace table afterwards to clean up.

First on how to get an ASN: you can buy it for reasonably cheap from a LIR. Some will even toss a free /48 with that. Happy to offer names in private so that there’s no advertising. Expect a budget quoted above.

Once you have an ASN, you need to get an upstream - actually two as RIPE mandates at least two (otherwise why’d you need an ASN). Some LIRs would offer transit with ASN purchase. You can upstream via your ISP, if they allow you to (that’s very rare). Another option is a tunnel (there are free and paid ones) or a VM somewhere (some cloud providers offer to set up bgp with VMs they host). Generally, free ones are enough for basic stuff. Not much bandwidth and oftentimes IPv6 only, but you don’t pay anything either. Besides, you can ask around in various network related chats. Practically, I can offer ip transit with some marginally low burstable bandwidth, and that’s pretty common. You can look/ask around https://discord.gg/ipv6 for example.

For getting ASN to your homelab you’re looking at a tunnel option, most probably. Great if you have static ipv4 - allows you to use more common tunnels, but is still doable with a floating IP (e.g. check bgptunnel).

BGP anycast person here. If you have any presence in the region RIPE operates in then the pricing is within the homelab reach. ASN and a block of /48 would be about 70 EUR/year.

Alternatively, something like route64 would happily tunnel you IPs they announce for about 2 EUR/month.

I’d suggest VictoriaMetrics over pure Prometheus - better ram footprint is pretty useful nowadays.

You probably want to mention that some prominent cryptographers think this just astroturfing?

I actually had lots of fun writing a clojure compiler in rust that would target wasm. This allows to write it all in rust and then get it to self-host bit by bit.

Or you could, idk, have some metrics. That’s a wild idea, of course. Who uses monitoring when you can just ask an LLM?

A second offsite NAS with your friend? That’s what I did when I grew out of my old synology. My new NAS capacity is noticeably impacted by things like frequent local snapshots but I don’t need to back those up remotely and it saves space.

While matter technically requires IPv6 there are non-confirming devices (glares at Hue) that will only do IPv4. It will work with most of the matter controller networks, because in the end it’s a conscious decision to disable IPv4 there.

You really want the ECC ram and the motherboard/cpu combo that supports it.

If your note’s type is JSON (or TW’s native dictionary), you can query it as such in filters.

My search problem is that I rely on metadata a lot. It’s natural for me to want a UI that renders machine readable metadata in a way that my brain can process and that requires rich querying capabilities.

I tried them all and, so far, TW wins for me, with orgmode being second close (I like orgmode in vim, but it had some fatal rendering flaws and I don’t feel like using emacs just for notes).

tiddlywiki has one of the most insane search engines from this list. They have a whole filters syntax that can express pretty much anything imaginable, no? I went back to TW from Obsidian because I was tired from Obsidian’s trivial search functionality.

Slothmud (https://slothmud.org/) is still fun and has that very classic MUD vibe.

Let’s untangle those problems. I have a similar setup so I just want to share some ideas to show that you don’t need to copy keys.

If I’m traveling or I wipe my device or get a new one, I would have to add the new key to many servers as authorized keys

If you oftentimes access ssh from untrusted systems you’re kind of in a bad spot to begin with. The best thing you can have is a yubikey on a keychain. Everything else means you leak secret material (a password or a key) to a machine you don’t inherently trust.

Also, I want a key backed up in case of disaster since all of my devices are in my home most of the time

Again, something that you can easily solve with a hardware key [in a safe]. But realistically, in case of a disaster a local shell password login should be good enough?

I’d recommend you to think about what attacks are you trying to prevent by using a shared private key. I’m not saying it’s a bad concept, inherently having it in your password manager (like 1Password that even has ssh-agent support) is pretty common. The problem with just the keys is that it’s non-trivial to expire them if needed. You might be indeed better off with some web based authentication that you can access from any place which would ask you secret questions/send you a text message or do whatever 2FA you deem sufficient and mint you a short-lived certificate for ssh.

Not an answer, but I’m curious: what’s wrong with just having several ssh keys, one per device?

Next step is discovering atuin! https://atuin.sh/