I’ll chime in: simplicity. It’s much easier to keep a few patches that apply to local OS builds: I use Nix, so my Mastodon microVM config just has an extra patch line. If there’s a new Mastodon update, the patch most probably will work for it too.

Yes, I could build my own Docker container, but you can’t easily build it with a patch (for Mastodon specifically, you need to patch js pre-minification). It’s doable, but it’s quite annoying. And then you need to keep track of upstream and update your Dockerfile with new versions.

One thing about grafana, though, is that you get logs, metrics and monitoring in the same package. You can use loki as the actual log store and it’s easy to integrate it with the likes of journald and docker.

Yes, you will have to spend more time learning LogQL, but it can be very handy where you don’t have metrics (or don’t want to implement them) and still want some useful data from logs.

After all, text logs are just very raw, unstructured events in time. You may think that you only look into them very occasionally when things break and you would be correct. But if you want to alert on them, oftentimes that means you’re going from raw logs to structured data. Loki’s LogQL does that, and it’s still ten times easier to manage than the elastic stack.

VictoriaMetrics has its own logging product too, now, and while I didn’t try it yet, VM for metrics is probably the best thing ever happened since Prometheus. Especially for resource constrained homelabs.

Storage box networking can be hit and miss. It’s ok for incremental uploads, but I went through hell and back to get the initial backup finish, which makes me wonder what it would take to download it in case I have to.

Scp breaks off once in a while, and WebDAV terminates the session. I didn’t try smb as I feel it’s a rather weird protocol for the public internet. In the end, I figured it’s not the networking per se, it’s something with the timeouts on the remote, and I was able to finish the backup using a Hetzner-hosted server as a jumpbox.

But it’s cheap, yeah.

That’s not exactly true, synology doesn’t do anything you can’t access from an off the shelf linux (it’s your usual mdraid and btrfs). But you better know what you’re doing if you go that route.

Conduit is in no way compact either. I tuned its caches because two gigs of ram seemed ridiculous for a single-user instance but I only got the mobile client sync lag as a result.

XMPP used to be so much nicer…

This is the best answer. Your router protects you from the outside, but a local firewall can protect you from someone prodding your lan from a hacked camera or some other IoT device. By having a firewall locally you just minimize the attack surface further.

Matrix is spectacularly cursed to the point of being unusable if you self-host it. The protocol is dumb enough to lock you out of rooms hosted on another server forever if anything goes wrong with the key rotation.

I just made a mirror out of two NVMes―they got cheap enough not to bother too much with the loss of capacity. Of course, that limits what I can put there, so I use a bit of a tiered storage between my NVMe and HDD pools.

Just think in terms of data loss: are you going to be ok if you lost the data between backups? If the answer is yes, one NVMe is enough.

Another alternative: https://pushover.net

You don’t need -it because you don’t run an interactive session in docker. It might be failing because you ask for a pseudoterminal in an environment where it doesn’t make sense.

Seq is expecting structured logs which yours aren’t. So you want to either convert your app’s logs into a structured format (which is generally hard for a random third-party application) or use a log collector that’s fine with non-structured logs (e.g. Loki+grafana don’t care about the shape is your logs and you can format the output while querying).

How does it compare to archivebox in regards to specifically saving content that’s a mix of websites and YT videos?

I’ve been using FreshRSS and Reeder (now Reeder Classic) since google reader stopped being a thing. It’s pretty great.

I have a dedicated vm for things that are crucial to the home network, either latency-critical or network related.

That’d be my dns resolver (I enforce it over VLANs by hijacking anyone trying to do DNS to other resolvers, like random IoT devices), homebridge for less important home automaton and my own matter controller for most important home automaton (controlling the lights).

My router of choice is RouterOS in another VM. I tried opnsense, pfsense, vyatta, and a bunch of others (even a containerized Cisco route), and I settled on ROS, because it was the only one who could do IPv6 properly (apart from Cisco, but that has other issues).

For the less important things I run them on k8s and really, there are only two bits worth mentioning as essential: ArgoCD and nixhelm. Together, they provide effortless and mostly automated software updates with very easy rollbacks. I don’t have to go and manually update every single bit of software and that saves huge amounts of time.

I wonder if NixOS is a vacuum coffee maker for how confusing nix looks when you see it for the first time or instant coffee for how reproducible it is…

That’s just Slackware.

That’s a good point. Mind that in most production environments you’d be firewalled rather hard (especailly when it comes to logs processing which oftentimes ends up having PII). I wouldn’t trust any service that tries to use DoT or DoH in there that I couldn’t snoop on. Many deployments nowadays allow you to “punch” firewall holes based on the outgoing dns requests to an allowlisted domain, so chances are you actually want to use the glibc resolver and not try to be fancy.

That said, smaller images are always good in my book!

You’re nailing your goal then!

I would still steer you slightly towards documenting your architectural decisions more. It’s a good skill to have and will help you in a long run.

You have dozens of crate dependencies and only you know why they are in there. A high-level document on how your system interconnects and how the algorithms under the hood work will be a huge help to anyone who comes looking through your source code. We become better programmers not by reading the source code, but by understanding what it actually does.

Here’s a random trivia: your server depends on trust-dns-resolver. Why? Why wasn’t the stock resolver enough? Is that a design choice or you just wanted to have fun? There is no wrong answer but without the design notes it’s hard to figure your intent.

This looks nice, but there’s plenty free alternatives in this space which warrants a section in the readme with the comparison to other products.

You mention ram usage, but it’s oftentimes a product of event size. Based on your numbers, your average event size is about 800 bytes. Let’s call it 1kb. That’s one million events per day. It’s surely sounds more promising than Elastic, but not reaching Loki numbers, or, if you focus on efficiency, is way behind Victoriametrics Logs (based on peeking at their benches).

I think the important bits you need to add is how you store the logs (i.e. which indices you build) and what are your trade-offs. Grep is an efficient logs processor which barely uses any ram but incurs dramatic I/O costs, after all.

Enterprises will be looking at different numbers and they have lots of SaaS products to choose from. Homelab users are absolutely your target audience and you can have it by making a better UI than the alternative (victoriametrics logs aren’t that comfortable to work with) or making resource usage lower (people run k8s clusters on RPis, they sure wonder about every megabyte of ram lost) or making the deployment easier (fire and forget, and when you come to it, it works).

It sounds like lots of things and I don’t want to be discouraging. What you started there is really nice-looking. Good job!

You can enforce an always-on VPN (for at least ipsec) via an MDM profile. This kind of features isn’t found in the casual user setup options, but there’s plenty of knobs to tune in the enterprise profile configurator.

And yes, you can easily install that profile on your phone after.