**beep ** bop.

  • 1 Post
  • 54 Comments
Joined 2 years ago
cake
Cake day: July 1st, 2023

help-circle

  • My fear is, that if i don’t document well or not use ansible, I will be hating my life once my server dies and I have to restore my data and also set um my services again in a few years.

    I’ve been there plenty of times, you’re not alone. There are two solutions to that problem, really, and it boils down to the classic pet vs cattle.

    1. Everything is a pet

    Pets mean you care about every server. If it breaks, it’s cheaper for you to fix it than redeploy. The overwhelming majority of your setup will be pets. Why? It’s simpler. Things don’t break that often, and when they do, it’s okay to be low-effort in fixing them.

    Write docs for yourself, even if it’s just notes on the sequences of commands to run to redeploy things. You will thank yourself when the server finally dies in two years and you have notes on how to bring everything back.

    1. Everything is a cattle

    Cattle means there’s no difference between server A and B. Everything is replaceable. Ultimately, whatever you run can run to the same extent in AWS, your basement NAS, or on your desk PC.

    Cattle is also a lot of work. You will learn an excruciating amount of things about storage, networking, visualisation, workload scheduling, and such. And it’s easy to be demotivated because of how much there is to learn.

    So take it easy. Concur that your hobby world is full of pets, but learn how to do the cattle approach at your leisure. You’ll realise that in every practical cattle setup, there are still pets, and that automating yourself from complexity only means you add layers of it somewhere else.



  • I’m in a same boat, honestly.

    Matrix has decent clients but managing a matrix instance is a world of pain, especially if you federate. And its resource use is really bad then: a single user instance can easily demand 4gb ram if you are in a couple popular chatrooms. Key propagation is oftentimes broken. Clients all have mixed support of features.

    Xmpp is a joy to host, but there are no decent clients for iOS.

    IRC is easy to host, but the IRCv3 coverage for clients is also meh.

    I was looking for something that I could throw at casual people with relative ease and there’s just not a thing. Even the “techy” chat is in discord nowadays.




  • One thing about grafana, though, is that you get logs, metrics and monitoring in the same package. You can use loki as the actual log store and it’s easy to integrate it with the likes of journald and docker.

    Yes, you will have to spend more time learning LogQL, but it can be very handy where you don’t have metrics (or don’t want to implement them) and still want some useful data from logs.

    After all, text logs are just very raw, unstructured events in time. You may think that you only look into them very occasionally when things break and you would be correct. But if you want to alert on them, oftentimes that means you’re going from raw logs to structured data. Loki’s LogQL does that, and it’s still ten times easier to manage than the elastic stack.

    VictoriaMetrics has its own logging product too, now, and while I didn’t try it yet, VM for metrics is probably the best thing ever happened since Prometheus. Especially for resource constrained homelabs.


  • Storage box networking can be hit and miss. It’s ok for incremental uploads, but I went through hell and back to get the initial backup finish, which makes me wonder what it would take to download it in case I have to.

    Scp breaks off once in a while, and WebDAV terminates the session. I didn’t try smb as I feel it’s a rather weird protocol for the public internet. In the end, I figured it’s not the networking per se, it’s something with the timeouts on the remote, and I was able to finish the backup using a Hetzner-hosted server as a jumpbox.

    But it’s cheap, yeah.









  • Seq is expecting structured logs which yours aren’t. So you want to either convert your app’s logs into a structured format (which is generally hard for a random third-party application) or use a log collector that’s fine with non-structured logs (e.g. Loki+grafana don’t care about the shape is your logs and you can format the output while querying).




  • I have a dedicated vm for things that are crucial to the home network, either latency-critical or network related.

    That’d be my dns resolver (I enforce it over VLANs by hijacking anyone trying to do DNS to other resolvers, like random IoT devices), homebridge for less important home automaton and my own matter controller for most important home automaton (controlling the lights).

    My router of choice is RouterOS in another VM. I tried opnsense, pfsense, vyatta, and a bunch of others (even a containerized Cisco route), and I settled on ROS, because it was the only one who could do IPv6 properly (apart from Cisco, but that has other issues).

    For the less important things I run them on k8s and really, there are only two bits worth mentioning as essential: ArgoCD and nixhelm. Together, they provide effortless and mostly automated software updates with very easy rollbacks. I don’t have to go and manually update every single bit of software and that saves huge amounts of time.