Blahaj.zone experienced a security breach and is handling it to properly reduce the risk of harm to their users. the current eta for their reture is in about 7 hours.
well, there goes 75% of shitposts on lemmy.
Also a vital support network for LGBT+ people, I guess?
Don’t worry, we’re back again :)
Also a vital support network for LGBT+ people, I guess?
Mhm and the instance where the only women’s community is hosted :(
Our piefed instance (which hosts our remaining communities) should be back up soon!
Thank you!!
i know this isn’t my place as a cis white dude, but can anarchist.nexus be a resource for a women’s space? i’ve been looking for gaps in the threadiverse we could nurture and foster. my #1 concern lately with the makeup of the hegemony of the threadiverse lately has been:
- eurocentricism
- authoritarianism by default
- black and white thinking (see above)
- unchecked and unchallenged mysoginy
my commitment to you as a cis white dude admin would be to stay the fuck out of the way of moderators looking to host a women’s issues community here, within reason. obviously it wouldn’t be okay to break instance rules against bigotry and such, but i can offer my commitment to shutting my mouth and knowing my role
edit: also, i bet @poVoq@slrpnk.net and @ProdigalFrog@slrpnk.net from my second original home instance would be happy to have a women’s issues community hosted on slrpnk.net, i’m still figuring out what my new relationship and responsibility to the threadiverse is now, but i think just like how there’s more than one world news community, there could be more than one women’s issues community so that people have a preferred and fallback space on days like today. the one thing i don’t want to diminish in this comment is the incredible work people like @ada@lemmy.blahaj.zone have done in creating a welcoming space on the threadiverse for marginalized people. i just don’t want that to be work that is only done alone. we should be collaborative towards a better, freer, world
For what it’s worth, I absolutely want to see duplicate communities across multiple instances. Even if it’s a backup community, folks shouldn’t be left without a space if an instance goes offline, or if an admin goes rogue. I want to see more communities for vulnerable folk across more instances.
I wish there were more queer first instances too, and hopefully, this incident pushes someone in to spinning one up!
i know one person who’s interesting in hosting some queer focused fediverse infrastructure, however they wants no part of the threadiverse since they considers reddit and reddit-alikes to be intrinsically prone to toxicity
There used to be a women’s space on slrpnk.net (although a bit unfortunately named after a controversial Reddit community), but we could never really find women willing to moderate it longer term and the amount of dudes showing up with trollish comments became a bit too much for us admins to handle.
the amount of dudes showing up with trollish comments became a bit too much for us admins to handle.
The mods in womensstuff work really, really hard.
Maybe! LadyButterfly is the mod of the womensstuff community and she does a great job, maybe we can get this info to her.
I really appreciate your acknowledgement of some of the issues with the diversity in the threadiverse. And appreciate your support <3
@LadyButterfly@piefed.blahaj.zone , when your instance is online again let’s have a discussion about what resources i can make available to you that will center your community’s needs without someone like me colonizing the discussion.
(unless i have the wrong ladybutterfly, then my apologies)
That’s her :)
Then they transfered a file to /tmp/exp which was linux kernel CVE-2026-43500, nicknamed ‘Dirty Frag’, an RxRPC local privilege escalation. I had not patched these internal servers that nobody should have access to against this.
Lessons Learned #1:
Install your patches.
“But I have a firewall!”
That is not a sufficient control.
Install.
Your.
Fucking.
Patches!“Just patch” is advice for a windows administrator, where updates break everything so you have to sit and baby them and apply them manually.
On Linux, there are ways to enable automatic security updates, including automatic reboots, so you can safely receive the mitigations your distro provides. That way, you don’t have to worry about forgetting to patch (until the distro release becomes unmaintained, at least).
Now, dirty frag was a zero day, meaning that it was released and probably in the wild before a mitigation was pushed out to handle it. So you did need to apply an actual configuration patch… unless you had some form of kernel based isolation, which I mention as #2 of my other comment in this thread: https://programming.dev/post/52129409/24414213
Sadly, a reluctance to install patches isn’t unique to Windows administration. I worked at a site with a well functioning Satellite infrastructure and support contracts with Red Hat. And we (InfoSec) were still chasing down admins to get their shit patched. Thankfully, we had NAC and authorization to disconnect systems that feel out of compliance. Most departments got with the program pretty quick when they ignored the "please patch all critical vulnerabilities in three days’ email and ended up with a “you are out of compliance and have been disconnected” email.
And Docker had made the whole Linux situation even worse. So many devs love to spin up containers, basically disable any sort of firewall, don’t bother with IP filtering. Oh and let’s just use passwords for ssh. Also, who needs logs? It’s a container, right. So, let’s disable all logging and not forward those anywhere. Then they promptly forget about the container until we run a vuln scan and find it’s got half a dozen RCE vulns and have to run them down and ask why the fuck it’s still running.
Linux is a much better base to build on. But bad security hygiene is still rife and still really bad for security.
“Should” is a four-letter word in fields like safety and security.
Thanks for sharing the details 👍
Fookin’ hell. Nightmare scenario.
Really appreciate the transparency. Time to change my blahaj password!
Excellent writeup, and I appreciate the transparency. I have some suggestions on how to mitigate something like this from happening in the future.
- Use a separate DBMS (that is, a separate postgres/mariasql/etc container) for each service. Give each one service unique passwords, which you can define in the docker compose.
This is simpler than trying to control postgres permissions granularity. Even if one application that connects to a database gets owned, it doesn’t have access to other postgres databases, preventing data leaks/exfiltration.
- Use a virtual machine or application container based runtime for your containers.
Kata containers is a container runtime, that is virtual machine.
There is also Gvisor and Syd Box, which are application kernels. Application kernels are reimplimentations of the parts of the Linux kernel needed to run apps, and in this case both Gvisor (Go) and Syd Box (Rust) are in memory safe langauges.
Kata containers are faster, but you will need nested virtualization in order to use them. Application kernels are slower, but you can install them anywhere, including hosts where virtualization is disabled (like a VPS that doesn’t let you enable nested virtualization.
Both take a tiny bit more resources intensive due to no longer being able to share the host kernel, but for most part, it is worth it. They don’t bring an entire kernel along, just what is needed to run apps.
Both offer similar levels of isolation, and preventing applications running inside them from touching the host kernel directly. They effectively manage to prevent issues like copy fail, dirty frag, and so on, from owning your host.
They are fairly easy to install, docker has some docs here: https://docs.docker.com/engine/daemon/alternative-runtimes/ . But if you are using podman or kubernetes, you can also install them there.
- Enable automatic security updates (and reboots) on stable distros.
A large part of the draw of stable Linux like Debian or Red Hat, is that they only do security updates. They don’t do feature updates, or even bug fixes (except for critical ones). In doing so, there is essentially a guarantee of reliability, where it is impossible for updates to break anything.
This makes it possible to enable automatic security updates, and you can even configure it to automatically reboot in order to load a new kernel that includes mitigations against issues like dirty frag. Make sure your docker containers are configured to automatically restart and everything will be smooth.
“Just patch” is a good but it is never enough, and I am frustrated hearing it so frequently. The way I view it is, any time I have to patch, what I really need to do is to improve my security architecture so I never have to “patch” this specific issues again. Patches are the exact kind of security toil that I complain about in this comment.
Use a separate DBMS (that is, a separate postgres/mariasql/etc container) for each service. Give each one service unique passwords, which you can define in the docker compose.
unique passwords is good practice, but separate db server for each of the services is extreme. it brings much more resource consumption. the solution here is being subscribed to security releases and updating soon. those application kernels also sound like a good idea. and as I understand, postgres permissions were not at fault, the permission system had a bug.
Even if one application that connects to a database gets owned, it doesn’t have access to other postgres databases, preventing data leaks/exfiltration.
except that because of the bug, anyone with query permission could have become postgres superuser.
except that because of the bug, anyone with query permission could have become postgres superuser.
If a user can’t log in to a DBMS, they don’t have query permission.
separate db server for each of the services is extreme. it brings much more resource consumption.
Yes. It consumes more resources. But it’s not that much more, and you can make it fit easily. Many users using docker compose unwittingly do this since docker composes often bring their own database containers. When done consciously, you make a trade off for peace of mind.
the solution here is being subscribed to security releases and updating soon.
I addressed takes like these in the last part of my previous comment. The linked comment also elaborates on my opinions about manual updates, manually watching security releases, and other forms of security toil.
If a user can’t log in to a DBMS, they don’t have query permission.
weren’t they having access through peertube? you can’t revoke access to something that needs it.
sure, they could run dedicated postgreses for each service, but I think this kind of bug is so rare that it is not worth wasting lots of RAM on that, and even then, it would rather just deter people from hosting services they want.
also at that point, the question arises why not just run everything in qubes OS, each service in its own VM and its own VLAN. and the response is that it is unfeasible, and it was unfeasible even before the RAM shortage, unless you have enterprise hardware with 100+ GB RAM, consuming way too much power compared to a server made from desktop PC hardware, even when there is no meaningful CPU load.But it’s not that much more, and you can make it fit easily.
I run 3 separate database servers on the same OS because I made a bad decision at one point, and I regret it. I just don’t have time to merge them.
weren’t they having access through peertube? you can’t revoke access to something that needs it.
The peertube database did not have the vulnerable extension enabled. They got access but connecting to another database:
We did not have pgcrypto installed in the peertube database, but I overlooked that someone could connect to the main postgres database if they say had a nodejs plugin running.
From that database they escaped. So if they only had access to the peertube database, that path would have been cut off for the attackers.
also at that point, the question arises why not just run everything in qubes OS, each service in its own VM and its own VLAN.
VLAN’s suck. They allow for traffic to travel within each VLAN them, unmonitored and unrestricted. Sometimes red teamers call that “hard outside, soft inside, like an eggshell”.
Better, is a private VLAN. Private VLAN’s enable the firewall to monitor or block all traffic within them. For many usecases, a single private VLAN can replace the complex, many VLAN setups that people spend so much effort setting up. You just block everything within itself from communicating with eachother, and then you can explicitly allow the stuff that actually needs to talk.
Often, there is no reason to allow devices with a VLAN to communicate with eachother. And even if you are allowing intra-pvlan communication within a private VLAN, you can now monitor the traffic, which you can’t do with a regular VLAN.
Qubes OS
Yeah. That’s basically what virtual machine or application kernel based runtimes are doing. The trick is that they are designed for this usecase, with the goal of making the isolation as cheap and performant as possible. Qubes runs full Linux VM’s since it is designed for a desktop isolation usecase.
Anyway. It’s a matter of threat model and effort. I like application containers/vm container runtimes due to how simple and easy they are to run, and the clear security benefits they provide. Install in < 30 min, configure the container runtime, restart your containers, done. Switching from a regular Linux desktop to Qubes is not as simple, so it’s harder for me to recommend that, and it becomes a matter of threat model.
It’s also common to separate services out in different Proxmox VM’s, which offers isolation benefits akin to Qubes.
I run 3 separate database servers on the same OS because I made a bad decision at one point, and I regret it. I just don’t have time to merge them.
Congrats on the security isolation!
In all seriousness, you should be able to migrate them into the same database by dumping the database with pg_dump or a similar utility, and then loading them up using pg_restore. Other databases have equivalent tools. I used the Dbeaver open source database GUI to do this a few days ago.
They got access but connecting to another database:
We did not have pgcrypto installed in the peertube database, but I overlooked that someone could connect to the main postgres database if they say had a nodejs plugin running.
how? did they use the same database user account for all databases? unless I misunderstood it, peertube’s database user shouldn’t be able to operate in other databases of the same server.
VLAN’s suck. They allow for traffic to travel within each VLAN them, unmonitored and unrestricted. Sometimes red teamers call that “hard outside, soft inside, like an eggshell”.
nonsense. if you don’t use VLANs, you are essentially using a single huge VLAN. of course using VLANs don’t exclude monitoring and firewall restrictions. could as well say, networking sucks, lets disconnect everything. if you don’t set up monitoring, and you don’t set up restrictions, then yeah, there will be no monitoring and no restrictions, like on any network.
Better, is a private VLAN. Private VLAN’s enable the firewall to monitor or block all traffic within them.
didn’t you just say VLANs wholesale suck?
also, peertube will require access to the internet…
You just block everything within itself from communicating with eachother,
only if it was so simple. you can set up routing restrictions between subnets on the IP level, but the switch will gladly forward all traffic anywhere inside of the VLAN, according to the destination MAC address. with that, a compromised system can confuse every other on the VLAN with ARP poisoning and faking DHCP servers.
In all seriousness, you should be able to migrate them into the same database by dumping the database with pg_dump or a similar utility, and then loading them up using pg_restore. Other databases have equivalent tools. I used the Dbeaver open source database GUI to do this a few days ago.
yeah, that’s the easier part. but one of the databases is mysql, which I want to get rid of completely. and conversion is nontrivial. probably dbveaver could handle it though.
only if it was so simple. you can set up routing restrictions between subnets on the IP level, but the switch will gladly forward all traffic anywhere inside of the VLAN, according to the destination MAC address. with that, a compromised system can confuse every other on the VLAN with ARP poisoning and faking DHCP servers.
This is what private VLAN’s fix. The switch isolates all traffic, forwarding it to the firewall instead, where the firewall can then monitor and control it. It’s usually done with a combination of port isolation, where layer2 traffic within a subnet is essentially blocked, and proxying the arp requests. Nothing can communicate with each other directly at the Layer 2 level, only with the switch, which then replies to all ARP requests instead. Then the switch forwards the traffic that was supposed to go to that IP address through the firewall and back as Layer 3 traffic.
with that, a compromised system can confuse every other on the VLAN with ARP poisoning and faking DHCP servers.
This is what Private VlAN’s fix. Any ARP requests a device attempts is simply blocked at the switch level via port isolation. Only the switch is able to communicate with attached devices unrestricted. It serves the DHCP, and assigns and IP address… and then if the device attached to the port attempts to lie about it’s IP address or the like, the switch simply ignores it since it knows what ports are supposed to have what IP addresses (port security).
This allows for much more granular isolation, but also a simpler architecture, since you can replace complex multi VLAN setups with fewer private VLAN’s. Maybe even just a single private VLAN. I say VLAN’s suck because by default, yeah they kinda suck. You pointed out the problems. I guess private VLAN’s are a type of VLAN but they are a different thing, just like not having any VLAN’s is really technically just one big VLAN.
yeah, that’s the easier part. but one of the databases is mysql, which I want to get rid of completely. and conversion is nontrivial. probably dbveaver could handle it though.
Is it a custom service? Or a service you are consuming from someone else? If the services supports mysql/maria/postgres, I would be worried that the adapter under the hood might be using different schema or something like that.
It does look like dbeaver has a data transfer feature: https://dbeaver.com/docs/dbeaver/Data-transfer/#supported-formats
Where you can transfer data across tables, across databases, and even across database types. There is also “Data export” mentioned on that page, where you can export data to json, or other non sql formats. If the schema match, maybe you could use that to transfer.
If the services supports both mysql and postgres, you might be able to start by making a backup of the mysql database, loading that into a test database, and then setting up a clone of the service, but pointed at postgres instead. Then you could investigate the schema, or safely attempt data transfer from the mysql replica into the postgres database. Or mariadb, but mariadb is compatible with mysql (fork of it), so I am assuming you meant postgres which is different.
I’ve been wondering why it’s down, that’s my home instance. That’s unfortunate :/
Oof, that sucks :(
Thank you very much for sharing this, it makes all Lemmy instances safer. Good job!
Im curious how they could execute that postgres archive command to write a marker. Did they use the oauth token to be able to do that?
Just not sure how they can run postgres queries as a normal user. What made that possible?
Damn, i hope the femcelmemes crew is fine
Ha that instance’s mods got all upset at me for pointing out you shouldn’t celebrate “kys posts”
ouch. i really feel for you.
Oof, hope it works out for them 😣
Bummer! I was wondering what had happened. Thanks for the update!
This is horrible. And I say this as someone who’s had horrible experiences with users over there, particularly when it comes to bullying and harassment. They didn’t deserve to get shut down like this. Hope they’re able to fix whatever the problem is and get back up and running again.
