Any way to compress game files on linux? I know that all games vary, content vary and so on, but is there any things i can look out for in like an average steam game which I can compress, and the game can still run fine, tools or programs or anything?
What?
The Btrfs and ZFS filesystems support compression.
This is how I do it. I put my Steam directory in an external BTRFS drive with compression turned on. Works great, and allows me to take my library between computers.
You’d get even more savings using something like
beesbecause it does block level deduplication.What
beesdoes is build a hash table of every block on your ssd, and compares them. If it finds any matches, it will delete one and just place a pointer to the other where the deleted one was, the pointer being much smaller than the duplicate data block.Functionally, any installed games with shared assets get space savings. It’s particularly helpful on with Steam games because of all the proton prefixes. Lots of opportunities for finding duplicate data blocks.
If you use snapshots, it can save even more.
Have you measured the difference? Is it huge? I’m curious to understand what the trade-offs are and how many more games can you store on the same drive.
Depends on the game. A title like dota will have a lot of savings (tens of gigabytes last I checked it). Most other games it seems don’t have obviously and easily compressible assets from a generic compression algorithm, and therefore will yield extremely minor savings.
Checked some installed games. DotA is down to 36 gb from 67. Almost 54% the original size.
RoboCop rogue city is 37->37gb. Negligible savings.
Btrfs is usually last in perf benchmarks, so I wouldn’t do it myself
My main goals are efficient space usage and portability, so I’ve never measured anything. I’ll get some stats for you.
I would appreciate even not very accurate comparison just to get the idea. A brief search meanwhile showed me it’s a great idea actually. But having someone who uses that to comment would be great too. Thanks.
I’ll be away for a while, but I’ll get some stats, because now I’m also curious.
This is another reason on why dodi / fitgirl repacks are so popular, they are masters on this.
If you make any changes to the game files in Steam, including compression, then Steam will update the files and redownload them. And for online games, you could even get a ban if an Anticheat system thinks you try to cheat because the files are tempered (changed).
However there is a transparent compression available on filesystem level. Meaning files are saved compressed and look and work like uncompressed files. Something similar what Windows has I believe. EXT4 does not have transparent compression, but I often read Btrfs and ZFS do. I have no experience with that, so cannot assist. But at least you should be able to search for this now. Have in mind that compressed filesystem would be slower.
Have in mind that compressed filesystem would be slower.
Often the oppositre is true, depending on case. Compressed files load faster, so if you have the cpu power to spare (which you usually have in games while loading) and loading speed is the bottle-neck then compression speeds things up, often considerably.
And even in the age of ssds processing data and moving it through ram is much faster than the disk, so even for writing some amount of transparent compression is possible without affecting speeds.
If you have a fast CPU adding compression can actually improve performance. https://www.phoronix.com/review/btrfs-zstd-compress
with fast nvme not that much… not like the ‘olden days’ of msdos and stacker where it was actually noticeable on some systems.
Hmm, looks like it can be slower on write, but faster on read depending on the CPU and drive probably. Games mostly read files, so this might even improve performance.
I would suggest Btrfs for regular use. ZFS is better for server use and redundancy in systems with critical data you don’t want to lose. It also uses all your ram as cache and will give it back to you if you need it, but still has a harder performance hit on things you run. Be overkill for just wanting to save a little space. A lot of new gaming distros like Bazzite suggest using Btrfs on all secondary drives.
Aren’t most games assets files already compressed?
With
compress=zstdZFS reports a 1.30 compression ratio on my Steam dataset, compressing it from ~1.2TB (reported by the file manager) down to 904GB (reported byzfs list -o name,usedds). Pretty good I think.Obviously it depends on the games. AAA crap will probably add tens/hundreds of GBs of pre-compressed asset blobs, while indie and older games will often have more loose file structures with config files, scripts, runtimes etc. that all compress extremely well. And with older games, even when compressed the algorithms are often far from ideal and zstd can still get a few more percent out of them.
Btrfs here, with zsdt:3, on my game data the compression ratio is a bit worse than that, but it does save some space to my surprise.
with default btrfs zstd compression here, i save a whole 2 percent on game install directories…but 48 percent elsewhere (trixie kde, ~ 2800 dpkg + flatpaks. ‘documents’ are on a diff filesystem)
deleted by creator
Btrfs disk with periodical dedup.
Does dedup no longer shred the file system?
Never did with my weekly cronjob. But i have ext4 on the system partition, btrfs on the 1.5 TB game partition.
Edit: system partition, boot partition is fat32.
Use ZFS
As noted by others, I’m pretty sure many different solutions exist. But, FWIW, I’ve had good results with DwarFS.
There are about a million ways to do what you’re asking. Which one is the right choice depends on the particular hardware you’re using and what you want to accomplish.
