I work on an HPC and often I have to share files with other users. The most approachable solution is to have an external cloud storage and recline back and forth. However there’s some projects that are quite heavy (several TB) and that is unfeasible. We do not have a shared group. The following is the only solution I found which is not to just set al permissions to 777, and I still don’t like it.
Create a directory and set ACL to give access to the selected users. This works fine if the users create new files in there, but it does not work if they copy from somewhere else as default umask is 022. Thus the only appropriate solution is to change default umask to 002, which however affects file creation system wide. The alternative is to change permissions every time you copy something, but you all know very well that is not going to happen.
Does it really have to be such a pain in the ass?
I think he meant self-hosting Opencloud
Yes. That’s what I recommended. Self-host whatever middleman software. Opencloud, WebDAV, S3, FTP, anything he puts in the middle can accomplish what he wants.
I see! Well, I currently do not have another server that has so much storage that we could use for thi purpose. Maybe in the future and that will solve a bunch of problems, this is only one of them.
We do have a storage server, but that is local only and backup only: not going to open it to the internet.
It is indeed a solution. What is absurd to me is to have to consider such a solution that requires two servers.
You don’t need additional storage. It’s one program you need to set up.
It is not something I can setup on that server, I would need a separate server to set up something of that kind.
If it’s a compliance problem, I get it. From a practical standpoint, FTP or WebDAV don’t require installing anything.
Not strictly about compliance, setting up FTP or WebDAV is technically complex without root access. You’d have to account for the fact that sessions on a HPC are time limited. Probably you can come up with some workaround that way, but I’m not sure that is any better than my current setup.
You could use Copyparty. No root needed.
I didn’t fully understand what this software does, but it looks pretty neat. However, regardless of what it does, this is not something that I could use in my case. This spawns a server and I imagine it can do it’s thing only as long as the server is running. Which in my case it would be around 8 hours which is the login session time limit on the HPC. Moreover, I’d be running a potentially resource hungry process on a login server, which is a big problem. I could request a compute job and run this there I guess, but this would still be limited by the queue max time. Moreover, while not impossible, the login node to login node communication would be a pain in the ass. I’d have to either always connect to the same node to spawn it or to let everyone know of the IP the server is currently running on. And I’d have to do this manually every 8 hours. It is feasible, but this is probably a better software for other kinds of problems.