I work on an HPC and often I have to share files with other users. The most approachable solution is to have an external cloud storage and recline back and forth. However there’s some projects that are quite heavy (several TB) and that is unfeasible. We do not have a shared group. The following is the only solution I found which is not to just set al permissions to 777, and I still don’t like it.
Create a directory and set ACL to give access to the selected users. This works fine if the users create new files in there, but it does not work if they copy from somewhere else as default umask is 022. Thus the only appropriate solution is to change default umask to 002, which however affects file creation system wide. The alternative is to change permissions every time you copy something, but you all know very well that is not going to happen.
Does it really have to be such a pain in the ass?
You don’t need additional storage. It’s one program you need to set up.
It is not something I can setup on that server, I would need a separate server to set up something of that kind.
If it’s a compliance problem, I get it. From a practical standpoint, FTP or WebDAV don’t require installing anything.
Not strictly about compliance, setting up FTP or WebDAV is technically complex without root access. You’d have to account for the fact that sessions on a HPC are time limited. Probably you can come up with some workaround that way, but I’m not sure that is any better than my current setup.
You could use Copyparty. No root needed.
I didn’t fully understand what this software does, but it looks pretty neat. However, regardless of what it does, this is not something that I could use in my case. This spawns a server and I imagine it can do it’s thing only as long as the server is running. Which in my case it would be around 8 hours which is the login session time limit on the HPC. Moreover, I’d be running a potentially resource hungry process on a login server, which is a big problem. I could request a compute job and run this there I guess, but this would still be limited by the queue max time. Moreover, while not impossible, the login node to login node communication would be a pain in the ass. I’d have to either always connect to the same node to spawn it or to let everyone know of the IP the server is currently running on. And I’d have to do this manually every 8 hours. It is feasible, but this is probably a better software for other kinds of problems.
Previously you said you didn’t want to duplicate the files because it was a ton of data. Now you’re saying that accessing it on demand is impractical.
It’s becoming difficult to help you. Not because of your technical context, but your attitude.
Copyparty introduction
I’m grateful for all the help and advice in here. Duplicating the data is not a problem, we can have several copies of the data on the server, not an issue.
Having the data on an external server on the other hand may be a problem, because that would require quite a large amount of storing capacity.
I’m unsure what you mean by accessing on demand: data already is on the server and people can access it. My main pain point is that if people copy stuff in there rather than creating it in place I do not get write access by default.
The copy party software looks interesting for other applications, and I may pick it up for something else, but it is not something that would work in this case. As I have explained extensively, while spawning a file server would not be impossible, it would be a huge hassle with no real advantages.