I wanted to start contributing to an open source software project yesterday evening, and they recommend virtual box to not mess with your default installation of the program and the databases it uses.

So I thought Debian would be a nice clean distro for developing Python… Gnome feels really unusual to me and I hate it, I guess I can replace it with KDE.

But I couldn’t install a specific Python version? System python is 3.13 but I needed 3.10. I tried adding the deadsnake ppa but Debian didn’t know the add-apt-repository command. So I tried to install software-properties-common which also failed because the package couldn’t be located. Someone on SO said it was removed because security but I mean wtf? So the solution is to add this package cgabbelt manually to sources.list but I couldn’t get it to run because I couldn’t verify the GPG key… Then I went to sleep.

I am pretty sure this community can help with the problem, but honestly, wtf? I am not a Linux power user but a data scientist who works on Linux for a couple of years now, how is it possible installing a specific Python version is such a hassle?

Is Debian just a poor choice for developing? The software I want to contribute to has many dependencies, they recommend Ubuntu but fuck Ubuntu. So I guess I can’t take something too exotic.

  • TechNom (nobody)@programming.dev
    link
    fedilink
    English
    arrow-up
    10
    ·
    14 hours ago

    Here’s my 2¢. Debian is a reasonable OS to develop on, since it provides a super stable and reasonably secure base platform. I’ve used it quite extensively for the same purpose without any issues whatsoever.

    However, it won’t satisfy the needs of the modern style of development. For that, you need a reproducible development environment so that every developer gets the same results. (That means avoiding the ‘it works for me’ type of bugs). That means you need the same version of runtimes (python), same version of libraries/dependencies and same developer tools on every development system, irrespective of the distro it’s running on.

    The problem with Debian is that it often ships older versions of software to keep it stable. It will likely not match the version of python and tools you need. For that matter, no distributions including the frequently updated Arch Linux are likely to meet those requirements. (There are exceptions - NixOS and Guix.) So the widely adopted solution is to create a dev environment independent of your core system, from your regular non-root account. That means separately installing a python runtime that’s different from your distro repo, etc. They don’t touch your core system and keep it clean and pristine.

    The way we achieve this is by using 4 tools:

    1. Runtime version manager: Manages the version of runtimes.
    • Eg: python: pyenv, node: nvm, rust toolchains: rustup
    1. Environment manager: Creates and maintains isolated environments where the correct runtimes, project dependencies and tools are available.
    • Eg: python: venv, node: node project, rust: rust project
    1. Dependency manager: Resolves and installs the correct version of all project dependencies into the environment.
    • Eg: python: pip, node: npm, pnpm, rust: cargo add
    1. Tool manager: Installs additional development tools for the project, like formatter, linter, etc.
    • Eg: python: pip, pipx, node: npx, rust: cargo install

    Often, many of these are combined and you may get less than 4 different tools. The current situation is extremely complicated and there are many different tool combinations you can use. So let’s address your specific requirements.

    If your project uses only Python

    In this case, the choice is pretty straightforward. Use uv.

    The package management situation in the Python ecosystem was an absolute mess until UV appeared on the scene. UV combines all the 4 functions I mentioned above. It replaces venv, pyenv, pip and pipx in a single fast binary. You’ll be surprised by its speed if you’re used to the speed of pip. It’s easy to use and very well integrated. It also integrates additional functionalities like:

    • Formatter (replaces black)
    • Project setup (helps with poetry, flit, hatch, etc)
    • Project publishing (replaces twine)

    If you will use more than Python

    If you will use tools or languages other than Python in a single project or in other future projects, you might want to use a ‘language-agnostic runtime and tool manager’. They can manage runtimes and tools of multiple languages. Dependencies have to be managed using language package managers (like UV, pip, cargo, npm, etc).

    The most well known tool manager is asdf. Others include aqua, vfox, etc. But the upcoming star is mise. I use mise for multiple projects including for Python projects. Mise uses UV behind the scenes for Python. So, mise projects play well with UV projects and with others who use UV.

    The only disadvantage with multitool managers like asdf and mise is that they tend to be more complex compared to single language tools like UV. They obviously handle more and provide more features. However, investing time in tools like mise pays in the long run when you’re handling multiple languages.

    Servers like databases

    The tools I mentioned above don’t handle daemons like postgresql, redis, etc. This is why your colleagues recommend VMs. I will talk about VMs in a while. But I want to show you some simpler solutions here.

    Another commenter has already mentioned the use of docker-compose files to set up such servers. It’s the easiest solution possible.

    Another more refined solution specifically for development containers is testcontainers. It’s essentially the same as the docker compose solution, but with more dials and switches to help with automated tests like unit tests during CI. You’ll have to learn a bit more than docker compose, to use it. However, those test servers are also readily available online and require little configuration.

    Do you need virtualbox?

    The methods explained above don’t ruin your base Debian install. So a dedicated VM is not really required. However, I’m leaving this information here for completeness.

    Use of VMs was widespread in the past. But they didn’t run virtualbox, VMware or Qemu directly. Instead, a CLI frontend tool was used to set up those VMs for development. The most common tool was Hashicorp’s Vagrant. Another tool available today is Lima. These tools mount your project directory into the VM, set up its network, install required tools, start required services (like DBs), attach a shell for you to work on, etc. These VMs are complete development environments and you don’t need to do anything on the host system other than starting them up.

    Since the advent of containers, the same idea has been implemented using containers instead of VMs. These are obviously less resource intensive than VMs. Most of them follow the devcontainers standard. So a devcontainer configuration works on multiple platforms, including GitHub’s famous codespaces. Local tools for it include devpod, ona, devbox and devenv.

    Conclusion

    There are a lot more solutions. But these are the ones you’re most likely to settle on. So I leave it at that. Please let me know if you have any questions about this reply. Hope you find your favorite setup soon.

    • gigachad@piefed.socialOP
      link
      fedilink
      English
      arrow-up
      1
      ·
      13 hours ago

      Thanks so much for your efforts writing this up. I will study it and come back if I have any doubts.

      The reason I use VirtualBox is simply, that it is the recommendation in the developers guidelines of the project, which is gramps by the way.

      I have basic experience with Docker, I managed to build a container that runs with alls dependencies based on alpine, then it struck me - I don’t have a GUI in my container which I need of course to run my IDE and the software itself. I read about X11 forwarding but then wasn’t sure if that is the right way. I am pretty sure one can isolate some things using docker-compose, but it can be frustrating if you want to get into the project codebase and then have to do so much infrastructure stuff.

      I guess it is a combination of lack of skills (I derive data from data for my job, ML etc.) and the project itself using outdated methods because of historical development.

      But as I said, I need to do my research. Thanks again for your friendly way of explaining things, much appreciated.

      • TechNom (nobody)@programming.dev
        link
        fedilink
        English
        arrow-up
        1
        ·
        10 hours ago

        The reason I use VirtualBox is simply, that it is the recommendation in the developers guidelines of the project

        That’s a perfectly reasonable requirement. The problem here is that solution space is like the JavaScript ecosystem. There are more solutions than you can count and no two developers ever agree on everything. Everyone eventually settles down on one solution or the other based on their personal preferences. That’s why I thought it was important to give you an overview rather than a definite solution.

        I just want to add that VirtualBox is a backend to ‘Vagrant’. Developers usually depend on Vagrant, rather than fiddle with VirtualBox directly. Vagrant takes care of some low-level VM configuration on its own and consistently recreates the VM every time with little fuss. You will need a configuration file for the VM (Vagrantfile), either readily available or newly written. I recommend Vagrant rather than direct vbox due to its convenience. However, I’m not pushing it further since I can’t judge your options better than you.

        I am pretty sure one can isolate some things using docker-compose, but it can be frustrating if you want to get into the project codebase and then have to do so much infrastructure stuff.

        Certainly! That’s what tools like vagrant are meant for. The only problem is that it replaces tediousness with choice fatigue!

        I don’t have a GUI in my container which I need of course to run my IDE and the software itself.

        This is where the convenience of tools like VM, devpod, etc shines. However, you can also implement this solution manually with VMs or Docker. It’s simple - bind mount the source directory on host into the VM/container. Then you can edit it using devtools and IDE from both the host and the VM. Tools like vagrant and devpod manages this for you.

        However, the situation gets a bit tricky when you want to use GUI applications inside the VM/container. That involves mounting the X11/Wayland socket from the host into the VM/Container, doing the correct UID/GID mapping and setting the correct file permissions. Developers usually don’t do this manually since it’s tedious. However, some tools manage that too. I don’t remember which one, off hand.

        Thanks again for your friendly way of explaining things, much appreciated.

        Glad I can help! Take your time - there are a lot of tools to explore, unless you’re on a mission. Good luck!