I wrote a program in Python to read some huge Json files. It took forever to finish. Turned out I could optimize it with some simple changes, which reduced 10 minutes to 10 seconds (not literally, just figurative) and made a difference in huge files only. Later I rewrote the program in Rust. And you know what happened? It was slower than the Python version. Only after understanding what I was doing and making right changes made the Rust version faster.
Most of the time its the design of the code or libraries that slows down. The language might play a huge role in performance, but it depends on what type of code and application it is.
I actually wanted write a blog post about this… if I still have the project data I might explain what happened. BTW this had nothing to do with Ai.
What about the fact that runtime startup alone auto-disqualifies python for all short-lived process use-cases?
Or the fact that you can’t do concurrency especially across the language barrier from another language without multiple python runtimes orchestrated with a queue and IPC because of possible process-wide global state and similar concerns/assumptions for any python library dependency that is not all native across its own dependency chain?
“languages are just tools” and “use the right tool for the job” were already stupid catch phrases. Pretending languages don’t matter is a big further step into this shallow faux-intellectualism masquerading as wisdom.
What about the fact that runtime startup alone auto-disqualifies python for all short-lived process use-cases?
While Python’s start-up time is a lot slower than a corresponding compiled program, in most cases it’s not slow enough to matter even if the process itself is short-lived:
$ hyperfine 'python3 -c "pass"'
Benchmark 1: python3 -c "pass" (412 runs)
mean ± σ min … max
Wall Time 7.1 ms ± 0.2 ms 6.7 ms … 8.4 ms
Memory 9.4 MiB ± 0.1 MiB 9.1 MiB … 9.5 MiB
If you are running a small program so often that this kind of start-up time matters, then the loop should probably already be implemented in the program itself. In line with the article’s thesis
Yeah nah, Lua runs circles around Python both at runtime and especially at startup. I’ve made me some desktop automation scripts in Lua to call from Alfred, I had to recheck that I didn’t leave static test data in there instead of calling the script, because the results were appearing instantly.
Recently had to develop a Python project, the tests took at least ten seconds to start each single time. This is while I made over a hundred tests for my module and re-ran them after every change.
Recently had to develop a Python project, the tests took at least ten seconds to start each single time. This is while I made over a hundred tests for my module and re-ran them after every change.
Python is dog slow, but if your tests are taking 10 seconds to start (not even run!), then something is catastrophically wrong with your setup. For comparison, for one of my projects, it takes a total of 12 seconds to run nox on a clean slate, which involves setting up venvs for five major versions of Python, installing the package and its dependencies in each, and finally running the 1500 test test-suite five times
It wasn’t my project. Afaict it’s a standard pytest setup with the Litestar framework (unused in my tests) and an in-memory mock Postgre transaction with like five records per test, so choose for yourself which part of this is ‘catastrophically wrong’. However, I can indeed agree that most of it is trash.
Same approach with dozens upon dozens of tests worked snappily as heck in PHP, haven’t yet tried it with my beloved Lua.
It is, yes. But I also commented on Lua vs Python, and if you even try to time them, Lua scripts will be finished before Python starts up. Lua is so fast that people mostly don’t bother making libs for it in C, the fucking interpreter is quick enough for folks to skip any kind of compiling and just write libraries in the interpreted language. Oh, didn’t I mention? The startup thing is with the interpreter, without any kind of bytescript, JIT and such.
In the same vein, the K and Q languages and the KDB+ database by Arthur Whitney are so goddamn fast at array processing that they’re used in banking for decades despite their extremely inscrutable syntax that would shear any Python programmer’s head clean off. They just stay in the cache at most times. And they are also interpreted, not compiled at any time.
Python suffers from the same shit as Java, it has to initialize hundreds of objects before the main program even starts, and then it creates hundreds more. Neither Lua nor JavaScript do that. PHP does to some extent, but the fucker is so fast that it can recreate the entire runtime context on every single request and then drop all of it afterwards, and it’s still quicker than Python.
Oh really? Well guess what, that’s a huge benefit for Lua because its entire total size is around 500 KB. It’s loaded into memory and executed start to finish faster than you blink. If you need a ‘standard library’, just use Penlight. As I already mentioned, vast majority of libraries for Lua are written in… Lua itself, because it’s so goddamn fast. I’m using Fennel, the Lua Lisp that’s transcribed on-the-fly, and it’s still faster than Python. Lua is great for interactive tasks like desktop automation, e.g. with Alfred, Hammerspoon (which latter uses Lua throughout), or for scripting in Termux on the phone where Python would kill the memory immediately.
Lua’s problems with the absence of popular libraries aren’t because of any language issues (even the lack of null can be dealt with), but solely due to its non-adoption in the masses. It is very popular as an embedded language in gaming, though, and that’s not a coincidence, because it fits that niche precisely because of its small size and extraordinary speed.
Well guess what, that’s a huge benefit for Lua because its entire total size is around 500 KB.
Filesize doesn’t matter, if the functionality is missing. Lua is just not a direct comparison and replacement for Python. Penlight isn’t standard library. In example if I want to write a commandline script, I better off doing it in Python. The language is much better to me, has better features, has a standard library builtin and is even installed in all Linux distributions.
Lua is meant to be embedded into other applications, not meant to be used on its own.
Lua’s problems with the absence of popular libraries aren’t because of any language issues (even the lack of null can be dealt with), but solely due to its non-adoption in the masses.
Wrong. This is a design choice, because Lua isn’t meant to be used by its own. The embedded environment is meant to provide the functionality. In example in Neovim they provide their own functionality embedded within the editor. I can’t use that script on a different environment.
Filesize doesn’t matter, if the functionality is missing.
Lua is meant to be embedded into other applications, not meant to be used on its own.
Wrong. Lua is meant to be used on its own, because it’s so freaking good. It provides everything one needs to begin, while allowing for a wide array of libraries to be made. But I know you haven’t made any serious number of scripts in either language, simply because you claim ‘filesize doesn’t matter’, while Lua is so fast partly because of its size.
This is a design choice, because Lua isn’t meant to be used by its own.
Wrong again. Lua is amazing on its own, but third-party libraries make it even better.
As an OPS guy you know what really grinds my gears? Developers not paying attention to shit like this.
We have a job in production that is a powershell script calling a powershell script calling a java applet. Now you may already be thinking “what is this abomination?” And you’re right, but that java application is set to DEBUG. IN PRODUCTION. So every time it runs it generates 1gb of logs.
Do they fix it? No of course not because if it’s not broken don’t fix it. But Jesus Christ we have people developing without understanding fundamentals like “YOU DONT LEAVE DEBUG ON IN PRODUCTION”.
The thing is also that you can’t separate a language from its ecosystem.
Because it is possible to write the fastest programs in C/C++ and Rust, folks that actually need the fastest performance will use these languages and will go around opening pull requests to fix performance problems in commonly-used libraries.
So, you do often end up with the most performant design anyways, just because the language itself is fast on paper.
Counterpoint, I wrote a Python program that reads a load of JSON and runs regex searches over it. It was quite slow. Rewrote it in Rust as a straight 1:1 port, no algorithmic changes. 50x speedup.
I think you must have either done something very wrong or your Python program didn’t really do anything other than call Python built-ins that are actually written in C.
It wasn’t a very wrong what I did in Python. I did the obvious thing. While I do not remember details, I remember it had something to do with adding items I think. But sadly I have not documented this issue, that’s on me. It was only noticeable for really huge files like 50MBytes json. Edit: Most weren’t this big, but it also combined over 100 Json files into one, so a search would search them all through and be able to save as a single file.
I wrote a draft of a service that would take hours to complete. when I optimized it for multithreaded operation instead of taking hours it took around 60 seconds.
write software like you write research papers. first draft > second draft > final draft > publication
I’d love that but it’s difficult when it’s not really my project and anything that is functioning well enough gets you assigned the next ticket at work.
I wrote a program in Python to read some huge Json files. It took forever to finish. Turned out I could optimize it with some simple changes, which reduced 10 minutes to 10 seconds (not literally, just figurative) and made a difference in huge files only. Later I rewrote the program in Rust. And you know what happened? It was slower than the Python version. Only after understanding what I was doing and making right changes made the Rust version faster.
Most of the time its the design of the code or libraries that slows down. The language might play a huge role in performance, but it depends on what type of code and application it is.
I actually wanted write a blog post about this… if I still have the project data I might explain what happened. BTW this had nothing to do with Ai.
yeah, sometimes it feels like the same people complaining about python speed are the ones writing “C” code in python.
What about the fact that runtime startup alone auto-disqualifies python for all short-lived process use-cases?
Or the fact that you can’t do concurrency especially across the language barrier from another language without multiple python runtimes orchestrated with a queue and IPC because of possible process-wide global state and similar concerns/assumptions for any python library dependency that is not all native across its own dependency chain?
“languages are just tools” and “use the right tool for the job” were already stupid catch phrases. Pretending languages don’t matter is a big further step into this shallow faux-intellectualism masquerading as wisdom.
And we didn’t even get to low level stuff yet.
While Python’s start-up time is a lot slower than a corresponding compiled program, in most cases it’s not slow enough to matter even if the process itself is short-lived:
$ hyperfine 'python3 -c "pass"' Benchmark 1: python3 -c "pass" (412 runs) mean ± σ min … max Wall Time 7.1 ms ± 0.2 ms 6.7 ms … 8.4 ms Memory 9.4 MiB ± 0.1 MiB 9.1 MiB … 9.5 MiBIf you are running a small program so often that this kind of start-up time matters, then the loop should probably already be implemented in the program itself. In line with the article’s thesis
Thanks for putting numbers. People really overblow python performance issues when it literally does not matter most times.
Yeah nah, Lua runs circles around Python both at runtime and especially at startup. I’ve made me some desktop automation scripts in Lua to call from Alfred, I had to recheck that I didn’t leave static test data in there instead of calling the script, because the results were appearing instantly.
Recently had to develop a Python project, the tests took at least ten seconds to start each single time. This is while I made over a hundred tests for my module and re-ran them after every change.
Python is dog slow, but if your tests are taking 10 seconds to start (not even run!), then something is catastrophically wrong with your setup. For comparison, for one of my projects, it takes a total of 12 seconds to run nox on a clean slate, which involves setting up venvs for five major versions of Python, installing the package and its dependencies in each, and finally running the 1500 test test-suite five times
It wasn’t my project. Afaict it’s a standard pytest setup with the Litestar framework (unused in my tests) and an in-memory mock Postgre transaction with like five records per test, so choose for yourself which part of this is ‘catastrophically wrong’. However, I can indeed agree that most of it is trash.
Same approach with dozens upon dozens of tests worked snappily as heck in PHP, haven’t yet tried it with my beloved Lua.
Then its the Test environment that is slow, not Python.
It is, yes. But I also commented on Lua vs Python, and if you even try to time them, Lua scripts will be finished before Python starts up. Lua is so fast that people mostly don’t bother making libs for it in C, the fucking interpreter is quick enough for folks to skip any kind of compiling and just write libraries in the interpreted language. Oh, didn’t I mention? The startup thing is with the interpreter, without any kind of bytescript, JIT and such.
In the same vein, the K and Q languages and the KDB+ database by Arthur Whitney are so goddamn fast at array processing that they’re used in banking for decades despite their extremely inscrutable syntax that would shear any Python programmer’s head clean off. They just stay in the cache at most times. And they are also interpreted, not compiled at any time.
Python suffers from the same shit as Java, it has to initialize hundreds of objects before the main program even starts, and then it creates hundreds more. Neither Lua nor JavaScript do that. PHP does to some extent, but the fucker is so fast that it can recreate the entire runtime context on every single request and then drop all of it afterwards, and it’s still quicker than Python.
Lua also doesn’t provide everything Python does and has a totally different use case.
Oh really? Well guess what, that’s a huge benefit for Lua because its entire total size is around 500 KB. It’s loaded into memory and executed start to finish faster than you blink. If you need a ‘standard library’, just use Penlight. As I already mentioned, vast majority of libraries for Lua are written in… Lua itself, because it’s so goddamn fast. I’m using Fennel, the Lua Lisp that’s transcribed on-the-fly, and it’s still faster than Python. Lua is great for interactive tasks like desktop automation, e.g. with Alfred, Hammerspoon (which latter uses Lua throughout), or for scripting in Termux on the phone where Python would kill the memory immediately.
Lua’s problems with the absence of popular libraries aren’t because of any language issues (even the lack of
nullcan be dealt with), but solely due to its non-adoption in the masses. It is very popular as an embedded language in gaming, though, and that’s not a coincidence, because it fits that niche precisely because of its small size and extraordinary speed.Filesize doesn’t matter, if the functionality is missing. Lua is just not a direct comparison and replacement for Python. Penlight isn’t standard library. In example if I want to write a commandline script, I better off doing it in Python. The language is much better to me, has better features, has a standard library builtin and is even installed in all Linux distributions.
Lua is meant to be embedded into other applications, not meant to be used on its own.
Wrong. This is a design choice, because Lua isn’t meant to be used by its own. The embedded environment is meant to provide the functionality. In example in Neovim they provide their own functionality embedded within the editor. I can’t use that script on a different environment.
Wrong. Lua is meant to be used on its own, because it’s so freaking good. It provides everything one needs to begin, while allowing for a wide array of libraries to be made. But I know you haven’t made any serious number of scripts in either language, simply because you claim ‘filesize doesn’t matter’, while Lua is so fast partly because of its size.
Wrong again. Lua is amazing on its own, but third-party libraries make it even better.
At that point just call the person messy and move on. They just want to flex their brain.
As an OPS guy you know what really grinds my gears? Developers not paying attention to shit like this.
We have a job in production that is a powershell script calling a powershell script calling a java applet. Now you may already be thinking “what is this abomination?” And you’re right, but that java application is set to DEBUG. IN PRODUCTION. So every time it runs it generates 1gb of logs.
Do they fix it? No of course not because if it’s not broken don’t fix it. But Jesus Christ we have people developing without understanding fundamentals like “YOU DONT LEAVE DEBUG ON IN PRODUCTION”.
I don’t understand how this is even possible.
People are dumb
The thing is also that you can’t separate a language from its ecosystem.
Because it is possible to write the fastest programs in C/C++ and Rust, folks that actually need the fastest performance will use these languages and will go around opening pull requests to fix performance problems in commonly-used libraries.
So, you do often end up with the most performant design anyways, just because the language itself is fast on paper.
Counterpoint, I wrote a Python program that reads a load of JSON and runs regex searches over it. It was quite slow. Rewrote it in Rust as a straight 1:1 port, no algorithmic changes. 50x speedup.
I think you must have either done something very wrong or your Python program didn’t really do anything other than call Python built-ins that are actually written in C.
It wasn’t a very wrong what I did in Python. I did the obvious thing. While I do not remember details, I remember it had something to do with adding items I think. But sadly I have not documented this issue, that’s on me. It was only noticeable for really huge files like 50MBytes json. Edit: Most weren’t this big, but it also combined over 100 Json files into one, so a search would search them all through and be able to save as a single file.
I mean, python has a very nice set of tools that run at C speed so it’s not crazy to think you can get good performance out of it.
I wrote a draft of a service that would take hours to complete. when I optimized it for multithreaded operation instead of taking hours it took around 60 seconds.
write software like you write research papers. first draft > second draft > final draft > publication
I’d love that but it’s difficult when it’s not really my project and anything that is functioning well enough gets you assigned the next ticket at work.