I once spent several angry hours researching zombie processes in a quest to kill them by any means necessary. Ended up rebooting, which was a sort of baby-with-the bath-water solution.
Zombie processes still infuriate me. While I’m not a Rust developer, nor do I particularly care about the language, I’m eagerly watching Redox OS, as it looks like the micro kernel OS with the best chance to make to it useful desktop status. A good micro kernel would address so so many of the worst aspects of Linux.
Zombie processes are already dead. They aren’t executing, the kernel is just keeping a reference to them so their parent process can check their return code (waitpid).
All processes becomes zombies briefly after they exit, just usually their parents wait on them correctly. If their parents exit without waiting on the child, then the child gets reparented to init, which will wait on it. If the parent stays alive, but doesn’t wait on the child, then it will remain zombied until the parent exits and triggers the reparenting.
Its not really Linux’s fault if processes don’t clean up their children correctly, and I’m 99% sure you can zombie a child on redox given its a POSIX OS.
I haven’t tried this, but if you just need the parent to call waitpid on the child’s pid then you should be able to do that by attaching to the process via gdb, breaking, and then manually invoking waitpid and continuing.
Zombie processes are hilarious. They are the unkillable package delivery person of the Linux system. They have some data that must be delivered before they can die. Before they are allowed to die.
Sometimes just listening to them is all they want. (Strace or redirect their output anywhere.)
Sometimes, the whole village has to burn. (Reboot)
Performance is the major flaw with microkernels that have prevented the half-dozen or more serious attempts at this to succeed.
Incurring context switching for low-level operations is just too slow.
An alternative might be a safe/provable language for kernel and drivers where the compiler can guarantee properties of kernel modules instead of requiring hardware guarantees, and it ends up in one address space/protection boundary. But then the compiler (and its output) becomes a trusted component.
Quite opposite. Most firmware microcontrollers run is giant kernel. Some microcontrollers don’t even have context switching at all. And I’m not even starting to talk about MMU.
I was not meaning to say that all microcontrollers (or microcontroller firmwares) run a microkernel but, rather, that microcontrollers are an environment where they can work well because the limited scope of what the device is expected to do and its necessarily supported peripherals can be much smaller, making the potential impact of context changes smaller.
For some good examples of microkernels for such purposes, take a look at FreeRTOS, ChibiOS, or Zephyr pre-1.6 (at which point architecture changed to a monolith because it is creeping towards general computing functionality).
Some microcontrollers don’t even have context switching at all.
As long as there’s some processing RAM and sufficient ROM, I’m sure that it can be crammed in there via firmware (in a sub-optimal way that makes people who have to maintain the code, including your future self, hate you and wish a more appropriate part were used).
At least in the docs, I see it described as a microkernel but, with a kernel that small, the differences are probably academic (and I’ll leave that to people with more formal background in CS than myself).
You can’t even emulate MPU without MPU. The only way is running bytecode, which is still not context switching.
You are correct here. Should have said MPU instead.
Oh yes! That makes a lot more sense. I’ve been on-and-off looking at implementing multithreading and multiprocessing in CircuitPython. Memory protection is a big problem with making it work reliably.
RedoxOS would likely never become feature complete enough to be a stable, useful and daily-drivable OS. It’s currently a hobbyist OS that is mainly used as a testbed for OS programming in Rust.
If the RedoxOs devs could port the Cosmic DE, they’d become one of the best Toy OS and maybe become used on some serious projects . This could give them enough funds to become a viable OS used by megacorps on infrastructures where security is critical and it may lead it to develop into a truly daily drivable OS.
They are planning to port Cosmic DE, and have already ported several applications from Cosmic including the file manager and text editor if I remember correctly.
Ok, how change of kernel would fix userspace program not reading return value? And if you just want to use microkernel, then use either HURD or whatever DragonflyBSD uses.
But generally microkernels are not solution to problems most people claim they would solve, especially in post-meltdown era.
This particular issue could be solved in most cases in a monolithic kernel. That it isn’t, is by design. But it’s a terrible design decision, because it can lead to situations where (for example) a zombie process locks a mount point and prevents unmounting because the kernel insists it’s still in use by the zombie process. Which the kernel provides no mechanism for terminating.
It is provable via experiment in Linux by use of fuse filesystems. Create a program that is guaranteed to become a zombie. Run it within a filesystem mounted by an in-kernel module, like a remote nfs mount. You now have a permanently mounted NFS mount point. Now, use mount something using fuse, say a WebDAV remote point. Run the same zombie process there. Again, the mount point is unmountable. Now, kill the fuse process itself. The mount point will be unmounted and disappear.
This is exactly how microkernels work. Every module is killable, crashable, upgradable - all without forcing a reboot or affecting any processes not using the module. And in a well-designed microkernel, even processes using the module can in many cases continue functioning as if the restarted kernel module never changed.
Fuse is really close to the capabilities of microkernels, except it’s only filesystems. In a microkernel, nearly everything is like fuse. A linux kernel compiled such that everything is a loadable module, and not hard linked into the kernel, is close to a microkernel, except without the benefits of actually being a microkernel.
Microkernels are better. Popularity does not prove superiority, except in the metric of popularity.
Create a program that is guaranteed to become a zombie. Run it within a filesystem mounted by an in-kernel module, like a remote nfs mount. You now have a permanently mounted NFS mount point.
Ok, this is not really good implementation. I’m not sure that standard requires zombie processes to keep mountpoints(unless its executable is located in that fs) untill return value is read. Unless there is call to get CWD of another process. Oh, wait. Can’t ptrace issue syscall on behalf of zombie process or like that? Or use vfs of that process? If so, then it makes sense to keep mountpoint.
Every module is killable, crashable, upgradable - all without forcing a reboot or affecting any processes not using the module.
except without the benefits of actually being a microkernel.
Except Linux does it too. If graphics module crashes, I still can SSH into system. And when I developed driver for RK3328 TRNG, it crashed a lot. Replaced it without reboot.
Microkernels are better. Popularity does not prove superiority, except in the metric of popularity.
As I said, we live in post-meltdown world. Microkernels are MUCH slower.
As I said, we live in post-meltdown world. Microkernels are MUCH slower.
I’ve heard this from several people, but you’re the lucky number by which I’d heard it enough that I bothered to gather some references to refute this.
First, this is an argument that derived from first generation microkernels, and in particular, MINIX, which - as a teaching aid OS, never tried to play the benchmark game. It’s been repeated, like dogma, through several iterations of microkernels which have, in the interim, largely erased most of those performance leads of monolithic kernels. One paper notes that, once the working code exceeds the L2 cache size, there is marginal advantage to the monolithic structure. A second paper running benchmarks on L4Linux vs Linux concluded that the microkernel penalty was only about 5%-10% slower for applications than the Linux monolithic kernel.
This is not MUCH slower, and - indeed - unless you’re doing HPC applications, is close enough to be unnoticeable.
Edit: I was originally going to omit this, as it’s propaganda from a vested interest, and includes no concrete numbers, but this blog entry from a product manager at QNX specifically mentions using microkernels in HPC problem spaces, which I thought was interesting, so I’m post-facto including it.
First, this is an argument that derived from first generation microkernels, and in particular, MINIX, which - as a teaching aid OS, never tried to play the benchmark game.
Indeed, first generation microkernels were so bad, that Jochen Liedtke in rage created L3 “to show how it’s done”. While it was faster than existing microkernels, it was still slow.
The paper is about hybrid kernels. And gutted Mach(XNU) is used as example.
Nowdays(after meltdown) all cache levels are usually invalidated during context switch. Processors try to add mechanisms to avoid this, but they create new vulnreabilities.
Can you elaborate? I am not an OS design expert, and I thought microkernels had some advantages.
Many people think that microcernels are only way to run one program on multiple machines without modyfing them. Counterexample to such statement is Plan 9, which had such capability with monolithic kernel.
That’s not something I ever associated with microkernels to be honest. That’s just clustering.
I was more interested in having minimal kernels with a bunch of processes handling low level stuff like file systems that could be restarted if they died. The other cool thing was virtualized kernels.
nah, you can have micro-kernel features on linux, but you can’t have monolithc kernel features on microkernel, there’s zero arguments in favor of a micro kernel, except being a novel project
Do explain how you can have micro kernel features on Linux. Explain, please, how I can kill the filesystem module and restart it when it bugs out, and how I can prevent hard kernel crashes when a bug in a kernel module causes a lock-up. I’m really interested in hearing how I can upgrade a kernel module with a patch without forcing a reboot; that’d really help on Arch, where minor, patch-level kernel updates force reboots multiple times a week (without locking me into an -lts kernel that isn’t getting security patches).
I’d love to hear how monolithic kernels have solved these.
I’ve been hoping that we can sneak more and more things into userspace on Linux. Then, one day, Linus will wake up and discover he’s accidentally made a microkernel.
I thought the point of lts kernels is they still get patches despite being old.
Other than that though you’re right on the money. I think they don’t know what the characteristics of a microkernel are. I think they mean that a microkernel can’t have all the features of a monolithic kernel, what they fail to realise is that might actually be a good thing.
I thought the point of lts kernels is they still get patches despite being old.
Well, yeah, you’re right. My shameful admission is that I’m not using LTS because I wanted to play with bcachefs and it’s not in LTS. Maybe there’s a package for LTS now that’d let me at it, but, still. It’s a bad excuse, but there you go.
I think a lot of people also don’t realize that most of the performance issues have been worked around, and if RedoxOS is paying attention to advances in the microkernel field and is not trying to solve every problem in isolation, they could end up with close to monolithic kernel performance. Certainly close to Windows performance, and that seems good enough for Industry.
I don’t think microkernels will ever compete in the HPC field, but I highly doubt anyone complaining about the performance penalty of microkernel architecture would actual notice a difference.
Windows is a hybrid kernel, and has some interesting layers of abstraction, all of which make it slower. It’s also full of junkware these days. So beating it shouldn’t be that hard.
Yeah to be fair in HPC it’s probably easier to just setup a watchdog and reboot that node in case of issues. No need for the extra resilience.
That’s my point. If you’re l33t gaming, what matters is your GPU anyway. If HPC, sure, use whatever architecture gets you the most bang for your buck, which is probably going to be a monolithic kernel (but, maybe not - nanokernels allow processes basically direct access to hardware, with minimal abstraction, like X11 DRI, and might allow even faster solutions to be programmed). For most people, the slight improvement in performance of a monolithic kernel over a modern, optimized microkernel design will probably not be noticeable.
I keep getting people telling me monolithic kernels are way faster, dude, but most are just parroting the state of things decades ago and are ignoring many of the advancements micro kernels like L4 have made in intervening years. But I need to go find links and put together references before I counter-claim, and right now I have other things I’d rather be doing.
you don’t need a micro kernel to install medules, nor to make a crash in certain module don’t bring the kernel down, you program it isolated, they don’t do that now because it’s unecessary, but android do that, and there’s work being doing in that way
https://www.phoronix.com/news/Ubuntu-Rust-Scheduler-Micro
the thing is that it’s harder todo that, that’s why no one does, but not impossible, you also need to give the kernel the foundation to support that
bro thinking a chromecast OS gonna run in google servers 💀, micro kernels has their utility in embedded system, we know, saying that they are replacement for monolithic kernel is dumb, also companies can’t do different/hacks project anymore?
Also fake because zombie processes.
I once spent several angry hours researching zombie processes in a quest to kill them by any means necessary. Ended up rebooting, which was a sort of baby-with-the bath-water solution.
Zombie processes still infuriate me. While I’m not a Rust developer, nor do I particularly care about the language, I’m eagerly watching Redox OS, as it looks like the micro kernel OS with the best chance to make to it useful desktop status. A good micro kernel would address so so many of the worst aspects of Linux.
Zombie processes are already dead. They aren’t executing, the kernel is just keeping a reference to them so their parent process can check their return code (
waitpid).All processes becomes zombies briefly after they exit, just usually their parents wait on them correctly. If their parents exit without waiting on the child, then the child gets reparented to init, which will wait on it. If the parent stays alive, but doesn’t wait on the child, then it will remain zombied until the parent exits and triggers the reparenting.
Its not really Linux’s fault if processes don’t clean up their children correctly, and I’m 99% sure you can zombie a child on redox given its a POSIX OS.
Edit: https://gist.github.com/cameroncros/8ae3def101efc08be2cd69846d9dcc81 - Rust program to generate orphans.
I haven’t tried this, but if you just need the parent to call waitpid on the child’s pid then you should be able to do that by attaching to the process via gdb, breaking, and then manually invoking waitpid and continuing.
I think that should do it. I’ll try later today and report back.
Of course, this risks getting into an even worse state, because if the parent later tries to correctly wait for its child, the call will hang.
Edit: Will clean up the orphan/defunct process.
If the parent ever tried to wait, they would either get ECHILD if there are no children, or it would block until a child exited.
Will likely cause follow on issues - reaping someone elses children is generally frowned upon :D.
Zombie processes are hilarious. They are the unkillable package delivery person of the Linux system. They have some data that must be delivered before they can die. Before they are allowed to die.
Sometimes just listening to them is all they want. (Strace or redirect their output anywhere.)
Sometimes, the whole village has to burn. (Reboot)
Performance is the major flaw with microkernels that have prevented the half-dozen or more serious attempts at this to succeed.
Incurring context switching for low-level operations is just too slow.
An alternative might be a safe/provable language for kernel and drivers where the compiler can guarantee properties of kernel modules instead of requiring hardware guarantees, and it ends up in one address space/protection boundary. But then the compiler (and its output) becomes a trusted component.
Thank you. Came here to say this. Microkernels are great for limited scope devices like microcontrollers but really suffer in general computing.
Quite opposite. Most firmware microcontrollers run is giant kernel. Some microcontrollers don’t even have context switching at all. And I’m not even starting to talk about MMU.
I was not meaning to say that all microcontrollers (or microcontroller firmwares) run a microkernel but, rather, that microcontrollers are an environment where they can work well because the limited scope of what the device is expected to do and its necessarily supported peripherals can be much smaller, making the potential impact of context changes smaller.
For some good examples of microkernels for such purposes, take a look at FreeRTOS, ChibiOS, or Zephyr pre-1.6 (at which point architecture changed to a monolith because it is creeping towards general computing functionality).
As long as there’s some processing RAM and sufficient ROM, I’m sure that it can be crammed in there via firmware (in a sub-optimal way that makes people who have to maintain the code, including your future self, hate you and wish a more appropriate part were used).
Some madlads forked Linux to get it to work without an MMU, even getting it merged into the mainline kernel: https://en.m.wikipedia.org/wiki/ΜClinux
So, doable. Adviseable? Probably not in most cases but that’s subjective.
AFAIK FreeRTOS always ran drivers in kernel.
You can’t even emulate MPU without MPU. The only way is running bytecode, which is still not context switching.
You are correct here. Should have said MPU instead.
At least in the docs, I see it described as a microkernel but, with a kernel that small, the differences are probably academic (and I’ll leave that to people with more formal background in CS than myself).
Oh yes! That makes a lot more sense. I’ve been on-and-off looking at implementing multithreading and multiprocessing in CircuitPython. Memory protection is a big problem with making it work reliably.
Then you would need to move compiler to kernel. Well, there is one: BPF(and derivatives). It’s turing-incomplete by design.
RedoxOS would likely never become feature complete enough to be a stable, useful and daily-drivable OS. It’s currently a hobbyist OS that is mainly used as a testbed for OS programming in Rust.
If the RedoxOs devs could port the Cosmic DE, they’d become one of the best Toy OS and maybe become used on some serious projects . This could give them enough funds to become a viable OS used by megacorps on infrastructures where security is critical and it may lead it to develop into a truly daily drivable OS.
I believe the next step after that would be to wake up
Hey, we can always dream!
They have already ported apps from Cosmic though.
HURD 2, the return of rust
They are planning to port Cosmic DE, and have already ported several applications from Cosmic including the file manager and text editor if I remember correctly.
They’ve shown the COSMIC terminal working on their last showcase video
What does this have to do with Rust? Or redox, or micro kernels or Linux?
Zombies are usually tied to some resource use. In microkernels, you have more control over the resources.
I don’t think a microkernel will help with zombies.
Ok, how change of kernel would fix userspace program not reading return value? And if you just want to use microkernel, then use either HURD or whatever DragonflyBSD uses.
But generally microkernels are not solution to problems most people claim they would solve, especially in post-meltdown era.
This particular issue could be solved in most cases in a monolithic kernel. That it isn’t, is by design. But it’s a terrible design decision, because it can lead to situations where (for example) a zombie process locks a mount point and prevents unmounting because the kernel insists it’s still in use by the zombie process. Which the kernel provides no mechanism for terminating.
It is provable via experiment in Linux by use of fuse filesystems. Create a program that is guaranteed to become a zombie. Run it within a filesystem mounted by an in-kernel module, like a remote nfs mount. You now have a permanently mounted NFS mount point. Now, use mount something using fuse, say a WebDAV remote point. Run the same zombie process there. Again, the mount point is unmountable. Now, kill the fuse process itself. The mount point will be unmounted and disappear.
This is exactly how microkernels work. Every module is killable, crashable, upgradable - all without forcing a reboot or affecting any processes not using the module. And in a well-designed microkernel, even processes using the module can in many cases continue functioning as if the restarted kernel module never changed.
Fuse is really close to the capabilities of microkernels, except it’s only filesystems. In a microkernel, nearly everything is like fuse. A linux kernel compiled such that everything is a loadable module, and not hard linked into the kernel, is close to a microkernel, except without the benefits of actually being a microkernel.
Microkernels are better. Popularity does not prove superiority, except in the metric of popularity.
It was(see CLONE_DETATCHED here) and is(source)
Ok, this is not really good implementation. I’m not sure that standard requires zombie processes to keep mountpoints(unless its executable is located in that fs) untill return value is read. Unless there is call to get CWD of another process. Oh, wait. Can’t ptrace issue syscall on behalf of zombie process or like that? Or use vfs of that process? If so, then it makes sense to keep mountpoint.
Except Linux does it too. If graphics module crashes, I still can SSH into system. And when I developed driver for RK3328 TRNG, it crashed a lot. Replaced it without reboot.
As I said, we live in post-meltdown world. Microkernels are MUCH slower.
I’ve heard this from several people, but you’re the lucky number by which I’d heard it enough that I bothered to gather some references to refute this.
First, this is an argument that derived from first generation microkernels, and in particular, MINIX, which - as a teaching aid OS, never tried to play the benchmark game. It’s been repeated, like dogma, through several iterations of microkernels which have, in the interim, largely erased most of those performance leads of monolithic kernels. One paper notes that, once the working code exceeds the L2 cache size, there is marginal advantage to the monolithic structure. A second paper running benchmarks on L4Linux vs Linux concluded that the microkernel penalty was only about 5%-10% slower for applications than the Linux monolithic kernel.
This is not MUCH slower, and - indeed - unless you’re doing HPC applications, is close enough to be unnoticeable.
Edit: I was originally going to omit this, as it’s propaganda from a vested interest, and includes no concrete numbers, but this blog entry from a product manager at QNX specifically mentions using microkernels in HPC problem spaces, which I thought was interesting, so I’m post-facto including it.
Indeed, first generation microkernels were so bad, that Jochen Liedtke in rage created L3 “to show how it’s done”. While it was faster than existing microkernels, it was still slow.
I’ll mark quotes from paper as doublequotes.
Wait, what? Co-located in-kernel? So, loadable module?
Right now I stopped at the end of second page of this paper. Maybe will continue later.
Will read.
Can you elaborate? I am not an OS design expert, and I thought microkernels had some advantages.
Many people think that microcernels are only way to run one program on multiple machines without modyfing them. Counterexample to such statement is Plan 9, which had such capability with monolithic kernel.
That’s not something I ever associated with microkernels to be honest. That’s just clustering.
I was more interested in having minimal kernels with a bunch of processes handling low level stuff like file systems that could be restarted if they died. The other cool thing was virtualized kernels.
Well, even monolithic Linux can restart fs driver if it dies. I think.
nah, you can have micro-kernel features on linux, but you can’t have monolithc kernel features on microkernel, there’s zero arguments in favor of a micro kernel, except being a novel project
ORLY.
Do explain how you can have micro kernel features on Linux. Explain, please, how I can kill the filesystem module and restart it when it bugs out, and how I can prevent hard kernel crashes when a bug in a kernel module causes a lock-up. I’m really interested in hearing how I can upgrade a kernel module with a patch without forcing a reboot; that’d really help on Arch, where minor, patch-level kernel updates force reboots multiple times a week (without locking me into an -lts kernel that isn’t getting security patches).
I’d love to hear how monolithic kernels have solved these.
I’ve been hoping that we can sneak more and more things into userspace on Linux. Then, one day, Linus will wake up and discover he’s accidentally made a microkernel.
I thought the point of lts kernels is they still get patches despite being old.
Other than that though you’re right on the money. I think they don’t know what the characteristics of a microkernel are. I think they mean that a microkernel can’t have all the features of a monolithic kernel, what they fail to realise is that might actually be a good thing.
Well, yeah, you’re right. My shameful admission is that I’m not using LTS because I wanted to play with bcachefs and it’s not in LTS. Maybe there’s a package for LTS now that’d let me at it, but, still. It’s a bad excuse, but there you go.
I think a lot of people also don’t realize that most of the performance issues have been worked around, and if RedoxOS is paying attention to advances in the microkernel field and is not trying to solve every problem in isolation, they could end up with close to monolithic kernel performance. Certainly close to Windows performance, and that seems good enough for Industry.
I don’t think microkernels will ever compete in the HPC field, but I highly doubt anyone complaining about the performance penalty of microkernel architecture would actual notice a difference.
Windows is a hybrid kernel, and has some interesting layers of abstraction, all of which make it slower. It’s also full of junkware these days. So beating it shouldn’t be that hard.
Yeah to be fair in HPC it’s probably easier to just setup a watchdog and reboot that node in case of issues. No need for the extra resilience.
That’s my point. If you’re l33t gaming, what matters is your GPU anyway. If HPC, sure, use whatever architecture gets you the most bang for your buck, which is probably going to be a monolithic kernel (but, maybe not - nanokernels allow processes basically direct access to hardware, with minimal abstraction, like X11 DRI, and might allow even faster solutions to be programmed). For most people, the slight improvement in performance of a monolithic kernel over a modern, optimized microkernel design will probably not be noticeable.
I keep getting people telling me monolithic kernels are way faster, dude, but most are just parroting the state of things decades ago and are ignoring many of the advancements micro kernels like L4 have made in intervening years. But I need to go find links and put together references before I counter-claim, and right now I have other things I’d rather be doing.
I wasn’t making a counter claim. I was agreeing with you. Like what?
you don’t need a micro kernel to install medules, nor to make a crash in certain module don’t bring the kernel down, you program it isolated, they don’t do that now because it’s unecessary, but android do that, and there’s work being doing in that way https://www.phoronix.com/news/Ubuntu-Rust-Scheduler-Micro
the thing is that it’s harder todo that, that’s why no one does, but not impossible, you also need to give the kernel the foundation to support that
Fun fact: Android’s next Kernel, Fuchsia, is a microkernel. So even Google acknowledges the superiority of microkernels.
bro thinking a chromecast OS gonna run in google servers 💀, micro kernels has their utility in embedded system, we know, saying that they are replacement for monolithic kernel is dumb, also companies can’t do different/hacks project anymore?