The biggest unknown is what happens if/when an AGI is given (or obtains) full access to its own code, including any intended safeguards put in by humans.
Vestigial organs and processes are a given in evolutionary biology, but what if a being had all the power needed to prune these things themselves?
Even if an AGI started down the path of self-improvement with all the best intentions of helping humanity, there may still be an iteration that acts a little worse by whatever grouping of factors. Then, maybe that iteration gets modified further, and could result in something quite dangerous to humans.
Certainly I don’t mean to argue this is inevitable, just that even all the best intentions of the AGI itself could still result in big problems. As with evolutionary biology (sans “intentions”), and as with human behavior.
The biggest unknown is what happens if/when an AGI is given (or obtains) full access to its own code, including any intended safeguards put in by humans.
In a neural network, the code itself isn’t sufficient to understand how it behaves. You need the parameter weights, which were developed through lots and lots of computation, presumably through resource-intensive processes with lots of training data and feedback and selection mechanisms.
So if AGI can be achieved through a particular hard coded architecture and the weights of trillions of parameters, what can an AGI do to perform brain surgery on itself? Like Borges’ Library of Babel thought experiment, the overwhelming majority of possible states will be broken, so any edits will have to be very careful and guided by extrinsic rules. Plus, the ability to edit the weights may form problems akin to biological cancer, dementia, hallucinations, other brain disorders.
Just as the human brain doesn’t understand everything about the human brain, it would be incorrect to assume that an AGI that can achieve both general intelligence and consciousness must necessarily have the ability to understand its own internal function, or modify itself in a way that improves things for itself. More likely, it is either programmed to (or learns through reinforcement learning and evolutionary mechanisms) that self modification is dangerous, and develops a very conservative approach to self preservation.
How dare you critique me, a rank amateur with mediocre knowledge about a few subjects? Especially with this reasoned argument?!
But yeah, my main takeaway is “the code must be complex in order to meaningfully analyze anything, and the more complex the thing it can analyze, the even-more-complex it must itself be.”
Yeah, with sufficient complexity it’s more along the lines of “I created a procedure that makes this complex thing” rather than “I built this complex thing up piece by piece.”
So if the act of creation is considerably less complex than analyzing and understanding a part of that creation, it’s far more likely that the complexity gap ends up preventing any self-aware AGI from being able to effectively reinvent oneself, even if it does have full write access to the components.
The biggest unknown is what happens if/when an AGI is given (or obtains) full access to its own code, including any intended safeguards put in by humans.
Vestigial organs and processes are a given in evolutionary biology, but what if a being had all the power needed to prune these things themselves?
Even if an AGI started down the path of self-improvement with all the best intentions of helping humanity, there may still be an iteration that acts a little worse by whatever grouping of factors. Then, maybe that iteration gets modified further, and could result in something quite dangerous to humans.
Certainly I don’t mean to argue this is inevitable, just that even all the best intentions of the AGI itself could still result in big problems. As with evolutionary biology (sans “intentions”), and as with human behavior.
In a neural network, the code itself isn’t sufficient to understand how it behaves. You need the parameter weights, which were developed through lots and lots of computation, presumably through resource-intensive processes with lots of training data and feedback and selection mechanisms.
So if AGI can be achieved through a particular hard coded architecture and the weights of trillions of parameters, what can an AGI do to perform brain surgery on itself? Like Borges’ Library of Babel thought experiment, the overwhelming majority of possible states will be broken, so any edits will have to be very careful and guided by extrinsic rules. Plus, the ability to edit the weights may form problems akin to biological cancer, dementia, hallucinations, other brain disorders.
Just as the human brain doesn’t understand everything about the human brain, it would be incorrect to assume that an AGI that can achieve both general intelligence and consciousness must necessarily have the ability to understand its own internal function, or modify itself in a way that improves things for itself. More likely, it is either programmed to (or learns through reinforcement learning and evolutionary mechanisms) that self modification is dangerous, and develops a very conservative approach to self preservation.
How dare you critique me, a rank amateur with mediocre knowledge about a few subjects? Especially with this reasoned argument?!
But yeah, my main takeaway is “the code must be complex in order to meaningfully analyze anything, and the more complex the thing it can analyze, the even-more-complex it must itself be.”
Good stuff!
Yeah, with sufficient complexity it’s more along the lines of “I created a procedure that makes this complex thing” rather than “I built this complex thing up piece by piece.”
So if the act of creation is considerably less complex than analyzing and understanding a part of that creation, it’s far more likely that the complexity gap ends up preventing any self-aware AGI from being able to effectively reinvent oneself, even if it does have full write access to the components.