The doomers are worried about evil AGI.



But moral knowledge, is just another kind of knowledge, like aesthetic knowledge or knowledge about math.

Our moral knowledge can grow, and has been growing over time.

For example most of the world feels slavery is wrong now. We gained that knowledge just like we gained knowledge about prime numbers. How? Popper say it is through "conjectures and refutations". It's a fancy way of saying we make intelligent guesses and criticize them.

So somehow, the doomers think that AGI is just going to be incredibly good at creating knowledge about everything EXCEPT morality.

Here is a question: what if the AIs create more moral knowledge than we do? What if they become more moral than we are?

Can you believe that AIs make breakthroughs in Chess and Go and Physics and Maths and Biology, but somehow can't make breakthroughs in Morality?

Is it moral to stop machines that can help us learn about what is moral?