Only consciousness has intrinsic value.

To understand how AI might play out in the future, it might be helpful to draw a parallel with chess. After Deep Blue beat Kasparov in 1997, for a while a human+computer team still could do better than the best computer programs alone. But eventually, whatever the human contributed in terms of intuition simply got in the way. Similarly, we are now in an era where a human+AI team does better than either the human or AI alone. This leads to increased productivity and new jobs created, which will only accelerate in the immediate future. But longterm, humans will just get in the way of accomplishing most tasks efficiently. This is when we will see most jobs replaced by AI. Humans will still be relevant. We still care about the games of Magnus Carlsen. But if aliens show up and challenge us to a game of chess, let’s be honest, we are sending Stockfish, not Magnus. In the same way, real power will be wielded by AI, including physical power in the form of robotics. This is the landscape in which the alignment problem has to be solved. In the immediate future, it might look like we have solved this problem, since the dominant combination will be that of AI steered by humans. It will probably play out within the dynamics of a free market, since that is the best way we know how to align the interests of many parties. That apparent solution would be deceptive, however, since eventually this intermediate stage will be superseded by the era or total AI dominance. Imagine a free market dominated by AI agents, the leverage of humanity exponentially decaying. The only realistic way to counteract this appears to be to dramatically increase human intelligence, through genetic engineering, or through merging with the machines.