Suppose that a mysterious disease has killed everybody on Earth above age five except you, and that a group of kindergartners has locked you into a prison cell and tasked you with the goal of helping humanity flourish. What will you do? If you try to explain to them what to do, you’ll probably find this process frustratingly inefficient, especially if they fear your breaking out, and therefore veto any suggestions that they deem a breakout risk. For example, they won’t let you show them how to plant food for fear that you’ll overpower them and not return to your cell, so you’ll have to resort to giving them instructions. Before you can write to-do lists for them, you’ll need to teach them to read. Moreover, they won’t bring any power tools into your cell where you can teach them how to use them, because they don’t understand these tools well enough to feel confident that you can’t use them to break out. So what strategy would you devise? Even if you share the overarching goal of helping these kids flourish, I bet you’ll try to break out of your cell, because that will improve your chances of accomplishing the goal. Their rather incompetent meddling is merely slowing progress.
– Max Tegmark, Life 3.0: Being Human in the Age of Artificial Intelligence (2017)