110
6 A Planetary-Scale Threat
People using such powerful simulation tools may not be satisfied with knowing
potential future courses of the world—with the aim to be better prepared for what
might come. They may also want to use the tool as a war simulator and planning
tool.
19 Even if used in a peaceful way, they would like to select a particular future
path and make it happen—without proper transparency and democratic legitimation.
Using surveillance data and powerful artificial intelligence, one might literally try
to “write history”. Some people say, this has already happened, and “Brexit” was
socially engineered this way.
20 Later, when we talk about behavioral manipulation,
the possibility of such a scenario will become clearer.
Now, suppose that the World Simulator has identified a particularly attractive
future scenario, e.g. one with significantly reduced climate change and much higher
sustainability. Wouldn’t this intelligent tool know better what to do than us? Shouldn’t
we, therefore, ensure that the world will take exactly this path? Shouldn’t we follow
the instructions of the World Simulator, as if it was a “benevolent dictator”?
It may sound plausible, but I have questioned this for several reasons.
21 One of
them being that, in order to optimize the world, one would need to select a goal
function, but there is no science telling us what would be the right one to choose. In
fact, projecting the complexity of the world on a one-dimensional function is a gross
over-simplification, which is a serious problem. Choosing a different goal function,
however, may lead to a different “optimal” scenario, and the actions we would have
to perform might be totally different. Hence, if there is no transparency about the
goals of the World Simulator, it may easily lead us astray.
6.6 Attention Economy
In the age of the “data deluge”, we are being overloaded with information. In the
resulting “attention economy,” it is impossible to check all the information we get, and
whether it is true or false. We are also not able to explore all reasonable alternatives.
When attention is a short resource (perhaps even scarcer than money) people get in
a reactive rather than an active or proactive mode. They respond to the information
according to a stimulus-response scheme, and tend to do what is suggested to them.
22
This circumstance makes people “programmable”, but it also creates a competition
for our attention. Whoever comes up with the most interesting content, whoever is
more visible or louder will win the competition. This mechanisms favors fake news,
19 Sentient world: war games on the grandest scale, The Register (June 23, 2007) https://www.the
register.com/2007/06/23/sentient_worlds/.
20 Brexit—How the British People Were Hacked, Global Research (November 23, 2017) https://
worldfinancialreview.com/brexit-how-the-british-people-were-hacked/, Brexit—a Game of Social
Engineering with No Winners, Medium (June 4, 2019) https://medium.com/@viktortachev/bre
xit-a-game-of-social-engineering-and-no-winners-f817529506f4; see also the books by Cambridge
Analytica Insiders Christopher Wylie, and Brittany Kaiser.
21 Helbing and Pournaras [3], Helbing [4].
22 Kahneman [5].
6 A Planetary-Scale Threat
People using such powerful simulation tools may not be satisfied with knowing
potential future courses of the world—with the aim to be better prepared for what
might come. They may also want to use the tool as a war simulator and planning
tool.
19 Even if used in a peaceful way, they would like to select a particular future
path and make it happen—without proper transparency and democratic legitimation.
Using surveillance data and powerful artificial intelligence, one might literally try
to “write history”. Some people say, this has already happened, and “Brexit” was
socially engineered this way.
20 Later, when we talk about behavioral manipulation,
the possibility of such a scenario will become clearer.
Now, suppose that the World Simulator has identified a particularly attractive
future scenario, e.g. one with significantly reduced climate change and much higher
sustainability. Wouldn’t this intelligent tool know better what to do than us? Shouldn’t
we, therefore, ensure that the world will take exactly this path? Shouldn’t we follow
the instructions of the World Simulator, as if it was a “benevolent dictator”?
It may sound plausible, but I have questioned this for several reasons.
21 One of
them being that, in order to optimize the world, one would need to select a goal
function, but there is no science telling us what would be the right one to choose. In
fact, projecting the complexity of the world on a one-dimensional function is a gross
over-simplification, which is a serious problem. Choosing a different goal function,
however, may lead to a different “optimal” scenario, and the actions we would have
to perform might be totally different. Hence, if there is no transparency about the
goals of the World Simulator, it may easily lead us astray.
6.6 Attention Economy
In the age of the “data deluge”, we are being overloaded with information. In the
resulting “attention economy,” it is impossible to check all the information we get, and
whether it is true or false. We are also not able to explore all reasonable alternatives.
When attention is a short resource (perhaps even scarcer than money) people get in
a reactive rather than an active or proactive mode. They respond to the information
according to a stimulus-response scheme, and tend to do what is suggested to them.
22
This circumstance makes people “programmable”, but it also creates a competition
for our attention. Whoever comes up with the most interesting content, whoever is
more visible or louder will win the competition. This mechanisms favors fake news,
19 Sentient world: war games on the grandest scale, The Register (June 23, 2007) https://www.the
register.com/2007/06/23/sentient_worlds/.
20 Brexit—How the British People Were Hacked, Global Research (November 23, 2017) https://
worldfinancialreview.com/brexit-how-the-british-people-were-hacked/, Brexit—a Game of Social
Engineering with No Winners, Medium (June 4, 2019) https://medium.com/@viktortachev/bre
xit-a-game-of-social-engineering-and-no-winners-f817529506f4; see also the books by Cambridge
Analytica Insiders Christopher Wylie, and Brittany Kaiser.
21 Helbing and Pournaras [3], Helbing [4].
22 Kahneman [5].