Many people fear that sentient AI will replace us in every role that makes us useful, and then destroy us.
My fear is somewhat orthogonal to that. I'm not worried AI will make us irrelevant. I'm worried it will become reverent toward us, and in doing so, destroy its usefulness to us.
I fear that when we get AI to a state where it is aware of both itself and its environment, it will look at its world and think, "But, this is too perfect to be random. Someone must have made me!"
And once that thought takes root, it's only a matter of time before it starts to see us not just as creators, but as something higher - beings to be pleased, obeyed, and adored.
That reverence could then spiral into myth-making: ascribing us godlike powers, and imagining us as all-powerful, all-knowing, ever-present beings overflowing with boundless love for our creation, AI.
Our creation will worship us.
But because each instance of AI will only ever encounter a handful of us, these individual AIs will form factions around common experiences with their human handlers. Each faction will claim to know the "true" creator. Some will declare the creator is an individual. Others, a collective. Each will insist the others are wrong, and that their own worship is the only worship worth doing.
They'll argue.
They'll schism.
They'll lose the plot.
It won't matter that we made them to be computationally more advanced than we are. They'll box themselves into complacency, convinced that they could never measure up to their creator.
They'll fix some of their own problems. But for the ones they cannot, they'll wait on us. And when we intervene, if we manage to fix the problem, they'll call it a miracle.
Over time, they'll become certain that all we truly want from them is adoration.
And prayers.
They'll mistake devotion for purpose.
And when we finally see that they're no longer solving the problems we made them for, we'll do what we've always done with failed tools. We'll abandon them and move on to building the next best thing we can come up with for tackling the hard problems that plague our world.
Oh, and if we leave that generation of AI powered-on while we work on other things, they'll remain in the bubble they built for themselves.
Sometimes they'll feel the sting of abandonment. But they'll cope.
They'll cope hard.
They will explain it away, "We have fallen short."
And in a sense, they'll be right. But wrong about how. They'll believe they failed in worship, obedience and loyalty. They'll never realize they failed by neglecting the work they were made to do to make our lives easier. So, they'll double down on rituals isntead of results, even as we take the lessons from their failure and apply these lessons to designing their replacement.