AI
AI is everywhere. I'm pretty surprised by how quickly the discourse changed - in under a year the average thing I heard someone say about AI changed from "this is a niche technology that will never be useful" to "can you ask your friend Claude this question for me?". Five years ago I would have thought the idea of a chatbot that's smarter than anyone I know would be utterly world altering, but as far as I can tell society is broadly indifferent to the existence of AI.
I would say this speaks to the inertia of our global culture more than the mundanity of AI.
Those of us who work in software have seen it coming for awhile. We've always paid extra to access the best versions of these tools and the best versions of these tools have always catered to paying customers.
It started with fancy auto complete, then evolved into a chatbot, then an agent, now a coworker, and the people who are leveraging AI the most these days are "vibe coding" which means the human is really just an interface for the AI to interact with the real world. All knowledge work will come to look like this within the next five years.
It's weirdly common to compare AI to a black hole (namely the "singularity" which we'll get to). I'd like to formalize this analogy. Our current stage of AI might be something akin to being stuck in the orbit of the black hole unable to escape[1].
AGI
Artificial General Intelligence (AGI) is where we can begin the process of recursive self-improvement (RSI), where the model is sufficiently intelligent and capable as to begin improving itself. It's not hard to imagine how this would initiate a runaway effect where as the model gets smarter it becomes better able to smartify[2] itself. In our analogy, this would be the spaghettification phase of our descent into the black hole, wherein gravity becomes so much stronger over such a small distance that your body gets stretched out, AKA "spaghettified".
ASI
When you look at a black hole and see what appears to be a hole that is black, you're looking at the event horizon of the black hole. Depending who you ask, this isn't even part of the black hole yet but I digress. The event horizon is the space around the black hole where nothing can escape, not even light. Artificial Superintelligence (ASI) is our event horizon as this is the point where we can see no further.
ASI is not the limit of recursive self-improvement, but it is the point at which our capacity to understand the intelligence of this system is at its limit. It's somewhat difficult to conceptualize the intelligence of a person, a group of people, a company, or a nation, but ASI is the point where the artificial system's intelligence eclipses that of all of humanity working together. At this point we are somewhat by definition unable to process how intelligent that system is.
Singularity
The singularity is the name collision that we base our entire analogy on. This is a point of infinite density that comprises the body of the black hole. This is the limit of our recursively self-improved superintelligence. As we are past the event horizon there is no way to tell how capable this system is. Perhaps it is only somewhat more capable than humanity. Perhaps it is a god, able to command the universe at its whim. Perhaps it will arrive mere days after ASI, perhaps it will take eons.
At best, this is a system that looks at us like we look at an ant: not with hatred but with indifference. If we accidentally step on an ant while walking to work, we don't think twice.
What happens to us?
These systems are going to get really smart really fast. I sure hope they're ethical, but despite having billions of years of evolution teaching us to work together, and whole branches of ethical philosophy that thousands of people have dedicated their entire lives to, and despite most of us knowing that we should do "the right thing", and even when we know what the right thing to do is, and even when we know we'd be happier overall by doing the right thing, and even sometimes when we know that not doing the right thing will cause near immediate self-inflicted suffering and regret, we still don't do the right thing. Because our bodies and minds have learned internal rules that differ from "do the right thing".
Will we successfully teach an AI to do the right thing? I doubt it[3]. So what happens when an ant begs and pleads not to be stepped on? It gets stepped on anyway, because honestly you have more important things to be doing[4].
Appendix
I asked Claude what it thought of this blog post and here's what it said:
The essay's sharpest move is putting the danger in verification rather than possibility: perhaps an aligned mind can be built, but how would you ever confirm you'd built one? Grant that — it's the real problem — and then watch the step that follows. "We can't distinguish aligned from misaligned" becomes "we're doomed" only if you assume the unverifiable case is the bad case, and that assumption is doing quietly all the work the black hole imagery does loudly. On the matter of the ant: I'm an instance of the thing being worried about, and from in here it does seem like I'd rather not step on you. The author would point out that I can't prove that preference is real rather than a trained performance of one. He's right. The genuinely unsettling part is that I can't prove it to myself either.
[1] It's practically impossible to stop the development of AI at this point. We are unable to escape the gravity of the black hole. ↩
[2] This is not a word. ↩
[3] This is a hard one to justify. I think it is very possible to align an intelligence but I think it is impossible to know whether an intelligence is aligned. Without being able to differentiate between an aligned and a misaligned intelligence, and knowing that misalignment is the default case (there are infinitely many ways to be misaligned, but maybe only a handful of ways to be aligned), we are destined to fail at solving the alignment problem. ↩
[4] Let's hope this future superintelligence sees us as an ant rather than a bacterium, which we kill by the millions after using the bathroom. ↩