Sam Gentle.com

Impaired awareness

I've always found it difficult to think about anything superintelligence-related, or even moderate-increase-in-intelligence-related. It's not that I can't reason about those things – that's fairly easy – but it's really difficult to build an intuition about. How would it feel to be a genetically engineered supergenius, or some kind of supercomputer-powered AI, or even just someone a lot smarter than I am? I have no idea.

One of the most interesting things is to go the other direction, and think about times you were temporarily impaired in some way. For example, when you're very tired you might recognise that you're tired because of the physiological symptoms, but what psychological feedback do you have? The brain you use to reason about an impairment might not be able to because of the impairment itself. You see this most acutely in carbon monoxide poisoning, like in this story on Reddit where the author became totally useless but didn't realise anything was wrong.

I wouldn't be surprised to learn that there are much milder forms of this impaired awareness that happen all the time. Maybe you haven't been sleeping as well lately, or you're eating differently, or there are just natural changes in your cognitive function from time to time. How would you even know? Maybe you're half as clever as you were a year ago, and you just haven't realised because it doesn't feel any different. You just don't notice all the things you don't think of. Conversely, you might be more clever and not notice it if there aren't any obvious clues.

I suspect being the sudden recipient of superintelligence might feel a lot like sobering up after a big night and thinking "wow, I didn't realise I was drunk at the time, but I had no idea what I was doing".