It's a real danger of movements and systems that they want to self-perpetuate. Any animal's first instinct is to keep itself alive. Any religion's first rule is "copy the book exactly". More generally, the primary goal of any system is to protect the system, because without that the system can't do anything else. That's well and good, but it often leads to systems whose non-survival goals aren't relevant anymore, or that have no other goals than self-perpetuation.
In biology there is a process called apoptosis, where a cell deliberately dies (as opposed to being killed by injury or another cell). Far from being a bad thing, apoptosis is vital in development and disease prevention. Despite the obvious biological imperative, it's important that cells only live for as long as it makes sense to do so. A lack of normal apoptosis is implicated in many autoimmune diseases, autism, and cancer.
Similarly, I think systems built for a purpose also need to be built with a mechanism to terminate themselves when that purpose is achieved or irrelevant. If you start the Cure AIDS Foundation, its articles should include a "good work everyone, we cured AIDS, let's shut down" provision – and what a happy day that would be! On the other hand, suggesting that we should terminate people, and even sometimes animals, who have achieved their purpose is a pretty horrifying proposition. In that case, we value the system, the life itself, more highly than we do its purpose.
In the case of people, we have a moral imperative, presumably one that springs from our own parallel desire for self-perpetuation. However, it seems dangerous to me that we so easily extend the right to self-perpetuation to other systems. To put it another way, once we have a system whose primary goal is self-perpetuation, how is it different from an animal? Built from different materials, perhaps, but still out there fighting for survival like the rest of us.
When we give primary self-perpetuation to a system we give it personhood, and we shouldn't be surprised that it keeps trying to exist even when it's no longer fulfilling any other purpose.
I remember watching some sci-fi show years ago with a sentient robot character who always said things like "entering sadness mode" or "activating my joy circuits". Of course, that's totally ridiculous. First of all, what the hell is a joy circuit? Secondly, the idea that a sentient robot would turn on or off their emotions at will is just a cheap grab at otherness to make sure we know that sentient robots can't be like us. The real world obviously wouldn't work like that.
But schlocky sci-fi may have had the last laugh, because not only is a neural circuit a real thing, but it seems like turning on and off certain patterns of thought may also be real (though, thankfully, the two aren't the same). I had a really interesting conversation with a friend in games research a while back who was saying that an important aspect of morality in games is whether the games make you feel like you're making moral decisions. You could be an incredibly moral person but still press a button that murders millions of virtual people without even considering morality, because that's not a moral situation.
That's not a moral situation! What a strange thought. Surely all situations are moral situations if morality is to have any meaning, but there's a certain uncomfortable truth to the idea that we would just forget to think about morality. It's C.S. Lewis's evil of quiet men in well-lit rooms. I doubt most people acting immorally would identify their actions as such, and not even because of some twisted appeal to moral relativism. It's just that the situation never seemed like a moral one in the first place.
And I don't think it's just morality. For example, everyone has the capacity to introspect, to analyse their situation and their decisions and change them for the better. But often we just... don't. Not because we decided that introspection was a bad idea, just because it never came up. Later on it seems so obvious, like, how did I not realise I was doing the wrong thing? But of course it's much easier to engage your introspection circuits in hindsight, much harder to do it at the time.
Every situation is an opportunity for introspection, morality, or emotion; the situation doesn't care what category you put it in, it's just a matter of which lenses you're willing to look at it through. I think we'd be a lot better off if we could distinguish ourselves from our shiny metallic cousins and activate those circuits more often.
I heard someone remark recently that it's great how easy it is for amateurs to create stuff now. I've said the same thing myself; you can go from an idea to something that actually exists with very little effort. And not just that, but it can be good. It's even within the reach of a motivated group of amateurs to self-fund a high production-value superhero short film, a genre that is challenging even for large studios. The story is the same in novels, music, video games: there's never been a better time to be an amateur.
But there's a dark side to this, in a way. With so many motivated amateurs making such high-quality work, you lose the diamond-in-the-rough crappiness that once marked amateur work. There's no need to have a badly-edited grainy short film with some stand-out moments when you can have a well-edited and polished short film with the same content. The high standard of amateur work means that, ironically, as the barrier to entry for production has fallen, the barrier to entry for skill has risen. You could make a professional-quality short film entirely on your own, if you only knew how.
I'm still on the side of technology here, but I think there's a new skill required in amateur content creation: humility. Whereas previously you could max out the capabilities of the crappy tools you have and get an impressive-for-what-you're-working-with result, that shield is gone. The rising tide of technology means you could always do better, if you were better.
"The only limit is yourself" is the kind of aphorism that's meant to sound motivating, but it's also a heavy burden. You have to learn to make things whose quality is limited by yourself, not by your tools, and be happy with that.
