Another one in my series of revisiting old lessons in the context of a world where we have AI. One thing I'm thinking about today is survivorship bias (or success bias, if you like; they get used to describe the same thing).

Survivorship bias is the error of drawing conclusions from the subjects that made it through some selection process, while never studying the ones that didn't, since they never show up in the data. The classic example is the World War II bomber planes. The military was studying the planes returning from missions, which were covered in bullet holes, and wanted to reinforce the places where the planes were getting shot. A statistician, Abraham Wald, pointed out that this was exactly backwards: you should reinforce the places the returning planes were not hit, because a plane hit there didn't come home to be counted.

So why do I bring up survivorship bias? Well, it seems to me that the success stories of people using AI with great success may be carrying too much weight, and a lot of those are toy examples anyway.

Toy Examples

What I mean by "toy examples" is when we see posts and articles about something that was created with AI that looks amazing and required little effort, but didn't really have a real value initiative behind it in the first place.

One example of that is the 100 webpages created by Opus 5.5. That's a good demo of what it's capable of: it did in fact create all of those, and they were unique and different. Repeating that may be good for getting ideas and inspiration. But the thing people take from it (that it took such a small prompt and it did all of this work) is not really what's impressive about it. What's impressive is just that it did something and completed the task.

Whenever you are already inspired and you have a particular vision in mind--you have something that you want Claude (or any AI) to accomplish--that's a very different story. Getting something created is now incredibly easy, but getting something of value or getting the thing that you wanted... that's the challenge.

Amazing Valuable Results

I'm not saying that the amazing results are only with toy examples. I'm only pointing out that the amazing value-creating examples are more rare and still require significant effort and/or luck.

A personal anecdote: a few months ago I was working on defining the brand personality and positioning for one of my companies. After a lot of collaboration and using AI to refine the idea, at some point it started to repeat things back in a way that I felt like it really got me. I had a bit of an emotional reaction when things started clicking into place. Through that exercise, the agent connected some dots between the things I had said, and they were connections that I didn't expect, but really solidified the entire strategy. There's no better sense of being "done" than having a moment like that!

Developing the strategy without using AI would have taken me so much longer, and I may not have realized the vision as completely as I had with it. But the fact is, I haven't consistently been able to intentionally repeat that process; there's a point where you just get lucky.

For every interaction that landed like that, I could take you through maybe 50 others where I am just fighting with it to try to understand where the gap is between what I want and what it's producing. Often the issue is that the AI is trying to achieve the literal goal I asked for, without evaluating potential gaps or conflicts in the context it's been given.

They say that the definition of "synergy" is that the interaction of the parts produce an effect that is better (greater) than the sum of the parts individually. I think most of the time, the problem with AI is that the sum of the parts are actually less than the sum of the parts because the understanding of the larger system and context is just absent. Sometimes you get lucky and everything snaps into place beautifully, but quite often it's just a tangled mess.

The point is, we need to be careful that we don't set the success stories as the standard.