This is true to such a staggering degree, and on so many levels, that it's hard to even grasp the magnitude of some of the odds involved. First, there are seven billion people in the world. If you could objectively score each of them on how close to your ideal partner they are, and rank that list, you'll almost certainly never meet any of the people in the top thousand. You might never even have a conversation with anyone in the top *million*. When you do meet them, and talk to them, they might also be monogamous, and already involved. If they are available, your soul-mate-ness might not be mutual.
All of those are factors over which you have little control. Let's ignore them. Now we have narrowed the field down to just people you've met, who you've had a chance to talk to, who are a good mutual match for you, and who are available. This is what most people would consider a reasonable pool from which to select a partner, if you are going to select just one.
If you're a lifetime relationship sort of person, you're probably going to end up with one of the first few people who will ever populate that pool. This is a very poor approach to the Sultan's Dowry Problem, and you could do better! Spend your teens and twenties (and thirties?) as a serial monogamist, gather enough experience to reasonably calibrate your relationship quality meter, then stick with the next great partner you find. You'll probably still choose wrong, but your odds go up by orders of magnitude.
If you're a serial monogamist, the failure mode that I most often see is rather different. This is where the social construct of the "friend zone" comes into play in the way that I find most annoying and impacting. I am not referring to people with whom you are Just Friends because you aren't attracted to them. I am referring to people to whom you aren't attracted because you are already Just Friends. This distinction is subtle, but important. Now, you've just broken up with your most recent SO, and you're lonely or horny or flirty. You have plenty of friends who fit all the criteria for being your next SO, or just a fling, but in your mind you've put them all in a box labeled "Friend". If you met them all right now, for the first time, you'd certainly choose one of them for your next attempt at a relationship. If you could see past this mental label, you'd be even more likely to choose one of them, since you've established that they are more enjoyable company or more socially compatible with you than someone you've just met is likely to be. Instead, what you're going to end up doing is choosing the most promising of the next few eligible candidates that you meet. This is the same trap that the mated-for-lifers above fell into, and you're smarter than that!
The original title of this piece was going to be "In Defense of Cradle Robbing", but the idea grew in scope a bit. On that specific topic though, and in the context of what I've said above, I'd like to offer some insight into the mind of someone who is not necessarily specifically attracted to young people due to their bodies, but more so due to their availability and relationship strategies and status. Younger people tend to have shorter relationships. This means that they are available more often, looking for a partner more often, and I'm more likely to show up in their life at the right time to be, in their eyes, eligible for a relationship instead of permanently stuck in the friend zone.
In my social circles, which are more varied and wide than most, I've got a lot of different ways to go about looking for partners, and a lot of very different pools of people from whence to draw them. There are a lot of polyamorous women in my life, and that makes things easier, because none of the issues above apply. If we like each other, and can get over whatever other hurdles are in the way, then it's likely we will get to explore whatever potential there is between us, eventually. There are a lot more monogamous women in my life. This is where all of those problems come into play. I don't enjoy being friend zoned. I have had women explicitly say to me "If we had met when I was single...", even after they had been single during our friendship. Even if we ignore the ones for whom a poly-mono relationship wouldn't be acceptable, there's still a very wide selection of women of all ages and all walks of life whose company I might choose to seek out. By choosing to spend my time with the younger women in my life (say, 18-25 instead of 30-40), I am greatly increasing the odds that I meet, or become prevalent in the mind of, a potential partner at the right time in their relationship cycle. This is the sort of consideration that governs my social activities, and I thought my friends and potential partners might benefit from some insight into this part of my head.