I haven't digested the whole pedophilia discussion in this thread as well as I should, but some thoughts after reading briefly:
1. Everything's OK. No thought or impulse is inherently "wrong". However, if acting in some way may cause hurt to another, you may choose another course of action, or inaction. Of course, this is a philosophical viewpoint, not a practical viewpoint, so it might be easy to disagree with me if you're talking from a different place, such as trying to find rules that maintain social order, etc.
But no thought pattern ever changes or goes away just because of someone calling it "wrong". If it's not going to change, then you might as well address it with love, caring, and full attention, rather than trying to turn it off or decide that it's wrong. Can we all agree this is one of the main problems of Christianity?
I get subconsciously attracted to my little students every now and then during class (ok not that little, but still little). A teacher I used to work with called it "pattern recognition", your eyes just see legs with nylons poking out from a short skirt, and woop! You look at them in "that way" without any thought at all. It just happens before you're even aware of it. I have pervy moments like this every now and then. Once I was at my brother's high school basketball game, and there was this little probably 14 year old sitting in front of me with a red thong showing at the back of her pants. I knew it was wrong, but I couldn't stop glancing at it, and it was definitely a sexual urge that kept making me glance. It's an age-old thing anyway, us old men getting turned on by young girls. Best just to shrug, remember you're old, and move on. It's not a big deal.
2. Sexual maturity/readiness - when does it happen? Nature has not quite caught up with our current human values (which I don't think are bad or wrong, and I think we're evolving to where our nature will catch up). Hormonally, who here was not hornier than hell by 15? OK, some of us probably weren't, but some of us were. I believe that some tribal cultures have mating ceremonies to pair up girls and boys who are around 14 - 18 or so. The babies they produce are taken care of by the elders. I think that's about where our nature is, in terms of hormone production, mental responses to sexuality, production of pheromones, etc. Evolutionally speaking, we're not even physically used to living past the age of 35 yet, though we're getting there.
3. Japanese people in general are under huge amounts of stress (from what I hear). I think their perverted fantasies (and oh my God do I think they're perverted) to some extent are a result of that. A successful man should be working around 14-16 hours a day, from what I hear, and jobs are not even that easy to come by (this is in Tokyo), so you always have to be trying to work harder than the next guy. Also, the cost of living is through the roof. Insanity reigns supreme. What I don't get is what all the fantasies seem to fit into like a group of like at most 10 patterns, and people apparently still get off on that. But whatever, that's kind of an Asian thing, doing what's been done a thousand times before, and having it still be cool. Probably somewhat a product of having a much older culture than we have in the states.
4. I think what's probably being fetishized is probably innocence? When you consciously and thoughtfully get turned on by taking someone's innocence away, I think that might be a little sick. It probably means you don't spend enough time with children, imo, you're not a very well-intentioned person, or maybe you have some childhood issues you need to work out.
5. The flipside of that: losing your sexual innocence probably ought to be a beautiful thing, and well-supported by someone with experience, and maybe that's why we have natural attractions towards young, or specifically innocent-seeming people as we get older. Getting back to tribal traditions, I know there were some Native American tribes where the young men would spend time with a mature woman learning about how to have sex properly and how to take care of a woman, before they were loosed on the young girls. I don't know how - it's so hard, almost impossible in our current repressed society, but we really need to get back to that. I had a good ex-hippie friend in California (not all hippies are bad by the way, or even really hippies. Some are just thoughtful people seeking a better way of life) who lost her virginity at 16 to an I forget how old guy. late 20's I think. And she said it was wonderful and she really treasures that memory. Apparently he was a friend, and a good guy, and he took good care of her. They didn't even have a relationship (another perverted creation of our sick society - monogamous relationships
), they just had some sex.
But anyway, I have been pleased, as I get older, to note that I am much more attracted to older women now, and not attracted at all to any girls less than 20. They just have a "not ready" look to me. I was never a milf guy in my 20's, and always thought teenagers were hot unless they looked really young, so I'm glad that my attractions seem to have matured a bit. A lot of it though I think is just from having good experiences with my wife, and therefore desiring a more mature woman (young girls talk so much!)
I'm really sorry for rambling on and on...
edit: but I agree with you that it's weird and I don't know why anime has to be like that. Sort of gets in the way of its other positive qualities, like you're saying.
edit2: Another thought: there's also the infinite bubbly energy of a young girl for an "old man" to be attracted to. Not sure about the young boy attraction, but I'm sure someone understands it.