I felt this way after I would masturbate to gay porn before I had come to terms with my sexuality. I would be thinking to myself something along the lines of, "Why am I turned on by this and not what I 'should' be attracted to?" Thinking I should be attracted to women instead of men, because I was still trying to live the life that my dad had laid for me.
You'll feel guilty about something if you think it's wrong, or if you could have done it differently and better, or if you're stuck worrying about the past and not concerned about the present or future. I can't tell you what you think you did wrong; I think you need to figure those specifics out on your own.
However, if I had any right to "diagnose" the root of the "problem," I would think that you're not comfortable with your sexuality. And that's perfectly okay. A lot of us have been there - I've been there. And we all live our own life at our own pace.
I don't think any sexuality is a crime. But who can define who/what you like beside yourself? You say that you've "seriously enjoyed" your sexual encounters with other men. I don't think there's anything wrong with enjoying yourself - why feel guilty about it? I think there's no need.
You also don't need to be concerned about labels. Just try to learn yourself, and I think as long as you don't hold yourself back, the rest will fall into place.
But as long as you continue to feel guilty about something you like, you will hold yourself back.
That's all just my opinion, at least. I am not very well-versed in life, so perhaps others may have more worthwhile opinions and suggestions. But I think I can relate to what you described, and the solution for ME was to become more comfortable and confident in my sexuality. Whether or not it could be a solution for you, as well, is for YOU to determine.
P.S. And, for what it's worth, welcome to LPSG.