I am having an affair while engaged to another man.?

Sounds to me like you have already made your decision. Don't marry this guy if u don't feel like you can live the rest of your life happily with him. Seems like he has already got on your nerves. Be honest with him and tell him your not ready for marriage just yet. Then wait a bit and break up with him and then date the guy you truly love.