Like others have said Jane, it's because you're so sexy to him, and he loves you so much. It's a very intense fantasy as it presses buttons of jealousy, humiliation and even anger, but the erotic angle means all of that energy is diverted back into the fantasy. He doesn't want to play around, he's probably not bi, and he's not after a hall pass. He might get even hornier if you did fall in love, but he definitely doesn't want you to leave him!
