You know what would look really weird? If you were to walk in on the invisible man having sex with his girlfriend who, just for the sake of the joke, isn't invisible.
Picture it. Go ahead, its funny.