I am glad that Joseph didn't feel that way.   It would have been sad for Jesus to have grown up without his mother's husband to teach him how to build and fish.    

 I thought Mary was a virgin. How did she have a husband?