I am glad that Joseph didn't feel that way.   It would have been sad for Jesus to have grown up without his mother's husband to teach him how to build and fish.