I think I found a bug in golang's xml unmarshal function. It seems to completely give up on the first type mismatch it encounters.
Okay, false alarm, omitempty tag solved this.