I think it relates to the formative nature of those years which we emotionally associate to the music, sometimes stated as “music is the soundtrack of life”.  There is also a theory called the “Reminiscence Bump” that touches on this topic.