I did this with Arch, and ever since, I haven't been able to let go of Arch (many years) - It's, essentially, my Arch. 
What you get is a super lean, lightweight but flexible systems with only the stuff you need to get it to work, and it's amazing. 
Sure, it's grown a bit as time goes, but at least it's easier to keep track of what's there and once in a while I'll just go in and take out anything that doesn't belong anymore. 

Running Gnome though, as I try to not multi-task so much.