Is not that I personally looked into it. But I would say for this purpose LFS (Linux from scratch) could may help:
https://linuxfromscratch.org/ 

 Sounds like what I’m looking for, thanks.