Is not that I personally looked into it. But I would say for this purpose LFS (Linux from scratch) could may help:
https://linuxfromscratch.org/