Your argument works great on micro plastics: if it doesn't decompose in hundreds of years, basically being inert, why would it harm to have some mg of it in my body?

For fiber though I think there is a case to be made that it increases the volume of what reaches your lower intestines and I think there is scientific evidence it lowers colesterol. Clearly it's not its chemical properties but its mechanical properties.