I have had allergies to mould, dust mite and grass for years. For a long time I thought it was wheat and dairy, but those things seem to just exasperate rather than be the cause. If you can afford it I recommend desensitisation treatment. It had reduced my symptoms significantly. 

Interesting to hear about the fasting (starving in your case) this is something I need to look into.