I like my doctor but...

OK, so i told you about the doctor wanting me to eat gluten free and sugar free. I also went the extra step and decided to give up meat except for fish. It is going well, I have lost weight and feel a lot better. So I went back to my doctor recently for an enlarged lymph node and some neck pain. It is nothing serious but while I was there she said that I needed to give up dairy.  I think she may be crazy or at least expecting way too much of me. She told me that some people have enlarged lymph nodes because of diary and dairy can cause a lot of skin issues. I switched to lactose free milk about a year ago because I found myself to be lactose intolerant but I love milk. I know I love bread too and I gave that up, but seriously I think it will be impossible to eat if I have to give up milk products as well. Additionally, there is the issue of no ice cream, sour cream, cheese or lots of other things. I know there are soy replacements for all of these items but I really just don't want to do this. I am considering switching my milk to soy since that is the dairy product I consume most often but I only really like one soy brand and have not found it here yet. I want to be able to eat in a restaurant and at this point I am already limited to fish, vegetables, potatoes and rice. If I give up dairy even the gluten free, meat free foods I can find on a menu will be off limits to me. I know I can do the same thing with dairy that I do with the rest, allow myself the cheat day and eliminate it at home. I also know that in the animal kingdom we are the only ones who consume milk as adults but still... Seriously, I already eat basically beans, rice, fruit, veggies, chex cereal, occasional rice crispy treats, fruit roll ups, yogurt and cheese. You do the math if I eliminate the cheese and milk on the cereal. So to dairy or not is the question.