AM I MAKING THIS INTO A BIGGER DEAL THAN IT IS?

Just go along with it, but if you really have a problem with it then speak your mind. You gotta let him know how you feel or otherwise he will think its ok with you. If you don't say anything then how will he know?