Complex guy situation: should i stay or go?

It would be uncomfortable for me. I would probably try talking to him and if he can't understand it I would leave.