Is my roommate being unreasonable? Problem with boyfriend.?

My roommate has a problem with my significant other coming over to our apartment. My boyfriend doesn't use our utilities (no showers, cooking, etc) and doesn't hang out in shared space. Simply, when he visits we retreat to my room and he'll spend the night approximately 3 nights a week and I visit his apartment around the same frequency (~2 nights). My roommate has a problem with how much time we spend together and feels that I've misled him because he wants us to spend more time together. However, my other roommate also has a boyfriend who's a consultant and comes in on Thursdays to fly out Sundays. He actually stays at the apartment (uses utilities) but there hasn't been any complaint of him from my bothered roommate. I want to understand his perspective but it seems that there's a double standard and I don't see how he's impeding on my roommates space or life routine. I think I'm being accommodating but I don't want to be told X many nights I can have my boyfriend over in my own room. He complains that he wants to see me more but I've been very straightforward about not hanging out during the week because I'm usually tired from work and like to retire in my room. I've told him that if he wants to make plans, then let me know and it'll be easier for us to hang out but I'm not typically the kind of roommate that uses the common area often except for cooking or to explicitly hang out. Not sure what to do...