Are you in the left lane with cars piling up behind you? If yes, you're lollygagging. If it's an individual aggressive driver and you're actively passing while in the left lane then yes, that driver is being the asshole. Just use the left lane for passing the right lane for riding. It's really a pretty simple social contract. And intentionally hanging out or slowing down on the left lane just to flex your rights and show somebody who's boss makes you an asshole.