No that isnt why they think that
They know that he is gay, and that conversation is just an example where they were talking about it. It is in the books.
I'm inclined to believe this is false, given the evidence. But, I think this is more of a "what you read" kind of thing. You may infer that he's gay, I infer that he's not. (Sort of like the whole "Dumbledore is gay!" thing a few years back) And that's fine, it's just a style of reading. (I wouldn't go as far as saying it is in the books. The dude was metro. Metro =/= gay.)
I don't like, however, that there are gay scenes with him and Loras. Because that's definitely not canon, not matter how much you perceive it to be inferred.
EDIT: The fact that there's a gay scene doesn't bother me (all God's children), and I could honestly care less. But, the fact that it's there when it's not in the book is stupid.