It refers to people who have never invited the person who mentions it to a certain event, outing or gathering.