Well, more advanced attractive properties can override that fact, such as intelligence, and tastes in various matters, but basic human sex looks purely at physical attractiveness, since it signifies health (supposedly) and is supposed to guarantee the best chance for offspring.
Males and females leer because they want a hot and healthy body to slide into bed and mate with, so there's no real reason why get upset when the boys want to run commentary on some random lady. The deeper issues comes when you're serious about a potential mate.