"A woman's voice is nakedness" (Brachot 24a)
The justification of this statement is not obvious to everyone today. Sure, many people go to concerts in order to swoon over the performer of the opposite sex, but there good looks and suggestive dancing are at least as much an attraction as a good voice. And those performers are generally required to have relatively good voices. For the rest of us, with our often ugly voices, the attraction is clearly much weaker. Yet Chazal called that voice "nakedness", and prohibited listening to it in the same line in which they prohibited staring at a woman's leg!
To me, this is best understood by reference to Greek mythology. Measures of sexual attractiveness are influenced by one's culture, and the Greeks and Romans whom Chazal lived among found the female voice to be very seductive. I am thinking of the myth of Odysseus and the Sirens. There are, it is told, sea-dwelling creatures which sing in a woman's voice, whose song is so seductive that any man who hears them will have an irresistible urge to abandon ship and swim to them, only to be killed. Unlike the mermaids of other cultures, who often had beautiful bodies, the sirens were attractive only because of their voices. The rabbinic decree of kol isha responded to what, at the time, was apparently a very strong temptation, as hinted to by the myth of the sirens.