Abayei heard a certain man say to a certain woman: "Let us go on our way". [Abayei] said: I will go and prevent them from committing a sin [together]. He went after them three parasangs in the field [a long distance, to an isolated place]. When they parted from each other, he heard them saying "Our company is pleasant but the way is long." Abayei said: "If it were me, I could not have restrained myself." He went and leaned in anguish against a doorpost, until a certain elder came and taught: "The greater the man, the greater the Evil Inclination" (כל הגדול מחבירו יצרו גדול הימנו). (Sukkah 52a)
It is surprising to hear that a great Torah scholar like Abayei is more, not less, likely to commit a severe sexual sin than the man he followed. Isn't Torah study, and Torah commitment in general, supposed to refine your character? Doesn't the same page of gemara (52b) say that if you encounter the Evil Inclination, you should "drag it to the beit midrash" in order to overcome it?
Therefore, I wonder if the "greatness" which is said to augment your Evil Inclination is social rather than spiritual.
Spiritual leaders as diverse as King David, Martin Luther King, and Shlomo Carlebach have been known to commit serious sexual crimes. It is hard to see this a function of their spiritual inclinations, but easy to see it as a function of their being respected leaders. As talented charismatic high-status men, such leaders can easily meet, attract, and seduce many women. As holders of communal authority, they can arrange for their affairs to be discrete, and silence those who would publicize or punish their crimes later on. If an accusation is ever made against them, their word will be trusted over that of their accusers. We should hope that such leaders have fewer affairs than similar secular leaders (JFK and Clinton come to mind), but not be surprised if they have more affairs than the average religious man.
Abayei, too, was a respected leader. He was more capable of getting away with sexual crimes than was the average man. This would make his challenge harder than theirs, even if in other respects his character was as good as theirs or better.