"The ‘ma’aleh ashan’ [herb?] would [cause the smoke from the burning incense] to rise straight up like a staff…" (Yoma 53a).
These sources discuss the smoke issuing from incense burning in the Temple. We see from both sources that the smoke was supposed to rise in a perfectly straight column.
This incense column is NOT the same smoke that Pirkei Avot 5:5 says was miraculously never dispersed by the wind. That smoke was from animal sacrifices burning on the large outdoor altar. We are talking about smoke from incense burned inside the Temple, where there was no wind. There is no need to assume that the vertical rising of incense smoke is miraculous. In fact, I will now provide a purely natural explanation for it.
Examine the following photograph of burning incense. At first, the smoke rises in a smooth vertical column. Then, past a certain point, the column collapses into a bunch of turbulent swirls. This is a common phenomenon, which I have seen with incense sticks and cigarettes, as well as in this picture.
It seems to me that the gemara is talking about this kind of smoke column and swirls. Smoke naturally begins as a straight column, and past a certain point becomes turbulent. Depending on the type of incense, the turbulence could begin sooner or later. The rabbis desired incense whose turbulence was delayed, so that the smoke column remained vertical for as long as possible.
I asked my father, a physics professor, to explain why this phenomenon occurs. Here is his response, (with the "bottom line" highlighted, in case you don't have patience for the technical details).
The rising column of hot air is unstable to shear instability (the simplest shear instability is Kelvin-Helmholtz instability that occurs between two half-spaces with relative velocity parallel to their boundary) that develops into turbulence. This can be stabilized by viscosity or density stratification (the latter is relevant to Kelvin-Helmholtz in the atmosphere, ocean, etc., but not to a rising column of hot air).
If the rise is slow and the column thin, viscosity will be more effective in stabilizing the flow. Very slow burning will then tend to produce a steady column. The burning rate will depend on how finely the incense is ground, what it is made of, moisture content, degree of packing, etc. The trick is to slow the burning without making it extinguish itself.
So there you go. Now you, too, know how to make perfectly rising incense, like the house of Avtinas did. Well, at least you know which factors to experiment with until by luck or skill you get it right. As I learned when my roommate did a class project on gas engine combustion chamber designs, the precise analysis of burning is very complicated and sometimes not fully predictable even by modern science.
Why was it desired that the smoke column rise straight upward? Possible reasons include: 1) It was simply more aesthetically pleasing. 2) It was cleaner – smoke concentrated at the ceiling rather than filling the entire room. 3) It was symbolic of the incense going directly to God and not getting "stuck" on the way there. 4) The rabbis noticed the link between vertical rising and slow burning; wanting the incense to last longer, they wanted to see vertical rising as an indication of this.
Whatever the reason, it seems this goal was not trivial to achieve. According to the gemara, a particular substance ("ma'aleh ashan") had to be mixed into the incense. But just the presence of that substance was not sufficient, and expert preparation was needed as well. The House of Avtinas were the experts, and the rabbis tried unsuccessfully to break their monopoly on the correct method.
One final word on the column of smoke from the outdoor altar. Pirkei Avot says that this smoke rose vertically as well, miraculously, despite the wind. What was the point of this vertical rising? Explanations 1), 2) and 3) from the incense column would also seem to apply here, with minor changes. 1) A vertical column of smoke would be more attractive, and like a skyscraper more visible to people far from the Temple. (Compare this to Akedat Yitzchak, where in the midrash Avraham saw a cloud over Mt. Moriah before arriving. Maybe he was prophetically envisioning smoke from sacrifices there?) 2) Anyone who has made a bonfire knows how annoying it is to stand downwind from the fire and get smoke blown in your face. With a vertical column that would not happen. 3) The burning sacrifices are supposed to provide a pleasing odor" to God (Bamidbar 28:1 and other verses). If the smoke is blown horizontally, then it never goes up to the heavens, giving the impression that God is literally pushing the sacrifice to the side and rejecting it.