ותתן לנו ה' אלקינו באהבה את יום הזכרון הזה, יום תרועה, מקרא קודש, זכר ליציאת מצרים
Every major Jewish holiday includes the phrase "zecher litziat mitzraim" in its prayers and kiddush. In most cases the reason is clear, or at least understandable. For Pesach, the reason is beyond obvious. Sukkot commemorates our dwelling in the desert after leaving Egypt; Shavuot marks the giving of the Torah which happened shortly after leaving Egypt, and Yom Kippur is the day Moshe descended with the second tablets a few months after that original Shavuot. Of all the holidays, Rosh Hashana alone is left without an explanation for why it is a "remembrance of leaving Egypt".
This question bothered me a little the past few days, and at Rosh Hashana dinner Wednesday - sitting next to a bunch of people I'd just met in the Gush dining room - I decided to ask them what they thought. After receiving the following answer, I felt no need to ask further.
The Ramban, commenting on Masechet Beitzah, explains the difference between the work prohibitions on Shabbat and Yom Tov. On Shabbat, all creative activity - "melacha" - is prohibited. This is to commemorate God's creation of the world, and as we say in the prayers, Shabbat is a "zecher lemaaseh breishit". On Yom Tov, the prohibition is different. In the Torah's language, what is prohibited is not "melacha" but "melechet avodah" - creative work. The purpose is not to commemorate creation, but to commemorate work - specifically, the work we had to do in Egypt until we were freed from there. Instead of serving Pharoah, we now serve God. We do this all year round, but particularly on holidays, when we visit Jerusalem and take part in the Temple "avodah". So it is then that the phrase "zecher litziat mitzraim" is most appropriate.
If so, then we should not expect to find any connection between Rosh Hashana and leaving Egypt. The connection is rather between leaving Egypt and holidays, regardless of each holiday's particular theme. A conclusive proof of this comes from Shabbat kiddush, which calls Shabbat a "yom techila lemikraei kodesh, zecher litziat mitzraim". Shabbat itself may not involve Egypt, but Shabbat does determine the character [?] of holidays (mikraei kodesh), and all mikraei kodesh by their very definition apparently involve Egypt.
I thought this such a good answer that I wanted to quote it here with the name of the person who told it to me. But he told me that this Ramban is "famous" and "everyone" now at Gush knows it, implying that there is no reason to quote him as the author. Wow. If that's common knowledge, then I wonder what went wrong with my own education. All I know is that this Rosh Hashana, I chose a pretty good place to be commemorating leaving Egypt.