"You shall keep the feast of Shavuot... You shall rejoice before Hashem your God - you, your son, your daughter, your male and female slave, the Levite within your gates, and the stranger and orphan and widow who are amongst you - in the place Hashem your God will choose for His name to dwell in." (Devarim 16:10-11)
It's worth noting where this holiday feast will take place, as well as where the participants come from. The holiday of course must be celebrated in Jerusalem. But "you", the average person, are coming from somewhere else in Israel. So, for that matter, are the stranger/orphan/widow/Levite who are sharing your hospitality - they, who live "amongst you" and "in your gates", not in Jerusalem. It is not just a matter of inviting them for one meal. You have to invite them for the entire holiday, and pay all their travel expenses as well, if they themselves cannot afford it. Otherwise they would not be able to come, and you must ensure that they can come.
When we make reservations for our exclusive glatt kosher hotel holiday vacations, how many of us remember to pay for the local poor people to come along with us? But, at least when the Temple stands, that is exactly what the Torah requires.