WHY DO WE USE THE SAME OLD FANTASY RACES?

The easiest answer to the question is that it's just easier that way. Whether it is ease of design (the work is already done) or easy recognition and understanding by players and GMs, by using existing, well known fantasy races the designer of a setting cuts down on the amount of work they have to do to get their game to the table and player engaged with it. But does a game setting need the standard races to be enjoyable, accessible, and playable? The answer to this question is a resounding and firm, "maybe".

The major problem with filling a fantasy world with new and exotic life is that it can become murky and mired down in detail that detracts from the game. This is not necessarily the case, but it is always a danger. Having fantasy races in your game that are too weird or too numerous can create a sense of wonder and imagination, but it can also leave players (including the GM) nothing to anchor too. If everything is new and alien then the barrier to understanding and engagement is that much greater. For many players this is enjoyable and desirable as the act of playing the game becomes a true exploration of something wholly new and wonderous. However, for many players this will just end up being a frustrating struggle to figure out who and what they are and where they fit into a world they do not understand.

The TTRPG "Talislanta", first published in 1987 and with several editions following up to the present, is a great example to use when examining the effect of trying to build a world where none of the typical fantasy races exist. Bringing hundreds of unique races and cultures to the players, right up front, Talislanta was championed by its pitch of "No Elves!", a motto that was not entirely true in practice even if it was mostly true in theory. 

I remember pouring over all of the insane, imaginative, fantastic races presented in the the rules and feeling an immense sense of wonder and possibility both as a GM and as a potential player. I read through all the races and the monsters, which are also mostly unique and unusual, and coming away feeling that initial sense of wonder withering under a shadow of something else - confusion. 

The problem with Talislanta is that there is just too much going on to make setting into the game easy. the best I could come up with was to pick a place in the world to start the game and limit options to what was common there. As imaginative and evocative as Talislanta was, it was just too much. The races there were a mix of strange and exotic and subtle variation on tropes and themes used elsewhere. While the "No Elves!" claim was mostly true, there were plenty of races in the game that were elves in everything but name.

If we do want to use new races in our games, we need to do so with a bit of restraint, something Talislanta lost sight of. D&D as a game, has always presented itself with a few core races and then expanded from there. Considering this design choice has been repeated across countless TTRPGs from dozens of genres, there is clearly something to it and it is worth taking note of when designing our own weird worlds of fantasy. 

When laying out your assortment of new fantasy species, it helps to have at least one that is potentially more accessible and less alien than the others. This will give your players a reference point by which they can gauge the weirdness and newness of the other species. Even Talislanta had familiar tropes even if it called them something else and there were "humans". Assigning familiar tropes, even just as GM notes, to your new races can go a long way in making them more playable and understandable to your players. A simple "like wood elves" or "like dwarves" or even "Klingon/Vulcan mashup" can go a long way in providing a useful shorthand for your new fantasy (or scifi) races. The main lesson here, is to keep the pallet simple and limited to a handful of starter races. 

All TTRPGS ultimately introduce more and more unique races over time, providing new options for players and GMs alike. Often these are added as the game world expands and new territories are explored. This keeps the game interesting and provides new ways to play but can lead to a loss of theme and overall feel if you are not careful but it does avoid the mire of options that I believe hampered Talislanta.

On a final note, it is important to consider that you do not need to completely reinvent the wheel when it comes to your fantasy races. A simple reskinning of existing races is often all that is needed to make a fantasy world that seems different even if it present the same options. An example of this, in my mind, is the classic TSR TTRPG "Star Frontiers" where we have four core races that have always seemed like analogs for common fantasy races, just given a nice scifi overhaul. While they didn't demonstrate the same appearances or abilities as D&D races, the Vrusk, Dralasite, and Yazirian of Star Frontiers did present family racial profiles and overviews. 

It is an easy exercise to create a world where the common D&D races have new identities. You could make them all into anthropomorphic animal races if that was the campaign you wanted or perhaps you want a world where there are no gnomes but you still want their abilities in the game and so they become some new fantasy race. It would be entirely possible to create a world of elves where every classic D&D race was now some form of elf, living in new and interesting ways. Races handled in this manner are less work for the DM, in terms of game design, and really need as little or as much background work as you want.