You are probably thinking of the
paragon classes, though these were only three level prestige classes.
In early D&D there were no races as such and "dwarf" was a class, which might be what you are actually thinking of. I suspect there are some OSR games that have revived this.
For
GURPS Dungeon Fantasy see "Races as Professions" in
Pyramid #3/50: Dungeon Fantasy II.