Whedon did, in an interview, admit that Firefly was based upon an RPG he played in college, IIRC, he used the words "a major science fiction RPG." That interview has been removed from the web for years now. He didn't say which one...
It is quite likely that it was Traveller - the trope set fits, except for the FTL drive... but we see a 2-mode STL drive. One that seems to do about 33 PSL, and one that is considerable slower, but uses far less fuel. (This speed is based upon taking 120 years to cross the 40 LY to the system.)
The odds of the other major Sci-Fi RPG's of 1984 being the game are slim. We're talking a handful of games. We can rule out Trek games out of hand.
- Traveller - major and best trope fit
- Space Opera - uses zapguns.
- Space Patrol - not major. Trekish. No ship rules.
- Starfaring - not major, zapguns
- Star Frontiers - major, but really poor fit. Lasers dominate personal weapons.
- Space Infantry - not major.
- Worlds of Wonder - possible, but unlikely. No ship rules.
- Champions - possible, but not seen as a sci-fi game by most; vehicles rules in a 1982 supplement. Star Hero comes out in 1989, Robot Warriors in 1986 - after college for Whedon.
- Other Suns - not major, loads of aliens.
- Spacemaster - zapguns. aliens galore.
- Starships & SPacemen - not major, trek
- STRPG - came out after he graduated - RIGHT after.
- STAGFF - Trek
Of these, the short list is
- Traveller - major and best trope fit (only FTL differs)
- Space Opera - uses zapguns and lots of aliens.
- Champions
- Other Suns - not major, loads of aliens.
- Spacemaster - zapguns. aliens galore.
Note that Traveller, despite being noted for aliens, has no aliens in core rules until 1987, and even then, only as setting elements not supported in rules.
Space Opera as a game has the rules needed - it covers much the same territory as Traveller - but lacks the plethora of habitable moons and mini-worlds. It also has lots of aliens and no reaction drives nor semi-newtonian motion.
Spacemaster was out (I got it Dec 26, 1983 same day as I got Traveller) despite some mis-listings (at RPGG, it's shown as 1985 - that's an error). But it was also brand new, and not well known. It has the same issue with FTL... but at least ships fly on reaction drives in N-space. Personal weapons are a poor fit, tho. No trade rules, either. Ships not expensive enough, either.
Other Suns - noted for aliens galore. It was from a major publisher (FGU) but was not itself major. I can't be more specific, as I never got into it.
Champions with Champions II - has the rules one needs in play, but lacks entirely the world generation process. Lacks cogent firearms examples, and Danger International, which is the HS game that does, post dates Whedon graduating.
Of the possibles, the number of coincidences is high if it's not based upon Traveller - the Serenity can be seen as a variant of either the type A or type R. My money's on a Type R - especially with the forward opening cargo bay. A lot depends on just what the dimensions of the cargo bay are. I make her externals to be roughly a 400-600 ton ship... a type R with 2 launches and a 30'x60' x30' cargo bay puts her with the right amount of cargo. (The Serenity RPG, however, would put her with a much smaller bay - 30'x40'x30' - commensurate with a variant type A - but the Type A carries an air raft, not launches. Come to think of it, Serenity does carry both a small ATV (the mule) and an air-raft (in the movie).
Preponderance of the evidence leads to a conclusion that Traveller PROBABLY is the RPG he spoke of, but it's not a given that it was.