Broadly speaking, the reasons for using a small group of elites over a larger group of less elite forces are
- Stealth: it's a lot easier to sneak up on a dragon with six people than with twenty people.
- Logistics and Mobility: the difficulty of transporting a unit is generally proportional to its size, as is the effort of getting through many obstacles, the amount of food and shelter it needs, and so on.
- Organizational Speed: the more people you have, the longer it takes to get organized.
- Concentration of Force: in constrained quarters, it's hard to bring the full power of a large unit to bear. An army is going to do a lot better in an open field than at getting through a tunnel ten feet wide.
If those things don't apply... maybe this isn't an adventurer job?