[A Ladder]

I believe the trick to keeping magic “Mysterious” is to actually do that. Keep the Rules of the system a mystery to the players.

By only giving the players partial information about the magic system, it’s possible to keep magic as a mysterious element. Instead of fleshing out the entirety of how the system works only give them a brief overview of what the penalties, rules, and possibilities entails at the outset. This would force players to pay attention to every bit of detail when spells were being cast to try to figure out what was going on. Net effect: Mystery.

Example: Say your magic system is based on a free-form keyword base like RPM, with costs coming from Threshold and Fatigue. Threshold starts at 0 and has a minimum cost of 1, with no upperbound. Fatigue is generally used to offset the Threshold cost but a minimum Fatigue expenditure is also based on key words used in the spell (IE: All Damage spells and ones that affect an area or multiple targets have a minimum 1 FP, etc.).
Threshold costs are assigned based on how “big” of magic the PCs want to use, but with random element added to them. Then, the effect of the spell is based on a Reaction Roll table, much similar to Divine Favor. A better reaction = a “better/bigger” effect or less Threshold costs.
Increasing your Reaction Roll on the spell table is based on a number of attributes: character behavior, the subjects involved, any esoteric items/ingredients involved, chanting, special locations, time of day, etc…By adding these “Traditional Trappings” to the spells they start to feel more eldritch as they don’t seem to follow the natural order of the how Science explains the universe to work. “Why does creating a magic circle out of the dust of a dead relative make teleportation spells easier?” etc. But also, the randomness adds a hint of unpredictability to the casting. You could think you know all the rules of how to get bonuses, but still get a bad roll and have a wimpy spell go off.

However, you only tell your players that spells use Fatigue and might require research or knowledge checks to make their spells better. The building of the spell is all in the GM’s hands. The GM merely asks the player “what do you want to do?” and assigns keywords and costs based on what the player describes. The GM can give the option of asking “Do you want to spend any FP?”, without telling them what that paying of FP does. Then the GM rolls the 3d Reaction Roll (in secret) every time a spell is cast and doesn’t tell the PCs what that does. Already the Pc’s are curious at all the unknowns. Ah! Mystery!
At this point PCs will start with just trying to cast spells with the least amount of FP. This generally results in faster Threshold accumulation and either less spells or more catastrophic ones. The GM should liberally allow them Per-based Research/Thaumatology/Occultism checks to notice one aspect of something affecting their spells: “Oh the moon was especially bright during your previous Ray of Frost and doesn’t seem to have the same effect during the day.” “Your research shows that those who spend more of their body’s own resources allow spells to be more stable.” Etc.

This gives your players incentive to figure out what the hell is going on in your magic system due to all the info they don’t know. The players will then have to make notes on what helped and what didn’t and slowly begin to unravel the rules, the mystery, of the system. But, the Reaction Roll keeps everything not quite "A+B+C = D"-like science.

**But make sure the players (not the PCs) know that there are things the GM is looking for behind the scenes when they cast spells and are on board with this system first!**