[T.K.]

Having played Mage The Ascension extensively, Witchcraft, Arcanum, Ignis Fatuus and quite some different magical systems from GURPS Thaum (Currently using 5 different magical systems in my setting), I'd say that to attain the mysterious and eldritch feeling you're looking for, you'll have to go away from the usual "D&D/DF/DungeonCrawling-esque" mechanics.

If you look at books, movies, tales, legends... normally what composes mysterious is something you quite can't grasp as a whole or understand the full extent of the system and its outcome, even if you manage to understand some bits of it .

For magic, more specifically, what I mean is you gotta break some part of the system and take that away from players, so that leaves them wondering what could happen exactly.

I'd say at a very basic level, a straight hermetical magic type spell would be something like:

Effect Visualization > Performer/Material/Targets Gathering > Ritual Performance > Manifestation > Check of Visualization X Manifestation

So, as I previously said, this full complete system and its understanding takes away any (or almost all) variance and mysticism, because all possible variables are controlled or already expected by the performer.

Taking away 1 or more of these steps from the control of the performer would add degrees of variance which in turn could add mystery (Not necessarily, though...variance can be taken as annoyance if done wrong).

Braking each step to see what we'd gain from removing the performer control:

Effect Visualization removed

I find taking this away would be more annoying than mystifying, since if you can't even decide what you're trying to do or alter, how are you supposed to expect it to even work in the first place.
For me, this is a no no to remove from the control of performers and thus players.

Performer/Material/Target Gathering

Here things start to become interesting... What if materials are not strictly defined and/or depending on what you use you have somewhat different results?
Performers would have a wild guess on what could be used as material, but often could try different or unusual components to try different effects. This could lead to quests often found in tales for special materials or leave a feeling of "What's gonna happen" at each casting with different or improvised materials.

Performer is also quite a good one, imo. WHO actually performs the magic? What if the magi only puts things in motion, put the real magic is done by a superior thing from the void? What if performers only open energy lines to the primordial planes and then the entropic forces act on their own? What if performers are bound to ancient pacts with fey Thuatá De Danaan and their whimsical desires is what actually can mold reality?
Even if they know what actually performs the magic, not being the ones in control of it allows room for exquisite things to happen.
This, I believe, is a step with great potential for mysterious depth and adding. Depending on the variance of the true magic force matrix, even castings with exact same steps on the other points could come out quite different due to this step possible variance while still being true to its set definition.

I wouldn't mess with Targets, since it tends to get annoying fast, when performers and thus players, start missing their expected targets or simply not affect them the way they thought.

Material, good to tweak. Performers, very good to tweak. Target, bad. imo.

Ritual Performance

I like to mess with this step quite a lot for the purpose of mystifying magic, specially coupled with the previous Material and/or Performer step.

If the forces moving the universe around and thus the ones peforming magic are subject to considerable change and mutability and the rituals that put those forces in motion are also not quite defined or known, you have a lot of wiggle room for the players to be wondering what exactly would happen, how it would happen and so on...

Personally, I find this couple with the previous step is where your gold is.

Manifestation

I normally find that tweaking with this gets frustrating quite fast. If players are performing the same rituals, with the same mats, to the same forces and the results are wildly different they tend to treat it as an unreliable force quite fast and will only apply it when they can withstand the chance/variance or almost not at all.

If tweaking this, do it only for fluffy mostly and not really sweeping, significant variances, imo.

Check

This is simply the stage where the magi, the player, checks if the effects meets his desire.
Put it simply, is the step where somehing like "I wanted to cast a Fireball...did I cast a Fireball?" happens.
Sometimes magic takes time or doesn't leave any visual/material manifestation and not knowing straight right away if your magic worked as you desired it to work, if done in the correct amount and tone, can also add some degree of eldritchness.
You know those scenes in movies where the main characters push something, or perform a ritual or do something and there's that 2-5 seconds with complete silence and stillness and camera shows the whole area and you kinda stop your breath waiting what's gonna happen and IF its gonna happen?
While it's an excitement buildup, if you look from the magi/player perspective, not being 100% sure if/how it'll work, even after performing correct known magical formulae, it could add a layer of mystery to it.

Just don't overdo it or magic will fall into the "unreliable" territory quite fast.