The point of making something scary, is that it should have some degree of "inevitability": too strong to be fought, too weird to remain sane, incurable.
The point is not to cripple permanently the PCs to the point players don't have fun anymore in playing them, but to bring back a sense of danger.
In D&D, fights are normal events, so is getting wounded and healed, or possibly ressurected. If death is not permanent and any injury can be healed easily, no fight, no opponent is ever going to be scary unless there is something else at stake that cannot be undone.
Destruction of a friend/family/city/nation/continent/world
Loss of something dear: family items, rare artefact
Loss or damage of something personal: sanity, hands, foot, eye... which could be restore, but in a form that will be a constant reminder of the encounter.
I personally love sanity loss: developping a phobia or other mental troubles is usually never enough to make a PC unplayable, but it has a "weight", and players know/feel that the more of these phobia they have, the closer to permanent retirement their character is.
To make something scary, you need to instill a sense of doom, a threat of something which cannot easily be undone.
In my modern fantasy game, there is no ressurection possible, hence any fight is carefully weighted before being started - and I roll all my dice in front of the players, so there is no fudging of dice. They know I won't try to kill them, but I won't save them neither of a awful roll, even less from their own bad decision.
Also, rarity of a threat makes it scarier. If demons pops out every adventures, PCs & players will become seasoned demon fighters and it is normal that they won't be as scared or impressed by Belzeball III jr after a while. They might even look for a worthy opponent to enhance their demon hunter status.
On the other hand, if suddenly you toss in front of them a powerful undead, with ability that they are not used to counter, spells that they have never seen, they will wonder "how can I protect myself from that ?" "Can i dodge it ? Survive it ? Cure it ?". Or maybe they discover that it comes back each time more powerful, with a better knolwedge of the PCs' weakness. It can be even better if the undead (or whatever suitable monster) looks like a demon, or is mistaken for a demon and suddenly proven techniques against demon do not work - maybe kicking once the whole party ass.
