1d4 daggers still have a very good chance of killing non-plot important characters (ie commoners, minions, etc) so I think any fight where the charcter can shrug off such damage it should be clear why they can. It's narrative, this is a tough guy (it's the bulky nazi under the plane from Raiders of the Lost arc)
Mechanically reducing a monster to zero hit points means you can do what you want with them, so damage can really be treated like stages in the fight. Narratively you can have the wound mean as much or as little as possible, as long as it registers as a story beat. A 20 damage can still be important even if mechanically there's still a ways to go. The monster might not be out of the fight by any means, but it could still be a blow that it cringes back from, initially grabs itself for. Perhaps it alters its attack style, perhaps it doens't use an AoO / OA, or perhaps the characters now have a narrative hook to cling to when attacking it, favouring its weak side or using its wound as intimidation. Stuff you can then throw circumstancial bonuses around for.
And it also gets back to the idea of monsters can leave a fight when the fight feels done, not when they are 0 hp. Did the 20 damage hit feel significant? Then it was. The creature staggers, and either tries to retreat while holding the wound, or holds up its hand and surrenders. A creature fights until it feels its mortality, unless it is fighting for its life or fighting for something more important than its life. If the creature still has a decent set of hit points, great. Don't heal it - if the wound wasn't that big a deal after all, it will not do anything more than a quick patch up to stop bleeding, if it is even capable of healing. That blow will still reduce its effectivenes next combat and still matter. If the monster ihas surrenders, then the fact that it is still a decent threat will stop the players from simply deciding they can just cut its throat the moment it might turn tail - it's still a force to be reckoned with, even if it is temporarily scared.
Another way is simply opening up the narrative for each successive good hit. The first great hit smashes in to the orc chieftain's neck guard, smashing it to pieces. The chieftsin's eye grow wide for a moment, and then it keeps attacking angrily. Then, the next critical/large blow of course lands right on the orc's shoulder or neck, giving a severe wound. This time the chieftain falls back, calling all its minions to itself, even if that actually removes an advantage it had.
And then the next hit that lands actually staggers the chieftain, causing him to think about quit the battle.
Again, he might still be on a 1/4 or 2/5 of his hp, but you've had a long significant battle and now it can end.
Anyway, that's how I'd handle such thigns in your case - regard the HP cloud as either a story buffer so that you don't have to just get rid of creatures right away and they can keep returning, or as a fight beat indicator so the players know they can't just hack this creature's head off with a single good roll, they need to throw some tactics and ploys in to make it a fun story and combat.