What formulas for damage have you been experimenting around with? Are you open to making defense divisive? You'll just need to make sure defense is never zero to avoid returning undefined unless you have some constant added to it in the damage calculation. If nothing else you could always put the resulting damage in a max function so it always deals at least whatever constant you put in the max function. If you're sticking with subtractive defense, you'll have to accept thresholds. Divisive defense will give you that gradual reduction. This will have other effects though as multi-hit attacks won't behave different from single hit attacks with divisive defense, so you have to carefully pick what you want.
Are there buffs and debuffs to your armor/attack power? How do want them to work? Should a move that reduces defense by 50% always make it so you deal 50% more damage to the debuffed enemy?
If you want, give me more details about what you desire from your systems and what you've been trying, then I can try to help you figure something out as someone who is a bit nutty about the numbers side of RPGs.
edit: Here's some very basic formulas to offer an idea
Divisive: Attack * 10 / (10 + Defense) --- Will never fully reach 0 (though can go below 1), can't return undefined even if defense is 0, damage will always be equal to attack power against enemies with no defense. Defense buffs/debuffs could get weird with the constant unless the buff/debuff percentage increase/decrease is applied to the constant as well, that'd have to be ironed out.
Subtractive: Max(1, (Attack - Defense)) --- Simple subtractive formula just wrapped in a max function so you always deal at least one point of damage. May be ideal for very low numbers. This is BotW's damage calculation by the way. One heart is 4HP, you'll always lose a quarter heart no matter your defense.