Will I lose my late husband's inheritance tax allowance on our home if my estate tops £2M?

Is it only my residence nil rate band that is tapered away, while my husband's remains intact from his death?