Backstab problems

When I'm wielding my fist (no weapon) and dual-wielding a dagger, I cannot backstab. I get an echo that says "You need to wield a dagger to backstab." But then when I wield, like, a sword and keep the dagger in the off-hand, now I'm allowed to backstab. If I need to wield a dagger to backstab, shouldn't I have to switch the dagger to the mainhand then?