Can't rewield a dagger if dual wielding a heavier weapon

I'm not sure if this falls under the general category of things that are wrong with wield/dual wield and are hard to fix or not. As a dagger spec, you can dual wield any one handed weapon regardless of weight (within strength limits). I assume this is intended. However, if your dagger gets disarmed you can't pick it up and wield it. You get the "doesn't complement your secondary weapon" message.