My character is focused on destruction as well, but my skill level in restoration and alteration is pretty high also. Conjuration is low for me.
Yeah, my conjuration has been a bit stagnant since I stopped meleeing and went back to pure destruction. It's still very useful for summons as they make a great distraction, but overall I definitely shouldn't have wasted time on smithing and one-handed as I'm not in a position of being somewhat mediocre in the magical skills. I'm doing okay, but definitely still playing catch-up.