Drywall anchors break in wall, or don't at all! Help?

I'm trying to install a closet shelf into a drywall closet, the first wall, an interior one, worked great, tthe anchors went in with no problems. Now when trying to drill the halls sin drywall on an exterior wall, im running into all kinds of problems. First, the plastic drywall anchors (Togglers, the big white screw in kind) wouldent even the drywall. Next I drilled a pilot hall, which just cut through seemingly very thick drywall, no wood behind it. Next, the anchor went in but it got stuck and snapped off. Why won't these anchors the drywall at all??