Small bodies of the solar system, like asteroids, trans-Neptunian objects, cometary nuclei, and planetary satellites, with diameters smaller than 1000 km usually have irregular shapes, often resembling dumb-bells or contact binaries. The spinning of such a gravitating dumb-bell creates around it a zone of chaotic orbits. We determine its extent analytically and numerically. We find that the chaotic zone swells significantly if the rotation rate is decreased; in particular, the zone swells more than twice if the rotation rate is decreased 10 times with respect to the "centrifugal breakup" threshold. We illustrate the properties of the chaotic orbital zones in examples of the global orbital dynamics about asteroid 243 Ida (which has a moon, Dactyl, orbiting near the edge of the chaotic zone) and asteroid 25143 Itokawa.