It was narrated that Aisha (RA) said: "The Quraish said: We are the neighbors of the House and we do not leave the sanctuary. Allah said: Then depart from the place whence all the people depart." (Sahih)