To a certain degree, it’s the fundamental construction. The X-Men were created and debuted as a team concept, both the originals and the later All-New incarnations. Whereas the Avengers were gathered up from established players who had their own strips, and later on, who’d appeared elsewhere more often than not. So statistically, it’s easier to consider X-MEN to be those characters’ “home book” (i.e.: the place where the most meaningful and important stuff will happen to them), whereas the Avengers’ home titles tend to be the solo books. So on some level, solo series featuring X-Men tend to be perceived by the audience as a whole as add-ons–nice to have in many cases, but not essential. Whereas new X-Teams–X-Factor, Excalibur, Generation X, etc are more readily accepted as genuinely counting.