The standard college football season is 12 games. (Fine - some play one or two fewer, some might play one or two more. but let's stick with 12 for the conversation)
What's the maximum number of teams that you could reasonably have within a conference and still keep the high profile/high profitable rivalry and meaningful in-conference games?
And let's not forget that there are those throwaway games for homecoming or "warmups" (hello, Mercer or GA State vs Alabama) that rely on big paydays for being perennially steamrolled that may lose those beneficial slots because a conference became so large that there was no room for non-division games (or Notre Dame) to be squeezed in.
Take an example: which one of these Alabama games would you sacrifice to make room for Michigan in a "required" annual rotation? ('Bama used as an example. Clemson or other major ACC/SEC school would have similar issues)