You may recall that NCSU beat Maryland in 73-74 tourney championship game 103-100 (OT) in what's known as one of the greatest games ever. NCSU was ranked #1 and went on to win the national championship and although MD was ranked #3, they didn't even into the national tourney.
I can see you're really young, because back in those days only conference champs made the tourney.
Expansion of field
The NCAA tournament has expanded a number of times throughout its history. This is a breakdown of the history of the tournament formats:
1939–1950: 8 teams
1951–1952: 16 teams
1953–1974: varied between 22 and 25 teams
1975–1978: 32 teams
1979: 40 teams
1980–1982: 48 teams
1983: 52 teams (four play-in games before the tournament)
1984: 53 teams (five play-in games before the tournament)
1985–2000: 64 teams
2001–2010: 65 teams (one play-in game to determine whether the 64th or 65th team plays in the first round)
2011–present: 68 teams (four play-in games in the first round before all remaining teams compete in the second round)