quality versus player size?

I've been working under the assumption that to have a larger player size on the screen requires that I push MORE data (higher bitrate) to get decent quality. Am I wrong about this? Will I get any major degradation of stream quality on a player 640 pixels wide versus 480 pixels wide (assuming I don't change the data rate).. In particular, I'm probably going to send H.264 at around 640k total bandwidth.  It seems if I push any more data than that, people viewing at home start to buffer or some such...