We don't artificially throttle things, if that is what you mean. We use a multi-tenancy architecture, so this site is running in exactly the same process, on exactly the same hardware, as every other site. There is some slight biasing, mainly for balancing in-memory cache for things like stackoverflow (basically, requests heading towards stackoverflow go to a particular set of nodes) - and at the database level obviously some databases are on different servers, but: your beta site here is treated no different to, say, physics.SE, programmers.SE, etc.
In terms of things like gravatars: that's nothing to do with us : ask your browser and gravatar why that is taking longer.
One interesting thing to note is that cache basically spreads pain over multiple users. If a piece of data is good, say, for 1 minute - and that site is getting used 100 times a minute, then that is 99 requests (99%) who don't see any latency. However, if it is getting used twice a minute, then 50% of the requests will actually need to do work. This will be exacerbated by things like database data loading: if the data happens to already be in memory, access is faster. If the site is used lots, it is more likely to be in memory. So quiet sites could indeed require more physical data access (rather than in-memory). OK, we use SSDs, but there's still time.
Finally, some actions are themselves triggered by ambient activity; this means that on a very quiet site you might see things one request later than you would on a busy site. For example, if you reload the questions page, and you are the only user, it might give you the stale version, even though it knows it is a little old. Then because it saw usage it will do the work to rebuild the lists in the background - so next time you refresh (etc) it may display the shiny new data.
However! For most things: less data means less things for the system to sift through; so you can actually get faster performance on a brand new site. Obviously, though, we don't specifically optimize for that case.
Does that cover most things?