I don't want to seem too dismissive of the size of the sun at noon so let me break it down a little, incase you actually want to do this thought exercise yourself.
At noon in LA it's 3pm in New York. There is over 2000 miles between them in the East-West direction.
I can see the Sun from New York at 3pm (it's 2000 miles west). Not only that but I can still see it at 6pm, closer to sunset. So if it's moved 2000 miles at 3pm then it's moved 4000 miles at 6pm, and I can still see it.
If we use a little basic math (sorry, has to be done) we can use the Pythagorean Theorm to find the distance from you to the Sun (assuming it's 3000 miles in the sky and 4000 miles west).
c2 = a2 + b2
c = sqrt (a2 + b2)
a = 3000 (height of the sun)
b = 4000 (sun traveled 4000 miles west at 6pm in New York)
c = distance from you to the Sun 6 hours after noon
c = sqrt(3000 * 3000 + 4000 * 4000)
therefore c = 5000
So the sun at 6pm in New York is 5000 miles away. But at noon it's only 3000 miles straight up.
The way perspective works, the sun would be MASSIVE at noon, relative to sunset.
Hope that makes sense.
For reference:
http://ncalculators.com/images/pythagoras-theorem.gif