The question of why "standard" top tubes are 1" or 26 mm in diameter, when they clearly could be larger, and sometimes have been larger, has been intriguing me, too. The one bike I have with a 1 1/8" top tube, but standard diameter tubes elsewhere, seems to ride quite differently from my "standard" bikes - in fact, I like it less.
Some research has turned up that in the late 1890s, the best French racing bikes were made from 28 mm OD, 0.5 mm wall thickness tubing. Quite lightweight stuff, by the way, lighter than for example Columbus SL. I have no idea whether it was butted or not. It appears they used the same tube as top, down and seat tubes. (I found this in Raymond Henry's "Velocio" book, he quotes an 1890s issue of Le Cycliste.)
I don't have any information about the following decades, but fast forward to the 1930s, and top tubes have decreased in diameter. I don't think it was a random fluke, but it appears that builders felt a smaller top tube was better. Maybe they wanted a more elegant frame? Or they felt the more flexible top tube improved the ride? Or they were concerned about the thinwall tubes breaking, so they increased the wall thickness and decreased the diameter to make up for some of the extra weight? (However, in the 1930s, there are lots of super-thinwall frames out there, all the way to 0.3 mm wall thickness. And those seem to have lasted just fine for the most part.)
From my riding experience, I suspect it is something about the ride, but that is just speculation.
So I don't know why the top tubes got skinnier, and I do know that - as Hilary pointed out - some disagree with that. The fact is that they used to be fatter, and then it appears to have been a conscious choice to decrease their diameter. I suspect that somebody ran out of tubes, and substituted a smaller diameter top tube, or was in the mood to experiment, and when the rider/racer on the new frame came back from their first ride, they rather liked the bike, and that may have started a trend.
There was a lot of experimentation early on, until the "standard" tubing diameters were more or less agreed upon. Those then remained constant for 30+ years.
For example, Rene Herse offered oversize down tubes during the 1940s and 1950s, but not all frames have them, not even all tall frames...
Hopefully, more research will dig up some source that sheds more light on this. -- Jan Heine Editor/Publisher Vintage Bicycle Quarterly c/o Il Vecchio Bicycles 140 Lakeside Ave, Ste. C Seattle WA 98122 http://www.vintagebicyclepress.com