wiki查到的 結論應該真的是地理位置=>有錯請更正

In American and Canadian English, downtown refers to the commercial heart of a city. This is by analogy with its coinage in New York City, where downtown was used to refer to the early business center of the island of Manhattan, located at the southern end and now properly referred to as the Financial District.

Downtown is now used in two senses in New York City. Since the city now has two major business districts, downtown is still used as an informal name for the Financial District (or more broadly for Lower Manhattan), in contradistinction to Midtown, the other commercial center. (Anything north of 59th Street, which is most of the island, is uptown.) However, the terms downtown and uptown are also used to indicate direction of travel: someone travelling south is heading downtown and someone heading north is heading uptown. By extension, the side of a north-south avenue on which traffic is travelling downtown (the west side) is referred to as the downtown side of the avenue, and analagously for the uptown side.