Quick correction... Times Square (around 42nd St) is in Midtown, not Downtown. Downtown consists of a whole bunch of smaller districts and it's much further South.
My bad! My, that is confusing. Perhaps that's why when one of the subway lines got shut down we got COMPLETELY lost.