It is widely thought that continental chemical weathering provides the key feedback that prevents large fluctuations in atmospheric CO2, and hence surface temperature, on geological time scales. However, low-temperature alteration of the upper oceanic crust in off-axis hydrothermal systems provides an alternative feedback mechanism. Testing the latter hypothesis requires understanding the timing of carbonate mineral formation within the oceanic crust. Here we report the first radiometric age determinations for calcite formed in the upper oceanic crust in eight locations globally via in-situ U-Pb laser ablation–inductively coupled plasma–mass spectrometry analysis. Carbonate formation occurs soon after crustal accretion, indicating that changes in global environmental conditions will be recorded in changing alteration characteristics of the upper oceanic crust. This adds support to the interpretation that large differences between the hydrothermal carbonate content of late Mesozoic and late Cenozoic oceanic crust record changes in global environmental conditions. In turn, this supports a model in which alteration of the upper oceanic crust in off-axis hydrothermal systems plays an important role in controlling ocean chemistry and the long-term carbon cycle.