Long-Term Monthly Climate Records from Stations Across the Contiguous United States

USHCN Home Daily Data Daily Data Documentation Monthly Data Web Interface

Investigators

M.J. Menne, C.N. Williams, Jr., and R.S. Vose
National Climatic Data Center, National Oceanic and Atmospheric Administration

Table of Contents

• Introduction
• Version 2 Monthly Temperature Homogenization Processing Steps
      Quality Evaluation and Database Construction
      Time of Observation Bias Adjustments
      Homogeneity Testing and Adjustment Procedures
      Estimation of Missing Values
      Urbanization Effects
• Station Siting and U.S. Surface Temperature Trends
• Station Information
• Data Files
• Data Access
• Pairwise Homogeneity Adjustment Software
• References
• Contacts

Introduction

The United States Historical Climatology Network (USHCN) is essentially a subset of the U.S. Cooperative Observer Network operated by NOAA's National Weather Service (NWS). The approximately 1200 USHCN stations were originally selected according to factors such as record longevity, percentage of missing values, spatial coverage, as well as the number of station moves and/or other station changes that may affect data homogeneity. Most USHCN stations are situated in rural areas or small towns; however, a smaller number of stations are also part of the NOAA NWS synoptic network, whose stations are generally located at airports in more urbanized environments. USHCN datasets have been developed at NOAA's National Climatic Data Center (NCDC) in collaboration with the Department of Energy's Carbon Dioxide Information Analysis Center (CDIAC).

The USHCN project dates to the mid-1980s (Quinlan et al. 1987). At that time, in response to the need for an accurate, unbiased, and modern historical climate record for the United States, personnel at the Global Change Research Program of the U.S. Department of Energy and at NCDC defined a network of 1219 stations in the contiguous United States whose observations would comprise a key baseline dataset for monitoring U.S. climate. Since then, the USHCN dataset has been updated several times (e.g., Karl et al., 1990; Easterling et al., 1996; Menne et al., 2009). USHCN version 2.0 (Menne et al. 2009; Menne and Williams 2009) was produced using a new set of quality control and homogeneity assessment algorithms, and Menne and Williams (2009) includes an assessment of the version 2.0 maximum and minimum temperature trends.

The USHCN database is used by NOAA to monitor temperature and precipitation over the U.S. This includes the calculation of trends over roughly the last century and regular updates to yearly and monthly state/regional rankings of temperature and precipitation (see NCDC's Climate Monitoring web page). Further background on the USHCN's use in this work may be found at NCDC's National Temperature Trends: The Science Behind the Calculations web page.

In October 2012, a revision to the version 2.0 dataset was released as version 2.5. The version 2.5 processing steps are essentially the same as in version 2.0, but the version number change reflects modifications to the underlying database as well as coding changes to the pairwise homogenization algorithm (PHA) that improve its overall efficiency. These modifications are listed in Table 1 below. Details regarding the PHA modifications are provided in NCDC Technical Reports GHCNM-12-01R (Williams et al., 2012a) and GHCNM-12-02 (Williams et al. 2012b).

Table 1. Differences between USHCN version 2.0 and version 2.5

A brief summary of version 2 processing steps is provided below. A more comprehensive summary, including discussions of the sources and magnitude of bias in the raw (unadjusted) data, is provided in Menne et al. (2009). An assessment specifically addressing the reliability of the USHCN temperature trends in light of station siting concerns is also provided below and in more detail by Menne et al. (2010). Details of the pairwise homogenization algorithm and its evaluation against synthetic benchmark datasets with realistic bias-error scenarios are provided in Williams et al. (2012c). A comparison of the USHCN v2 homogenization algorithm (version 52d) and other approaches to homogenization is provided in Venema et al. (2012). A comparison of adjusted and unadjusted USHCN temperature trends with a number of atmospheric reanalysis datasets is described in Vose et al. (2012). The methodology used in previous releases of the version 1.0 monthly data is described at the NCDC USHCN Version 1 web site.

Version 2 Monthly Temperature Homogenization Processing Steps

The data from each USHCN station were subject to the following quality control and homogeneity testing and adjustment procedures.

Quality Evaluation and Database Construction

See Table 1 for an overview of the database construction and quality control steps.

Time of Observation Bias Adjustments

Next, monthly temperature values were adjusted for the time-of-observation bias (Karl, et al. 1986; Vose et al., 2003). The Time of Observation Bias (TOB) arises when the 24-hour daily summary period at a station begins and ends at an hour other than local midnight. When the summary period ends at an hour other than midnight, monthly mean temperatures exhibit a systematic bias relative to the local midnight standard (Baker, 1975). In the U.S. Cooperative Observer Network, the ending hour of the 24-hour climatological day typically varies from station to station and can change at a given station during its period of record. The TOB-adjustment software uses an empirical model to estimate and adjust the monthly temperature values so that they more closely resemble values based on the local midnight summary period. The metadata archive is used to determine the time of observation for any given period in a station's observational history.

Homogeneity Testing and Adjustment Procedures

Following the TOB adjustments, the homogeneity of the TOB-adjusted temperature series is assessed. In previous releases of the USHCN monthly dataset, homogeneity adjustments were performed using the procedure described in Karl and Williams (1987). This procedure was used to evaluate non-climatic discontinuities (artificial changepoints) in a station's temperature or precipitation series caused by known changes to a station such as equipment relocations and changes. Since knowledge of changes in the status of observations comes from the station history metadata archive maintained at NCDC, the original USHCN homogenization algorithm was known as the Station History Adjustment Program (SHAP).

Unfortunately, station histories are often incomplete so artificial discontinuities in a data series may occur on dates with no associated record in the metadata archive. Undocumented station changes obviously limit the effectiveness of SHAP. To remedy the problem of incomplete station histories, the version 2 homogenization algorithm addresses both documented and undocumented discontinuities.

The potential for undocumented discontinuities adds a layer of complexity to homogeneity testing. Tests for undocumented changepoints, for example, require different sets of test-statistic percentiles than those used in analogous tests for documented discontinuities (Lund and Reeves, 2002). For this reason, tests for undocumented changepoints are inherently less sensitive than their counterparts that are used when changes are documented. Tests for documented changes should, therefore, also be conducted where possible to maximize the power of detection for all artificial discontinuities. In addition, since undocumented changepoints can occur in all series, accurate attribution of any particular discontinuity between two climate series is more challenging (Menne and Williams, 2005).

The USHCN version 2 "pairwise" homogenization algorithm addresses these and other issues according to the following steps, which are described in detail in Menne and Williams (2009). At present, only temperature series are evaluated for artificial changepoints.

1. First, a series of monthly temperature differences is formed between numerous pairs of station series in a region. Specifically, difference series are calculated between each target station series and a number (up to 40) of highly correlated series from nearby stations. In effect, a matrix of difference series is formed for a large fraction of all possible combinations of station series pairs in each localized region. The station pool for this pairwise comparison of series includes USHCN stations as well as other U.S. Cooperative Observer Network stations.
2. Tests for undocumented changepoints are then applied to each paired difference series. A hierarchy of changepoint models is used to distinguish whether the changepoint appears to be a change in mean with no trend (Alexandersson and Moberg, 1997), a change in mean within a general trend (Wang, 2003), or a change in mean coincident with a change in trend (Lund and Reeves, 2002) . Since all difference series are comprised of values from two series, a changepoint date in any one difference series is temporarily attributed to both station series used to calculate the differences. The result is a matrix of potential changepoint dates for each station series.
3. The full matrix of changepoint dates is then "unconfounded" by identifying the series common to multiple paired-difference series that have the same changepoint date. Since each series is paired with a unique set of neighboring series, it is possible to determine whether more than one nearby series share the same changepoint date.
4. The magnitude of each relative changepoint is calculated using the most appropriate two-phase regression model (e.g., a jump in mean with no trend in the series, a jump in mean within a general linear trend, etc.). This magnitude is used to estimate the "window of uncertainty" for each changepoint date since the most probable date of an undocumented changepoint is subject to some sampling uncertainty, the magnitude of which is a function of the size of the changepoint. Any cluster of undocumented changepoint dates that falls within overlapping windows of uncertainty is conflated to a single changepoint date according to
   • a known change date as documented in the target station's history archive (meaning the discontinuity does not appear to be undocumented), or
   • the most common undocumented changepoint date within the uncertainty window (meaning the discontinuity appears to be truly undocumented)
5. Finally, multiple pairwise estimates of relative step change magnitude are re-calculated (as a simple difference in mean) at all documented and undocumented discontinuities attributed to the target series. The range of the pairwise estimates for each target step change is used to calculate confidence limits for the magnitude of the discontinuity. Adjustments are made to the target series using the estimates for each shift in the series.

Estimation of Missing Values

Following the homogenization process, estimates for missing data are calculated using a weighted average of values from highly correlated neighboring stations. The weights are determined using a procedure similar to the SHAP routine. This program, called FILNET, uses the results from the TOB and homogenization algorithms to obtain a more accurate estimate of the climatological relationship between stations. The FILNET program also estimates data across intervals in a station record where discontinuities occur in a short time interval, which prevents the reliable estimation of appropriate adjustments.

Urbanization Effects

In the original USHCN, the regression-based approach of Karl et al. (1988) was employed to account for urban heat islands. In contrast, no specific urban correction is applied in USHCN version 2 because the change-point detection algorithm effectively accounts for any "local" trend at any individual station. In other words, the impact of urbanization and other changes in land use is likely small in USHCN version 2. Figure 2 - the minimum temperature time series for Reno, Nevada - provides anecdotal evidence in this regard. In brief, the black line represents unadjusted data, and the blue line represents fully adjusted data. The unadjusted data clearly indicate that the station at Reno experienced both major step changes (e.g., a move from the city to the airport during the 1930s) and trend changes (e.g., a possible growing urban heat island beginning in the 1970s). In contrast, the fully adjusted (homogenized) data indicate that both the step-type changes and the trend changes have been effectively addressed through the change-point detection process used in USHCN version 2.

Figure 1. (a) Mean annual unadjusted and fully adjusted minimum temperatures at Reno, Nevada. Error bars indicating the magnitude of uncertainty (±1 standard error) were calculated via 100 Monte Carlo simulations that sampled within the range of the pairwise estimates for the magnitude of each inhomogeneity; (b) difference between minimum temperatures at Reno and the mean from its 10 nearest neighbors.

Station Siting and U.S. Surface Temperature Trends

Recent photographic documentation of poor siting conditions at stations in the USHCN has led to questions regarding the reliability of surface temperature trends over the conterminous U.S. (CONUS).

To evaluate the potential impact of poor siting/instrument exposure on CONUS temperatures, Menne et al. (2010) compared trends derived from poor and well-sited USHCN stations using both unadjusted and bias-adjusted data. Results indicate that there is a mean bias associated with poor exposure sites relative to good exposure sites in the unadjusted USHCN version 2 data; however, this bias is consistent with previously documented changes associated with the widespread conversion to electronic sensors in the USHCN during the last 25 years (see e.g., Menne et al. 2009). Moreover, the sign of the bias is counterintuitive to photographic documentation of poor exposure because associated instrument changes have led to an artificial negative (“cool”) bias in maximum temperatures and only a slight positive (“warm”) bias in minimum temperatures.

Adjustments applied to USHCN Version 2 data largely account for the impact of instrument and siting changes, although a small overall residual negative (“cool”) bias appears to remain in the adjusted USHCN version 2 CONUS average maximum temperature. Nevertheless, the adjusted USHCN CONUS temperatures are well aligned with recent measurements from the U.S. Climate Reference Network (USCRN). This network was designed with the highest standards for climate monitoring and has none of the siting and instrument exposure problems present in USHCN. The close correspondence in nationally averaged temperature from these two networks is further evidence that the adjusted USHCN data provide an accurate measure of the U.S. temperature.

The Menne et al. (2010) results underscore the need to consider all changes in observation practice when determining the impacts of siting irregularities. Further, the influence of non-standard siting on temperature trends can only be quantified through an analysis of the data which do not indicate that the CONUS average temperature trends are inflated due to poor station siting.

Four sets of USCHN stations were used in the Menne et al. (2010) analysis, and these are available via the following direct links to the NCDC web site and will open in a separate browser window. Set 1 includes stations identified as having good siting by the volunteers at surfacestations.org. Set 2 is a subset of set 1 consisting of the set 1 stations whose ratings are in general agreement with an independent assessment by NOAA’s National Weather Service. Set 3 are those stations with moderate to poor siting ratings according to surfacestations.org. Set 4 is a subset of set 3 consisting of the set 3 stations whose ratings are in agreement with an independent assessment by NOAA’s National Weather Service. For further information, please see Menne et al. (2010). The set of Maximum Minimum Temperature Sensor (MMTS) stations and Cotton Region Shelter (Stevenson Screen) sites used in Menne et al. (2010) are also available. Access to the unadjusted, time of observation adjusted, and fully adjusted USHCN version 2 temperature data is described in the DATA ACCESS section below.

Station Information

The format of each record in the USHCN station inventory file (ushcn-stations.txt) is as follows.

These variables have the following definitions:

Table 2. State numbers and abbreviations for the contiguous United States

Data Files

USHCN data files may be downloaded from CDIAC's anonymous FTP area (see the USHCN Data Access page). Filenames and further descriptions are as follows. The use of '2014' in the filenames indicates that data values may be available through December 2014 (and are available in most records). These filenames differ somewhat from NCDC's version 2.5 filenames, but the key identifiers (e.g., "FLs_52i_tmax") are essentially equivalent. Temperature values are given in tenths of degrees Fahrenheit (e.g., 725 indicates 72.5F) and precipitation amounts are given in hundredths of inches (e.g., 468 indicates 4.68").

Each USHCN data file contains data for all 1218 stations for one of the four meteorological variables. Each record of a file contains one year of 12 monthly values plus an annual mean/total value derived from the monthly means, with formatting as follows:

These variables have the following definitions:

Data Access

The USHCN monthly data are available via FTP or a Web interface that allows users to query, plot, and download individual station data. Please see the USHCN Data Access page.

Pairwise Homogeneity Adjustment Software

The automated pairwise bias adjustment software (Menne and Williams 2009) used to detect and adjust for documented and undocumented inhomogeneities in the USHCN version 2 monthly temperature dataset is available via the NCDC ftp site at:

ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2/monthly/software.

Please refer to the README text file in this directory for guidance on how to download, uncompress, compile and run the pairwise homogenization software. The "tar/txtipped" file contains all of the necessary software to run the pairwise homogenization procedure. A simulated test dataset is included with the software along with a file of the expected output that can be used to verify proper execution of the code. Updates to this file will be provided as enhancements become available.

References

• Alexandersson, H. and A. Moberg, 1997: Homogenization of Swedish temperature data. Part I: Homogeneity test for linear trends. Int. J. Climatol., 17, 25-34.
• Baker, D. G., 1975: Effect of observation time on mean temperature estimation. J. Appl. Meteor., 14, 471-476.
• Easterling, D. R., T. R. Karl, E.H. Mason, P. Y. Hughes, and D. P. Bowman. 1996. United States Historical Climatology Network (U.S. HCN) Monthly Temperature and Precipitation Data. ORNL/CDIAC-87, NDP-019/R3. Carbon Dioxide Information Analysis Center, Oak Ridge National Laboratory, U.S. Department of Energy, Oak Ridge, Tennessee.
• Karl, T.R., H.F. Diaz, and G. Kukla, 1988: Urbanization: its detection and effect in the United States climate record, J. Climate. 1, 1099-1123.
• Karl, T.R., and C.N. Williams Jr., 1987: An approach to adjusting climatological time series for discontinuous inhomogeneities. J. Climate Appl. Meteor., 26, 1744-1763.
• Karl, T.R., C.N. Williams, Jr., P.J. Young, and W.M. Wendland, 1986: A model to estimate the time of observation bias associated with monthly mean maximum, minimum, and mean temperature for the United States, J. Climate Appl. Meteor., 25, 145-160.
• Karl, T.R., and C.N. Williams Jr., 1987: An approach to adjusting climatological time series for discontinuous inhomogeneities. J. Climate Appl. Meteor., 26, 1744-1763.
• Lawrimore, J.H., M.J. Menne, B.E. Gleason, C.N. Williams, D.B. Wuertz, R.S. Vose, and J. Rennie, 2011: An overview of the Global Historical Climatology Network Monthly Mean Temperature Dataset, Version 3. Journal of Geophysical Research-Atmospheres, 116, D19121, doi:10.1029/2011JD016187. (PDF Version)
• Lund, R., and J. Reeves, 2002: Detection of undocumented changepoints: a revision of the two-phase regression model. J. Climate, 15, 2547-2554.
• Menne, M.J., and C.N. Williams, Jr., 2005: Detection of undocumented changepoints using multiple test statistics and composite reference series. J. Climate, 18, 4271-4286.
• Menne, M.J., and C.N. Williams, Jr., 2009: Homogenization of temperature series via pairwise comparisons. Journal of Climate, 22, 1700-1717, doi: 10.1175/2008JCLI2263.1. (PDF Version)
• Menne, M.J., C.N. Williams, and R.S. Vose, 2009: The United States Historical Climatology Network Monthly Temperature Data - Version 2. Bull. Amer. Meteor. Soc., 90, 993-1007, doi: 10.1175/2008BAMS2613.1. (PDF Version)
• Menne, M.J., C.N. Willaims, Jr., and M.A. Palecki, 2010: On the reliability of the U.S. surface temperature record. J. Geophys. Res. , doi:10.1029/2009JD013094, in press. (accepted 7 January 2010). (PDF Version)
• Quinlan, F. T., T. R. Karl, and C. N. Williams, Jr. 1987. United States Historical Climatology Network (HCN) Serial Temperature and Precipitation Data. NDP-019. Carbon Dioxide Information Analysis Center. Oak Ridge National Laboratory, U.S. Department of Energy, Oak Ridge, Tennessee.
• Venema, V. K. C., O. Mestre, E. Aguilar, I. Auer, J.A. Guijarro, P. Domonkos, G. Vertacnik, T. Szentimrey, P. Stepanek, P. Zahradnicek, J. Viarre, G. Müller-Westermeier, M. Lakatos, C.N. Williams, M.J. Menne, R. Lindau, D. Rasol, E. Rustemeier, K. Kolokythas, T. Marinova, L. Andresen, F. Acquaotta, S. Fratianni, S. Cheval, M. Klancar, M. Brunetti, C. Gruber, M. Prohom Duran, T. Likso, P. Esteban, and T. Brandsma, 2012: Benchmarking homogenization algorithms for monthly data. Climates of the Past, 8, 89-115, doi:10.5194/cp-8-89-2012. ( (PDF Version))
• Vose, R.S., S. Applequist, M.J. Menne, C.N. Williams and P.W. Thorne, 2012: An intercomparison of temperature trends in the U.S. Historical Climatology Network and recent atmospheric reanalyses. Geophyical.Research Letters, 39, L10703, doi:10.1029/2012GL051387. ( (PDF Version)
• Vose, R.S., C.N. Williams, T.C. Peterson, T.R. Karl, and D.R. Easterling, 2003: An evaluation of the time of observation bias adjustment in the US Historical Climatology Network. Geophysical research letters, 30 (20), 2046, clim3-1--3-4 doi:10.1029/2003GL018111.
• Wang, X.L., 2003: Comments on "Detection of undocumented changepoints: A revision of the two-phase model". J. Climate, 16, 3383-3385.
• Williams, C.N, M.J. Menne and J.H Lawrimorei, 2012a: Modifications to Pairwise Homogeneity Adjustment software to improve run-time efficiency. NOAA Technical Report NCDC No. GHCNM-12-01R
• Williams, C.N, M.J. Menne and J.H Lawrimore, 2012b: Modifications to Pairwise Homogeneity Adjustment software to address coding errors and improve run-time efficiency. NOAA Technical Report NCDC No. GHCNM-12-02