King's College London

Research portal

Quantifying and reducing researcher subjectivity in the generation of climate indices from documentary sources

Research output: Contribution to journalArticlepeer-review

George C. D. Adamson, David J. Nash, Stefan W. Grab

Original languageEnglish
Pages (from-to)1071-1081
Number of pages11
JournalClimate of the past
Issue number5
Published17 May 2022

Bibliographical note

Funding Information: Financial support. This research has been supported by the British Academy (Small Grant SG-40838). Publisher Copyright: © 2022 George C. D. Adamson et al.


  • Adamson et al. 2022

    Adamson_et_al._2022.pdf, 2.38 MB, application/pdf

    Uploaded date:17 May 2022

    Version:Final published version

King's Authors


The generation of index-based series of meteorological phenomena, derived from narrative descriptions of weather and climate in historical documentary sources, is a common method to reconstruct past climatic variability and effectively extend the instrumental record. This study is the first to explicitly examine the degree of inter-rater variability in producing such series, a potential source of bias in index-based analyses. Two teams of raters were asked to produce a five-category annual rainfall index series for the same dataset, consisting of transcribed narrative descriptions of meteorological variability for 11 "rain years"in nineteenth-century Lesotho, originally collected by Nash and Grab (2010). One group of raters (n = 71) was comprised of students studying for postgraduate qualifications in climatology or a related discipline; the second group (n = 6) consisted of professional meteorologists and historical climatologists working in southern Africa. Inter-rater reliability was high for both groups at r = 0.99 for the student raters and r = 0.94 for the professional raters, although ratings provided by the student group disproportionately averaged to the central value (0: normal/seasonal rains) where variability was high. Back calculation of intraclass correlation using the Spearman-Brown prediction formula showed that a target reliability of 0.9 (considered "excellent"in other published studies) could be obtained with as few as eight student raters and four professional raters. This number reduced to two when examining a subset of the professional group (n = 4) who had previously published historical climatology papers on southern Africa. We therefore conclude that variability between researchers should be considered minimal where index-based climate reconstructions are generated by trained historical climatologists working in groups of two or more.

View graph of relations

© 2020 King's College London | Strand | London WC2R 2LS | England | United Kingdom | Tel +44 (0)20 7836 5454