There has been a long argument over whether schizophrenia is a neurodegenerative disorder associated with progressive cognitive impairment. Given high heritability of schizophrenia, ascertaning if genetic susceptibility to schizophrenia is also associated with cognitive decline in healthy people would support the view that schizophrenia leads to an accelerated cognitive decline. Using the population representative sample of 6817 adults aged >50 years from the English Longitudinal Study of Ageing, we investigated associations between the biennial rate of decline in cognitive ability and the schizophrenia polygenic score (SZ-PGS) during the 10-year follow-up period. SZ-PGS was calculated based on summary statistics from the Schizophrenia Working Group of the Psychiatric Genomics Consortium. Cognition was measured sequentially across four time points using verbal memory and semantic fluency tests. The average baseline verbal memory was 10.4 (SD = 3.4) and semantic fluency was 20.7 (SD = 6.3). One standard deviation (1-SD) increase in SZ-PGS was associated with lower baseline semantic fluency (β = −0.25, 95%CI = −0.40 to −0.10, p = 0.002); this association was significant in men (β = −0.36, 95%CI = −0.59 to −0.12, p = 0.003) and in those who were aged 60–69 years old (β = −0.32, 95%CI = −0.58 to −0.05, p = 0.019). Similarly, 1-SD increase in SZ-PGS was associated with lower verbal memory score at baseline in men only (β = −0.12, 95%CI = −0.23 to −0.01, p = 0.040). However, SZ-PGS was not associated with a greater rate of decline in these cognitive domains during the 10-year follow-up. Our findings highlight that while genetic susceptibility to schizophrenia conveys developmental cognitive deficit, it is not associated with an ongoing cognitive decline, at least in later life. These results do not support the neo-Kraepelinian notion of schizophrenia as a genetically determined progressively deteriorating brain disease.