Validation of Identifying Cancer Diagnosis Based on Self-Reported Information in the Japan Nurses' Health Study.

BACKGROUND: The Japan Nurses' Health Study (JNHS) is a large-scale, nationwide prospective cohort study of female nurses. This study aimed to examine the validity of self-reported diagnosis of cancer among the JNHS cohort members (N=15,019).
METHODS: For women who reported any diagnosis of five cancers (stomach, colorectal, liver, lung and thyroid) in the biennial follow-up surveys, an additional outcome survey, medical facility survey, and confirmation of death certificate (DC) were conducted. The JNHS Validation Study Committee (referred to as "the committee") made a final decision on the reported outcomes. To examine the validity of self-reported diagnosis of cancer, the positive predictive value (PPV) was calculated using the committee's decision as the gold standard. To examine the validity of the committee's decision based on self-reports and DCs, PPV was calculated using physician-reported information as the gold standard.
RESULTS: The PPV of self-reported diagnosis in the biennial follow-up surveys was 77.8% for stomach, 66.2% for colorectal, 41.7% for liver, 60.2% for lung, and 64.6% for thyroid cancer. The corresponding PPVs in the additional outcome survey were 96.2%, 80.7%, 62.5%, 82.5%, and 96.9%, respectively. The PPV of the committee's decision was 100% for stomach, 87.5% for colorectal, 94.7% for lung, and 100% for thyroid cancer (data not available for liver cancer). The proportion of DC-only cases among committee-defined cases was below 10% for all cancers except liver cancer (28.6%).
CONCLUSIONS: The validity of identifying cancer diagnosis based on self-reported information in the JNHS was favorable for stomach, colorectal, lung and thyroid cancer.

Introduction

Identifying and validating disease outcomes are some of the biggest challenges in epidemiological research. One strategy is to use information provided in physician records and to conduct periodic examinations, which is typically adopted in hospital-based studies or community-based studies with good access to healthcare data (Boring III et al., 1996). Another way is through linkage with population-based disease registries, a strategy that is widely used for large-scale cohort studies on cancer and requires high-quality population-based cancer registry systems in the study area (Calle et al., 2002; Inoue et al., 2011; Sado et al., 2017). Alternatively, large-scale cohort studies often use self-reported disease history as a source of information due to time and cost constraints. A large number of validation studies have compared self-reported outcome information with data from objective sources, such as disease registries and medical records. For cancer or related diseases, previous findings vary by a number of factors such as study population, disease outcome and time since cancer diagnosis (Colditz et al., 1986; Kriegsman et al., 1996; Bergmann et al., 1998; Yoshinaga et al., 2001; Manjer et al., 2004; Navarro et al., 2006; Inoue et al., 2011; Cho et al., 2017; Patel et al., 2017; Cowdery et al., 2020). Validation studies conducted in Japanese cohorts from the general population have shown that sensitivity, calculated using registered cancer as the gold standard, is as low as 36% for any cancer (ranging from 8% for liver to 81% for female breast cancers), (Yoshinaga et al., 2001), although this improved to 53% (ranging from 34% for liver to 82% for female breast cancers) in a more recent cohort (Inoue et al., 2011). By contrast, the U.S. Nurses’ Health Study, a large-scale cohort of medical professionals, showed that over 90% of self-reported cancer cases were confirmed by histopathological records (Colditz et al., 1986). The validity of self-reports for cancer identification among Japanese medical professionals has not been studied.

The Japan Nurses’ Health Study (JNHS) is a large-scale, nationwide prospective cohort study of female nurses (Fujita et al., 2007; Hayashi et al., 2007). It was established by modeling the U.S. Nurses’ Health Study. A total of approximately 15,000 participants were recruited for the follow-up cohort from all 47 prefectures in Japan. One of the study’s primary endpoints is the incidence of cancers. The present study aimed to examine the validity of self-reported diagnosis of cancer among female nurses in the JNHS.

Materials and Methods

Study design

This validation study was conducted using postal questionnaires sent to the participants and medical facilities and information obtained from death certificates.

Setting and participants

The JNHS is an ongoing prospective cohort study of female nurses (Hayashi et al., 2007). The baseline data were collected between 2001 and 2007 from 49,927 female nurses aged 25 years or older using a self-administered questionnaire, of which 15,019 provided written informed consent to be followed-up for 10 years. Prior to this nationwide recruitment, a pilot cohort was set up in Gunma prefecture in 1999, which comprised 698 female nurses (called GNHS: Gunma Nurses’ Health Study) (Maeno et al., 2005; Kato et al., 2012). The present study population was pooled from the two follow-up cohorts.

Outcomes

The JNHS and GNHS target a wide range of diseases and symptoms, with particular focus on reproductive and health outcomes common to women (Fujita et al., 2007). The present study covered cancer of the stomach, colon/rectum, liver, lung and thyroid. Other outcomes including cancer of the breast and female genital organs will be examined elsewhere.

Baseline and follow-up surveys

The baseline questionnaire was distributed with an invitation letter; respondents completed the questionnaire and mailed it back to the JNHS office with a signed form providing informed consent. The questionnaire included items on lifestyle, drug use, physical and reproductive condition, and history of disease. Participants were followed up for 10 years. Follow-up questionnaires inquiring about changes in lifestyle, other exposure items, and incidence of disease were sent every 2 years. For the baseline and follow-up surveys, except the 2-year follow-up, history or incidence of each disease was asked using the question “Have you ever been diagnosed with a disease?” This was followed by a question about the respondent’s age at diagnosis. In the 2-year follow-up survey, the question “Have you been diagnosed with a disease during the past two years?” was used, followed by a question inquiring about the year and month of the diagnosis. Details of the survey design are provided in our previous report (Hayashi et al., 2007).

Validation

To validate of self-reported diagnosis of diseases identified in the biennial follow-up surveys, an additional outcome survey, medical facility survey, and confirmation of death certificate were conducted (Figure 1). Women eligible for these validation checks were cohort members who completed at least one of the routine biennial follow-up surveys (N=15,019 for JNHS and N=698 for GNHS) and who did not report a history of diagnosis of each outcome at baseline. First, for each outcome, we defined the self-reported answer to the question about disease diagnosis in the biennial follow-up surveys as “definition 1”. Second, the additional outcome survey questionnaire was sent to participants who answered “yes” to definition 1. This additional survey was conducted separately for each outcome (i.e. each type of cancer in the present study), and included the following question items (examples for stomach cancer):

Figure 1

Flow Chart of the Outcome Surveys. The flow of the outcome surveys in the Japan Nurses’ Health Study and the comparison diagrams for this validation study are illustrated

“Have you ever been diagnosed with stomach cancer by a medical doctor?” (yes or no)

“When were you diagnosed?” (year and month)

“How were you diagnosed?” (answer chosen from a list: screening, abdominal discomfort, stomach upset, etc.)

“Which test(s) did you undergo for the diagnosis?” (answer chosen from a list: upper gastrointestinal endoscopy, X-ray, Helicobacter pylori test, etc.)

“Were you hospitalized for the diagnosis and/or treatment?” (yes or no)

“What type of treatment(s) did you receive?” (answer chosen from a list: gastric resection with open surgery, laparoscopic gastrectomy, endoscopic surgery, etc.)

The self-reported diagnosis indicated in the additional survey was defined as “definition 2”.

Third, the medical facility survey was conducted among participants who provided written informed consent to participate in this survey and the contact information of the medical doctor in charge of their case. A questionnaire was sent to the provided postal address of the medical facility inquiring about the following items regarding the patient (examples for stomach cancer): date of diagnosis, primary / recurrent / multiple cancer, smoking history, family history of stomach cancer, height and weight, modality of diagnosis (histology, cytology, imaging, clinical, and/or other), primary site (upper, middle, lower, esophagogastric junction), tumor size, invasion into lymph nodes or veins, clinical and pathological TNM (Tumor, lymph Nodes, and Metastasis) classifications, site of metastasis, if any. The physician-reported diagnosis provided in the medical facility survey was defined as “definition 3”.

Death certificate confirmation is routinely conducted in the JNHS. Individual records of death certificates were obtained from the Office of Vital Statistics of Japan under official approval, and linked to the personal identification data of the cohort. This linkage was performed using date of birth, date of death, and municipality of residence data as key variables.

Combining the information obtained from the four sources (definitions 1-3 and death certificate confirmation), the JNHS Validation Study Committee (referred to as “the committee”) made a final decision on the incidence of the reported outcome (definition 4). In cases where there was no physician-reported diagnosis (definition 3), the committee made their decision (definition 4’) based on three information sources (definitions 1-2 and death certificate confirmation). The majority of the incident cases were decided upon definition 4’.

The primary aim of the present study was to examine the validity of self-reported diagnosis of cancer using the following indices (Figure 1):

A. Positive predictive value (PPV) of definition 1 using definition 4 as the gold standard

B. PPV of definition 2 using definition 4 as the gold standard

C. Proportion of death certificate-only cases among diagnosed cases defined by the committee’s overall decision (definition 4) (i.e. potentially false-negative cases in the self-reporting scheme)

The secondary aim was to examine the validity of the committee’s overall decision on cancer diagnosis made based on self-reported diagnosis and death certificate confirmation (definition 4’) using physician-reported information as the gold standard (definition 3). Specifically, the following indices were calculated:

D-1. PPV of definition 4’ using definition 3 as the gold standard

D-2. Negative predictive value (NPV) of definition 4’ using definition 3 as the gold standard (calculated only for cases identified as “positive” in self-reported sources)

The GNHS study protocol was approved by the institutional review board of Gunma University, Japan (Approval No. 3, 1999), and the JNHS study protocol was approved by the institutional review board of Gunma University, Japan (Approval No. 101, 2001) and by the ethics review board of Japan’s National Institute of Public Health, Japan (Approval No. 03007, 2003). Informed consent was obtained from all individual participants included in the study.

Results

Table 1 shows the basic characteristics of the participants of the JNHS and its pilot study, the GNHS. Median age of the cohort members was 41 and 37 years old in the JNHS and GNHS, respectively. Women aged 30-49 years accounted for 77% of the JNHS and 59% of the GNHS cohort, and the median follow-up period was 14 and 18 years, respectively.

Table 1

Basic Characteristics of the Japan and Gunma Nurses' Health Studies

	JNHS		GNHS
Total number	15,019		698
Age at baseline
Mean (SD)	41.8	(8.2)	37.3	(10.1)
Median (Q1-Q3)	41	(35-48)	37	(28-45)
<30 years	498	(3.3%)	194	(27.8%)
30-34 years	2859	(19.0%)	96	(13.8%)
35-39 years	3061	(20.4%)	115	(16.5%)
40-44 years	3026	(20.1%)	107	(15.3%)
45-49 years	2672	(17.8%)	95	(13.6%)
50-54 years	1959	(13.0%)	53	(7.6%)
55-59 years	764	(5.1%)	33	(4.7%)
60-64 years	138	(0.9%)	5	(0.7%)
65+ years	42	(0.3%)	0	(0.0%)
Follow-up years
Mean (SD)	11.9	(4.4)	14.3	(5.3)
Median (Q1-Q3)	14	(10-16)	18	(10-18)

JNHS, Japan Nurses' Health Study; GNHS, Gunma Nurses' Health Study; SD, standard deviation; Q1, 25 percentile; Q3, 75 percentile

Table 2 shows the results of the outcome surveys. The percentage of potential incident cases identified through the biennial follow-up surveys (among all eligible women) was 0.5% for stomach, 0.9% for colorectal, 0.1% for liver, 0.6% for lung, and 0.7% for thyroid cancer. After excluding cases due to withdraw, loss to follow-up, and/or death, over 90% of potential incident cases were subjected to the additional outcome survey. We did not calculate response rates to the biennial follow-up surveys because the cohort members provided responses to at least one of the routine biennial follow-up surveys.

Table 2

Results of the Outcome Surveys in the Japan Nurses' Health Study

Cancer site	Stomach		Colorectal		Liver		Lung		Thyroid
Biennial follow-up surveys (definition 1)
Eligible womena	14,984		14,984		15,016		15,010		14,981
Self-reported diagnosis = yes (potential incident cases) [% among eligible women]	72	[0.5%]	136	[0.9%]	12	[0.1%]	83	[0.6%]	99	[0.7%]
Withdraw/lost of follow-up/death	2		5		1		4		6
Proceed to the additional outcome surveys [% among potential incident cases]	70	[97.2%]	131	[96.3%]	11	[91.7%]	79	[95.2%]	93	[93.9%]
Additional outcome survey (definition 2)
Eligible women	70		131		11		79		93
Responded [%]c	64	[91.4%]	117	[89.3%]	9	[81.8%]	65	[82.3%]	76	[81.7%]
Self-reported diagnosis = yes [% among respondents]	53	[82.8%]	109	[93.2%]	8	[88.9%]	57	[87.7%]	64	[84.2%]
Self-reported diagnosis = no	9		7		1		8		11
Self-reported diagnosis = unknown	2		1		0		0		1
Agreed on the medical facility survey [% among respondents]	27	[50.9%]	49	[45.0%]	2	[25.0%]	29	[50.9%]	37	[57.8%]
Medical facility survey (definition 3)
Eligible women	27		49		2		29		37
Responded [%]c	20	[74.1%]	33	[67.3%]	0	[0.0%]	22	[75.9%]	32	[86.5%]
Physician-reported diagnosis = yes [% among respondents]	20	[100.0%]	32	[97.0%]	0	(NA)	19	[86.4%]	31	[96.9%]
Death certificate
Death from cancer during the follow-upb	7		6		2		10		2
Self-reported diagnosis at follow-up = yes	5		3		0		6		2
Death certificate only	2		3		2		4		0
Overall decision by Validation Study Committee (definition 4)
Eligible women (=potential incident cases)	72		136		12		83		99
Diagnosis = yes [% among potential cases]	58	[80.6%]	93	[68.4%]	7	[58.3%]	54	[65.1%]	64	[64.6%]
[% among all eligible women]		[0.4%]		[0.6%]		[0.05%]		[0.4%]		[0.4%]

JNHS: Japan Nurses' Health Study, NA: not applicable; a. Eligible women were cohort members without a self-reported diagnosis of each disease at baseline (total cohor menbers: N=15,019); b, As of Apr. 30, 2019; c, As of Jul. 31, 2019

In the additional outcome survey, the response rate ranged from 81.7% to 91.4%. Almost all respondents re-reported their diagnosis (82.8% to 93.2%). Among the respondents of the additional outcome survey, half or less agreed to participate in the medical facility survey (50.9% for stomach, 45.0% for colorectal, 25.0% for liver, 50.9% for lung, and 57.8% for thyroid cancer). The medical facility survey revealed good agreement between self-reported and physician-reported diagnoses (100.0% for stomach, 97.0% for colorectal, 86.4% for lung, and 96.9% for thyroid cancer) (there were no respondents for liver cancer). The number of deaths identified through death certificates (death certificate-only cases) was 7 (2) for stomach, 6 (3) for colorectal, 2 (2) for liver, 10 (4) for lung, and 2 (0) for thyroid cancer. Finally, the committee confirmed the diagnosis of 58 stomach, 93 colorectal, 7 liver, 54 lung, and 64 thyroid cancer cases, corresponding to 80.6%, 68.4%, 58.3%, 65.1%, 64.6% of the potential incident cases in the biennial follow-up surveys, respectively. The percentage of all eligible women confirmed to have a diagnosis by the committee was 0.4% for stomach, 0.6% for colorectal, 0.05% for liver, 0.4% for lung, and 0.4% for thyroid cancer.

Table 3 shows the results of the PPV and other validation indices. The PPV of self-reported diagnosis in the biennial follow-up surveys was highest for stomach (77.8%) and lowest for liver (41.7%), and intermediate for colorectal (66.2%), lung (60.2%), and thyroid (64.6%) cancer. The PPV of self-reported diagnosis in the additional outcome survey was also lowest for liver cancer (62.5%), while the highest values were observed for stomach (96.2%) and thyroid (96.9%) cancer. The proportion of death certificate-only cases among diagnosed cases defined by the committee’s overall decision was less than 10% for all cancers except liver cancer (28.6%). The PPV of the overall decision compared to the physician-reported diagnosis was as high as 87.5% for colorectal cancer and even higher for stomach (100%), lung (94.7%), and thyroid (100%) cancer (the PPV could not be calculated for liver cancer because there were no respondents). The NPV of the overall decision compared to the physician-reported diagnosis was 100% for colorectal and lung cancer and 0% for thyroid cancer, although the number of cases available to calculate this index was very small. Corresponding results from the GNHS are shown in Supplementary Tables 1-2.

Table 3

Positive Predictive Values and Other Validation Idicies for the Japanese Nurses' Health Study

		Stomach		Colorectal		Liver		Lung		Thyroid
A	PPV of self-reported diagnosis in the biennial follow-up survey (definition 1 vs. definition 4)	77.8%	(56/72)	66.2%	(90/136)	41.7%	(5/12)	60.2%	(50/83)	64.6%	(64/99)
B	PPV of self-reported diagnosis in the additional outcome survey (definition 2 vs. definition 4)	96.2%	(51/53)	80.7%	(88/109)	62.5%	(5/8)	82.5%	(47/57)	96.9%	(62/64)
C	Proportion of death certificate only cases among diagnosed cases defined by the overall decision (definition 4)	3.4%	(2/58)	3.2%	(3/93)	28.6%	(2/7)	7.4%	(4/54)	0.0%	(0/64)
D-1	PPV of overall decision in comparison with physician-reported diagnosis (definition 4’ vs. definition 3)	100.0%	(20/20)	87.5%	(28/32)	NA	(0/0)	94.7%	(18/19)	100.0%	(31/31)
D-2	NPV of overall decision in comparison with physician-reported diagnosisa (definition 4’ vs. definition 3)	NA	(0/0)	100.0%	(1/1)	NA	(0/0)	100.0%	(3/3)	0.0%	(0/1)

PPV, positive predictive value; NPV, negative predictive value; NA, not applicable; a, Calculated only for the cases identified as “positive” in self-reported information

Discussion

The present study examined the validity of self-reported diagnosis of cancer among female nurses in the JNHS by combining information from the participants medical facilities, and death certificates. We observed favorable PPVs for stomach, colorectal, lung, and thyroid cancers (approximately 60-80% using the committee’s overall decision based on the additional outcome survey and death certificate confirmation as the gold standard). For these outcomes, death certificate-only cases, which can be considered false-negative cases, were as low as 10%, and PPVs calculated for a sub-sample using information obtained from medical facilities as the gold standard were over 87%. Our results indicate that the validity of identifying cancer diagnosis based on self-reported information in the JNHS was favorable for stomach, colorectal, lung, and thyroid cancer.

The PPVs of self-reported diagnosis of cancer in this study were higher than those reported in a population-based cohort study of community-dwelling populations. In a validation study of the Japan Public Health Center-based Prospective Study (JPHC Study), PPVs calculated by comparison with cancer registry data were 52% for stomach, 47% for colorectal, 46% for lung, 58% for female breast cancer.(Inoue et al., 2011) This discrepancy in PPV values may be due to differences in the cohort profiles: our participants were female nurses, who may have provided more accurate responses about their health conditions than women in the general population. Consistent with our hypothesis, the original U.S. Nurses’ Health Study demonstrated the accuracy of self-reported outcomes from female nurses.(Colditz et al., 1986)

The validity of self-reported diagnosis of liver cancer was relatively low in our study: the PPV (vs. the committee’s overall decision) and proportion of death certificate-only cases were approximately 40% and 30%, respectively (the PPV vs. information from medical facilities could not be calculated). This result was consistent with those of previous studies. In the JPHC Study cited above, the PPV was 31% for liver cancer.(Inoue et al., 2011) An earlier validation study from the same JPHC group also reported a very low sensitivity for self-reported liver cancer.(Yoshinaga et al., 2001) The inaccuracy of self-reported liver cancer may be due to the smaller number of cases (liver cancer: 12 cases, colorectal cancer: 136 cases in the present study). Unfavorable prognosis of this cancer may also be a contributing factor, which was reflected in the higher proportion of death certificate-only cases (29%).

The biennial follow-up surveys, additional outcome survey, medical facility survey, and death certificate confirmation are conducted as part of the JNHS’s routine follow-up process. In the present study, PPVs of self-reports using the committee’s overall decision as the gold standard were higher in the additional outcome survey than in the biennial follow-up surveys (B vs. A in Table 3). This is a matter of course because the overall decision was based on detailed clinical information obtained from the additional outcome survey. However, from a quality assurance perspective on the routine follow-up, this additional survey, which received response rates of over 80%, is useful for improving the accuracy of outcome information in our study. Additionally, we identified 3-7% of cancer cases (other than liver cancer) from death certificates, suggesting that death certificate confirmation is useful for supplementing cases that cannot be identified by self-report.

Use of physician records data is an ideal way to confirm self-reported disease outcomes. While this is the main strategy adopted by the U.S. Nurses’ Health Study, in which they contact each patient to obtain permission and the patients’ hospitals to obtain copies of pertinent records,(Colditz and Hankinson, 2005) other studies obtained medical records only from a sub-sample of cases.(Hunter et al., 1992) Studies examining cancer outcomes have generally been successful at obtaining medical records from a high percentage of reported cases (e.g. 93-94%).(Colditz et al., 1995; Feskanich et al., 2007) Although the JNHS adopts the same method, the proportion of self-reported cases providing their hospitals’ contact information has been much lower; the proportion ranged from 25.0% (liver cancer) to 57.8% (thyroid cancer) (Table 2). Furthermore, the response rate from medical facilities was also low, ranging from 0.0% (liver cancer) to 86.5% (thyroid cancer). Therefore, although there was good agreement between self-report and medical facilities data (86.4% or higher), this method of confirmation was only feasible for a portion of cases. While it may be possible to increase the proportion of medical facility-confirmed cases by calling the hospitals or using other reminder methods, this requires financial and human resources. Based on the fact that we found good agreement between self-report and medical facilities data, we think that contacting medical facilities for a sub-sample of self-reported cases (i.e. participants who provided their hospitals’ contact information) is a feasible and valid method for the JNHS study.

One of the strengths of the present study is the high response rates in the outcome surveys at least from the participants. We were also able to obtain comprehensive information from death certificates. Nevertheless, there are several limitations in our study. First, we did not examine the sensitivity or specificity of the self-reported outcomes. To do this, we would need high-quality population-based cancer registry data from the study areas. Although population-based cancer registry data were available in all 47 prefectures for cancer cases diagnosed in 2012 and later, not all of them met the standard quality criteria.(Hori et al., 2015) Because our cohort members were recruited from all over Japan, it was difficult to obtain high-quality cancer registry data for all members. The National Cancer Registry was started in 2016 in Japan based on the Act on Promotion of Cancer Registries.(Matsuda and Sobue, 2015; Tanaka and Matsuda, 2015) This new data source allows us to calculate the sensitivity and specificity for cases diagnosed in 2016 and later.

Second, some of the information sources were not independent of each other. Specifically, the decision of the committee, one of the gold standards was based on the results of the follow-up and outcome surveys, which were also used for comparisons. However, we also conducted comparisons between independent sources for a sub-sample, namely, the biennial follow-up and additional outcome surveys and information from the medical facilities survey.

In conclusion, the validity of identifying cancer diagnosis based on self-reported information in the JNHS was favorable for stomach, colorectal, lung and thyroid cancers.

Author Contribution Statement

KK wrote the manuscript, KH supervised the whole study, YI, NM, and KN collected and analyzed the data, and YT, MM, and TH examined the clinical data.

Availability of Data

The tabulated data that support the findings of this study are available from the last author (KH) upon reasonable request. The raw data are not publicly available because providing such data was not included in the informed consent.

Funding Statement

This work was supported by the Japan Society for the Promotion of Science [grant number 18H04069 to KH] and the Japan Agency for Medical Research and Development [grant number 21gk0210027h0001 to KH].

Potential Conflict of Interest

KK, YI, and KN received a JMWH Bayer Grant (1 million JPY) via the Japan Society for Menopause and Women’s Health from Sep. 2017 to Aug. 2019, from Sep. 2019 to Aug. 2020, and from Sep. 2020 to Aug. 2021, respectively. None for the other authors.