Title: Reproducibility and validity in dietary assessment measurements
Key words: dietary assessment, validity, reproducibility, criteria, variability, subjects, reference systems, biomarkers, bias, misclassification, dietary surveys, excretion rates, protocols, gold standard, questionnaires, correlation coefficient, errors, bias, standardisation, statistical analysis
Date: Sept 2006
Author: Morgan, G
Reproducibility and validity in dietary assessment measurements
Dietary assessments, in order to deliver measurements that are reliable and repeatable, depend on the validity and reproducibility of the methods used to collate dietary information. Reproducibility is obviously a prerequisite for any test purporting to measure a dietary index but does not necessarily imply that the test itself has internal validity. On the other hand, the validity of a test, defined as its ability to measure the intended dietary variable as truthfully as possible, must also possess reproducibility in order to meet these criteria.
Variability within-subjects, between subjects and over time renders any hope of finding a gold standard for any biological system a fruitless search. The most that can be achieved is a degree of relative or congruent validity for any test of dietary measurement. Establishing validity in this context requires the use of models and methodologies which most closely mirror the state of nutritional medical knowledge at the time. The reference systems which are the product of this process may involve either the use of chemical biomarkers or dietary surveys such as 16-day weighed records to establish their validity. Both have their limitations.
Biomarkers are only able to measure a restricted range of dietary variables, and factors such as absorption, bioavailability and homeostasis lead to internal bias which compromises their validity. Dietary surveys more accurately reflect the dynamics of dietary intake but are handicapped by factors such as biased data collection, underreporting and misclassification. Practical considerations will determine which of these two methods is used: e.g. excretion rates for sodium are a more accurate assessment of salt intake than dietary surveys.
As far as it is possible, cross-validation studies should be used to assess the validity of any dietary assessment measure. Although there are no gold standards, protocols based on, for example, 16 day weighed records with the necessary checks and controls, and urinary nitrogen excretion levels with PABA controls can be used as reference systems with which to validate other protocols which may be more cost effective, practical and yield more information, e.g. food frequency questionnaires.
Correlation coefficients are the most widely used statistical method of validating the test measurements relative to the reference values. The only proviso is that, in carrying out a test validation, the variables tested should be independent of one another so that auto validation does not occur, i.e. that the data are true unbiased measures of the variable.
All biological models such as nutritional models are compromised by the issues of reproducibility and confounding factors. Confounding factors such as age, sex, weight and socio-economic group introduce variables into these models which, if not allowed for statistically, will introduce errors and bias into any measurement or analysis. Even allowing for this, subject, observer and experimental bias may seriously compromise the validity of dietary data by affecting its reproducibility. Standardisation of protocols and techniques is of key importance here, both in the laboratory and with other dietary assessment measurements such as weighed records.
It is important to recognise that both the validity and reproducibility of a test are necessary for accurate data to be elicited from any dietary survey. The two are closely related and any discrepancy is liable to lead to a loss of information or biased information. This most usually leads to the so-called ‘bias towards the null’ whereby useful information has to be discarded and is rendered unavailable for statistical analysis. This is regrettable and needs to guarded against in the design stage of any dietary assessment trial.
1. New SA. (2002) Lecture notes.
2. Nelson M. (2002) Lecture notes.
3. Margetts BM & Nelson M. (1997) Design
Concepts in Nutritional Epidemiology. 2nd Edition.