The use of subjective rankings in clinical trials with an application to cardiovascular disease.

Evaluating a clinical trial can be problematic if the studied treatments affect patients in many ways. A possible method for evaluating treatments is to have raters rank all the patients' trial experiences and then test whether the distribution of ranks differ between treatments. Before one can advocate such a procedure, however, one would like to be assured that raters agree fairly well with one another. As a first step in examining whether raters tend to agree, we conducted a small study with 20 raters evaluating 43 trial experiences from an imaginary cardiovascular clinical trial. Raters showed a high degree of consensus. Moreover, the average ranks agreed fairly well with two quantitative ranking rules. On the other hand, the average ranks did not agree very well with weightings usually selected in cardiovascular trials. These results suggest ranking may be a feasible approach to analysing certain clinical trials with multiple outcomes.