bootstrapping

[James B. Whitfield]

Sylvia (and TAXACOM listers)

        A few personal views on the bootstrap/reliability issue...and
thanks for the interesting distinctions from psychology -

        As Hillis and Bull (1993. Syst. Biol. 42:182-192, on bootstrapping
and confidence) described it, we should keep straight the distinctions
between repeatability, accuracy and precision.  All three are desirable in
some respects as properties of our methods, although as systematists what
we are principally after is accuracy in the long run.

with respect to bootstraps

        accuracy = the probability that a given result represents the
                   true phylogeny
        repeatability = probability that a given phylogenetic result (or
                        boostrap value) will be
                        found again using a subsequent sampling of characters
        precision =  correspondence between multiple sets of bootstrap
                     pseudoreplicates taken from the same initial sample
                     (i.e. how well does a limited sample of bootstraps reflect
                     the values that would be obtained from an infinite set of
                     pseudosamples)

        Thus we can (at least to some extent) measure both repeatability
and precision with the bootstrap, but especially easy is repeatability, as
you suggest.  Unless we actually know the "true phylogeny", as with the
experimental data of Hillis, Bull et al., we can only get at *accuracy*
through corroboration from additional data.
        I think you are right, however, that the term "reliability" is
often being used rather loosely in the literature (to involve any or all of
the above three concepts as the author wishes).
        I strongly disagree that the bootstrap is a misleading measure of
confidence in a phylogeny. I do not feel confident that the measure should
be used in an explicit  "confidence interval" context with p-values, but it
does provide an index of support with a number of well-understood
properties.  It IS an INCOMPLETE measure, in that it does not immediately
get at the accuracy issue (this is true of Bremer support and other indices
as well).
        Most importantly, the bootstrap helps to tell you how well your
data support the CURRENT hypothesis (as opposed to the ultimate true
phylogeny).  This is, in my view, the first step - further steps come from
getting additional data for corroboration.

                                                        Cheers, Jim Whitfield









>          This is in re all the discussion of "corroboration" - just
>          want to suggest that bootstrapping is not corroboration and
>          is misleading as a measure of confidence in a phylogeny.
>
>          In psychological parlance, the distinction between
>          reliability and validity is clear (or used to be back when I
>          studied psych).  Reliability is how repeatable an outcome
>          is, given the same data set.  Validity is how well a data
>          set corresponds to some outside criterion (corroboration).
>          For example "split-half" reliability correlates scores on
>          half of test items with scores on the other half.  The
>          extent of correlation indicates how well individual items
>          are measuring the *same thing*, NOT how well they are
>          measuring any *particular* thing.  No one trained in psych.
>          testing mistakes reliability for validity.
>
>          Bootstrapping is a test of internal consistency of a data
>          set and therefore a test of *reliability*, not *validity.*
>
>***************************************
>Sylvia Hope
>Dept. of Ornithology and Mammalogy    *
>California Academy of Sciences        *
>Golden Gate Park                      *
>San Francisco, CA, USA 94118          *
>                                      *
>email:  sylvia.hope at calacademy.org    *
>phone:  (415) 750-7176                *
>fax:    (415) 750-7137                *
>***************************************

J. B. Whitfield
Department of Entomology
University of Arkansas
Fayetteville, AR 72701
(501)575-2482 FAX -2452
jwhitfie at comp.uark.edu