Ubiquity of synonymity: almost all large binary trees are not uniquely identified by their spectra or their immanantal polynomials.

BACKGROUND: There are several common ways to encode a tree as a matrix, such as the adjacency matrix, the Laplacian matrix (that is, the infinitesimal generator of the natural random walk), and the matrix of pairwise distances between leaves. Such representations involve a specific labeling of the vertices or at least the leaves, and so it is natural to attempt to identify trees by some feature of the associated matrices that is invariant under relabeling. An obvious candidate is the spectrum of eigenvalues (or, equivalently, the characteristic polynomial).
RESULTS: We show for any of these choices of matrix that the fraction of binary trees with a unique spectrum goes to zero as the number of leaves goes to infinity. We investigate the rate of convergence of the above fraction to zero using numerical methods. For the adjacency and Laplacian matrices, we show that the a priori more informative immanantal polynomials have no greater power to distinguish between trees.
CONCLUSION: Our results show that a generic large binary tree is highly unlikely to be identified uniquely by common spectral invariants.

Background

Tree shape theory furnishes numerical statistics about the structure of a tree [1,2]. (Because we are interested in applications of tree statistics to trees that describe the structure of branching events in evolutionary histories, we will, for convenience, always take the term tree without any qualifiers to mean a rooted, binary tree without any branch length information or labeling of the vertices.) Such statistics have two related uses. Firstly, they can be used in an attempt to tell whether two trees are actually the same and, secondly, they can be used to indicate the degree of similarity between two trees with respect to some criterion.

Examples of the latter use are the testing of hypotheses about macroevolutionary processes and the detection of bias in phylogenetic reconstruction. Historically, numerical statistics for such purposes have attempted to capture the notion of the balance of a tree, which is the degree to which daughter subtrees are the same size. The balance is typically measured by ad-hoc formulae that are often selected for statistical power to distinguish between two different distributions on trees [3,4]. In previous work we investigated the possibility of describing the shape of the tree using a list of numbers rather than just a single number [5,6].

A mathematically "canonical" approach to finding a list of such numbers is to use information derived from matrix representations of the trees. We first describe the matrix representations of a tree that we will consider.

In algebraic graph theory [7], the basic matrix associated to a graph is the adjacency matrix A(G), whose ijth entry is one if i and j are connected by an edge, and zero otherwise. From a probabilistic point of view, the more natural matrix to associate with a graph is the Laplacian matrix L(G), which is the infinitesimal generator of the natural random walk on the graph and is given by A(G) - D(G), where D(G) is the diagonal matrix of vertex degrees. It is clear that a graph can be recovered from either its adjacency of Laplacian matrix. Some authors, such as [8], define the Laplacian to be D(G)-1/2L(G)D(G)1/2. Note that this difference is not relevant if one is only considering characteristics of the matrix L such as eigenvalues that are invariant under similarity transformations.

Readers in the phylogenetics community may be more familiar with the pairwise distance matrix [1,9]. The distance matrix P given a leaf-labeling 1, . . ., n has as its ijth entry the length of the path between leaf i and leaf j. Any leaf-labeled tree is uniquely determined by its distance matrix. These matrices have also been extensively studied as discrete metric spaces [10,11].

The definition of the adjacency and Laplacian matrices requires a numbering of the vertices, while the definition of the distance matrix requires a numbering of the leaves. Because we are considering unlabeled trees (that is, we identify trees that are equivalent in the usual sense of graph-theoretic isomorphism), we are only interested in tree statistics that are invariant under renumbering. Algebraically, this means that we are only interested in features of the associated matrix that are unaffected by similarity transformations via a permutation matrix. The most obvious such statistics are the eigenvalues.

The adjacency and Laplacian matrices and their eigenvalues are familiar objects in the area of spectral graph theory [7,8,12]. The eigenvalues of the adjacency matrix tend to contain combinatorial information about the graph, such as bounds on the chromatic number. The eigenvalues of the Laplacian give information of a more geometric flavor, such as the equivalent of the surface area to volume ratio of subgraphs of a graph. As well as having connections to the theory of random walks on graphs, the Laplacian eigenvalues can be used to define the expander graphs, an important class of graphs that have applications in coding theory. Therefore, it would not be too surprising if the these eigenvalues were a convenient way to summarize information about a tree, thus giving a nice collection of tree statistics.

Similarly, it seems plausible that the eigenvalues of the pairwise distance matrix could contain quite a lot of information about the tree that could be used to compare trees. Moreover, although the distance matrix formally contains the same information as the adjacency or Laplacian matrices, the transformation that takes the distance matrix to one of the other two is distinctly non-linear, and hence there is no reason to believe that there is any simple connection between the corresponding eigenvalues.

We demonstrate below that not only do there exist pairs of trees that have the same spectrum as another tree for the adjacency, Laplacian, and distance matrices, but that this is the rule rather than the exception as the trees become large, in the sense that the fraction of trees with a given number of leaves that have a unique adjacency, Laplacian, or distance spectrum goes to zero as the number of leaves goes to infinity.

The basic methodology that we use to prove this result was first established in [13] and developed in [14] for general (that is, not necessarily bifurcating) graph-theoretic trees in the case of the adjacency and Laplacian matrices. The present paper provides the first results of this type concerning rooted bifurcating trees, as well as the first examination of such results for the pairwise distance matrix. The key idea is to establish that certain pairs of trees T1 and T2 have the following exchange property for a given matrix representation: that exchanging T1 for T2 as subtrees of a given tree does not change the spectrum for that matrix representation. This is a stricter requirement than simply having the same spectrum (Figure 1). It then becomes a matter of showing that the number of trees with a given number of leaves is asymptotically of larger order than the number of trees with the same number of leaves that don't have a particular subtree. For this we build on the generating function argument used in [15] for asymptotic estimates of the number of unlabeled rooted bifurcating trees (see the section Asymptotic numbers of trees).

Figure 1

Pairs of trees with similar algebraic properties. Figure (a) shows the smallest pair of rooted binary trees with the same adjacency and Laplacian spectrum; these trees do not have the corresponding exchange property. Figure (b) shows two trees with the exchange property for the adjacency, Laplacian, and pairwise distance matrices.

One possible explanation for this phenomenon is that two diagonalizable matrices have the same spectrum if they are similar via an arbitrary similarity transformation rather than just via a permutation transformation, and this suggests considering features of a matrix that are invariant under permutation similarities but not more general ones. We will now describe a feature of a matrix, its immanantal polynomial, that has this property.

The immanant is a generalization of the determinant. Recall that the determinant of a matrix A = (a) is given by

where the sum is over the symmetric group of permutations of {1, 2,..., n} and sgn(σ) is the sign of the permutation σ.

The function sgn is a particular example of a character of an irreducible representation of the symmetric group. It would take us too far afield to define these notions here, but excellent treatments may be found in [16-19]. We note, however, that the irreducible characters are constant on the conjugacy classes of the symmetric group (recall that two permutations belong to the same conjugacy class if and only if they have the same cycle structure) and they form a basis for the vector space of functions with this property (the class functions).

Our use of characters is simply to define the immanant

of a matrix A for the irreducible character χ. A discussion of immanants may be found in [20,21]. The immanantal polynomial for a character χ of a matrix is the corresponding generalization of the characteristic polynomial; that is, it is the polynomial . Because the characters are class functions, an immanantal polynomial is invariant under similarity by permutation matrices, but it will not typically be invariant under more general similarities.

Unfortunately, as we show in Lemma 2, for either the adjacency or Laplacian matrix the following two conditions on a pair of trees are equivalent:

• the spectra are equal,

• the immanantal polynomials are equal for all irreducible characters.

Consequently, the immanantal polynomials for the adjacency and Laplacian matrices provide no more distinguishing power than the spectra and, in particular, a vanishing fraction of large trees have a unique set of immanantal polynomials for these matrices. We do not know if the same fact is true for the immanantal polynomials of the distance matrix.

Our main result is thus the following.

Theorem 1. Let tbe the number of trees with n leaves. For either the adjacency, Laplacian, or pairwise distance matrix, let lbe the number of trees with n leaves that do not share their spectrum with another tree. Then, the fraction l/tgoes to zero as n goes to infinity. For the adjacency and Laplacian matrices, the same result holds if we replace the spectrum by the complete set of immanantal polynomials.

The rate of convergence of the fraction in Theorem 1 is also of interest. If it is extremely slow then the existence of trees with shared spectra may not be practically relevant for the construction of informative tree shape statistics. We investigate this matter numerically towards the end of the paper.

Several other research groups have investigated problems that are related to, but different from, those investigated here. Steyaert and Flajolet [22] investigate the occurrences of subtrees in the case of "planar" trees, i.e. trees that are equipped with an order of subtrees at every internal node. These planar trees are substantially easier to analyze: for example, there is a nice closed form generating function for the numbers of such trees (the Catalan numbers). In contrast, the generating function for the number of trees considered here is only given as the solution of a functional equation and there is no closed form expression for the numbers of such trees. Graham and Lovasz [23] investigate the spectra of distance matrices of trees, but their distance matrices are defined in terms of vertex-to-vertex distances, rather than the leaf-to-leaf definition. The leaf-to-leaf distance matrix is a principal sub-matrix of the vertex-to-vertex one, and there are interlacing relations between the spectrum of a matrix and one of its principal submatrices (2). However it is not a priori the case that if two matrices have the same spectrum, then the principal submatrices with the same rows and columns will also have the same spectrum. In a similar vein, the vertex-to-vertex distance matrix can be constructed from the leaf-to-leaf distance, but the construction involves considering whether certain linear inequalities hold and so it isn't a procedure that will, a priori, transform spectra in a simple way.

More recently, Bhamidi, Evans, and Sen [24] have proven that, subject to weak general conditions, many ensembles of random trees have the property that, with probability converging to one as the number of leaves goes to infinity, a realization shares its spectrum with another tree. Their conditions are easiest to check when the ensemble can be embedded in a general continuous-time branching process where individuals give birth to a possibly random number of offspring at the arrival times of a point process up to a possibly infinite death time, and those offspring go on to behave as independent copies of their parent. This particular framework covers examples such as random recursive trees, linear preferential attachment trees, uniform rooted unordered labeled trees, and Yule trees. However, we have been unable to embed the ensemble considered here in a suitable continuous-time branching process or otherwise check the general conditions of [24].

The computer code used in this paper was written in OCaml[25] and has been made available at http://github.com/matsen/ubiquity_synonymity, along with the results produced by this code.

Algebraic preliminaries concerning spectra and immanantal polynomials

Equality of adjacency and Laplacian spectra implies equality of immanantal polynomials

In order to prove results for the adjacency and Laplacian matrices simultaneously, we define for a tree T and arbitrary real numbers y and z the generalized Laplacian (recall that A(T) is the adjacency matrix and D(T) is the diagonal matrix of vertex degrees). We define the corresponding generalized Laplacian immanantal polynomial of the tree T with r vertices to be

for an irreducible character χ of the symmetric group S.

The generalized Laplacian immanantal polynomial can be computed in a simple combinatorial fashion as follows. Define a k-matching to be a set of k pairwise disjoint edges of the tree (that is, a set of edges such that no two share a common vertex). Let M(T) denote the set of k-matchings on the tree T. We think of an edge as a pair of vertices, so when we use the notation for a vertex i and a matching p, we mean that i is not one of the ends of any edge in p. Let Cdenote the conjugacy class of the symmetric group Sconsisting of permutations that are the product of k disjoint transpositions, and write χ(C) for the common value of the character χ on such permutations. The following lemma appears in [14] and is included for completeness.

Lemma 1. The generalized Laplacian immanantal polynomial of the tree T for the character χ is given by

where d(T) is the degree of vertex i in the tree T.

Proof. Set so that the generalized Laplacian immanantal polynomial is

The matrix entries mare zero unless i = j or there is an edge between i and j. If the permutation σ has a cycle of length 3 or greater, then corresponding term in (1) must be zero because otherwise the tree would have a loop. Therefore we need only consider permutations that are products of disjoint transpositions where, moreover, each transposition exchanges the two vertices of an edge. Such a permutation is equivalent in an obvious way to a k-matching for some k, and the lemma follows.

Lemma 2. Two trees have the same spectrum for their generalized Laplacian if and only if they have the same generalized Laplacian immanantal polynomial for all characters.

Proof. One direction is trivial: if two trees have the same generalized Laplacian immanantal polynomial for all characters, then their generalized Laplacians have the same characteristic polynomial and hence the same spectrum.

Conversely, if the generalized Laplacians of two trees have the same spectrum, then the characteristic polynomials of the generalized Laplacians are the same. Lemma 1 in the case χ = sgn, the fact that sgn(C) = ±1 ≠ 0 for all k, and the fact that two equal polynomials have the same coefficients imply that the quantity

is the same for both trees for all k. Another application of Lemma 1 completes the proof.

A sufficient condition for two trees to have the same adjacency or Laplacian spectrum

We use the phylogenetic rather than graph-theoretic definition of a subtree as follows. Define the distal vertex of an edge to be the vertex farthest from the root that touches that edge. Then, a subtree of a given rooted tree is what results from separating an edge from its distal vertex, which then becomes the root of the subtree.

Recall that M(T) is the set of k-matchings of the tree T. Let N(T) be the set of k-matchings where the chosen edges do not contact the root.

Define

The following lemma is implicit in [14], but again we include a proof for completeness.

Lemma 3. Let S1 and S2 be trees with the same number of leaves. If P(S1) = P(S2) and Q(S1) = Q(S2) for all k, then any tree with S1 as a subtree has the same generalized Laplacian spectrum as the tree obtained by substituting S2 for S1.

Proof. Let T1 be a tree with S1 as a subtree, and write T2 for the tree obtained by substituting S2 for S1. Denote by e0 the edge that connects the rest of T1 (resp. T2) to the root of S1 (resp. S2).

We differentiate between two types of k-matchings of T: those that contain e0 and those that do not. Note that a k-matching of Tthat does not contain e0 restricts to an ℓ-matching of Sfor some ℓ, and all matchings of Sarise via such a restriction. Similarly, a k-matching of Tthat does contain e0 restricts to an ℓ-matching of Swith the property that the root of Sdoes not belong to any edge in the matching, and all matchings of Swith this property arise via such a restriction.

Consider the formula for the characteristic polynomial of the generalized Laplacian matrix that comes from Lemma 1 with χ = sgn. Apply this formula to T1 and T2. The assumption P(S1) = P(S2) (resp. Q(S1) = Q(S2)) ensures that the matchings that do not include (resp. do include) e0 make the same contribution to the respective characteristic polynomials.

The trees depicted in Figure 1 (b) are the smallest pair of rooted bifurcating trees satisfying the criteria of this lemma. The verification of this fact was done by computer, and the corresponding Pand Qpolynomials are available in the code repository.

A sufficient condition for two trees to have the same distance matrix spectrum

We first recall an identity for determinants of partitioned matrices. If

then

Lemma 4. Form two trees T1 and T2 by gluing the roots of trees S1 and S2 with distance matrices A1 and A2 onto the same leaf of a common tree R. Write afor the column vector of distances from the leaves of Sto the root of S. Suppose that the following pairs of matrices have the same spectra (where ' denotes transpose):

and

where 1 is a column vector with each entry 1. Then, the distance matrices of T1 and T2 have the same spectrum.

Proof. Write B for the distance matrix of R. Then, B has the partitioned form

where is the distance matrix of the tree obtained from R by deleting the last leaf, b is the column vector of distances from the other leaves of R to the last leaf. Assume without loss of generality that this last leaf is the attachment point of the S.

Denote by Dthe distance matrix of T. Observe that

Hence, by (2), Dhas the characteristic polynomial

Using (2) again, we see that a partitioned matrix of the form

where g and h are column vectors, has characteristic polynomial

and the result follows.

It was verified by computer that the trees in Figure 1 (b) are the smallest such that have distance matrices Aand vectors asatisfying the criteria of this lemma. The corresponding characteristic polynomials are available in the code repository. We note with surprise that the smallest pair of trees with the exchange property for the distance matrix are the same as the smallest pair with the exchange property for the generalized Laplacian; this is a curiosity for which we do not have an explanation.

Asymptotic numbers of trees

As outlined in the Introduction, the proof of Theorem 1 follows immediately from Lemma 2, Lemma 3, Lemma 4, the existence of the trees in Figure 1 (b), and the following result.

Proposition 1. Let T be a rooted tree. Let tbe the number of trees with n leaves. Let sbe the number of such trees that do not contain T as a rooted subtree. Then, the fraction s/tgoes to zero as n goes to infinity.

Proof. Suppose that T has a leaves. Let and denote the generating functions for tand s, respectively. Write ρ for the radius of convergence of the power series f and ρfor the radius of convergence of the power series f. Note that ρ ≤ ρ< 1.

It is shown in [26] that ρ = 0.402698 ... and

where η = 0.7916032 ... (see [27] for an asymptotic expansion of tthat extends this result and [28-31] for reviews of general methods for determining asymptotic numbers of trees of various sorts from a knowledge of the functional equations that their generating functions solve). Since sis o(α-) for any 0 <α <ρ, it follows that s/tis o(β) for any β > ρ/ρ, and the proposition will hold if we can show that ρ <ρ.

For the sake of completeness and because it serves as a good introduction to the derivation of the functional equation satisfied by the generating function of s, we first derive the well-known functional equation satisfied by the generating function of the t. See the comments after the proof of the lemma for some remarks about the history of the latter generating function.

By decomposing a tree into the two subtrees rooted at the daughters of the root, it is clear that

These expressions are equivalent to the statement

where tis set to zero if n is odd.

From (3) the generating function f satisfies the functional equation

It will be convenient to consider the function g: = 1 - f, which satisfies the functional equation

It is shown in [15] that:

• The radius of convergence ρ is strictly positive.

• The functional equation (4) has a unique solution in the whole complex plane, and this solution agrees with our power series in {x ∈ ℂ: |x| <ρ}.

• If, with a slight abuse of notation, we also denote this solution by g, then g(ρ) = 0.

• The point ρ is the only zero of g within {x ∈ ℂ : |x| < 1}.

It is clear from the power series that g is continuous and decreasing on [0, ρ) and g(0) = 1. Hence g is strictly positive on [0, ρ).

As observed in [15], these observations suggest a method for computing ρ. Put , so that h satisfies h(x2) = 2 + h2(x). Set

and

so that each function qis strictly increasing on [-2, +∞) and q1 ≤ q2 ≤ .... In particular, limq(y) exists in ℝ ∪ {+∞} for each y ∈ [-2, +∞). Moreover,

for 0

Conveniently, (3) holds with sin place of tfor all n except for n = a; in this case one simply adds two to the right hand side of the equation to make up for the fact that s= t- 1. Hence fsatisfies the functional equation.

Set g: = 1 - f, so that

It is clear that gis continuous and decreasing on [0, ρ) and g(0) = 1. Following the arguments in [15], the functional equation (6) has a unique solution in the whole complex plane, and this solution agrees with our power series in {x ∈ ℂ: |x| <ρ}. Moreover, analogues of the other properties of g obtained in [15] hold for g.

Set , so that

Put

and

Then

for 0

Now

and so

It therefore suffices to show that the function y ↦ limq(y) is strictly increasing on (-ε, +∞) for some 0 <ε < 2.

Observe that the derivative of qsatisfies

For k ≥ 1,

so that is non-decreasing for x > 0. For y ∈ (-ε, +∞),

and hence

Taking 0 <ε < 1, the proof will be completed by demonstrating for any x > 1 that

Taking the logarithm gives

and this series clearly converges by the ratio test.

In relation to Proposition 1, we note from [13] that the number of rooted strictly bifurcating trees without a given subtree is asymptotically smaller than the number of all graph-theoretic trees (see also [32] for more about the enumeration of general trees without a given subtree), but this is not enough for our purposes. We needed to show that it is asymptotically smaller than the space of all rooted strictly bifurcating trees. The generating function for tseems first to have been investigated in [15] in connection with enumerating "types of arrangements" in a commutative but non-associative algebra, such as a1(a2(a3a4)) or (a1a2)(a3a4); these are identical to rooted bifurcating trees in the "Newick" format [1]. The recursion behind the generating function has been re-discovered independently several times such as in [33] - see [34] for a discussion and several further references. We remark that numerically iterating the quantity qof the proof converges quickly to the value of ρ-1 calculable by other means [26,35]. We also observe that the methods of [27-31] can be used to show, in the notation of the proof, that for some positive constant ηand hence lim(ρ/ρ)(s/t) = η/η, but we don't pursue this matter here.

Numerical experiments

Proposition 1 says nothing about the rate of convergence of the fraction. Here we investigate this rate using computation. The characteristic polynomials for the generalized Laplacian were calculated via a doubly-recursive algorithm to enumerate matchings. The characteristic polynomials for the distance matrices were calculated via the Leverrier-Faddeev algorithm [36].

Table 1 shows that the fraction of trees with unique spectra does not go to zero very quickly. We can't compute this fraction for large numbers of leaves, but we can get some idea of the convergence by using the recursion relation corresponding to the generating function (5). Figure 2 shows the number of trees that do not have one of two subtrees of size seventeen as a subtree. This is an actual fraction that can be used with Proposition 1 in order to prove Theorem 1 for the generalized Laplacian matrix.

Table 1

The number of trees, the number of spectra for the generalized Laplacian (GLS), and the number of spectra for the distance matrix (DS).

leaves	trees	GLS	DS
2	1	1	1
3	1	1	1
4	2	2	2
5	3	3	3
6	6	6	6
7	11	11	11
8	23	22	23
9	46	45	46
10	98	95	98
11	207	203	207
12	451	443	451
13	983	972	983
14	2179	2159	2179
15	4850	4827	4850
16	10905	10870	10905
17	24631	24580	24630
18	56011	55931	56009
19	127912	127830	127908

Figure 2

The fraction of trees not containing a given pair of subtrees of size 17 .

Figure 2 shows that this fraction converges extremely slowly, despite the fact that as shown above it is asymptotically equivalent to βfor some 0 <β < 1. It is important to note, however, that this fraction is probably a very crude upper bound on the fraction of trees that share a spectrum with another tree. As can be seen in Table 1, the actual number not sharing a spectrum goes down considerably more quickly, though it is probably still the case that the vast majority of trees of intermediate size should have their own spectra.

Conclusions

Spectral invariants of matrix formulations of trees are a natural way to quantify the shape of phylogenetic trees. However, in this paper we show that a complete classification of tree shapes using common spectral invariants of generalized Laplacian and distance matrices is not possible. For either of these choices of matrix we show that the fraction of binary trees with a unique spectrum goes to zero as the number of leaves goes to infinity, but the rate of convergence of the above fraction to zero appears to be slow. For the adjacency and Laplacian matrices, we show that the a priori more informative immanantal polynomials have no greater power to distinguish between trees.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

FAM conceived of the project, proved the theorems, performed the numerical experiments, and wrote the paper. SNE applied the immanant, proved the theorems, and wrote the paper.