INCREASING SUBSEQUENCES AND THE CLASSICAL GROUPS

E. M. Rains

AT&T Research

Submitted: November 24, 1997; Accepted: January 30, 1998

Abstract. We show that the moments of the trace of a random unitary matrix have combinatorial interpretations in terms of longest increasing subsequences of permutations. To be precise, we show that the 2n-th moment of the trace of a random k-dimensional unitary matrix is equal to the number of permutations of length n with no increasing subsequence of length greater than k. We then generalize this to other expectations over the unitary group, as well as expectations over the orthogonal and symplectic groups. In each case, the expectations count objects with restricted “increasing subsequence” length.

Introduction

Much work has been done in the combinatorial literature on the “increasing subsequence problem”, that of studying the distribution of the length of the longest increasing subsequence of a random permutation. The problem was first considered by Hammersley ([5]); good summaries can be found in [1] and [10], which gives an alternate proof of Theorem 1.1. This problem is also closely connected to the representation theory of S_{n}, particularly the theory of Young tableaux. The representation theory aspects are covered in [13]; section 5.1.4 in [8] gives a good treatment of the more elementary Young tableaux results.

The results reported here arose from the observation that a certain partial sum of characters of the symmetric group that occurs naturally in the increasing subsequence problem also appears when calculating certain expectations over the unitary group. In particular, it turns out that the distribution of the length of the longest increasing subsequence can be expressed exactly in terms of the moments of the trace of a random (uniformly distributed) unitary matrix. This correspondence generalizes both to other moments for the unitary group, and to the moments of the trace of a random orthogonal or symplectic matrix. In each case, the moments count objects (colored permutations, signed permutations, or fixed-point-free involutions) with restricted increasing subsequence length.

Section 1 states and proves the connection between the classical increasing subsequence problem and the unitary group. Section 2 extends this to other increasing subsequence problems connected to the unitary group, including an increasing subsequence problem for signed permutations (the hyperoctahedral group). Section

Typeset by AMS-TEX

1991 Mathematics Subject Classification. Primary 05E15, Secondary 05A15 05A05.

3 gives the corresponding results for the other classical groups. Finally, section 4 proves an alternate form of theorem 1.1, originally given in [10].

This work is taken from section 5 of the author’s Ph.D. thesis ([11]).

1. The classical increasing subsequence problem

In [4], Diaconis and Shashahani give the following result:

E_{U∈U(k)} (|Tr(U)^{n}|^{2}) = n!,

for n ≤ k, where the expectation is taken with respect to Haar measure. A natural question is then: what happens for n > k? By refining their methods, one can prove the following:

Theorem 1.1. Define

f_{nk} = E_{U∈U(k)} (|Tr(U)^{n}|^{2}) .  (1.1)

Then f_{nk} is the number of permutations π of {1 … n} such that π has no increasing subsequence of length greater than k. (An increasing subsequence is a sequence i_{1} < i_{2} < … < i_{m} such that π(i_{1}) < π(i_{2}) < … < π(i_{m}).)

Proof. As in [4], we can think of this as an inner product of Tr(U)^{n} with itself; since the nonzero Schur functions are orthonormal with respect to this inner product, we can simplify things by expanding Tr(U)^{n} = p_{1^{n}}(U) (a power-sum symmetric function) into Schur functions. In general, for any expression of the form E_{U∈U(k)}(p_{λ}(U)\overline{p_{μ}(U)}), we can expand the power-sum functions into Schur functions, getting the following formula:

E_{U∈U(k)} (p_{λ}(U)\overline{p_{μ}(U)}) = ∑_{ν,κ⊢n} χ_{λ}^{ν}χ_{μ}^{κ}E_{U∈U(k)} (s_{ν}(U)s_{κ}(U))
= ∑_{{ν⊢n \atop ℓ(ν)≤k}} χ_{λ}^{ν}χ_{μ}^{ν},  (1.2)

where χ_{λ}^{ν} is the character of the symmetric group with label ν evaluated at a permutation of cycle type λ. In particular, for λ = μ = 1^{n}, we get

E_{U∈U(k)} (|p_{1^{n}}(U)|^{2}) = ∑_{{λ⊢n \atop ℓ(λ)≤k}} (χ_{1^{n}}^{λ})^{2} .  (1.3)

Thus, we are left with the purely combinatorial question of evaluating the sum on the right.

Recall that χ_{1^{n}}^{λ} is equal to the number of Young tableaux of shape λ. Thus (1.3) is counting the number of pairs of Young tableaux with the same shape, with size n, and with width at most k. But, by Schensted’s correspondence, this is exactly equal to the number of permutations of S_{n} with longest increasing subsequence of length at most k, and the theorem is proved. □

(Notation remark: in the sequel, f_{nk} will refer to the combinatorial quantity, rather than to the formula over U(k); similar remarks apply to the generalizations of f_{nk} defined below)

2. Other increasing subsequence problems

Given this result, it is natural to investigate the possibility of generalizing Theorem 1.1 to give combinatorial interpretations of other similar formulae on U(k). By (1.2), this boils down to finding combinatorial interpretations of

∑_{{ν⊢n \atop ℓ(ν)≤k}} χ_{λ}^{ν}χ_{μ}^{ν}.

Now, [17] and [15] give a generalization of Schensted’s correspondence connecting “hook permutations” and pairs of rim-hook tableaux of the same shape, analogous to Schensted’s correspondence, including, in some special cases, an increasing subsequence style result.

Let S_{n}^{(m)} be the set of functions from {1 … n} to {1 … n} × {1 … m} such that the function yields a permutation when projected onto the first component. In other words, the elements of S_{n}^{(m)} are essentially colored permutations. (This is a special case of Stanton and White’s concept of a hook permutation; the general definition is unnecessary for our purposes.) Then [15] gives a bijection between S_{n}^{(m)} and pairs of rim-hook tableaux of the same shape and content m^{k}.

Let us define an increasing subsequence of π ∈ S_{n}^{(m)} as a sequence i_{1} < i_{2} < … < i_{k} such that the first components of π(i_{j}) are increasing in j, and such that the second components of π(i_{j}) are all equal. If p is the second component, then the length of the increasing subsequence is defined to be m(k − 1) + p.

Lemma 2.1. For π ∈ S_{n}^{(m)}, the length of the longest increasing subsequence of π is equal to the width of the rim-hook tableaux corresponding to π.

Proof. Theorem 36 of [15] gives only that ⌈\frac{l}{m}⌉ = ⌈\frac{w}{m}⌉, where l is the length of the longest increasing subsequence, and w is the width of the rim-hook tableaux. However, it is straightforward to get this slight refinement using essentially the same proof. □

Then, using (1.2) and the Murnaghan-Nakayama formula for the characters of the symmetric group (where we remark that the sign of a rim-hook tableau of content m^{k} depends only on its shape, so can be ignored), we have immediately

Theorem 2.2. Define

f_{nk}^{(m)} = E_{U∈U(k)} (|Tr(U^{m})^{n}|^{2}) .

Then f_{nk}^{(m)} is the number of π ∈ S_{n}^{(m)} such that π has no increasing subsequence of length greater than k.

Remark. [4] gives the following formula:

E_{U∈U(k)} (|Tr(U^{m})^{n}|^{2}) = m^{n}n!,

for k ≥ mn. This also follows from theorem 2.2; if k ≥ mn, then no element of S_{n}^{(m)} has an increasing subsequence of length greater than k. Thus f_{nk}^{(m)} = |S_{n}^{(m)}| = m^{n}n!.

For other values of λ, (or, especially, when λ ≠ μ), we encounter a major difficulty in finding a combinatorial interpretation of the character sum (1.2). The problem is that the Murnaghan-Nakayama formula is, in general, an alternating sum. This causes problems with the generalized Schensted correspondence; when counting pairs of tableaux, it is necessary to consider signs. In order to make the total character sum work out, Stanton and White’s construction needs to pair up all pairs with opposite signs with pairs having the same signs. This pairing, unfortunately, does not respect shape. Thus, only in those cases in which the signs in the Murnaghan-Nakayama formula are constant for each shape can we expect a nice result as above. (At least for k small; in the case |λ|, |μ| ≤ k, [4] gives a closed form for E_{U∈U(k)}(p_{λ}(U)\overline{p_{μ}(U)}).) The signs in the case λ = μ = m^{n} are constant, as we mentioned above; the only remaining case (as far as the author knows) with constant signs is λ = μ = 2^{n}1, so it is natural to look for a combinatorial interpretation in this case.

Define B_{n} to be the hyperoctahedral group, defined as the group of permutations ρ of {−n, −n + 1, …, −1, 1, …, n − 1, n} such that ρ(−x) = −ρ(x). An element of B_{n} is determined by two data: |ρ(|x|)|, which permutes {1, 2, …, n}, and the sign of ρ(i) for 1 ≤ i ≤ n. This gives a bijection between B_{n} and the wreath product Z_{2} ≀ S_{n}; thus, |B_{n}| = 2^{n}n!. The order-preserving map between {−n, −n + 1 …, −1, 1, …, n − 1, n} and {1, 2, … n} gives an imbedding of B_{n} in S_{2n}; the defining relation of B_{n} becomes ρ(2n + 1 − x) = 2n + 1 − ρ(x). For our purposes, we also need an imbedding of B_{n} in S_{2n+1}. This is done by adding 0 to the set B_{n} permutes (naturally, ρ(0) = 0); we get a permutation on {1, 2, … 2n + 1} by a translation. The defining relation in that case is ρ(2n + 2 − x) = 2n + 2 − ρ(x).

Theorem 2.3. Define

b_{(2n)k} = E_{U∈U(k)} (|T r(U^{2})^{n}|^{2})
b_{(2n+1)k} = E_{U∈U(k)} (|T r(U^{2})^{n}T r(U)|^{2}) .

Then b_{Nk} is equal to the number of elements of B_{⌊ \frac{N}{2} ⌋} that have no increasing subsequence of length greater than k, considered as an element of S_{N}.

Proof. A “domino tableau” is defined (see, for instance, [7]) as a rim-hook tableau with content 2^{n}; every value appears in two neighboring positions. Similarly we define an “almost-domino tableau” as a rim-hook tableau with content 2^{n}1. Using the Murnaghan-Nakayama formula, we have that χ_{2^{n}}^{ν} is equal (up to sign) to the number of domino tableaux of size 2n, and χ_{2^{n}1}^{ν} is equal (again up to sign) to the number of almost-domino tableaux of size 2n + 1. Thus, we need to relate domino or almost-domino tableaux to the hyperoctahedral group.

We associate to each element of B_{n} a pair of tableaux of size N of the same shape, by applying the Schensted correspondence to the corresponding element of S_{N}. In order to count these tableaux, then, we need a more useful characterization of them. Since B_{n} ⊂ S_{N} is characterized as the subgroup of ρ such that N + 1 − ρ(N + 1 − x) = ρ(x), we need to understand how this transformation acts on tableaux. This action can be stated in terms of a certain dualization operation on tableaux (due to Schützenberger [14]). The dual of a tableau is defined as follows:

Let D_{1} be the following operation on tableaux: delete the top-left element of the tableaux, then move the lesser of the element immediately below and the element immediately to the right into the vacated spot, and continue until the edge of the tableaux is reached. For example:

1 3 5 2 4 6 → 3 5 2 4 6 → 2 3 5 4 6 → 2 3 5 4 6

To construct the dual of a tableaux, apply operation D_{1} repeatedly until the tableaux is empty; the dual tableaux has a 1 in the last position vacated, a 2 in the second-to-last position vacated, etc. Clearly, then the dual of a tableaux has the same shape. Furthermore, we have the following lemma ([8], section 5.1.4, theorem D):

Lemma 2.4. Let π ∈ S_{N} correspond to the pair of Young tableaux (T_{1}, T_{2}). Then the pair of tableaux corresponding to x ↦ N + 1 − π(N + 1 − x) is (T_{1}^{∗}, T_{2}^{∗}), where T^{∗} is the dual of T.

(Note that it follows immediately from this that (T^{∗})^{∗} = T).

Thus, under the Schensted correspondence in S_{N}, B_{n} corresponds to the set of pairs of self-dual tableaux. The following result is known (proposition 17 in [2]; see also [7]):

Lemma 2.5. There is a bijection between the set of self-dual tableaux of shape λ and the set of domino (almost-domino) tableaux of shape λ.

Theorem 2.3 follows immediately. □

Remark. We have two interpretations of E_{U∈U(k)}(| Tr(U^{2})^{n}|^{2}), namely the combinatorial versions of f_{nk}^{(2)} and b_{(2n)k}. It is unclear whether there is a simpler proof that these two combinatorial quantities are the same; unfortunately, the natural bijection between S_{n}^{(2)} and B_{n} does not preserve increasing subsequence length.

Remark. Again, [4] gives theorem 2.3, in the special case k ≥ N.

3. The other classical groups

The above results can be extended, in the simpler cases, to the orthogonal and symplectic groups, using some facts on the expectations of Schur functions over these groups. In particular, for any partition λ, E_{O∈O(k)}(s_{λ}(O)) is either 1 or 0; its value is 1 if and only if λ has at most k elements, each of which is even. This follows from the fact that only those representations of U(n) contain a copy of the trivial representation when restricted to O(n). Similarly, E_{S∈Sp(2k)}(s_{λ}(S)) is 1 if and only if λ has at most 2k elements, and every number appears an even number of times in λ (alternatively, every element of the transpose λ′ of λ is even). A straightforward modification of the first part of the proof of theorem 1.1 gives:

Lemma 3.1. E_{O∈O(k)}(Tr(O)^{n}) is equal to the number of Young tableaux of size n of width at most k with each column of even length.

And similarly for the symplectic group:

Lemma 3.2. E_{S∈Sp(k)}(Tr(S)^{n}) is equal to the number of Young tableaux of size n of width at most 2k with each row of even length.

It remains then to interpret this number in a more straightforward way. Now, just as there is a correspondence between pairs of Young tableaux and permutations, there is a similar correspondence between single Young tableaux and involutions (take the permutation corresponding to the pair (T, T)). Furthermore, the following is known ([8], exercise 5.1.4.4):

Lemma 3.3. Let π be an involution with k fixed points. Then the tableau corresponding to π has exactly k columns of odd length.

Thus a tableau with no odd columns corresponds to a permutation with no fixed points. Putting this together with the increasing subsequence result used in theorem 1.1, we get:

Theorem 3.4. E_{O∈O(k)}(T r(O)^{n}) is equal to the number of fixed-point-free involutions of length n with no increasing subsequence of length greater than k. Similarly, E_{S∈Sp(2k)}(T r(S)^{n}) is equal to the number of fixed-point-free involutions of length n with no decreasing subsequence of length greater than 2k.

Proof. The result for the orthogonal group follows easily from the above. For the symplectic case, it suffices to note that if one transposes the tableaux being counted, one ends up counting tableaux of height at most 2k where each column has even length. But the height of a tableaux gives the length of the longest decreasing subsequence of the corresponding permutation, and, as already established, tableaux with even-length columns correspond to fixed-point-free involutions. The desired result follows immediately. □

Remark. It should be noted that the above methods can also give combinatorial interpretations of E_{O(k)}(T r(O^{2})^{n}) and E_{Sp(2k)}(T r(S^{2})^{n}), given by replacing n by 2n in theorem 3.4 and adding the condition that the fixed-point-free involutions must also be elements of the hyperoctahedral group, as embedded above. However, there is apparently no immediate extension to E_{O(k)}(T r(O^{m})^{n}) and E_{Sp(2k)}(T r(S^{m})^{n}).

Remark. Again, [4] give a formula for E_{O∈O(k)}(p_{λ}(O)) and E_{S∈Sp(2k)}(p_{λ}(S)), for k sufficiently large compared to |λ|, using properties of the Brauer algebra; proofs of their results along the above lines are given as theorems 6.2 and 6.4 of [11].

4. An alternate form

One possible application of (1.1) and its generalizations is that it may be easier to get asymptotics for trace formulae on U(n) than for the associated combinatorial quantities; one can write explicit integrals for the trace formulae. [10] contains preliminary work in an attempt to use a related formula to prove asymptotic formulae for f_{nk}, which is a quantity of some interest to combinatorialists. They use the following formula (which they derived independently):

Corollary 4.1.

f_{nk} = \frac{2^{2n}(n!)^{2}}{(2π)^{k}k!(2n)!}I(n, k),

where

I(n, k) = ∫_{−π}^{π} … ∫_{−π}^{π} ( ∑_{1≤j≤k} cos(θ_{j}))^{2n} ∏_{1≤j≠l≤k} |e^{iθ_{j}} − e^{iθ_{l}} | dθ_{1} … dθ_{k}.

Proof. Noting that the density of Haar measure on U(k) in terms of the θ_{j} is given by

\frac{1}{2π}^{k} \frac{1}{k!} ∏_{1≤j≠l≤k} |e^{iθ_{j}} − e^{iθ_{l}} |

(see, for example, [16]), we can immediately rewrite I(n, k) as an expectation over U(k):

I(n, k) = (2π)^{k}k!E_{U∈U(k)} (( ∑_{1≤j≤k} cos(θ_{j}))^{2n}) .

Writing

∑_{1≤j≤k} cos(θ_{j}) = \frac{1}{2} ( ∑_{1≤j≤k}(e^{iθ_{j}} + e^{−iθ_{j}})) ,

we get

I(n, k) = \frac{(2π)^{k}k!}{2^{2n}}E_{U∈U(k)} ((Tr(U) + \overline{Tr(U)})^{2n}) .

We can expand this using the binomial theorem. Since Haar measure on U(k) is invariant under change of phase, it follows that the only term that contributes to the expectation is the Tr(U)^{n}\overline{Tr(U)^{n}} term; consequently, we have

I(n, k) = \frac{(2π)^{k}k!}{2^{2n}} \frac{2n!}{n!^{2}}E_{U∈U(k)} (|Tr(U)^{n}|^{2}) .

The result follows immediately from theorem 1.1. □

Remarks. (1) Zeilberger (personal communication) has reported a third independent proof of corollary 4.1, using Schenstead’s correspondence and a variant of the hook formula. (2) One can similarly rewrite the other trace formulae to use sums of cosines in place of the traces; it is not entirely clear which form would be more convenient for doing asymptotics.

Johansson has recently used this formula to give another proof that the mean of the length of the longest increasing subsequence of a random permutation of length N is asymptotically 2\sqrt{N} [6]. He also has similar results for the quantities of theorem 2.2 (personal communication), as well as heuristics for the variance.

It is also worth remarking that Regev ([12,3]) has considered the asymptotics of f_{nk} for k fixed, as well as the (combinatorial) quantities of theorem 3.4. Regev finds that these quantities (as well as a number of generalizations) are asymptotically given by certain multiple integrals (which can then be explicitly evaluated). In the cases of particular interest to us, the resulting multiple integrals turn out to be integrals appearing in the theory of random matrices (related to the Gaussian matrix ensembles). Consequently, it should be possible to derive Regev’s asymptotic formula for f_{nk} from theorem 1.1, together with standard results from random matrix theory relating the behavior of the eigenvalues of random unitary matrices to the behavior of the eigenvalues of Gaussian Hermitian matrices [9].

Acknowledgements

The greatest thanks are due to P. Diaconis (the author’s thesis advisor), for his many suggestions of problems (one of which appears in the present work).

Thanks are also due to the following people and institutions, in no particular order: A. Odlyzko, for many helpful comments; AT&T Bell Laboratories (Murray Hill) and the Center for Communications Research (Princeton) for generous summer support; the Harvard University Mathematics Department and the National Science Foundation, for generous support for the rest of the year. Also, thanks are due, for helpful comments, to N. Bergeron, M. Grassl, M. Rojas, R. Stanley, and D. Stroock. Last, but not least, the author owes thanks to W. Woyczynski, both for introducing him to probability theory and for introducing him to Prof. Diaconis; the thesis of which this work is a part would be very different, had either introduction not been made.

References

1. Aldous, D. and Diaconis, P., Hammersley’s interacting particle process and longest increasing subsequences, Prob. Th. and Rel. Fields 103 (1995), no. 2, 199–213.

2. Barbasch, D. and Vogan, D., Primitive ideals and orbital integrals in complex classical groups, Math. Ann. 259 (1982), 153–199.

3. Cohen, P. S. and Regev, A., Asymptotics of combinatorial sums and the central limit theorem, SIAM J. Math. Anal. 19 (1988), no. 5, 1204–1215.

4. Diaconis, P. and Shahshahani, M., On the Eigenvalues of Random Matrices, J. Appl. Prob. 31 (1994), 49–61.

5. Hammersley, J. M., A few seedlings of research, Proc. Sixth Berkeley Symp. Math. Statist. and Probability, vol. 1, University of California Press, Berkeley, Calif., 1972, pp. 345–394.

6. Johansson, K., The longest increasing subsequence in a random permutation and a unitary random matrix model (to appear).

7. Kirillov, A. N., Lascoux, A., Leclerc, B., and Thibon, J-Y., Séries génératrices pour les tableaux de dominos, C. R. Acad. Sci. Paris Sér. I Math. 318 (1994), no. 5, 395–400.

8. Knuth, D. E., The Art of Computer Programming, vol. 3: Sorting and Searching, 2nd. ed., Addison Wesley, Reading, Mass., 1973.

9. Mehta, M. L., Random Matrices, 2nd ed., Academic Press, Boston, Mass., 1991.

10. Odlyzko, A. M., Poonen, B., Widom, H. and Wilf, H. S., On the distribution of longest increasing subsequences in random permutations, in preparation.

11. Rains, E. M., Topics in Probability on Compact Lie Groups, Ph.D. thesis, Harvard University, Cambridge, Mass., 1995.

12. Regev, A., Asymptotic values for degrees associated with strips of young diagrams, Adv. in Math. 41 (1981), 115–136.

13. Sagan, B., The Symmetric Group: Representations, Combinatorial Algorithms, and Symmetric Functions, Wadsworth & Brooks/Cole, Pacific Grove, Calif., 1991.

14. Schützenberger, M. P., Quelqes remarques sur une construction de Schensted, Math. Scand. 12 (1963), 117–128.

15. Stanton, D. W. and White, D. E., A Schensted Algorithm for Rim Hook Tableaux, J. Comb. Th., series A 40 (1985), 211–247.

16. Weyl, H., Classical Groups, Princeton University Press, Princeton, N.J., 1942.

17. White, D. E., A Bijection Proving Orthogonality of the Characters of S_{n}, Adv. in Math. 50 (1983), 160–186.

AT&T Research, 180 Park Avenue, Florham Park, NJ 07932-0971 E-mail address: rains@research.att.com