Journal of Lie Theory Volume 9 (1999) 369–382 ◯^{C}1999 Heldermann Verlag

Maximal degenerate representations of SL(n+1,H)

Angela Pasquale

Communicated by J. Faraut

Abstract. The infinitesimal method is applied to study some maximal degenerate representations of SL(n + 1, ℍ). The irreducibility and unitarizability of these representations are determined and the composition series is given in the reducible case. The existence of intertwining operators is established for the differentiated Lie algebra representations. On the group level, the intertwining operators are given by an explicit integral formula.

Introduction

The intimate connection between maximal degenerate representations of SL(n + 1, 𝔽) and canonical representations of SU(1, n; 𝔽) associated with a finite dimensional irreducible representation of U(1, 𝔽) has been recognized in [4],

[3] and [5] respectively for 𝔽 = ℝ, ℂ and ℍ. The structure of these representations of SL(n + 1, 𝔽) is therefore a key ingredient for the understanding of the corresponding canonical representations. The cases of SL(n + 1, ℝ) and SL(n + 1, ℂ) have been respectively treated in [4] and [7]. This paper deals with the case of SL(n + 1, ℍ).

Precisely, to each finite dimensional irreducible representation of Sp(1) we associate two families of degenerate representations of SL(n + 1, ℍ) induced by opposite maximal parabolic subgroups. We study the irreducibility and unitarizability of these representations, and, when reducible, we determine their composition series. Moreover, we determine the existence of intertwining operators among them. The explicit integral formula for these intertwining operators allows us to compute the so called η-function of [12] in a very direct manner.

Our analysis is based on the infinitesimal method, which easily produces very explicit results. After its first application by Bargmann for SL(2, ℝ), this method has been successfully applied for the study of some degenerate principal series representations of several other classical groups. Among these we mention the following: SO_{0}(p, q), see [14], [15] and [8]; SU(p, q), see [8], and [9] for p = q; Sp(p, q), see [8] and [9] for p = q; Sp(2n, ℝ), see [10] and [13]; SL(n + 1, ℝ), see [4]; SL(n + 1, ℂ), see [7].

ISSN 0949–5932 / $2.50 ◯^{C} Heldermann Verlag

Our results are somehow related to those determined for SL(4(n+1), ℝ), but for us the procedure to obtain them is more complicated because the restriction of our representations to the maximally compact subgroup Sp(n + 1) of SL(n + 1, ℍ) is not multiplicity free. This forces us to modify the usual application of the infinitesimal method as employed in the case of either SL(n+1, ℝ) or SL(n+1, ℂ).

The author wishes to express her thanks to Prof. van Dijk for having suggested the analysis of the representations considered in this paper and for having explained to her the general correspondence between canonical representations for the hyperbolic spaces and the restriction of maximal degenerate representations of groups SL. The author also gratefully acknowledges financial support from the Dutch Organization for Scientific Research (N.W.O.) and the Thomas Stieltjes Institute for Mathematics.

1. Preliminaries

Let ℍ denote the skew field of the quaternions, with units 1, i, j, k (i^{2} = j^{2} = k^{2} = −1). If q = a + ib + j c + kd ∈ ℍ (with a, b, c, d ∈ ℝ), then the quaternionic conjugate of q is q̄ = a − ib − j c − kd. The real and the imaginary parts of q are respectively ℜq = a and ℑq = ib + j c + kd. If q = ℑq, then q is said to be purely quaternionic.

Let 𝔤 = 𝔰𝔩(n + 1, ℍ) denote the real Lie algebra of the (n + 1) × (n + 1) matrices over ℍ with purely quaternionic trace. The simply connected real Lie group SL(n + 1, ℍ) associated with 𝔰𝔩(n + 1, ℍ) will be also denoted by G. Using Dieudonné notion of determinant for square matrices with entries from ℍ (described for example in [1], pp. 151–158), one can realize SL(n + 1, ℍ) as the group of the (n + 1) × (n + 1) matrices over ℍ with determinant equal to 1. It turns out that SL(n + 1, ℍ) can also be identified with the group SU^{∗}(2(n + 1)) consisting of the (2n + 2) × (2n + 2) matrices over ℂ with determinant 1 which are of the form

[ \begin{matrix}Z_{1} & −Z̄_{2} \\ Z_{2} & Z̄_{1} \end{matrix} ]

for (n + 1) × (n + 1) complex matrices Z_{1} and Z_{2}.

Endow the right ℍ-vector space ℍ^{n+1} with the inner product

(x, y) = ȳ_{0}x_{0} + ȳ_{1}x_{1} + … + ȳ_{n}x_{n}.

The subgroup U = Sp(n + 1) of G consisting of all matrices preserving (·, ·) is simply connected and maximally compact in G. Its Lie algebra 𝔲 = 𝔰𝔭(n + 1) consists of all the skew-Hermitian (n + 1) × (n + 1) matrices over ℍ: X + X̄^{t} = 0 for X ∈ 𝔲, the symbol ^{t} denoting transposition. If 𝔭 indicates the vector space of the Hermitian elements of 𝔤, then 𝔤 = 𝔲 + 𝔭 is the Cartan decomposition of 𝔤 associated with the Cartan involution θ : X ↦ −X̄^{t}. The corresponding involution on G is also denoted by θ (so θ(g) = (ḡ^{t})^{−1}).

Consider the maximal parabolic subgroups P^{+} and P^{−} = θ(P^{+}) of G, respectively consisting of the matrices in G of the form

[ \begin{matrix}a & b \\ 0 & C \end{matrix} ]  and  [ \begin{matrix}a & 0 \\ b & C \end{matrix} ]

with a ∈ ℍ \ {0} and C an n × n matrix.

For l ∈ ℕ/2, let τ_{l} denote the (2l + 1)-dimensional irreducible representation of Sp(1) ≅ SU(2), realized as a unitary representation on the space (V_{l}, (·, ·)_{l}). The elements of Sp(1) will be often considered as quaternions λ with norm (λ̄λ)^{1/2} = 1.

For μ ∈ ℂ define representations ω_{l,μ}^{±} of P^{±} by

ω_{l,μ}^{±}(p) = |a|^{μ±ρ}τ_{l}(a/|a|),  p = [ \begin{matrix}a & · \\ · & · \end{matrix} ] ∈ P^{±},  (1)

where ρ := 2(n + 1).

Let π_{l,μ}^{±} denote the representation of G induced from P^{±}:

π_{l,μ}^{±} = Ind_{P^{±}}^{G}ω_{l,μ}^{±}.  (2)

The natural action of G on ℍ_{∗}^{n+1} := ℍ^{n+1}\{0} is transitive. The subgroup P_{1}^{+} of G stabilizing the point e_{0} = [1, 0, …, 0]^{t} consists of all elements of P^{+} of the

form [ \begin{matrix}1 & b \\ 0 & C \end{matrix} ]. On it, ω_{l,μ}^{±} is trivial. Let D_{μ}(ℍ_{∗}^{n+1}, τ_{l}) be the space of compactly

supported smooth V_{l} -valued functions on ℍ_{∗}^{n+1} satisfying

F (xa) = |a|^{−(μ+ρ)}τ_{l}(a/|a|)^{−1}F (x),  x ∈ ℍ_{∗}^{n+1}, a ∈ ℍ \ {0}.  (3)

A noncompact realization of π_{l,μ}^{+} can then be given on D_{μ}(ℍ_{∗}^{n+1}, τ_{l}) by

[π_{l,μ}^{+}(g)F](x) = F (g^{−1}x),  g ∈ G, x ∈ ℍ_{∗}^{n+1}.  (4)

(gx denoting multiplication of the matrix g ∈ G by x ∈ ℍ_{∗}^{n+1} considered as a column vector).

Let S := {s ∈ ℍ^{n+1} : ∥s∥ = 1} be the unit sphere in ℍ^{n+1}, where ∥x∥ := (x, x)^{\frac{1}{2}} for x ∈ ℍ^{n+1}. G acts on S by

g · s = \frac{gs}{∥gs∥},  g ∈ G, s ∈ S.  (5)

A compact realization of π_{l,μ}^{+} on the space D(S, τ_{l}) of all smooth functions φ : S → V_{l} satisfying φ(sλ) = τ_{l} (λ)^{−1} φ(s) for all s ∈ S and λ ∈ Sp(1) is given by

[π_{l,μ}^{+}(g)φ](s) = φ(g^{−1} · s)∥g^{−1}s∥^{−(μ+ρ)},  g ∈ G, s ∈ S.  (6)

The realizations of π_{l,μ}^{−} are obtained by considering the transitive action of G on ℍ_{∗}^{n+1} given by g : x ↦ θ(g)(x), for which the stabilizer of e_{0} is P_{1}^{−} = θ(P_{1}^{+}). The noncompact and compact realizations of π_{l,μ}^{−} are again on the spaces D_{−μ}(ℍ_{∗}^{n+1}, τ_{l}) and D(S, τ_{l}), respectively. They are

[π_{l,μ}^{−}(g)F](x) = F (θ(g^{−1})x),  g ∈ G, F ∈ D_{−μ}(ℍ_{∗}^{n+1}, τ_{l}), x ∈ ℍ_{∗}^{n+1}  (7)
[π_{l,μ}^{−}(g)φ](s) = φ(θ(g^{−1}) · s)∥θ(g^{−1})s∥^{μ−ρ},  g ∈ G, φ ∈ D(S, τ_{l}), s ∈ S. (8)

It follows, in particular, that π_{l,μ}^{−} = π_{−μ,l}^{+} ∘ θ.

In the remainder of the paper we will only work with the compact realizations. We endow every space of C^{∞} functions with the Schwartz topology. The π_{l,μ}^{±} are therefore admissible differentiable representations of G. We refer to [2] Chapter III §7 for a discussion on the Schwartz topology and for the relationship between the notion of induced representation of this section and other notions commonly found in the literature.

2. Reduction of the problem

The usual application of the infinitesimal method (cf. f.i. [14] or [4]) cannot be directly employed for the representations π_{l,μ}^{±}. In fact their restriction to the maximally compact subgroup U = Sp(n + 1) is not multiplicity free. The problem is overcome by a trick similar to the one used in [8] for the study of the degenerate principal series representations of Sp(p, q). The main idea is to use the commutativity of the G-action (5) on S with the right multiplication by elements of Sp(1) in order to replace the representations π_{l,μ}^{±} of G with suitable “extended” representations of G×Sp(1) having multiplicity free decomposition over U ×Sp(1).

We first substitute the representation π_{l,μ}^{±} of G with the representation π_{l,μ}^{±} ⊗ τ_{l} of G̃ := G × Sp(1). Since τ_{l} is irreducible and unitary, the irreducibility and unitarizability of (a submodule of) π_{l,μ}^{±} and of (the corresponding submodule in) π_{l,μ}^{±} ⊗ τ_{l} are equivalent.

The representation τ_{l} is self-dual (i.e. equivalent to its contragredient representation). Using [17] p. 99, it is possible to show the existence of an orthogonal basis {e_{j}}_{j=0}^{2l} for V_{l} with dual basis {ε_{j}}_{j=0}^{2l} for the dual space V_{l}^{∗} so that the linear operator A_{l} : V_{l} → V_{l}^{∗} of matrix [A_{l;i,j}]_{i,j=0}^{2l} with

A_{l;i,j} = (−1)^{i}δ_{j,2l−i}  (9)

intertwines τ_{l} and its contragredient representation. Here δ_{j,k} is the Kronecker delta: δ_{jk} = 1 if j = k and = 0 otherwise.

The bilinear form

⟨v, w⟩_{l} := (A_{l}v)(w)  (10)

is nondegenerate, symmetric if 2l is even, skew-symmetric if 2l is odd, and it satisfies

⟨τ_{l}(k)v, τ_{l}(k)w⟩_{l} = ⟨v, w⟩_{l}, k ∈ k, v, w ∈ V_{l}.

Let D(S) denote the space of C^{∞} functions on S endowed with the C^{∞}-topology. Define β_{l} : D(S, τ_{l}) ⊗ V_{l} → D(S) by

β_{l}(φ ⊗ v)(s) = ⟨φ(s), v⟩_{l},  φ ∈ D(S, τ_{l}), v ∈ V_{l}, s ∈ S.  (11)

Let T_{l,μ}^{±} be the representations of G̃ on D(S) given by

[T_{l,μ}^{+}(g, λ)f](s) = f(g^{−1} · sλ)∥g^{−1}s∥^{−(μ+ρ)}
[T_{l,μ}^{−}(g, λ)f](s) = f(θ(g^{−1}) · sλ)∥θ(g^{−1})s∥^{μ−ρ}

for g ∈ G, λ ∈ Sp(1), f ∈ D(S), s ∈ S. Then β_{l} is a continuous linear isomorphism of D(S, τ_{l})⊗V_{l} onto its image D_{l}(S) := β_{l}(D(S, τ_{l})⊗V_{l}) with the induced topology from D(S), and it intertwines the representations π_{l,μ}^{±} ⊗ τ_{l} and T_{l,μ}^{±} of G̃.

The study of the representation π_{l,μ}^{±} of G can be therefore equivalently replaced by the study of the representation T_{l,μ}^{±} of G̃.

3. Irreducibility of π_{l,μ}^{±}

Let Ũ := Sp(n + 1) × Sp(1), which is maximally compact in G̃, and let D_{l}(S)Ũ denote the space of Ũ -finite vectors in D_{l}(S). The representation T_{l,μ}^{±} induces a representation of the universal enveloping algebra 𝔘(𝔤̃) of the complexification of the Lie algebra 𝔤̃ of G̃ on D_{l}(S)Ũ. We also denote this representation by T_{l,μ}^{±}. It is a well known property that the irreducibility of D_{l}(S)Ũ as a 𝔘(𝔤̃)-module is equivalent to the irreducibility of D_{l}(S) as a G̃-module. We can therefore recover the properties of the G̃-module out of those of the 𝔘(𝔤̃)-module.

Observe that for (k, λ) ∈ Ũ and f ∈ D_{l}(S)Ũ

[T_{l,μ}^{±}(k, λ)f](s) = f(k^{−1}sλ),  s ∈ S.  (12)

T_{l,μ}^{±} is therefore the restriction to D_{l}(S)Ũ of the representation of Ũ on D(S) by the same formula (12). The Ũ -type decomposition of this representation on D(S) has been studied by Johnson and Wallach [11] (see also [16]).

Let M̃ denote the group of all matrices in Ũ of the form

[ \begin{matrix}u & 0 & 0 \\ 0 & V & 0 \\ 0 & 0 & u \end{matrix} ]

with u ∈ Sp(1) and V ∈ Sp(n). Theorem 3.1(3) in [11] proves that the Ũ -finite vectors in D(S) decompose according to

D(S)Ũ = ∑ {V^{p,q} : p, q integers, p ≥ q ≥ 0, p ≡ q (mod 2)} .  (13)

Here V^{p,q} is the Ũ -cyclic subspace with M̃ -fixed vector e_{p,q} (unique up to a constant multiple) described as follows. Let x_{0}, x_{1}, …, x_{n} be the usual coordinates in ℍ^{n+1}. Set

r^{2} = ∑_{j=0}^{n} |x_{j}|^{2} ,  r cos ξ = |x_{0}| ,  r cos ξ cos t = ℜx_{0}  (14)

with 0 ≤ ξ ≤ \frac{π}{2} and 0 ≤ t ≤ π. e_{p,q} is the function on S (depending only on |x_{0}| and on ℜx_{0}, i.e. only on ξ and t) given by

e_{p,q}(ξ, t) = Ξ_{q/2}(t)h_{p,q/2}(ξ)  (15)

where

Ξ_{q/2}(t) = \frac{sin(q + 1)t}{sin t}  (16)

and

h_{p,q/2}(ξ) = cos^{p} ξ F (\frac{−p + q}{2} , \frac{−p − q − 2}{2}; 2n; − tan^{2} ξ)  (17)

(F (a, b; c; z) is Gauss’ hypergeometric function). If the point of coordinates (p, q) denotes V^{p,q}, then D(S)Ũ can be depicted as in Fig.1a.

Let χ_{l}(λ) := trace τ_{l}(λ) be the character of τ_{l}. Then Ξ_{l}(t) = χ_{l}(λ) if ℜλ = cos t. The explicit expressions for β_{q/2} and ⟨·, ·⟩_{q/2} give

e_{p,q}(s) = β_{q/2} (∑_{j=0}^{q} F_{p,q;j} ⊗ e_{j}) (s)  (18)

where {e_{j}}_{j=0}^{q} is the fixed orthogonal basis for V_{q/2} and

F_{p,q;j}(s) := (−1)^{j}τ_{\frac{q}{2}} (s_{0}/|s_{0}|)^{−1}(e_{2l−j})h_{p,q}(Arccos |s_{0}|).  (19)

We obtain the following lemma.

[figure: p; p; q; q; D(S)Ũ; D_{l}(S)Ũ; 2l; 2l; (a); (b)]

Figure 1: Ũ -type decomposition for (a) D(s) and (b) D_{l}(S)

Lemma 3.1. e_{p,q} ∈ D_{l}(S)Ũ if and only if q = 2l.

We therefore have the Ũ -type decomposition

D_{l}(S)Ũ = ∑ {V^{p,2l} : p integer ≥ 2l, p ≡ 2l (mod 2)} ,  (20)

which is represented in Fig.1b.

Let

H := diag (\frac{n}{n + 1}, −\frac{1}{n + 1}, …, −\frac{1}{n + 1}) .  (21)

The group G̃ is generated by Ũ and by {exp(tH) : t ∈ ℝ} ≡ {exp(tH) : t ∈ ℝ} × {1}. The action of 𝔘(𝔤̃) on the Ũ -types is therefore completely determined by the action of the differential operators T_{l,μ}^{±}(H) = ±T_{l,±μ}^{+}(H). In particular, it is enough to know the image of e_{p,2l} under T_{l,μ}^{+}(H).

Let τ ∈ ℝ. For the generic element s ∈ S, of coordinates ξ and t, let ξ(τ) and t(τ) respectively denote the values of the coordinates ξ and t of the element

s(τ) := \frac{exp(−τH)s}{∥ exp(−τH)s∥} ∈ S.  (22)

Then

cos ξ(τ) = \frac{e^{− \frac{nτ}{n+1}} cos ξ}{[e^{\frac{2τ}{n+1}} sin^{2} ξ + e^{− \frac{2nτ}{n+1}} cos^{2} ξ]^{1/2}}  (23)
cos t(τ) = cos t.  (24)

On functions of (ξ, t), we have

T_{l,μ}^{+}(H) = (ρ + μ) (\frac{−1}{1 + n} + cos^{2} ξ) + sin ξ cos ξ \frac{∂}{∂ξ} .  (25)

The Gauss’ relation (cf. [6] 2.8(33))

(γ − α − β)F (α, β; γ; z) + α(1 − z)F (α + 1, β; γ; z)
−(γ − β)F (α, β − 1; γ; z) = 0

and the differentiation property (cf. [6] 2.8(21))

\frac{d}{dz} [z^{α}F (α, β; γ; z)] = αz^{α−1}F (α + 1, β; γ; z)

prove the following lemma.

Lemma 3.2.

(a) \frac{b}{c}zF (a + 1, b + 1; c + 1; z) = F (a + 1, b; c; z) − F (a, b; c; z) .

(b) F (a + 1, b; c; z) = −\frac{b}{c − a − b − 1}(1 − z)F (a + 1, b + 1; c; z)

+\frac{c − a − 1}{c − a − b − 1}F (a, b; c; z).

(c) \frac{1}{1 − z} F (a, b; c; z) =

= \frac{ab}{(c − a − b)(c − a − b + 1)}(1 − z)F (a + 1, b + 1; c; z)

−\frac{a(c − a) + (c − b − 1)(b − 1)}{(c − a − b − 1)(c − a − b + 1)}F (a, b; c; z)

+\frac{(c − b)(c − a)}{(c − a − b)(c − a − b + 1)} \frac{1}{1 − z} F (a − 1, b − 1; c; z) .

A long but straightforward computation gives

T_{l,μ}^{+}(H)e_{p,2l} = α_{+}(p, 2l, μ)e_{p+2,2l} + α_{0}(p, 2l, μ)e_{p,2l} + α_{−}(p, 2l, μ)e_{p−2,2l}  (26)

where

α_{+}(p, 2l, μ) = \frac{(4n + p − 2l)(4n + p + 2l + 2)}{4(2n + p + 1)(2n + p)} β_{+}(p, μ)  (27)

α_{0}(p, 2l, μ) = \frac{p(n − 1)(4n + p + 2) − 4l(l + 1)(n + 1)}{2(n + 1)(2n + p)(2n + p + 2)} β_{0}(p, μ)  (28)

α_{−}(p, 2l, μ) = \frac{(p − 2l)(p + 2l + 2)}{4(2n + p + 1)(2n + p)} β_{−}(p, μ)  (29)

and

β_{+}(p, μ) = μ + ρ + p  (30)
β_{0}(p, μ) = μ  (31)
β_{−}(p, μ) = μ − ρ − p + 2.  (32)

Observe that the functions β_{+}(p, μ), β_{0}(p, μ) and β_{−}(p, μ) are equal to the analogue functions for SL(4(n + 1), ℝ) ^{1}. T_{l,μ}^{±} is reducible if and only if

β_{−}(p, μ) = 0 for some p > 2l, p ≡ 2l (mod 2)

or

β_{+}(p, μ) = 0 for some p ≥ 2l, p ≡ 2l (mod 2).

We therefore have the following theorem.

1cf. [4], p.5. In [4] the groups G = SL(n, ℝ) acts from the right, and no ρ-shift is considered in the inducing representation of P^{±}. Our β_{+}(p, μ) and β_{−}(p, μ) must therefore be compared with the functions −β_{1}(−μ − ρ, p) and −β_{2}(−μ − ρ, p) for SL(4(n + 1), ℝ), the functions β_{1} and β_{2} being as in [4].

Theorem 3.3. The representations T_{l,μ}^{±} of G×Sp(1) (hence the representations π_{l,μ}^{±} of G) are irreducible except for the following cases:

(a) μ ∈ ℤ, μ ≥ 2l + ρ, μ ≡ 2l (mod 2).

(b) μ ∈ ℤ, μ ≤ −(2l + ρ), μ ≡ 2l (mod 2).

In Case (a), T_{l,±μ}^{±} has in D_{l}(S) one infinite dimensional irreducible submodule E_{l,μ} and one finite dimensional irreducible quotient Ẽ_{l,μ} = D_{l}(S)/E_{l,μ}. In Case (b), T_{l,±μ}^{±} has in D_{l}(S) one finite dimensional irreducible submodule E_{l,μ} and one infinite dimensional irreducible quotient Ẽ_{l,μ} = D_{l}(S)/E_{l,μ}.

Consequently, in Case (a), π_{l±μ}^{±} has in D_{l}(S, τ_{l}) one infinite dimensional irreducible submodule E_{l,μ} and one finite dimensional irreducible quotient Ẽ_{l,μ} = D_{l}(S, τ_{l})/E_{l,μ}. In Case (b), T_{l,±μ}^{±} has in D_{l}(S, τ_{l}) one finite dimensional irreducible submodule E_{l,μ} and one infinite dimensional irreducible quotient Ẽ_{l,μ} = D_{l}(S, τ_{l})/E_{l,μ}.

For a fixed l ∈ ℕ/2, the situation for T_{l,μ}^{+} is described by the diagrams in Figure 2. The dot {• \atop m} denotes the Ũ -type in D_{l}(S)Ũ corresponding to the parameter (m, 2l). A barrier • ] • cannot be crossed from left to right using the 𝔘(𝔤̃)-action, but the crossing in the opposite direction is allowed. In this situation, the Ũ -types on the left of the barrier form a 𝔘(𝔤̃)-invariant subspace; those at the right represent a 𝔘(𝔤̃)-invariant quotient. A similar interpretation is given to a barrier of the form • [ •.

[figure: p = 2l; p = 2l; 2l+2; 2l+2; μ−ρ; −μ−ρ; μ−ρ + 2; −μ −ρ+2; E_{l,μ}; Ẽ_{l,μ}; Ẽ_{l,μ}; E_{l,μ}; μ ≤ −(2l + ρ) :; μ ≥ 2l + ρ :]

Figure 2: Composition series for the reducible T_{l,μ}^{+} (μ ∈ ℤ, μ ≡ 2l (mod 2))

4. Unitarizability

The representation π_{l,μ}^{±} is said unitarizable if the space of U -finite vectors in D(S, τ) can be endowed with a positive definite Hermitian form ⟨·, ·⟩ for which the 𝔤-action is given by skew-adjoint operators:

⟨π_{l,μ}^{±}(X)φ, ψ⟩ = −⟨φ, π_{l,μ}^{±}(X)ψ⟩, X ∈ 𝔤, φ, ψ ∈ D(S, τ_{l})_{U}.

As for the irreducibility of the π_{l,μ}^{±}, also the unitarizability of the π_{l,μ}^{±} can be deduced from the unitarizability of the T_{l,μ}^{±}.

Let us first determine the existence of 𝔤̃-intertwining operators between the various T_{l,μ}^{±}, that is the existence of (nonzero) linear operators Ã : D_{l}(S)Ũ → D_{l′}(S)Ũ satisfying

Ã ∘ T_{l,μ}^{±}(X) = T_{l′,μ′}^{±}(X) ∘ Ã,  X ∈ 𝔤̃  (33)

(in (33) any possible combination of ± signs is allowed). Let 𝔲̃ = 𝔰𝔭(n + 1)⊕𝔰𝔭(1) be the Lie algebra of Ũ. Since different V^{p,q} ’s are inequivalent Ũ -modules, such Ã cannot exist unless l = l′. Moreover, the restriction of Ã to each V^{p,2l} must be a scalar multiple of the identity operator:

Ã|_{V^{p,2l}} = a_{p,2l}I

for some nonzero constants a_{p,2l}. These constants can be determined by applying both sides of the equation

Ã ∘ T_{l,μ}^{+}(H) = ±T_{l,±μ′}^{+}(H) ∘ Ã  (34)

to the M̃ -fixed vectors e_{p,2l}. In (34) the + signs correspond to the case of interwining operators for (T_{l,μ}^{+}, T_{l,μ′}^{+}) or (T_{l,−μ}^{−}, T_{l,−μ′}^{−}), whereas the − signs correspond to intertwining operators for (T_{l,μ}^{+}, T_{l,μ′}^{−}) or (T_{l,−μ}^{−}, T_{l,−μ′}^{+}).

Using (27)–(29), we obtain

a_{p+2,2l} (μ + ρ + p) = a_{p,2l} (μ′ ± ρ ± p)
a_{p,2l} μ = a_{p,2l} μ′
a_{p−2,2l} (μ − ρ − p + 2) = a_{p,2l} (μ′ ± (2 − ρ − p)),

i.e.

a_{p+2,2l} (μ + ρ + p) = a_{p,2l}(μ ± ρ ± p)  (35)

Suppose first μ ∈ ℂ is so that T_{l,μ}^{±} is irreducible. Then we can uniquely determine a_{p,2l} in terms of a_{2l,2l}. When the + signs are considered, (35) gives a_{p,2l} = a_{2l,2l} for all p ≥ 2l, p ≡ 2l (mod 2). So the unique 𝔤̃-intertwining operators between T_{l,μ}^{±} and itself are the scalar multiples of the identity operator. When the − signs are considered, (35) shows that the unique 𝔤̃-intertwining operators for (T_{l,μ}^{+}, T_{l,μ}^{−}) and for (T_{l,−μ}^{−}, T_{l,−μ}^{+}) are given by Ã|_{V^{p,2l}} = a_{p,2l}I with

a_{p,2l} = \frac{Γ (\frac{β_{−}(2l,μ)}{2}) Γ (\frac{β_{+}(2l,μ)}{2})}{Γ (\frac{β_{−}(p,μ)}{2}) Γ (\frac{β_{+}(p,μ)}{2})} a_{2l,2l}  (36)

= \frac{Γ (−\frac{β_{−}(p,μ)}{2} + 1) Γ (−\frac{β_{+}(p,μ)}{2} + 1)}{Γ (−\frac{β_{−}(2l,μ)}{2} + 1) Γ (−\frac{β_{+}(2l,μ)}{2} + 1)} a_{2l,2l}.  (37)

Suppose then μ ∈ ℂ corresponds to a reducible T_{l,μ}^{±}.

Let T_{l,μ}^{±,0} and T̃_{l,μ}^{±} respectively denote the restriction of T_{l,μ}^{±} to E_{l,μ} and the factor representation of T_{l,μ}^{±} on Ẽ_{l,μ}. We indicate with the same symbols also the corresponding differentiated representations of 𝔤̃. The choice of the + signs in (35) gives the scalar multiples of the identity as the unique 𝔤̃-intertwining operators between each of these representations and itself. When the − signs are considered in (35), we have to distinguish two further cases, μ ≤ −(ρ+2l) and μ ≥ ρ+2l. In the first case, (35) is solved for 2l ≤ p ≤ −μ−ρ by (37), which defines intertwining operators for (T_{l,μ}^{+,0}, T̃_{l,μ}^{−}) and for (T_{l,−μ}^{−,0}, T̃_{l,−μ}^{+}). The solution for p ≥ −μ − ρ + 2 is

a_{p,2l} = (−1)^{\frac{β_{+}(p,μ)}{2} +1} \frac{Γ (−\frac{β_{−}(p,μ)}{2} + 1)}{Γ (−μ + 1) Γ (\frac{β_{+}(p,μ)}{2})} a_{−μ−ρ+2,2l}.  (38)

Observe that the right-hand side of (38) vanishes for 2l ≤ p ≤ −μ − ρ, so to define intertwining operators for (T̃_{l,μ}^{+}, T_{l,μ}^{−,0}) and for (T̃_{l,−μ}^{−}, T_{l,−μ}^{+,0}). In the second case μ ≥ ρ + 2l, (35) is solved for 2l ≤ p ≤ μ − ρ by (36), which vanishes for p ≥ μ − ρ + 2. Interwining operators for (T̃_{l,μ}^{+}, T_{l,μ}^{−,0}) and for (T̃_{l,−μ}^{−}, T_{l,−μ}^{+,0}) are therefore defined. For p ≥ μ − ρ + 2, (35) is solved by

a_{p,2l} = (−1)^{\frac{β_{−}(p,μ)}{2}} \frac{Γ (−\frac{β_{−}(p,μ)}{2} + 1) Γ (μ + 1)}{Γ (\frac{β_{+}(p,μ)}{2})} a_{μ−ρ+2,2l}.  (39)

This solution gives intertwining operators for (T_{l,μ}^{+,0}, T̃_{l,μ}^{−}) and for (T_{l,−μ}^{−,0}, T̃_{l,−μ}^{+}). We collect the results in the following proposition.

Proposition 4.1. Besides the trivial case of scalar multiples of the identity which intertwine a representation with itself, there exist 𝔤̃-intertwining operators only for the following pairs.

Irreducible case: (T_{l,μ}^{+}, T_{l,μ}^{−}) and (T_{l,μ}^{−}, T_{l,μ}^{+}).

Reducible case: (T_{l,μ}^{+,0}, T̃_{l,μ}^{−}), (T̃_{l,μ}^{−}, T_{l,μ}^{+,0}), (T_{l,μ}^{−,0}, T̃_{l,μ}^{+}) and (T̃_{l,μ}^{+}, T_{l,μ}^{−,0}).

In particular, the following 𝔤̃-representations are equivalent: T_{l,μ}^{+} and T_{l,μ}^{−} (when irreducible); T_{l,μ}^{+,0} and T̃_{l,μ}^{−}; T_{l,μ}^{−,0} and T̃_{l,μ}^{+}.

Observe that a 𝔤̃-intertwining operator Ã for the T_{l,μ}^{±} corresponds to a

𝔤̃-intertwining operator ˜ Ã for the differentiated representations d(π_{l,μ}^{±} ⊗ τ_{l}) ≡ π_{l,μ}^{±} ⊗ I + I ⊗ τ_{l} of 𝔤̃ on (D_{l}(S, τ_{l}) ⊗ V_{l}))Ũ = D_{l}(S, τ_{l})_{U} ⊗ V_{l}. In turn, because of

the irreducibility of τ_{l}, the ˜ Ã’s are exactly the linear operators of the form A ⊗ I for a 𝔤-intertwining operator A : D_{l}(S, τ_{l})_{U} → D_{l}(S, τ_{l})_{U} of the π_{l,μ}^{±}’s.

Let π_{l,μ}^{±,0} and π̃_{l,μ}^{±} respectively denote the restriction of π_{l,μ}^{±} to E_{l,μ} and the factor representation of π_{l,μ}^{±} on Ẽ_{l,μ}. We indicate with the same symbols also the corresponding differentiated representations of 𝔤. Then Proposition 4.1 immediately implies

Corollary 4.2. Besides the trivial case of scalar multiples of the identity which intertwine a representation with itself, there exist 𝔤-intertwining operators only for the following pairs.

Irreducible case: (π_{l,μ}^{+}, π_{l,μ}^{−}) and (π_{l,μ}^{−}, π_{l,μ}^{+}).

Reducible case: (π_{l,μ}^{+,0}, π̃_{l,μ}^{−}), (π̃_{l,μ}^{−}, π_{l,μ}^{+,0}), (π_{l,μ}^{−,0}, π̃_{l,μ}^{+}) and (π̃_{l,μ}^{+}, π_{l,μ}^{−,0}).

In particular, the following 𝔤-representations are equivalent: π_{l,μ}^{+} and π_{l,μ}^{−} (when irreducible); π_{l,μ}^{+,0} and π̃_{l,μ}^{−}; π_{l,μ}^{−,0} and π̃_{l,μ}^{+}.

We now determine which among the T_{l,μ}^{±} and their irreducible subrepresentations and subquotients are unitarizable.

Note first that the standard inner product in L^{2}(S)

(φ|ψ) = ∫_{S} φ(s)\overline{ψ(s)} ds  (40)

is invariant with respect to the pairs (T_{l,μ}^{+}, T_{l,−μ̄}^{+}) and (T_{l,μ}^{−}, T_{l,−μ̄}^{−}). Therefore T_{l,μ}^{±} is always unitarizable when μ̄ = −μ.

In the general case, if a 𝔤̃-invariant Hermitian form (·, ·) exists for T_{l,μ}^{±} (or for some of its subrepresentations or quotients when reducible), then its restriction to each Ũ -type must be a constant multiple of the standard inner product from L^{2}(S), say

(φ, ψ) = a_{p,2l}(φ|ψ) on V^{p,2l}.

Moreover, different Ũ -types must be mutually orthogonal, so that (·, ·) is completely determined by the a_{p,2l} ’s. The operator Ã : D_{l}(S)Ũ → D_{l}(S)Ũ given by Ã|_{V^{p,2l}} = a_{p,2l}I then intertwines (T_{l,μ}^{±}, T_{l,−μ̄}^{±}). Proposition 4.1 implies −μ̄ = μ. Therefore a 𝔤̃-invariant Hermitian form (·, ·) exists on D_{l}(S)Ũ if and only if μ ∈ iℝ, and in this case it is a scalar multiple of the standard inner product of L^{2}(S). A similar analysis can be done for the reducible T_{l,μ}^{±}.

Theorem 4.3. An irreducible T_{l,μ}^{±} (hence π_{l,μ}^{±}) is unitarizable if and only if μ ∈ iℝ. When T_{l,μ}^{±} (or π_{l,μ}^{±}) is reducible, neither its irreducible submodule nor its irreducible quotients are unitarizable.

We remark that the inner product making π_{l,μ}^{±} unitary is the standard inner product in L^{2}(S, V_{l}), that is

(φ|ψ)_{l} := ∫_{S}(φ(s), ψ(s))_{l} ds.  (41)

5. Intertwining operators and the η-function

An intertwining operator for (π_{l,μ}^{±}, π_{l′,μ′}^{±}), with arbitrary choice of signs, is a (nonzero) continuous linear map A : D(S, τ_{l}) → D(S, τ_{l′}) satisfying

A ∘ π_{l,μ}^{±}(g) = π_{l′,μ′}^{±}(g) ∘ A, g ∈ G.

In particular, A maps D(S, τ_{l})_{U} into itself and, by continuity, it intertwines the differentiated representations. Corollary 4.2 implies that in the irreducible case a nontrivial intertwining operator can exist only for (π_{l,μ}^{+}, π_{l,μ}^{−}) and (π_{l,μ}^{−}, π_{l,μ}^{+}). In this case, it is uniquely determined by its restriction to the U -finite vectors, so it is unique up to constant multiples. With the same argument of Lemma 5.3 in [14], it is possible to show that the 𝔤-intertwining operators Ã determined by (36) indeed extend to intertwining operators for (π_{l,μ}^{+}, π_{l,μ}^{−}) and (π_{l,−μ}^{−}, π_{l,−μ}^{+}). We will not pursue this argument further. Rather, we explicitely present them in integral form.

Recall that an operator A_{μ} : D(S, τ_{l}) → D(S, τ_{l}) is said to be holomorphic (resp. meromorphic) in μ if ω(A_{μ}φ) is a holomorphic (resp. meromorphic) function of μ for all φ ∈ D(S, τ_{l}) and all ω ∈ D(S, τ_{l})′, the dual space of D(S, τ_{l}).

For every φ ∈ D(S, τ_{l})
A_{l,μ}φ(s) = ∫_{S} |(s, t)|^{μ−ρ}τ_{l}( \overline{(s, t)}/|(s, t)|) φ(t) dt,  s ∈ S  (42)

defines a norm convergent V_{l} -valued integral provided ℜμ > 2(n + 1). On this region, A_{l,μ} is holomorphic in μ and the regularization of (42) gives a meromorphic extension of A_{l,μ} to ℂ.

It can be easily checked that A_{l,μ} intertwines (π_{l,μ}^{+}, π_{l,μ}^{−}) and (π_{l,−μ}^{−}, π_{l,−μ}^{+}). It follows that A_{l,−μ} ∘ A_{l,μ} intertwines π_{l,μ}^{±} with itself, so it is a scalar multiple of the identity operator:

A_{l,−μ} ∘ A_{l,μ} = η_{l,μ}I  (43)

for some even meromorphic function η_{l,μ}.

We now want to determine the eigenvalues of A_{l,μ} on the U -types and the function η_{l,μ}. Since A_{l,μ} intertwines π_{l,μ}^{+} and π_{l,μ}^{−}, the operator Ã_{l,μ} := β_{l} ∘ (A_{l,μ} ⊗ I) ∘ β_{l}^{−1} intertwines T_{l,μ}^{+} and T_{l,μ}^{−}.

Observe that if f = β_{l}(φ ⊗ v) ∈ D_{l}(S), then

(Ã_{l,μ}f)(s) = ⟨A_{l,μ}φ(s), v⟩_{l} = ∫_{S} |(s, t)|^{μ−ρ}⟨τ_{l} (\frac{\overline{(s, t)}}{|(s, t)|}) φ(t), v⟩_{l} dt.

Setting s = e_{0}, we obtain

(Ã_{l,μ}f)(e_{0}) = ∫_{S} |t_{0}|^{μ−ρ}⟨τ_{l}( \overline{t_{0}}/|t_{0}|)φ(t), v⟩_{l} dt.

For every p ≡ 2l (mod 2), p ≥ 2l, e_{p,2l} is an eigenfunction of Ã_{l,μ}. Let a_{p,l;μ} be the corresponding eigenvalue. a_{p,l;μ} can be determined by evaluating at e_{0} both sides of the equation

Ã_{l,μ}e_{p,2l} = a_{p,l;μ}e_{p,2l}.

Because of (36) it is enough to know γ_{l,μ} := a_{2l,l;μ}. From (18) we have

e_{2l,2l}(s) = β_{l} (∑_{j=0}^{2l} F_{2l,2l;j} ⊗ e_{j}) (s)

with

F_{2l,2l;j}(s) = (−1)^{j}τ_{l}( s_{0}/|s_{0}|)^{−1}(e_{2l−j})|s_{0}|^{2l}.

Hence e_{2l,2l}(e_{0}) = 2l + 1 and

Ã_{l,μ}e_{2l,2l}(e_{0}) = ∫_{S} |t_{0}|^{μ−ρ+2l} ∑_{j=0}^{2l}(−1)^{j}⟨τ_{l}(t_{0}/|t_{0}|)τ_{l}(t_{0}/|t_{0}|)^{−1}e_{2l−j}, e_{j}⟩_{l} dt

= ∫_{S} |t_{0}|^{μ−ρ+2l} [∑_{j=0}^{2l}(−1)^{j}⟨e_{2l−j}, e_{j}⟩_{l}] dt
= (2l + 1) ∫_{S} |t_{0}|^{μ−ρ+2l} dt

Lemma 5.1. Let F : S → ℂ be of the form F (t) = f(|t_{0}|) for some function f ∈ L^{1}(0, 1) and let dt denote the U -invariant measure on S normalized by the condition dt(S) = 1 then

∫_{S} F (t) dt = 4n(2n + 1) ∫_{0}^{1} f(x)(1 − x^{2})^{2n−1}x^{3} dx.

Lemma 5.1 and the integral defining the Beta function therefore give

γ_{l,μ} = Γ(ρ)\frac{Γ (\frac{μ−ρ+2l+4}{2})}{Γ (\frac{μ+ρ+2l}{2})}

a_{p,2l;μ} = Γ(ρ)\frac{Γ (\frac{μ−ρ−2l+2}{2}) Γ (\frac{μ−ρ+2l+4}{2})}{Γ (\frac{μ−ρ−p+2}{2}) Γ (\frac{μ+ρ+p}{2})}, p ≡ 2l (mod 2), p ≥ 2l

η_{l,μ} = γ_{l,μ}γ_{l,−μ} = Γ(ρ)^{2} \frac{Γ (\frac{μ−ρ+2l+4}{2})}{Γ (\frac{μ+ρ+2l}{2})} \frac{Γ (\frac{−μ−ρ+2l+4}{2})}{Γ (\frac{−μ+ρ+2l}{2})}

Observe that, as in Proposition 7.4 of [12], we have

γ_{l,\overline{μ}} = \overline{γ_{l,μ}},
η_{l,μ} = η_{l,−μ},
η_{l,iμ} ≥ 0 for μ ∈ ℝ.

Moreover the normalized intertwining operators A_{l,μ} := γ_{l,μ}^{−1}A_{l,μ} satisfy

A_{l,μ}A_{l,−μ} = I,
A_{l,μ}^{∗} = A_{l,−μ̄} the adjoint being defined U−type by U−type,
A_{l,μ} is a unitarily equivalence between π_{l,μ}^{+} and π_{l,μ}^{−} for μ ∈ iℝ.

References

[1]  Artin, E., “Geometric Algebra,” Interscience, New York, 1957.

[2] Borel, A., and N. Wallach, “Continuous cohomology, discrete subgroups and representations of reductive groups, ” Princeton University Press, Princeton, 1980.

[3] van Dijk, G., and S. C. Hille, Maximal degenerate representations, Berezin kernels and canonical representations, in: B. Komrakov, J. Krasil’shchik, G. Litvinov and A. Sossinky, Editors, “Lie groups and Lie algebras, their representations, generalizations, and applications,” 1–15, Kluwer Academic, Dordrecht, 1997.

[4] van Dijk, G., and V.F. Molchanov, Tensor product of maximal degenerate series representations of the group SL(n, ℝ), J. Math. Pures et Appl. (1997), to appear.

[5] van Dijk, G., and A. Pasquale, Canonical representations of Sp(1, n) associated with representations of Sp(1), Preprint, Leiden University, 1998.

[6] Erdélyi, A., et al, “Higher transcendental functions,” McGraw-Hill, New York, 1953.

[7] Hille, S. C., Maximal degenerate representations of SL(n+1, ℂ), Preprint, Leiden University, 1997.

[8] Howe, R. E., and E. Tan, Homogeneous functions on light cones: the infinitesimal structure of some degenerate principal series representations, Bull. Amer. Math. Soc. 28 (1993), 1–74.

[9] Johnson, K. D., Degenerate principal series and compact groups, Math. Ann. 287 (1990), 703–718.

[10] —, Degenerate principal series on tube domains, Contemporary Math. 138 (1992), 175–187.

[11] Johnson, K. D., and N. R. Wallach, Composition and intertwining operators for the spherical principal series, Trans. Amer. Math. Soc. 229 (1977), 137–173.

[12] Knapp, A. W., and E. M. Stein, Intertwining operators for semisimple Lie groups, II, Inventiones Math. 60 (1980), 9–84.

[13] Lee, S. T., Degenerate principal series representations of Sp(2n, ℝ), Compositio Math. 103 (1996), 123–151.

[14] Molchanov, V. F., Representations of pseudo-orthogonal groups associated with a cone, Math USSR Sbornik 10 (1970), 333–347.

[15] Rossmann, W., Analysis on real hyperbolic spaces, J. Funct. Anal 30 (1978), 448–477.

[16] Smith, R. T., The spherical representations of groups transitive on S^{n}, Indiana Univ. Math. Journal 24 (1974), 307–325.

[17] Zelobenko, D. P., “Compact Lie groups and their representations,” Translations of Mathematical Monographs 40, Amer. Math. Soc., Providence, 1973.

Fachbereich Mathematik Technische Universität Erzstrasse 1 D-38678 Clausthal-Zellerfeld Germany mapa@math.tu-clausthal.de

Received May 5, 1998 and in final form September 30, 1998