Journal of Lie Theory Volume 12 (2002) 409–421 C ◯2002 Heldermann Verlag

Vanishing Properties of Analytically Continued Matrix Coefficients

Bernhard Krötz^{∗} and Michael Otto

Communicated by K.-H. Neeb

Abstract. We consider (generalized) matrix coefficients associated to irreducible unitary representations of a simple Lie group G which admit holomorphic continuation to a complex semigroup domain S⊆G_{ℂ}. Vanishing theorems for these analytically continued matrix coefficients, one of Howe-Moore type and one for cusp forms, are proved.

Introduction

Recall the Howe-Moore Theorem (cf. [9]; see also [18] and [20]) on the vanishing of matrix coefficients:

Theorem. Let G be a semisimple Lie group with no compact simple factors and compact center. If (π, H) is a non-trivial irreducible unitary representation of G, then for all v, w ∈ H one has

lim_{g→∞}⟨π(g).v, w⟩ = 0.

Now, if G happens to be hermitian and (π, H) is a unitary highest weight representation of G, then it was discovered by Olshanski and Stanton (cf. [16], [19]) that (π, H) analytically extends to a complex G × G-biinvariant domain S ⊆ G_{ℂ}. These domains turn out to be complex semigroups, so-called complex Olshanski semigroups. There is a maximal one S_{max} which is the compression semigroup of the bounded symmetric domain G/K ⊆ G_{ℂ}/K_{ℂ}P^{+} . Here G ⊆ P^{−}K_{ℂ}P^{+} denotes the Harish-Chandra decomposition. Hence one always has S ⊆ P^{−}K_{ℂ}P^{+} . Our interest however lies in the minimal complex Olshanski semigroup which is given by

S_{min} = G exp(iW_{min})

with W_{min} a minimal Ad(G)-invariant closed convex cone in Lie(G) of nonempty interior. Our first result is (cf. Theorem 2.5):

∗ Supported in part by the NSF-grant DMS-0097314

ISSN 0949–5932 / $2.50 ^{_{C} ◯} Heldermann Verlag Theorem A. (Vanishing at infinity of analytically continued matrix coefficients) Let G be a linear hermitian group and (π_{λ}, H_{λ}) a unitary highest weight representation of G analytically continued to the minimal complex Olshanski semigroup S_{min} . Then for all v, w ∈ H_{λ} we have that

lim_{{s→∞ \atop s∈S_{min}}} ⟨π_{λ}(s).v, w⟩ = 0,

i.e, the analytically continued matrix coefficients s ↦ ⟨π_{λ}(s).v, w⟩, s ∈ S_{min} , vanish at infinity.

It is interesting to observe that the proof of this theorem relies on geometric facts only: firstly that the middle projection κ: P^{−}K_{ℂ}P^{+} → K_{ℂ} restricted to S_{min} is a proper mapping (cf. Proposition 1.2) and secondly an explicit description of κ(S_{min}) (cf. Corollary 2.4). Since G ⊆ S_{min} is closed, our methods imply a simple new proof of the Howe-Moore Theorem for the special case of unitary highest weight representations.

Let now Γ < G be a lattice and η ∈ (H_{λ}^{−∞})^{Γ} a Γ-invariant distribution vector for (π_{λ}, H_{λ}). Then for all K -finite vectors v of (π_{λ}, H_{λ}) the prescription

θ_{v,η}: Γ\G → ℂ, Γg ↦ ⟨π_{λ}(g).v, η⟩: = \overline{η(π_{λ}(g).v)}

defines an automorphic form of Γ\G. One can show that θ_{v,η} naturally extends to a function on Γ\S_{min} ⊆ Γ\G_{ℂ}. We denote this extension by the same symbol. Then our next result is (cf. Theorem 3.3):

Theorem B. (Vanishing at infinity of analytically continued automorphic forms) Let Γ < G be a lattice and η ∈ (H_{λ}^{−∞}) a cuspidal element for a non-trivial unitary highest weight representation (π_{λ}, H_{λ}) of the hermitian Lie group G. Then for all K -finite vectors v of (π_{λ}, H_{λ}) the analytically continued automorphic forms θ_{v,η} vanish at infinity:

lim_{{Γs→∞ \atop Γs∈Γ\S_{min}}} θ_{v,η}(Γs) = 0.

Theorem B has applications to complex analysis. For example it implies that the bounded holomorphic functions on Γ\ int S_{min} separate the points (cf. [1]).

For G = Sl(2, ℝ) the results in this paper were first proved in the diplome thesis of the second named author (cf. [17]).

It is our pleasure to thank the referee for his careful work.

1. Preliminaries on hermitian Lie groups

Let 𝔤 be a real semisimple Lie algebra with Cartan decomposition 𝔤 = 𝔨 ⊕ 𝔭. Then 𝔤 is called hermitian if 𝔤 is simple and 𝔷(𝔨) ≠ {0}. Here 𝔷(𝔨) denotes the center of 𝔨.

Hermitian Lie algebras are classified. The complete list is as follows (cf. [6, p. 518]):

𝔰𝔲(p, q), 𝔰𝔬^{∗}(2n), 𝔰𝔭(n, ℝ), 𝔰𝔬(2, n), 𝔢_{6(−14)}, 𝔢_{7(−25)}.

That 𝔤 is hermitian implies in particular that 𝔷(𝔨) = ℝX_{0} is one dimensional, and after a renormalization of X_{0} we can assume that

Spec(ad X_{0}) = {−i, 0, i}

(cf. [6, Ch. VIII]). If 𝔩 is a Lie algebra we denote by 𝔩_{ℂ} its complexification. The spectral decomposition of ad X_{0} then reads as follows

𝔤_{ℂ} = 𝔭^{+} ⊕ 𝔨_{ℂ} ⊕ 𝔭^{−}

with 𝔭^{±} = {X ∈ 𝔤_{ℂ}: [X_{0}, X] = ∓iX}. Note that 𝔭^{±} are abelian, [𝔨_{ℂ}, 𝔭^{±}] ⊆ 𝔭^{±} and 𝔭_{ℂ} = 𝔭^{+} ⊕ 𝔭^{−}.

We extend 𝔷(𝔨) to a compact Cartan subalgebra 𝔱 of 𝔤. We may assume that 𝔱 ⊆ 𝔨. Let Δ = Δ(𝔤_{ℂ}, 𝔱_{ℂ}) denote the root system with respect to 𝔱_{ℂ}. Then

𝔤_{ℂ} = 𝔱_{ℂ} ⊕ ⨁_{α∈Δ} 𝔤_{ℂ}^{α}

with 𝔤_{ℂ}^{α} the root spaces.

A root α ∈ Δ is called compact if α(X_{0}) = 0 and non-compact otherwise. The collection of compact roots, resp. non-compact roots, is denoted by Δ_{k}, resp. Δ_{n}. Note that Δ = Δ_{k} ˙∪Δ_{n} and that α ∈ Δ_{k} if and only if 𝔤_{ℂ}^{α} ⊆ 𝔨_{ℂ} and α ∈ Δ_{n} iff 𝔤_{ℂ}^{α} ⊆ 𝔭_{ℂ}.

If Δ^{+} is a positive system of Δ we set Δ^{−} = −Δ^{+} , Δ_{k}^{±} = Δ_{k} ∩ Δ^{±} and Δ_{n}^{±} = Δ_{k} ∩ Δ^{±}. We can choose Δ^{+} such that

Δ_{n}^{+} = {α ∈ Δ: α(X_{0}) = −i}.

Note that 𝔭^{±} = ⊕_{α∈Δ_{n}^{±}}𝔤_{ℂ}^{α}.

If 𝔩 is a Lie algebra and 𝔞 < 𝔩 is a subalgebra of 𝔩, then we define Inn_{𝔤}(𝔞): = ⟨e^{ad X}: X ∈ 𝔞⟩.

Define the little Weyl group of (𝔤, 𝔱) by W_{𝔨}: = N_{Inn_{𝔨}(𝔱)}/Z_{Inn_{𝔨}(𝔱)} . If α ∈ Δ we write α̌ ∈ i𝔱 for its coroot, i.e., α̌ ∈ [𝔤_{ℂ}^{α}, 𝔤_{ℂ}^{−α}] ⊆ 𝔱_{ℂ} with α(α̌) = 2.

If X is a topological space and Y ⊆ X, then we write cl Y for the closure and int Y for the interior of Y . If V is a vector space and E ⊆ V , then we write conv E for the convex hull of E and cone E for the convex cone generated by E.

Define the minimal cone in 𝔱 by

C_{min}: = cl(cone{−iα̌: α ∈ Δ_{n}^{+}}).

Note that C_{min} is a W_{𝔨} -invariant convex cone with non-empty interior in 𝔱. Define the minimal cone in 𝔤 by

W_{min}: = cl ( conv(Inn(𝔤).ℝ^{+}X_{0})).

Note that W_{min} is a convex Inn(𝔤)-invariant cone in 𝔤 with non-empty interior and W_{min} ∩ 𝔱 = C_{min} (cf. [7, Sect. 7]). In the sequel we set W: = int W_{min} . Then cl W = W_{min} .

We write G for a linear connected Lie group with Lie algebra 𝔤. Then G ⊆ G_{ℂ} with G_{ℂ} the universal complexification of G. The prescription

S: = G exp(iW)

defines a subsemigroup of G_{ℂ}, a so-called complex Olshanski semigroup. The closure of S is given by cl S = G exp(i cl W). This is a consequence of Lawson’s Theorem which states that the polar mapping

G × cl W → cl S, (g, X) ↦ g exp(iX)

is a homeomorphism (cf. [13] or [15, Th. XI.1.7]).

Write G_{ℂ} → G_{ℂ}, g ↦ \overline{g} for the complex conjugation of G_{ℂ} with respect to the real form G. Then the prescription

cl S → cl S, s = g exp(iX) ↦ s^{∗}: = \overline{s}^{−1} = exp(iX)g^{−1}

defines an involution on cl S which is antiholomorphic when restricted to the open subset S of G_{ℂ}.

Write K, K_{ℂ}, P^{+} and P^{−} for the analytic subgroups of G_{ℂ} corresponding to 𝔨, 𝔨_{ℂ}, 𝔭^{+} and 𝔭^{−}. A theorem of Harish-Chandra states that the multiplication mapping

P^{−} × K_{ℂ} × P^{+} → G_{ℂ}, (p^{−}, k, p^{+}) ↦ p^{−}kp^{+}

is a biholomorphism onto its open image and that G ⊆ P^{−}K_{ℂ}P^{+} (cf. [6, Ch. VIII]). If s ∈ P^{−}K_{ℂ}P^{+} , then s = l^{−}(s)κ(s)l^{+}(s) with holomorphic maps l^{±}: P^{−}K_{ℂ}P^{+} → P^{±} and κ: P^{−}K_{ℂ}P^{+} → K_{ℂ}. The Harish-Chandra realization D ⊆ 𝔭^{−} of the hermitian symmetric space G/K is the image of the injective holomorphic map

ζ: G/K → 𝔭^{−}, gK ↦ log l^{−}(g).

Note that D is a bounded symmetric domain (cf. [6, Ch. VIII]). The compression semigroup of D is defined by

comp(D): = {g ∈ G_{ℂ}: g.D ⊆ D}
= {g ∈ G_{ℂ}: g exp(D)K_{ℂ}P^{−} ⊆ exp(D)K_{ℂ}P^{−}}.

Then the G-biinvariance of comp(D) together with exp(iℝ^{+}X_{0}) ⊆ comp(D) imply that

cl S ⊆ comp(D).

This was first realized by Olshanski (cf. [16] or [15, Th. XII.3.3]).

The idea behind the following Lemma is not new and can also be found in [8].

Lemma 1.1. We have

cl S ⊆ exp(D)K_{ℂ}\overline{exp(D)}

with \overline{exp(D)} ⊆ P^{+} the complex conjugate of exp(D).

Proof. Since cl S compresses D, we conclude that

cl S ⊆ exp(D)K_{ℂ}P^{+}.

Now cl(S) is ∗-invariant and so together with D = −D we get that

cl S = (cl S)^{∗} ⊆ P^{−}K_{ℂ}\overline{exp(D)}.

Finally

cl S ⊆ exp(D)K_{ℂ}P^{+} ∩ P^{−}K_{ℂ}\overline{exp(D)} = exp(D)K_{ℂ}\overline{exp(D)}.

Proposition 1.2. The middle projection restricted to cl S

κ: cl S → K_{ℂ}, s ↦ κ(s)

is a proper mapping.

Proof. Let A ⊆ K_{ℂ} be a compact subset. Then κ^{−1}(A) is closed in cl S by the continuity of κ. By Lemma 1.1 we have that κ^{−1}(A) ⊆ exp(D)A\overline{exp(D)} and the latter set is relatively compact in G_{ℂ} by the boundedness of D. Hence the assertion follows.

Remark 1.3. There are many other interesting complex Olshanski semigroups than the one associated to the minimal cone. There is a distinguished maximal cone W_{max} characterized by

C_{max}: = W_{max} ∩ 𝔱 = {X ∈ 𝔱: (∀α ∈ Δ_{n}^{+}) α(iX) ≥ 0}

and with it comes a continuous family of closed convex Inn(𝔤)-invariant cones W_{0} lying between W_{min} and W_{max} :

W_{min} ⊆ W_{0} ⊆ W_{max}.

To each W_{0} one can associate a complex Olshanski semigroup

S_{0} = G exp(i int W_{0})

featuring the same properties as S. In particular Lemma 1.1 and Proposition 1.2 remain true for cl S_{0} . One has S_{max} = G exp(iW_{max}) = comp(D) (cf. [7, Th. 8.49]). However, for the applications we have in mind, namely vanishing properties of matrix coefficients on S and Γ\S, the assumption on the minimalilty of the cone is crucial. For more details we refer to [15, Sect. VII.3, Ch. X-XI].

2. Matrix coefficients on S

In the sequel (π_{λ}, H_{λ}) denotes a unitary highest weight representation of G with highest weight λ ∈ i𝔱^{∗} and with respect to the positive system Δ^{+} . We refer to [15, Ch. XI] for more on unitary highest weight representations.

Let H be a Hilbert space with bounded operators B(H). By a holomorphic representation of S we understand a holomorphic semigroup homomorphism

π: S → B(H)

which in addition satisfies π(s^{∗}) = π(s)^{∗} for all s ∈ S.

If V is a finite dimensional real vector space, V^{∗} its dual and C ⊆ V a subset, then we define the dual cone of C by

C^{⋆}: = {α ∈ V^{∗}: (∀X ∈ C) α(X) ≥ 0}.

Note that C is a closed convex subcone of V^{∗}.

The central ideas of part (ii) in the next theorem go back to Olshanski and Stanton (cf. [16], [19]); a very systematic approach to these results is due to Neeb (cf. [14]).

Theorem 2.1. Let G be a hermitian Lie group and S an associated minimal complex Olshanski semigroup. Then for every non-trivial unitary highest weight representation of G the following statements hold:

(i) λ ∈ i int C_{min}^{⋆}.

(ii) (π_{λ}, H_{λ}) extends to a strongly continuous and contractive representation π_{λ}: cl S → B(H_{λ}) with π_{λ} |_{S} a holomorphic representation.

Proof. (i) [15, Th. IX.2.17]. (ii) This follows from (i) and [15, Th. XI.4.8].

We now take a closer look at the inclusion cl S ⊆ P^{+}K_{ℂ}P^{−} and prove a refinement of Lemma 1.1. This will be accomplished with tools provided by representation theory.

Let (π_{λ}, H_{λ}) be a unitary highest weight representation of G. In view of Theorem 2.1(ii) we henceforth consider (π_{λ}, H_{λ}) as a representation of cl S. We denote by V_{λ} ⊆ H_{λ} the space of K -finite vectors. Since every vector in V_{λ} is 𝔭^{+} -finite we have a natural representation σ_{λ} of the semidirect product group K_{ℂ} ⋊ P^{+} on V_{λ} obtained by exponentiating the derived representation dπ_{λ} |_{𝔨_{ℂ}⋊𝔭^{+}} .

If v_{λ} ∈ V_{λ} is a highest weight vector, then we set

F (λ): = span_{ℂ}{π_{λ}(K).v_{λ}}

for the finite dimensional subspace of the highest K -type.

Lemma 2.2. Let (π_{λ}, H_{λ}) be a unitary highest weight representation of G. Then we have for all s ∈ cl S and v, w ∈ F (λ) that

⟨π_{λ}(s).v, w⟩ = ⟨σ_{λ}(κ(s)).v, w⟩.

Proof. This follows from [11, Prop. 2.20].

We write HW (G) for those λ ∈ i𝔱^{∗} for which there exists a unitary highest weight representation of G with respect to Δ^{+} . Recall that HW (G) ⊆ i int C_{min}^{⋆} ∪ {0} (cf. Theorem 2.1(i)). Moreover, from our knowledge on the unitarizable highest weight modules for G we have

(2.1)  iHW (G)^{⋆} = C_{min}

(cf. [10, Lemma II.5]; this follows basically from the fact that HW (G) contains a subset of the form Γ ∩ (x + i int C_{min}^{⋆}) with Γ ⊆ i𝔱^{∗} a vector lattice and x ∈ i𝔱^{∗} a certain element). Write W_{K}: = Ad(K).C_{min} and note that W_{K} is a convex cone, a consequence of Kostant’s convexity theorem. Define now the semigroup

S_{K}: = K exp(iW_{K}) = K exp(iC_{min})K ⊆ K_{ℂ}

and note that

S_{K} ⊆ cl S.

Proposition 2.3. The following inclusion holds

cl S ⊆ exp(D)S_{K} \overline{exp(D)}.

Proof. We define

U: = ⋂_{λ∈HW (G)}{k ∈ K_{ℂ}: σ_{λ}(k)|_{F (λ)} is a contraction}.

Note that Lemma 2.2 together with Lemma 1.1 and the fact that the representation (π_{λ}, H_{λ}) of cl S is contractive (cf. Theorem 2.1(ii)) imply that cl S ⊆ exp(D)U\overline{exp(D)}. Hence it is sufficient to show that U = S_{K}.

From the definition of U it is clear that U is K -biinvariant and so U = K exp(iC)K with C ⊆ i𝔱 a convex cone (note that 𝔱 is abelian). By a theorem of Kostant we know that the 𝔱_{ℂ} -weight spectrum of F (λ) is contained in conv(W_{𝔨}.λ). Thus we obtain that

U = K( ⋂_{λ∈HW (G)} exp({X ∈ i𝔱: (∀w ∈ W_{𝔨}) (w.λ)(X) ≤ 0}))K,

and so (2.1) implies that C = C_{min} , concluding the proof of the proposition.

Corollary 2.4. We have that κ(cl S) = S_{K}. Proof. Since S_{K} ⊆ cl S ∩ K_{ℂ} the inclusion ′′ ⊇′′ is clear. The converse inclusion follows from Proposition 2.3.

We now come to the main result of this Section.

Theorem 2.5. (Vanishing at infinity of analytically continued matrix coefficients) Let (π_{λ}, H_{λ}) be a unitary highest weight representation of G analytically continued to cl S. Then for all v, w ∈ H_{λ} we have that

lim_{{s→∞ \atop s∈cl S}} ⟨π_{λ}(s).v, w⟩ = 0,

i.e, the analytically continued matrix coefficients ⟨π_{λ}(s).v, w⟩, s ∈ cl S, vanish at infinity. Proof. Since V_{λ} ⊆ H_{λ} is a dense subspace and ∥π_{λ}(s)∥ ≤ 1 for all s ∈ cl S, it is sufficient to prove the theorem for v, w ∈ V_{λ}. For v, w ∈ V_{λ} the proof of [11, Prop. 2.20] shows that

⟨π_{λ}(s).v, w⟩ = ⟨σ_{λ}(κ(s))σ_{λ}(l^{+}(s)).v, σ_{λ}(\overline{l^{−}(s)}^{−1}).w⟩.

Write l^{+}(s) = exp(X), \overline{l^{−}(s)}^{−1} = exp(Y) for elements X, Y ∈ \overline{D} ⊆ 𝔭^{+} (cf. Lemma 1.1). Hence there exists an N ∈ ℕ, independent from s ∈ cl S, such that

⟨π_{λ}(s).v, w⟩ = ∑_{j,k=1}^{N} \frac{1}{j!k!}⟨σ_{λ}(κ(s))dπ_{λ}(X)^{j}.v, dπ_{λ}(Y)^{k}.w⟩.

Note that

sup_{{1≤j,k≤N \atop s∈cl S}} \{∥dπ_{λ}(X)^{j}.v∥, ∥dπ_{λ}(Y)^{k}.w∥\} < ∞

since \overline{D} is bounded. Hence it is sufficient to show that

(2.2)  ⟨σ_{λ}(κ(s)).v, w⟩ → 0

for s → ∞ in cl S and v, w ∈ V_{λ}. As κ: cl(S) → K_{ℂ} is proper by Proposition 1.2, Corollary 2.4 implies that (2.2) is equivalent to

(2.3)  lim_{{s→∞ \atop s∈S_{K}}} ⟨σ_{λ}(s).v, w⟩ = 0

for all v, w ∈ V_{λ}.

Now we make a final reduction from which the theorem will follow. Write C_{min}^{+}: = {X ∈ C_{min}: (∀α ∈ Δ^{+}) iα(X) ≥ 0} and note that C_{min}^{+} is a fundamental domain in C_{min} for the W_{𝔨} -action (see also Remark 1.3 for the inclusion C_{min} ⊆ C_{max} which is needed here).

Since S_{K} = K exp(iC_{min})K, we obtain that S_{K} = K exp(iC_{min}^{+})K. Hence the fact that K is compact, and v, w are K -finite implies that (2.3) is equivalent to

(2.4)  lim_{{X→∞ \atop X∈C{+ \atop min}}} ⟨σ_{λ}(exp(iX)).v, w⟩ = 0

for all v, w ∈ V_{λ}. W.l.o.g. we may assume that v, w are 𝔱_{ℂ} -weight vectors. Recall that

Spec(dπ_{λ} |_{𝔱_{ℂ}}) ⊆ λ − ℕ_{0}[Δ^{+}].

The fact that λ(iX) < 0 for all X ∈ C_{min}\{0} (cf. Theorem 2.1(i)) proves (2.4) and hence the theorem.

3. Analytic continuation of holomorphic automorphic forms

Let H_{λ}^{∞} be the G-Fréchet module of smooth vectors of (π_{λ}, H_{λ}). Then the strong antidual (the space of antilinear continuous functionals equipped with the strong topology) of H_{λ}^{∞} is denoted by H_{λ}^{−∞} and we refer to it as the space of distribution vectors of (π_{λ}, H_{λ}). Recall the chain of continuous inclusions

H_{λ}^{∞} ↪ H_{λ} ↪ H_{λ}^{−∞}.

For a discrete subgroup Γ < G we write (H_{λ}^{−∞})^{Γ} for the Γ-invariants of H_{λ}^{−∞}. If η ∈ (H_{λ}^{−∞})^{Γ} and v ∈ H_{λ}^{∞}, then we consider the general matrix coefficient

θ_{v,η}: Γ\G → ℂ, Γg ↦ ⟨π_{λ}(g).v, η⟩: = \overline{η(π_{λ}(g).v)}.

Note that θ_{v,η} ∈ C^{∞}(Γ\G).

Since Γ acts properly discontinuously on G_{ℂ}, we get Hausdorff quotients Γ\S, Γ\ cl S ⊆ Γ\G_{ℂ}. Note that Γ\S is also a complex submanifold of Γ\G_{ℂ}.

In view of the results of [12, App.], we have π_{λ}(cl S).H_{λ}^{∞} ⊆ H_{λ}^{∞} and so the functions θ_{v,η} naturally extend to functions on Γ\ cl S. We denote these extensions also by θ_{v,η}. Note that θ_{v,η} |_{Γ\S} is a holomorphic map since π_{λ}(S).H_{λ}^{−∞} ⊆ H_{λ} (cf. [12, App.]).

Remark 3.1. If v ∈ V_{λ} is a K -finite vector of (π_{λ}, H_{λ}), then θ_{v,η} |_{Γ\G} is an automorphic form in the sense of Borel and Wallach (cf. [Wal92, 11.9.1]).

If v ∈ F (λ), then θ_{v,η} is a so-called holomorphic automorphic form (cf. [2, §6]).

From now on Γ < G denotes a lattice, i.e, Γ is a discrete subgroup with 12(Γ\G) < ∞. We call an element η ∈ (H_{λ}^{−∞})^{Γ} cuspidal if for all v ∈ V_{λ} the automorphic form θ_{v,η} |_{Γ\G} is a cusp form (cf. [5, Ch. I, §4] for the definition of cusp forms).

Remark 3.2. The definition of cusp forms is technical and we restrained to give it here and refered to [5] instead. However, some remarks are appropriate.

(a) In [5] automorphic forms are defined for arithmetic lattices Γ < G only. In view of more recent results, this is no major constraint anymore: Margulis’ “arithmeticity theorem” (cf. [21, Th. 6.1.2]) implies that every lattice is arithmetic if rank_{ℝ}(G) ≥ 2; if rank_{ℝ}(G) = 1, then the difficulties (in particular the existence of a Siegel set) can be overcome by the work of Garland and Raghunathan (cf. [4]).

(b) If η ∈ (H_{λ}^{−∞})^{Γ} such that θ_{v,η} |_{Γ\G} belongs to L^{2}(Γ\G) for all v ∈ V_{λ}, then η is cuspidal. This is a special feature related to holomorphic automorphic forms; a conceptual proof of this fact for the group G = Sl(2, ℝ) is for example given in [3, Cor. 7.10].

(c) In [1, Th. 3.11] it is shown that the Poincaré series P (v_{λ}) of v_{λ}

P (v_{λ}) = ∑_{γ∈Γ} π_{λ}(γ).v_{λ}

converges for almost all parameters λ in the module of hyperfunction vectors H_{λ}^{−ω} to a non-zero Γ-fixed element. Since convergent Poincaré series define cuspidal elements (cf. [3, Th. 8.9]), the existence of sufficiently many non-trivial cuspidal elements is hence guaranteed.

Theorem 3.3. (Vanishing at infinity of analytically continued automorphic forms) Let Γ < G be a lattice and η ∈ (H_{λ}^{−ω})^{Γ} a cuspidal element for a nontrivial unitary highest weight representation of the hermitian Lie group G. Then for all K -finite vectors v ∈ V_{λ} the analytically continued automorphic forms θ_{v,η} vanish at infinity:

lim_{{Γs→∞ \atop Γs∈Γ\ cl S}} θ_{v,η}(Γs) = 0.

Remark 3.4. (a) For Γ < G cocompact Theorem 3.2 was proved in [1] with different methods coming from representation theory.

(b) Theorem 3.2 together with [1, Th. 4.7] implies in particular that the bounded holomorphic functions on Γ\S separate the points. Here it might by interesting to observe that the surrounding complex manifold Γ\G_{ℂ} admits no holomorphic functions except the constants: Hol(Γ\G_{ℂ}) = ℂ1. For more information we refer to [1].

Proof of Theorem 3.3. First we reduce the assertion of the theorem to the case where v = v_{λ} is a highest weight vector. Assume that θ_{v_{λ},η} vanishes at infinity on Γ\ cl S. Then it follows that θ_{v,η} vanishes at infinity for all v ∈ E_{λ}: = span_{ℂ}{π_{λ}(G).v_{λ}}. Note that E_{λ} is dense in H_{λ} since (π_{λ}, H_{λ}) is irreducible.

If (χ, U_{χ}) is an irreducible representation of K, then we write V_{λ}^{[χ]} for the χ-isotypical part of the K -module V_{λ}. By the density of E_{λ} ⊆ H_{λ} we conclude that the orthogonal projection

P_{χ}: E_{λ} → V_{λ}^{[χ]}, v ↦ \frac{1}{dim U_{χ}} ∫_{K} \overline{tr χ(k)}π(k).v dk

is onto. In particular, if v ∈ V_{λ}^{[χ]} with v = P_{χ}(w) for some w ∈ E_{λ}, then we have

θ_{v,η}(Γs) = \frac{1}{dim U_{λ}} ∫_{K} \overline{tr χ(k)}θ_{π(k).w,η}(s) dk.

Hence the compactness of K implies that θ_{v,η} vanishes at infinity, completing the proof of our reduction.

We now show that θ_{v_{λ},η} vanishes at infinity. First we need some notation. Write

p_{F (λ)}: H_{λ}^{−∞} → F (λ)

for the projection onto the highest K -type along the other K -types. Define the function

f: G → F (λ), g ↦ p_{F (λ)}(π_{λ}(g^{−1}).η).

Note that f is smooth, left Γ-invariant and that

θ_{v_{λ},η}(Γg) = ⟨v_{λ}, f(g)⟩ (g ∈ G).

Further we define

μ_{λ}(s): = σ_{λ}(κ(s))|_{F (λ)} ∈ Gl(F (λ)) (s ∈ cl S).

Then on D ≅ G/K the prescription

(3.1)  F (gK): = μ_{λ}(g^{−1})^{−1}f(g)  (g ∈ G)

defines an anti-holomorphic function on D (cf. [2, §6]).

We claim that F is bounded. Let ∥ · ∥ be a norm on G_{ℂ}. Denote by S ⊆ G a Siegel set for Γ. Recall that a Siegel set has the properties that ΓS = G and |ΓS ∩S| < ∞. Then the fact that θ_{v,η} is a cusp form for all v ∈ F (λ) implies that there exists for all N ∈ ℕ a constant C = C_{N} > 0 such that

(3.2)  (∀g ∈ S)  |θ_{v,η}(Γg)| ≤ C_{N}∥v∥ · ∥g∥^{−N}

(cf. [3, Th. 7.5] for G = Sl(2, ℝ) and [5, Ch. I, Lemma 10] for the general case). By Lemma 1.1 there exists constants C_{1}, C_{2} > 0 such that C_{1}∥g∥ ≤ ∥κ(g^{−1})^{−1}∥ ≤ C_{2}∥g∥ for all g ∈ G. Hence there exists an M ∈ ℕ and a constant C > 0 such that ∥μ_{λ}(g^{−1})^{−1}∥ ≤ C∥g∥^{M}. In view of (3.1) and (3.2), our claim now follows.

From (3.1) we get that

f(g) = μ_{λ}(g^{−1})F (gK)

and so

(3.3)  θ_{v_{λ},η}(Γg) = ⟨v_{λ}, μ_{λ}(g^{−1})F (gK)⟩.

Write F̃: cl S → F (λ), s ↦ F (s.K) and note that F̃ is anti-holomorphic on S (Recall that cl S.D ⊆ D). Thus analytic continuation of (3.3) yields

(3.4)  θ_{v_{λ},η}(Γs) = ⟨v_{λ}, μ_{λ}(s^{∗}).F̃(s)⟩

Since F is bounded, F̃ is bounded. By (3.4) it hence suffices to show μ_{λ}(s^{∗}) → 0 for s → ∞ in cl S. But since s ↦ s^{∗} is a homeomorphism of cl S this now follows from Proposition 1.2 and Proposition 2.3.

References

[1] Achab, D., F. Betten, and B. Krötz, Discrete group actions on Stein domains in complex Lie groups, submitted.

[2] Borel, A., Introduction to automorphic forms, Proc. Symp. Pure Math. 9, Amer. Math. Soc. (1966), 199–210.

[3] —, “Automorphic Forms on Sl(2, ℝ),” Cambridge Tracts in Mathematics 130, Cambridge University Press, 1997.

[4] Garland, H., and R.S. Raghunathan, Fundamental domains for lattices in (R-)rank 1 semisimple Lie groups, Ann. of Math. 92:2 (1970), 279–326.

[5] Harish-Chandra, “Automorphic Forms on Semisimple Lie Groups,” Springer Lecture Notes in Mathematics 62, Springer, 1968.

[6] Helgason, S., “Lie Groups, Differential Geometry and Symmetric Spaces,” Academic Press, London, 1978.

[7] Hilgert, J., and K.-H. Neeb, “Lie Semigroups and their Applications,” Lecture Notes in Math. 1552, Springer Verlag, Berlin, Heidelberg, New York, 1993.

[8] — , Maximality of compression semigroups, Semigroup Forum 50 (1995), 205–222.

[9] Howe, R., and C. C. Moore, Asymptotic properties of unitary representations, J. Funct. Analysis 31 (1979), 72–96.

[10] Krötz, B., Equivariant embeddings of Stein domains sitting inside of complex semigroups, Pacific J. Math. 189:1 (1999), 55–73.

[11]  —, On the dual of complex Ol’shanskii semigroups, Math. Z., to appear.

[12] Krötz, B., K.-H. Neeb, and G. Ólafsson, Spherical representations and mixed symmetric spaces, Represent. Theory 1 (1997), 424–461.

[13] Lawson, J., Polar and Olshanski type decompositions, J. Reine ang. Math. 448 (1994), 183–202.

[14] Neeb, K.–H., Holomorphic representation theory II, Acta math. 173:1 (1994), 103–133.

[15] —, “Holomorphy and Convexity in Lie Theory,” Expositions in Mathematics 28, de Gruyter, Berlin, 2000.

[16] Olshanski, G. I., Invariant cones in Lie algebras, Lie semigroups, and the holomorphic discrete series, Funct. Anal. and Appl. 15 (1982), 275–285.

[17] Otto, M., Vanishing properties of generalized matrix-coefficients, Diplome Thesis, TU Clausthal, 2001.

[18] Sherman, T., A weight theory for unitary representations, Canad. J. Math. 18 (1966), 159–168.

[19] Stanton, R. J., Analytic extension of the holomorphic discrete series, Amer. J. Math. 108 (1986), 1411–1424.

[20] Zimmer, R. J., Orbit spaces of unitary representations, ergodic theory, and simple Lie groups, Ann. Math. (2) 106 (1977), no. 3, 573–588.

[21] —, “Ergodic Theory and Semisimple Lie Groups,” Monographs in Mathematics 81, Birkhäuser, 1984.

B. Krötz, M. Otto The Ohio State University Department of Mathematics 231 West 18th Avenue Columbus, OH 43210-1174 USA kroetz@math.ohio-state.edu otto@math.ohio-state.edu

Received April 4, 2001 and in final form June 12, 2001