Journal of Lie Theory Volume 9 (1999) 383–402 C ◯1999 Heldermann Verlag

Invariant Hilbert spaces of holomorphic functions

J. Faraut and E. G. F. Thomas

Communicated by J. Hilgert

Abstract. A Hilbert space of holomorphic functions on a complex manifold Z, which is invariant under a group G of holomorphic automorphisms of Z, can be decomposed into irreducible subspaces by using Choquet theory. We give a geometric condition on Z and G which implies that this decomposition is multiplicity free, with application to several examples.

In a paper of 1947 Cartan and Godement gave a proof of the classical Bochner theorem as an application of Krein-Milman Theorem [4]. The same idea was used by van Dijk in his thesis for proving the Bochner-Godement theorem for Gelfand Pairs [6]. This was extended by Thomas to generalized Gelfand Pairs [29,30]. There Choquet integral representation theory was used, which complements the Krein-Milman Theorem [28]. In the setting of invariant Hilbert spaces of holomorphic functions it is also possible to use a recent version of Choquet theory [31].

We consider a complex manifold Z with an action of a group G. The set of invariant Hilbert spaces of holomorphic functions can be identified with a convex cone Γ_{G}(Z) in a topological vector space. The extremal rays correspond to irreducible Hilbert spaces and the integral representation leads to the decomposition of an invariant Hilbert subspace of holomorphic functions into irreducible subspaces. We give a geometric condition on Z and G which implies that this decomposition is multiplicity free. This is our main result. Then we give several examples of multiplicity free decompositions, not all of which require the general theory for their proof.

This result has been extended by Kobayashi [13] to the case of equivariant holomorphic line bundles, and this extension is applied to study the decomposition of a unitary representation of a simple Lie group G when restricted to a subgroup H for a symmetric pair (G, H).

The paper [18], which was published as this article was in preparation, is closely related to our subject, and so is also the paper [14].

ISSN 0949–5932 / $2.50 ^{_{C} ◯} Heldermann Verlag

1. General setting

Let Z be a connected complex manifold, and O(Z) be the space of holomorphic functions on Z, equipped with the topology of uniform convergence on compact sets. Let G be a group of holomorphic automorphisms of Z. The action of G on O(Z) is given by

(π(g)f)(z) = f(g^{−1} · z).

A Hilbert subspace of O(Z) is a Hilbert space H with a continuous injection H ⊂→ O(Z). Such a space has a reproducing kernel. In fact, for z ∈ Z, the point evaluation

f ↦ f(z), H → ℂ (z ∈ Z),

is continuous, and, by the Riesz Representation Theorem, there exists K_{z} ∈ H such that

f(z) = (f|K_{z}) (f ∈ H).

The reproducing kernel K of H is defined by

K(z, w) = K_{w}(z).

It is holomorphic in z, antiholomorphic in w, and it is Hermitian of positive type. The kernel K completely characterizes the Hilbert space H. Conversely a kernel with these properties is the reproducing kernel of a unique Hilbert subspace of O(Z) (see §2 below).

The Hilbert subspace H is said to be G-invariant if, for every g ∈ G, H is invariant under the operator π(g), and the restriction π^{H}(g) of π(g) to H is unitary. One then obtains a unitary representation of G on H. The Hilbert subspace is G-invariant if and only if its reproducing kernel K is invariant:

K(z, w) = K(g · z, g · w) (g ∈ G, z, w ∈ Z).

We are interested in the following problems:

1) Determine the minimal (i.e. irreducible) invariant Hilbert spaces.

2) Decompose an invariant Hilbert subspace H ⊂ O(Z) into a direct sum (or direct integral) of minimal invariant Hilbert subspaces.

3) Determine conditions implying the uniqueness of such decompositions, or equivalently, implying that the decompositions are multiplicity free.

4) Determine the reproducing kernels of minimal invariant Hilbert subspaces.

If Z̃ is a holomorphic extension of Z, every holomorphic function f ∈ O(Z) has an extension f̃ ∈ O(Z̃). If H ⊂ O(Z) is a Hilbert space of holomorphic functions, then H̃ = {f̃ | f ∈ H}, with the norm ∥f̃∥ = ∥f∥, is a Hilbert space of holomorphic functions on Z̃. This means that, for every compact set Q ⊂ Z̃, there exists a constant M = M(Q) such that, for f ∈ H, z ∈ Q, |f̃(z)| ≤ M∥f∥. This follows from Lemma 5.4.1 in [12]. If Z̃ is a Stein extension of Z the action of G on Z extends to Z̃. This follows from a remark of Hörmander ([12], p. 192 and 5.1.3) according to which the continuous multiplicative linear functionals L : O(Z̃) ⟶ ℂ are of the form L(f̃) = f̃(z̃) for precisely one z̃ ∈ Z̃. More generally, if Z_{1} and Z_{2} are complex manifolds having Stein extensions Z̃_{1} and Z̃_{2} , every holomorphic map F : Z_{1} ⟶ Z_{2} has a unique holomorphic extension z̃_{1} ↦ F̃(z̃_{1}) = z̃_{2} from Z̃_{1} to Z̃_{2} defined by the relations f̃(z̃_{2}) = ˜ (f ∘ F)(z̃_{1}), the right-hand side being a multiplicative linear functional of f̃ ∈ O(Z̃_{2}). It therefore seems natural to assume that Z is a Stein manifold.

2. Application of Choquet theory

Let Γ = Γ (Z) be the convex cone of Hermitian kernels K(z, w) of positive type, which are holomorphic in z, antiholomorphic in w. By Hartog’s theorem these are holomorphic on the space Z × \overline{Z}, where \overline{Z} denotes the manifold Z equipped with the opposite complex structure. This implies that these kernels are continuous. It follows, by the classical theory of reproducing kernels (e.g.

[1] or [24]) that they reproduce Hilbert spaces of continuous functions, which in the case of kernels K ∈ Γ (Z) are easily seen to be composed of holomorphic functions. Thus there is a bijective correspondence H ↦ K between the set Hilb(O(Z)) of Hilbert subspaces of O(Z) and the set of kernels belonging to Γ (Z).

The set Hilb(O(Z)) also has a natural structure of convex cone: The sum of two Hilbert subspaces

H = H_{1} + H_{2}

is defined as the usual vector sum, endowed with the norm which makes the map (h_{1}, h_{2}) ↦ h = h_{1} + h_{2} from the product H_{1} × H_{2} to H a partial isometry. In particular, if H_{1} ∩ H_{2} = {0}, it is an isomorphism, and the sum is said to be direct and written H = H_{1} ⊕ H_{2}, the spaces H_{1} and H_{2} then being closed orthogonal subspaces of H.

For a number λ ≥ 0 one defines the space λH to be {0} if λ = 0 or else equal to the linear space H equipped with the inner product equal to the inner product of H divided by λ.

Then it is a classical result that if the kernels of H_{1}, H_{2} are K_{1} and K_{2} respectively, the kernel of H_{1} + H_{2} is K_{1} + K_{2} ; if K is the kernel of H then λK is the kernel of λH ([24] §6). Thus the correspondence between Hilb(O(Z)) and Γ (Z) preserves the cone structures. This property extends to finite and infinite sums, as well as to integrals of kernels and Hilbert subspaces ([24], [30]).

Let Γ_{G} = Γ_{G}(Z) be the cone of G-invariant kernels in Γ (Z). Then we have a bijective correspondence between the set Hilb_{G}(O(Z)) of G-invariant Hilbert subspaces of O(Z) and the cone Γ_{G} ([24] p. 182).

We consider the cones Γ and Γ_{G} as embedded in the topological vector space O(Z × \overline{Z}),

Γ_{G} ⊂ Γ ⊂→ O(Z × \overline{Z}).

Recall that an element k in a convex cone C is extremal if it does not have a non-trivial decomposition as a sum of two elements in the cone: k = k_{1}+k_{2} with k_{1}, k_{2} in C, implies that there exist numbers λ_{i} ≥ 0 such that k_{i} = λ_{i}k, i = 1, 2. Equivalently: k is extremal if the inequality 0 ≤ ℓ ≤ k, in the sense of the order defined by the cone, i.e., ℓ ∈ C, k−ℓ ∈ C, implies that ℓ is proportional to k ([21],[5]).

The extremal elements (also called extremal generators) form a subcone ext(C) ⊂ C, i.e. a subset stable under multiplication by non-negative numbers. It may happen that ext(C) = {0} (for instance if C is the cone of non-negative elements in L^{2}[0, 1]) but in the case of the cone Γ_{G} it will be shown that, on the contrary, Γ_{G} is the closed convex hull of its extremal generators. The importance of the extremal generators comes from the fact that they correspond with the irreducible Hilbert spaces:

Proposition 1. The invariant Hilbert subspace H is irreducible iff its reproducing kernel K is extremal. Proof. This is true more generally if the space O(Z) is replaced by any other sequentially complete locally convex space E on which a group acts by continuous linear transformations. Other particular cases of the result have been published before, e.g. in [25] ch.4, and in [29]. For the convenience of the reader we repeat the proof in the present framework: If K_{1} and K_{2} belong to Γ_{G} and K_{1} ≤ K_{2} i.e. K_{2} − K_{1} ∈ Γ_{G}, it follows by the general properties of reproducing kernels (e.g. [24]) that the corresponding Hilbert space H_{1} is continuously embedded in H_{2} . If i denotes the G-equivariant injection of H_{1} into H_{2} and i^{∗} its adjoint H_{2} ⟶ H_{1} , the operator ii^{∗} : H_{2} ⟶ H_{2} commutes with the representation of G on H_{2} , and so if H_{2} is irreducible, it is equal to a multiple of the identity by Schur’s lemma. This implies that the Hilbert space H_{1} has the same underlying linear space as H_{2} and that its inner product is proportional to that of H_{2} , implying that K_{1} is proportional to K_{2} , i.e. implying that K_{2} is extremal. Conversely, if K is extremal the corresponding space H is irreducible, for a non trivial decomposition H = H_{1} ⊕ H_{2} into invariant subspaces would entail a non trivial decomposition of K in Γ_{G}.

Let ext(Γ_{G}) be the set of extremal generators, and let λ ↦ K_{λ}, Λ → ext(Γ_{G}), be an admissible parametrization of ext(Γ_{G}) defined on a topological Hausdorff space Λ. This means that it is a continuous injection Λ → ext (Γ_{G}), such that each extremal generator is proportional to K_{λ} for precisely one λ ∈ Λ, and moreover, such that the inverse map is universally measurable. This last condition is always satisfied if Λ is a second countable locally compact space. The existence of admissible parametrizations and the last assertion are proved in [31], §1.3. The choice of such a parametrization is in general totally arbitrary, (but can be avoided by the use of Choquet’s conical measures [31]). However, in the examples, such as those below, there often is a natural choice for a parametrization.

Theorem 1. For every kernel K ∈ Γ_{G} there exists a positive Radon measure μ on Λ such that

K(z, w) = ∫_{Λ} K_{λ}(z, w)dμ(λ),

the integral converging uniformly on compact sets of z and w.

Proof. First note that the integral defines a kernel K ∈ Γ_{G}(Z) if and only if the measure μ has the following property: for every compact set Q ⊂ Z, there is a constant M such that, for z ∈ Q,

∫_{Λ} K_{λ}(z, z)dμ(λ) ≤ M.

The existence of the integral representation is an application of the nuclear integral representation theorem, p.226 in [31], according to which a closed convex cone in a conuclear space, having bounded order intervals, is spanned, in the sense of integral representations, by its extremal generators. In fact, Z × \overline{Z} being, like Z, a complex manifold, countable at infinity, O(Z×\overline{Z}) is a nuclear Fréchet space, which implies that it is a conuclear space ([9], Chapitre II Espaces Nucléaires, p. 56 Corollaire, and p. 40 Théorème 7, and [26] pp.227-231 Theorem 1a. See also [22], IV, Theorem 9.6, and [10]). Thus, it is enough to show that the order intervals in Γ are bounded. Let K be fixed in Γ, and L ∈ Γ with 0 ≤ L ≤ K, in the sense of the ordering defined by Γ, i.e., for all N ∈ ℕ, z_{1}, …, z_{N} ∈ Z, and α_{1}, …, α_{N} ∈ ℂ,

0 ≤ ∑_{j,k=1}^{N} L(z_{k}, z_{j})α_{j}\overline{α_{k}} ≤ ∑_{j,k=1}^{N} K(z_{k}, z_{j})α_{j}\overline{α_{k}}.

In particular, with N = 1, it follows that 0 ≤ L(z, z) ≤ K(z, z). With N = 2, and by the elementary facts regarding matrices of positive type, one obtains

|L(z, w)|^{2} ≤ L(z, z)L(w, w) ≤ K(z, z)K(w, w).

The right-hand side being uniformly bounded for z and w in a compact set Q, we have

sup_{0≤L≤K} sup_{z,w∈Q} |L(z, w)| < +∞,

which implies that the order intervals are bounded in O(Z × \overline{Z}). A fortiori the order intervals of Γ_{G} are bounded.

Alternatively one could proceed as follows, avoiding the above mentioned theorems of Grothendieck and Schwartz on nuclear and conuclear spaces, but using the classical fact that closed bounded subsets of O(Z) are compact: Let ϱ be a continuous strictly positive measure on Z, i.e. one which in each chart has a continuous strictly positive density with respect to Lebesgue measure. Then the topology of O(Z) of uniform convergence on compact sets is identical to the topology induced by L_{loc}^{2}(Z, ϱ). More precisely: for every compact set K ⊂ Z and relatively compact open neighborhood ω of K, there exists a constant M such that

sup_{z∈K} |f(z)| ≤ M( ∫_{ω} |f(z)|^{2}dϱ(z))^{1/2}, f ∈ O(Z).

It is enough to prove this when ω is contained in a chart, in which case it easily follows from the mean value property of holomorphic, hence harmonic, functions, and from the Cauchy-Schwarz inequality. This estimate can be used as a basis for a direct proof of the fact that the space O(Z) is both nuclear and conuclear, but we can avoid these concepts altogether by showing directly that the cones Γ and Γ_{G} are well-capped, i.e. the union of their caps, compact convex subsets in the cone such that the complement in the cone is convex. The importance of caps comes from the fact that their extreme points lie on extreme rays of the cone, so that a well-capped cone is the closed convex hull of its extreme rays ([21], p.88).

Proposition 2. The cones Γ and Γ_{G} are well capped (i.e. union of their caps). Proof. If a is a strictly positive continuous function on Z, then the set

C_{a} = {K ∈ Γ : ∫_{Z} K(z, z)a(z)dϱ(z) ≤ 1}

is a cap. In fact, by Fatou’s lemma, C_{a} is closed. If ω is a relatively compact open set in Z, there exists a constant A > 0 such that a(z) ≥ A for z ∈ ω. If K belongs to C_{a}, then

∫_{ω} ∫_{ω} |K(z, w)|^{2}dϱ(z)dϱ(w) ≤ ∫_{ω} K(z, z)dϱ(z) ∫_{ω} K(w, w)dϱ(w) ≤ 1/A^{2},

showing that C_{a} is bounded in O(Z × \overline{Z}). Being closed and bounded C_{a} is compact. Since both C_{a} and Γ \ C_{a} are convex, it follows that C_{a} is a cap in Γ. Moreover the intersection Γ_{G} ∩ C_{a} is a cap in Γ_{G}.

It remains to show that every kernel K in Γ or Γ_{G} belongs to such a cap. Let h be a strictly positive continuous function such that ∫_{Z} h(z)dϱ(z) = 1. The kernel K belongs to the cap C_{a} with a(z) = \frac{h(z)}{K(z,z)+1}.

Then applying Choquet theory such as in Theorems 5.3 and 1.21 of [31] one again obtains the above integral representation as well as the equivalence between (a) and (b) in the following theorem:

Theorem 2. With the notations of Theorem 1, the following properties are equivalent:

(a) For every K ∈ Γ_{G}, the measure μ on Λ is unique.

(b) The cone Γ_{G} is a lattice (i.e. any two elements in the cone have a smallest common majorant in the cone).

(c) For every G-invariant Hilbert subspace H the commutant {π^{H}}′ in B(H) of the representation π^{H} is commutative.

(d) Every G-invariant Hilbert subspace H has a unique decomposition as direct integral

(∗) H = ∫_{Λ}^{⊕} H_{λ}dμ(λ).

(e) Any two irreducible G-invariant Hilbert subspaces of O(Z) either coincide as linear spaces, and have proportional inner products, or they yield inequivalent representations of G.

Moreover, under these conditions the algebra of diagonal operators in the direct integral (∗) equals the commutant {π^{H}}′.

When these conditions are satisfied, in particular condition (e), we shall say that the action of G on O(Z) is multiplicity free.

Corollary 1. If G_{1} and G_{2} are two groups of holomorphic transformations acting on Z, with G_{1} ⊂ G_{2}, then if the action of G_{1} is multiplicity free, so is the action of G_{2} .

Proof. This follows from condition (c).

Corollary 2. If the action of G is multiplicity free, every minimal invariant space H is composed of eigenvectors of the G-invariant linear operators u : O(Z) ⟶ O(Z), i.e. there exists a number α = α(u, H) such that uf = αf for all f ∈ H. In particular, two invariant operators commute on H.

Proof. This is an immediate consequence of Schur’s lemma (cf. [30], Theorem B).

The proof of the theorem, based on Theorem 5.1 or 5.3 of [31], is essentially the same as in the case where the space O(Z) is replaced by the space of distributions on a real manifold, ([30], Theorem A, §4 and §6), and we will not repeat it.

In the present context of complex manifolds we will give a simple geometric condition which implies that the action of G is multiplicity free: (H) There exists an antiholomorphic involution τ : Z → Z such that, for every z ∈ Z, there exists g ∈ G with

τ(z) = g · z.

Theorem 3. The condition (H) implies that the action of G on O(Z) is multiplicity free.

Proof. We will prove condition (c). For f ∈ O(Z), let Jf(z) = \overline{f(τ(z))}. Notice that the function Jf is holomorphic. Let H ⊂→ O(Z) be a Hilbert subspace, with reproducing kernel K. The reproducing kernel K̃ of the Hilbert subspace H̃ = J(H) is

K̃(z, w) = K(τ(w), τ(z)).

We will show that, if H is G-invariant, then K̃ = K.

Lemma 1. Let F (z, w) be holomorphic in z, antiholomorphic in w. If F (z, z) = 0 for all z, then F vanishes identically.

Proof. In fact diag(Z) = {(z, z) | z ∈ Z} is a totally real submanifold in Z × \overline{Z}. Hence a holomorphic function F on Z × \overline{Z}, which vanishes on diag(Z), vanishes identically.

Let z ∈ Z. By assumption (H), there exists g ∈ G such that τ(z) = g·z. Therefore

K̃(z, z) = K(τ(z), τ(z)) = K(g · z, g · z) = K(z, z),

since K is G-invariant. By Lemma 1 it follows that K̃ = K, and H̃ = H. Hence J is an antilinear automorphism of O(Z) leaving the G-invariant Hilbert subspaces unitarily invariant. Thus Theorem E of [30] is essentially applicable, but we recall the proof. The restriction J : H → H is an antilinear unitary automorphism. If A is a positive self-adjoint operator on H which commutes with π(g) (g ∈ G), then the inner product

(f_{1}|f_{2})_{A} = (Af_{1}|f_{2})

defines a Hilbert subspace H_{A} of H which is invariant, therefore J(H_{A}) = H_{A}. It follows that JAJ^{−1} = A. If B ∈ B(H) commutes with π(g) (g ∈ G), then, by decomposing B into positive parts and using the antilinearity of J, it follows that JBJ^{−1} = B^{∗}, and, if B, C ∈ B(H) commute with π(g),

BC = J^{−1}(BC)^{∗}J = J^{−1}C^{∗}B^{∗}J = J^{−1}C^{∗}JJ^{−1}B^{∗}J = CB.

Let 𝔻_{G}(Z) be the algebra of G-invariant differential operators on Z with holomorphic coefficients. Under the condition (H), if H is an irreducible invariant subspace, and D ∈ 𝔻_{G}(Z), then H is an eigenspace of D: there exists α ∈ ℂ such that, for all f ∈ H, Df = αf (Corollary 2 above). The direct integral decomposition (∗) means that every element f ∈ H has an integral decomposition

f = ∫_{Λ} f(λ)dμ(λ)

in O(Z) with f(λ) ∈ H_{λ} and

||f||^{2} = ∫_{Λ} ||f(λ)||_{H_{λ}}^{2}dμ(λ)

(cf. [24], [30]). This implies that f is the limit of expressions ∑ c_{i}f(λ_{i}) on which the invariant operators commute. It follows that, under the condition (H), and if there exists an invariant Hilbert subspace H which is dense in O(Z), or more generally the union of the irreducible spaces is total, the algebra 𝔻_{G}(Z) is commutative.

Let ω be a holomorphic differential form on Z of maximal degree, which does not vanish. Then we can define the Bergman space

B_{ω}^{2}(Z) = \{f ∈ O(Z) | ∥f∥^{2} = ∫_{Z} |f|^{2}ω ∧ \overline{ω} < ∞\}.

If ω is G-invariant then B_{ω}^{2}(Z) is an invariant Hilbert subspace of O(Z). We can state:

Theorem 4. Assume that (i) (H) is satisfied, (ii) There exists a G-invariant holomorphic differential form ω on Z of

maximal degree which does not vanish,

(iii) The Bergman space B_{ω}^{2}(Z) is dense in O(Z). Then the algebra 𝔻_{G}(Z) is commutative.

We propose the following question: is it possible to drop the third assumption? In our opinion this question should be related to a result of Lichnerowicz [17]: if G/H is a symmetric homogeneous space with an invariant volume form, then the algebra 𝔻(G/H) of invariant differential operators is commutative.

3. Examples

a) The unit ball. Let Z = D = {z ∈ ℂ^{n} | ∥z∥ < 1} be the unit ball in ℂ^{n}, with ∥z∥ = \sqrt{|z_{1}|^{2} + ⋯ + |z_{n}|^{2}} . Assumption (H) is satisfied for the unitary group U = U(n). In fact define τ(z) = z̄. Since every z ∈ D can be written z = ru·e_{1}, with e_{1} = (1, 0, …, 0), 0 ≤ r < 1, u ∈ U, it follows that τ(z) = z̄ = g · z, with g = ū u^{−1} ∈ U.

Every kernel K ∈ Γ_{U}(D) can be written K(z, w) = ∑_{m=0}^{∞} μ_{m}(z|w)^{m}, with a sequence of numbers μ_{m} ≥ 0 such that

∀r, 0 ≤ r < 1, ∑_{m=0}^{∞} μ_{m}r^{2m} < ∞.

The sequence μ_{m} is clearly unique.

b) Bounded symmetric domain. More generally let Z = D ⊂ V ≃ ℂ^{n} be a bounded symmetric domain (see [11], p.382). We assume that D is circled and irreducible. Let G be the group of holomorphic automorphisms of D, and U be the isotropy subgroup of 0; U is compact, and is the group of ℂ-linear automorphisms of D. Let 𝔤 and 𝔲 be the Lie algebras of G and U, and let 𝔤 = 𝔲 + 𝔭 be the Cartan decomposition with respect to the Cartan involution θ of G associated to U,

θ(g) · z = −g · (−z) (g ∈ G, z ∈ D).

An element in 𝔤 can be seen as a vector field ξ on D, and the map ξ ↦ ξ(0), 𝔭 → V , is a ℝ-linear isomorphism. Therefore one can identify 𝔭 with V . In particular a Cartan subspace 𝔞 of 𝔭, i.e. a maximal abelian subspace 𝔞 of 𝔭, can be seen as a real vector subspace of V .

For a complex conjugation τ of V such that τ(D) = D, let us consider the following property (P): there exists a Cartan subspace 𝔞 ⊂ V such that τ(z) = z for all z ∈ 𝔞. This property does not hold for every complex conjugation τ such that τ(D) = D. In fact let D be the Lie ball, i.e. the unit ball in ℂ^{n} for the norm

N(z) = (∥x∥^{2} + ∥y∥^{2} + 2\sqrt{∥x∥^{2}∥y∥^{2} − (x|y)^{2}})^{\frac{1}{2}} (z = x + iy).

Then an element U ∈ G can be written

g · z = e^{iθ}uz (θ ∈ ℝ, u ∈ O(n)).

For the conjugation τ, τ(z) = z̄, the property (P) does not hold. However, for τ defined by τ(z) = (z̄_{1}, …, z̄_{n−1}, −z̄_{n}) the property (P) holds.

For every irreducible bounded symmetric domain there exists a conjugation τ with the property (P). This fact will be checked by using the classification of the irreducible bounded symmetric domains. See the appendix.

Let σ be the involutive automorphism of U defined by

σ(g) = τ ∘ g ∘ τ,

or τ(g · z) = σ(g) · τ(z). Every z ∈ D can be written z = u · x, with u ∈ U, x ∈ 𝔞 ∩ D. Therefore τ(z) = g · z, with g = σ(u)u^{−1} ∈ U. This proves that (H) holds for the group U.

To make explicit the integral representation of Theorem 1, we will use the results of Schmid [23], and Takeuchi [27], notation and results of [7] (Sections 2,3) (see also [8], Chapter XI). The space P of polynomials on V decomposes into irreducible subspaces under the action of U as

P = ⨁_{m≥0} P_{m}.

Here m = (m_{1}, …, m_{r}) ∈ ℕ^{r}, r is the rank of D, and m ≥ 0 means that m_{1} ≥ ⋯ ≥ m_{r}. We equip P with the Fischer inner product

(p|q)_{F} = p(\frac{∂}{∂z} )q̄(z)|_{z=0} = \frac{1}{π^{n}} ∫_{V} p(z)\overline{q(z)}e^{−∥z∥^{2}}dλ(z),

where λ denotes the Lebesgue measure. Let K^{m} be the reproducing kernel of P_{m} . Every kernel K ∈ Γ_{U}(D) can be written

K(z, w) = ∑_{m≥0} μ_{m}K^{m}(z, w),

with a sequence μ_{m} ≥ 0 such that ∑_{m≥0} μ_{m}K^{m}(z, z) < ∞ for all z ∈ 𝔞 ∩ D. The sequence μ_{m} is unique.

c) Let Ω be a convex domain in a real vector space V ≃ ℝ^{n}, and Z = T_{Ω} = V + iΩ ⊂ V^{ℂ} ≃ ℂ^{n} be the tube domain with base Ω. Let G ≃ V be the group of real translations. For z ∈ T_{Ω} define

τ(z) = −z̄, τ(x + iy) = −x + iy.

Then τ(z) = g · z, where g is the translation of vector −2x. Since G is commutative, an irreducible Hilbert subspace H of O(T_{Ω}) is one dimensional,

H = {ce^{i⟨z,u⟩} | c ∈ ℂ} (u ∈ V^{∗}),

whose reproducing kernel is K(z, w) = e^{i⟨z−w̄,u⟩}. Every kernel K ∈ Γ_{G}(Z) can be written

K(z, w) = ∫_{V^{∗}} e^{i⟨z−w̄,u⟩}dμ(u),

where μ is a positive measure on V^{∗} such that

∀y ∈ Ω, ∫_{V^{∗}} e^{−2⟨y,u⟩}dμ(u) < ∞.

d) Let G_{0}^{ℂ} be a connected reductive complex Lie group, and G_{0} be a maximal compact subgroup of G_{0}^{ℂ}. Then G_{0} is a real form of G_{0}^{ℂ}. Denote by 𝔤_{0} and 𝔤_{0}^{ℂ} = 𝔤_{0} + i𝔤_{0} the Lie algebras of G_{0} and G_{0}^{ℂ}. Every g ∈ G_{0}^{ℂ} can be uniquely written

g = k exp iX (k ∈ G_{0}, X ∈ 𝔤_{0}).

A G_{0} -biinvariant domain Z ⊂ G_{0}^{ℂ} can be written Z = G_{0} exp iΩ, where Ω is an Ad(G_{0})-invariant domain in 𝔤_{0} . If Ω is convex then Z is a Stein manifold, and, in general, the envelope of holomorphy of Z is Ẑ = G_{0} exp iconv(Ω), where conv(Ω) is the convex hull of Ω (Theorems 6.1 and 7.9 in [19], see also Théorèmes 3 and 4 in [15]). Therefore we may assume that Ω is convex.

Let G = G_{0} × G_{0} act on Z by

g · z = k_{1}zk_{2}^{−1} (g = (k_{1}, k_{2})).

For z ∈ Z, z = k exp iX (k ∈ G_{0}, X ∈ 𝔤_{0} ), we define

τ(z) = exp iXk^{−1} = k^{−1} exp i Ad(k)X.

Then, for z = k exp iX ∈ Z, τ(z) = k^{−1}zk^{−1} = g · z, with g = (k^{−1}, k^{−1}) ∈ G.

Let Ĝ_{0} be the set of equivalence classes of irreducible unitary representations of G_{0} , and, for λ ∈ Ĝ_{0} , let ω_{λ} be a representative of the class λ. Then ω_{λ} has a holomorphic extension to G_{0}^{ℂ}. Every kernel K ∈ Γ_{G}(Z) can be written

K(z, w) = ∑_{λ∈_{G_{0}}} μ_{λ} tr ω_{λ}(zτ(w)),

with a sequence μ_{λ} ≥ 0 such that

∀X ∈ Ω, ∑_{λ∈_{G_{0}}} μ_{λ} tr ω_{λ}(exp 2iX) < ∞.

More generally one can consider a G-invariant domain Z in the complexification of a compact symmetric space G/H.

e) Biinvariant domain in a complex Olshanski semi-group. Let G_{0} be a connected simple real linear Lie group whose Lie algebra 𝔤_{0} is Hermitian: if K_{0} is a maximal compact subgroup of G_{0} , then the center 𝔷_{0} of its Lie algebra 𝔨_{0} has dimension one. A cone C ⊂ 𝔤_{0} is said to be regular if it is closed, convex, pointed, and generating (or equivalently with non empty interior), and to be invariant if, for every g ∈ G_{0} , Ad(g)(C) = C. Under the condition that 𝔤_{0} is Hermitian, there exist invariant regular cones. Fix X_{0} ∈ 𝔷_{0} (X_{0} ≠ 0). Among the invariant regular cones in 𝔤_{0} containing X_{0} , there is a maximal one C_{max} and a minimal one C_{min} . If C is an invariant regular cone, then S(C) = G_{0} exp(iC) ⊂ G_{0}^{ℂ} is a closed semi-group (G_{0}^{ℂ} is the complexification of G_{0} ). Such a semi-group is called a complex Olshanski semi-group.

Let Ω ⊂ C_{max} be an Ad(G_{0})-invariant domain. The set Z = G_{0} exp iΩ is a G_{0} -biinvariant domain in G_{0}^{ℂ}, contained in S(C_{max}), which is homeomorphic to G_{0} × Ω. If Ω is convex then Z is a Stein manifold, and, in general, the envelope of holomorphy of Z is Ẑ = G_{0} exp iconv(Ω) (Theorems 6.1 and 7.9 in [19]). Therefore we may assume that Ω is convex. The group G = G_{0} × G_{0} acts on Z by

g · z = g_{1}zg_{2}^{−1} (g = (g_{1}, g_{2})).

Then the action on O(Z) is multiplicity free. This can be seen as a consequence of Theorem 3. In fact, for z = g exp iX (g ∈ G_{0} , X ∈ Ω), we define

τ(z) = exp iX g^{−1} = g^{−1} exp i Ad(g)X ∈ Z.

Since τ(z) = g^{−1}zg^{−1}, the property (H) holds.

Let C be an invariant regular cone in 𝔤_{0} , and ω a unitary representation of G_{0} on a Hilbert space W. The representation ω is said to be C -positive if the essentially self-adjoint operator −idπ(X) is ≥ 0 for every X ∈ C. Such a representation has an extension as a representation ω̃ of the semi-group S(C) such that

ω̃(τ(γ)) = ω̃(γ)^{∗}, ∥ω̃(γ)∥ ≤ 1 (γ ∈ S(C)),

and, for every u, v ∈ W, the function γ ↦ (ω(γ)u|v) is continuous on S(C), and holomorphic on its interior S(C)^{0} .

Let H ⊂ O(Z) be a G_{0} -biinvariant Hilbert space of holomorphic functions. Then the representation of G = G_{0} × G_{0} on H defined by

(π(g)f)(z) = f(g_{1}^{−1}zg_{2}) (g = (g_{1}, g_{2}))

extends as a representation π̃ of S(C_{min}) × S(C_{min}),

(π̃(γ_{1}, γ_{2})f)(z) = f(τ(γ_{1})zγ_{2}),

and

∥π̃(γ_{1}, γ_{2})∥ ≤ 1 (γ_{1}, γ_{2} ∈ S(C_{min}))

(Theorem III.8 in [20]).

Let Ŝ(C) be the set of equivalence classes of C -positive irreducible unitary representations of G_{0} , and, for λ ∈ Ŝ(C), let ω_{λ} be a representative of λ. The set Ŝ(C) is countable. Every kernel K ∈ Γ_{G}(Z) can be written

K(z, w) = ∑_{λ∈_{S(C_{min})}} μ_{λ} tr ω̃_{λ}(zτ(w)),

with a sequence μ_{λ} ≥ 0 such that

∀X ∈ Ω, ∑_{λ∈_{S(C_{min})}} μ_{λ} tr ω̃_{λ}(exp 2iX) < ∞.

4. An example associated with a symmetric cone

We present an example for which the set Λ has a continuous part and a discrete one. Let us consider again Example (c), and assume now that Ω is an irreducible symmetric cone, i.e. a self-dual and homogeneous cone (see [8]), and let G(Ω) be the group of linear automorphisms of Ω,

G(Ω) = {g ∈ GL(V) | g(Ω) = Ω}.

We assume that n = dim V > 1. Define

G_{1} = {g ∈ G(Ω) | det g = 1},

and let G = G_{1} ⋉ V be the group of affine transformations of V ,

g · z = g_{1}z + v (g = (g_{1}, v), g_{1} ∈ G_{1}, v ∈ V).

The symmetric cone is associated with a structure of a Euclidean Jordan algebra on V . Since Ω is irreducible, the Jordan algebra V is simple. We denote by tr x and det x the trace and the determinant of x ∈ V relatively to this Jordan algebra structure, and by r the rank (see [8], Chapters I,II).

For instance, the cone Ω of positive definite symmetric matrices in V = Sym(m, ℝ) is a symmetric cone. The Jordan product on V is given by

x ∘ y = \frac{1}{2} (xy + yx).

Then the rank is r = m. Also tr x and det x are the trace and the determinant of x in the usual sense.

For Z = T_{Ω} = V + iΩ, we saw that a kernel in Γ_{G}(Z) can be written

K(z, w) = ∫_{V^{∗}} e^{i(z−w̄|u)}dμ(u),

and in the present case the measure μ is G_{1} -invariant.

Proposition 3. The support of μ is contained in \overline{Ω}.

The proof will use the following lemma

Lemma 2. Let Ω be an open convex cone in V and let μ be a positive measure on V^{∗} such that, for every x ∈ Ω,

φ(x) = ∫_{V^{∗}} e^{−⟨x,u⟩}dμ(u) < ∞.

Furthermore assume that, for every x ∈ Ω, the function t ↦ φ(tx) is decreasing on ]0, ∞[. Then

supp(μ) ⊂ \overline{Ω}^{∗} = \{u ∈ V^{∗} | ∀x ∈ Ω, ⟨x, u⟩ ≥ 0\}.

Proof. Let Q be a compact set in V^{∗} such that Q ∩ \overline{Ω}^{∗} = Ø. There exists x_{0} ∈ Ω and α > 0 such that, for every u ∈ Q, ⟨x_{0}, u⟩ ≤ −α. For t > 0, φ(tx_{0}) ≥ e^{αt}μ(Q) since the function t ↦ φ(tx_{0}) is decreasing, it follows that μ(Q) = 0.

Proof of Proposition 3. For x ∈ Ω,

φ(x) = ∫_{V} e^{−(x|u)}dμ(u) < ∞.

Since φ is G_{1} -invariant, it can be written φ(x) = ψ(log det x), where ψ is a function defined on ℝ. The functions φ and ψ are analytic. Let us write the first and second derivatives of φ:

(Dφ)_{x}(u) = ψ′(log det x)(x^{−1}|u),
(D^{2}φ)_{x}(u, v) = ψ′′(log det x)(x^{−1}|u)(x^{−1}|v) − ψ′(log det x)(P (x)^{−1}u|v),

where P (x) is the quadratic representation of the Jordan algebra V . We have used the formulas

D(log det x)(u) = (x^{−1}|u), D(x^{−1})(u) = −P (x)^{−1}u

([8], Propositions II.3.3 and III.4.2). The function φ is convex, therefore, for x ∈ Ω, u ∈ V ,

(D^{2}φ)_{x}(u, u) ≥ 0.

Since dim V > 1 there exists u ≠ 0 in V such that (x^{−1}|u) = 0, and we obtain ψ′(log det x) ≤ 0, since P (x)^{−1} is positive definite. Therefore ψ is decreasing, and the statement of Proposition 3 follows from Lemma 2.

The function x ↦ det x has no critical point in Ω. Therefore, for a function f ∈ D(Ω), there exists a function Mf which is defined on ]0, ∞[, such that, for every continuous function ψ on ]0, ∞[,

∫_{Ω} ψ(det x)f(x)d^{∗}x = ∫_{0}^{∞} ψ(t)Mf(t)\frac{dt}{t},

where d^{∗}x = (det x)^{− \frac{n}{r}} dλ(x), a G(Ω)-invariant measure on Ω (λ is the Euclidean measure).

For t > 0, the map f ↦ Mf(t) defines a positive G_{1} -invariant measure σ_{t} whose support is the G_{1} -orbit Ω_{t} = {x ∈ Ω | det x = t}. Furthermore

Mf(t) = ∫ f(x)dσ_{t}(x) = ∫ f(t^{\frac{1}{r}} x)dσ_{1}(x).

The zeta integral of the function f ∈ S(\overline{Ω}) is defined by

Z(f, s) = ∫_{Ω} f(x)(det x)^{s}d^{∗}x (s ∈ ℂ).

It converges for ℜs > \frac{d}{2}(r −1) ([8], Section VII.2) and, as a function of s, admits a meromorphic continuation to ℂ. The zeta integral can be written as a Mellin transform

Z(f, s) = ∫_{0}^{∞} Mf(t)t^{s} \frac{dt}{t}.

If f(x) = e^{i(z|x)} , z ∈ T_{Ω} , then

∫_{Ω} e^{i(z|x)}(det x)^{s}d^{∗}x = Γ_{Ω}(s)(det \frac{z}{i})^{−s} (ℜs > \frac{d}{2}(r − 1)),

where Γ_{Ω} is the gamma function of the symmetric cone Ω,

Γ_{Ω}(s) = ∫_{Ω} e^{−tr(x)}(det x)^{s}d^{∗}x,

which has been computed by Gindikin (see [8], Proposition VII.1.2, Corollary VII.1.3),

Γ_{Ω}(s) = (2π)^{\frac{n−r}{2}} ∏_{j=0}^{r−1} Γ (s − j \frac{d}{2}).

The integer d is related to the dimension n and the rank r by

n = r + \frac{d}{2}r(r − 1).

One regularizes the zeta integral Z(f, s) by letting

Z′(f, s) = \frac{1}{Γ_{Ω}(s)}Z(f, s).

Then, as a function of s, Z′(f, s) is an entire function. For s = \frac{d}{2}j (j = 0, …, r − 1) the map f ↦ Z′(f, \frac{d}{2}j) defines a positive measure ω_{j},

Z′(f, \frac{d}{2}j) = ∫ f(x)dω_{j}(x).

The measure ω_{j} is supported by the boundary of Ω, more precisely,

supp(ω_{j}) = \overline{\{x ∈ \overline{Ω} | rank(x) = j\}}.

The Laplace transform of ω_{j} is given by

∫ e^{i(z|x)}dω_{j}(x) = (det \frac{z}{i})^{− \frac{d}{2} j}.

(See [16], Proposition 17). In particular ω_{0} = δ (the Dirac measure).

Proposition 4. Let μ be a positive G_{1} -invariant measure whose support in contained in \overline{Ω}. There exist a positive measure dν on ]0, ∞[, and non negative numbers a_{j}, j = 0, …, r − 1, such that

∫ fdμ = ∫_{0}^{∞} Mf(t)dν(t) + ∑_{j=0}^{r−1} a_{j} ∫ fdω_{j}.

Proof. The orbits of G_{1} in \overline{Ω} are the following sets

Ω_{t} = {x ∈ Ω | det x = t}  (t > 0),
O_{j} = {x ∈ ∂Ω | rank(x) = j}  (j = 0, …, r − 1).

The orbits Ω_{t} are closed and the closure of the orbit O_{j} is given by

\overline{O}_{j} = \{x ∈ ∂Ω | rank(x) ≤ j\}.

Let μ be a positive G_{1} -invariant measure whose support is contained in \overline{Ω}. Using once more the fact that the determinant function x ↦ det x has no critical point in Ω, one shows that there exists a positive measure ν on ]0, ∞[ such that, if f is supported in Ω,

∫ fdμ = ∫_{0}^{∞} Mf(t)dν(t),

and, for an arbitrary function f,

∫ fdμ = ∫_{0}^{∞} Mf(t)dν(t) + ∫ fdμ_{r−1},

where μ_{r−1} is a G_{1} -invariant positive measure supported by ∂Ω = \overline{O_{r−1}} . There exists a number a_{r−1} ≥ 0 such that μ_{r−1} = a_{r−1}ω_{r−1} + μ_{r−2} , where μ_{r−2} is a G_{1} -invariant measure supported by \overline{O_{r−2}} . The statement is obtained inductively by using the following fact. A G_{1} -invariant positive measure on the set

Ω^{j} = \{x ∈ \overline{Ω} | rank(x) ≥ j\},

which is supported by O_{j} is proportionnal to the restriction of ω_{j} to Ω^{j}.

Let H be the function defined by

H(ζ) = \frac{1}{2iπ} ∫_{ℜs=α} Γ_{Ω}(s)ζ^{−s}ds, α > (r − 1)\frac{d}{2}.

The function H is defined on the domain of the universal covering of ℂ^{∗} given by −r \frac{π}{2} < arg ζ < r \frac{π}{2} . It is the inverse Mellin transform of Γ_{Ω} ,

∫_{0}^{∞} H(t)t^{s} \frac{dt}{t} = Γ_{Ω}(s).

It follows that

∫ e^{i(z|x)}dσ_{1}(x) = H(det \frac{z}{i}).

(This function H has been introduced by Blind in his thesis [3], where its connection with the analysis on symmetric cones is studied.)

Theorem 5. Every kernel K ∈ Γ_{G}(Z) can be written

K(z, w) = K_{0}(det \frac{z − w̄}{i} ), with K_{0}(ζ) = ∫_{0}^{∞} H(tζ)dν(t) + ∑_{j=0}^{r−1} a_{j}ζ^{− \frac{d}{2} j},

where ν is a positive measure on ]0, ∞[, and the a_{j} are non negative numbers. Proof. Let K be a kernel in Γ_{G}(Z). It can be written

K(z, w) = ∫ e^{i(z−w̄|u)}dμ(u),

where μ is a G_{1} -invariant positive measure. By Proposition 3 the measure μ is supported by \overline{Ω}, and, by Proposition 4 there exists a positive measure ν on ]0, ∞[ and numbers a_{j} ≥ 0 such that

∫ fdμ = ∫_{0}^{∞} Mf(t)dν(t) + ∑_{j=0}^{r−1} a_{j} ∫ fdω_{j}.

Let us take f(u) = e^{i(z−w̄|u)} , then

Mf(t) = H(t det \frac{z − w̄}{i} ), ∫ fdω_{j} = (det \frac{z − w̄}{i} )^{− \frac{d}{2} j}.

Therefore

K(z, w) = ∫_{0}^{∞} H(t det \frac{z − w̄}{i} )dν(t) + ∑_{j=0}^{r−1} a_{j}(det \frac{z − w̄}{i} )^{− \frac{d}{2} j}.

Appendix

Type I_{p,q}

\begin{aligned}V & = M(p, q, ℂ) (p ≤ q), τ(z) = z̄, \\ 𝔞 & = \{( \begin{matrix}t_{1} & & & & & 0 & … & 0 \\ & . & . & . \\ & & & & t_{p} & 0 & … & 0 \end{matrix} ) | t_{j} ∈ ℝ\}. \end{aligned}

Type II_{n}

V = Skew(n, ℂ),  τ(z) = z̄.

If n is even, n = 2m,

𝔞 = \{( \begin{matrix}& & & & & & & & & t_{1} \\ & & & & & & . & . & . \\ & & & & & t_{m} \\ & & & & −t_{m} \\ & . & . & . \\ −t_{1} \end{matrix} ) | t_{j} ∈ ℝ\} .

If n is odd, n = 2m + 1,

𝔞 = \{( \begin{matrix}& & & & & & & & & & t_{1} \\ & & & & & & & . & . & . \\ & & & & & & t_{m} \\ & & & & & 0 \\ & & & & −t_{m} \\ & . & . & . \\ −t_{1} \end{matrix} ) | t_{j} ∈ ℝ\} .

Type III_{n}

\begin{aligned}V & = Sym(n, ℂ), τ(z) = z̄, \\ 𝔞 & = \{( \begin{matrix}t_{1} \\ & . & . & . \\ & & & & t_{n} \end{matrix} ) | t_{j} ∈ ℝ\} \end{aligned} .

Type IV_{n}

\begin{aligned}V & = ℂ^{n}, τ(z) = (z̄_{1}, …, z̄_{n−1}, −z̄_{n}), \\ 𝔞 & = \{(t_{1}, 0, …, 0, it_{2}) | t_{1}, t_{2} ∈ ℝ\}. \end{aligned}

Type V

The algebra 𝕆 of Cayley octave numbers is the following Cayley-Dickson extension of ℝ,

𝕆 = ℝ(−1, −1, −1).

The vector space is V = 𝕆_{ℂ}^{2} . An element z ∈ 𝕆_{ℂ} can be written

z = ∑_{j=0}^{7} z_{j}e_{j} (z_{j} ∈ ℂ),

where {e_{j}} is the canonical basis of 𝕆. Let α be the complex conjugation on 𝕆_{ℂ} defined by

α(∑_{j=1}^{8} z_{j}e_{j}) = ∑_{j=0}^{3} z̄_{j}e_{j} − ∑_{j=4}^{7} z̄_{j}e_{j}.

We consider on V the conjugation τ defined by

τ(z, z′) = (α(z), α(z′)),

and the Cartan subspace

𝔞 = {(t_{1}e_{0}, it_{2}e_{4}) | t_{1}, t_{2} ∈ ℝ}.

Type VI

\begin{aligned}V & = Herm(3, ℂ)_{ℂ}, τ(z) = z̄, \\ 𝔞 & = \{( \begin{matrix}t_{1} \\ & t_{2} \\ & & t_{3} \end{matrix} ) | t_{1}, t_{2}, t_{3} ∈ ℝ\}. \end{aligned}

References

[1] Aronszajn, N., Theory of reproducing kernels, Trans. Amer. Math. Soc. 68 (1950), 337–404.

[2] Berg, C., Christensen, J. P. R., and P. Ressel, “Harmonic Analysis on Semigroups,” Springer Verlag, 1984.

[3] Blind, B., Analyse de Fourier sur une algèbre de Jordan, Thèse, Université de Nancy I, 1991.

[4] Cartan, H., and R. Godement, Théorie de la dualité et analyse harmonique dans les groupes abéliens localement compacts, Ann. Sci. E.N.S. 64 (1947), 79–99.

[5]  Choquet, G., “Lectures on Analysis,” Benjamin, 1969.

[6] van Dijk, G., Spherical functions on the p-adic group P GL(2), Indagationes Math. 31 (1969), 213–241.

[7] Faraut, J., and A. Korányi, Function spaces and reproducing kernels on bounded symmetric domains, J. Functional Anal. 88 (1990), 64–89.

[8]  —, “Analysis on symmetric cones,” Oxford University Press, 1994.

[9] Grothendieck, A., Produits tensoriels topologiques et espaces nucléaires, Memoirs of the A. M. S., No 16, 1956.

[10] Harzallah, K., and E. G. F. Thomas, Espaces nucléaires, Sém. Brelot-Choquet-Deny (Théorie du potentiel), 1966/67, exposé No 13.

[11] Helgason, S., “Differential geometry, Lie groups, and symmetric spaces,” Academic Press, 1978.

[12] Hörmander, L., “An introduction to complex analysis in several variables,” North-Holland, 1979.

[13] Kobayashi, T., Multiplicity free branching laws for unitary highest weight modules, Proceedings of the Symposium on Representation theory held at Saga (1997), Kyushu (eds. K. Mimachi), 9–17.

[14] Krötz, B., The Plancherel theorem for biinvariant Hilbert spaces, Publ. RIMS, Kyoto Univ. 35 (1999), to appear.

[15] Lassalle, M., Sur la transformation de Fourier-Laurent dans un groupe analytique complexe réductif, Ann. Inst. Fourier 28 (1978), 115–138.

[16] —, Algèbre de Jordan et ensemble de Wallach, Invent. math. 89 (1987), 375–393.

[17] Lichnerowicz, A., Opérateurs différentiels sur un espace homogène, Ann. scient. Éc. Norm. Sup. 81 (1964), 341–385.

[18] Neeb, K.-H., On some classes of multiplicity free representations, manuscripta math. 92 (1997), 389–407.

[19] —, On the complex and convex geometry of Ol’shanskii semigroups, Ann. Inst. Fourier 48 (1998), 149–203.

[20]  —, Representation theory and convexity, Preprint, 1998.

[21]  Phelps, R. R., “Lectures on Choquet’s Theorem,” Van Nostrand, 1966.
[22]  Schaefer, H. H., “Topological vector spaces,” Springer Verlag, 1971.

[23] Schmid, W., Die Randwerte holomorpher Funktionen auf hermitesch symmetrischen Räumen, Inv. Math. 9 (1969), 61–80.

[24] Schwartz, L., Sous-espaces hilbertiens d’espaces vectoriels topologiques et noyaux associés, J. Anal. math. 13 (1964), 115–256.

[25] —, “Application of distributions to the theory of elementary particles in quantum mechanics,” Gordon and Breach, 1968.

[26] —, “Radon measures on arbitrary topological spaces and cylindrical measures,” Oxford University Press, 1973.

[27] Takeuchi, M., Polynomial representations associated with symmetric bounded domains, Osaka J. Math. 10 (1973), 441–475.

[28] Thomas, E. G. F., Représentation Intégrale dans les cones convexes, C. R. Acad. Sci. 286 (1978), 515–518.

[29] —, Integral representations of invariant reproducing kernels, Proceedings, Bicentennial Congress Wiskundig genootshap (Amsterdam, 1978) Part II, 391–404, Mathematisch Centrum, Amsterdam.

[30] —, The theorem of Bochner-Schwartz-Godement for generalized Gelfand pairs, Proc. 3rd Paderborn conference on functional analysis, North Holland, 1983.

[31] —, Integral representations in conuclear cones, J. Convex Analysis 1 (1994), 225–258.

Institut de Mathématiques de Jussieu (UMR 7586 du CNRS) Université Pierre et Martie Curie Case 247, 4 place Jussieu 75252 Paris Cedex 05 faraut@mathp6.jussieu.fr University of Groningen Department of Mathematics Postbus 800, NL-9700 AV Groningen e.thomas@math.rug.nl

Received May 5, 1998 and in final form February 22, 1999