ON OUTER AUTOMORPHISM GROUPS OF COXETER GROUPS

R. B. HOWLETT, P. J. ROWLEY, AND D. E. TAYLOR

Abstract. It is shown that the outer automorphism group of a Coxeter group W of finite rank is finite if the Coxeter graph contains no infinite bonds. A key step in the proof is to show that if the group is irreducible and Π_{1} and Π_{2} any two bases of the root system of W, then Π_{2} = ±wΠ_{1} for some w ∈ W. The proof of this latter fact employs some properties of the dominance order on the root system introduced by Brink and Howlett.

1. Introduction

The main aim of this paper is to prove the following result.

Theorem 1.1. If W is a Coxeter group of finite rank whose diagram has no infinite bonds, then Out(W) is a finite group.

We recall that a Coxeter group is a group with a presentation of the form

(1.1) W = gp⟨ { r_{a} | a ∈ Π } | (r_{a}r_{b})^{m_{ab}} = 1 for all a, b ∈ Π ⟩

where Π is some indexing set, whose cardinality is called the rank of W, and (m_{ab}) is a Coxeter matrix, that is, the parameters m_{ab} satisfy m_{ab} = m_{ba} ∈ { m ∈ ℤ | m ≥ 1 } ∪ {∞} and m_{ab} = 1 if and only if a = b. (A “relation” (r_{a}r_{b})^{∞} = 1 is to be interpreted as vacuous.)

The diagram of W is a graph with vertex set Π in which an edge (or bond) labelled m_{ab} joins a, b ∈ Π whenever m_{ab} ≥ 3. Conventionally, we omit the label when m_{ab} = 3. The diagram provides a compact encoding of the presentation (1.1). Every Coxeter group can be realized geometrically—possibly in several inequivalent ways—as a group of linear transformations of a real vector space V. We shall deal only with geometric realizations which embed W in the orthogonal group of some symmetric bilinear form F on V, as described in more detail in §2 below. In this situation we call V a reflection module for W.

For Coxeter groups whose diagram is of the form

◯ ^{\underline{∞}} ◯ ^{\underline{∞}} ◯ ⋯ ◯ ^{\underline{∞}} ◯

1991 Mathematics Subject Classification. 20F55, 20F28.

the automorphism group is completely described by James [7]. Tits [11] extended this to cover all finite rank Coxeter groups whose diagram has no triangles and only infinite bonds. An important ingredient in Tits’ work is the result that every finite subgroup of a Coxeter group is contained in a finite parabolic subgroup. This also plays a vital role in our work. Specifically, it allows us to prove that [Aut(W) : Aut_{V} (W)] < ∞, where Aut_{V} (W) consists of automorphisms of W induced by orthogonal transformations of a reflection module V. Our second main theorem, which we consider to be of independent interest, provides information about Aut_{V} (W), and thereby enables us to complete the proof of Theorem 1.1.

Theorem 1.2. Let W be an irreducible Coxeter group of finite rank and Φ its root system in a reflection module V. Let Π be a root basis in Φ. If φ : V → V is an orthogonal transformation permuting Φ, then φΠ = ±wΠ, for some w ∈ W.

We understand that J. Y. Hée has an unpublished proof of Theorem 1.2. See also Kac’s book [8], Proposition 5.9.

Our strategy for proving Theorem 1.2 is to show that (after replacing φ by −φ if necessary) the set

{ a ∈ Φ^{+} | φ(a) ∈ Φ^{−} } (1.2)

is finite. Here Φ^{+} and Φ^{−} denote, respectively, the positive and negative roots of Φ. This is equivalent to showing that φ preserves the Tits cone of W. (For the definition of the Tits cone see §3.) For affine and hyperbolic Coxeter groups we use a characterization of the Tits cone in terms of the bilinear form to achieve this. In particular, this proves Theorem 1.2 for Coxeter groups of rank at most 3. To pass to the general case we employ the dominance partial order on Φ which was first introduced in [2]. In particular, we show that φ or −φ preserves dominance, and then the finiteness of the set

E = { a ∈ Φ^{+} | a does not dominate any b ∈ Φ^{+} \ {a} }

yields that the set in (1.2) is finite.

We close this section with an example which shows that the assumption of no infinite bonds in Theorem 1.1 cannot be omitted. Let W be the Coxeter group whose diagram is

\begin{aligned}& b \\ ∞ & ∞ \\ & ◯ & \\ a^{◯} & c^{◯} \\ & ∞ \end{aligned}

Put x = r_{a}, y = r_{b}, z = r_{c} and t = xy. Let α : W → W be the homomorphism for which

α(x) = x, α(y) = y and α(z) = tzt^{−1}.

It is easy to check that, in fact, α ∈ Aut(W). Suppose that α^{i} ∈ Inn(W) for some i ∈ ℤ. Then α^{i}(x) = x = wxw^{−1}, α^{i}(y) = y = wyw^{−1} and α^{i}(z) = t^{i}zt^{−i} = wzw^{−1} for some w ∈ W. Hence

w ∈ C_{W} (x) ∩ C_{W} (y) = ⟨ x ⟩ ∩ ⟨ y ⟩ = 1,

and so t^{i} ∈ C_{W} (z) = ⟨ z ⟩. Since t has infinite order, this implies that i = 0. Consequently the image of α in Out(W) has infinite order.

2. Notation and Preliminary Results

Assume that V is a vector space over ℝ equipped with a symmetric bilinear form F : V × V → ℝ and assume that Π is a finite subset of V which spans V and is a root basis in the sense of the next definition.

Definition 2.1. A subset Π ⊆ V is a root basis (relative to the bilinear form F) if

(i) for all a, b ∈ Π,

F (a, b) = − cos \frac{π}{m_{ab}}  if m_{ab} ≠ ∞,
F (a, b) ≤ −1  if m_{ab} = ∞,

where (m_{ab}) is a Coxeter matrix, and (ii) the zero vector is not contained in the set

X = { ∑_{a∈Π} λ_{a}a | λ_{a} ≥ 0 for all a, and λ_{a} ≠ 0 for some a }.

For each x ∈ V with F (x, x) = 1 we define r_{x} : V → V (the reflection corresponding to x) by

r_{x}v = v − 2F (x, v)x

for all v ∈ V. It is easily checked that r_{x} is linear and preserves F.

It can be shown that (1.1) is a presentation for the subgroup of O(V, F) (the orthogonal group with respect to F) generated by { r_{a} | a ∈ Π }, identifying the generators in (1.1) with the reflections as the notation suggests. We refer the reader to Bourbaki [1] for the proof of this and all other assertions about Coxeter groups whose proofs we omit (unless we give an alternative reference). Note, however, that unlike Bourbaki we do not assume that Π is linearly independent, as the weaker assumption 2.1(ii) suffices for the proofs. If Π is linearly independent we call it a free root basis, and W a standard geometric realization of the Coxeter group (1.1). Note that, given any presentation of the form (1.1), we can construct a standard geometric realization by defining V to be the vector space which has the indexing set Π as a linear basis, and then defining the form F so that 2.1(i) is satisfied.

The root system of W in V is the set

Φ = { wa | w ∈ W and a ∈ Π },

and the elements of Φ^{+} = Φ∩X and Φ^{−} = Φ∩(−X) are called positive and negative roots respectively. The elements of Π are called simple roots.

By Definition 2.1(ii) it is clear that Φ^{+} ∩ Φ^{−} = ∅, and it can be shown that Φ = Φ^{+} ∪ Φ^{−}. For each w ∈ W define

N(w) = { a ∈ Φ^{+} | wa ∈ Φ^{−} },

and note that N(r_{a}) = {a} whenever a ∈ Π. It follows that for each w ∈ W and a ∈ Π,

N(wr_{a}) = \begin{cases}r_{a}N(w) ∪ \{a\} & if wa ∈ Φ^{+} \\ r_{a}(N(w) \ \{a\}) & if wa ∈ Φ^{−}, \end{cases} (2.1)

and |N(w)| = ℓ(w), the length of a shortest possible expression for w as a product of the generators { r_{a} | a ∈ Π }.

We say that W (or the root system Φ) is irreducible if it is impossible to express Φ as the union of two non-empty subsets which are orthogonal relative to F. It is easily seen that Φ is irreducible if and only if Π cannot be expressed as the union of two non-empty orthogonal subsets and this in turn is equivalent to connectedness of the associated diagram. Clearly every root system is uniquely expressible as a union of irreducible root subsystems.

The group W is finite if and only if the form F is positive definite. In consequence, if W is finite then the root basis Π is necessarily free. If Φ is infinite and irreducible and F is positive semidefinite, then W is a so-called “affine Weyl group”, corresponding to one of the diagrams on page 34, §2.5 of [6]. In each case the radical of F is one-dimensional.

It is easily checked that if g ∈ O(V, F) and if x ∈ V satisfies F (x, x) = 1, then gr_{x}g^{−1} = r_{gx}. Applying this with x = a ∈ Π and g ∈ W it follows that r_{b} ∈ W for all b ∈ Φ. In the converse direction we have the following result

Lemma 2.2. Let b ∈ V such that F (b, b) = 1 and r_{b} ∈ W. Then b ∈ Φ.

Proof. We use induction on ℓ(r_{b}). If ℓ(r_{b}) = 1, then r_{b} = r_{a} for some a ∈ Π, and hence b = ±a ∈ Φ. Now suppose that ℓ(r_{b}) > 1, and choose a ∈ Π such that ℓ(r_{a}r_{b}) = ℓ(r_{b}) − 1. By (2.1), r_{b}a ∈ Φ^{−}. Note that ℓ(r_{a}r_{b}) = ℓ(r_{b}r_{a}) since these elements are inverses of each other. So by (2.1) again,

ℓ(r_{a}r_{b}r_{a}) = \begin{cases}ℓ(r_{b}) & if r_{a}(r_{b}a) ∈ Φ^{+}, \\ ℓ(r_{b}) − 2 & if r_{a}(r_{b}a) ∈ Φ^{−}. \end{cases}

However, if r_{a}(r_{b}a) ∈ Φ^{+}, then −r_{b}a = a because −r_{b}a ∈ Φ^{+} and N(r_{a}) = {a}. But then b = ±a ∈ Φ, and so we may assume that r_{a}(r_{b}a) ∈ Φ^{−}, and hence ℓ(r_{a}r_{b}r_{a}) = ℓ(r_{b}) − 2. But since r_{a}r_{b}r_{a} = r_{r_{a}(b)} it follows by the inductive hypothesis that r_{a}b ∈ Φ, and hence b ∈ Φ, as required.

For each Γ ⊆ Φ the subgroup W_{Γ} generated by { r_{a} | a ∈ Γ } is called a reflection subgroup of W. The set Φ_{Γ} = { a ∈ Φ | r_{a} ∈ W_{Γ} } is called the (root) subsystem generated by Γ. Let Φ_{Γ}^{+} = Φ_{Γ} ∩ Φ^{+} and Φ_{Γ}^{−} = Φ_{Γ} ∩ Φ^{−}, and define

Π_{Γ} = { a ∈ Φ_{Γ}^{+} | |N(r_{a}) ∩ Φ_{Γ}| = 1 }.

The main theorem of Deodhar [4] (in particular, steps 2 and 3 of its proof) and also Theorem (3.3) of Dyer [5] yield the following result.

Theorem 2.3. If Γ ⊆ Φ, then the subgroup W_{Γ} is a Coxeter group on the generating set { r_{a} | a ∈ Π_{Γ} }.

The following is Theorem (4.4) of Dyer [5].

Theorem 2.4. A subset Π′ ⊆ Φ^{+} equals Π_{Γ} for some Γ ⊆ Φ if and only if the following condition holds: for all a, b ∈ Π′ either F (a, b) = − cos \frac{π}{m} for some integer m, or else F (a, b) ≤ −1.

It is clear from Definition 2.1(ii) that if Π′ ⊆ Φ^{+} then the zero vector cannot be nontrivially expressed in the form ∑_{a∈Π′} λ_{a}a with all λ_{a} ≥ 0. Hence Theorem 2.4 shows that Π_{Γ} is a root basis in all cases. Thus Φ_{Γ} is finite if and only if the restriction of F to Span(Γ) is positive definite. We remark that for an arbitrary Γ it is not necessarily the case that Π_{Γ} is finite, despite our assumption that Π is.

If J ⊆ Π then it is easily shown that Π_{J} = J. In this case the subgroup W_{J} is called a standard parabolic subgroup of W, and Φ_{J} a standard parabolic subsystem of Φ. A parabolic subgroup of W is any subgroup of the form wW_{J}w^{−1}, where J ⊆ Π and w ∈ W. Similarly, a parabolic subsystem is a root subsystem of the form wΦ_{J} for some w ∈ W and J ⊆ Π.

For a standard geometric realization, if Ψ is a parabolic subsystem of Φ then Ψ contains all roots expressible as linear combinations of Ψ; that is, Ψ = Φ ∩ Span Ψ. In the converse direction, if Φ is finite and Ψ is a root subsystem of Φ, then Φ ∩ Span Ψ is a parabolic subsystem of rank equal to dim(Span Ψ).

As mentioned in §1, we require the following result, which appears in [1], exercise 2d, p. 130.

Theorem 2.5 (Tits). If H is a finite subgroup of W, then H is contained in some finite parabolic subgroup of W.

Since a Coxeter group acts faithfully on its root system, the reflection subgroup corresponding to a finite root system will be finite, and so it follows from Theorem 2.5 that every such subsystem is contained in a finite parabolic subsystem. Thus we obtain the following result, which is essentially contained in [9], Lemma 3.2.3.

Proposition 2.6. If Ψ is a finite subsystem of Φ then Φ ∩ Span Ψ is a finite parabolic subsystem of Φ.

Proof. Since |Ψ| < ∞ the subspace Span Ψ is Euclidean, and so the subsystem Λ = Φ∩Span Ψ is finite. By Theorem 2.5 there exist w ∈ W and J ⊆ Π such that W_{Λ} ≤ w^{−1}W_{J}w, and |W_{J}| < ∞. Replacing Ψ by wΨ permits us to assume that W_{Λ} ≤ W_{J}, and hence that Λ = { a ∈ Φ | r_{a} ∈ W_{Λ} } ⊆ Φ_{J}. Since Λ = Φ_{J} ∩ Span Λ, we conclude that Λ is a parabolic subsystem of Φ_{J}, and hence a parabolic subsystem of Φ.

The following result can be found in Deodhar [3], Proof of Proposition 4.2.

Proposition 2.7. If Φ is infinite and irreducible then Φ\Φ_{J} is infinite whenever J is a proper subset of Π.

We also need the following technical lemma.

Lemma 2.8. Suppose that Γ ⊆ Φ has the property that W_{Γ} = W. Then Γ spans V.

Proof. An easy induction shows that if a_{1}, a_{2}, …, a_{ℓ} ∈ Γ, then the vector r_{a_{1}}r_{a_{2}} … r_{a_{ℓ}}v − v is a linear combination of a_{1}, a_{2} …, a_{ℓ}, for all v ∈ V. Since the simple reflections r_{a} can be expressed as products r_{a_{1}}r_{a_{2}} … r_{a_{ℓ}}, it follows that 2a = a − r_{a}a ∈ Span Γ for all a ∈ Π. Because Π spans V, so too does Γ.

The following was proved in a preliminary version of [2].

Proposition 2.9. If Γ ⊆ Φ, then |Π_{Γ}| ≤ |Γ|.

Proof. Let Ṽ be a real vector space with basis Π̃ in bijective correspondence with Π_{Γ}, and define a bilinear form F̃ on Ṽ so that the linear map ψ : Ṽ → V induced by the bijection Π̃ → Π_{Γ} satisfies F (ψu, ψv) = F̃(u, v) for all u, v ∈ Ṽ.

Then W̃ = ⟨ { r_{a} | a ∈ Π̃ } ⟩ is a standard geometric realization of the Coxeter group W_{Γ}, and there is an isomorphism σ : W̃ → W_{Γ} such that σr_{a} = r_{ψa}. It is easily checked that ψ(r_{a}v) = r_{ψa}(ψv) for all a ∈ Π̃ and v ∈ Ṽ. By induction on ℓ(w) it follows that

ψ(wv) = (σw)(ψv) (2.2)

for all w ∈ W̃ and v ∈ Ṽ. Hence ψ maps the root system Φ̃ of W̃ in Ṽ to Φ_{Γ}, and since ψΠ̃ = Π_{Γ} it follows that ψ(Φ̃^{+}) = Φ_{Γ}^{+}.

Since σ maps the reflections in W̃ bijectively to the reflections in W_{Γ}, the restriction of ψ to Φ̃^{+} is a bijection Φ̃^{+} → Φ_{Γ}^{+}. Hence a ↦ ψa is a bijection from Φ̃ to Φ_{Γ}.

The required conclusion is now an easy consequence of Lemma 2.8. Let Γ̃ = { a ∈ Φ̃ | ψa ∈ Γ }. Since { r_{a} | a ∈ Γ } generates W_{Γ} it follows that { σ^{−1}(r_{a}) | a ∈ Γ } = { r_{b} | b ∈ Γ̃ } generates W̃, and hence Γ̃ spans Ṽ. Since Π̃ is a basis for Ṽ it follows that |Π̃| ≤ |Γ̃|, whence |Π_{Γ}| ≤ |Γ| as claimed.

We conclude this section with another elementary technical lemma which will be needed in Section 4 below.

Lemma 2.10. If Φ is irreducible and a, b ∈ Φ then there exists c ∈ Φ with F (a, c) ≠ 0 and F (b, c) ≠ 0.

Proof. If F (a, b) ≠ 0 we may take c = a. So assume that F (a, b) = 0. Now Φ_{{a,b}} = {±a, ±b} is finite, and so, by Proposition 2.6, Span{a, b}∩ Φ is a parabolic subsystem of rank 2. If this system contains c ∉ {±a, ±b} then clearly F (a, c) and F (b, c) are both nonzero, as required. So we may assume that Span{a, b} ∩ Φ = {±a, ±b} is a parabolic subsystem, and replacing a and b by wa and wb (if necessary) for some w ∈ W, we may assume that a, b ∈ Π.

Since Φ is irreducible, we may choose a_{0}, a_{1}, …, a_{k} ∈ Π with a = a_{0}, b = a_{k} and F (a_{i−1}, a_{i}) ≠ 0 for i = 1, 2, …, k. Do this so as to minimize k. Then d = r_{a_{k−1}}r_{a_{k−2}} … r_{a_{2}}(a_{1}) has the required properties (since d = ∑_{i=1}^{k−1} μ_{i}a_{i} for some positive coefficients μ_{i}).

3. Cones, Hyperbolic Coxeter Groups and Dominance

A cone in a finite dimensional real vector space V is a subset of V which is closed under addition and multiplication by positive scalars.

If V^{∗} is the dual space and S is a cone in V then the polar cone of S is the set

{ f ∈ V^{∗} | fv ≤ 0 for all v ∈ S }.

It is more convenient for us to work with the negative of this, and so we define

S^{∗} = { f ∈ V^{∗} | fv ≥ 0 for all v ∈ S }.

It is clear that S^{∗} is topologically closed (relative to the usual topology for real vector spaces) and also that S ⊆ S^{∗∗}.

The following is well-known and not hard to prove.

Proposition 3.1. If S is any cone in V, then S^{∗∗} = \overline{S}, the topological closure of S.

Note that the set X appearing in Definition 2.1(ii) is a cone. It is also well known that the assumption that 0 ∉ X is equivalent to the statement that there exists an element f ∈ V^{∗} such that fa > 0 for all a ∈ Π. That is, the set

C = { f ∈ V^{∗} | fv > 0 for all v ∈ X. }

is non-empty. Note that \overline{C} = X^{∗}.

For w ∈ W and f ∈ V^{∗} we define wf ∈ V^{∗} by

(wf)v = f(w^{−1}v) for all v ∈ V.

The Tits cone is defined to be the set

U = ⋃_{w∈W} w\overline{C}.

See Tits [12] (or Bourbaki [1]) for the proof that U is in fact a cone.

If Φ is reducible then the Tits cone for Φ is the sum of the Tits cones for the irreducible components. Note that we continue to assume that the space V is spanned by Φ. For finite root systems the Tits cone equals V^{∗}. For infinite irreducible systems we have the following result, which appears in [9], Theorem 2.1.6; see also Vinberg [13], Lemma 15, p. 1112.

Proposition 3.2. If Φ is infinite and irreducible, then U ∩ (−U) = {0}.

Proof. Since U ∩ (−U) = ∪_{w_{1},w_{2}∈W} (w_{1}\overline{C} ∩ w_{2}(−\overline{C})) it follows that if U ∩ (−U) ≠ \{0\} then there exists a nonzero f ∈ \overline{C} such that −wf ∈ \overline{C} for some w ∈ W. Let J = { a ∈ Π | fa = 0 } so that fa > 0 for all a ∈ Π \ J. If b ∈ Φ^{+} \ Φ_{J} then fb > 0, whence (−wf)(wb) = −fb < 0 and so wb ∈ Φ^{−} (since −wf ∈ \overline{C}). Hence Φ^{+} \ Φ_{J} is contained in the finite set N(w). By Proposition 2.7 it follows that J = Π, contradicting f ≠ 0.

Observe that

U^{∗} = { v ∈ V | fv ≥ 0 for all v ∈ U }
= \{ v ∈ V | (wf)v ≥ 0 for all f ∈ \overline{C} and w ∈ W \}
= \{ v ∈ V | f(w^{−1}v) ≥ 0 for all f ∈ \overline{C} and all w ∈ W \}
= \{ v ∈ V | w^{−1}v ∈ \overline{X} for all w ∈ W \}

since \overline{X} = X^{∗∗} = \overline{C}^{∗}. Note also that \overline{X} = X ∪ \{0\} is the set of all non-negative linear combinations of Π. Thus U^{∗} consists of all v ∈ V such that wv is a non-negative linear combination of Π for all w ∈ W. Although we make no subsequent use of it, the following fact may be of some interest.

Proposition 3.3. Suppose that v ∈ \overline{X} and F (a, v) ≤ 0 for all a ∈ Π. Then v ∈ U^{∗}.

Proof. We use induction on ℓ(w) to show that wv ∈ \overline{X} for all w ∈ W. This holds by hypothesis when ℓ(w) = 0. Assume then that ℓ(w) > 0, so that N(w) ≠ ∅ and hence N(w) ∩ Π ≠ ∅, and choose a ∈ N(w) ∩ Π. By (2.1) we have that w = w′r_{a} where ℓ(w′) = ℓ(w) − 1, and so the inductive hypothesis yields w′v ∈ \overline{X}. Now

wv = (w′r_{a})v
= w′(v − 2F (a, v)a)
= w′v − 2F (a, v)w′a ∈ \overline{X}

since w′a = −wa ∈ Φ^{+} ⊆ \overline{X} and −2F (a, v) ≥ 0.

We remark that it is not necessarily the case that every element of U^{∗} has the form wv for some w ∈ W and some v satisfying F (a, v) ≤ 0 for all a ∈ Π. For example, if Π = {a, b} with F (a, b) = −2, then a + (2 + \sqrt{3})b is in U^{∗} but cannot be expressed in the stated form.

The next result was proved by Maxwell [10] for the case of a free root basis. The proof for an arbitrary root basis is somewhat more complicated.

Proposition 3.4. If v ∈ U^{∗} then F (v, v) ≤ 0.

Proof. To prove this we first choose a free root basis Π̃ in a space Ṽ, relative to a form F̃, such that there is a linear map η : Ṽ → V which restricts to a bijection Π̃ → Π and carries F̃ to F. Then η is a W -module homomorphism and Z = ker η is contained in the radical of F̃.

Let

Y = { ∑_{a∈Π̃} λ_{a}a | λ_{a} ≥ 0 for all a ∈ Π̃ },

and for v = ∑_{a∈Π̃} λ_{a}a ∈ Ṽ define S(v) = ∑_{a∈Π̃} λ_{a}. Observe that η^{−1}(U^{∗}) = ⋂_{w∈W} (wY + Z), and note also by Definition 2.1(ii) that Y ∩ Z = {0}.

Lemma 3.5. For each v ∈ Ṽ the set (v + Z) ∩ Y is bounded.

Proof. If not, then there exist elements v + z_{n} ∈ (v + Z) ∩ Y such that S(v + z_{n}) → ∞ as n → ∞. Now the elements S(v + z_{n})^{−1}(v + z_{n}) lie in the compact set Y_{1} = { y ∈ Y | S(y) = 1 }. A convergent subsequence gives a limit in Z ∩ Y_{1}, contradicting Z ∩ Y = {0}.

For each v ∈ Z + Y define Z(v) = { z ∈ Z | v + z ∈ Y } and M_{v} = sup{ S(v + z) | z ∈ Z(v) } (which is well-defined in view of Lemma 3.5).

Now suppose, for a contradiction, that η^{−1}(U^{∗}) contains elements v with F̃(v, v) > 0, and choose v_{0} ∈ η^{−1}(U^{∗}) with F̃(v_{0}, v_{0}) = 1. Put M = M_{v_{0}} and ε = 2/nM, where n = |Π̃|. Clearly v_{0} is contained in

U = { v ∈ η^{−1}(U^{∗}) | F̃(v, v) = 1 and M_{v} ≤ M }.

We show that for any given v ∈ U there exists v′ ∈ U with M_{v′} ≤ M_{v} − ε. A finite number of iterations of this process yields a v ∈ U with M_{v} < 0, which is absurd.

Given v ∈ U, since v ∈ η^{−1}(U^{∗}) ⊆ Z + Y, we may choose z ∈ Z(v), and write v + z = ∑_{a∈Π̃} λ_{a}a. Then

1 = F̃(v + z, v + z) = ∑_{a∈Π̃} λ_{a}F̃(a, v),

and so there exists an a ∈ Π̃ with λ_{a}F̃(a, v) ≥ 1/n. Since λ_{a} ≤ S(v + z) ≤ M_{v} ≤ M it follows that F̃(a, v) ≥ 1/nM = ε/2.

Put v′ = r_{a}v. Then v′ ∈ η^{−1}(U^{∗}) (since η^{−1}(U^{∗}) is W -invariant) and F̃(v′, v′) = F̃(v, v) = 1; so to show that v′ ∈ U it remains to verify that M_{v′} ≤ M.

Since v′ = r_{a}v = v − 2F̃(a, v)a we see that

S(v′) = S(v) − 2F̃(a, v) ≤ S(v) − ε.

Hence if z′ ∈ Z(v′) then

v + z′ = v′ + z′ + 2F̃(a, v)a ∈ Y,

and so z′ ∈ Z(v). Hence,

S(v′ + z′) = S(v′) + S(z′)
≤ S(v) − ε + S(z′)
≤ M_{v} − ε,

and consequently M_{v′} ≤ M_{v} − ε. Since M_{v} ≤ M this yields M_{v′} ≤ M, so that v′ is an element of U satisfying M_{v′} ≤ M_{v} − ε, as required to complete the proof of the proposition.

Corollary 3.6. If W is an irreducible affine Weyl group then U^{∗} = { λρ | λ ≥ 0 } where ρ spans the radical of F.

Proof. Since W is infinite we have U ∩ (−U) = {0} by Proposition 3.2, and so U^{∗} ≠ \{0\} (since otherwise \overline{U} = U^{∗∗} would equal V^{∗}). By Proposition 3.4 there exists nonzero ρ ∈ U^{∗} such that F (ρ, ρ) ≤ 0, whence F (ρ, ρ) = 0 and ρ spans the radical. Since U^{∗} ∩ (−U^{∗}) ⊆ \overline{X} ∩ (−\overline{X}) = \{0\} it follows that U^{∗} consists only of the nonnegative multiples of ρ.

For our next two results we assume that W is a hyperbolic Coxeter group, by which we mean that V has a basis x_{1}, x_{2}, …, x_{n} such that F (x_{i}, x_{j}) = 0 for i ≠ j and

F (x_{i}, x_{i}) = \begin{cases}1 & 1 ≤ i ≤ n − 1, \\ −1 & i = n. \end{cases}

Let H = \{ ∑ λ_{i}x_{i} | λ_{n} ≥ \sqrt{∑_{i=1}^{n−1} λ_{i}^{2}} \}, and observe that

{ v ∈ V | F (v, v) ≤ 0 } = H ∪ (−H).

Furthermore, it can be checked (using the Cauchy-Schwarz inequality) that H is a cone.

Since U^{∗} ≠ {0} (as in the proof of Corollary 3.6) it follows from Proposition 3.4 that U^{∗} ∩ (H ∪ (−H)) ≠ {0}. Replacing x_{n} by −x_{n} if necessary, we may suppose that U^{∗} ∩ H ≠ {0}. Define

L = { f ∈ V^{∗} | f(x) > 0 for all nonzero x ∈ H }.

Proposition 3.7 (Maxwell [10]). (i) U^{∗} ⊆ H. (ii) L ⊆ U.

Proof. Let 0 ≠ v ∈ H ∩ U^{∗} and suppose, for a contradiction, that there exists a nonzero vector v′ ∈ U^{∗} ∩ (−H). Write v = ∑ λ_{i}x_{i} and v′ = ∑ μ_{i}x_{i} and put v′′ = λ_{n}v′ − μ_{n}v. Since v′′ is a linear combination of x_{1}, …, x_{n−1} we have that F (v′′, v′′) ≥ 0. But v′′ ∈ U^{∗}, since v, v′ ∈ U^{∗} and the coefficients λ_{n} and −μ_{n} are both positive. So F (v′′, v′′) ≤ 0, by Proposition 3.4, whence v′′ = 0 since F is positive definite on Span{x_{1}, …, x_{n−1}}. So v′ is a positive multiple of −v, which shows that v and −v both lie in U^{∗} ⊆ \overline{X} = X ∪ \{0\}. Since X is a cone and 0 ∉ X, this forces v = 0, a contradiction. Hence U^{∗} ∩ (−H) = ∅ and so (i) holds.

From U^{∗} ⊆ H we obtain H^{∗} ⊆ U^{∗∗} = \overline{U}. Since U is convex, U contains the interior of \overline{U}. As L ⊆ int(H^{∗}) ⊆ int \overline{U}, the assertion of (ii) follows.

Corollary 3.8. C ∩ L ≠ ∅.

Proof. Clearly W preserves L∪(−L), but since it also preserves U, and as U ∩ (−U) = {0}, it follows from Proposition 3.7 that W preserves L. Since U = ⋃_{w∈W} w\overline{C} this implies that \overline{C} ∩ L ≠ ∅, and hence, since L is open, C ∩ L ≠ ∅.

We conclude this section with some results concerning the dominance partial order.

Definition 3.9. Let a, b ∈ Φ. If

{ w ∈ W | wa ∈ Φ^{−} } ⊆ { w ∈ W | wb ∈ Φ^{−} }

we say that a dominates b, and write a ≽ b (and a ≻ b if a ≠ b).

Although in [2] dominance was defined only on Φ^{+}, it is clear that defining it on all of Φ, as we have done, yields a partial order on Φ. Furthermore, this partial order is W -invariant. Clearly, also, if a ≽ b then −b ≽ −a.

Lemma 2.3 of [2] yields the following criterion for dominance.

Proposition 3.10. If a, b ∈ Φ, then F (a, b) ≥ 1 if and only if either a ≽ b or b ≽ a.

Suppose that Ψ is a subsystem of Φ and (temporarily) let ⪰ denote dominance on Ψ defined with respect to W_{Ψ}. Using Proposition 3.10 it is not hard to show, for all a, b ∈ Ψ that a ≽ b if and only if a ⪰ b.

A positive root a is said to be elementary if { b ∈ Φ^{+} | a ≻ b } is empty. The main result of [2] (Theorem 2.8) says that the set E = { a ∈ Φ^{+} | a is elementary } is finite.

Lemma 3.11. Suppose that Π = {a, b, c} and that Φ is infinite and irreducible. Then there exists α, β, γ ∈ Φ with α ≻ a, β ≻ b and γ ≻ c. Suitable values for α, β and γ for the various cases are as follows.

(i) ^{\begin{aligned}& c \\ t & & s \\ & ◯ & \\ ◯ a & & {◯ \atop b} \\ & r \end{aligned}}, where r, s, t ≥ 3: α = (r_{a}r_{b})c, β = (r_{b}r_{c})a, γ = (r_{c}r_{a})b.

(ii) ◯ a ^{\underline{r}} {◯ \atop b} ^{\underline{s}} ◯ c, where r, s ≥ 4: α = (r_{a}r_{b}r_{c}r_{b})a, β = (r_{b}r_{a}r_{c})b,

γ = (r_{c}r_{b}r_{a}r_{b})c.

(iii) ◯ a ^{\underline{r}} {◯ \atop b} ^{\underline{3}} ◯ c, where r ≥ 6: α = (r_{a}r_{b}r_{c}r_{a}r_{b})a, β = (r_{b}r_{a}r_{c})b,

γ = (r_{c}r_{b}r_{a})b.

Proof. To prove that α ≻ a it suffices to show that α ∈ Φ^{+} \ {a} and F (α, a) ≥ 1 since r_{a}(a) ∈ Φ^{−} and r_{a}(α) ∈ Φ^{+} shows that a does not dominate α.

So, writing α′ = r_{a}α, it suffices to show that α′ ∈ Φ^{+} and F (α′, a) ≤ −1. Similarly to prove that β ≻ b and γ ≻ c it suffices to show that β′ = r_{b}β and γ′ = r_{c}γ are positive roots satisfying F (β′, b) ≤ −1 and F (γ′, c) ≤ −1.

In case (i), α′ = c + 2(cos \frac{π}{s})b and so

F (α′, a) = − cos \frac{π}{t} − 2(cos \frac{π}{s})(cos \frac{π}{r}) ≤ −1,

since the cosines are all at least 1/2. Symmetrical arguments apply for β′ and γ′.

In case (ii), α′ = a + 8 cos \frac{π}{r} cos^{2} \frac{π}{s} b + 4 cos \frac{π}{r} cos \frac{π}{s} c, and so

F (α′, a) = 1 − 8 cos^{2} \frac{π}{r} cos^{2} \frac{π}{s} ≤ −1,

since the cosines are both at least 1/\sqrt{2}. A symmetrical argument applies for γ′. We leave the calculation for β′ to the reader.

In case (iii) similar calculations show that

α′ = (4λ^{2} − 1)a + 8(λ^{3} − 2λ)b + 2λc

and F (α′, a) = −8λ^{4} + 6λ^{2} − 1, where λ = cos \frac{π}{r}. Elementary calculus,

using the fact that λ ≥ ^{\sqrt{3}}/2, yields F (α′, a) ≤ −1. The calculations for β′ and γ′ are again left to the reader.

Proposition 3.12. In every infinite irreducible root system Φ, for each a ∈ Φ there exists b ∈ Φ with b ≻ a.

Proof. Since dominance is preserved by the W -action it suffices to prove that for each a ∈ Π there exists b ∈ Φ with b ≻ a. Since E is finite and Φ^{+} is infinite there certainly exist c, d ∈ Φ^{+} with c ≻ d, and by W -invariance of dominance we may choose d ∈ Π.

By irreducibility of Π it now suffices to show that if e ∈ Π with F (d, e) ≠ 0, there exists f ∈ Φ with f ≻ e. If F (d, e) ≤ −1 we may take f = r_{e}(d); so we may assume that the parabolic subsystem generated by d and e is finite. Let Γ = {c, d, e}, and note that the subsystem Φ_{Γ} generated by Γ is infinite since F (c − d, c − d) ≤ 0. Because Π_{Γ} contains d and e (as is immediate from the definition of Π_{Γ}) it follows from Proposition 2.9 that Φ_{Γ} has rank 3 and is irreducible. By Lemma 3.11 there is an f ∈ Φ_{Γ} with the required property.

Our final result for this section is a key ingredient in the proof of Theorem 1.2.

Proposition 3.13. Suppose that φ : Φ → Φ is a bijection that preserves dominance. Then

A = { a ∈ Φ^{+} | φa ∈ Φ^{−} }

is a finite set.

Proof. Let

B = { b ∈ Φ^{+} | −φe ≽ b for some e ∈ A ∩ E }.

Note that if e ∈ A ∩ E then f = −φe ∈ Φ^{+}, and so if f ≽ b then b ∈ N(r_{f}) (since f ∈ N(r_{f})). Hence B is a subset of the finite set ⋃_{f∈−φE} N(r_{f}).

Let a ∈ A and choose e ∈ E such that a ≽ e. Then φa ≽ φe since φ preserves dominance. Since φa ∈ Φ^{−} it follows that φe ∈ Φ^{−}. So e ∈ A. Also φa ≽ φe gives −φe ≽ −φa, and so −φa ∈ B. Hence A ⊆ φ^{−1}(−B), a finite set.

4. Proof of Theorem 1.2

In this section we shall often be dealing with two Coxeter groups simultaneously. We retain the notation of §3, using subscripts 1 and 2 to distinguish the groups.

Theorem 4.1. Let Φ_{1} and Φ_{2} be irreducible root systems spanning the spaces V_{1} and V_{2} with root bases Π_{1} and Π_{2}. Suppose that φ : V_{1} → V_{2} maps Φ_{1} bijectively to Φ_{2} and takes the bilinear form on V_{1} to that on V_{2}. Then Π_{2} = ±φwΠ_{1} for some w ∈ W_{1}.

Proof. If the root systems are finite this is well-known—indeed, Π_{2} = φwΠ_{1} for some w in this case—and so we assume that they are infinite. Note that x ↦ φxφ^{−1} is an isomorphism O(V_{1}, F_{1}) → O(V_{2}, F_{2}) which takes W_{1} to W_{2}.

Assume first that W_{1} is an irreducible affine Weyl group. Then so is W_{2} (since if F_{1} is positive semidefinite and Φ_{1} cannot be expressed as a union of two nonempty orthogonal subsets, then the same is true of F_{2} and Φ_{2}), and by what was shown in §3 the duals of the Tits cones are

U_{1}^{∗} = { λρ_{1} | λ ≥ 0 }
U_{2}^{∗} = { λρ_{2} | λ ≥ 0 }

where ρ_{1} and ρ_{2} span the radicals of V_{1} and V_{2} respectively. Since φ maps the radical of V_{1} to the radical of V_{2}, it follows that φU_{1}^{∗} = ±U_{2}^{∗}, and, replacing φ by −φ if necessary, we may assume that φU_{1}^{∗} = U_{2}^{∗}.

Denoting by φ^{∗} the map V_{2}^{∗} → V_{1}^{∗} induced by φ : V_{1} → V_{2}, we have

φ^{∗}\overline{U}_{2} = φ^{∗}(U_{2}^{∗∗}) = U_{1}^{∗∗} = \overline{U}_{1}.

Hence φ^{∗}(int(\overline{U}_{2})) = int(\overline{U}_{1}) = int U_{1}, and in particular, choosing f ∈ C_{2} arbitrarily, we have φ^{∗}f ∈ int U_{1}. We conclude that, in the case of affine Weyl groups, there exists f ∈ C_{2} with φ^{∗}f ∈ int U_{1} (φ having possibly been replaced by −φ).

Assume alternatively that the Coxeter groups W_{1} and W_{2} are hyperbolic. Choose f ∈ C_{2} ∩ L_{2}. Then φ^{∗}f ∈ φ^{∗}L_{2}, and after possibly replacing φ by −φ so that φH_{1} = H_{2} we have

φ^{∗}L_{2} = { φ^{∗}f | f ∈ V_{2}^{∗} and f(x) > 0 for all nonzero x ∈ H_{2} }

= { φ^{∗}f | f ∈ V_{2}^{∗} and (φ^{∗}f)(φ^{−1}x) > 0 for all nonzero x ∈ H_{2} }

= { f ∈ V_{1}^{∗} | f(φ^{−1}x) > 0 for all nonzero x ∈ H_{2} }
= { f ∈ V_{1}^{∗} | f(x) > 0 for all nonzero x ∈ H_{1} }
= L_{1} ⊆ int U_{1}

So in the hyperbolic case, as well as in the affine, there exists f ∈ C_{2} with φ^{∗}f ∈ int U_{1}. For such an f, we have that φ^{∗}f ∈ w\overline{C}_{1} for some w ∈ W_{1}. But since f(a) ≠ 0 for all a ∈ Φ_{2}, and since φ takes Φ_{1} to Φ_{2}, it follows that

(w^{−1}φ^{∗}f)(a) = f(φ(wa)) ≠ 0 for all a ∈ Φ_{1}.

So w^{−1}φ^{∗}f ∈ C_{1}. Now

Φ_{2}^{+} = { a ∈ Φ_{2} | f(a) > 0 }
= φw({ a ∈ Φ_{1} | f(φ(wa)) > 0 })
= φw({ a ∈ Φ_{1} | (w^{−1}φ^{∗}f)(a) > 0 })
= φw(Φ_{1}^{+}) (since w^{−1}φ^{∗}f ∈ C_{1}).

So Π_{2} = φw(Π_{1}), and so the result holds for affine and hyperbolic Coxeter groups (as well as finite ones).

Consider now the general case. If Ψ_{1} is a subsystem of Φ_{1} of rank 3 or less, corresponding to a reflection subgroup T_{1} ≤ W_{1}, then the restriction of φ yields a bijection from Ψ_{1} to Ψ_{2} = φ(Ψ_{1}) satisfying the hypotheses of the theorem. Since root systems of rank at most 3 are either finite, affine or hyperbolic, we conclude by the first part of the proof that there exists w ∈ T_{1} such that either Ψ_{2}^{+} = φwΨ_{1}^{+} or Ψ_{2}^{+} = −φwΨ_{1}^{+}. We show that the dominance partial order on Ψ_{1} is preserved by φ in the former case, reversed by φ in the latter.

Write T_{2} for the reflection subgroup of W_{2} corresponding to the subsystem Ψ_{2}, noting that the isomorphism x ↦ φxφ^{−1} takes T_{1} to T_{2}. Assuming the former of our two cases, if a, b ∈ Ψ_{1} with a ≽ b, then

{ x ∈ T_{2} | xφa ∈ Ψ_{2}^{−} } = φ{ y ∈ T_{1} | φya ∈ Ψ_{2}^{−} }φ^{−1}
= φ{ y ∈ T_{1} | φya ∈ φwΨ_{1}^{−} }φ^{−1}
= φ{ y ∈ T_{1} | w^{−1}ya ∈ Ψ_{1}^{−} }φ^{−1}
⊆ φ{ y ∈ T_{1} | w^{−1}yb ∈ Ψ_{1}^{−} }φ^{−1}
= φ{ y ∈ T_{1} | φyb ∈ φwΨ_{1}^{−} }φ^{−1}
= φ{ y ∈ T_{1} | φyb ∈ Ψ_{2}^{−} }φ^{−1}
= { x ∈ T_{2} | xφb ∈ Ψ_{2}^{−} },

and hence φa ≽ φb. So φ preserves dominance on Ψ_{1}, as claimed. Since multiplication by −1 reverses dominance it is clear that dominance is reversed in the other case.

We now show that φ either preserves dominance or reverses it on the entire root system. Let a, b ∈ Φ_{1} with a ≻ b. If φb ≻ φa replace φ by −φ, so that φa ≻ φb. We show that now whenever c, d ∈ Φ_{1} with c ≻ d then φc ≻ φd.

Let c, d ∈ Φ_{1} with c ≻ d. If F (b, c) ≠ 0 then Φ_{{a,b,c}} is an infinite irreducible root subsystem of rank at most 3, and since a, b ∈ Φ_{{a,b,c}} the restriction of φ to Φ_{{a,b,c}} must preserve dominance rather than reverse it. Furthermore, there exists f ∈ Φ_{{a,b,c}} with f ≻ c. Now φf ≻ φc and hence φ preserves dominance on Φ_{{f,c,d}}. Thus φc ≻ φd.

We may suppose that F (b, c) = 0. By Lemma 2.10 there exists f ∈ Φ with F (f, b) ≠ 0 and F (f, c) ≠ 0. Since Φ_{{a,b,f}} is infinite and irreducible, there exists g ∈ Φ_{{a,b,f}} with g ≻ f. Furthermore φ preserves dominance on Φ_{{a,b,f}} and so φg ≻ φf. Now Φ_{{g,f,c}} is infinite and irreducible and so there exists h ∈ Φ_{{g,f,c}} with h ≻ c. Moreover, φ preserves dominance on Φ_{{g,f,c}} (since φg ≻ φf) and so φh ≻ φc. Now applying the same argument to Φ_{{h,c,d}} we conclude that φc ≻ φd, as required.

Since φ preserves dominance, the set { a ∈ Φ_{1}^{+} | φa ∈ Φ_{2}^{−} } is finite, by Proposition 3.13. Now choose w ∈ W so as to minimize the size of the set S = { a ∈ Φ_{1}^{+} | φwa ∈ Φ_{2}^{−} }. If S ≠ ∅ then S must contain a simple root b, and then it is clear that

{ a ∈ Φ_{1}^{+} | (φwr_{b})a ∈ Φ_{2}^{−} } = r_{b}(S \ {b})

has fewer elements than S, contradicting the choice of w. So (φw)Φ_{1}^{+} = Φ_{2}^{+}, and hence (φw)Π_{1} = Π_{2}, as required.

Of course, Theorem 1.2 is simply the case Φ_{1} = Φ_{2} of Theorem 4.1.

5. Proof of Theorem 1.1

Recall that our aim is to prove that if W is a Coxeter group such that |Π| < ∞ and m_{ab} < ∞ for all a and b ∈ Π, then Out(W) is a finite group.

Proof. Choose a standard geometric realization of W, so that Π is a basis for a real vector space V on which W acts in the manner described in § 2 above. Since |Π| < ∞ clearly J = { J ⊆ Π | |W_{J}| < ∞ } is a finite set. Let I be the set of involutions of W and P = { (r, s) | r, s ∈ I and |rs| is finite }. The group W acts on P by conjugation and Aut W permutes the orbits. By Theorem 2.5 if (r, s) ∈ P then there exists w ∈ W and a J ∈ J such that w⟨ r, s ⟩w^{−1} ⊆ W_{J}. Since J is finite and for each J ∈ J there are only finitely many pairs of involutions in W_{J}, it follows that W has only finitely many orbits on P.

Let K be the kernel of the action of Aut W on the set of W -orbits on P. Since the number of orbits is finite, the index of K in Aut(W) is finite. If a, b ∈ Π then (r_{a}, r_{b}) ∈ I, since m_{ab} < ∞, and if σ ∈ K then (σ(r_{a}), σ(r_{b})) is in the same W orbit of I as (r_{a}, r_{b}). Hence for each σ ∈ K there exists w = w(σ, a, b) ∈ W such that

σ(r_{a}) = wr_{a}w^{−1}, σ(r_{b}) = wr_{b}w^{−1}.

We may choose these elements so that w(σ, a, b) = w(σ, b, a) for each two element subset {a, b} of Π.

Given σ ∈ K and a ∈ Π, define σ̂(a) = w(a), where w = w(σ, a, a). Note that if v = w(σ, a, b), where b ∈ Π, then

r_{w(a)} = wr_{a}w^{−1} = σ(r_{a}) = vr_{a}v^{−1} = r_{v(a)}

and so v(a) = ±w(a). It follows that σ̂(a) = ±w(σ, a, b)(a) for all b ∈ Π.

Let a, b ∈ Π. Then for each σ ∈ K there exist signs ε(σ, a, b) = ±1 such that

F (σ̂(a), σ̂(b)) = ε(σ, a, b)F (w(σ, a, b)(a), w(σ, a, b)(b))
= ε(σ, a, b)F (a, b).

We show that if σ, τ ∈ K and ε(σ, a, b) = ε(τ, a, b) for all a, b ∈ Π, then τ ∈ (Aut_{V} (W) ∩ K)σ.

By Lemma 2.8 { σ̂(a) | a ∈ Π } and { τ̂(a) | a ∈ Π } both span V, and hence are both bases of V. Furthermore, F (σ̂(a), σ̂(b)) = F (τ̂(a), τ̂(b)) for all a, b ∈ Π. Hence there exists an orthogonal transformation T: V → V such that T σ̂(a) = τ̂(a) for all a ∈ Π. Now

T σ(r_{a})T^{−1} = T r_{σ̂(a)}T^{−1} = r_{T σ̂(a)} = r_{τ̂(a)} = τ(r_{a}) = (τσ^{−1})(σ(r_{a}))

and since the elements σ(r_{a}) generate W it follows that T wT^{−1} = (τσ^{−1})(w) for all w ∈ W. Hence τσ^{−1} ∈ Aut_{V} (W); so τ ∈ (Aut_{V} (W)∩ K)σ, as claimed.

As there only finitely many possibilities for the signs ε(σ, a, b) for each σ, it follows that the index of Aut_{V} (W) ∩ K in K is finite. Since the index of K in Aut(W) is finite, the index of Aut_{V} (W) in Aut(W) is also finite. So it remains to show that the index of Inn(W) in Aut_{V} (W) is finite. But if σ ∈ Aut_{V} (W) then there exists φ ∈ O(V, F) such that σ(x) = φxφ^{−1} for all x ∈ W, and by Theorem 1.2 there exists a permutation π of Π, a sign ε = ±1 and a w ∈ W such that φa = w(επa) for all a ∈ Π. It follows that F (επa, επb) = F (a, b) for all a, b ∈ Π, and so there exists a T ∈ O(V, F) satisfying T a = επa for all a ∈ Π. Now φ = wT, and, furthermore, since T preserves Φ there exists τ ∈ Aut_{V} (W) given by τ(x) = T xT^{−1} for all x ∈ W. Thus

σ(x) = φxφ^{−1} = wT xT^{−1}w^{−1} = wτ(x)w^{−1}

for all x ∈ W, whence σ ∈ Inn(W)τ. As there are only finitely many possibilities for the permutation π and the sign ε, there are only finitely many possibilities for τ, and so the index of Inn(W) in Aut_{V} (W) is finite, as required.

References

[1] N. Bourbaki. Groupes et algèbres de Lie. Chapitres 4, 5 et 6. Hermann, Paris, 1968.

[2] B. Brink and R. B. Howlett. A finiteness property and an automatic structure for Coxeter groups. Math. Ann., 296:179–190, 1993.

[3] V. V. Deodhar. On the root system of a Coxeter group. Comm. Algebra, 10:611–630, 1982.

[4] V. V. Deodhar. A note on subgroups generated by reflections in Coxeter groups. Arch. Math. (Basel), 53:543–546, 1989.

[5] M. Dyer. Reflection subgroups of Coxeter systems. J. Algebra, 135:57–73, 1990.

[6] James E. Humphreys. Reflection Groups and Coxeter Groups, volume 29 of Cambridge Studies in Advanced Mathematics. Cambridge University Press, [7] Lynne D. James. Complexes and Coxeter groups – operations and outer auto-

morhisms. J. Algebra, 113:339–345, 1988.

[8] Victor G. Kac. Infinite Dimensional Lie Algebras. Cambridge University Press, third edition, 1990.

[9] Daan Krammer. The conjugacy problem for Coxeter groups. Dissertation, Universiteit Utrecht, 1994.

[10] G. A. Maxwell. Sphere packings and hyperbolic reflection groups. J. Algebra, 79:78–97, 1982.

[11] J. Tits. Sur le groupe des automorphismes de certains groupes de Coxeter. J. Algebra, 113:346–357, 1988.

[12] Jacques Tits. Groupes et géométries de Coxeter. Technical report, Institut des Hautes Études Scientifiques, Juin 1961.

[13] E. B. Vinberg. Discrete linear groups generated by reflections. Math. USSR-Izv., 5:1083–1119, 1971.

R. B. Howlett, School of Mathematics and Statistics, University of Sydney, NSW 2006 Australia

  E-mail address: howlett r@maths.su.oz.au

P. J. Rowley, Department of Mathematics, University of Manchester Institute of Science and Technology, Manchester M60 1QD, United Kingdom

  E-mail address: Peter.Rowley@umist.ac.uk

D. E. Taylor, School of Mathematics and Statistics, University of Sydney, NSW 2006 Australia

  E-mail address: taylor d@maths.su.oz.au