The purpose of this note is to prove the following result.
Let , and with (see Fig. 1). If satisfies the Helmholtz equation
(1)Δu+k2u=0in Ω in the distributional sense, then .


 Fig. 1. The conical domain Ω in the twodimensional case (d =1). Zoom 


 Fig. 1. The conical domain Ω in the twodimensional case (d =1). Zoom 
This theorem is optimal in the sense that it becomes false if . Indeed it is easy to construct solutions to the Helmholtz equation that are squareintegrable in a halfplane (see Remark 3).
As the wellknown Rellich's uniqueness theorem [[8]] and succeeding results (see, e.g., [[9]]), this theorem points out a forbidden behavior of solutions to the Helmholtz equation in an unbounded domain. In particular, it is not related to boundary conditions (no assumption is made on the behavior of u near the boundary of Ω). Most Rellich type results involve a particular Besov space related to the boundedness of the energy flux and lead to the uniqueness of the solution to scattering problems. Our theorem involves a more restrictive functional framework: the assumption rather expresses the boundedness of the total energy in Ω, which leads to the absence of socalled trapped modes (or bound states ) or equivalently, the absence of eigenvalues embedded in the continuous spectrum of a large class of operators that coincide with the Laplacian in Ω. For instance, if we consider the equation
Δu+k2n2u=0in Rd+1, with a variable index of refraction (say, bounded) such that in Ω, Theorem 1.1 together with the unique continuation principle (see, e.g., [[6]]) shows that this equation has no squareintegrable nonzero solutions.
Our initial motivation concerned the possible existence of trapped modes in waveguides. Such solutions are known to occur for local perturbations of closed uniform waveguides, that is, cylindrical infinite pipes with bounded crosssection (see [[7]] for a review). Trapped modes also appear in the case of curved closed waveguides (see, e.g., [[4]] and more recently [[3]]). The situation differs singularly in the case of open waveguides, that is, when the transverse section becomes unbounded, as for instance optical fibers or immersed pipes. It is now understood that trapped modes do not exist either in local perturbations of open straight waveguides [[1], [5]] or in straight junctions of waveguides [[2]]. The example above shows that this result holds true for curved open waveguides provided that the core of the waveguide and other possible inhomogeneities are located in .
The proof of Theorem 1.1 is based on the twodimensional case, which is explained in sections 2 and 3. Section 4 then shows how to deal with higher dimensions as well as some possible extensions of Theorem 1.1. Following [[1], [2], [5]] (inspired by the pioneering work of Weder [[10]]), the main ingredients of the proof are, on the one hand, a Fourier representation of a solution u to ((1)) in a halfplane and on the other hand, an analyticity property . The Fourier representation consists in decomposing u as a superposition of modes, which are either propagative or evanescent. Since we are only interested in squareintegrable solutions, the components associated with propagative modes must vanish. As in the abovementioned papers, the fact that the components associated with evanescent modes also vanish results from the analyticity property . Here the key idea to obtain this property is to reuse the Fourier representation in two oblique directions.


Fourier representation in a halfplane 
Let denote the usual Fourier transform defined for a function by
Fφ(ξ):=12π∫Rφ(x)e−ixξdxfor ξ∈R, which can be classically extended to the Schwartz space of tempered distributions.
For given and , let u be a solution to the Helmholtz equation ((1)) in the halfplane such that . Then
(2)u(x,y)=12π∫ξ>kφˆ(ξ)eixξ−yξ2−k2dξfor all (x,y)∈Π0‾, where has the following properties: (3)φˆ(ξ)=0 if ξ<kandφˆ(ξ)eεξξ2−k21/4∈L2(R).
Formula ((2)) appears as a modal representation of u (analogous to the plane wave spectrum representation of Fourier optics) in the sense that it can be interpreted as a superposition of the modes where stands for the modal amplitude. These modes are propagative in the xdirection and evanescent in the ydirection, since in ((2)). The first property in ((3)) expresses actually the absence of propagative modes in the ydirection, which results from the assumption .
The domain in which u is supposed to satisfy the Helmholtz equation, is larger than the domain where the representation ((2)) is written. This allows us to avoid the consequences of a possible poor regularity of u at the boundary of and yields a strong decay of as tends to ∞, which is expressed by the exponential term in the second property of ((3)). Note that by the Cauchy–Schwarz inequality, this property implies that , which shows that the integral in ((2)) makes sense.
Using the arguments of the proof below, it is readily seen that conversely to the statement of Proposition 2.1, if satisfies ((3)), then the function u defined by ((2)) belongs to and is a solution to the Helmholtz equation ((1)) in . This shows that Theorem 1.1 is no longer true for .
As , we know that for almost every , function belongs to so that we can define its Fourier transform . By the Parseval's identity,
uˆ(⋅,y)∈L2(R)and‖uˆ(⋅,y)‖L2(R)=‖u(⋅,y)‖L2(R). As a consequence,(4)uˆ∈L2(R×(−ε,+∞))and‖uˆ‖L2(R×(−ε,+∞))=‖u‖L2(Πε).
Applying the Fourier transform to the Helmholtz equation yields
∂2uˆ∂y2+(k2−ξ2)uˆ=0 in D′(−ε,+∞)for a.e. ξ∈R. Henceuˆ(ξ,y)=Aˆ(ξ)e−(y+ε)ξ2−k2+Bˆ(ξ)e+(y+ε)ξ2−k2, for some functions and , where denotes a given determination of the complex square root such that if . From ((4)), we have for almost every , which implies that, on the one hand, if , on the other hand, if . Therefore(5)uˆ(ξ,y)=Aˆ(ξ)e−(y+ε)ξ2−k2whereAˆ(ξ)=0 if ξ<k. Noticing that‖uˆ‖L2(R×(−ε,+∞))2=∫ξ>kAˆ(ξ)2∫−ε+∞e−2(y+ε)ξ2−k2dydξ=∫ξ>kAˆ(ξ)2dξ2ξ2−k2, we infer that . Setting and using the inverse Fourier transform of ((5)), the conclusion follows. □
The following corollary plays an essential role in the proof of Theorem 1.1.
For any halfline where , a solution to the Helmholtz equation ((1)) in is such that .
By the Fourier representation ((2)) of u , we have
∫0+∞u(tcosα,tsinα)dt=12π∫0+∞∫ξ>kφˆ(ξ)eiξtcosα−ξ2−k2tsinαdξdt≤12π∫0+∞∫ξ>kφˆ(ξ)e−ξ2−k2tsinαdξdt≤12π∫ξ>kφˆ(ξ)ξ2−k2sinαdξ, using Fubini's theorem. We deduce from the Cauchy–Schwarz inequality that∫0+∞u(tcosα,tsinα)dt≤12π‖φˆ(ξ)eεξξ2−k21/4‖L2(R)(∫ξ>ke−2εξξ2−k2sin2αdξ)1/2, where the righthand side is bounded, according to ((3)). □