A planar polygon is convex if it contains all the line segments connecting any pair of its points. Thus, for example, a regular pentagon is convex (left figure), while an indented pentagon is not (right figure). A planar polygon that is not convex is said to be a concave polygon.
Let a simple polygon have 
vertices 
for 
, 2, ..., 
, and define the edge vectors as
 |
(1)
|
where 
is understood to be equivalent to 
. Then the polygon is convex iff
all turns from one edge vector to the next have the same sense. Therefore, a simple
polygon is convex iff
 |
(2)
|
has the same sign for all 
, where 
denotes the perp
dot product (Hill 1994). However, a more efficient test that doesn't require
a priori knowledge that the polygon is simple is known (Moret and Shapiro 1991).
The happy end problem considers convex 
-gons and the minimal number of points 
(in the general position)
in which a convex 
-gon
can always be found. The answers for 
, 4, 5, and 6 are 3, 5, 9, and 17. It is conjectured that
, but only proven that
 |
(3)
|
where 
is a binomial coefficient.
