Foundations of Data Science

model, the energy of the system is given by
(
f(x 1 , x 2 , . . . , x n ) = exp c ∑ i∼j
|x i − x j |
)
.
The constant c can be positive or negative. If c < 0, then energy is lower if many adjacent
pairs have opposite spins and if c > 0 the reverse holds. The model was first
used to model probabilities of spin configurations. The hypothesis was that for each
{x 1 , x 2 , . . . , x n } in {−1, +1} n , the energy of the configuration with these spins is proportional
to f(x 1 , x 2 , . . . , x n ).
In most computer science settings, such functions are mainly used as objective functions
that are to be optimized subject to some constraints. The problem is to find the
minimum energy set of spins under some constraints on the spins. Usually the constraints
just specify the spins of some particles. Note that when c > 0, this is the problem of
minimizing ∑ |x i − x j | subject to the constraints. The objective function is convex and
i∼j
so this can be done efficiently. If c < 0, however, we need to minimize a concave function
for which there is no known efficient algorithm. The minimization of a concave function
in general is NP-hard.
A second important motivation comes from the area of vision. It has to to do with
reconstructing images. Suppose we are given observations of the intensity of light at
individual pixels, x 1 , x 2 , . . . , x n , and wish to compute the true values, the true intensities,
of these variables y 1 , y 2 , . . . , y n . There may be two sets of constraints, the first stipulating
that the y i must be close to the corresponding x i and the second, a term correcting possible
observation errors, stipulating that y i must be close to the values of y j for j ∼ i. This
can be formulated as ( ∑
min |x i − y i | + ∑ )
|y i − y j | ,
y
i
i∼j
where the values of x i are constrained to be the observed values. The objective function
is convex and polynomial time minimization algorithms exist. Other objective functions
using say sum of squares instead of sum of absolute values can be used and there are
polynomial time algorithms as long as the function to be minimized is convex.
More generally, the correction term may depend on all grid points within distance two
of each point rather than just immediate neighbors. Even more generally, we may have n
variables y 1 , y 2 , . . . y n with the value of some already specified and subsets S 1 , S 2 , . . . S m of
these variables constrained in some way. The constraints are accumulated into one objective
function which is a product of functions f 1 , f 2 , . . . , f m , where function f i is evaluated
on the variables in subset S i . The problem is to minimize ∏ m
i=1 f i(y j , j ∈ S i ) subject to
constrained values. Note that the vision example had a sum instead of a product, but by
taking exponentials we can turn the sum into a product as in the Ising model.
310