Foundations of Data Science

12 Appendix
12.1 Asymptotic Notation
We introduce the big O notation here. The motivating example is the analysis of the
running time of an algorithm. The running time may be a complicated function of the
input length n such as 5n 3 +25n 2 ln n−6n+22. Asymptotic analysis is concerned with the
behavior as n → ∞ where the higher order term 5n 3 dominates. Further, the coefficient
5 of 5n 3 is not of interest since its value varies depending on the machine model. So we
say that the function is O(n 3 ). The big O notation applies to functions on the positive
integers taking on positive real values.
Definition 12.1 For functions f and g from the natural numbers to the positive reals,
f(n) is O(g(n)) if there exists a constant c >0 such that for all n, f(n) ≤ cg(n).
Thus, f(n) = 5n 3 + 25n 2 ln n − 6n + 22 is O(n 3 ). The upper bound need not be tight.
Not only is f(n), O(n 3 ), it is also O(n 4 ). Note g(n) must be strictly greater than 0 for all n.
To say that the function f(n) grows at least as fast as g(n), one uses a notation called
omega of n. For positive real valued f and g, f(n) is Ω(g(n)) if there exists a constant
c > 0 such that for all n, f(n) ≥ cg(n). If f(n) is both O(g(n)) and Ω(g(n)), then f(n) is
Θ(g(n)). Theta of n is used when the two functions have the same asymptotic growth rate.
Many times one wishes to bound the low order terms. To do this, a notation called
f(n)
little o of n is used. We say f(n) is o(g(n)) if lim = 0. Note that f(n) being
n→∞ g(n)
O(g(n)) means that asymptotically f(n) does not grow faster than g(n), whereas f(n)
being o(g(n)) means that asymptotically f(n)/g(n) goes to zero. If f(n) = 2n + √ n, then
asymptotic upper bound
f(n) is O(g(n)) if for all n, f(n) ≤ cg(n) for some constant c > 0.
asymptotic lower bound
f(n) is Ω(g(n)) if for all n, f(n) ≥ cg(n) for some constant c > 0.
≤
≥
asymptotic equality
f(n) is Θ(g(n)) if it is both O(g(n)) and Ω(g(n)). =
f(n)
f(n) is o(g(n)) if lim = 0 . <
n→∞ g(n)
f(n)
f(n) ∼ g(n) if lim = 1. =
n→∞ g(n)
f(n)
f(n) is ω (g (n)) if lim = ∞. >
n→∞ g(n)
375