[Home]

Table of contents

Finite existence of $E(X)$

We know from the definition of expectation that it exists finitely if and only if the sum defining the expectation converges absolutely. It sometimes helps to know some sufficient conditions for this that are easier to check.
Theorem If $X$ takes only finitely many values, then $E(X)$ must exist finitely.

Proof: No infinite series involved here at all. Just a finite sum! [QED]

We shall use a concept from real analysis called the comparison test repeatedly below. Click on the link to learn about it.
Theorem If $X$ is bounded, then $E(X)$ must exist finitely.

Proof: Let $X$ take values $x_1,x_2,...$ with probabilities $p_1,p_2,...$ We are assumin $\exists B\in{\mathbb R}~~\forall i~~|x_i|\leq B.$ So $$ \sum_n |x_n|p_n\leq \sum_n B p_n = B\sum p_n. $$ We know the $\sum p_n$ converges (to $1$).

So, by comparison test, $\sum |x_n|p_n$ must converge. [QED]

Theorem Let $X$ be a random variable and $f,g$ be two functions such that $P(|f(x)|\leq |g(X)|) =1$ and $E|g(X)|<\infty.$ Then $E(f(X))$ must also have finite expectation.

Proof: Use comparison test between $\sum |f(x_n)|p_n$ and $\sum |g(x_n)|p_n.$ [QED]

Theorem Let $f,g:{\mathbb R}\rightarrow{\mathbb R}$ be such that $E(f(X))$ and $E(g(X))$ both exist finitely. Let $h:{\mathbb R}\rightarrow{\mathbb R}$ be defined as $h(x) = \max\{f(x),g(x)\}.$ Then $E(h(X))$ must also exist finitely.

Proof: Apply comparison test between $\sum \max\{f(x_n)p_n,g(x_n)p_n\}$ and $\sum (f(x_n)+g(x_n))p_n.$ [QED]

Theorem Let $m<n$ be any two positive integers. If $E(X^n)$ exists finitely, then $E(X^m)$ must also exist finitely.

Proof: Use the fact that $|X^m| \leq \max\{1,|X^n|\}.$ Now use the last theorem. [QED]

Variance

Of course, a random variable is random, and so may differ from its expectation. By how much? A lot or a little? We can use expectation to find that out.
Definition: Variance If $E|X|<\infty,$ the we define variance of $X$ as $$ V(X) = E\big[ (X-E(X))^2 \big]. $$ It is either finite or $\infty.$
Theorem $V(X)\geq 0.$
Theorem If $E(X^2)<\infty,$ then $V(X)$ exists finitely, and $V(X) = E(X^2)-\big( E(X) \big)^2.$
Theorem $V(aX+b) = a^2 V(X).$
Theorem $V(X)=0$ if and only if $X$ is a degenerate random variable.
Chebyshev is also credited with designing a quadruped robot-like linkage.
Chebyshev inequality Let $V(X)<\infty.$ Then $$ \forall \epsilon>0~~P(|X-E(X)| \geq \epsilon) \leq \frac{V(X)}{\epsilon^2}. $$

Proof: Take any $\epsilon>0.$

Let $E(X)$ be denoted by $\mu.$

Define $$ f(x) = \left\{\begin{array}{ll} \epsilon^2 &\text{if }|x-\mu|\geq \epsilon\\ 0 &\text{otherwise.} \end{array}\right. $$ Then $\forall x~~f(x)\leq (x-\mu)^2.$

So $$\begin{eqnarray*} V(X) & = & E(X-\mu)^2\\ & \geq & E(f(X))\\ & = & \epsilon^2 P(|X_i-\mu| \geq\epsilon) + 0\times P(|X_i-\mu| <\epsilon). \end{eqnarray*}$$ Hence the result. [QED]

Moments

Definition: Raw and central moments The $k$-th raw moment of $X$ is $$ E(X^k) $$ and the $k$-th central moment of $X$ is $$ E\big[ (X-E(X))^k \big]. $$

Moment generating function

Definition: Moment generating function (MGF) For any random variable $X$ we define its moment generating function as the function $$ m_X(t) = E(e^{tX}). $$ The domain of this function conists of all $t\in{\mathbb R}$ for which the expectation exists.
Clearly, $m_X(0)$ always exists and equals $1.$
Theorem If, for some $k\in{\mathbb N}$, the moment $E(X^k)$ exists finitely, then the $k$-th derivative of $m_X(t)$ exists at $t=0,$ and equals $E(X^k).$

Proof: We shall not do the proof here. But here is the main idea: $$ e^{tX} = 1 + \frac{tX}{1!} + \frac{t^2X^2}{2!} + \frac{t^3X^3}{3!} + \cdots. $$ From this we want to write $$ E(e^{tX}) = 1 + \frac{tE(X)}{1!} + \frac{t^2E(X^2)}{2!} + \frac{t^3E(X^3)}{3!} + \cdots. $$ This is not a precise statement, because we do not know if all raw moments of $X$ exist finitely. Also, even if they do, is it valid to "distribute" expectation over an infinite sum?

Answers to these questions require deeper real analysis results than we know at this point.

However, assuming that this is valid, we may try to differentiate both sides to get $$ \frac{d}{dt} E(e^{tX}) = E(X) + \frac{2tE(X^2)}{2!} + \frac{3t^2E(X^3)}{3!} + \cdots. $$ Again this step needs justification. Can we "distribute" differentiation over an infinite sum?

Assuming that we can, puting $t=0$ indeed gives us $E(X).$

SImilarly, differentiating once again, and putting $t=0$ gives us $E(X^2),$ and so on. [QED]

We shall not spend much time with MGFs, because there is a better alternative called the characteristic function (CF).
Definition: Characteristic function (CF) For any (real-valued) random variable $X$ we define its characteristic function as the function $$ \phi_X(t) = E(e^{itX}),~~t\in{\mathbb R}. $$
Don't be nervous to see expectation of a complex random variable. It is simply $$ E(\cos tX) + i E(\sin tX). $$ CFs are better than MGFs because of two reasons, that we give as theorems below.
Theorem For any (real-valued) random variable, the CF is defined over entire ${\mathbb R}.$

Proof: This is obvious, since $\sin tX$ and $\cos tX$ are both bounded random variables, and hence have finite expectations. [QED]

Theorem If $X,Y$ are two random variables with the same CF, then they must have the same distribution.

Proof: Not in this course. [QED]

Indeed, this property has earned characteristic functions their name.

MGFs do not have this proprty. It is possible to get (rather ugly) counter-examples of random variables $X$ and $Y$ that both have the same MGF (in particluar both have the same domain $D\subseteq{\mathbb R}$), but still $X$ and $Y$ have different distributions. However, if the domain includes a neighbourhood of $0,$ then $X,Y$ must have the same distribution. This is stated in the following theorem.
TheoremLet $m_X(t)$ be defined for $t\in (-a,a)$ for some $a>0.$ Let $Y$ be a random variable with the same MGF. Then $X$ and $Y$ must have the same distribution.

Proof: Too difficult for this course. [QED]

We shall not spend proving any result on MGF here. We shall learn the proofs for CFs in the next semester.

Problems for practice

  1. A box has 6 red balls an 4 black balls. An SRSWR of size $n$ is selected. If $X$ is the number of red balls selectrd, then find PMF and $E(X).$ Also solve the problem in the case of SRSWOR.
  2. Let $N$ be a positive integer. Let $$ f(x) = \left\{\begin{array}{ll}c 2^x &\text{if }x=1,2,...,N\\0&\text{otherwise.}\end{array}\right. $$ be a PMF. Find $c.$ Find $E(X)$ and $V(X)$ if $X$ has this PMF.
  3. An SRSWR of size 2 is drawn from $\{1,2,...,12\}.$ Let $X$ be the maximum of the two numbers selected. Find $E(X).$
  4. An SRSWR of size $n$ is selected from $\{1,2,...,12\}.$ Let $a_n $ be the expected value of the maximum of the sample. Show that $a_n \leq a_{n+1}$ without explicily finding $a_n$ in terms of $n.$