Definitions and Representations

stochastic process:$X=\{X_t:t\in T\}$

• $X_t$: the state of the process at time $t$.
• discrete space process: $X_t$ assumes values from a countably infinite set.
  finite process: $X_t$ assumes values from a finite set.
• discrete time process: $t$ is a countably infinite set.

Markov chain: $X_0,X_1,X_2,…$ is a Markov chain if

$$ \begin{aligned} Pr(X_t=a_t|X_{t-1}=a_{t-1},X_{t-2}=a_{t-2},...,X_0=a_0)&=Pr(X_t=a_t|X_{t-1}=a_{t-1})\\ &= P_{a_{t-1},a_t} \end{aligned} $$

Assume $P_{i,j}=Pr(X_t=j|X_{t-1}=i)$.
transition matrix:

$$ P=\left[ \begin{matrix} P_{0,0} & P_{0,1} & \cdots & P_{0,j} & \cdots \\ P_{1,0} & P_{1,1} & \cdots & P_{1,j} & \cdots\\ \vdots & \vdots & \ddots & \vdots &\ddots\\ P_{i,0} & P_{i,1} & \cdots & P_{i,j} &\cdots \\ \vdots & \vdots & \ddots & \vdots &\ddots \end{matrix} \right] $$