STAT3911 STOCHASTIC PROCESSES AND TIME SERIES (ADV ...

STAT3911 STOCHASTIC PROCESSES AND TIME SERIES (ADV) COURSE NOTES

Contents Introduction: ........................................................................................................................................... 6 Assessments: ....................................................................................................................................... 6 Schedule: ............................................................................................................................................. 6 Stochastic Processes: .............................................................................................................................. 7 Basic concenpts:.................................................................................................................................. 8 Definition: Stochastic process ......................................................................................................... 8 Definition: Random walk................................................................................................................. 8 Definition: Strong stationary........................................................................................................... 9 Definition: State space of stochastic process ................................................................................. 9 Definition: Increments .................................................................................................................... 9 Markov Property: .............................................................................................................................. 10 Definition: Markov property ......................................................................................................... 10 Definition: filtration ...................................................................................................................... 11 Definition: Markov Process ........................................................................................................... 12 Markov Chains ...................................................................................................................................... 12 Basic concepts: .................................................................................................................................. 12 Notation: ....................................................................................................................................... 12 Gambler ruin problem: ..................................................................................................................... 13 Solution ......................................................................................................................................... 13 Classification of states: ..................................................................................................................... 15 𝑛 step transition probability notation: ......................................................................................... 15 Recurrence and transience ............................................................................................................... 19 Recurrence definition: .................................................................................................................. 19 Transient definition:...................................................................................................................... 19 Green Function: ............................................................................................................................ 19 Period: ........................................................................................................................................... 22 Limiting theorems and Stationarity of Markov Chains ......................................................................... 24 Expected number of transitions from 𝑗 to 𝑗...................................................................................... 24 Definition: ..................................................................................................................................... 24 Number of visits to 𝑗 by the 𝑛𝑡ℎ step: .......................................................................................... 25 Stationary Distributions .................................................................................................................... 29 Discussion: .................................................................................................................................... 29 Set up: ........................................................................................................................................... 30 Theorem: irreducible aperiodic Markov chains and classes: ........................................................ 30 Poisson Processes ................................................................................................................................. 34 1

Introduction: ..................................................................................................................................... 34 Theory from notes: ....................................................................................................................... 35 Revision of Distributions ................................................................................................................... 35 Poisson distribution ...................................................................................................................... 35 Exponential Distribution ............................................................................................................... 37 Definition of Poisson Process:........................................................................................................... 39 Counting process (definition)........................................................................................................ 39 Poisson Process Definition: ........................................................................................................... 40 Interarrival and waiting time distributions: ...................................................................................... 41 Distribution of 𝐸𝑘: Interarrival time ............................................................................................. 41 Distribution of 𝑇𝑘: Waiting time................................................................................................... 42 Conditional Distribution of the Arrival Times ............................................................................... 43 Superposition and Thinning of Poisson Processes ............................................................................ 46 Theorem: Superposition position of Poisson Processes: .............................................................. 46 Sampling (thinning) ....................................................................................................................... 47 Revision: ................................................................................................................................................ 49 Random Sums (STAT2911) ................................................................................................................ 49 Examples: ...................................................................................................................................... 49 STAT3911 Random sums:.............................................................................................................. 51 Moment Generating Function .......................................................................................................... 52 Definition: ..................................................................................................................................... 52 Calculating MGF: ........................................................................................................................... 52 Moment Generating function of a random Sum .......................................................................... 56 Branching Processes: ............................................................................................................................ 57 Set up: ............................................................................................................................................... 58 Class .............................................................................................................................................. 58 Notes: ............................................................................................................................................ 59 Formulation of branching process: ................................................................................................... 59 Expectation and variance of 𝑍𝑛 + 1 ............................................................................................. 60 Probability of dying out:................................................................................................................ 61 Brownian motion .............................................................................................................................. 68 Definition 1.................................................................................................................................... 68 Definition 2.................................................................................................................................... 69 Time Series: ........................................................................................................................................... 69 Basic concepts: .................................................................................................................................. 71 Regular time seris: ........................................................................................................................ 71 2

Notation for TS data:..................................................................................................................... 71 Graphs of Time Series Data ........................................................................................................... 73 Basic Terminology of TS analysis................................................................................................... 80 Analysis of Components of time series: ............................................................................................ 82 Estimation and elimination of trend in absence of seasonality:................................................... 82 Estimation and elimination of both trend and seasonal components of a TS .............................. 89 Stationary Processes and Time Series 1:............................................................................................... 91 Autocovariance and Autocorrelation Functions ............................................................................... 92 Definition: autocovariance function 𝛾 .......................................................................................... 92 Definition: Autocorrelation (acf) 𝜌 ............................................................................................... 93 Estimation of 𝛾𝑘 and 𝜌𝑘 ............................................................................................................... 93 Sampling Properties of 𝑋𝑛; 𝐶𝑛, 𝑘; 𝑅𝑛, 𝑘 ........................................................................................... 95 Sampling properties of 𝑋𝑛 ............................................................................................................ 95 Sampling properties of 𝐶𝑛, 𝑘 ........................................................................................................ 96 Sampling properties for 𝑅𝑛, 𝑘 ....................................................................................................... 96 Sample Correlogram ..................................................................................................................... 97 Detection of Randomness, short term and long term correlations of a TS .................................. 98 Autocorrelation Plot as Diagnostic Tool ..................................................................................... 101 Partial Autocorrelation Function (PACF) ..................................................................................... 103 Stationary Time Series: ............................................................................................................... 104 Some Stochastic Models for Time Series ............................................................................................ 104 White Noise Process ....................................................................................................................... 104 Definition: ................................................................................................................................... 104 Statistical Properties of WN 𝑍𝑡 ................................................................................................... 105 Linear Combination of 𝑍𝑡 ............................................................................................................ 105 Some useful time series Models: .................................................................................................... 108 Moving average (MA) process .................................................................................................... 108 Autocorrelation function, acf of 𝑀𝐴(𝑞) process ........................................................................ 111 Simulating MA process in R......................................................................................................... 111 Useful Operations in Time Series .................................................................................................... 112 Backshift Operator (Lag operator) .............................................................................................. 112 Differencing Operator: ................................................................................................................ 112 Seasonal Differencing Operator .................................................................................................. 113 Inevitability of MA Processes .............................................................................................................. 114 Invertible solution: .......................................................................................................................... 115 Theorem: Invertible 𝑀𝐴(1) process ........................................................................................... 115 3

Invertability of general 𝑀𝐴(𝑞) process: ......................................................................................... 115 Theorem: invertibility of 𝑀𝐴(𝑞) process.................................................................................... 116 PACF of invertible MA process: ................................................................................................... 118 Autoregressive Processes and the Properties: ................................................................................... 119 Autoregressive (𝐴𝑅) Processes: ..................................................................................................... 120 Definition: Autoregressive process of order 𝑝 ............................................................................ 120 Analysis of an 𝐴𝑅(1) Process: .................................................................................................... 122 Analysis of AR(2) process (2nd order AR) ..................................................................................... 128 Autoregressive Processes of Order 𝑝 (AR(p)) ..................................................................................... 129 Theorems ........................................................................................................................................ 129 Theorem 1: .................................................................................................................................. 130 Theorem 2: .................................................................................................................................. 130 Yule Walker Equations for stationary AR processes ....................................................................... 130 Yule Walker Equation:................................................................................................................. 130 PACF of stationary AR(p) process: .............................................................................................. 136 Mixed autoregressive moving average (ARMA) process: ................................................................... 139 Notation: ......................................................................................................................................... 139 Note: ........................................................................................................................................... 139 Stationarity and invertibility of ARMA(p,q) Process ....................................................................... 139 Theorems .................................................................................................................................... 140 Special cases of ARMA(𝑝, 𝑞) ....................................................................................................... 140 Moments of Xt ∼ ARMA(p, q) ....................................................................................................... 142 Mean 𝐸𝑋𝑡 ................................................................................................................................... 142 Autocovariance function: 𝛾𝑘 ...................................................................................................... 143 Homogeneous Nonstationary Processes ........................................................................................ 145 Example data:.............................................................................................................................. 145 Modelling homogeneous nonstationary time series: ................................................................. 147 Autoregressive Integrated moving average ARIMA(p,d,q) ......................................................... 150 Identification and estimation:......................................................................................................... 151 ARMA/ARIMA Models:................................................................................................................ 151 Hypothesis testing of orders or ARMA(p,a) and estimation ............................................................... 156 Identification: .................................................................................................................................. 157 1: Test whether the series is a white noise of 𝐻0: 𝑋𝑡 ∼ 𝐴𝑅𝑀𝐴(0,0) ........................................ 157 2: Test 𝑋𝑡 ∼ 𝐴𝑅𝑀𝐴(0, 𝑞) or 𝑋𝑡 ∼ 𝑀𝐴(𝑞) ................................................................................. 158 3: Test 𝑋𝑡 ∼ 𝐴𝑅𝑀𝐴𝑝, 0; or 𝑋𝑡 ∼ 𝐴𝑅(𝑝) .................................................................................... 159 Parameter estimation of ARMA models: ........................................................................................ 161 4

1: MA(1) ...................................................................................................................................... 161 Estimation of Parameters continued: ......................................................................................... 162 Diagnostic Checking: ........................................................................................................................... 163 Residual analysis ............................................................................................................................. 163

5

STAT3911 STOCHASTIC PROCESSES AND TIME SERIES (ADV) COURSE NOTES STANDARD LECTURES Lecture 1.

Monday, 6 March 2017

Introduction: Lecturer: Ray Kawai week 1-7 (stochastic processes) Shelton Reiris week 8-13 (time series) Qiuing Wang (ADV) week 1-7 (Stochastic processes)

Assessments: Quizes -

Wednesday week 7 common (April 26) (Stochastic processes)

-

Friday advanced week 7 (April 28) (Stochastic processes advanced material)

-

Friday week 12 (June 2) (Time series)

Computer exam: week 13

Schedule: 1. Review of probability theory. Elements of stochastic processes and time series. 2. Markov chains. 3.

Markov chains.

4.

Markov chains. 6

5.

The Poisson process.

6.

The Poisson process.

7.

The Poisson process.

8.

Time series data, components of a time series. Filtering to remove trends and seasonal components.

9.

Stationarity time series. Sample autocorrelations and partial autocorrelations. Probability models for stationary time series. Moving Average (MA) models and properties.

10. Invertibility of MA models. Autoregressive (AR) models and their properties. Stationarity of AR models. Mixed Autoregressive Moving Average (ARMA) models and their properties. 11. Homogeneous nonstationary time series (HNTS). Simple models for HNTS. Autoregressive Integrated Moving Average (ARIMA) models and related results. Review of theoretical patterns of ACF and PACF for AR, MA and ARMA processes. Identification of possible AR, MA, ARMA and ARIMA models for a set of time series data. 12. Estimation and fitting ARIMA models via MM and MLE methods. Hypothesis testing, diagnostic checking and goodness-of-fit tests. AIC for ARIMA models. Forecating methods for ARIMA models. 13. Minimum mean square error (mmse) forecasting and its properties. Derivation of l-step ahead mmse forecast function. Forecast updates. Forecast errors, related results and applications.

Stochastic Processes: In probability theory and related fields, a stochastic or random process is a mathematical object usually defined as a collection of random variables. Historically, the random variables were associated with or indexed by a set of numbers, usually viewed as points in time, giving the interpretation of a stochastic process representing numerical values of some system randomly changing over time, such as the growth of a bacterial population, an electrical current fluctuating due to thermal noise, or the movement of a gas molecule.[1][4][5] Stochastic processes are widely used as mathematical models of systems and phenomena that appear to vary in a random manner. They have applications in many disciplines including sciences such as biology,[6] chemistry,[7] ecology,[8] neuroscience,[9] and physics[10] as well as technology and engineering fields such as image processing, signal processing,[11] information theory,[12] computer science,[13] cryptography[14] and telecommunications.[15] Furthermore,

7

seemingly random changes in financial markets have motivated the extensive use of stochastic processes in finance.[16][17][18]

Basic concenpts: -

Randomness, indexed by time {𝑋𝑡 : 𝑡 ∈ 𝕋}

Where 𝑡 ∈ [0, 𝑇] for some endtime 𝑡. -

Can be either discrete or continuous State

Time

Continuous

Height, temperature

Continuous time

discrete

Dice, coinflip, number of

1 day, 1 second, 1 year

people

Definition: Stochastic process A stochastic process is a model of time-dependent random phenomena. A single random variable describes a static random phenomena; a stochastic process is a collection of random variables {𝑋𝑡 : 𝑡 ∈ 𝕋}, one for each time 𝑡 ∈ 𝕋. -

Can be either discrete or continuous

Definition: Random walk We have a time set 𝕋 and state space 𝑆. We defininte the stochastic process {𝑋𝑡 : 𝑡 ∈ 𝕋}. In principle, we need to know the joint distribution of (𝑋𝑡1 , 𝑋𝑡2 , … , 𝑋𝑡𝑛 ), where 𝑡𝑘 ∈ 𝕋, 𝑛 ∈ ℕ. This is difficult. We define a random walk as a stochastic process {𝑋𝑡 : 𝑡 ∈ 𝕋}, where 𝕋 = ℕ in such a way that 𝑋𝑡 = 𝑋𝑡−1 + 𝑍𝑡 ; (𝑡 ∈ 𝕋) Where {𝑍𝑘 }𝑘∈ℕ is a sequence of iid RV with ℙ(𝑍𝑘 = 1) = 𝑝, ℙ(𝑍𝑘 = −1) = 1 − 𝑝 for some 𝑝 ∈ (0,1). The equation 𝑋𝑡 = 𝑋𝑡−1 + 𝑍𝑡 is an example of a difference equation. It is an implicit definition of 𝑋𝑡 , since it is only given in terms of 𝑋𝑡−1 . In continuous time, this becomes the differential equation

8

Definition: Strong stationary A SP {𝑋𝑡 : 𝑡 ∈ 𝕋} is said to be strong stationary, if the two joint distributions of (𝑋𝑡1 , 𝑋𝑡2 , … , 𝑋𝑡𝑛 ) and (𝑋𝑘+𝑡1 , 𝑋𝑘+𝑡2 , … , 𝑋𝑘+𝑡𝑛 ) are identical ∀𝑡1 , … , 𝑡𝑛 ; 𝑘 + 𝑡1 , … , 𝑘 + 𝑡𝑛 ∈ 𝕋

Definition: State space of stochastic process The set of values that the 𝑋𝑡 ‘s can take is called the state space, 𝑆, of the stochastic process

Definition: Increments An increment of a stochastic process is the amount by which its value changes over a period of time, for example 𝑋𝑡𝑘+1 − 𝑋𝑡𝑘 , where 𝑡𝑘 < 𝑡𝑘+1 ∈ 𝕋 Definition: Stationary increments The SP {𝑋𝑡 : 𝑡 ∈ 𝕋} is said to have stationary increments if the distribution of the increment depends only on the difference between the two time points. -

If, for 𝑡1 ≤ 𝑡2 and 𝑡3 ≤ 𝑡4 ⟹ 𝑡2 − 𝑡1 = 𝑡4 − 𝑡3 𝑋𝑡2 − 𝑋𝑡1

ℒ 𝑋 − 𝑋𝑡3 = 𝑡4

(the increments have the same distribution) Eg: temperature between today and tomorrow is distributed the same

Example: stock prices Let {𝑆𝑡 : 𝑡 ∈ ℝ+ } denote the price of one share of a specific stock. It might be considered reasonable to assume that the distribution of the return over a period of duraction Δ > 0 𝑆𝑡+Δ − 𝑆𝑡 𝑆𝑡+Δ = −1 𝑆𝑡 𝑆𝑡 9

𝑆𝑡+Δ :𝑡 𝑆𝑡

Depends on Δ but not 𝑡. Generally, we assume that {

∈ ℝ+ } is a stationary stochastic process.

Accordingly, the log-price processes 𝑋 ≔ ln 𝑆𝑡 would have stationary increments 𝑋𝑡+Δ − 𝑋𝑡 = 𝑆𝑡+Δ ), 𝑆𝑡

ln (

even though the stochastic process {𝑋𝑡 : 𝑡 ∈ ℝ+ } might not be stochastic. In other words,

for fixed Δ, the stochastic process 𝑌𝑡Δ ≔ 𝑋𝑡+Δ − 𝑋𝑡 is stationary. Definition: Independent increments A stochastic process {𝑋𝑡 : 𝑡 ∈ 𝕋} has independent increments, if ∀𝑡 ∈ 𝕋, and Δ > 0|𝑡 + Δ ∈ 𝕋, the increment 𝑋𝑡+Δ − 𝑋𝑡 is independent of all the past {𝑋𝑠 : 𝑠 ∈ 𝕋} of the SP. -

The increments at some time period are independent of previous events.

o

The first half of this course will assume stationary and independent increments

Markov Property: In probability theory and related fields, a Markov process (or Markoff process), named after the Russian mathematician Andrey Markov, is a stochastic process that satisfies the Markov property[1][2] (sometimes characterized as "memorylessness"). Loosely speaking, a process satisfies the Markov property if one can make predictions for the future of the process based solely on its present state just as well as one could knowing the process's full history; i.e., conditional on the present state of the system, its future and past states are independent. A Markov chain is a type of Markov process that has either discrete state space or discrete index set (often representing time), but the precise definition of a Markov chain varies.[3] For example, it is common to define a Markov chain as a Markov process in either discrete or continuous time with a countable state space (thus regardless of the nature of time)[4][5][6][7], but it is also common to define a Markov chain as having discrete time in either countable or continuous state space (thus regardless of the state space).[8]

-

Future events only depend upon current time, and not previous events

Definition: Markov property A SP {𝑋𝑡 : 𝑡 ∈ 𝕋} is said to have a Markov Property if 10

𝑃(𝑋𝑡𝑚+1 ∈ 𝐴𝑚+1 |𝑋𝑡1 ∈ 𝐴1 , … , 𝑋𝑡𝑚 ∈ 𝐴𝑚 ) = 𝑃(𝑋𝑡𝑚+1 ∈ 𝐴𝑚+1 |𝑋𝑡𝑚 ∈ 𝐴𝑚 ) Where 0 ≤ 𝑡1 ≤ 𝑡2 ≤ ⋯ ≤ 𝑡𝑚 ≤ 𝑡𝑚+1 and {𝐴𝑘 }(𝑘∈ℕ} is a sequence of measurable sets in 𝑆

Corollary: Independent increments have the Markov property A SP {𝑋𝑡 : 𝑡 ∈ 𝕋} with independent increments has the Markov property. Example: State space of natural numbers Eg, let 𝑆 = ℕ, then 𝑃(𝑋𝑡𝑚+1 = 𝑥𝑚+1 |𝑋𝑡1 ∈ 𝐴1 , … , 𝑋𝑡𝑚−1 ∈ 𝐴𝑚−1 , 𝑋𝑡𝑚 = 𝑥𝑚 ) = 𝑃(𝑋𝑡𝑚+1 − 𝑋𝑡𝑚 = 𝑥𝑚+1 − 𝑥𝑚 |𝑋𝑡1 ∈ 𝐴1 , … , 𝑋𝑡𝑚−1 ∈ 𝐴𝑚−1 , 𝑋𝑡𝑚 = 𝑥𝑚 ) = 𝑃(𝑋𝑡𝑚+1 − 𝑋𝑡𝑚 = 𝑥𝑚+1 − 𝑥𝑚 |𝑋𝑡𝑚 = 𝑥𝑚 ) (𝑏𝑦 𝑖𝑛𝑑𝑒𝑝𝑒𝑛𝑑𝑒𝑛𝑡 𝑖𝑛𝑐𝑟𝑒𝑚𝑒𝑛𝑡𝑠) = 𝑃(𝑋𝑡𝑚+1 = 𝑥𝑚+1 |𝑋𝑡𝑚 = 𝑥𝑚 ) Note, independent increments imply markov property, but not the reverse (as what if 𝑋𝑛+1 = 𝑋𝑛 + 𝜖𝑛 , where 𝜖_𝑛|𝑥0 , . . , 𝑥𝑛 ∼ 𝑁(−𝑋𝑛 , 1) for example)?. This brings in the concept of filtration, where we need to model the flow of public information. Lecture 2.

Tuesday, 7 March 2017

Definition: filtration Let (Ω, ℱ) be a measureable space, and 𝕋 ⊆ [0, ∞). 1. Assume ∀𝑡 ∈ 𝕋 ∃ 𝜎 −field, ℱ𝑡 ⊆ ℱ. Assume that, for 𝑠 ≤ 𝑡 ⟹ ℱ𝑠 ⊆ ℱ𝑡 . We call the collection of 𝜎 fields (𝐹𝑡 )𝑡∈𝕋 a filtration. 2. A SP {𝑋𝑡 : 𝑡 ∈ 𝕋} is said to be (ℱ)𝑡∈𝕋 adapted if, ∀𝑡 ∈ 𝕋, the RV 𝑋𝑡 is ℱ𝑡 − measurable.

Remark: filtration generated by a stochastic process and information If the filtration (ℱ𝑡 )𝑡∈𝕋 is generated by a stochastic process {𝑋𝑡 : 𝑡 ∈ 𝕋}, then ∀𝑡 ∈ 𝕋 ℱ𝑡 = 𝜎(𝑋𝑠 : 𝑠 ∈ 𝕋, 𝑠 ≤ 𝑡) In this case, the 𝜎 −field ℱ𝑠 contains all the information of the SP up till time 𝑠. -

The concept of filtration can then generalise the definition of the Markov property

11

Definition: Markov Process Let (Ω, ℱ, ℙ) be a probability space, and (ℱ𝑡 )𝑡∈𝕋 be a filtration. A (ℱ𝑡 )𝑡∈𝕋 adapted SP {𝑋𝑡 : 𝑡 ∈ 𝕋} is called a Markov process, if, ∀𝐵 ∈ 𝜎(𝑋𝑠 : 𝑠 ≥ 𝑡) ℙ(𝐵|ℱ𝑡 ) = ℙ(𝐵|𝑋𝑡 ) (note that 𝐵 depends only on {𝑋𝑠 : 𝑠 ≥ 𝑡}).

Lecture 3.

Wednesday, 8 March 2017

Markov Chains In probability theory and related fields, a Markov process (or Markoff process), named after the Russian mathematician Andrey Markov, is a stochastic process that satisfies the Markov property[1][2] (sometimes characterized as "memorylessness"). Loosely speaking, a process satisfies the Markov property if one can make predictions for the future of the process based solely on its present state just as well as one could knowing the process's full history; i.e., conditional on the present state of the system, its future and past states are independent. A Markov chain is a type of Markov process that has either discrete state space or discrete index set (often representing time), but the precise definition of a Markov chain varies.[3] For example, it is common to define a Markov chain as a Markov process in either discrete or continuous time with a countable state space (thus regardless of the nature of time)[4][5][6][7], but it is also common to define a Markov chain as having discrete time in either countable or continuous state space (thus regardless of the state space).[8]

-

Discrete time and discrete space problem o

-

After we will look at continuous time and discrete space (Poisson processes)

That is to say, they are indexed by 𝑡 ∈ 𝕋 = ℕ = {0,1,2, … }

Basic concepts: Independence of increments are replaced by the Markov property -

That the future and the past are independent given the present

Notation: Given 𝑛, 𝑟, 𝑘0 , … , 𝑘𝑛+𝑟 , with 𝑃(𝑋𝑛 = 𝑘𝑛 ) > 0 : -

we write the past event as 12

{𝑋0 = 𝑘0 , 𝑋1 = 𝑘1 , … , 𝑋𝑛−1 = 𝑘𝑛−1 } ≔ 𝐴 -

And all future events as {𝑋𝑛+1 = 𝑘𝑛+1 , 𝑋𝑛+2 = 𝑘𝑛+2 , … , 𝑋𝑛+𝑟 = 𝑘𝑛+1 } ≔ 𝐵

We then get the conditional probability that:

-

𝑃(𝐴 ∩ 𝐵|𝑋𝑛 = 𝑘𝑛 ) = 𝑃(𝐴|𝑋𝑛 = 𝑘𝑛 )𝑃(𝐵|𝑋𝑛 = 𝑘𝑛 ) This is equivalent that ∀𝑛 ∈ ℕ, 𝑘0 , 𝑘1 , … , 𝑘𝑛+1 with 𝑃(𝑋0 = 𝑘0 , 𝑋1 = 𝑘1 , … , 𝑋𝑛−1 = 𝑘𝑛−1 ) > 0 that: 𝑃(𝑋𝑛+1 = 𝑘𝑛+1 |𝑋0 = 𝑘0 , … , 𝑋𝑛 = 𝑘𝑛 ) = 𝑃(𝑋𝑛+1 = 𝑘𝑛+1 |𝑋𝑛 = 𝑘𝑛 )

(by the Markov property) Transiontion probability The right hand side of the above equation is known as the transition probability. It does not depend on time, but only on the states 𝑘𝑛 and 𝑘𝑛+1 . We write that 𝑃(𝑋𝑛+1 = 𝑗 |𝑋𝑛 = 𝑘) ≔ 𝑝𝑘,𝑗

Gambler ruin problem: -

Start with an amount of money $𝑛, with probability 𝑝 you win $1, and 𝑞 = 1 − 𝑝 you lose $1. You stop after you either:

-

o

Lose all money

o

Win up to a certain amount $𝑁

What is the probability that we will succeed with $𝑁 without losing everything?

Solution -

His fortune is a MC on {0,1, … , 𝑁} ∈ ℕ.

13

$ (discrete) 𝑁

𝑝

𝑛+1

𝑛 1−𝑝

𝑛−1

Time (discrete)

0

Definite: 𝐴𝑛 ={eventual success starting from $𝑛}; and 𝑎𝑛 = 𝑃(𝐴𝑛 ) [the eventual success when starting from initial state 𝑛]. -

We have that 𝑎0 = 0, 𝑎𝑁 = 1 𝑃(𝐴𝑛 ) = 𝑃(𝐴𝑛 |{𝑤𝑖𝑛 𝑎𝑡 𝑛})𝑃({𝑤𝑖𝑛 𝑎𝑡 𝑛}) + 𝑃(𝐴𝑛 |{𝑙𝑜𝑠𝑒 𝑎𝑡 𝑛})𝑃({𝑙𝑜𝑠𝑒 𝑎𝑡 𝑛}) = 𝑃(𝐴𝑛 |{𝑤𝑖𝑛 𝑎𝑡 𝑛})𝑝 + 𝑃(𝐴𝑛 |𝑙𝑜𝑠𝑒 𝑎𝑡 𝑛}) (1 − 𝑝) = 𝑃(𝐴𝑛+1 |{𝑤𝑖𝑛 𝑎𝑡 𝑛})𝑝 + 𝑃(𝐴𝑛−1 |{𝑙𝑜𝑠𝑡 𝑎𝑡 𝑛})(1 − 𝑝)

(as, if we win at 𝑛, we go to game 𝑛 + 1, and if we lose at 𝑛 we go to game 𝑛 − 1) = 𝑃(𝐴𝑛+1 )𝑝 + 𝑃(𝐴𝑛−1 )(1 − 𝑝) (using the markov property) ∴ 𝑎𝑛 = 𝑎𝑛+1 𝑝 + 𝑎𝑛−1 𝑞 Which is difference equation, with 𝑃(𝐴0 ) = 𝑎0 = 0; 𝑃(𝐴𝑛 ) = 𝑎𝑛 = 1 The solution to this is: 1−𝑝 𝑛 1−( 𝑝 ) 𝑁

; 𝑖𝑓 𝑝 ≠

𝑎𝑛 = 1 − (1 − 𝑝) 𝑝 𝑛 1 ; 𝑖𝑓 𝑝 = { 𝑁 2

1 2

14

-

𝑛

Note that this expression is continuous in 𝑝; and that the 𝑁 expression can be derived based 𝑞 𝑛

1

1

on the one for 𝑝 ≠ 2 using the asymptotic behavious 1 − (𝑝) = 4𝑛 (𝑝 − 2) + 1 2

1

𝑂 ((𝑝 − 2) ) as 𝑝 → 2.

We get the same solution if we let 𝛽𝑛 be the probability of eventual ruin when starting at 𝑛, finding that 𝛽𝑛 = 𝑝𝛽𝑛+1 + 𝑞𝛽𝑛−1 , with 𝛽0 = 1 and 𝛽𝑁 = 0. This means that the gambler must either succed or be ruined, and the gambler will not be able to converge to a steady state of some other amount of money.

Also; observe that in the limit: 𝑞 𝑛 1 1 − ( ) [> 0]; 𝑖𝑓 𝑝 > 𝑝 2 lim 𝑎𝑛 = 𝑁↑+∞ 1 0; 𝑝 ≤ { 2 So, taking 𝑁 ↑ +∞ means that in the limit, the gambler will only ever stop if ruined. In this situation, 1

if each gamble is in the player’s favour (𝑝 > 2), then there is a positive probability that the gambler 1 2

will never get ruined, but instead become infinitely rich. If each gamble is out of favour, (𝑝 < ), then the gambler will get ruined almost surely.

Classification of states: 𝑛 step transition probability notation: Let (𝑛)

𝑝𝑘0 ,𝑘𝑛 ≔ ℙ𝑘0 (𝑋𝑛 = 𝑘𝑛 ) = ℙ(𝑋𝑛 = 𝑘𝑛 |𝑋0 = 𝑘0 ) -

The probability that we are at state 𝑘𝑛 , given that we started at 𝑘0 𝑛 steps ago.

-

For ease of notation, 𝑝𝑖,𝑗 = 𝑝𝑖,𝑗

(1)

Additional 𝑚 steps - Where are we after an additional amount of steps?

15

𝑗

𝑘

𝑖

𝑛

(𝑛+𝑚)

𝑝𝑖,𝑗

𝑛+𝑚

= 𝑃(𝑋𝑛+𝑚 = 𝑗|𝑋0 = 𝑖)

= ∑ 𝑃(𝑋𝑛+𝑚 = 𝑗, 𝑋𝑛 = 𝑘 |𝑋0 = 𝑖) [𝑖. 𝑒. −𝑔𝑜𝑖𝑛𝑔 𝑡ℎ𝑟𝑜𝑢𝑔ℎ 𝑠𝑡𝑎𝑡𝑒 𝑘] = ∑ 𝑃(𝑋𝑛+𝑚 = 𝑗|𝑋𝑛 = 𝑘, 𝑋𝑜 = 𝑖)𝑃(𝑋𝑛 = 𝑘|𝑋0 = 𝑖) [𝑐𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛𝑎𝑙 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦] 𝑘

= ∑ 𝑃(𝑋𝑛+𝑚 |𝑋𝑛 = 𝑘)𝑃(𝑋𝑛 = 𝑘|𝑋𝑜 = 𝑖) [𝑏𝑦 𝑡ℎ𝑒 𝑀𝑎𝑟𝑘𝑜𝑣 𝑝𝑟𝑜𝑝𝑒𝑟𝑡𝑦] 𝑘 (𝑚) (𝑛) = ∑ 𝑝𝑘,𝑗 𝑝𝑖,𝑘 [𝑢𝑠𝑖𝑛𝑔 𝑛𝑜𝑡𝑎𝑡𝑖𝑜𝑛] 𝑘 (𝑛+𝑚) 𝑝𝑖,𝑗

(𝑛) (𝑚)

= ∑ 𝑝𝑘,𝑗 𝑝𝑖,𝑘 𝑘

-

Which looks like the definition of matrix multiplication‼

𝑛 step transitional probability matrix (𝑛+𝑚)

(𝑛) (𝑚)

The above equation: 𝑝𝑖,𝑗 = ∑𝑘 𝑝𝑘,𝑗 𝑝𝑖,𝑘 is called the Chapman-Kolmogorov equation, and can be written as the matrix equation: 𝑃(𝑛+𝑚) = 𝑃(𝑛) 𝑃(𝑚)

-

From which we see that 𝑛

𝑃(𝑛) = (𝑃(1) )

In the matrix:

16

𝐷𝑒𝑠𝑖𝑛𝑎𝑡𝑖𝑜𝑛 𝑜𝑟𝑖𝑔𝑖𝑛 (

)

Accessibility: (𝑛)

We say that state 𝑗 is accessible from state 𝑖, ∃ some number of steps 𝑛 ∈ ℕ+ |𝑝𝑖,𝑗 > 0 -

i.e.: it is possible to get to state 𝑗 from state 𝑖 after some amount of states (as probability > 0) can write as 𝑖 → 𝑗

Communicate If 𝑖 and 𝑗 are accessible from one another, they are said to communicate, written as 𝑖 ↔ 𝑗 Properties of communicating states: 1. reflexivity: 𝑖 ↔ 𝑖 2. symmetry: 𝑖 ↔ 𝑗 ⟺ 𝑗 ↔ 𝑖 3. transitivity: if 𝑖 ↔ 𝑗, and 𝑗 ↔ 𝑘; then 𝑖 ↔ 𝑘 Communicating classes: Communicating states can be partitioned into communicating classes: -

largest set A of states such that all 𝑖, 𝑗 ∈ 𝐴 communicate. all states in a communicating class communicate with one another o in the gamblers ruin problem; there are 3 communicating classes:  {𝑁}  {0}  {1,2, … , 𝑁 − 1}

Closed communicating classes We say a communicating class 𝐶 is closed if no state outside of 𝐶 can be reached from any state in 𝐶. i.e: 𝑝𝑖,𝑗 = 0 for 𝑖 ∈ 𝐶 and 𝑗 ∉ 𝐶 Irreducible: If the markov chain consists of only 1 communicating class, then the MC is said to be irreducible. Absorbing state A state is said to be absorbing, if we cannot go anywhere after it. -

i.e; the set {𝑗} is a closed class, that is 𝑝𝑗,𝑗 = 1

o

Lecture 4.

in the gambler’s guin problem: 𝑝0,0 = 1 and 𝑝𝑁,𝑁 = 1

Monday, 13 March 2017

17

Example: Markov Chain Consider the MC with states {1,2,3} 1 2 1 2

1 2 1 𝑃= 4 1 0 ( 3

0 1 4 2 3)

Eg: (probability of 1 → 3 is 0) Accessible states:1,1; 1,2; 2,1; 2,2; 2,3; 3,2; 3,3 1↔2↔3 {1,2,3} is a communicating class Example 2: markov chain 4 states with 1 2 1 𝑃= 2 1 4 (0

1 2 1 2 1 4 0

0

0

0

0

1 4 0

1 4 1)

4 is absorbing sate 1 and 2 communicate 1,2,3,4 is accessible from 3. -

This is not irreducible (3 communicating classes)

Example 3: weather Probability of fine or rain (0,1). Chances of rain tomorrow depends on today’s conditions. If it is fine today, probability of fine tomorrow is 0.7. if it is rainy today, probability of fine tomorrow is 0.4 0.7 0.3 𝑃=( ) . 4 0.6 -

This is irreducible

3 step transition: 0.7 0.3 3 0.583 . 417 𝑃3 = ( ) =( ) . 4 0.6 . 556 . 441 (3)

Ie: 𝑝0,1 = 0.417

18

Recurrence and transience (𝑛)

Let 𝑓𝑖,𝑗 denote the probability that the first transition into 𝑗 takes place at time 𝑛, when the chain starts at state 𝑖 (𝑛)

𝑓𝑖,𝑗 ≔ ℙ({𝑋𝑛 = 𝑗} ∩ {𝑋𝑛−1 , 𝑋𝑁−2 , … , 𝑋1 ≠ 𝑗}) -

i.e. the “first” probability from 𝑖 → 𝑗 after 𝑛 steps, without having vistited 𝑗 in between.

-

If 𝑓𝑖,𝑗 = 0 if 𝑖 ≠ 𝑗 and 𝑓𝑖,𝑗 = 1 if 𝑖 = 𝑗. Then the quantity becomes:

(0)

(0)

+∞ (𝑛)

𝑓𝑖,𝑗 ≔ ∑ 𝑓𝑖,𝑗 = ℙ(𝑋𝑛 = 𝑗 for some 𝑛 ≥ 1|𝑋0 = 1) 𝑛=1

Recurrence definition: Indicates the probability of ever making a transition into state 𝑗 when the chain starts at 𝑖. We call state 𝑗 recurrent if 𝑓𝑗,𝑗 = 1 (i.e.: starting at 𝑗, the chain will almost surely return to itself in a finite number of steps).

Transient definition: A non recurrent state is said to be transient. -

For example, if 𝑓𝑗,𝑗 < 1, then state 𝑗 is transient Not sure that we will return

Green Function: The Green function of the MC is the expected number of visits to 𝑗 for the chain starting at 𝑖. +∞

𝐺(𝑖, 𝑗) ≔ +∞

(𝑛) ∑ 𝑝𝑖,𝑗 𝑛=0

+∞

= ∑ 𝔼(𝕀(𝑋𝑛 = 𝑗)|𝑋0 = 𝑖) 𝑛=0

= 𝔼 ∑(𝕀(𝑋𝑛 = 𝑗)|𝑋0 = 𝑖) 𝑛=0

-

Is the expected number of visits to 𝑗 starting from state 𝑖.

Transience and green function: State 𝑗 is transient iff 𝐺(𝑖, 𝑗) < ∞ (i.e. the expected number of moves before returning is finite) 𝑡𝑟𝑎𝑛𝑠𝑖𝑒𝑛𝑐𝑒 ⟺ 𝐺(𝑗, 𝑗) < ∞ Proof: time j

j

19

After visiting 𝑗, the probability of returning is 𝑓𝑗,𝑗 after some time. This happens every time we return to state 𝑗. As we are counting the number of successes of visits to 𝑗, this means the distribution is geometric; with 𝑝 = 𝑓𝑗,𝑗 . ∼ 𝐺𝑒𝑜𝑚𝑒𝑡𝑟𝑖𝑐(𝑓𝑗,𝑗 ) 𝑝

Mean of geometric is 1−𝑝 𝐸[𝑋] =

𝑓𝑗,𝑗 1 − 𝑓𝑗,𝑗

But we already know that 𝐺(𝑗, 𝑗) = 𝐸(𝑋), so 𝑓𝑘,𝑘 < 1 iff 𝐺(𝑗, 𝑗) < ∞

Lecture 5.

Tuesday, 14 March 2017

Recurrence of communicating states If 𝑖 ↔ 𝑗 and if state 𝑖 is recurrent, then state 𝑗 is recurrent.

Proof: 𝑚 𝑛 As 𝑖 ↔ 𝑗∃𝑚, 𝑛|𝑝𝑖,𝑗 , 𝑝𝑖,𝑗 > 0. We fix such 𝑚, 𝑛, it then holds that ∀𝑠 ≥ 0 𝑛+𝑠+𝑚 𝑛 𝑝𝑗,𝑗 ≥ 𝑝𝑗,𝑖 𝑝𝑖 , 𝑖 𝑠 𝑝𝑖 , 𝑗 𝑚

Due to the chapman kolmorgov equation when: 𝑛+𝑠+𝑚 𝑛 𝑠 𝑚 𝑛 𝑚 𝑠 𝑛 𝑚 ∑ 𝑝𝑗,𝑗 ≥ ∑ 𝑝𝑗,𝑖 𝑝𝑖,𝑖 𝑝𝑖,𝑗 = 𝑝𝑗,𝑖 𝑝𝑖,𝑗 ∑ 𝑝𝑖,𝑖 = 𝑝𝑗,𝑖 𝑝𝑖,𝑗 𝐺(𝑖, 𝑖) = ∞ 𝑠

𝑠

𝑠

Recurrence and accessibility If state 𝑖 is recurrent and state 𝑗 is accessible from 𝑖, then 𝑓𝑖,𝑗 = 1 and 𝑖 ↔ 𝑗 Proof: (𝑛)

Let 𝑋0 = 𝑖. As 𝑗 is accessible from 𝑖∃𝑛|𝑝𝑖,

> 0. Fix such 𝑛, we define the following:

𝐴0 = {𝑋𝑛 = 𝑗}; 𝑇1 = min{𝑘 ≥ 𝑛: 𝑋𝑘 = 𝑖} 𝐴1 = {𝑋𝑇1 +𝑛 = 𝑗}; 𝑇2 = min{𝑘 ≥ 𝑇1 + 𝑛: 𝑋𝑘 = 𝑖} … 𝐴𝑟 = {𝑋𝑇𝑟 +𝑛 = 𝑗}; 𝑇𝑟+1 = min{𝑘 ≥ 𝑇𝑟 + 𝑛: 𝑋𝑘 = 𝑖}

𝑛 Since 𝑖 is recurrent; 𝑇𝑘 are finite. Then, the MC {𝐴𝑛 } are independent and have th 𝑝𝑖,𝑗 so one of them occurs.

20

Remark: - We have used the Strong Markov Property to show this. Which always holds for discrete time markov chains, and usually for continuous ones. It says that: if 𝑁 is the stopping time of a MC {𝑋𝑛 }𝑛∈ℕ+ and we write 𝜓(𝑖, 𝐵) = 𝑃𝑖 ((𝑋0 , 𝑋1 , … ) ∈ 𝐵)

The Strong markov property says that: 𝑃((𝑋𝑁 , 𝑋𝑁+1 , … ) ∈ 𝐵|𝑋0 , 𝑋1 , … , 𝑋𝑁 ) = 𝜓(𝑋𝑁 , 𝐵) Example: Consider the 4 staes with probability: 1 2 𝑃= 1 0 0 0 1 0 (0 0 0 0 0

1 2 0 0 0)

It is easy to see that all states communicate: 1 → 3 → 2 → 1 → 4 → 2 → 1 -

All states must be recurrent (using the above property)

Example 2: 1 2 1 2 𝑃=

0 0 1 (4

1 0 2 1 0 2 1 0 2 1 0 2 1 0 4

0 0 0 0 1 0 2 1 0 2 1 0 2)

The chain consists of 3 classes: {1,2}, {3,4}, {5}.

The first two classes are recurrent, if the chin starts at 1 it will come back to 1 sometime a.s. however; if it starts at 5, it may never come back, as if it goes to state {1,2} it can never return.

21

Period: Definition The period of state 𝑘 is the greatest common divisor of the number of steps to come back to 𝑖, (𝑛)

starting from 𝑖 {𝑛 ∈ ℕ: 𝑝𝑘,𝑘 > 0}, often written as 𝑑(𝑘) If 𝑑(𝑘) = 1 then the state is called aperiodic.

Lecture 6.

Wednesday, 15 March 2017

Remark on periodicity: The minimum number of steps required to return is purely irrelevant to the concept of periodicity, eg: consider

1 0 2 𝑃=( 0 0 1 0

0 1 0

)

Observe that: starting from state 2; the chain returns to state 2 after 3,5,6 steps. The minimum number of steps required to return to state 2 startin at 2 is 3. Nevertheless, the greatest common divisor is 1, so the periodicity is 1. Communicating states and periodicity If 𝑖 ↔ 𝑗 then 𝑑(𝑖) = 𝑑(𝑗) Proof: If 𝑖 = 𝑗, result is trivial. Suppose 𝑖 ≠ 𝑗. We show that𝑖 ↔ 𝑗 means 𝑑(𝑖) divides 𝑑(𝑗). Find a positive 𝑠 interer 𝑠, such that 𝑝𝑖,𝑖 > 0. By definition, this integer divides 𝑑(𝑖). Moreover, there exists positive 𝑛 𝑛 interges 𝑚, 𝑛|𝑝𝑖,𝑗 , 𝑝𝑗,𝑖 > 0. Then we have that: 𝑛+𝑚 𝑛 𝑚 𝑝𝑗,𝑗 ≥ 𝑝𝑗,𝑖 𝑝𝑖,𝑗 > 0

So (𝑛 + 𝑚) divides 𝑑(𝑗). By visiting 𝑖 in the middle, we have: 𝑛+𝑠+𝑚 𝑛 𝑠 𝑚 𝑝𝑗,𝑗 ≥ 𝑝𝑗,𝑖 𝑝𝑖,𝑖 𝑝𝑖,ℎ > 0

So (𝑛 + 𝑠 + 𝑚) divides 𝑑(𝑗). ∴ 𝑠 divides 𝑑(𝑗) and 𝑑(𝑖). Meaning that 𝑑(𝑖) = 𝑑(𝑗) as they are the greatest common divisor

Example: Random walk: Consider the MC on ℤ such that, for a given 𝑝 ∈ [0,1]; 𝑘 ∈ ℤ 𝑝𝑘,𝑘+1 = 𝑝; 𝑝𝑘,𝑘−1 = 1 − 𝑝

22

(example of a MC with period 2). 1

We show that, for 𝑝 = 2 it is recurrent. First, observe that the n step transition probability is binomial 𝑛 1 𝑛 (2𝑛) 2𝑛 1 𝑝0,0 = ( ) ( ) ( ) 𝑛 2 2

Which we approximiate (using Stirlings approximation), as (2𝑛)

𝑝0,0 ∼

1 √𝜋𝑛

Hence, 𝐺(0,0) = ∞. -

One of a famous theorems in probability extends to higher dimensions (Polya theorem), which says symmetric random walks on ℤ𝑘 are transient iff 𝑘 ≥ 3

-

Computing the probability of this for a 1D random walk will ever return if 𝑝 ≠ 2: WLOF, suppose we start at 0. Let 𝑌𝑘 be the 𝑘𝑡ℎ step of the walk, {𝑌𝑘 }𝑘∈ℕ is a sequence of iid RV with

1

+1; 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑝 −1; 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 1 − 𝑝 𝑋𝑛 denotes the position of the walk after 𝑛 steps, with 𝑋0 = 0. So 𝑋𝑛 = ∑𝑛𝑘−1 𝑌𝑘 . Considering the first transition, 𝑌𝑘 = {

𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0) = 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = 1)𝑃(𝑌1 = 1) + 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = −1)𝑃(𝑌1 = −1) = 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = 1)𝑝 + 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = −1)(1 − 𝑝) 1

If 𝑝 > 2, (walk tends in positive direction), observe by the law of large numbers that as 𝑛 ↑ +∞

𝑛

1 ∑ 𝑌𝑘 → 𝐸(𝑌1 ) = 2𝑝 − 1 (> 0)𝑎. 𝑠. 𝑛 𝑘=1

Which implies that ∑𝑛𝑘−1 𝑌𝑘 tends to infinity almost surely. If we investigate the transitional probability 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = +1) conditioning on the second transition, we get

= 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = +1, 𝑌2 = −1)𝑃(𝑌2 = −1) + 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = +1, 𝑌2 = +1)𝑃(𝑌2 = +1) = 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = 1, 𝑌1 = −1)(1 − 𝑝) + 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = 1, 𝑌2 = 1)𝑝 = 1 − 𝑝 + 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = +1, 𝑌2 = +1)𝑝 = 1 − 𝑝 + 𝑃(𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 0|𝑋0 = 0, 𝑋1 = 1, 𝑋2 = 2)𝑝 23

Which holds as the walk restrats after 2 steps. If the walk is at state 𝑋2 = 2, in order to ever return to state 0, we must first ever enter state 1. The probability that the walk ever enters state 1 starting from 2 is identical to 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛 𝑡𝑜 0|𝑋1 = 1). Similary, the probability that the walk ever enters state 0 from 1 is 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛 𝑡𝑜 0|𝑋1 = 1). And so: 𝑃(𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 0|𝑋2 = 2) = 𝑃(𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 0|𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 1 𝑠𝑡𝑎𝑟𝑡𝑖𝑛𝑔 𝑓𝑟𝑜𝑚 2)𝑃(𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 1|𝑠𝑡𝑎𝑟𝑡𝑖𝑛𝑔 𝑓𝑟𝑜𝑚 2) + 𝑃(𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 0|𝑛𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 1 𝑠𝑡𝑎𝑟𝑡𝑖𝑛𝑔 𝑓𝑟𝑜𝑚 2)𝑃(𝑛𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 1|𝑠𝑡𝑎𝑟𝑡𝑖𝑛𝑔 𝑓𝑟𝑜𝑚 2) = 𝑃(𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 0|𝑠𝑡𝑎𝑟𝑡 𝑓𝑟𝑜𝑚 1)𝑃(𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 1|𝑠𝑡𝑎𝑟𝑡 𝑓𝑟𝑜𝑚 2) = 𝑃(𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 0|𝑠𝑡𝑎𝑟𝑡 𝑓𝑟𝑜𝑚 1)2

We get that: 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = +1) = 1 − 𝑝 + 𝑝𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = +1)2 Givins us: 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = +1) = 1 𝑜𝑟

1−𝑝 𝑝

The probability =1 is impossible, as we know by transience that it is strictly less than 1. Giving us that 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = +1) = ∴ 𝑃(𝐸𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛) =

1−𝑝 𝑝

2 1−𝑝

1

For 𝑝 < 2, we get 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛_ = 2𝑝 Generally thr probability of returning in a Random Walk is: 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛) = 2 min{𝑝, 1 − 𝑝}

2 3

1 3

For example, if 𝑝𝑘,𝑘+1 = ; 𝑝𝑘,𝑘−1 = , starting at 0, the walk will come back to 0 again with 2 3

1 2

probability . (note the above result inclues 𝑝 = → 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛) = 1

Limiting theorems and Stationarity of Markov Chains Expected number of transitions from 𝑗 to 𝑗 Definition: Let 𝜇𝑗,𝑗 be the expected number of transitions required to return to state 𝑗 starting from 𝑗, for the first time; +∞ (𝑛)

𝜇𝑗,𝑗 ≔ ∑ 𝑛𝑓𝑗,𝑗 𝑛=0

24