Joint state and parameter estimation in particle filtering and stochastic optimization

In this paper, an adaptive estimation algorithm is proposed for non-linear dynamic systems with unknown static parameters based on combination of particle filtering and Simultaneous Perturbation Stochastic Approximation (SPSA) technique. The estimations of parameters are obtained by maximum-likelihood estimation and sampling within particle filtering framework, and the SPSA is used for stochastic optimization and to approximate the gradient of the cost function. The proposed algorithm achieves combined estimation of dynamic state and static parameters of nonlinear systems. Simulation result demonstrates the feasibility and efficiency of the proposed algorithm.


Introduction
Dynamic state-space models are useful for describing data in many different areas, such as engineering, finance mathematics, environmental data, and physical science. An important task when analyzing data by state-space models is estimation of the underlying state process based on measurements from the observation process. Bayesian filtering represents a solution of considerable importance for this type of problem definition as demonstrated by many existing algorithms based on the Kalman filter and particle filtering (PF) (Arulampalam 2002, Yang et al. 2006. The PF has been extensively studied in the situation where the unknown attributes are time-varying dynamic states. Although PF have been successful in many applications, a main problem with it is how to handle the presence of the unknown static parameters, especially in models with realistically large numbers of fixed parameters. The estimation of both the dynamic state and static parameters is commonly known in literatures as the dual estimation. Numerous papers have been written on the construction of estimation algorithms based on Markov chain Monte Carlo (MCMC) (Spall 2003). Although such methods may be effective for offline estimation, they are not suitable for online estimation because the MCMC algorithm needs to be restart at each time point. In engineering, a common trick to problem is to include the parameters as part of the state space vector. Berzuini et al. (Berzuini et al. 1997) put this approach into Bayesian estimation. However, the non-dynamics in the parameters cause the degeneracy of the algorithm. Jane and West (Jane & Mike 2001) introduced diversity in the particles by Kernel method, which is similar to replace the original static parameter with an alternative dynamic model. Polson et al (Polson & Stroud 2008) proposed a sequential parameter learning and filtering based on approximating the target posterior by a mixture of fixed lag smoothing distributions. Lee and Chia (Lee & Chia 2002) combined the particle filtering and MCMC to achieve an estimation algorithm in which the measurements are processed sequentially by particle filtering. When the degeneration occurs, the particles are rejuvenated by MCMC. Storvik (Storvic 2002) considered models with sufficient statistics for the parameters and applied particle filters to an augmented vector of states and sufficient statistics. Djuric et al (Djuric & Miguez 2002) proposed an alternative approach for a certain class of state-space model, which suppose that the marginal distribution of parameter can be analytically tractable. However, both algorithms suffer from an accumulation of error over times, albeit more slowly, leading to instability eventually. On the other hand, Andrieu et al (Andrieu et al. 2003, 2004, 2005, Yang et al. 2008) considered maximum likelihood parameter point estimation based on gradient. But in reality the graduate computation is intractable for complex nonlinear system function. In this chapter, we propose a new algorithm that preserves the static nature of the unknown parameters. The maximum likelihood parameter estimation is performed based on particle filtering and an effective stochastic approximation gradient algorithm is used to optimize cost function. The estimation of static parameters and dynamic state variables is performed simultaneously.

Problem formulation
The state-space models have the form . For nonlinear and non-Gaussian dynamic models, the particle filtering can achieve approximated estimation of PDF based on Monte Carlo simulation. Although particle filtering has been successful in many simulation experiments and in analysis of real data, a main problem with it is how to handle the presence of unknown static parameters. In this paper, we present a method referred to as point estimation, i.e. we do not aim to estimating the PDF of θ . We focus rather on the estimation of θ directly by maximum-likelihood (ML) principle.
The dynamic state is estimated by particle filtering and static parameter is estimated by recursive ML method online. Given a set of measurements 0:t y , the estimation of ML requires maximization of likelihood with respect to parameter θ . Firstly, the cost function is presented, and the likelihood of measurements 0:t y is given by In practice, one uses the log-likelihood which is numerically better behaved and satisfies www.intechopen.com To simplify the computation, the cost function is chosen as predicted likelihood, i.e.
However, except in a few simple cases, it is impossible to compute the optimal filter and the likelihood in closed-form, the numerical approximation schemes are required. The problem of maximizing the cost function can be translated into finding the zeros of the gradient () Under appropriate conditions, the iteration in (5) will converge to the true value of θ in some stochastic sense. The essential part of (5) is how to obtain the gradient estimate, however, it is impossible to compute the closed-form gradient and we must resort to the numerical approximation. The particle filtering (Gordon 1993, Yang et al. 2006) is based on importance sampling where t x is simulated sequentially from some importance distribution 1: () tt qx y , and the whole trajectory 1:t x is given importance weight xiN ω = at each time point t . The problem with the particle filtering is the degeneracy phenomenon, where the variance of the importance weights can only increase over time, making the estimate unstable (Kong et al. 1994). A common trick to avoid this is to re-sample from particle set with probabilities proportional to the importance weight (Gordon et al. 1993 where the particles are obtained using a one-step ahead state evolution prediction. Stochastic optimization techniques apply in the cases where a closed-form solution to the optimization problem of interest is not available and where the input information into optimization method may be contaminated with noise. One of the techniques that have attracted considerable recent attention for difficult multivariate problems is the simultaneous perturbation stochastic approximation (SPSA) method introduced by Spall (Spall 1987(Spall , 1998. SPSA is based on a highly efficient and easily implemented "simultaneous perturbation" approximation to the gradient. The SPSA technique requires all elements of θ to be varied randomly simultaneously to obtain two estimates of the cost function. Only two cost function measurements are required regardless of the dimension of the parameters be optimized. The SPSA has proven to be an effective and easy implemented algorithm and success among other finite difference methods with reduced number of estimates required for convergence (Chan et al. 2003, Andrieu et al. 2003. A step-by-step guide to implementation of SPSA for stochastic optimization is presented in (Spall 1998). It is assumed that () f θ is a differentiable function of θ and that the minimum point of θ corresponding to a zero point of the gradient. In SPSA, the gradient estimate In cases where the gradient has more than one zero point, then the algorithm may only converge to a local minimum, Spall further gives some modifications to the basic SPSA algorithm to allow it to search for the global solution among multiple local solutions[15].

Sampling algorithms for combined estimation of parameter and state
We present here how to incorporate maximum-likelihood algorithm within the particle filtering framework. To enhance the global convergence and Robust of the parameter estimate, for each state particle, i.e. a possible state trajectory, we produce a particle of www.intechopen.com parameter and resample correspondingly. The ultimate parameter estimate is produced by weighted sum of parameter particles. This process can alleviate the divergence of estimate of parameter. The algorithm proceeds as follows.
Step 1. Initialization: Step 2. State sampling: Given a set of weighted state and parameter particles Step 3. Cost function evaluation: Step 4. Gradient approximation: For each parameter particle, the corresponding gradient , www.intechopen.com Step 5. Parameter update: Step 6. Re-sampling: For each particle Re-assign even importance weights () 1 Step 7. Output: The obtained weighted particles

Simulation results
Here, we consider the following set of equations as an illustrative example which has been analyzed before in many publications (Gordon et al. 1993, Chan et al. 2003. We present two algorithms to deal with the unknown parameters. The first algorithm, titled "Augmented State", includes the parameters as part of the state vector (,) tt x θ which www.intechopen.com proposed in paper (Jane & Mike 2001). θ is replaced by t θ at time t , then add an independent, zero-mean normal increment increment to the parameter at each time. That is, For some specified variance matrix t W . We use 10 t W = in simulations. The second algorithm is our algorithm, titled "Adaptive estimate", which includes an on-line adaptive estimation of the parameters as proposed in this paper. We perform 50 independent Monte Carlo runs with 1000 N = particles in each run. The initial values of parameters are selected randomly in interval [0,1] . For reference, the true states for the exemplar run are shown in Fig.1 and the measurements in Fig.2. The sequences of parameter 1 θ and 2 θ estimate are illustrated in Fig.3 and Fig.4 respectively where the solid line with the label "adaptive estimation" indicates the estimate by our algorithm, the dashed line with the label "augmented state" indicates the estimate by the first algorithm. Fig.5 shows the RMSE of dynamic state t x by particle filtering where dashed line represents RMSE with true value of parameters, the solid line represent RMSE with augmented state estimates of parameters by the first algorithm. Fig.6 shows the RMSE of particle filtering where dashed line represents RMSE with true value of parameters, the solid line represent RMSE with adaptive estimates of parameters by our algorithm. From the simulation results, it can be seen that the parameters converge to true values quickly by the proposed algorithm and RMSE of dynamic state with adaptive estimates of parameters diminish with time and approach to RMSE with the true values of parameters.    We also compare the performance measure of our results with the "augmented state estimate" algorithm. The performance measure is root mean square error as follows: where (, ) xmt is the estimate of () xt in the mth Monte Carlo simulation, 50, 5000 MT = = . The performance of the first algorithm, our algorithm and the particle filtering with true parameter for various number of particles are presented in Table 1

Conclusions
In this chapter, we proposed an adaptive estimation algorithm for non-linear dynamic systems with unknown parameters based on combination of particle filtering and SPSA technique. We have demonstrated how to combine the maximum-likelihood parameter estimation with particle filtering. The estimates of parameters are obtained by state samples and maximum-likelihood estimation within particle filtering. The SPSA is used to approximate the gradient of cost function. The proposed algorithm achieves joint estimation of dynamic state and static parameters.