A Robust Design Criterion for Interpretable Fuzzy ... - Semantic Scholar
314
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 14, NO. 2, APRIL 2006
A Robust Design Criterion for Interpretable Fuzzy Models With Uncertain Data Mohit Kumar, Associate Member, IEEE, Regina Stoll, and Norbert Stoll
Abstract—We believe that nonlinear fuzzy filtering techniques may be turned out to give better robustness performance than the filtering techexisting linear methods of estimation ( 2 and niques), because of the fact that not only linear parameters (consequents), but also the nonlinear parameters (membership functions) attempt to identify the uncertain behavior of the unknown system. However, the fuzzy identification methods must be robust to data uncertainties and modeling errors to ensure that the fuzzy approximation of unknown system’s behavior is optimal in some sense. This study presents a deterministic approach to the robust design of fuzzy models in the presence of unknown but finite uncertainties in the identification data. We consider online identification of an interpretable fuzzy model, based on the robust solution of a regularized least-squares fuzzy parameters estimation problem. The aim is to resolve the difficulties associated with the robust fuzzy identification method due to lack of a priori knowledge about upper bounds on the data uncertainties. The study derives an optimal level of regularization that should be provided to ensure the robustness of fuzzy identification strategy by achieving an upper bound on the value of energy gain from data uncertainties and modeling errors to the estimation errors. A time-domain feedback analysis of the proposed identification approach is carried out with emphasis on stability, robustness, and steady-state issues. The simulation studies are provided to show the superiority of the proposed fuzzy estimation over the classical estimation methods. Index Terms— -optimality regularization, interpretability, least-squares, 2 -stability, min–max identification, normalized least mean squares algorithm (NLMS) algorithm.
I. INTRODUCTION
M
ANY real-world physical processes are generally characterized by the presence of nonlinearity, complexity and uncertainty. These processes cannot be represented by linear models used in conventional system identification [1]. The capability of fuzzy systems paradigm for not only learning complex input-output mappings but also to interpret these mappings with linguistic terms stimulates the study of approximating ill-defined and complex processes using a fuzzy inference system. Therefore, the fuzzy identification of nonlinear systems from input–output data has become an important topic of scientific research with a wide range of applications [2]–[9]. A
Manuscript received December 22, 2003; revised December 30, 2004, March 8, 2005, and May 2, 2005. This work was supported by European Space Agency under ESTEC Contract 14350/01/NL/SHMAP project AO-99-058. M. Kumar and R. Stoll are with the Institute of Occupational and Social Medicine, Faculty of Medicine, University of Rostock, D-18055 Rostock, Germany (e-mail: [email protected]; [email protected]). N. Stoll is with the Institute of Automation, College of Computer Science and Electrical Engineering, University of Rostock, D-18119 Rostock-Warnemünde, Germany (e-mail: [email protected]). Digital Object Identifier 10.1109/TFUZZ.2005.861614
large number of techniques have been developed for the fuzzy identification of nonlinear systems from measured input-output data. These techniques can be grouped into three approaches and their combinations: ad-hoc data covering approaches [4], [10]), neural networks ([6], [11]), and genetic algorithms [9], [12]–[14]. However, these methods do not consider the situations when the training data is uncertain. Regularization was suggested as a method for improving the robustness of fuzzy identification scheme in [7], [15], and [16]. However, the choice of regularization parameter is usually not obvious and application dependent. An iterative method for the robust (min–max) identification of fuzzy parameters with uncertain data, was suggested in [3], by solving an equivalent optimally regularized identification problem. However, the method is offline and requires a priori knowledge about upper bounds on the data uncertainties. At present, the literature lacks robustness, stability, and steady-state analysis of online fuzzy identification methods that don’t require the knowledge of upper bounds on uncertainties, in the deterministic or stochastic framework. We consider the fuzzy identification problem for a special class of fuzzy systems (Sugeno type fuzzy systems), since they ideally combine simplicity with good analytical properties [17]. Also, if we take into account the appropriate restrictions in terms of interpretability, the data-driven construction of Sugeno fuzzy systems allows qualitative insight into the relationships [18]–[21]. This paper starts with the mathematical formulation of Sugeno type fuzzy systems and considers the fuzzy parameters estimation with uncertain data, based on the robust solution of a regularized least-squares problem. By a robust solution we mean one that attempts to alleviate the worst-case effect of data uncertainties on fuzzy parameters estimation performance. However, the knowledge of an upper bound on the values of data uncertainties is needed to compute the robust solution of fuzzy parameter estimation problem. Therefore, we consider in particular, the derivation of a robust fuzzy identification method that does not require a priori knowledge about upper bounds on the data uncertainties, by providing a suitable choice of regularization parameters. The proposed method is shown to be robust and stable by providing a time-domain feedback analysis. The analysis highlights and exploits an intrinsic feedback structure, mapping the data uncertainties to the estimation errors. The optimal choice of regularization parameters is motivated by the robustness analysis of the mapping that can be associated with the proposed method of fuzzy parameters estimation. The performance of a fuzzy parameter estimation scheme should be measured with its transient behavior and its steady-state behavior. The transient behavior is characterized
1063-6706/$20.00 © 2006 IEEE
KUMAR et al.: A ROBUST DESIGN CRITERION FOR INTERPRETABLE FUZZY MODELS WITH UNCERTAIN DATA
315
the construction of membership functions based on knot vector , consider the following examples. a) Trapezoidal membership function: Let , input, such that for holds . Now, trapezoidal membership input can be functions for defined as if if otherwise if if Fig. 1. Trapezoidal membership functions.
if otherwise
by the stability and convergence rate whereas the steady-state behavior provides information about the mean-square-error once it reaches steady-state. In particular, the paper performs transient analysis in a purely deterministic framework without assuming a priori statistical information and steady-state analysis is carried out under the often realistic statistical assumptions. In the field of estimation theory, the classical linear least) estimation techniques and robust estimean-squares (or mation methods (such as ), are considered as two extremes estimation techniques, under in terms of their goals. The certain statistical assumptions on the signals, minimize the expected estimation error energy. On the other hand, robust estimation methods, safeguard against the worst-case uncertainties and therefore make no assumptions on the statistical nature of signals. Therefore, the feasibility of proposed fuzzy estimation strategy is shown through simulation studies by its comparison and ). with the existing extreme methods of estimation (
if if otherwise Fig. 1 shows an example with the choice of antecedent , , , , , parameters as: . and b) Gaussian membership functions with unit dispersion: Let , such that for input, holds for all . Now, Gaussian membership functions assuming unit dispersion input can be defined as for
II. SUGENO FUZZY INFERENCE SYSTEM Let us consider the problem of tuning a Sugeno fuzzy infer, mapping -dimensional input ence system to one dimensional real line, space rule is in the the form: consisting of different rules. The If is and is and is , then ; for all , where are nonempty fuzzy , respectively, such that membership subsets of functions fulfill for all . The values are real numbers. So, we have (1) We assume that belongs to a nonempty real intervals i.e. for all . We define a real vector such that the membership functions can be constructed from the elements of vector . To illustrate
Fig. 2 shows an example with the choice of antecedent , , and . parameters as: Total number of possible rules depends on the number of , where membership functions for each input i.e. is the number of membership functions defined over input. Depending upon the choice of membership functions, (1) can be rewritten as function of , i.e., (2) where (3) Let us introduce the notation
316
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 14, NO. 2, APRIL 2006
III. ROBUST ONLINE FUZZY IDENTIFICATION Let us consider a nonlinear system, mapping -dimensional to one-dimensional real line, described by input vector following unknown equation:
Our aim is to identify the previous nonlinear system using an interpretable Sugeno type fuzzy model from uncertain input–output identification data sequence . Here, is the multiplicative uncertainty present in input vector , and is the uncertainty present in output measurement . present in A practical assumption about the uncertainty is that the inequality holds good. That is, the uncertainty present in the output measurement is less than 100%. is the true value of unknown fuzzy Assume that, parameters, which approximate the nonlinear system, then we have
Fig. 2. Gaussian membership functions.
, , . Now, it can be written that
(4) Hence, the output of above defined type of fuzzy inference system comes out to be linear in consequent parameters but nonlinear in antecedent parameters. Expression (4) is a general representation that allows also to include multivariate membership functions. That is, if we consider a fuzzy model If
is
then
such that antecedent fuzzy set is defined by a multivariate membership function in the product-space domain of the vector , then we could define
Now, for example, Gaussian-type multivariate membership functions can be defined as
where we have
and
denotes the Euclidean norm. In this case,
If we denote the additive uncertainties as , then
and
(5) also includes modeling errors. If is the unNote that in certainty vector in regression vector due to uncertainty , then we have
where , is some un, , and . The vector known bounded function of is bounded due to the fact that each element of vector has a positive value between zero and one and sum of all its elements is equal to one [see (3)]. Therefore
Assume that we already have estimation of fuzzy parameters at time , say ( , ). Given the new measure, one can seek to improve the estimate of ments , by solving
where and, hence
where are the regularization parameters. Also, we want to preserve the interpretability of fuzzy system during learning. So the membership functions can be prevented from overlapping by imposing some constraints on the position of knots, for
KUMAR et al.: A ROBUST DESIGN CRITERION FOR INTERPRETABLE FUZZY MODELS WITH UNCERTAIN DATA
instance, in case of trapezoidal membership functions the constraints can be formulated, i.e., for all
for all
These constraints can be formulated in term of a matrix in(as in [3], [5], [7], [16], [22]). Let and equality be the upper bounds on and respectively (i.e., ). Now, the robust online identification problem is reduced to
317
corresponds Since each element of vector to the normalized firing strength of a rule (3), having any nonnegative value less than one and sum of all elements of vector is equal to 1, therefore, . It follows from (5) that
Therefore, a reasonable condition on the estimate is that it should also satisfy
of
This implies that (6) Noting
Define the quantities
, we have
Since by definition
, therefore for
The optimization problem (6) can be shown to be equivalent to
(9)
This implies that
(7) To show the equivalence of problems (6) and (7), we define for a given value of , and
If , then it follows from the triangle inequality of norms . Conversely, if , then define for a given the that perturbations
Then
, so that
This shows that two variables replaced by a single variable rewritten as
We find that the minimum value of is equal to . Therefore, there is no harm, so far as robustness is concerned, in solving a more conservative optimization problem
(10) The advantage for solving a more conservative problem is that can be seen as an equivthe measurement noise uncertainty in the realent additional uncertainty of gression vector uncertainty , and that would allow the explicit solution of the optimization problem. Let
, and . Hence,
in problem (6) can be in (7). Problem (7) can be
(8)
For any fixed values of parameters imization problem
For constant , the cost function , so that the maximization over in
, we consider the max-
is convex is achieved at the
318
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 14, NO. 2, APRIL 2006
boundary, . Therefore, the previous constrained optimization problem can be reformulated by introducing a nonnegative Lagrange multiplier , as follows:
Now, the optimization problem is reduced to
Let and be the optimal solution of above optimization problem obtained by differentiating the cost function with rewhich satisfy the equations spect to and
Let us define a measure of total data uncertainty
and Also the solution should satisfy hessian of the cost function w.r.t. inite. Now, the maximum cost
as
. This is because the must be nonpositive–defbecomes
so that and . is always positive and bounded for any Note from (9) that . Now, solving for from nonzeros output measurement (11), we get
if otherwise
It should be noted that . Applying matrix inversion lemma and then after some algebra, we get (assuming )
where
Noting that ,
, i.e., can be expressed as
Let us substitute the aforementioned value of and then simplify to get
The original problem (10) is, therefore, equivalent to
Let us introduce a three-variable cost function
into
as
Hence, the estimation equations can be written as Then, it can verified that
Thus, the original min–max problem becomes equivalent to
Now, to find the minimum of cost function , the gradient of with respect to resulting in
over is set to zero,
and
(11) Let be the optimal solution obtained by solving the previous equation, which obviously is a function of and , so we write
Thus, we see that a priori knowledge of parameter is required using the previous for the computation of parameters can estimation equations. However, in a practical situation, not be computed because of the lack of knowledge about noise ). Therefore, our concern in the signal magnitude (i.e.,
KUMAR et al.: A ROBUST DESIGN CRITERION FOR INTERPRETABLE FUZZY MODELS WITH UNCERTAIN DATA
319
next section is to estimate the fuzzy model parameters without a priori knowledge of . IV. ROBUST ONLINE FUZZY IDENTIFICATION WITHOUT A PRIORI KNOWLEDGE OF It is observed through repeated simulation studies that the tend to reach its minimum value at arguments function that are generally close to 1. Therefore, a practical approxi, for mation for the optimal is to set it close to 1, say some tuning parameter . Let us choose sufficiently higher value of regularization parameter such that . With these approximations of , and , the estimation equations are reduced to
Fig. 3. Time-variant mapping from data uncertainties and modeling error to the fuzzy estimation error.
We rewrite (13) in the following form:
(12) and (16)
(13) Now, the question is whether the estimation of fuzzy parameters using [see (12) and (13)] is robust to data uncertainties. If yes, then is it possible to justify the estimation scheme in a more “rigorous” manner. To answer these questions, we provide a time-domain feedback analysis to investigate the stability, robustness, and steady-state behavior of the estimation scheme. The analysis is motivated by the ideas presented in [23], where gradient-based adaptive schemes were considered. Let us assume that nonlinear system can be modeled as
Now,
can be expressed as
By evaluating the energies (i.e., squared norm) of both sides of this equation, we obtain
(17) In other words
where includes not only the data uncertainties, but also the error resulting from a difference between and . We define the following error measures for our analysis: denotes the difference between and its estimate , , and denotes the a priori estimation error, . It follows from (13) that
(18)
(19) (14) for simplicity. Let where we write denotes the a posteriori estimation error, . Using (14), we have
(15)
Relations (18) and (19) establish a time-variant mapping from the data uncertainties and modeling error to the resulting a priori estimation error , as shown in Fig. 3. The mapping from to , shown as feedforward gain of the loop in Fig. 3, is lossless, i.e., it preserves energy (18). Therefor, the feedforward gain of the loop has (2-induced) norm equal to one while the feedback loop consists of a gain factor that is equal to .
320
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 14, NO. 2, APRIL 2006
A. Stability, Robustness, and Convergence
We define the following:
The feedback structure of Fig. 3 allows us to study the -stability of the system using small gain theorem [24]. The -stability of the system means that it maps a bounded ento a bounded energy sequence ergy sequence , established by achieving an upper bound on the to . value of energy gain from For studying -stability, robustness and convergence, consider the estimation of fuzzy parameters using [see (12) and up to time . It follows from (18) (13)] from time that for every time instant , we have
Then
(20) Therefore
(22) The previous equality implies that
Since
corresponds to the firing strength of rules, therefore
(21) From (19), we have
where is the number of elements of vector of fuzzy rules). So
(total number
and The upper bounds on and , and inequality (22) can be combined to conclude the equation shown at the bottom of the next page. That is
where Therefore
Therefore
(23)
KUMAR et al.: A ROBUST DESIGN CRITERION FOR INTERPRETABLE FUZZY MODELS WITH UNCERTAIN DATA
Now, we establish the -stability of the system (shown in Fig. 3), according to small gain theorem. The small gain theorem states that the -stability of a feedback configuration requires that the product of the norms of the feedforward and the feedback operators is strictly bounded by one. For the system of Fig. 3, the feedforward (2-induced) norm is equal to one while . Therefore, the the feedback 2-induced norm is equal to , stability condition, is always satisfied. Inequality (23) explains the robustness property of fuzzy estimation scheme in a sense that an -stable mapto achieves an upper ping from bound on the value of energy gain from ); 1) initial guess deviation from the true value (i.e. 2) data uncertainties and modeling errors (i.e. ), to the a priori estimation error , in the sense of inequality (23). reflects a priori knowledge The term as to how close is to the initial guess and also determine the learning rate (or step-size) of the estimation algorithm. For a good initial guess, the value of should be low and vice-versa. This makes our basis of choosing the regularization parameter . Our approach is to choose a positive number between zero and one, say , depending upon a priori knowledge as to how close is to the initial guess . So, we have
for
321
In order to study the convergence properties of the fuzzy esand assume that timation scheme, we ignore the term . In this case, (19) is reduced to
Using (20) and the previous equality for the optimal choice of given by (24), we have
(26) Expression (26) shows the convergence property in a sense that consequent fuzzy parameters estimation energy is a nonincreasing function of time index . A higher value of paresults the higher rate of decrease of error energy. rameter may results in higher value of However the higher value of steady-state error that we will show in the next section. B. Steady-State Analysis This section is concerned with the steady-state error analysis of the fuzzy estimation strategy defined by (12) and (13), for the optimal choice of regularization parameter , in a stochastic setting. The aim is to determine the steady-state values of , , and/or . We start by substituting from (24) in (13),
(24)
We state that the previous choice of regularization parameter is optimal, since it results in an -stable mapping by achieving a robustness level, given as [from (23)]
Introduce the following notation:
and (25) We use the previous notations to rewrite the equations as where
and Therefore,
and
are related by (27)
322
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 14, NO. 2, APRIL 2006
We take expectation on both sides of (17) to get
By assumption A.1), it follows that
(28) and replace
by an equivalent expression (27), which yields (32)
(29) For the steady-state analysis, we make an often realistic and heavily used assumption in the field of adaptive filtering (see, e.g., [25] and [26]) that is independent, idenA.1) The zero-mean sequence tically distributed and statistically independent of the re. gressor sequence Since the fuzzy estimation scheme is stable, it should attain a steady-state where it holds that
where square error by
. We denote the steady-state mean. Using A.2), and (32), we have
(33)
Using (31) and (33), the steady-state (30) becomes
That is
Therefore, (29) becomes in steady state (30) Considering the left-hand side of (30) under assumption A.1), we have
(34)
We call upon the following approximation:
(31) in which case (34) leads to Before, we proceed to evaluate the other side of (30), we introduce the following “asymptotic” assumption. A.2) The random variables and are asymptotically independent, that is, as ,
By assumption A.2), the previous expression is reduced to where and Now, consider
are the functions of random variable.
Expressing
in terms of other variables, we get
(35)
KUMAR et al.: A ROBUST DESIGN CRITERION FOR INTERPRETABLE FUZZY MODELS WITH UNCERTAIN DATA
Noting that , the upper bound on steadysate mean-square error is given by
(36) The previous discussion presents the robustness (25), convergence (26), and steady-state analysis (36), of fuzzy estimation scheme of (12), (13) for the optimal choice of parameter given by (24). Until now, we have made no comment about the choice of other regularization parameter . Indeed, the choice of affects the performance of the estimation scheme through , , and in the expressions (25), (26), the terms and (36). The derivation of exact mathematical expressions, on the robustness, convergence, and showing the effect of steady-state properties does not seem to be an easy job due to the involved nonlinearities and interpretability constraints. However, motivated by the optimal choice of , we choose as
where and is a tuning parameter chosen according to the sensitivity of fuzzy output on the shape of mem, then for the bership functions. If we denote optimal choice of and previously suggested choice of , the estimation expressions (12), (13) are reduced to
323
which is a nonlinear constrained optimization problem that can be solved using Gauss–Newton method. We summarize our discussion by stating an algorithm for the robust design of an interpretable fuzzy model with uncertain data that does not require a priori knowledge of upper bounds on data uncertainty. The algorithm consists of following steps. , , , , , , and . 1) Choose and
2) Define
be the unique solution of following constrained let optimization problem solved by the algorithm suggested in [27]:
where is the Jacobian matrix of vector with respect to , determined by the method of finite-differences. The Jacobian is a full-rank matrix, as a result of using regularization. . 3) Compute 4) Compute
5)
and go to step 2). V. SIMULATIONS
That is (37) and
Now, we provide simulation studies to illustrate the proposed method of robust fuzzy identification. First, we consider three different nonlinear system identification problems to compare our approach with the existing standard techniques. Afterwards, we validate the steady-state analysis presented in the previous section. Based on the commonly used estimation methods, that does not require any priori knowledge of upper bounds, statistics and distribution of data uncertainties, there may be following approaches to solve the problem. 1) Gradient Descent: In the literature, gradient descent is a typical estimation method used in neural networks and fuzzy models, that seeks to decrease the value of the objective function based on the instantaneous error
(38) Let
The estimation of
A gradient descent learning law for updating the parame, has the form ters set
by (37) is equivalent to where is a step-size. It was shown in [28] that the instantaneous-gradient-based algorithms (such as backpropaga-optimal. tion) are locally
324
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 14, NO. 2, APRIL 2006
2) Gradient Descent + RLS: Another common method is to estimate the nonlinear parameters (antecedents) using gradient-descent and linear parameters (consequents) using recursive least-squares estimation (RLS). To highlight the robustness properties of this kind of hybrid training algorithm, assume that nonlinear system can be modeled as
where includes not only the data uncertainties, but also the error resulting from a difference between graand some true padient-descent estimated parameters rameters . For the estimation of linear parameter , we have following results: Adaptive Filtering: If and the are zero-mean, uncorrelated random variables with and unity, respectively, then variances by RLS algorithm estimation of
Fig. 4. Initial membership functions.
minimizes the maximum value of energy gain from disturbances to estimation error and ensures that
where
satisfies the (Riccati) recursion
for all . To compare our method with above three approaches, we consider three different examples. 1) One-Dimensional Example: Consider the fuzzy identification of an unknown nonlinear system described by
minimize the expected error energy
for all . 3) Gradient Descent + NLMS: Again, we estimate the nonlinear parameters using gradient-descent and model the nonlinear system as
where includes not only the data uncertainties, but also the error resulting from a difference between graand some true padient-descent estimated parameters rameters . Now, the estimation of according to NLMS algorithm follows as: Adaptive Filtering: The estimation of -optimal NLMS algorithm
using
Consider a fuzzy model with 4 gaussian shaped membership functions with initial guess of . The initial guess about the membership is taken functions is shown in Fig. 4. The initial guess equal to a zero vector. The nonlinear system was simulated by choosing from a uniform distribution on the interval [ 0.5,2.5]. The uncertain input–output identification data is generated by the sequence
where , are random entries, chosen from a normal distribution with mean zero and variance 0.01. To measure the estimation performance, we define instantaneous absolute estimation error (AE) at time instant as
KUMAR et al.: A ROBUST DESIGN CRITERION FOR INTERPRETABLE FUZZY MODELS WITH UNCERTAIN DATA
325
TABLE I A FUZZY MODEL FOR THE IDENTIFICATION OF A ONE-DIMENSIONAL NONLINEAR SYSTEM
Fig. 5. Membership functions at k = 10 000.
Fig. 7. Fuzzy identification of a one-dimensional nonlinear system.
Let us define and choose a fuzzy model, having gaussian type multivariate membership functions, consisting of following rule base:
where
is defined by
such that
Fig. 6. Comparison of different techniques for a one-dimensional example.
where the points are uniform distributed in , the one dimensional input space. Let us take , and choose and in such a way that two knots must be separated at least by a distance of 0.01. Fig. 6 is shows the simulation results, where error index plotted with index . Fig. 5 shows the identified membership functions and Table I shows the identified rule-base by the proposed algorithm. Finally, Fig. 7 shows the identification of the nonlinear system using a fuzzy model. 2) Two-Dimensional Example: Consider an unknown process described by
The initial guess about membership functions is taken as (equally spaced):
The initial guess is equal to a zero vector. The system and from a uniform diswas simulated by choosing tribution on the interval [ 0.5,2.5]. The uncertain inputoutput identification data consists of a sequence
326
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 14, NO. 2, APRIL 2006
TABLE II A FUZZY MODEL FOR THE IDENTIFICATION OF A TWO-DIMENSIONAL NONLINEAR SYSTEM
simulating the chaotic Mackey–Glass differential delay equation, i.e.,
with , , for . Fourth-order Runge-Kutta method was used for the simulation of above equation. The aim of the problem is to predict the value of by using a set of past values i.e. . The uncertain input-output identification data is a sequence
Fig. 8. Comparison of different techniques for a two-dimensional example.
where , , and are random, chosen from a normal distribution with mean zero and variance 0.01. Again, we define instantaneous absolute estimation error (AE) at time instant as
where the points are uniform distributed in the two-dimensional input space. Let us take and . The simulation results for the different techniques have been shown in Fig. 8. Table II describes the identified rule base using proposed approach . at 3) Chaotic Time Series Estimation: Let us consider a four-dimensional example to predict the future values of a chaotic time series. The time series is generated by
where , , , and are random entries, chosen from a normal distribution with mean zero and variance 0.01. Let us choose the trapezoidal type of membership functions such that number of membership functions assigned to each of four inputs (i.e. is equal to three. Let
and . Define instantaneous absolute estimation error (AE) at index as
KUMAR et al.: A ROBUST DESIGN CRITERION FOR INTERPRETABLE FUZZY MODELS WITH UNCERTAIN DATA
Fig. 9. Comparison of different techniques for chaotic time series estimation.
Fig. 10. Learning curves at experiments).
327
= 0:01, 0.05, and 0.1 (averaged over 1000
We run the simulations from to , taking and . The simulation results have been with . shown in Fig. 9 by plotting The aforementioned simulation studies (see Figs. 6, 8, and 9) clearly show the better robustness properties of the proposed algorithm in comparison to other techniques, since the bottom curve in all three cases refers to robust fuzzy filtering. We see that all other techniques (i.e., gradient-descent, gradient-descent+RLS, and gradient-descent+NLMS) reach their limits and the proposed approach outperforms them. Now, we provide simulation studies to validate the steadystate analysis. For this purpose, consider a model
where , is a vector that takes randomly the value either or , and is the noise taking randomly the value either 0.1 or 0.1. For this model, we have
The steady-state error for this model according to expression (35) is given as
(39) The proposed method was used to estimate . Fig. 10 shows the simulation studies at three different values of , where has been plotted with time index . The expected value of (i.e. ) has been calculated by averaging over 1000 experiments. Fig. 10 shows that with increase in the value of , both rate of convergence and steady-state error increases, as expected. To validate the expression (39), the simulations studies were carried out for different values of ranging from 0.01
Fig. 11. Steady-state error as a function of .
to 0.9. For each value of , the average value of over 1000 experiments is calculated, as an approximation to steady-state error . Fig. 11 shows the comparison of so calculated steady-state error in the simulations with the steady-state error calculated using expression (39). The simulation results are quite close to the theoretically predicted approximate results [i.e., using expression (39)]. We also verify the inequality (25) through simulations. VI. CONCLUDING REMARKS This study has outlined a criterion for the robust design of fuzzy models with uncertain data, when upper bounds on the data uncertainties are unknown. We derive an algorithm together with a stability, robustness, convergence, and steady-state analysis, for the problem of fuzzy filtering. The resulting procedure turns out to have better performance than the existing linear and filtering techniques). The methods of estimation ( better performance of fuzzy filtering is due to the fact that not
328
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 14, NO. 2, APRIL 2006
only linear parameters (consequents), but also the nonlinear parameters (antecedents) attempt to match the output of the fuzzy system to the unknown system output in an optimal manner. REFERENCES [1] L. Ljung, System Identification, Theory for the User. Englewood Cliffs, NJ: Prentice-Hall, 1987. [2] R. Babuˇska, Fuzzy Modeling for Control. Boston: Kluwer, 1998. [3] M. Kumar, R. Stoll, and N. Stoll, “Robust solution to fuzzy identification problem with uncertain data by regularization. Fuzzy approximation to physical fitness with real world medical date: an application,” Fuzzy Optim. Decision Making, vol. 3, no. 1, pp. 63–82, Mar. 2004. [4] L. X. Wang and J. M. Mendel, “Generating fuzzy rules by learning from examples,” IEEE Trans. Syst., Man, Cybern., vol. 22, no. 6, pp. 1414–1427, Nov. 1992. [5] M. Kumar, R. Stoll, and N. Stoll, “Robust adaptive fuzzy identification of time-varying processes with uncertain data. Handling uncertainties in the physical fitness fuzzy approximation with real world medical data: an application,” Fuzzy Optim. Decision Making, vol. 2, pp. 243–259, Sep. 2003. [6] J.-S. R. Jang, “ANFIS: Adaptive-network-based fuzzy inference systems,” IEEE Trans. Syst., Man, Cybern., vol. 23, no. 3, pp. 665–685, May 1993. [7] M. Kumar, R. Stoll, and N. Stoll, “Regularized adaptation of fuzzy inference systems. Modeling the opinion of a medical expert about physical fitness: An application,” Fuzzy Optim. Decision Making, vol. 2, Dec. 2003. [8] D. Nauck and R. Kruse, “A neuro-fuzzy approach to obtain interpretable fuzzy systems for function approximation,” in Proc. IEEE Int. Conf. Fuzzy Systems 1998 (FUZZ-IEEE’98), Anchorage, AK, May 1998, pp. 1106–1111. [9] F. Herrera, M. Lozano, and J. Verdegay, “Generating fuzzy rules from examples using genetic algorithms,” in Proc. 5th Int. Conf. Information Processing and Management of Uncertainty in Knowledge-Based Systems (IPMU’94), Paris, France, Jul. 1994, pp. 675–680. [10] K. Nozaki, H. Ishibuchi, and H. Tanaka, “A simple but powerful heuristic method for generating fuzzy rules from numerical data,” Fuzzy Sets Syst., vol. 86, pp. 251–270, 1997. [11] J. J. Shan and H. C. Fu, “A fuzzy neural network for rule acquiring on fuzzy control systems,” Fuzzy Sets Syst., vol. 71, pp. 345–357, 1995. [12] P. Thrift, “Fuzzy logic synthesis with genetic algorithms,” in Proc. 4th Int. Conf. Genetic Algorithms, 1991, pp. 509–513. [13] J. Liska and S. S. Melsheimer, “Complete design of fuzzy logic systems using genetic algorithms,” in Proc. 3rd IEEE Int. Conf. Fuzzy Systems, 1994, pp. 1377–1382. [14] A. Gonzáalez and R. Pérez, “Completeness and consistency conditions for learning fuzzy rules,” Fuzzy Sets Syst., vol. 96, pp. 37–51, 1998. [15] T. Johansen, “Robust identification of takagi-sugeno-kang fuzzy models using regularization,” in Proc. IEEE Conf. Fuzzy Systems, New Orleans, LA, 1996, pp. 180–186. [16] M. Burger, H. Engl, J. Haslinger, and U. Bodenhofer, “Regularized data-driven construction of fuzzy controllers,” J. Inverse Ill-Posed Prob., vol. 10, pp. 319–344, 2002. [17] T. Takagi and M. Sugeno, “Fuzzy identification of systems and its applications to modeling and control,” IEEE Trans. Syst., Man, Cybern., vol. SMC-15, no. l, pp. 116–132, Jan. 1985. [18] R. Babuˇska, “Construction of fuzzy systems-interplay between precision and transparency,” in Proc. ESIT 2000, Aachen, Germany, 2000, pp. 445–452. [19] U. Bodenhofer and P. Bauer, “Toward an axiomatic treatment of “interpretability’,” in Proc. IIZUKA2000, Oct. 2000, pp. 334–339. [20] J. Espinosa and J. Vandewalle, “Constructing fuzzy models with linguistic integrity from numerical data-AFRELI algorithm,” IEEE Trans. Fuzzy Syst., vol. 8, no. 5, pp. 591–600, Oct. 2000. [21] M. Setnes, R. Babuˇska, and H. B. Verbruggen, “Rule-based modeling: precision and transparency,” IEEE Trans. Syst., Man, Cybern., C, Appl. Rev., vol. 28, no. 1, pp. 165–169, Feb. 1998.
[22] M Kumar, R. Stoll, and N. Stoll, “Robust adaptive identification of fuzzy systems with uncertain data,” Fuzzy Optim. Decision Making, vol. 3, no. 3, pp. 195–216, Sept. 2004. [23] A. H. Sayed and M. Rupp, “A time-domain feedback analysis of adaptive gradient algorithms via the small gain theorem,” in Proc. SPIE Conf. Advanced Signal Processing, San Diego, CA, 1995, vol. 2563, pp. 458–469. [24] M. Vidyasagar, Nonlinear Systems Analysis, 2nd ed. Upper Saddle River, NJ: Prentice-Hall, 1993. [25] B. Widrow and S. D. Stearns, Adaptive Signal Processing. Englewood Cliffs, NJ: Prentice-Hall, 1985. [26] S. Haykin, Adaptive Filter Theory, 3rd ed. Upper Saddle River, NJ: Prentice-Hall, 1996. [27] C. L. Lawson and R. J. Hanson, Solving Least Squares Problems. Philadelphia, PA: SIAM, 1995. [28] B. Hassibi, A. H. Sayed, and T. Kailath, “ optimality criteria for LMS and backpropagation,” in Advances in Neural Information Processing Systems, J. D. Cowan, G. Tesauro, and J. Alspector, Eds. San Francisco, CA: Morgan-Kaufmann, Apr. 1994, vol. 6, pp. 351–359.
H
Mohit Kumar (A’05) received the B.Tech. degree in electrical engineering from the National Institute of Technology, Hamirpur, India, in 1999, the M.Tech. degree in control engineering from Indian Institute of Technology, Delhi, India, in 2001, and the Ph.D. degree in electrical engineering from Rostock University, Rostock, Germany, in 2004. He served as a Research Scientist in Institute of Occupational and Social Medicine, Rostock, Germany, from 2001 to 2004. Currently, he is with the Center for Life Science Automation, Rostock. His research interests include robust adaptive fuzzy identification, fuzzy logic in medicine, and robust adaptive control.
Regina Stoll received the diploma in medicine (Dip.-Med.), the Dr.Med. degree in occupational medicine, and the Dr.Med.Habil. degree in occupational and sports medicine from Rostock University, Rostock, Germany, in 1980, 1984, and 2002, respectively. She is head of the Institute of Occupational and Social Medicine, Rostock, Germany. She is a faculty member in the medicine faculty and faculty associate in the College of Computer Science and Electrical Engineering of Rostock University. She also holds the adjunct faculty member position in the industrial engineering department of North Carolina State University. Her research interests include occupational physiology, preventive medicine, and cardiopulmonary diagnostics.
Norbert Stoll received the diploma (Dip.-Ing.) in automation engineering in 1979, and the Ph.D. degree in measurement technology from Rostock University, Rostock, Germany, in 1985. He served as Head of Section Analytical Chemistry at the Academy of Sciences of GDR, Central Institute for Organic Chemistry till 1991. From 1992 to 1994, he was the Associate Director of Institute of Organic Catalysis, Rostock, Germany. Since 1994, he is a Professor of measurement technology in the Engineering Faculty of Rostock University. From 1994 to 2000, he directed the Institution of Automation in Rostock University. He is also holding, since 2003, the position of Vice President in Center for Life Science Automation, Rostock. His fields of interests include medical process measurement, lab automation, and smart systems and devices.
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 14, NO. 2, APRIL 2006
A Robust Design Criterion for Interpretable Fuzzy Models With Uncertain Data Mohit Kumar, Associate Member, IEEE, Regina Stoll, and Norbert Stoll
Abstract—We believe that nonlinear fuzzy filtering techniques may be turned out to give better robustness performance than the filtering techexisting linear methods of estimation ( 2 and niques), because of the fact that not only linear parameters (consequents), but also the nonlinear parameters (membership functions) attempt to identify the uncertain behavior of the unknown system. However, the fuzzy identification methods must be robust to data uncertainties and modeling errors to ensure that the fuzzy approximation of unknown system’s behavior is optimal in some sense. This study presents a deterministic approach to the robust design of fuzzy models in the presence of unknown but finite uncertainties in the identification data. We consider online identification of an interpretable fuzzy model, based on the robust solution of a regularized least-squares fuzzy parameters estimation problem. The aim is to resolve the difficulties associated with the robust fuzzy identification method due to lack of a priori knowledge about upper bounds on the data uncertainties. The study derives an optimal level of regularization that should be provided to ensure the robustness of fuzzy identification strategy by achieving an upper bound on the value of energy gain from data uncertainties and modeling errors to the estimation errors. A time-domain feedback analysis of the proposed identification approach is carried out with emphasis on stability, robustness, and steady-state issues. The simulation studies are provided to show the superiority of the proposed fuzzy estimation over the classical estimation methods. Index Terms— -optimality regularization, interpretability, least-squares, 2 -stability, min–max identification, normalized least mean squares algorithm (NLMS) algorithm.
I. INTRODUCTION
M
ANY real-world physical processes are generally characterized by the presence of nonlinearity, complexity and uncertainty. These processes cannot be represented by linear models used in conventional system identification [1]. The capability of fuzzy systems paradigm for not only learning complex input-output mappings but also to interpret these mappings with linguistic terms stimulates the study of approximating ill-defined and complex processes using a fuzzy inference system. Therefore, the fuzzy identification of nonlinear systems from input–output data has become an important topic of scientific research with a wide range of applications [2]–[9]. A
Manuscript received December 22, 2003; revised December 30, 2004, March 8, 2005, and May 2, 2005. This work was supported by European Space Agency under ESTEC Contract 14350/01/NL/SHMAP project AO-99-058. M. Kumar and R. Stoll are with the Institute of Occupational and Social Medicine, Faculty of Medicine, University of Rostock, D-18055 Rostock, Germany (e-mail: [email protected]; [email protected]). N. Stoll is with the Institute of Automation, College of Computer Science and Electrical Engineering, University of Rostock, D-18119 Rostock-Warnemünde, Germany (e-mail: [email protected]). Digital Object Identifier 10.1109/TFUZZ.2005.861614
large number of techniques have been developed for the fuzzy identification of nonlinear systems from measured input-output data. These techniques can be grouped into three approaches and their combinations: ad-hoc data covering approaches [4], [10]), neural networks ([6], [11]), and genetic algorithms [9], [12]–[14]. However, these methods do not consider the situations when the training data is uncertain. Regularization was suggested as a method for improving the robustness of fuzzy identification scheme in [7], [15], and [16]. However, the choice of regularization parameter is usually not obvious and application dependent. An iterative method for the robust (min–max) identification of fuzzy parameters with uncertain data, was suggested in [3], by solving an equivalent optimally regularized identification problem. However, the method is offline and requires a priori knowledge about upper bounds on the data uncertainties. At present, the literature lacks robustness, stability, and steady-state analysis of online fuzzy identification methods that don’t require the knowledge of upper bounds on uncertainties, in the deterministic or stochastic framework. We consider the fuzzy identification problem for a special class of fuzzy systems (Sugeno type fuzzy systems), since they ideally combine simplicity with good analytical properties [17]. Also, if we take into account the appropriate restrictions in terms of interpretability, the data-driven construction of Sugeno fuzzy systems allows qualitative insight into the relationships [18]–[21]. This paper starts with the mathematical formulation of Sugeno type fuzzy systems and considers the fuzzy parameters estimation with uncertain data, based on the robust solution of a regularized least-squares problem. By a robust solution we mean one that attempts to alleviate the worst-case effect of data uncertainties on fuzzy parameters estimation performance. However, the knowledge of an upper bound on the values of data uncertainties is needed to compute the robust solution of fuzzy parameter estimation problem. Therefore, we consider in particular, the derivation of a robust fuzzy identification method that does not require a priori knowledge about upper bounds on the data uncertainties, by providing a suitable choice of regularization parameters. The proposed method is shown to be robust and stable by providing a time-domain feedback analysis. The analysis highlights and exploits an intrinsic feedback structure, mapping the data uncertainties to the estimation errors. The optimal choice of regularization parameters is motivated by the robustness analysis of the mapping that can be associated with the proposed method of fuzzy parameters estimation. The performance of a fuzzy parameter estimation scheme should be measured with its transient behavior and its steady-state behavior. The transient behavior is characterized
1063-6706/$20.00 © 2006 IEEE
KUMAR et al.: A ROBUST DESIGN CRITERION FOR INTERPRETABLE FUZZY MODELS WITH UNCERTAIN DATA
315
the construction of membership functions based on knot vector , consider the following examples. a) Trapezoidal membership function: Let , input, such that for holds . Now, trapezoidal membership input can be functions for defined as if if otherwise if if Fig. 1. Trapezoidal membership functions.
if otherwise
by the stability and convergence rate whereas the steady-state behavior provides information about the mean-square-error once it reaches steady-state. In particular, the paper performs transient analysis in a purely deterministic framework without assuming a priori statistical information and steady-state analysis is carried out under the often realistic statistical assumptions. In the field of estimation theory, the classical linear least) estimation techniques and robust estimean-squares (or mation methods (such as ), are considered as two extremes estimation techniques, under in terms of their goals. The certain statistical assumptions on the signals, minimize the expected estimation error energy. On the other hand, robust estimation methods, safeguard against the worst-case uncertainties and therefore make no assumptions on the statistical nature of signals. Therefore, the feasibility of proposed fuzzy estimation strategy is shown through simulation studies by its comparison and ). with the existing extreme methods of estimation (
if if otherwise Fig. 1 shows an example with the choice of antecedent , , , , , parameters as: . and b) Gaussian membership functions with unit dispersion: Let , such that for input, holds for all . Now, Gaussian membership functions assuming unit dispersion input can be defined as for
II. SUGENO FUZZY INFERENCE SYSTEM Let us consider the problem of tuning a Sugeno fuzzy infer, mapping -dimensional input ence system to one dimensional real line, space rule is in the the form: consisting of different rules. The If is and is and is , then ; for all , where are nonempty fuzzy , respectively, such that membership subsets of functions fulfill for all . The values are real numbers. So, we have (1) We assume that belongs to a nonempty real intervals i.e. for all . We define a real vector such that the membership functions can be constructed from the elements of vector . To illustrate
Fig. 2 shows an example with the choice of antecedent , , and . parameters as: Total number of possible rules depends on the number of , where membership functions for each input i.e. is the number of membership functions defined over input. Depending upon the choice of membership functions, (1) can be rewritten as function of , i.e., (2) where (3) Let us introduce the notation
316
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 14, NO. 2, APRIL 2006
III. ROBUST ONLINE FUZZY IDENTIFICATION Let us consider a nonlinear system, mapping -dimensional to one-dimensional real line, described by input vector following unknown equation:
Our aim is to identify the previous nonlinear system using an interpretable Sugeno type fuzzy model from uncertain input–output identification data sequence . Here, is the multiplicative uncertainty present in input vector , and is the uncertainty present in output measurement . present in A practical assumption about the uncertainty is that the inequality holds good. That is, the uncertainty present in the output measurement is less than 100%. is the true value of unknown fuzzy Assume that, parameters, which approximate the nonlinear system, then we have
Fig. 2. Gaussian membership functions.
, , . Now, it can be written that
(4) Hence, the output of above defined type of fuzzy inference system comes out to be linear in consequent parameters but nonlinear in antecedent parameters. Expression (4) is a general representation that allows also to include multivariate membership functions. That is, if we consider a fuzzy model If
is
then
such that antecedent fuzzy set is defined by a multivariate membership function in the product-space domain of the vector , then we could define
Now, for example, Gaussian-type multivariate membership functions can be defined as
where we have
and
denotes the Euclidean norm. In this case,
If we denote the additive uncertainties as , then
and
(5) also includes modeling errors. If is the unNote that in certainty vector in regression vector due to uncertainty , then we have
where , is some un, , and . The vector known bounded function of is bounded due to the fact that each element of vector has a positive value between zero and one and sum of all its elements is equal to one [see (3)]. Therefore
Assume that we already have estimation of fuzzy parameters at time , say ( , ). Given the new measure, one can seek to improve the estimate of ments , by solving
where and, hence
where are the regularization parameters. Also, we want to preserve the interpretability of fuzzy system during learning. So the membership functions can be prevented from overlapping by imposing some constraints on the position of knots, for
KUMAR et al.: A ROBUST DESIGN CRITERION FOR INTERPRETABLE FUZZY MODELS WITH UNCERTAIN DATA
instance, in case of trapezoidal membership functions the constraints can be formulated, i.e., for all
for all
These constraints can be formulated in term of a matrix in(as in [3], [5], [7], [16], [22]). Let and equality be the upper bounds on and respectively (i.e., ). Now, the robust online identification problem is reduced to
317
corresponds Since each element of vector to the normalized firing strength of a rule (3), having any nonnegative value less than one and sum of all elements of vector is equal to 1, therefore, . It follows from (5) that
Therefore, a reasonable condition on the estimate is that it should also satisfy
of
This implies that (6) Noting
Define the quantities
, we have
Since by definition
, therefore for
The optimization problem (6) can be shown to be equivalent to
(9)
This implies that
(7) To show the equivalence of problems (6) and (7), we define for a given value of , and
If , then it follows from the triangle inequality of norms . Conversely, if , then define for a given the that perturbations
Then
, so that
This shows that two variables replaced by a single variable rewritten as
We find that the minimum value of is equal to . Therefore, there is no harm, so far as robustness is concerned, in solving a more conservative optimization problem
(10) The advantage for solving a more conservative problem is that can be seen as an equivthe measurement noise uncertainty in the realent additional uncertainty of gression vector uncertainty , and that would allow the explicit solution of the optimization problem. Let
, and . Hence,
in problem (6) can be in (7). Problem (7) can be
(8)
For any fixed values of parameters imization problem
For constant , the cost function , so that the maximization over in
, we consider the max-
is convex is achieved at the
318
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 14, NO. 2, APRIL 2006
boundary, . Therefore, the previous constrained optimization problem can be reformulated by introducing a nonnegative Lagrange multiplier , as follows:
Now, the optimization problem is reduced to
Let and be the optimal solution of above optimization problem obtained by differentiating the cost function with rewhich satisfy the equations spect to and
Let us define a measure of total data uncertainty
and Also the solution should satisfy hessian of the cost function w.r.t. inite. Now, the maximum cost
as
. This is because the must be nonpositive–defbecomes
so that and . is always positive and bounded for any Note from (9) that . Now, solving for from nonzeros output measurement (11), we get
if otherwise
It should be noted that . Applying matrix inversion lemma and then after some algebra, we get (assuming )
where
Noting that ,
, i.e., can be expressed as
Let us substitute the aforementioned value of and then simplify to get
The original problem (10) is, therefore, equivalent to
Let us introduce a three-variable cost function
into
as
Hence, the estimation equations can be written as Then, it can verified that
Thus, the original min–max problem becomes equivalent to
Now, to find the minimum of cost function , the gradient of with respect to resulting in
over is set to zero,
and
(11) Let be the optimal solution obtained by solving the previous equation, which obviously is a function of and , so we write
Thus, we see that a priori knowledge of parameter is required using the previous for the computation of parameters can estimation equations. However, in a practical situation, not be computed because of the lack of knowledge about noise ). Therefore, our concern in the signal magnitude (i.e.,
KUMAR et al.: A ROBUST DESIGN CRITERION FOR INTERPRETABLE FUZZY MODELS WITH UNCERTAIN DATA
319
next section is to estimate the fuzzy model parameters without a priori knowledge of . IV. ROBUST ONLINE FUZZY IDENTIFICATION WITHOUT A PRIORI KNOWLEDGE OF It is observed through repeated simulation studies that the tend to reach its minimum value at arguments function that are generally close to 1. Therefore, a practical approxi, for mation for the optimal is to set it close to 1, say some tuning parameter . Let us choose sufficiently higher value of regularization parameter such that . With these approximations of , and , the estimation equations are reduced to
Fig. 3. Time-variant mapping from data uncertainties and modeling error to the fuzzy estimation error.
We rewrite (13) in the following form:
(12) and (16)
(13) Now, the question is whether the estimation of fuzzy parameters using [see (12) and (13)] is robust to data uncertainties. If yes, then is it possible to justify the estimation scheme in a more “rigorous” manner. To answer these questions, we provide a time-domain feedback analysis to investigate the stability, robustness, and steady-state behavior of the estimation scheme. The analysis is motivated by the ideas presented in [23], where gradient-based adaptive schemes were considered. Let us assume that nonlinear system can be modeled as
Now,
can be expressed as
By evaluating the energies (i.e., squared norm) of both sides of this equation, we obtain
(17) In other words
where includes not only the data uncertainties, but also the error resulting from a difference between and . We define the following error measures for our analysis: denotes the difference between and its estimate , , and denotes the a priori estimation error, . It follows from (13) that
(18)
(19) (14) for simplicity. Let where we write denotes the a posteriori estimation error, . Using (14), we have
(15)
Relations (18) and (19) establish a time-variant mapping from the data uncertainties and modeling error to the resulting a priori estimation error , as shown in Fig. 3. The mapping from to , shown as feedforward gain of the loop in Fig. 3, is lossless, i.e., it preserves energy (18). Therefor, the feedforward gain of the loop has (2-induced) norm equal to one while the feedback loop consists of a gain factor that is equal to .
320
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 14, NO. 2, APRIL 2006
A. Stability, Robustness, and Convergence
We define the following:
The feedback structure of Fig. 3 allows us to study the -stability of the system using small gain theorem [24]. The -stability of the system means that it maps a bounded ento a bounded energy sequence ergy sequence , established by achieving an upper bound on the to . value of energy gain from For studying -stability, robustness and convergence, consider the estimation of fuzzy parameters using [see (12) and up to time . It follows from (18) (13)] from time that for every time instant , we have
Then
(20) Therefore
(22) The previous equality implies that
Since
corresponds to the firing strength of rules, therefore
(21) From (19), we have
where is the number of elements of vector of fuzzy rules). So
(total number
and The upper bounds on and , and inequality (22) can be combined to conclude the equation shown at the bottom of the next page. That is
where Therefore
Therefore
(23)
KUMAR et al.: A ROBUST DESIGN CRITERION FOR INTERPRETABLE FUZZY MODELS WITH UNCERTAIN DATA
Now, we establish the -stability of the system (shown in Fig. 3), according to small gain theorem. The small gain theorem states that the -stability of a feedback configuration requires that the product of the norms of the feedforward and the feedback operators is strictly bounded by one. For the system of Fig. 3, the feedforward (2-induced) norm is equal to one while . Therefore, the the feedback 2-induced norm is equal to , stability condition, is always satisfied. Inequality (23) explains the robustness property of fuzzy estimation scheme in a sense that an -stable mapto achieves an upper ping from bound on the value of energy gain from ); 1) initial guess deviation from the true value (i.e. 2) data uncertainties and modeling errors (i.e. ), to the a priori estimation error , in the sense of inequality (23). reflects a priori knowledge The term as to how close is to the initial guess and also determine the learning rate (or step-size) of the estimation algorithm. For a good initial guess, the value of should be low and vice-versa. This makes our basis of choosing the regularization parameter . Our approach is to choose a positive number between zero and one, say , depending upon a priori knowledge as to how close is to the initial guess . So, we have
for
321
In order to study the convergence properties of the fuzzy esand assume that timation scheme, we ignore the term . In this case, (19) is reduced to
Using (20) and the previous equality for the optimal choice of given by (24), we have
(26) Expression (26) shows the convergence property in a sense that consequent fuzzy parameters estimation energy is a nonincreasing function of time index . A higher value of paresults the higher rate of decrease of error energy. rameter may results in higher value of However the higher value of steady-state error that we will show in the next section. B. Steady-State Analysis This section is concerned with the steady-state error analysis of the fuzzy estimation strategy defined by (12) and (13), for the optimal choice of regularization parameter , in a stochastic setting. The aim is to determine the steady-state values of , , and/or . We start by substituting from (24) in (13),
(24)
We state that the previous choice of regularization parameter is optimal, since it results in an -stable mapping by achieving a robustness level, given as [from (23)]
Introduce the following notation:
and (25) We use the previous notations to rewrite the equations as where
and Therefore,
and
are related by (27)
322
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 14, NO. 2, APRIL 2006
We take expectation on both sides of (17) to get
By assumption A.1), it follows that
(28) and replace
by an equivalent expression (27), which yields (32)
(29) For the steady-state analysis, we make an often realistic and heavily used assumption in the field of adaptive filtering (see, e.g., [25] and [26]) that is independent, idenA.1) The zero-mean sequence tically distributed and statistically independent of the re. gressor sequence Since the fuzzy estimation scheme is stable, it should attain a steady-state where it holds that
where square error by
. We denote the steady-state mean. Using A.2), and (32), we have
(33)
Using (31) and (33), the steady-state (30) becomes
That is
Therefore, (29) becomes in steady state (30) Considering the left-hand side of (30) under assumption A.1), we have
(34)
We call upon the following approximation:
(31) in which case (34) leads to Before, we proceed to evaluate the other side of (30), we introduce the following “asymptotic” assumption. A.2) The random variables and are asymptotically independent, that is, as ,
By assumption A.2), the previous expression is reduced to where and Now, consider
are the functions of random variable.
Expressing
in terms of other variables, we get
(35)
KUMAR et al.: A ROBUST DESIGN CRITERION FOR INTERPRETABLE FUZZY MODELS WITH UNCERTAIN DATA
Noting that , the upper bound on steadysate mean-square error is given by
(36) The previous discussion presents the robustness (25), convergence (26), and steady-state analysis (36), of fuzzy estimation scheme of (12), (13) for the optimal choice of parameter given by (24). Until now, we have made no comment about the choice of other regularization parameter . Indeed, the choice of affects the performance of the estimation scheme through , , and in the expressions (25), (26), the terms and (36). The derivation of exact mathematical expressions, on the robustness, convergence, and showing the effect of steady-state properties does not seem to be an easy job due to the involved nonlinearities and interpretability constraints. However, motivated by the optimal choice of , we choose as
where and is a tuning parameter chosen according to the sensitivity of fuzzy output on the shape of mem, then for the bership functions. If we denote optimal choice of and previously suggested choice of , the estimation expressions (12), (13) are reduced to
323
which is a nonlinear constrained optimization problem that can be solved using Gauss–Newton method. We summarize our discussion by stating an algorithm for the robust design of an interpretable fuzzy model with uncertain data that does not require a priori knowledge of upper bounds on data uncertainty. The algorithm consists of following steps. , , , , , , and . 1) Choose and
2) Define
be the unique solution of following constrained let optimization problem solved by the algorithm suggested in [27]:
where is the Jacobian matrix of vector with respect to , determined by the method of finite-differences. The Jacobian is a full-rank matrix, as a result of using regularization. . 3) Compute 4) Compute
5)
and go to step 2). V. SIMULATIONS
That is (37) and
Now, we provide simulation studies to illustrate the proposed method of robust fuzzy identification. First, we consider three different nonlinear system identification problems to compare our approach with the existing standard techniques. Afterwards, we validate the steady-state analysis presented in the previous section. Based on the commonly used estimation methods, that does not require any priori knowledge of upper bounds, statistics and distribution of data uncertainties, there may be following approaches to solve the problem. 1) Gradient Descent: In the literature, gradient descent is a typical estimation method used in neural networks and fuzzy models, that seeks to decrease the value of the objective function based on the instantaneous error
(38) Let
The estimation of
A gradient descent learning law for updating the parame, has the form ters set
by (37) is equivalent to where is a step-size. It was shown in [28] that the instantaneous-gradient-based algorithms (such as backpropaga-optimal. tion) are locally
324
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 14, NO. 2, APRIL 2006
2) Gradient Descent + RLS: Another common method is to estimate the nonlinear parameters (antecedents) using gradient-descent and linear parameters (consequents) using recursive least-squares estimation (RLS). To highlight the robustness properties of this kind of hybrid training algorithm, assume that nonlinear system can be modeled as
where includes not only the data uncertainties, but also the error resulting from a difference between graand some true padient-descent estimated parameters rameters . For the estimation of linear parameter , we have following results: Adaptive Filtering: If and the are zero-mean, uncorrelated random variables with and unity, respectively, then variances by RLS algorithm estimation of
Fig. 4. Initial membership functions.
minimizes the maximum value of energy gain from disturbances to estimation error and ensures that
where
satisfies the (Riccati) recursion
for all . To compare our method with above three approaches, we consider three different examples. 1) One-Dimensional Example: Consider the fuzzy identification of an unknown nonlinear system described by
minimize the expected error energy
for all . 3) Gradient Descent + NLMS: Again, we estimate the nonlinear parameters using gradient-descent and model the nonlinear system as
where includes not only the data uncertainties, but also the error resulting from a difference between graand some true padient-descent estimated parameters rameters . Now, the estimation of according to NLMS algorithm follows as: Adaptive Filtering: The estimation of -optimal NLMS algorithm
using
Consider a fuzzy model with 4 gaussian shaped membership functions with initial guess of . The initial guess about the membership is taken functions is shown in Fig. 4. The initial guess equal to a zero vector. The nonlinear system was simulated by choosing from a uniform distribution on the interval [ 0.5,2.5]. The uncertain input–output identification data is generated by the sequence
where , are random entries, chosen from a normal distribution with mean zero and variance 0.01. To measure the estimation performance, we define instantaneous absolute estimation error (AE) at time instant as
KUMAR et al.: A ROBUST DESIGN CRITERION FOR INTERPRETABLE FUZZY MODELS WITH UNCERTAIN DATA
325
TABLE I A FUZZY MODEL FOR THE IDENTIFICATION OF A ONE-DIMENSIONAL NONLINEAR SYSTEM
Fig. 5. Membership functions at k = 10 000.
Fig. 7. Fuzzy identification of a one-dimensional nonlinear system.
Let us define and choose a fuzzy model, having gaussian type multivariate membership functions, consisting of following rule base:
where
is defined by
such that
Fig. 6. Comparison of different techniques for a one-dimensional example.
where the points are uniform distributed in , the one dimensional input space. Let us take , and choose and in such a way that two knots must be separated at least by a distance of 0.01. Fig. 6 is shows the simulation results, where error index plotted with index . Fig. 5 shows the identified membership functions and Table I shows the identified rule-base by the proposed algorithm. Finally, Fig. 7 shows the identification of the nonlinear system using a fuzzy model. 2) Two-Dimensional Example: Consider an unknown process described by
The initial guess about membership functions is taken as (equally spaced):
The initial guess is equal to a zero vector. The system and from a uniform diswas simulated by choosing tribution on the interval [ 0.5,2.5]. The uncertain inputoutput identification data consists of a sequence
326
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 14, NO. 2, APRIL 2006
TABLE II A FUZZY MODEL FOR THE IDENTIFICATION OF A TWO-DIMENSIONAL NONLINEAR SYSTEM
simulating the chaotic Mackey–Glass differential delay equation, i.e.,
with , , for . Fourth-order Runge-Kutta method was used for the simulation of above equation. The aim of the problem is to predict the value of by using a set of past values i.e. . The uncertain input-output identification data is a sequence
Fig. 8. Comparison of different techniques for a two-dimensional example.
where , , and are random, chosen from a normal distribution with mean zero and variance 0.01. Again, we define instantaneous absolute estimation error (AE) at time instant as
where the points are uniform distributed in the two-dimensional input space. Let us take and . The simulation results for the different techniques have been shown in Fig. 8. Table II describes the identified rule base using proposed approach . at 3) Chaotic Time Series Estimation: Let us consider a four-dimensional example to predict the future values of a chaotic time series. The time series is generated by
where , , , and are random entries, chosen from a normal distribution with mean zero and variance 0.01. Let us choose the trapezoidal type of membership functions such that number of membership functions assigned to each of four inputs (i.e. is equal to three. Let
and . Define instantaneous absolute estimation error (AE) at index as
KUMAR et al.: A ROBUST DESIGN CRITERION FOR INTERPRETABLE FUZZY MODELS WITH UNCERTAIN DATA
Fig. 9. Comparison of different techniques for chaotic time series estimation.
Fig. 10. Learning curves at experiments).
327
= 0:01, 0.05, and 0.1 (averaged over 1000
We run the simulations from to , taking and . The simulation results have been with . shown in Fig. 9 by plotting The aforementioned simulation studies (see Figs. 6, 8, and 9) clearly show the better robustness properties of the proposed algorithm in comparison to other techniques, since the bottom curve in all three cases refers to robust fuzzy filtering. We see that all other techniques (i.e., gradient-descent, gradient-descent+RLS, and gradient-descent+NLMS) reach their limits and the proposed approach outperforms them. Now, we provide simulation studies to validate the steadystate analysis. For this purpose, consider a model
where , is a vector that takes randomly the value either or , and is the noise taking randomly the value either 0.1 or 0.1. For this model, we have
The steady-state error for this model according to expression (35) is given as
(39) The proposed method was used to estimate . Fig. 10 shows the simulation studies at three different values of , where has been plotted with time index . The expected value of (i.e. ) has been calculated by averaging over 1000 experiments. Fig. 10 shows that with increase in the value of , both rate of convergence and steady-state error increases, as expected. To validate the expression (39), the simulations studies were carried out for different values of ranging from 0.01
Fig. 11. Steady-state error as a function of .
to 0.9. For each value of , the average value of over 1000 experiments is calculated, as an approximation to steady-state error . Fig. 11 shows the comparison of so calculated steady-state error in the simulations with the steady-state error calculated using expression (39). The simulation results are quite close to the theoretically predicted approximate results [i.e., using expression (39)]. We also verify the inequality (25) through simulations. VI. CONCLUDING REMARKS This study has outlined a criterion for the robust design of fuzzy models with uncertain data, when upper bounds on the data uncertainties are unknown. We derive an algorithm together with a stability, robustness, convergence, and steady-state analysis, for the problem of fuzzy filtering. The resulting procedure turns out to have better performance than the existing linear and filtering techniques). The methods of estimation ( better performance of fuzzy filtering is due to the fact that not
328
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 14, NO. 2, APRIL 2006
only linear parameters (consequents), but also the nonlinear parameters (antecedents) attempt to match the output of the fuzzy system to the unknown system output in an optimal manner. REFERENCES [1] L. Ljung, System Identification, Theory for the User. Englewood Cliffs, NJ: Prentice-Hall, 1987. [2] R. Babuˇska, Fuzzy Modeling for Control. Boston: Kluwer, 1998. [3] M. Kumar, R. Stoll, and N. Stoll, “Robust solution to fuzzy identification problem with uncertain data by regularization. Fuzzy approximation to physical fitness with real world medical date: an application,” Fuzzy Optim. Decision Making, vol. 3, no. 1, pp. 63–82, Mar. 2004. [4] L. X. Wang and J. M. Mendel, “Generating fuzzy rules by learning from examples,” IEEE Trans. Syst., Man, Cybern., vol. 22, no. 6, pp. 1414–1427, Nov. 1992. [5] M. Kumar, R. Stoll, and N. Stoll, “Robust adaptive fuzzy identification of time-varying processes with uncertain data. Handling uncertainties in the physical fitness fuzzy approximation with real world medical data: an application,” Fuzzy Optim. Decision Making, vol. 2, pp. 243–259, Sep. 2003. [6] J.-S. R. Jang, “ANFIS: Adaptive-network-based fuzzy inference systems,” IEEE Trans. Syst., Man, Cybern., vol. 23, no. 3, pp. 665–685, May 1993. [7] M. Kumar, R. Stoll, and N. Stoll, “Regularized adaptation of fuzzy inference systems. Modeling the opinion of a medical expert about physical fitness: An application,” Fuzzy Optim. Decision Making, vol. 2, Dec. 2003. [8] D. Nauck and R. Kruse, “A neuro-fuzzy approach to obtain interpretable fuzzy systems for function approximation,” in Proc. IEEE Int. Conf. Fuzzy Systems 1998 (FUZZ-IEEE’98), Anchorage, AK, May 1998, pp. 1106–1111. [9] F. Herrera, M. Lozano, and J. Verdegay, “Generating fuzzy rules from examples using genetic algorithms,” in Proc. 5th Int. Conf. Information Processing and Management of Uncertainty in Knowledge-Based Systems (IPMU’94), Paris, France, Jul. 1994, pp. 675–680. [10] K. Nozaki, H. Ishibuchi, and H. Tanaka, “A simple but powerful heuristic method for generating fuzzy rules from numerical data,” Fuzzy Sets Syst., vol. 86, pp. 251–270, 1997. [11] J. J. Shan and H. C. Fu, “A fuzzy neural network for rule acquiring on fuzzy control systems,” Fuzzy Sets Syst., vol. 71, pp. 345–357, 1995. [12] P. Thrift, “Fuzzy logic synthesis with genetic algorithms,” in Proc. 4th Int. Conf. Genetic Algorithms, 1991, pp. 509–513. [13] J. Liska and S. S. Melsheimer, “Complete design of fuzzy logic systems using genetic algorithms,” in Proc. 3rd IEEE Int. Conf. Fuzzy Systems, 1994, pp. 1377–1382. [14] A. Gonzáalez and R. Pérez, “Completeness and consistency conditions for learning fuzzy rules,” Fuzzy Sets Syst., vol. 96, pp. 37–51, 1998. [15] T. Johansen, “Robust identification of takagi-sugeno-kang fuzzy models using regularization,” in Proc. IEEE Conf. Fuzzy Systems, New Orleans, LA, 1996, pp. 180–186. [16] M. Burger, H. Engl, J. Haslinger, and U. Bodenhofer, “Regularized data-driven construction of fuzzy controllers,” J. Inverse Ill-Posed Prob., vol. 10, pp. 319–344, 2002. [17] T. Takagi and M. Sugeno, “Fuzzy identification of systems and its applications to modeling and control,” IEEE Trans. Syst., Man, Cybern., vol. SMC-15, no. l, pp. 116–132, Jan. 1985. [18] R. Babuˇska, “Construction of fuzzy systems-interplay between precision and transparency,” in Proc. ESIT 2000, Aachen, Germany, 2000, pp. 445–452. [19] U. Bodenhofer and P. Bauer, “Toward an axiomatic treatment of “interpretability’,” in Proc. IIZUKA2000, Oct. 2000, pp. 334–339. [20] J. Espinosa and J. Vandewalle, “Constructing fuzzy models with linguistic integrity from numerical data-AFRELI algorithm,” IEEE Trans. Fuzzy Syst., vol. 8, no. 5, pp. 591–600, Oct. 2000. [21] M. Setnes, R. Babuˇska, and H. B. Verbruggen, “Rule-based modeling: precision and transparency,” IEEE Trans. Syst., Man, Cybern., C, Appl. Rev., vol. 28, no. 1, pp. 165–169, Feb. 1998.
[22] M Kumar, R. Stoll, and N. Stoll, “Robust adaptive identification of fuzzy systems with uncertain data,” Fuzzy Optim. Decision Making, vol. 3, no. 3, pp. 195–216, Sept. 2004. [23] A. H. Sayed and M. Rupp, “A time-domain feedback analysis of adaptive gradient algorithms via the small gain theorem,” in Proc. SPIE Conf. Advanced Signal Processing, San Diego, CA, 1995, vol. 2563, pp. 458–469. [24] M. Vidyasagar, Nonlinear Systems Analysis, 2nd ed. Upper Saddle River, NJ: Prentice-Hall, 1993. [25] B. Widrow and S. D. Stearns, Adaptive Signal Processing. Englewood Cliffs, NJ: Prentice-Hall, 1985. [26] S. Haykin, Adaptive Filter Theory, 3rd ed. Upper Saddle River, NJ: Prentice-Hall, 1996. [27] C. L. Lawson and R. J. Hanson, Solving Least Squares Problems. Philadelphia, PA: SIAM, 1995. [28] B. Hassibi, A. H. Sayed, and T. Kailath, “ optimality criteria for LMS and backpropagation,” in Advances in Neural Information Processing Systems, J. D. Cowan, G. Tesauro, and J. Alspector, Eds. San Francisco, CA: Morgan-Kaufmann, Apr. 1994, vol. 6, pp. 351–359.
H
Mohit Kumar (A’05) received the B.Tech. degree in electrical engineering from the National Institute of Technology, Hamirpur, India, in 1999, the M.Tech. degree in control engineering from Indian Institute of Technology, Delhi, India, in 2001, and the Ph.D. degree in electrical engineering from Rostock University, Rostock, Germany, in 2004. He served as a Research Scientist in Institute of Occupational and Social Medicine, Rostock, Germany, from 2001 to 2004. Currently, he is with the Center for Life Science Automation, Rostock. His research interests include robust adaptive fuzzy identification, fuzzy logic in medicine, and robust adaptive control.
Regina Stoll received the diploma in medicine (Dip.-Med.), the Dr.Med. degree in occupational medicine, and the Dr.Med.Habil. degree in occupational and sports medicine from Rostock University, Rostock, Germany, in 1980, 1984, and 2002, respectively. She is head of the Institute of Occupational and Social Medicine, Rostock, Germany. She is a faculty member in the medicine faculty and faculty associate in the College of Computer Science and Electrical Engineering of Rostock University. She also holds the adjunct faculty member position in the industrial engineering department of North Carolina State University. Her research interests include occupational physiology, preventive medicine, and cardiopulmonary diagnostics.
Norbert Stoll received the diploma (Dip.-Ing.) in automation engineering in 1979, and the Ph.D. degree in measurement technology from Rostock University, Rostock, Germany, in 1985. He served as Head of Section Analytical Chemistry at the Academy of Sciences of GDR, Central Institute for Organic Chemistry till 1991. From 1992 to 1994, he was the Associate Director of Institute of Organic Catalysis, Rostock, Germany. Since 1994, he is a Professor of measurement technology in the Engineering Faculty of Rostock University. From 1994 to 2000, he directed the Institution of Automation in Rostock University. He is also holding, since 2003, the position of Vice President in Center for Life Science Automation, Rostock. His fields of interests include medical process measurement, lab automation, and smart systems and devices.
