Nondifferentiable optimization via approximation - MIT
Mathematical Programming Study 3 (1975) 1-25. North-Holland Publishing Company
NONDIFFERENTIABLE OPTIMIZATION VIA APPROXIMATION* Dimitri P. BERTSEKAS University of Illinois, Urbana, IlL, U.S.A. Received 8 November 1974 Revised manuscript received I 1 April 1975
This paper presents a systematic approach for minimization of a wide class of nondifferentiable functions. The technique is based on approximation of the nondifferentiable function by a smooth function and is related to penalty and multiplier methods for constrained minimization. Some convergence results are given and the method is illustrated by means of examples from nonlinear programming.
I. Introduction
Optimization problems with nondifferentiable cost functionals, particularly minimax problems, have received considerable attention recently since they arise naturally in a variety of contexts. Optimality conditions for such problems have been derived by several authors while a number of computational methods have been proposed for their solution (the reader is referred to [1] for:a fairly complete list of references up to 1973). Among the computational algorithms currently available are the subgradient methods of [10, 15, 19], the s-subgradient method [1, 2] coupled with an interesting implementation of the direction finding step given in [12], the minimax methods of [6, 7, 9, 17] which were among the first proposed in the nondifferentiable area, and the recent interesting methods proposed in [-5, 20]. While the advances in the area of computational algorithms have been significant, the methods mentioned above are by no means capable of handling all problems encountered in practice since they are often limited in their scope by assumptions such as convexity, cannot handle * This work was supported in part by the Joint Services Electronics Program (U.S. Army, U.S. Navy and U.S. Air Force) under Contract DAAB-07-72-C-0259, and in part by the U.S. Air Force under Grant AFOSR-73-2570.
D.P. Bertsekas / Nondifferentiable optimization via approximation
nonlinear constraints or they are applicable only to a special class of problems such as minimax problems of particular form. Furthermore several of these methods are similar in their behavior to either the method of steepest descent or first order methods of feasible directions and converge slowly when faced with relatively ill-conditioned problems. Thus there is considerable room for new methods and approaches for solution of nondifferentiable problems and the purpose of this paper is to provide a class of methods which is simple to implement, is quite broad in its scope, and relies on an entirely different philosophy than those underlying methods already available. We consider minimization problems of the form minimize g(x), subject to x ~ Q c R",
(1)
where g is a real-valued function on R" (n-dimensional Euclidean space). We consider the case where the objective function O is nondifferentiable exclusively due to the presence of several terms of the form V[fi(x)] = max {0, fi(x)},
i ~ I,
(2)
where {fi: ir I} is an arbitrary collection of real-valued functions on R". By this we mean that if the terms y(.) in the functional expression of g were replaced by some continuously differentiable functions ~(-) then the resulting function would also be everywhere continuously differentiable. For purposes of easy reference we shall call a term of the form (2), a simple kink. It should be emphasized that while we concentrate attention on simple kinks, the approach is quite general since we do not necessarily require that the functions f in (2) are differentiable but rather we allow them to contain in their functional expressions other simple kinks. In this way some other kinds of nondifferentiable terms such as for example terms of the form max {fl(x) .... ,f,,(x)}
(3)
can be expressed in terms of simple kinks by writing max {fl, ...,fro} = fl + Y[f2 - f , + Y[---7[fm-1 - fm-z + Y[f., -- f m - , ] ] . " ] ] '
(4)
Since there are no restrictions on the manner in which simple kinks enter in the functional expression of g, a little reflection should convince the
D.P. Bertsekas / Nondifferentiable optimization via approximation
reader that the class of nondifferentiable problems that we are considering is indeed quite broad. The basic idea of our approach for numerical solution of problems of the form (1) is to approximate every simple kink in the functional expression of 9 by a smooth function and solve the resulting differentiable problem by conventional methods. In this way an approximate solution of problem (1) will be obtained which hopefully converges to an exact solution as the approximation of the simple kinks becomes more and more accurate. While, as will be explained shortly, other approximation methods are possible, we shall concentrate on the following two-parameter approximation ~[f(x), y, c] of a simple kink 7If(x)],
( f ( x ) - (1 - y)2/2c ~[f(x), y, c] = _0 for a l l x ~ Q ;
yi = 1
/ff(x) > O,
(28)
/ff, f~) = 0.
Proposition 3.1 together with Proposition 2.1 and its corollaries may be used to provide a simple proof of an optimality condition for problem (16). We shall say that ~ is a local minimum for problem (16) if ~ is a minimizing point of g[x, y[fl(x)], ..., ~[fm(x)]] over a set of the form Q c~ {x: [x - ~] _< e}, where [.I denotes the Euclidean norm and e > 0 is some positive scalar. If~ is a unique minimizing point over Q n {x: Ix - ~] _< e}, we shall say that ~ is an isolated local minimum. Proposition 3.2. Let Q be a closed convex set and ~ be an isolated local minimum for problem (16). Then there exists a multiplier vector = (-ill,..., ym) satisfying
I
1
i=1 ~ y vJi Ix=" (x - x--) >_ 0 for all x e Q,
(29)
y' = 0
if f~(2) < O,
i = 1..... m,
(30)
yi= 1
/ff~(2)>O,
i = 1..... m,
(31)
0 < y' 0, i = 1,2 ..... m}, 1- = { i : f ~ ( 2 ) < 0 , i = 1 , 2 .... ,m}, I ~ = {i: f~(2) = 0, i = 1,2 .... ,m}. Assume that 5 > 0 is taken sufficiently small to guarantee that fi(x) > 0 ft(x) < 0
for all x ~ S(2; 5), for all x s S(2; 5),
i e 1 +, i e 1-.
Let us first consider the case where the objective function O in problem (16) has the particular form
g[x, 7[f,(x)],...,
7[fm(x)]] =
go(x) + i=1 ~ gi(x)7[fi(x)],
(35)
where g~: R" ~ R, i = 0 . . . . , m, are continuously differentiable functions. Now if we make the assumption gi(2) 4:0
for all i s I ~
(36)
14
D.P. Bertsekas
/ Nondifferentiable
optimization
via a p p r o x i m a t i o n
we have that, when g has the form (35), problem (16), locally within a neighborhood of ~, may be written as
min~go(x) + ~ g,(x) fi(x) ieI + ( + ,~,o+ ~ max[-O, g,(x) f(x)] + ,~,o ~ min[O, g,(x) f~(x)]} (37) where I ~247 = {i:
gi(~) >
O, f~(x) = 0},
I ~ = {i:
gi~)
< O, f(x) = 0}.
Since 2 is an isolated local minimum of the above problem, it follows under the mild assumption
V[gi(x) fi(x)]
x=x'
i ~ 10
are linearly independent vectors
(38)
that the set I ~ is empty and we have
gift)
> 0
for all i ~ I ~
(39)
Notice that the previous assumption (36) is implied by assumption (39). This fact can be verified by noting that 2 is an optimal solution of the problem min
~go(X)+ i~, gi(x) fi(x)+ i e~l o + z,+ i~, gi(x)fi(x)~ d + ~ "o -
g i ( x ) f t (x) k, where k is sufficiently large, are identical to the ones that would be generated by a method of multipliers for problem (41) for which: (a) Only the constraints g ~ ( x ) f ( x ) < zi, i e I ~ are eliminated by means of a generalized quadratic penalty. (b) The penalty parameter for the (k + 1)'hminimization corresponding to the ith constraint, i e I ~ depends continuously on x and is given by = ck/g
(x).
(c) The multiplier vector ~ at the beginning at the (k + 1)th minimization is equal to Yr. Alternatively, the vectors Xk, Yk for k > k, where k is sufficiently large, are identical to the ones that would be generated by the method of multipliers for problem (41) for which: (a) Both constraints g~(x) f ( x ) < z~, i e I ~ and 0 < z~, i e I ~ are eliminated by means of a generalized quadratic penalty.
D.P. Bertsekas / Nondifferentiable optimization via approximation
17
(b) The penalty parameter for the (k + 1)th minimization corresponding to the ith constraints depends continuously on x and is given by ~(x) =
2cdgi(x). (c) The multiplier vectors ~ , ~ at the beginning of the (k + 1)thminimization (where ~ corresponds to the constraints #~(x) f~(x) < zi, i ~ I ~ and ~ corresponds to the constraints 0 < z~, i t I ~ satisfy ~ = Yk-and ~ = 1 -- y~.
The equivalence described above may be seen by verifying the following relations which hold for all scalars y ~ [0, 1], c > 0, g > 0, f . ~ ( f , y, c) = min[z + (g/2c) {[max(0,y + (c/g)(gf - z))] 2 - y2}] 0 0, {y~}, {W~,} satisfy 0 n,
a,(m,m) -- 2 Isin (i)1 i/m + Y~ la,(m,j)l, j~p rn
bi(m)
=
e "/i
s i n ( / " m),
i =
We represented max{f1, f2 ..... fs} by
1.....
5,
m =
1, ..., 10.
D.P. Bertsekas / Nondifferentiable
optimization
0
.=.~. I I I I I I I I
~
~ t r l
I I I I I 1 ~ 1 r
r162
N
I
.~
I
I
I r
I
I
,...~ t'q
0
tr
tr
I
~'%
t
I
.8 t"q ("4
II
.=.
I
t,r
II
I
I
tr
I
I
it-i
I
I
O 0
0,.,.
[.-o
via approximation
23
24
D.P. Bertsekas / Nondifferentiable optimization via approximation
m a x { A , . . . , fs} = A + 7[f2 - A
+ ~:[f3 - A + ~'[A - f3 + ~:[f5 - A ] ] ] ]
and used our approximation procedure in conjunction with iteration (34). The starting points were x ~ = x z = ..- = x 1~ = 0 and y01 = y~ = yo3 = yo4 = 0. The optimal value obtained is -0.51800 and the corresponding multiplier vector was = (1.00000, 0.99550, 0.89262, 0.58783). It is worth noting that for minimax problems of this type the optimal values of the approximate objective obtained during the computation constitute useful lower bounds for the optimal value of the problem. Table 3 shows the results of the computation for the case where unconstrained minimization was "exact" (i.e., e = 10- 5 in the D F P routine% It also shows the results of the computation when the unconstrained minimization was inexact in the sense that the k th minimization was terminated when t h e / t - n o r m of the direction vector in the D F P was less than max[lO -s, lO-k].
References [17 D.P. Bertsekas and S.K. Mitter, "A descent numerical method for optimization problems with nondifferentiable cost functionals", SlAM Journal on Control 11 (1973) 637-652. [2] D.P. Bertsekas and S.K. Mitter, "Steepest descent for optimization problems with nondifferentiable cost functionals", Proceedinos of the 5th annual Princeton conference on information sciences and systems, Princeton, N.J., March 1971, pp. 347-351. [3] D.P. Bertsekas, "Combined primal-dual and penalty methods for constrained minimization", SIAM Journal on Control to appear. [4] D.P. Bertsekas, "On penalty and multiplier methods for constrained minimization", SIAM Journal on Control 13 (1975) 521-544. [5] J. Cullum, W.E. Donath and P. Wolfe, "An algorithm for minimizing certain nondifferentiable convex functions", RC 4611, IBM Research, Yorktown Heights, N.Y. (November 1973). [6] V.F. Demyanov, "On the solution of certain minimax problems", Kibernetica 2 (1966). [7] V.F. Demyanov and A.M. Rubinov, Approximate methods in optimization problems (American Elsevier, New York, 1970). [8] A.V. Fiacco and G.P. McCormick, Nonlinear programming : sequential unconstrained minimization techniques (Wiley, New York, 1968). [9] A.A. Goldstein, Constructive real analysis (Harper & Row, New York, 1967). [10] M. Held, P. Wolfe and H.P. Crowder, "Validation of subgradient optimization", Mathematical Programmin0 6 (1) (1974) 62-88. [11] M.R. Hestenes, "Multiplier and gradient methods", Journal of Optimization Theory and Applications 4 (5) (1969) 303-320. [12] C. Lemarechal, "An algorithm for minimizing convex functions", in : J.L. Rosenfeld, ed., Proceedings of the IFIP Conoress 74 (North-Holland, Amsterdam, 1974) pp. 552-556.
D.P. Bertsekas / Nondifferentiable optimization via approximation
25
[13] B.W. K o n and D.P. Bertsekas, "Combined primal-dual and penalty methods for convex programming", SIAM Journal on Control, to appear. [14] B.W. Kort and D.P. Bertsekas, "Multiplier methods for convex programming", Proceedings of 1973 IEEE conference on decision and control, San Diego, Calif., December 1973, pp. 428-432. [15] B.T. Polyak, "Minimization of unsmooth functionals", ~,urnal Vy~islitel'noi Matematiki i Matemati~eskoi Fiziki 9 (3) (1969) 509-521. [16] M.J.D. Powell, "A method for nonlinear constraints in minimization problems", in : R. Fletcher, ed., Optimization (Academic Press, New York, 1969) pp. 283-298 [17] B.N. Pschenishnyi, "Dual methods in extremum problems", Kibernetica I (3) (1965) 89-95. [18] R.T. Rockafr "The multiplier method of Hestenes and Powell applied to convex programming", Journal of Optimization Theory and Applications 12 (6) (1973). [19] N.Z. Shot, "On the structure of algorithms for the numerical solution of planning and design problems", Dissertation, Kiev (1964). ['20] P. Wolfe, "A method of conjugate subgradients for minimizing nondifferentiable functions", Mathematical Programming Study 3 (1975) 145-173 (this volume).
NONDIFFERENTIABLE OPTIMIZATION VIA APPROXIMATION* Dimitri P. BERTSEKAS University of Illinois, Urbana, IlL, U.S.A. Received 8 November 1974 Revised manuscript received I 1 April 1975
This paper presents a systematic approach for minimization of a wide class of nondifferentiable functions. The technique is based on approximation of the nondifferentiable function by a smooth function and is related to penalty and multiplier methods for constrained minimization. Some convergence results are given and the method is illustrated by means of examples from nonlinear programming.
I. Introduction
Optimization problems with nondifferentiable cost functionals, particularly minimax problems, have received considerable attention recently since they arise naturally in a variety of contexts. Optimality conditions for such problems have been derived by several authors while a number of computational methods have been proposed for their solution (the reader is referred to [1] for:a fairly complete list of references up to 1973). Among the computational algorithms currently available are the subgradient methods of [10, 15, 19], the s-subgradient method [1, 2] coupled with an interesting implementation of the direction finding step given in [12], the minimax methods of [6, 7, 9, 17] which were among the first proposed in the nondifferentiable area, and the recent interesting methods proposed in [-5, 20]. While the advances in the area of computational algorithms have been significant, the methods mentioned above are by no means capable of handling all problems encountered in practice since they are often limited in their scope by assumptions such as convexity, cannot handle * This work was supported in part by the Joint Services Electronics Program (U.S. Army, U.S. Navy and U.S. Air Force) under Contract DAAB-07-72-C-0259, and in part by the U.S. Air Force under Grant AFOSR-73-2570.
D.P. Bertsekas / Nondifferentiable optimization via approximation
nonlinear constraints or they are applicable only to a special class of problems such as minimax problems of particular form. Furthermore several of these methods are similar in their behavior to either the method of steepest descent or first order methods of feasible directions and converge slowly when faced with relatively ill-conditioned problems. Thus there is considerable room for new methods and approaches for solution of nondifferentiable problems and the purpose of this paper is to provide a class of methods which is simple to implement, is quite broad in its scope, and relies on an entirely different philosophy than those underlying methods already available. We consider minimization problems of the form minimize g(x), subject to x ~ Q c R",
(1)
where g is a real-valued function on R" (n-dimensional Euclidean space). We consider the case where the objective function O is nondifferentiable exclusively due to the presence of several terms of the form V[fi(x)] = max {0, fi(x)},
i ~ I,
(2)
where {fi: ir I} is an arbitrary collection of real-valued functions on R". By this we mean that if the terms y(.) in the functional expression of g were replaced by some continuously differentiable functions ~(-) then the resulting function would also be everywhere continuously differentiable. For purposes of easy reference we shall call a term of the form (2), a simple kink. It should be emphasized that while we concentrate attention on simple kinks, the approach is quite general since we do not necessarily require that the functions f in (2) are differentiable but rather we allow them to contain in their functional expressions other simple kinks. In this way some other kinds of nondifferentiable terms such as for example terms of the form max {fl(x) .... ,f,,(x)}
(3)
can be expressed in terms of simple kinks by writing max {fl, ...,fro} = fl + Y[f2 - f , + Y[---7[fm-1 - fm-z + Y[f., -- f m - , ] ] . " ] ] '
(4)
Since there are no restrictions on the manner in which simple kinks enter in the functional expression of g, a little reflection should convince the
D.P. Bertsekas / Nondifferentiable optimization via approximation
reader that the class of nondifferentiable problems that we are considering is indeed quite broad. The basic idea of our approach for numerical solution of problems of the form (1) is to approximate every simple kink in the functional expression of 9 by a smooth function and solve the resulting differentiable problem by conventional methods. In this way an approximate solution of problem (1) will be obtained which hopefully converges to an exact solution as the approximation of the simple kinks becomes more and more accurate. While, as will be explained shortly, other approximation methods are possible, we shall concentrate on the following two-parameter approximation ~[f(x), y, c] of a simple kink 7If(x)],
( f ( x ) - (1 - y)2/2c ~[f(x), y, c] = _0 for a l l x ~ Q ;
yi = 1
/ff(x) > O,
(28)
/ff, f~) = 0.
Proposition 3.1 together with Proposition 2.1 and its corollaries may be used to provide a simple proof of an optimality condition for problem (16). We shall say that ~ is a local minimum for problem (16) if ~ is a minimizing point of g[x, y[fl(x)], ..., ~[fm(x)]] over a set of the form Q c~ {x: [x - ~] _< e}, where [.I denotes the Euclidean norm and e > 0 is some positive scalar. If~ is a unique minimizing point over Q n {x: Ix - ~] _< e}, we shall say that ~ is an isolated local minimum. Proposition 3.2. Let Q be a closed convex set and ~ be an isolated local minimum for problem (16). Then there exists a multiplier vector = (-ill,..., ym) satisfying
I
1
i=1 ~ y vJi Ix=" (x - x--) >_ 0 for all x e Q,
(29)
y' = 0
if f~(2) < O,
i = 1..... m,
(30)
yi= 1
/ff~(2)>O,
i = 1..... m,
(31)
0 < y' 0, i = 1,2 ..... m}, 1- = { i : f ~ ( 2 ) < 0 , i = 1 , 2 .... ,m}, I ~ = {i: f~(2) = 0, i = 1,2 .... ,m}. Assume that 5 > 0 is taken sufficiently small to guarantee that fi(x) > 0 ft(x) < 0
for all x ~ S(2; 5), for all x s S(2; 5),
i e 1 +, i e 1-.
Let us first consider the case where the objective function O in problem (16) has the particular form
g[x, 7[f,(x)],...,
7[fm(x)]] =
go(x) + i=1 ~ gi(x)7[fi(x)],
(35)
where g~: R" ~ R, i = 0 . . . . , m, are continuously differentiable functions. Now if we make the assumption gi(2) 4:0
for all i s I ~
(36)
14
D.P. Bertsekas
/ Nondifferentiable
optimization
via a p p r o x i m a t i o n
we have that, when g has the form (35), problem (16), locally within a neighborhood of ~, may be written as
min~go(x) + ~ g,(x) fi(x) ieI + ( + ,~,o+ ~ max[-O, g,(x) f(x)] + ,~,o ~ min[O, g,(x) f~(x)]} (37) where I ~247 = {i:
gi(~) >
O, f~(x) = 0},
I ~ = {i:
gi~)
< O, f(x) = 0}.
Since 2 is an isolated local minimum of the above problem, it follows under the mild assumption
V[gi(x) fi(x)]
x=x'
i ~ 10
are linearly independent vectors
(38)
that the set I ~ is empty and we have
gift)
> 0
for all i ~ I ~
(39)
Notice that the previous assumption (36) is implied by assumption (39). This fact can be verified by noting that 2 is an optimal solution of the problem min
~go(X)+ i~, gi(x) fi(x)+ i e~l o + z,+ i~, gi(x)fi(x)~ d + ~ "o -
g i ( x ) f t (x) k, where k is sufficiently large, are identical to the ones that would be generated by a method of multipliers for problem (41) for which: (a) Only the constraints g ~ ( x ) f ( x ) < zi, i e I ~ are eliminated by means of a generalized quadratic penalty. (b) The penalty parameter for the (k + 1)'hminimization corresponding to the ith constraint, i e I ~ depends continuously on x and is given by = ck/g
(x).
(c) The multiplier vector ~ at the beginning at the (k + 1)th minimization is equal to Yr. Alternatively, the vectors Xk, Yk for k > k, where k is sufficiently large, are identical to the ones that would be generated by the method of multipliers for problem (41) for which: (a) Both constraints g~(x) f ( x ) < z~, i e I ~ and 0 < z~, i e I ~ are eliminated by means of a generalized quadratic penalty.
D.P. Bertsekas / Nondifferentiable optimization via approximation
17
(b) The penalty parameter for the (k + 1)th minimization corresponding to the ith constraints depends continuously on x and is given by ~(x) =
2cdgi(x). (c) The multiplier vectors ~ , ~ at the beginning of the (k + 1)thminimization (where ~ corresponds to the constraints #~(x) f~(x) < zi, i ~ I ~ and ~ corresponds to the constraints 0 < z~, i t I ~ satisfy ~ = Yk-and ~ = 1 -- y~.
The equivalence described above may be seen by verifying the following relations which hold for all scalars y ~ [0, 1], c > 0, g > 0, f . ~ ( f , y, c) = min[z + (g/2c) {[max(0,y + (c/g)(gf - z))] 2 - y2}] 0 0, {y~}, {W~,} satisfy 0 n,
a,(m,m) -- 2 Isin (i)1 i/m + Y~ la,(m,j)l, j~p rn
bi(m)
=
e "/i
s i n ( / " m),
i =
We represented max{f1, f2 ..... fs} by
1.....
5,
m =
1, ..., 10.
D.P. Bertsekas / Nondifferentiable
optimization
0
.=.~. I I I I I I I I
~
~ t r l
I I I I I 1 ~ 1 r
r162
N
I
.~
I
I
I r
I
I
,...~ t'q
0
tr
tr
I
~'%
t
I
.8 t"q ("4
II
.=.
I
t,r
II
I
I
tr
I
I
it-i
I
I
O 0
0,.,.
[.-o
via approximation
23
24
D.P. Bertsekas / Nondifferentiable optimization via approximation
m a x { A , . . . , fs} = A + 7[f2 - A
+ ~:[f3 - A + ~'[A - f3 + ~:[f5 - A ] ] ] ]
and used our approximation procedure in conjunction with iteration (34). The starting points were x ~ = x z = ..- = x 1~ = 0 and y01 = y~ = yo3 = yo4 = 0. The optimal value obtained is -0.51800 and the corresponding multiplier vector was = (1.00000, 0.99550, 0.89262, 0.58783). It is worth noting that for minimax problems of this type the optimal values of the approximate objective obtained during the computation constitute useful lower bounds for the optimal value of the problem. Table 3 shows the results of the computation for the case where unconstrained minimization was "exact" (i.e., e = 10- 5 in the D F P routine% It also shows the results of the computation when the unconstrained minimization was inexact in the sense that the k th minimization was terminated when t h e / t - n o r m of the direction vector in the D F P was less than max[lO -s, lO-k].
References [17 D.P. Bertsekas and S.K. Mitter, "A descent numerical method for optimization problems with nondifferentiable cost functionals", SlAM Journal on Control 11 (1973) 637-652. [2] D.P. Bertsekas and S.K. Mitter, "Steepest descent for optimization problems with nondifferentiable cost functionals", Proceedinos of the 5th annual Princeton conference on information sciences and systems, Princeton, N.J., March 1971, pp. 347-351. [3] D.P. Bertsekas, "Combined primal-dual and penalty methods for constrained minimization", SIAM Journal on Control to appear. [4] D.P. Bertsekas, "On penalty and multiplier methods for constrained minimization", SIAM Journal on Control 13 (1975) 521-544. [5] J. Cullum, W.E. Donath and P. Wolfe, "An algorithm for minimizing certain nondifferentiable convex functions", RC 4611, IBM Research, Yorktown Heights, N.Y. (November 1973). [6] V.F. Demyanov, "On the solution of certain minimax problems", Kibernetica 2 (1966). [7] V.F. Demyanov and A.M. Rubinov, Approximate methods in optimization problems (American Elsevier, New York, 1970). [8] A.V. Fiacco and G.P. McCormick, Nonlinear programming : sequential unconstrained minimization techniques (Wiley, New York, 1968). [9] A.A. Goldstein, Constructive real analysis (Harper & Row, New York, 1967). [10] M. Held, P. Wolfe and H.P. Crowder, "Validation of subgradient optimization", Mathematical Programmin0 6 (1) (1974) 62-88. [11] M.R. Hestenes, "Multiplier and gradient methods", Journal of Optimization Theory and Applications 4 (5) (1969) 303-320. [12] C. Lemarechal, "An algorithm for minimizing convex functions", in : J.L. Rosenfeld, ed., Proceedings of the IFIP Conoress 74 (North-Holland, Amsterdam, 1974) pp. 552-556.
D.P. Bertsekas / Nondifferentiable optimization via approximation
25
[13] B.W. K o n and D.P. Bertsekas, "Combined primal-dual and penalty methods for convex programming", SIAM Journal on Control, to appear. [14] B.W. Kort and D.P. Bertsekas, "Multiplier methods for convex programming", Proceedings of 1973 IEEE conference on decision and control, San Diego, Calif., December 1973, pp. 428-432. [15] B.T. Polyak, "Minimization of unsmooth functionals", ~,urnal Vy~islitel'noi Matematiki i Matemati~eskoi Fiziki 9 (3) (1969) 509-521. [16] M.J.D. Powell, "A method for nonlinear constraints in minimization problems", in : R. Fletcher, ed., Optimization (Academic Press, New York, 1969) pp. 283-298 [17] B.N. Pschenishnyi, "Dual methods in extremum problems", Kibernetica I (3) (1965) 89-95. [18] R.T. Rockafr "The multiplier method of Hestenes and Powell applied to convex programming", Journal of Optimization Theory and Applications 12 (6) (1973). [19] N.Z. Shot, "On the structure of algorithms for the numerical solution of planning and design problems", Dissertation, Kiev (1964). ['20] P. Wolfe, "A method of conjugate subgradients for minimizing nondifferentiable functions", Mathematical Programming Study 3 (1975) 145-173 (this volume).
