the construction of huffman codes is a submodular ... - CiteSeerX
THE CONSTRUCTION OF HUFFMAN CODES IS A SUBMODULAR (`CONVEX') OPTIMIZATION PROBLEM OVER A LATTICE OF BINARY TREES D. STOTT PARKER AND PRASAD RAMy
Abstract. We show that the space of all binary Hu man codes for a nite alphabet de nes a lattice, ordered by the imbalance of the code trees. Representing code trees as path-length sequences, we show that the imbalance ordering is closely related to a majorization ordering on real-valued sequences that correspond to discrete probability density functions. Furthermore, this tree imbalance is a partial ordering that is consistent with the total orderings given by either the external path length (sum of tree path lengths), or the entropy determinedby the tree structure. On the imbalance lattice, we show the weighted path-length of a tree (the usual objective function for Hu man coding) is a submodular function, as is the corresponding function on the majorization lattice. Submodular functions are discrete analogues of convex functions. These results give perspective on Hu man coding and suggest new approaches to coding as optimization over a lattice. Key words. Hu man coding, adaptive coding, pre x codes, enumeration of trees, lattices, combinatorial optimization, convexity, submodular functions, entropy, tree imbalance, Schur convex functions, majorization, Moebius inversion, combinatorial inequalities, Fortuin-Kasteleyn-Ginibre (FKG) inequality, quadrangle inequality, Monge matrices, dynamic programming, greedy algorithms. AMS subject classi cations. 94A15, 94A24, 94A29, 94A45, 90C25, 90C27, 90C39, 90C48, 52A41, 68Q20, 68R05, 05A05, 05A20, 05C05, 05C30, 06A07, 26B25, 26D15
1. Introduction. The Hu man algorithm has been used heavily to produce ecient binary codes for almost half a century now. It has inspired a large literature with diverse theoretical and practical contributions. A comprehensive, very recent survey is [1]. Although the algorithm is quite elegant, it is tricky to prove correct and to reason about. While there may be little hope of improving on the O(n log n) complexity of the Hu man algorithm itself,1 there is still room for improvement in our understanding of the algorithm. There is also plenty of room for improvement in our understanding of variants of Hu man coding. Although the Hu man algorithm is remarkably robust in general and has widespread use, it is far from optimal in many real applications. Hu man coding is optimal only when the symbols to be coded are random and occur with xed probabilities. Timevarying dependencies are not captured by the Hu man coding model, and optimal encoding of nite messages is not captured either. Our motivation came from analysis of dynamic Hu man coding, a speci c extension of Hu man coding in which the code used evolves over time. Recently dynamic coding algorithms have been studied heavily. Our initial idea was to de ne \rebalancing" operations on code trees and to use these dynamically (\on the y") in producing better codes, in situations where the distribution of symbols to be coded varies over time and/or is not accurately predictable in advance. This paper reconstructs Hu man coding as an optimization over the space of binary trees. A natural representation for this space is sequences of ascending path-lengths, since this captures what is signi cant in producing optimal codes. We show that the set of path-length sequences representing binary trees forms a lattice, which we call the imbalance lattice. This lattice orders trees by their imbalance and gives an organization for them that is useful in optimization. Our belief is that having a better UCLA Computer Science Department, University of California, Los Angeles, CA 90095-1596 ([email protected]). y Xerox Corporation, El Segundo, CA 90245 ([email protected]). 1 The algorithm is closely related to sorting, in the sense that the sorted sequence of a sequence of integer values h x1 xn i is obtainable directly from the optimal code tree for the values x x h 2 1 2 n i (e.g., [26, p.335]). 1
D.S. PARKER AND P. RAM
2
mathematical (and not purely procedural) understanding of coding will ultimately pay o in improved algorithms. The imbalance lattice and its imbalance ordering on trees depend on majorization in an essential way. Majorization is an important ordering on sequences that has many applications in pure and applied mathematics [27]. We have related it to greedy algorithms directly [33]. Earlier majorization was recognized as an important property of the internal node weights produced by the Hu man algorithm [13, 32], and in this work we go further to clarify its pervasive role. By viewing the space of trees as a lattice, a variety of new theorems and algorithms become possible. For example, the objective functions commonly used in evaluating codes are submodular on this lattice. Submodular functions are closely related to convex functions (as we explain later; see Theorem 4.5) and are often easy to optimize [6, 9, 23, 24, 25]. Hu man coding gives a signi cant example of the importance of submodularity in algorithms.
2. Ordered Sequences, Rooted Binary Trees, and Hu man Codes. 2.1. Ordered Sequences. By a sequence we mean an ordered collection of non-
negative real values
x = h x x xn i: 1
2
Repetition of values in the sequence is permitted: the values xj need not be distinct. The length of this sequence is nn, and for simplicity we also refer to the set of such sequences with the vector notation =R h 1 0 0 0 i: Proof. The transformation x 7! x de nes a lattice isomorphism between the vector and majorization lattices, and between the distribution and density lattices. Here x u y and x t y are de ned just so as to be the majorization glb and lub:
z xR ; z y R R R , (R z) vec ( Rx); ( z) vec ( y) R , ( z) vecR ( ( x) minvec R ( y) ) , z @ ( ( x) minvec ( y) ) , z x u y:
x Rz; y z R R R , ( Rx) vec ( zR); ( y) vecR ( z) , ( ( xR) maxvec ( yR) ) vec ( z) , @ ( ( x) maxvec ( y) ) z , x t y z:
Thus the majorization algebra also forms a distributive lattice. Even when x and y are in descending order, the sequences (x u y) and (x t y) de ned here are not necessarily in descending order: x = h 2?2 2?2 2?3 2?4 2?4 2?4 2?4 2?4 2?4 i and y = h 2?2 2?3 2?3 2?3 2?3 2?3 2?4 2?5 2?5 i yield the least upper bound x t y = R R @ ( x maxvec y) = h 2?2 2?2 2?3 2?4 2?4 2?3 2?4 2?5 2?5 i: See Figure 3.6.
D.S. PARKER AND P. RAM
12
ss = = s = t= R 22??tt == R 2?s maxvec R 2?t = R R @ ( 2?s maxvec 2?t ) = = 2?s t 2?t = R 2?s min Rs 2_?tt == vecR R @ ( 2?s minvec 2?t ) = = 2?s u 2?t = s^t =
R 22??
( ( ( ( ( ( ( ( ( ( ( ( ( (
2?7 2?7 2?7 2?7 2?7 2?7
1?1 2 64 2?2 2 32 64 64 2?1 1 32 32 2?2 2
2?2 2 96 2?2 2 64 96 32 2?2 2 64 32 2?2 2
4?4 2 104 2?2 2 96 104 8 2?4 4 96 32 2?2 2
4?4 2 112 3?3 2 112 112 8 2?4 4 112 16 2?3 3
5?5 2 116 4?4 2 120 120 8 2?4 4 116 4 2?5 5
5?5 2 120 5?5 2 124 124 4 2?5 5 120 4 2?5 5
5?5 2 124 6?6 2 126 126 2 2?6 6 124 4 2?5 5
6?6 2 126 7?7 2 127 127 1 2?7 7 126 2 2?6 6
6?6 2 128 7?7 2 128 128 1 2?7 7 128 2 2?6 6
) ) ) ) ) ) ) ) ) ) ) ) ) )
Fig. 3.5. Related points in the majorization and imbalance lattices, showing their connection
s = = s = t = R 22??tt == R 2?s maxvec R 2?t = R R @ ( 2?s maxvec 2?t ) = = 2?s t 2?t = s_t =
R 22??
s
2?7 2?7 2?7 2?7
( ( ( ( ( ( ( ( ( (
2?2 2 32 2?2 2 32 32 32 2?2 2
2?2 2 64 3?3 2 48 64 32 2?2 2
3?3 2 80 3?3 2 64 80 16 2?3 3
4?4 2 88 3?3 2 80 88 8 2?4 3
4?4 2 96 3?3 2 96 96 8 2?4 4
4?4 2 104 3?3 2 112 112 16 2?3 4
4?4 2 112 4?4 2 120 120 8 2?4 4
4?4 2 120 5?5 2 124 124 4 2?5 5
4?4 2 128 5?5 2 128 128 4 2?5 5
) ) ) ) ) ) ) ) ) )
Fig. 3.6. Results of (2?s u 2?t) and (2?s t 2?t) are not necessarily in descending order
s= = s = t= R 22??tt == R 2?s min R 2?t = vecR R ? s @ ( 2 minvec 2?t ) = = 2?s u 2?t =
R 22??
s
s^t =
2?7 2?7 2?7 2?7
( ( ( ( ( ( ( ( (
1?1 2 64 2?2 2 32 32 32 2?2
2?2 2 96 2?2 2 64 64 32 2?2
4?4 2 104 2?2 2 96 96 32 2?2
( 2?2 2?2 2?2 ( 2
2
2
5?5 2 108 3?3 2 112 108 12 2? ? = 2?4 4
5?5 5?5 5?5 5?5 5?5 2 2 2 2 2 112 116 120 124 128 4?4 5?5 6?6 7?7 7?7 2 2 2 2 2 120 124 126 127 128 112 116 120 124 128 4 4 4 4 4 2?5 2?5 2?5 2?5 2?5 (?7 + log2 (12)) ?3:4150375 2?4 2?5 2?5 2?5 2?5 4
5
5
5
5
2? (s ^ t) (2?s u 2?t ); the two di er where indicated. Non-integral exponents occur for n 9. Fig. 3.7. The imbalance lattice is not simply conjugate to a sublattice of the majorization lattice
) ) ) ) ) ) ) ) ) ) )
HUFFMAN CODING AS SUBMODULAR OPTIMIZATION OVER A LATTICE
13
3.6. The Imbalance Lattice: A Discrete Cousin of the Majorization Lattice. Since every pair of sequences in Figures 3.2 and 3.3 has a unique glb and lub, the imbalance ordering is not only a partial order, but also a lattice. In this section we prove this by showing that every pair of sequences s, t has a glb s ^ t and lub s _ t. We also relate the imbalance lattice directly to the majorization lattice, as illustrated in Figures 3.5|3.7. Theorem 3.17. On tree path-length sequences, the imbalance ordering is isomorphic to the majorization ordering. Speci cally, whenever s and t are tree path-length sequences, then 2?s 2?t : Proof. We show rst that balancing exchanges cause a reduction in the majorization ordering. Let s be the result of a balancing exchange on t (so s t). Then:
s t
R R
t = s =
( (
2 t = 2 s =
( (
? ? 2?t 2?s
R
=
(
=
(
p (p + 1) 2?p 2?(p+1) S S +2?p S S +2?(p+1) 0 + 2?(p+1)
( 0
R
i
2?u 2?(p+1) 2?u S +2?p +2?u S +2?p + 2?u u p
( + 1)
u
v
2?v T T ? 2?v + 2?v
(q + 1) v 2?(q +1) 2?v
(q + 1) q 2?(q +1) 2?q
T +2?(q+1) T + 2?(q +1)
T +2?q T +2?q 0
) ) ) )
1 ) 1 ) 0 )
Thus 2?s and 2?t di er only in the values appearing between p and q, and each element R R ? t ? s in 2 ? 2 is nonnegative, so 2?s 2?t . The proof of the converse, that 2?s 2?t implies s t for tree path-length sequences s, t, can proceed by assuming a counterexample for which the di erence in the levels of balance
m = (level of balance of s) ? (level of balance of t): is minimal. Since 2?s 2?t , let a, b, c, d be the rightmost aligned pairwise-di ering values among the two sorted sequences such that s = h a b i and t = h c d i, where c < a, b < d because of the majorization inequality, a b and c d because the sequences are ascending, c 6= d since c < a b < d, and nally 2?a + + 2?b = 2?c + + 2?d ; which is always possible by the Kraft equality. Because b < d necessarily t = h c d d i, since otherwise we reach a contradiction (multiplying both sides of the equality by 2d makes the left side even but the right side odd). Thus, if we de ne the result t0 = h (c + 1) (c + 1) (d ? 1) i of a balancing exchange on t =0 h c d d i, then the level di erence between s and t0 is at most (m ? 1), and 2?t 2?t . Furthermore ? s ? t0 we claim 2 2 , using the following schematic: t = t = s =
( ( (
2 t = 2 t = 2 s =
( (
0
? ?0 ? 2?t ? 2?s 0 2?t ? 2?t 0 ? t ? s 2 ? 2
R R R
R R R
(
=
( 0
=
( 0
=
( 0
c u (c + 1) (c + 1) a w 2?c 2?u 2??a(c+1) 2?(c+1) 2 2?w 0 +(2?c ? 2?a ) +S1 0 +2?(c+1) +2?u 0 +(2?(c+1) ? 2?a ) +(S1 ? 2?u )
d
2
d d ? 1) b 2?d 2?(d?1) 2?b
(
?d Sk
+
+2
?d
Sk ? 2?d )
+(
0 0 0
) ) ) ) ) )
0 ) 0 ) 0 )
Because 2?s 2?t , the running totals S1 ; : : : ; Sk are nonnegative. Also, (2?(c+1) ? 2?a ) 0 since c < a. Furthermore c (a ? 1) w, implying S1 ? 2?u = (2?c ? 2?a) ? 2w 2?(a?1) ? 2?w 0. Finally Sk +2?d ?R2?b = 0, soR b 0 (d ? 1) implies Sk0? 2?d = (2?b ? 2?d ) ? 2?d = 2?b ? 2?(d?1) 0: Thus 2?s vec 2?t (i.e., 2?s 2?t ), contradicting the assumed minimality of m, and existence of a counterexample.
D.S. PARKER AND P. RAM
14
Theorem 3.18. The imbalance ordering on binary trees determines a bona- de lattice in which, for all s and t, the glb s ^ t and lub s _ t are de ned with the following recursive algorithms, where the expansion used is chosen from among the lower and upper expansions: 8 s if s t > < t if ts s ^ t = > the greatest expansion of s? ^ t? : that is also a lower bound for s and t otherwise;
s_t =
8 t > < s the least expansion of s? _ t? > : that is also an upper bound for s and t
if if
st ts
otherwise:
Proof. We must show that, whenever s and t are tree path-length sequences of length
n, there are unique path-length sequences s ^ t and s _ t such that: s ^ t s; t; also if ` is any path-length sequence, then ` s; t i ` s ^ t: s; t s _ t; also if ` is any path-length sequence, then s; t ` i s _ t `: This can be done by induction on n. We consider only the glb here, the proof for the lub being similar. The theorem holds trivially for n 6, since then the trees are totally ordered. Assume it holds for sequences of size n ? 1 or less.
First, s and t must have a common lower bound: The glb s? ^ t? exists by induction, and (Theorems 3.9 and 3.10) lower expansion gives a lower bound (s? ^ t? )+ (s? )+ s; (s? ^ t? )+ (t? )+ t: Second, if s and t have two greatest lower bounds ` and `0 , then they must be equal: From ` s; t and `0 s; t we infer `? s? ^ t? and `0? s? ^ t? by Theorem 3.10. Since furthermore ` and `0 are greatest lower bounds, s? ^ t? `? and s? ^ t? `0? : Thus `? = `0? : By Theorem 3.9, the only way ` 6= `0 can arise is that ` = (`? )+ ; `0 = (`? )+ or ` = (`? )+ ; `0 = (`? )+ so ` `0 or `0 `, contradicting their both being greatest lower bounds. Thus ` = `0 . Third, the algorithm produces a glb that is as good as any other lower bound: Assuming this for (s? ^ t? ) by induction, there can be no lower bound ` 6= (s ^ t) such that (s? ^ t? )+ `; since otherwise (s? ^ t? ) `? s? ; t? contradicting our assumption. The table of nontrivial examples in Figure 3.3 gives an appreciation for glbs and lubs. The rst example (which is illustrated in the gure) is expanded in Table 3.2. Note the nal pairs of entries in s and t are the same as the nal pairs of entries in s ^ t and s _ t, and the sux lengths of s and t are never shorter than those of s ^ t and s _ t. Theorem 3.19. If s and t are path-length sequences of length n, then:
(s? _ t? ) ? ? ((ss? _^ tt? ))
s = (s?? ) and t = (t?? ) s = (s ) and t = (t ) s = (s?? ) and t = (t?? ) s^t = (s? ^ t? ) s = (s ) and t = (t ) Otherwise, if either s = (s? ) and t = (t? ) , or s = (s? ) and t = (t? ) , then either s ^ t = (s? ^ t? ) and s _ t = (s? _ t? ) ; or s ^ t = (s? ^ t? ) and s _ t = (s? _ t? ) : Furthermore: if the nal pairs of entries of s and t are hp pi and hq qi, where p q, then the nal pairs of entries of s ^ t and s _ t are respectively hp pi and hq qi. Also: the sux lengths of s and t are at least as long as those of (s ^ t) and (s _ t). s_t =
if if if if
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
HUFFMAN CODING AS SUBMODULAR OPTIMIZATION OVER A LATTICE
s
n
9 8 7 6
h
144444444
t
i
222355566
h
" lower expansion h
13444444
i
22235555
h
1334444
h
133344
223444444
i
h
2223455
i
22334444
h
h
222344
h
133355566
i
h
2224444
i
h
222344
h
13335555
i
" lower expansion
i
1333455
h
" lower expansion
i
i
" upper expansion
" lower expansion
" upper expansion
i
_
i
" lower expansion
" lower expansion
" lower expansion
s t
^
h
" upper expansion
" lower expansion h
s t
i
15
i
" upper expansion
i
h
133344
i
Table 3.2
Elaboration of the rst example of representative bounds in Figure 3.3, showing how s ^ t and s _ t can be derived with their recursive algorithms.
Proof. These properties follow by induction on n. For the basis, they all hold trivially when n 6, since then the imbalance lattice is a total order and f s; t g = f s _ t; s ^ t g; and the nal two entries of any path-length sequence are a pair by Theorem 3.1. For the induction step, we can write
s = h (p ? i) pz s? = h (p ? i) (p ? 1)
}|h p{ i zp h}|? p{ i
k
t = h (q ? j ) zq }| q{ i k? z }| q{ i ? t = h (q ? j ) (q ? 1) q where i; j;h; k > 0, and we assume with no loss of generality that p q. There are four cases to consider, depending on the the sux lengths 2h of s and 2k of t. In the rst, h = 1 and k = 1, i.e., s = (s? ) and t = (t? ) . Then i = 1 and j = 1 by Theorem 3.1. By induction (s? ^ t? ) and (s? _ t? ) have respective nal pairs h(p ? 1) (p ? 1)i and h(q ? 1) (q ? 1)i, and have sux lengths not exceeding those of s? and t? . Now, by Theorem 3.10 (s? ^ t? ) (s? ) and (s? ^ t? ) (t? ) : Because (s? ) = s and (t? ) = t; the recursive algorithm in Theorem 3.18 will nd (s? ^ t? ) = s ^ t: Thus the nal pair of s ^ t will be hp pi, and it will have sux length 2. Similarly s _ t = (s? _ t? ) because s _ t 2 f (s? _ t? ) ; (s? _ t? ) g; and choosing (s? _ t? ) gives a contradiction: if s = (s? ) (s? _ t? ) and t = (t? ) (s? _ t? ) then (because of Theorem 3.10) s? = ((s? ) )? ((s? _ t? ) )? = 6 ((?s? _?t? ) )?? = ?s? _ t? and correspond? ? ? ? ? ? ? ingly t = ((t ) ) ((s _ t ) ) = 6 ((s _ t ) ) = s _ t so the lub of s? and t? is not s? _ t? , a contradiction. Again the nal pair of s _ t will be hq q i, with sux length 2
2
2
2
2
+
+
2
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
2. The other three cases, where h > 1 and/or k > 1, are similar.
4. Submodularity of Weighted Path-Length over the Lattices. Hu man codes for a positive descending weight sequence w = h w1 w2 wn i are binary tree pathlength sequences ` = h `1 `2 `n i that minimize the weighted path-length gw (`) =
n X i=1
wi `i :
In this section we show that gw is submodular over the lattice of trees. which helps explain why ecient algorithms for nding optimal trees are possible at all. 4.1. Submodularity. Most work on submodular functions assumes the lattice is the lattice of subsets of a given set, the case originally emphasized by Edmonds [6]. However, the de nition applies to any lattice: Definition 4.1. A real-valued function f : L ! < de ned on a lattice hL; v; u; ti is submodular if f (x u y) + f (x t y) f (x) + f (y)
D.S. PARKER AND P. RAM
16
for all x; y 2 L. Equivalently, f is submodular if a \di erential" inequality holds:
2 f (x; y) def = f (x) + f (y) ? f (x u y) ? f (x t y) 0: Section 4.4 discusses the relationship between submodularity and convexity.
4.2. Submodularity of weighted path-length on the Majorization lattice. In this section we show that weighted path-length on the imbalance lattice of trees (or a logarithmic variant on the majorization lattice of densities) is a submodular function. De ne the function Gw on the majorization lattice of densities by
Gw (x) = gw (?log2 (x)) =
?
X i
wi log2 (xi ):
1 Note that Gw is convex on 1): Thus gw (s) > gw (s? ) and gw (t) > gw (t? ) in all cases. However, it can happen that gw (s ^ t) < gw (s? ^ t? ) or gw (s _ t) < gw (s? _ t? ) because it is possible either that s? ^ t? 6= (s ^ t)? or that s? _ t? 6= (s _ t)? : Speci cally, it is
z }|k { 2
z }|k { 2
possible s ^ t = h (q ? j ) (q ? j ) q q i and s? ^ t? = h (q ? j ? 1) q q i; i.e., s ^ t = (s? ^ t? )+ and s ^ t has sux increment j > 1, in which case gw (s ^ t) = gw (s? ^ t? ) + (wn?2k+1 ? j wn?2k + q wn ) and the parenthesized expression can be negative. From Theorem 3.19, the nal pairs of entries of s and t are always the same as the nal pairs of entries of s ^ t and s _ t, and the sux lengths for each of s and t cannot be less than those for each of (s ^ t) and (s _ t). We now consider the same four cases addressed in the proof of Theorem 3.19. In the case where both s is the upper expansion of s? , and t is the upper expansion of ? t , then by Theorem 3.19 s ^ t = (s? ^ t? )+ and s _ t = (s? _ t? )+ ; so 2 gw (s; t) = (gw (s) + gw (t)) ? (gw (s ^ t) + gw (s _ t)) = (gw (s? ) + gw (t? )) ? (gw (s? ^ t? ) + gw (s? _ t? )) = 2 gw (s? ; t? ) with the preceding analysis for gw (`) with k = 1. In this situation only the nal pairs of entries of s, t and of s ^ t, s _ t can cause the two di erences to be unequal, but we now know them to give the same two pairs. So in this case the theorem follows by induction. It remains to treat the cases where s is the lower expansion of s? or t is the lower expansion of t? . In these cases it can happen that gw (s ^ t) < gw (s? ^ t? ) or gw (s _ t) < gw (s? _ t? ) as previously noted. In the case where either s is the lower expansion of s? or t is the lower expansion of t? , but not both, then by Theorem 3.19, either s ^ t = (s? ^ t? )+ and s _ t = (s? _ t? )+ ; or s ^ t = (s? ^ t? )+ and s _ t = (s? _ t? )+ : The lower expansions among these two cannot yield as large a gw increase as the lower expansions giving s and t, because they expand higher-indexed positions (their sux lengths are never longer), and the sux increment of s? ^ t? or s? _ t? can be greater than 1. Therefore 2 gw (s?; t) 2 gw (s? ; t? ): In the nal case where both s is the lower expansion of s and t is the lower expansion of t? , then s ^ t = (s? ^ t? )+ and s _ t = (s? _ t? )+ (Theorem 3.19), and because the lower expansions giving s ^ t and s _ t cannot yield as large a gw increase as the lower expansions giving s and t, again 2 gw (s; t) 2 gw (s? ; t? ): To see an example, the submodularity of gw can be veri ed on the lattice for n = 9 and the weight sequence shown in Figure 4.1.
D.S. PARKER AND P. RAM
18
s s s s s s s s
123456788 1329
123457777 1340
123466677 1324
b123555677 b1351 b b ` gw (`) b b 124445677 123556666 " 123456788 1329 " 1330 1362 " 123457777 1340 " 123466677 1324 " 123555677 1351 "124446666 b133345677 b 1341 123556666 1362 b1281 b 124445677 1330 b b 124446666 1341 b b b 124455566 1325 b 222345677 b 133346666b 124455566 b 1292 b 1325 124555555 1386 1302 b b 133345677 1281 b b 133346666 1292 b b 133355566 1276 124555555bbb133355566bb 222346666 1313 e 1386 133444566 1303 b1276 b 133445555 1314 e b 134444455 1348 b e b 222355566 144444444 1433 133444566 b 1303 e 1297 222345677 1302 b e 222346666 1313 b b 222355566 1297 ee b b 222444566 222444566 1324 133445555 b 1314 b 1324 222445555 1335 b b 223334566 1303 b b b b 223335555 1314 b b 223344455 1298 134444455 222445555bb " 223334566 " 223444444 1359 1335 1303 e 1348 " 233333455 1349 e " 233334444 1360 " e 333333344 1510 144444444 "223335555 1314 e 1433 e e e e 223344455 b 1298 e b e b b ee 223444444bb " 233333455 " 1349 1359 " " " The transitively reduced imbalance lattice for n = 9 showing, "233334444 for w = h 189 95 73 71 23 21 18 9 1 i, the code cost g (`) for
Code Path-length Sequence
Cost Weighted Path-length
s s s fs s s s s
s s s s s s fs s s s
w
each path-length sequence `. The Hu man code 133355566, with cost 1276, is the global minimum. The code 223344455, with cost 1298, is a local minimum. (Because the graph shows only the transitive reduction of the lattice, it omits some edges corresponding to exchanges, but the minimum is localized.) Fig. 4.1. Costs of all possible codes for the weights
s s
1360
333333344 1510
w = 189 95 73 71 23 21 18 9 1 h
.
i
HUFFMAN CODING AS SUBMODULAR OPTIMIZATION OVER A LATTICE
19
4.4. Submodularity as a discrete analogue of Convexity. Although it is very simply de ned, submodularity is dicult to appreciate. Using only standard vector calculus, we now clarify some basic relationships between submodularity and notions of convexity. We have not seen this done elsewhere. There are several reasons why submodularity plays an important role here, at the crossroads between information and coding theory. First, submodularity is directly related to the Fortuin-Kasteleyn-Ginibre (FKG) \correlation" inequalities, which generalize a basic inequality of Tchebyche on mean values of functions (hence expected values of random variables). A ne survey of results with FKG-like inequalities is [15]. Second, submodularity is closely related to convexity. Book-length surveys by Fujishige [9] and Narayanan [28] review connections between submodularity and optimization (and even electrical network theory). The relationship between convexity and submodularity was neatly summarized by Lovasz with the following memorable de nition and result. Definition 4.4. Given a nite set S of cardinality n, we can identify a f0; 1g-vector t 2 i i : wi =zi + wi=zi ij == ji + 1 P and, for example, G (@ x) = ? w log (x ) ? n? wi log (xi ? xi ) satis es 2
w
2
2
w
+1
2
w
+1
2
w
+1
+1
+1
+1
w
1
2
1
1
i=1
@ 2 (Gw (@ x)) = ?wi+1 1 @xi @xj ln(2) (xi+1 ? xi )2 0
2
2 +1 2 +1
2
+1
when j = i + 1:
This gives two alternative proofs of Theorem 4.2, showing how such results can be derived more easily. Definition 4.8. A function F : 0; integer (1 i n): i=1
HUFFMAN CODING AS SUBMODULAR OPTIMIZATION OVER A LATTICE
25
Dropping the integrality constraint gives an interesting continuous relaxation of Hu man coding that can be attacked numerically. For example, by treating the constraint as a penalty function, the problem with ? P above can be solved numerically something like the P system of equations @=@`j ni=1 wi `i + 1010 ( 1 ? ni=1 2?`i )2 = 0 (1 j n): Using the example weight sequence w = h 189 95 73 71 23 21 18 9 1 i studied earlier, a simple program found a unique real solution ` h 1:4 2:4 2:8 2:8 4:4 4:8 4:8 5:8 9:0 i for these equations, with objective 1241. As expected, this solution is near the optimal Hu man code h 1 3 3 3 5 5 5 6 6 i, with cost 1276. When the relaxation is faithful to the original, it will be possible to nd optimal solutions quickly. The relaxed solution can be used to jump to the right neighborhood in the imbalance lattice, from which balancing exchanges will walk to the optimal code. The penalty function could clearly be varied, and perhaps could be changed to encourage near-integral solutions. Interior point methods on the majorization lattice may? also be possible. Among other things, it may ?be possible to de ne s ^ t in terms of ?log2 2?s u 2?t and de ne s _ t in terms of ?log2 2?s t 2?t : they are often identical, and always satisfy 2?s ^ t 2?s u 2?t ; 2?s t 2?t 2?s _ t (because 2?s u 2?t and 2?s t 2?t are the glb and lub with respect to majorization). For perspective, if = 7 ? log 2 (12) 3:4150375 and =? ( ? 1), the following set of examples represent the unusual cases with n = 9 where ?log2 2?s u 2?t 6= (s ^ t) or ? ? s ? t ?log 2 2 t 2 6= (s _ t):
?
?log 2?s u 2?t h133 55555i h222 55555i h22 444455i ? ?log 2?s t 2?t h14 345677i h14 346666i h14 355566i h14 444566i h14 445555i
s^t h133445555i h222445555i h223344455i s_t h133345677i h133346666i h133355566i h133444566i h133445555i ? These examples suggest there may? be algorithms that `round up' ?log 2?s u 2?t to give s ^ t, and `round down' ?log 2?s t 2?t to give s _ t. s
h124555555i h124555555i h134444455i s h144444444i h144444444i h144444444i h144444444i h144444444i
t
h133345677i h222345677i h222345677i t h222345677i h222346666i h222355566i h222444566i h222445555i
2
2
2
2
6.2. Practical Applications in Adaptive Coding. In many practical situations it is dicult or impossible to know a priori the weights w used in Hu man coding. A natural idea, which occurred independently to Faller [8] and Gallager [11], is to allow the weights to be determined dynamically, and have the Hu man code `evolve' over time. Dynamic Hu man coding is the strategy of repeatedly constructing the Hu man code for the input so far, and using it in transmitting the next input symbol. Knuth presented an ecient algorithm for dynamic Hu man coding in [22], and his performance results for the algorithm show it consistently producing compression very near (though not surpassing) the compression attained with static Hu man code for the entire input. Vitter [40, 41] then developed a dynamic Hu man algorithm that improves on Knuth's in the following way: rather than simply revise the Hu man tree after each P input symbol, Vitter also nds a new Hu man tree of minimal external path length i `i and height maxi `i . With this modi cation Vitter was actually able to surpass the performance of static Hu man coding on several benchmarks. A small contribution we can make is to clarify the improvement of Vitter. Basically, Vitter's algorithm di ers from Knuth's in constructing the optimal path-length sequence P that is also as balanced as possible. Note that minimizing the external path length i `i is
26
D.S. PARKER AND P. RAM
identical to maximizing the level of balance. Since there can be more than one optimal code, and unnecessary imbalance tends to penalize the symbol currently being encoded, insisting on maximally balanced codes improves performance. Another contribution of the lattice perspective here is to encourage development of new adaptive coding schemes. As suggested in Section 5.1, a move between adjacent points in the lattice corresponds to minor alteration of codes, and by moving through the lattice we incrementally modify the cost of a code. Hill-climbing then gives greedy coding algorithms, and on-line hill-climbing gives adaptive coding algorithms. Although we have shown that the codes produced by hill-climbing are not guaranteed to be optimal, lattice-oriented adaptive coding algorithms may still have a role to play in some coding situations, since the Hu man notion of optimality is not really what is needed in the (currently popular and enormously important) adaptive context. For example, adaptive coding algorithms can start at any point in the lattice, as long as both ends of the communication know which one. Rather than rely on the dynamic Hu man algorithm to derive reasonable operating points for the code, or rely on Knuth's `windowed' algorithm [22], one can immediately begin with a mutually-agreed-upon, `reasonable' initial code (depending on the type of information being transmitted), and then adapt this code using some mutually-agreed-upon greedy algorithm for moving in the imbalance lattice.
Acknowledgements. We are very grateful to Pierre Hasenfratz for insightful comments that improved this paper. A conversation with Mordecai Golin, who provided us with an expanded version of [14], inspired us to discuss dynamic programming explicitly in this paper. He also pointed out the survey [4] to us. Also, we are indebted to two anonymous referees for clari cations of the exposition, especially of the signi cance of submodularity and of Shannon's work [38]. REFERENCES [1] J. Abrahams, Code and Parse Trees for Lossless Source Encoding, Proc. Compression & Complexity of Sequences (SEQUENCES'97), Positano, Italy, 1997, IEEE Press, to appear. [2] A. Aggarwal, A. Bar-Noy, S. Khuller, D. Kravets, B. Schieber, Ecient Minimum Cost Matching and Transportation using the Quadrangle Inequality, J. Algorithms, 19:1 (July 1995), pp. 116{143. [3] A. Berman, R.J. Plemmons, Nonnegative matrices in the mathematical sciences, SIAM, 1994. [4] R.E. Burkard, B. Klinz, R. Rudolf, Perspectives of Monge properties in optimization, Discrete Applied Math., 70 (1996), pp. 95{161. [5] B.A. Davey, H.A. Priestley, Introduction to Lattices and Order, Cambridge U. Press, 1990. [6] J. Edmonds, Submodular Functions, Matroids and Certain Polyhedra, in Combinatorial Structures and their Applications, R. Guy et al., eds., Gordon & Breach, 1970, pp. 69{87. [7] D. Eppstein, Z. Galil, R. Giancarlo, G.F. Italiano, Sparse dynamic Programming. II. Convex and concave cost functions, Journal of the ACM, 39:3 (July 1992), pp. 546{567. [8] N. Faller, An adaptive system for data compression, Paci c Grove, CA: Record of the 7th Asilomar Conference on Circuits, Systems, and Computers, 1973, pp. 593{597. [9] S. Fujishige, Submodular functions and optimization, North-Holland Elsevier, 1991. [10] R.G. Gallager, Information Theory and Reliable Communications, J. Wiley, 1968. [11] R.G. Gallager, Variations on a theme by Hu man, IEEE Trans. Information Theory, IT-24:6 (November 1978), pp. 668{674. [12] E.N. Gilbert, Codes Based on Inaccurate Source Probabilities, IEEE Trans. Information Theory, IT-17:3 (May 1971), pp. 304{314. ( ) is analyzed on p. 309. [13] C.R. Glassey and R.M. Karp, On the optimality of Hu man trees, SIAM J. Appl. Math., 31 (1976), pp. 368{378. [14] M.J. Golin, G. Rote, A dynamic programming algorithm for constructing optimal pre x-free codes for unequal letter costs, Proc. ICALP 95, Z. Fulop, F. Gecseg, eds., Springer-Verlag, 1995, pp. 256{267. g N
HUFFMAN CODING AS SUBMODULAR OPTIMIZATION OVER A LATTICE
27
[15] R.L. Graham, Applications of the FKG Inequality and its Relatives, in MathematicalProgramming: The State of the Art, B. Korte, A. Bachem, M. Grotschel, eds., Springer-Verlag, 1983, pp. 115{131. [16] G.H. Hardy, J.E. Littlewood, G. Polya, Inequalities, Cambridge University Press, 1934. [17] A.J. Hoffman. On Simple Linear Programming Problems, in Convexity, Proc. Seventh Symposium in Pure Mathematics, Vol. VII, V. Klee, ed., AMS, 1961, pp. 317{327. [18] D.A. Huffman, A method for the construction of minimum redundancy codes, Proc. IRE, 40 (1951), pp. 1098{1101. [19] F.K. Hwang, Generalized Hu man Trees, SIAM J. Appl. Math., 37 (1979), pp. 124{127. [20] C.M. Klein, A submodular approach to discrete dynamic programming, European J. Operational Research, 80:1 (Jan. 1995), pp. 145{155. [21] D.E. Knuth, Optimum Binary Search Trees, Acta Informatica, 1 (1971), pp. 14{25. [22] D.E. Knuth, Dynamic Hu man Coding, J. Algorithms, 6 (1985), pp. 163{180. [23] E. Lawler, Combinatorial Optimization: Networks & Matroids, Holt-Rinehart-Winston, 1976. [24] E.L. Lawler, Submodular Functions & Polymatroid Optimization, in Combinatorial Optimization: Annotated Bibliographies, A.H.G. Rinnooy Kan, M. O'hEigeartaigh, J.K. Lenstra, eds., J. Wiley & Sons, 1985, pp. 32{38. [25] L. Lovasz, Submodular functions and convexity, in Mathematical Programming: The State of the Art, B. Korte, A. Bachem, M. Grotschel, eds., Springer-Verlag, 1983, pp. 235{257. [26] U. Manber, Introduction to Algorithms, Addison-Wesley, 1989. [27] A.W. Marshall, I. Olkin, Inequalities: Theory of Majorization and Its Applications, Academic Press, 1979. [28] H. Narayanan, Submodular functions and electrical networks, North-Holland Elsevier, 1997. [29] A. Ostrowski, Sur quelques applications des fonctions convexes et concaves au sens de I. Schur (o ert en homage a P. Montel), J. Math. Pures Appl., 31 (1952), pp. 253{292. [30] J.M. Pallo, Enumerating, ranking and unranking binary trees, Computer Journal, 29:2 (April 1986), pp. 171{175. [31] J.M. Pallo, Some properties of the rotation lattice of binary trees, Computer Journal, 31:6 (Dec. 1988), pp. 564{565. [32] D.S. Parker, Conditions for Optimality of the Hu man Algorithm, SIAM J. Comput., 9:3 (August 1980), pp. 470{489. [33] D.S. Parker, P. Ram, Greed and Majorization, November 1994. Issued as Technical Report CSD-960003, UCLA Computer Science Dept., March 1996. [34] D.S. Parker, P. Ram, A Linear Algebraic Reconstruction of Majorization, Technical Report CSD-970036, UCLA Computer Science Dept., September 1997. [35] U. Pferschy, R. Rudolf, G.J. Woeginger, Monge matrices make maximization manageable, Operations Research Letters, 16 (1994), pp. 245{254. [36] G.-C. Rota, On the Foundations of Combinatory Theory I. Theory of Mobius Functions, Z. Wahrscheinlichkeitstheorie, 2 (1964), pp. 340{368. eine Klasse von Mittelbildungen mit Anwendungen auf die Determinantenthe[37] I. Schur, Uber orie, Sitzungsber. Berl. Math. Ges., 22 (1923), pp. 9{20. [38] C.E. Shannon, The Lattice Theory of Information, Proc. IRE Trans. Information Theory, 1 (1950). Reprinted in Claude Elwood Shannon: Collected Papers, IEEE Press, 1993. [39] N.J.A. Sloane, S. Plouffe, The Encyclopedia of Integer Sequences, Academic Press, 1995. [40] J.S. Vitter, Design and Analysis of Dynamic Hu man Codes, J. ACM, 34:4 (October 1987), pp. 825{845. [41] J.S. Vitter, Algorithm 673: Dynamic Hu man Coding, ACM TOMS, 15:2 (June 1989), pp. 158{167. [42] F.F. Yao, Ecient dynamic programming using quadrangle inequalities, Los Angeles, CA: Proc. 12th Annual ACM Symp. on Theory of Computing, 1980, pp. 429{435.
Abstract. We show that the space of all binary Hu man codes for a nite alphabet de nes a lattice, ordered by the imbalance of the code trees. Representing code trees as path-length sequences, we show that the imbalance ordering is closely related to a majorization ordering on real-valued sequences that correspond to discrete probability density functions. Furthermore, this tree imbalance is a partial ordering that is consistent with the total orderings given by either the external path length (sum of tree path lengths), or the entropy determinedby the tree structure. On the imbalance lattice, we show the weighted path-length of a tree (the usual objective function for Hu man coding) is a submodular function, as is the corresponding function on the majorization lattice. Submodular functions are discrete analogues of convex functions. These results give perspective on Hu man coding and suggest new approaches to coding as optimization over a lattice. Key words. Hu man coding, adaptive coding, pre x codes, enumeration of trees, lattices, combinatorial optimization, convexity, submodular functions, entropy, tree imbalance, Schur convex functions, majorization, Moebius inversion, combinatorial inequalities, Fortuin-Kasteleyn-Ginibre (FKG) inequality, quadrangle inequality, Monge matrices, dynamic programming, greedy algorithms. AMS subject classi cations. 94A15, 94A24, 94A29, 94A45, 90C25, 90C27, 90C39, 90C48, 52A41, 68Q20, 68R05, 05A05, 05A20, 05C05, 05C30, 06A07, 26B25, 26D15
1. Introduction. The Hu man algorithm has been used heavily to produce ecient binary codes for almost half a century now. It has inspired a large literature with diverse theoretical and practical contributions. A comprehensive, very recent survey is [1]. Although the algorithm is quite elegant, it is tricky to prove correct and to reason about. While there may be little hope of improving on the O(n log n) complexity of the Hu man algorithm itself,1 there is still room for improvement in our understanding of the algorithm. There is also plenty of room for improvement in our understanding of variants of Hu man coding. Although the Hu man algorithm is remarkably robust in general and has widespread use, it is far from optimal in many real applications. Hu man coding is optimal only when the symbols to be coded are random and occur with xed probabilities. Timevarying dependencies are not captured by the Hu man coding model, and optimal encoding of nite messages is not captured either. Our motivation came from analysis of dynamic Hu man coding, a speci c extension of Hu man coding in which the code used evolves over time. Recently dynamic coding algorithms have been studied heavily. Our initial idea was to de ne \rebalancing" operations on code trees and to use these dynamically (\on the y") in producing better codes, in situations where the distribution of symbols to be coded varies over time and/or is not accurately predictable in advance. This paper reconstructs Hu man coding as an optimization over the space of binary trees. A natural representation for this space is sequences of ascending path-lengths, since this captures what is signi cant in producing optimal codes. We show that the set of path-length sequences representing binary trees forms a lattice, which we call the imbalance lattice. This lattice orders trees by their imbalance and gives an organization for them that is useful in optimization. Our belief is that having a better UCLA Computer Science Department, University of California, Los Angeles, CA 90095-1596 ([email protected]). y Xerox Corporation, El Segundo, CA 90245 ([email protected]). 1 The algorithm is closely related to sorting, in the sense that the sorted sequence of a sequence of integer values h x1 xn i is obtainable directly from the optimal code tree for the values x x h 2 1 2 n i (e.g., [26, p.335]). 1
D.S. PARKER AND P. RAM
2
mathematical (and not purely procedural) understanding of coding will ultimately pay o in improved algorithms. The imbalance lattice and its imbalance ordering on trees depend on majorization in an essential way. Majorization is an important ordering on sequences that has many applications in pure and applied mathematics [27]. We have related it to greedy algorithms directly [33]. Earlier majorization was recognized as an important property of the internal node weights produced by the Hu man algorithm [13, 32], and in this work we go further to clarify its pervasive role. By viewing the space of trees as a lattice, a variety of new theorems and algorithms become possible. For example, the objective functions commonly used in evaluating codes are submodular on this lattice. Submodular functions are closely related to convex functions (as we explain later; see Theorem 4.5) and are often easy to optimize [6, 9, 23, 24, 25]. Hu man coding gives a signi cant example of the importance of submodularity in algorithms.
2. Ordered Sequences, Rooted Binary Trees, and Hu man Codes. 2.1. Ordered Sequences. By a sequence we mean an ordered collection of non-
negative real values
x = h x x xn i: 1
2
Repetition of values in the sequence is permitted: the values xj need not be distinct. The length of this sequence is nn, and for simplicity we also refer to the set of such sequences with the vector notation =R h 1 0 0 0 i: Proof. The transformation x 7! x de nes a lattice isomorphism between the vector and majorization lattices, and between the distribution and density lattices. Here x u y and x t y are de ned just so as to be the majorization glb and lub:
z xR ; z y R R R , (R z) vec ( Rx); ( z) vec ( y) R , ( z) vecR ( ( x) minvec R ( y) ) , z @ ( ( x) minvec ( y) ) , z x u y:
x Rz; y z R R R , ( Rx) vec ( zR); ( y) vecR ( z) , ( ( xR) maxvec ( yR) ) vec ( z) , @ ( ( x) maxvec ( y) ) z , x t y z:
Thus the majorization algebra also forms a distributive lattice. Even when x and y are in descending order, the sequences (x u y) and (x t y) de ned here are not necessarily in descending order: x = h 2?2 2?2 2?3 2?4 2?4 2?4 2?4 2?4 2?4 i and y = h 2?2 2?3 2?3 2?3 2?3 2?3 2?4 2?5 2?5 i yield the least upper bound x t y = R R @ ( x maxvec y) = h 2?2 2?2 2?3 2?4 2?4 2?3 2?4 2?5 2?5 i: See Figure 3.6.
D.S. PARKER AND P. RAM
12
ss = = s = t= R 22??tt == R 2?s maxvec R 2?t = R R @ ( 2?s maxvec 2?t ) = = 2?s t 2?t = R 2?s min Rs 2_?tt == vecR R @ ( 2?s minvec 2?t ) = = 2?s u 2?t = s^t =
R 22??
( ( ( ( ( ( ( ( ( ( ( ( ( (
2?7 2?7 2?7 2?7 2?7 2?7
1?1 2 64 2?2 2 32 64 64 2?1 1 32 32 2?2 2
2?2 2 96 2?2 2 64 96 32 2?2 2 64 32 2?2 2
4?4 2 104 2?2 2 96 104 8 2?4 4 96 32 2?2 2
4?4 2 112 3?3 2 112 112 8 2?4 4 112 16 2?3 3
5?5 2 116 4?4 2 120 120 8 2?4 4 116 4 2?5 5
5?5 2 120 5?5 2 124 124 4 2?5 5 120 4 2?5 5
5?5 2 124 6?6 2 126 126 2 2?6 6 124 4 2?5 5
6?6 2 126 7?7 2 127 127 1 2?7 7 126 2 2?6 6
6?6 2 128 7?7 2 128 128 1 2?7 7 128 2 2?6 6
) ) ) ) ) ) ) ) ) ) ) ) ) )
Fig. 3.5. Related points in the majorization and imbalance lattices, showing their connection
s = = s = t = R 22??tt == R 2?s maxvec R 2?t = R R @ ( 2?s maxvec 2?t ) = = 2?s t 2?t = s_t =
R 22??
s
2?7 2?7 2?7 2?7
( ( ( ( ( ( ( ( ( (
2?2 2 32 2?2 2 32 32 32 2?2 2
2?2 2 64 3?3 2 48 64 32 2?2 2
3?3 2 80 3?3 2 64 80 16 2?3 3
4?4 2 88 3?3 2 80 88 8 2?4 3
4?4 2 96 3?3 2 96 96 8 2?4 4
4?4 2 104 3?3 2 112 112 16 2?3 4
4?4 2 112 4?4 2 120 120 8 2?4 4
4?4 2 120 5?5 2 124 124 4 2?5 5
4?4 2 128 5?5 2 128 128 4 2?5 5
) ) ) ) ) ) ) ) ) )
Fig. 3.6. Results of (2?s u 2?t) and (2?s t 2?t) are not necessarily in descending order
s= = s = t= R 22??tt == R 2?s min R 2?t = vecR R ? s @ ( 2 minvec 2?t ) = = 2?s u 2?t =
R 22??
s
s^t =
2?7 2?7 2?7 2?7
( ( ( ( ( ( ( ( (
1?1 2 64 2?2 2 32 32 32 2?2
2?2 2 96 2?2 2 64 64 32 2?2
4?4 2 104 2?2 2 96 96 32 2?2
( 2?2 2?2 2?2 ( 2
2
2
5?5 2 108 3?3 2 112 108 12 2? ? = 2?4 4
5?5 5?5 5?5 5?5 5?5 2 2 2 2 2 112 116 120 124 128 4?4 5?5 6?6 7?7 7?7 2 2 2 2 2 120 124 126 127 128 112 116 120 124 128 4 4 4 4 4 2?5 2?5 2?5 2?5 2?5 (?7 + log2 (12)) ?3:4150375 2?4 2?5 2?5 2?5 2?5 4
5
5
5
5
2? (s ^ t) (2?s u 2?t ); the two di er where indicated. Non-integral exponents occur for n 9. Fig. 3.7. The imbalance lattice is not simply conjugate to a sublattice of the majorization lattice
) ) ) ) ) ) ) ) ) ) )
HUFFMAN CODING AS SUBMODULAR OPTIMIZATION OVER A LATTICE
13
3.6. The Imbalance Lattice: A Discrete Cousin of the Majorization Lattice. Since every pair of sequences in Figures 3.2 and 3.3 has a unique glb and lub, the imbalance ordering is not only a partial order, but also a lattice. In this section we prove this by showing that every pair of sequences s, t has a glb s ^ t and lub s _ t. We also relate the imbalance lattice directly to the majorization lattice, as illustrated in Figures 3.5|3.7. Theorem 3.17. On tree path-length sequences, the imbalance ordering is isomorphic to the majorization ordering. Speci cally, whenever s and t are tree path-length sequences, then 2?s 2?t : Proof. We show rst that balancing exchanges cause a reduction in the majorization ordering. Let s be the result of a balancing exchange on t (so s t). Then:
s t
R R
t = s =
( (
2 t = 2 s =
( (
? ? 2?t 2?s
R
=
(
=
(
p (p + 1) 2?p 2?(p+1) S S +2?p S S +2?(p+1) 0 + 2?(p+1)
( 0
R
i
2?u 2?(p+1) 2?u S +2?p +2?u S +2?p + 2?u u p
( + 1)
u
v
2?v T T ? 2?v + 2?v
(q + 1) v 2?(q +1) 2?v
(q + 1) q 2?(q +1) 2?q
T +2?(q+1) T + 2?(q +1)
T +2?q T +2?q 0
) ) ) )
1 ) 1 ) 0 )
Thus 2?s and 2?t di er only in the values appearing between p and q, and each element R R ? t ? s in 2 ? 2 is nonnegative, so 2?s 2?t . The proof of the converse, that 2?s 2?t implies s t for tree path-length sequences s, t, can proceed by assuming a counterexample for which the di erence in the levels of balance
m = (level of balance of s) ? (level of balance of t): is minimal. Since 2?s 2?t , let a, b, c, d be the rightmost aligned pairwise-di ering values among the two sorted sequences such that s = h a b i and t = h c d i, where c < a, b < d because of the majorization inequality, a b and c d because the sequences are ascending, c 6= d since c < a b < d, and nally 2?a + + 2?b = 2?c + + 2?d ; which is always possible by the Kraft equality. Because b < d necessarily t = h c d d i, since otherwise we reach a contradiction (multiplying both sides of the equality by 2d makes the left side even but the right side odd). Thus, if we de ne the result t0 = h (c + 1) (c + 1) (d ? 1) i of a balancing exchange on t =0 h c d d i, then the level di erence between s and t0 is at most (m ? 1), and 2?t 2?t . Furthermore ? s ? t0 we claim 2 2 , using the following schematic: t = t = s =
( ( (
2 t = 2 t = 2 s =
( (
0
? ?0 ? 2?t ? 2?s 0 2?t ? 2?t 0 ? t ? s 2 ? 2
R R R
R R R
(
=
( 0
=
( 0
=
( 0
c u (c + 1) (c + 1) a w 2?c 2?u 2??a(c+1) 2?(c+1) 2 2?w 0 +(2?c ? 2?a ) +S1 0 +2?(c+1) +2?u 0 +(2?(c+1) ? 2?a ) +(S1 ? 2?u )
d
2
d d ? 1) b 2?d 2?(d?1) 2?b
(
?d Sk
+
+2
?d
Sk ? 2?d )
+(
0 0 0
) ) ) ) ) )
0 ) 0 ) 0 )
Because 2?s 2?t , the running totals S1 ; : : : ; Sk are nonnegative. Also, (2?(c+1) ? 2?a ) 0 since c < a. Furthermore c (a ? 1) w, implying S1 ? 2?u = (2?c ? 2?a) ? 2w 2?(a?1) ? 2?w 0. Finally Sk +2?d ?R2?b = 0, soR b 0 (d ? 1) implies Sk0? 2?d = (2?b ? 2?d ) ? 2?d = 2?b ? 2?(d?1) 0: Thus 2?s vec 2?t (i.e., 2?s 2?t ), contradicting the assumed minimality of m, and existence of a counterexample.
D.S. PARKER AND P. RAM
14
Theorem 3.18. The imbalance ordering on binary trees determines a bona- de lattice in which, for all s and t, the glb s ^ t and lub s _ t are de ned with the following recursive algorithms, where the expansion used is chosen from among the lower and upper expansions: 8 s if s t > < t if ts s ^ t = > the greatest expansion of s? ^ t? : that is also a lower bound for s and t otherwise;
s_t =
8 t > < s the least expansion of s? _ t? > : that is also an upper bound for s and t
if if
st ts
otherwise:
Proof. We must show that, whenever s and t are tree path-length sequences of length
n, there are unique path-length sequences s ^ t and s _ t such that: s ^ t s; t; also if ` is any path-length sequence, then ` s; t i ` s ^ t: s; t s _ t; also if ` is any path-length sequence, then s; t ` i s _ t `: This can be done by induction on n. We consider only the glb here, the proof for the lub being similar. The theorem holds trivially for n 6, since then the trees are totally ordered. Assume it holds for sequences of size n ? 1 or less.
First, s and t must have a common lower bound: The glb s? ^ t? exists by induction, and (Theorems 3.9 and 3.10) lower expansion gives a lower bound (s? ^ t? )+ (s? )+ s; (s? ^ t? )+ (t? )+ t: Second, if s and t have two greatest lower bounds ` and `0 , then they must be equal: From ` s; t and `0 s; t we infer `? s? ^ t? and `0? s? ^ t? by Theorem 3.10. Since furthermore ` and `0 are greatest lower bounds, s? ^ t? `? and s? ^ t? `0? : Thus `? = `0? : By Theorem 3.9, the only way ` 6= `0 can arise is that ` = (`? )+ ; `0 = (`? )+ or ` = (`? )+ ; `0 = (`? )+ so ` `0 or `0 `, contradicting their both being greatest lower bounds. Thus ` = `0 . Third, the algorithm produces a glb that is as good as any other lower bound: Assuming this for (s? ^ t? ) by induction, there can be no lower bound ` 6= (s ^ t) such that (s? ^ t? )+ `; since otherwise (s? ^ t? ) `? s? ; t? contradicting our assumption. The table of nontrivial examples in Figure 3.3 gives an appreciation for glbs and lubs. The rst example (which is illustrated in the gure) is expanded in Table 3.2. Note the nal pairs of entries in s and t are the same as the nal pairs of entries in s ^ t and s _ t, and the sux lengths of s and t are never shorter than those of s ^ t and s _ t. Theorem 3.19. If s and t are path-length sequences of length n, then:
(s? _ t? ) ? ? ((ss? _^ tt? ))
s = (s?? ) and t = (t?? ) s = (s ) and t = (t ) s = (s?? ) and t = (t?? ) s^t = (s? ^ t? ) s = (s ) and t = (t ) Otherwise, if either s = (s? ) and t = (t? ) , or s = (s? ) and t = (t? ) , then either s ^ t = (s? ^ t? ) and s _ t = (s? _ t? ) ; or s ^ t = (s? ^ t? ) and s _ t = (s? _ t? ) : Furthermore: if the nal pairs of entries of s and t are hp pi and hq qi, where p q, then the nal pairs of entries of s ^ t and s _ t are respectively hp pi and hq qi. Also: the sux lengths of s and t are at least as long as those of (s ^ t) and (s _ t). s_t =
if if if if
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
HUFFMAN CODING AS SUBMODULAR OPTIMIZATION OVER A LATTICE
s
n
9 8 7 6
h
144444444
t
i
222355566
h
" lower expansion h
13444444
i
22235555
h
1334444
h
133344
223444444
i
h
2223455
i
22334444
h
h
222344
h
133355566
i
h
2224444
i
h
222344
h
13335555
i
" lower expansion
i
1333455
h
" lower expansion
i
i
" upper expansion
" lower expansion
" upper expansion
i
_
i
" lower expansion
" lower expansion
" lower expansion
s t
^
h
" upper expansion
" lower expansion h
s t
i
15
i
" upper expansion
i
h
133344
i
Table 3.2
Elaboration of the rst example of representative bounds in Figure 3.3, showing how s ^ t and s _ t can be derived with their recursive algorithms.
Proof. These properties follow by induction on n. For the basis, they all hold trivially when n 6, since then the imbalance lattice is a total order and f s; t g = f s _ t; s ^ t g; and the nal two entries of any path-length sequence are a pair by Theorem 3.1. For the induction step, we can write
s = h (p ? i) pz s? = h (p ? i) (p ? 1)
}|h p{ i zp h}|? p{ i
k
t = h (q ? j ) zq }| q{ i k? z }| q{ i ? t = h (q ? j ) (q ? 1) q where i; j;h; k > 0, and we assume with no loss of generality that p q. There are four cases to consider, depending on the the sux lengths 2h of s and 2k of t. In the rst, h = 1 and k = 1, i.e., s = (s? ) and t = (t? ) . Then i = 1 and j = 1 by Theorem 3.1. By induction (s? ^ t? ) and (s? _ t? ) have respective nal pairs h(p ? 1) (p ? 1)i and h(q ? 1) (q ? 1)i, and have sux lengths not exceeding those of s? and t? . Now, by Theorem 3.10 (s? ^ t? ) (s? ) and (s? ^ t? ) (t? ) : Because (s? ) = s and (t? ) = t; the recursive algorithm in Theorem 3.18 will nd (s? ^ t? ) = s ^ t: Thus the nal pair of s ^ t will be hp pi, and it will have sux length 2. Similarly s _ t = (s? _ t? ) because s _ t 2 f (s? _ t? ) ; (s? _ t? ) g; and choosing (s? _ t? ) gives a contradiction: if s = (s? ) (s? _ t? ) and t = (t? ) (s? _ t? ) then (because of Theorem 3.10) s? = ((s? ) )? ((s? _ t? ) )? = 6 ((?s? _?t? ) )?? = ?s? _ t? and correspond? ? ? ? ? ? ? ingly t = ((t ) ) ((s _ t ) ) = 6 ((s _ t ) ) = s _ t so the lub of s? and t? is not s? _ t? , a contradiction. Again the nal pair of s _ t will be hq q i, with sux length 2
2
2
2
2
+
+
2
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
2. The other three cases, where h > 1 and/or k > 1, are similar.
4. Submodularity of Weighted Path-Length over the Lattices. Hu man codes for a positive descending weight sequence w = h w1 w2 wn i are binary tree pathlength sequences ` = h `1 `2 `n i that minimize the weighted path-length gw (`) =
n X i=1
wi `i :
In this section we show that gw is submodular over the lattice of trees. which helps explain why ecient algorithms for nding optimal trees are possible at all. 4.1. Submodularity. Most work on submodular functions assumes the lattice is the lattice of subsets of a given set, the case originally emphasized by Edmonds [6]. However, the de nition applies to any lattice: Definition 4.1. A real-valued function f : L ! < de ned on a lattice hL; v; u; ti is submodular if f (x u y) + f (x t y) f (x) + f (y)
D.S. PARKER AND P. RAM
16
for all x; y 2 L. Equivalently, f is submodular if a \di erential" inequality holds:
2 f (x; y) def = f (x) + f (y) ? f (x u y) ? f (x t y) 0: Section 4.4 discusses the relationship between submodularity and convexity.
4.2. Submodularity of weighted path-length on the Majorization lattice. In this section we show that weighted path-length on the imbalance lattice of trees (or a logarithmic variant on the majorization lattice of densities) is a submodular function. De ne the function Gw on the majorization lattice of densities by
Gw (x) = gw (?log2 (x)) =
?
X i
wi log2 (xi ):
1 Note that Gw is convex on 1): Thus gw (s) > gw (s? ) and gw (t) > gw (t? ) in all cases. However, it can happen that gw (s ^ t) < gw (s? ^ t? ) or gw (s _ t) < gw (s? _ t? ) because it is possible either that s? ^ t? 6= (s ^ t)? or that s? _ t? 6= (s _ t)? : Speci cally, it is
z }|k { 2
z }|k { 2
possible s ^ t = h (q ? j ) (q ? j ) q q i and s? ^ t? = h (q ? j ? 1) q q i; i.e., s ^ t = (s? ^ t? )+ and s ^ t has sux increment j > 1, in which case gw (s ^ t) = gw (s? ^ t? ) + (wn?2k+1 ? j wn?2k + q wn ) and the parenthesized expression can be negative. From Theorem 3.19, the nal pairs of entries of s and t are always the same as the nal pairs of entries of s ^ t and s _ t, and the sux lengths for each of s and t cannot be less than those for each of (s ^ t) and (s _ t). We now consider the same four cases addressed in the proof of Theorem 3.19. In the case where both s is the upper expansion of s? , and t is the upper expansion of ? t , then by Theorem 3.19 s ^ t = (s? ^ t? )+ and s _ t = (s? _ t? )+ ; so 2 gw (s; t) = (gw (s) + gw (t)) ? (gw (s ^ t) + gw (s _ t)) = (gw (s? ) + gw (t? )) ? (gw (s? ^ t? ) + gw (s? _ t? )) = 2 gw (s? ; t? ) with the preceding analysis for gw (`) with k = 1. In this situation only the nal pairs of entries of s, t and of s ^ t, s _ t can cause the two di erences to be unequal, but we now know them to give the same two pairs. So in this case the theorem follows by induction. It remains to treat the cases where s is the lower expansion of s? or t is the lower expansion of t? . In these cases it can happen that gw (s ^ t) < gw (s? ^ t? ) or gw (s _ t) < gw (s? _ t? ) as previously noted. In the case where either s is the lower expansion of s? or t is the lower expansion of t? , but not both, then by Theorem 3.19, either s ^ t = (s? ^ t? )+ and s _ t = (s? _ t? )+ ; or s ^ t = (s? ^ t? )+ and s _ t = (s? _ t? )+ : The lower expansions among these two cannot yield as large a gw increase as the lower expansions giving s and t, because they expand higher-indexed positions (their sux lengths are never longer), and the sux increment of s? ^ t? or s? _ t? can be greater than 1. Therefore 2 gw (s?; t) 2 gw (s? ; t? ): In the nal case where both s is the lower expansion of s and t is the lower expansion of t? , then s ^ t = (s? ^ t? )+ and s _ t = (s? _ t? )+ (Theorem 3.19), and because the lower expansions giving s ^ t and s _ t cannot yield as large a gw increase as the lower expansions giving s and t, again 2 gw (s; t) 2 gw (s? ; t? ): To see an example, the submodularity of gw can be veri ed on the lattice for n = 9 and the weight sequence shown in Figure 4.1.
D.S. PARKER AND P. RAM
18
s s s s s s s s
123456788 1329
123457777 1340
123466677 1324
b123555677 b1351 b b ` gw (`) b b 124445677 123556666 " 123456788 1329 " 1330 1362 " 123457777 1340 " 123466677 1324 " 123555677 1351 "124446666 b133345677 b 1341 123556666 1362 b1281 b 124445677 1330 b b 124446666 1341 b b b 124455566 1325 b 222345677 b 133346666b 124455566 b 1292 b 1325 124555555 1386 1302 b b 133345677 1281 b b 133346666 1292 b b 133355566 1276 124555555bbb133355566bb 222346666 1313 e 1386 133444566 1303 b1276 b 133445555 1314 e b 134444455 1348 b e b 222355566 144444444 1433 133444566 b 1303 e 1297 222345677 1302 b e 222346666 1313 b b 222355566 1297 ee b b 222444566 222444566 1324 133445555 b 1314 b 1324 222445555 1335 b b 223334566 1303 b b b b 223335555 1314 b b 223344455 1298 134444455 222445555bb " 223334566 " 223444444 1359 1335 1303 e 1348 " 233333455 1349 e " 233334444 1360 " e 333333344 1510 144444444 "223335555 1314 e 1433 e e e e 223344455 b 1298 e b e b b ee 223444444bb " 233333455 " 1349 1359 " " " The transitively reduced imbalance lattice for n = 9 showing, "233334444 for w = h 189 95 73 71 23 21 18 9 1 i, the code cost g (`) for
Code Path-length Sequence
Cost Weighted Path-length
s s s fs s s s s
s s s s s s fs s s s
w
each path-length sequence `. The Hu man code 133355566, with cost 1276, is the global minimum. The code 223344455, with cost 1298, is a local minimum. (Because the graph shows only the transitive reduction of the lattice, it omits some edges corresponding to exchanges, but the minimum is localized.) Fig. 4.1. Costs of all possible codes for the weights
s s
1360
333333344 1510
w = 189 95 73 71 23 21 18 9 1 h
.
i
HUFFMAN CODING AS SUBMODULAR OPTIMIZATION OVER A LATTICE
19
4.4. Submodularity as a discrete analogue of Convexity. Although it is very simply de ned, submodularity is dicult to appreciate. Using only standard vector calculus, we now clarify some basic relationships between submodularity and notions of convexity. We have not seen this done elsewhere. There are several reasons why submodularity plays an important role here, at the crossroads between information and coding theory. First, submodularity is directly related to the Fortuin-Kasteleyn-Ginibre (FKG) \correlation" inequalities, which generalize a basic inequality of Tchebyche on mean values of functions (hence expected values of random variables). A ne survey of results with FKG-like inequalities is [15]. Second, submodularity is closely related to convexity. Book-length surveys by Fujishige [9] and Narayanan [28] review connections between submodularity and optimization (and even electrical network theory). The relationship between convexity and submodularity was neatly summarized by Lovasz with the following memorable de nition and result. Definition 4.4. Given a nite set S of cardinality n, we can identify a f0; 1g-vector t 2 i i : wi =zi + wi=zi ij == ji + 1 P and, for example, G (@ x) = ? w log (x ) ? n? wi log (xi ? xi ) satis es 2
w
2
2
w
+1
2
w
+1
2
w
+1
+1
+1
+1
w
1
2
1
1
i=1
@ 2 (Gw (@ x)) = ?wi+1 1 @xi @xj ln(2) (xi+1 ? xi )2 0
2
2 +1 2 +1
2
+1
when j = i + 1:
This gives two alternative proofs of Theorem 4.2, showing how such results can be derived more easily. Definition 4.8. A function F : 0; integer (1 i n): i=1
HUFFMAN CODING AS SUBMODULAR OPTIMIZATION OVER A LATTICE
25
Dropping the integrality constraint gives an interesting continuous relaxation of Hu man coding that can be attacked numerically. For example, by treating the constraint as a penalty function, the problem with ? P above can be solved numerically something like the P system of equations @=@`j ni=1 wi `i + 1010 ( 1 ? ni=1 2?`i )2 = 0 (1 j n): Using the example weight sequence w = h 189 95 73 71 23 21 18 9 1 i studied earlier, a simple program found a unique real solution ` h 1:4 2:4 2:8 2:8 4:4 4:8 4:8 5:8 9:0 i for these equations, with objective 1241. As expected, this solution is near the optimal Hu man code h 1 3 3 3 5 5 5 6 6 i, with cost 1276. When the relaxation is faithful to the original, it will be possible to nd optimal solutions quickly. The relaxed solution can be used to jump to the right neighborhood in the imbalance lattice, from which balancing exchanges will walk to the optimal code. The penalty function could clearly be varied, and perhaps could be changed to encourage near-integral solutions. Interior point methods on the majorization lattice may? also be possible. Among other things, it may ?be possible to de ne s ^ t in terms of ?log2 2?s u 2?t and de ne s _ t in terms of ?log2 2?s t 2?t : they are often identical, and always satisfy 2?s ^ t 2?s u 2?t ; 2?s t 2?t 2?s _ t (because 2?s u 2?t and 2?s t 2?t are the glb and lub with respect to majorization). For perspective, if = 7 ? log 2 (12) 3:4150375 and =? ( ? 1), the following set of examples represent the unusual cases with n = 9 where ?log2 2?s u 2?t 6= (s ^ t) or ? ? s ? t ?log 2 2 t 2 6= (s _ t):
?
?log 2?s u 2?t h133 55555i h222 55555i h22 444455i ? ?log 2?s t 2?t h14 345677i h14 346666i h14 355566i h14 444566i h14 445555i
s^t h133445555i h222445555i h223344455i s_t h133345677i h133346666i h133355566i h133444566i h133445555i ? These examples suggest there may? be algorithms that `round up' ?log 2?s u 2?t to give s ^ t, and `round down' ?log 2?s t 2?t to give s _ t. s
h124555555i h124555555i h134444455i s h144444444i h144444444i h144444444i h144444444i h144444444i
t
h133345677i h222345677i h222345677i t h222345677i h222346666i h222355566i h222444566i h222445555i
2
2
2
2
6.2. Practical Applications in Adaptive Coding. In many practical situations it is dicult or impossible to know a priori the weights w used in Hu man coding. A natural idea, which occurred independently to Faller [8] and Gallager [11], is to allow the weights to be determined dynamically, and have the Hu man code `evolve' over time. Dynamic Hu man coding is the strategy of repeatedly constructing the Hu man code for the input so far, and using it in transmitting the next input symbol. Knuth presented an ecient algorithm for dynamic Hu man coding in [22], and his performance results for the algorithm show it consistently producing compression very near (though not surpassing) the compression attained with static Hu man code for the entire input. Vitter [40, 41] then developed a dynamic Hu man algorithm that improves on Knuth's in the following way: rather than simply revise the Hu man tree after each P input symbol, Vitter also nds a new Hu man tree of minimal external path length i `i and height maxi `i . With this modi cation Vitter was actually able to surpass the performance of static Hu man coding on several benchmarks. A small contribution we can make is to clarify the improvement of Vitter. Basically, Vitter's algorithm di ers from Knuth's in constructing the optimal path-length sequence P that is also as balanced as possible. Note that minimizing the external path length i `i is
26
D.S. PARKER AND P. RAM
identical to maximizing the level of balance. Since there can be more than one optimal code, and unnecessary imbalance tends to penalize the symbol currently being encoded, insisting on maximally balanced codes improves performance. Another contribution of the lattice perspective here is to encourage development of new adaptive coding schemes. As suggested in Section 5.1, a move between adjacent points in the lattice corresponds to minor alteration of codes, and by moving through the lattice we incrementally modify the cost of a code. Hill-climbing then gives greedy coding algorithms, and on-line hill-climbing gives adaptive coding algorithms. Although we have shown that the codes produced by hill-climbing are not guaranteed to be optimal, lattice-oriented adaptive coding algorithms may still have a role to play in some coding situations, since the Hu man notion of optimality is not really what is needed in the (currently popular and enormously important) adaptive context. For example, adaptive coding algorithms can start at any point in the lattice, as long as both ends of the communication know which one. Rather than rely on the dynamic Hu man algorithm to derive reasonable operating points for the code, or rely on Knuth's `windowed' algorithm [22], one can immediately begin with a mutually-agreed-upon, `reasonable' initial code (depending on the type of information being transmitted), and then adapt this code using some mutually-agreed-upon greedy algorithm for moving in the imbalance lattice.
Acknowledgements. We are very grateful to Pierre Hasenfratz for insightful comments that improved this paper. A conversation with Mordecai Golin, who provided us with an expanded version of [14], inspired us to discuss dynamic programming explicitly in this paper. He also pointed out the survey [4] to us. Also, we are indebted to two anonymous referees for clari cations of the exposition, especially of the signi cance of submodularity and of Shannon's work [38]. REFERENCES [1] J. Abrahams, Code and Parse Trees for Lossless Source Encoding, Proc. Compression & Complexity of Sequences (SEQUENCES'97), Positano, Italy, 1997, IEEE Press, to appear. [2] A. Aggarwal, A. Bar-Noy, S. Khuller, D. Kravets, B. Schieber, Ecient Minimum Cost Matching and Transportation using the Quadrangle Inequality, J. Algorithms, 19:1 (July 1995), pp. 116{143. [3] A. Berman, R.J. Plemmons, Nonnegative matrices in the mathematical sciences, SIAM, 1994. [4] R.E. Burkard, B. Klinz, R. Rudolf, Perspectives of Monge properties in optimization, Discrete Applied Math., 70 (1996), pp. 95{161. [5] B.A. Davey, H.A. Priestley, Introduction to Lattices and Order, Cambridge U. Press, 1990. [6] J. Edmonds, Submodular Functions, Matroids and Certain Polyhedra, in Combinatorial Structures and their Applications, R. Guy et al., eds., Gordon & Breach, 1970, pp. 69{87. [7] D. Eppstein, Z. Galil, R. Giancarlo, G.F. Italiano, Sparse dynamic Programming. II. Convex and concave cost functions, Journal of the ACM, 39:3 (July 1992), pp. 546{567. [8] N. Faller, An adaptive system for data compression, Paci c Grove, CA: Record of the 7th Asilomar Conference on Circuits, Systems, and Computers, 1973, pp. 593{597. [9] S. Fujishige, Submodular functions and optimization, North-Holland Elsevier, 1991. [10] R.G. Gallager, Information Theory and Reliable Communications, J. Wiley, 1968. [11] R.G. Gallager, Variations on a theme by Hu man, IEEE Trans. Information Theory, IT-24:6 (November 1978), pp. 668{674. [12] E.N. Gilbert, Codes Based on Inaccurate Source Probabilities, IEEE Trans. Information Theory, IT-17:3 (May 1971), pp. 304{314. ( ) is analyzed on p. 309. [13] C.R. Glassey and R.M. Karp, On the optimality of Hu man trees, SIAM J. Appl. Math., 31 (1976), pp. 368{378. [14] M.J. Golin, G. Rote, A dynamic programming algorithm for constructing optimal pre x-free codes for unequal letter costs, Proc. ICALP 95, Z. Fulop, F. Gecseg, eds., Springer-Verlag, 1995, pp. 256{267. g N
HUFFMAN CODING AS SUBMODULAR OPTIMIZATION OVER A LATTICE
27
[15] R.L. Graham, Applications of the FKG Inequality and its Relatives, in MathematicalProgramming: The State of the Art, B. Korte, A. Bachem, M. Grotschel, eds., Springer-Verlag, 1983, pp. 115{131. [16] G.H. Hardy, J.E. Littlewood, G. Polya, Inequalities, Cambridge University Press, 1934. [17] A.J. Hoffman. On Simple Linear Programming Problems, in Convexity, Proc. Seventh Symposium in Pure Mathematics, Vol. VII, V. Klee, ed., AMS, 1961, pp. 317{327. [18] D.A. Huffman, A method for the construction of minimum redundancy codes, Proc. IRE, 40 (1951), pp. 1098{1101. [19] F.K. Hwang, Generalized Hu man Trees, SIAM J. Appl. Math., 37 (1979), pp. 124{127. [20] C.M. Klein, A submodular approach to discrete dynamic programming, European J. Operational Research, 80:1 (Jan. 1995), pp. 145{155. [21] D.E. Knuth, Optimum Binary Search Trees, Acta Informatica, 1 (1971), pp. 14{25. [22] D.E. Knuth, Dynamic Hu man Coding, J. Algorithms, 6 (1985), pp. 163{180. [23] E. Lawler, Combinatorial Optimization: Networks & Matroids, Holt-Rinehart-Winston, 1976. [24] E.L. Lawler, Submodular Functions & Polymatroid Optimization, in Combinatorial Optimization: Annotated Bibliographies, A.H.G. Rinnooy Kan, M. O'hEigeartaigh, J.K. Lenstra, eds., J. Wiley & Sons, 1985, pp. 32{38. [25] L. Lovasz, Submodular functions and convexity, in Mathematical Programming: The State of the Art, B. Korte, A. Bachem, M. Grotschel, eds., Springer-Verlag, 1983, pp. 235{257. [26] U. Manber, Introduction to Algorithms, Addison-Wesley, 1989. [27] A.W. Marshall, I. Olkin, Inequalities: Theory of Majorization and Its Applications, Academic Press, 1979. [28] H. Narayanan, Submodular functions and electrical networks, North-Holland Elsevier, 1997. [29] A. Ostrowski, Sur quelques applications des fonctions convexes et concaves au sens de I. Schur (o ert en homage a P. Montel), J. Math. Pures Appl., 31 (1952), pp. 253{292. [30] J.M. Pallo, Enumerating, ranking and unranking binary trees, Computer Journal, 29:2 (April 1986), pp. 171{175. [31] J.M. Pallo, Some properties of the rotation lattice of binary trees, Computer Journal, 31:6 (Dec. 1988), pp. 564{565. [32] D.S. Parker, Conditions for Optimality of the Hu man Algorithm, SIAM J. Comput., 9:3 (August 1980), pp. 470{489. [33] D.S. Parker, P. Ram, Greed and Majorization, November 1994. Issued as Technical Report CSD-960003, UCLA Computer Science Dept., March 1996. [34] D.S. Parker, P. Ram, A Linear Algebraic Reconstruction of Majorization, Technical Report CSD-970036, UCLA Computer Science Dept., September 1997. [35] U. Pferschy, R. Rudolf, G.J. Woeginger, Monge matrices make maximization manageable, Operations Research Letters, 16 (1994), pp. 245{254. [36] G.-C. Rota, On the Foundations of Combinatory Theory I. Theory of Mobius Functions, Z. Wahrscheinlichkeitstheorie, 2 (1964), pp. 340{368. eine Klasse von Mittelbildungen mit Anwendungen auf die Determinantenthe[37] I. Schur, Uber orie, Sitzungsber. Berl. Math. Ges., 22 (1923), pp. 9{20. [38] C.E. Shannon, The Lattice Theory of Information, Proc. IRE Trans. Information Theory, 1 (1950). Reprinted in Claude Elwood Shannon: Collected Papers, IEEE Press, 1993. [39] N.J.A. Sloane, S. Plouffe, The Encyclopedia of Integer Sequences, Academic Press, 1995. [40] J.S. Vitter, Design and Analysis of Dynamic Hu man Codes, J. ACM, 34:4 (October 1987), pp. 825{845. [41] J.S. Vitter, Algorithm 673: Dynamic Hu man Coding, ACM TOMS, 15:2 (June 1989), pp. 158{167. [42] F.F. Yao, Ecient dynamic programming using quadrangle inequalities, Los Angeles, CA: Proc. 12th Annual ACM Symp. on Theory of Computing, 1980, pp. 429{435.
