Inferring Causal History from Shape - CSJ Archive
COGNITIVE
SCIENCE
13.357487
(1989)
Inferring CausalHistory from Shape MICHAEL LEYTON Rutgers University
The shape
of on object
often
seems
to tell
us something
about
the object’s
that is, the processes of growth, pushing, stretching, resistance, so on, that formed the object. A theory is offered here of how
history;
indentation, ond people are able to
infer the causal history of natural objects such as clouds, tumors, embryos, leoves, geological formations, and the like. Two inference problems are examined: the first is the inference of causal history from a single shape. It is claimed that this inference consists metry and curvature
of two simple and yet powerful structure of the shapes. When
a large collection remarkably well Intervening two stages fested, of the that from
of shapes, it is found that with intuition. The second
a grammar the earlier
Finally, relations detail.
a basic between
when a doctor compares and is able to conjecture
of only six operations one vio psychologically heuristic different
by which
shapes that are known to be This type of inference is mani-
two X-rays, the intervening
taken a month apart, growth. It is found
is sufficient to generate the later meaningful process-extrapolations.
human
processes
based upon the symtwo rules are applied to
the resulting causal histories accord inference problem is the recovery of
causal history between two successive in the development of the same object.
for example, same tumor,
rules, these
beings
in the past
seem
to infer
is proposed,
complex
shape
temporal
and investigated
in
1. Introduction The shapeof an object often seemsto tell us somethingabout the object’s history; that is, the causalprocessesthat formed it. For example,the shape in Figure 1 can be understoodas the result of various processesof growth, pushing, pulling, resistance,and so on. This type of understandingis, in fact, a remarkablephenomenon.A shapeis simply a singlestate, a frozen moment, a stepoutsidethe flow of time; and yet it can beusedas a window into the past. How is this possible? The author gratefully acknowledges the receipt of a Presidential Young Investigator Award (NSF IRI-88%110), part of which supported the writing of this article. Prior preliminary research was supported by NSF grants ET-8418164 and IST-8312240; and AFOSR grant F4%20-83-C-0135. Correspondence and requests for reprints should be sent to Michael Leyton, Department of Psychology, Busch Campus, Rutgers University, New Brunswick, NJ 08903. 357
LEMON
Flguro
1. A natural
shape
It will be argued here that people apply two very simple rules to a shape in order to recover a remarkably rich picture of the history of an object. The objects that will be considered are natural ones, such as leaves, tumors, clouds, and embryos. This is because such objects have much more complicated shapes than manufactured objects, which are usually the simple concatenation of box-like elements. Natural objects grow and evolve; and it is this history that is seen in them. Artificial objects never grow and evolve; and are thus seen as dead, unless they were specifically designed to mimic a sense of development. This article will examine two inference problems. The first is the inference of history from a single shape. As indicated above, this inference is comprised of two simple but powerful rules. The second inference problem is this: When given two shapes which are assumed to be the same object at two different stages of development, people seem able to infer the history that occurred between the two stages. For example, a doctor will examine two X-rays of a tumor, taken a month apart, precisely because he or she is able to infer the intervening history of the tumor. In order to see how this inference problem is solved, a grammar will be elaborated that generates the later shape via a series of developmental stages, from an earlier shape. Finally, it will be seen that in both the inference problems, people are capable not only of inferring causal processes in the past of some object, but also of inferring complex temporal relationships between those processes. The last part of this article will give a detailed analysis of the inference of these relationships.’ The discussion begins with the inference of history from a single shape. The two inference rules to be proposed, are based upon two simple structural features, curvature and symmetry. These two features will first be briefly described before proposing the two inference rules. I The first part of this article gives a cognitively oriented, nonmathematical exposition of work, which can be found in a much less cognitive, and much more technical version in Leyton (1988). The second part of the article consists of new material on the inference of complex temporal relations, and elaborates a new status for the process-grammar with respect to the inference of history from a single shape.
E w INFERRING
Figure point
2. A sequence of lines of increasing E, at which there is greater curvature
CAUSAL
curvature (bend)
HISTORY
359
(left to right). The far right one has a than at the other points on that line.
2. Curvature In what follows, it will be argued that curvature variation is a crucial means by which a person concludes that processes have taken place on a shape. What exactly is curvature variation? Observe first that curvature is the amount of bend. For example, consider the series of lines in Figure 2. The far left line has no bend and thus is regarded as having zero curvature. The amount of bend (curvature) increases in the progression of lines from left to right. Finally, observe that the right-hand curve has a special point E where the curvature on that line is greatest. That is, at any other point on that line, the curve is flatter, and has less bend, than at E. Thus E is called a curvature extremum. Curvature extrema have been regarded as having two important roles in shape perception: Attneave (1954) showed that such extrema are the most frequently used points in creating a visual summary of a shape; and Hoffman and Richards (1984) have proposed that particular kinds of curvature extrema define the endpoints of the perceptual parts of a shape. In this article, it will be argued that curvature extrema have a third psychological role: They are crucial to the inference of processes. 3. Symmetry It will also be argued that process-inference is intimately related to symmetry axes. A symmetry axis is usually defined to be a straight line along which a mirror will reflect one half of a figure onto the other. However, observe that, in complex natural objects, such as Figure 1, a straight axis might not exist. Nevertheless, one might still wish to regard the figure, or part of it, as symmetrical about some curved axis. For example, a branch of a tree tends not to have a straight reflectional axis. Nevertheless, one understands the branch to have an axial core that runs along its center. How can such a generalized axis be constructed? There are good mathematical reasons to proceed in the following way (Blum, 1973; Brady, 1983; Leyton, 1988). Consider Figure 3. It shows two curves cl and cz (the bold curves), which can be understood as two sides of an object. Now introduce
860
LEYTON
Figure
3. A rymmelry
axIs created
by the traiectory
of polntr
Q
a circle that is tangentialsimultaneouslyto the two curves,as shown.The two tangentpoints A and B arethen definedassymmetricalto eachother. Now move the circle continuouslyalongthe two curvescr and cs; while alwaysmaintainingdoubletouching. Onecan then defme a symmetryaxis to be the trajectory of the midpoint Q of the arc AB, asthe circle moves. For example,in Figure 3, the trajectory of Q is representedby the locus of dots shown. This definition was proposedin Leyton (1988),whereit was arguedthat it is more suitedfor the particulartask of process-inference than are other symmetry analyscss, suchas thoseproposedby Blum (1973)and Brady (1983).The intuitively satisfying resultsof this alternativeanalysis will be seenlater. 4. Symmetry-CurvatureDuality Symmetryand curvatureare two very different descriptorsof shape.However,a theoremproposedand provedin Leyton (1987b)showsthat thereis an intimate relationshipbetweenthesetwo descriptors.This relationship will be the basisof the entire article: SYMMETRY-CURVATURE DUALITY THEOREM (Leyton,1987b. p. 329): Any section of curve, that has one and only one curvature ewemum, has one and only one symmetry axis. This axis isforced to terminate at the extremum itself.
To illustrate: Considerthe shapewith which the article began.It is presentedagainin Figure4. The shapehas 14curvatureextrema.Thus, by the abovetheorem,thereare 14uniquesymmetryaxesassociatedwith, andterminating at, the extrema.Thesearethe dottedlines shownin Figure 4. 5. SymmetryAxes and Processes As skited earlier, thi reasonfor involving symmetry axesis that it will be argued&at they arecloselyrelatedto processes. This proposedrelationship is givenby the -followingprinciple:
INFERRING
CAUSAL
HISTORY
361
...* .*.. .. .-1.*.** :: :. .:....*.*. Gx : ‘.. . Figure 4. The natural shape curvature duality theorem
of Figure
1, with
the symmetry
axes
predicted
by the symmetry-
IN’IERACTION PRINCIPLE (Leyton, 1984, p. 149): The symmetry ayes of a perceptual organization are interpreted as the directions along whichpro&s.ses are most likely to act or have acted.
The principle was advancedand extensivelycorroboratedin Leyton (1984,1985,1986a,1986b,1986c,1987a,1987b,1987c)in severalareasof perception,including motion perceptionas well as shapeperception.The argumentusedin Leyton (1984,1986b)to justify the principle,involvesthe following two steps:(1)A processthat actsalonga symmetryaxis tendsto preservethe symmetry,that is, to be structurepreserving.(2)Structure-preservingprocessesare perceivedas the most likely processesto occur or to haveoccurred. 6. The Inference of Processes
The tools requiredto understandhow processesare recoveredfrom shape arenow in hand.In fact, the systemto beproposedconsistsof two inference rulesthat areappliedsuccessively to a shape.The rulescanbeillustratedby returningto Figure4. The first rule is the symmetry-curvatureduality theorem which statesthat, to eachcurvatureextremum,thereis a uniquesymmetry axisterminatingat that extremum.The secondrule is the interaction principle,which statesthat eachof the axesis a directionalongwhich a processhasacted.The implication is that the boundarywasdeformedalongthe axes;for example,eachprotrusionin Figure4 was the resultof pushing out along its axis, and eachindentationwas the result of pushing in along its axis. In fact, eachaxis is the truce or record of boundarymovement! Under this analysis,processesare understoodas creatingthe curvature extrema;for example,they introduceprotrusionsand indentationsinto the shapeboundary.This meansthat, if onewereto go backwardsin time, undoingall theinferredprocesses, onewould eventuallyremoveall theextrema. Observethat thereis only oneclosedcurvewithout extrema:the circle.Thus the implication is that the ultimate startingshapemust havebeena circle, andthis wasdeformedundervariousprocesses,eachof which producedan extremum.
362
LEYTON
7. Corroborating
Examples
To seethat thesetwo rulesconsistentlyyield satisfyingprocess-explanations, the explanationsthat theserules give for a large set of shapeswill be examined.Considerthe setof all possibleshapeswhich have8 extremaor less. Fiiure 5 presentsall theseshapes(asgivenby Richards,Koenderink,& Hoffman, 1987),and superimposes,on eachshape,the processesthat areinferred underthe inferencerules.The arrowsarethe inferredprocesses.As the readercan see,the resultsaccordremarkablywell with intuition. In orderto preparefor the restof the article, two observationsshouldbe made.First, the setof shapesdividesinto threelevelsaccordingto the number,of extremainvolved.LevelI shapeshave4 extrema,LevelII have6, and Level III have8. It hasbeenshownmathematicallythat no other possibilities exist with 8 extremaor less(Leyton, 1988). Second,it will now be seenthat, although only structural aspectsof shapehavebeenconsideredso far, somestrongsemantic constraintsemerge from theseaspects.To seethis, note first that, on eachshape,eachof the curvatureextremais labeledby oneof the four symbolsM, + m, - m, + or M. - Thesesymbolscorrespondto the four typesof extremathat canexist. They areillustratedin Figure 6. Undereachsymbolin the figure, thereis a curve.The dot on eachcurverepresentsthe curve’sextremum.The shading in eachcasemarks the sideof the curveupon which the body of the shape exists.The four extrematypesare characterizedin the following way. The dots on the first and secondcurvesare the points wherethosecurvesbend the most. The dots on the third and fourth curvesare the points wherethe curvesbendthe least; that is, the points wherethosecurvesarethe flattest. The shadingshowsthat the first and secondcurvesare figure-groundreversalsof eachother. Similarly, the third and fourth curvesare figure-ground reversalsof eachother.’ Thesearethe only four kindsof extremawhich can exist. The important point to observenow is that, surveyingthe process-analysesshownin Figure5, one finds that the four typesof extremacorrespond consistentlyto four typesof process: SEUANTIC
INTERPRETATION
RULE
M+ protrusion m- indentation m+ squashing
M- internalresistance 1 The symbols, M,+ m,- m,+ M,- are ch osen because they refer, respectively, to positive maxima, negative minima, positive minima, and negative maxima, of the curvature function. The convention for the specification of curvature on a planar curve is that one goes around the curve such that the inside of the figure is to the left of the curve: and one measures positive curvature as the rate of one’s anticlockwise rotation, and negative curvatwe as the rate of one’s clockwise rotation.
INFERRING
CAUSAL
HISTORY
363
That is, the purely structuralcharacterizations,M, + m, - m, + M, - correspond to semantic terms people useto describethe associatedprocesses. To seehow powerful this effectis, the readershouldlook at severalextrema on the shapesin Figure5, andcomparethem with the abovesemanticcharacterizations. 8. The Generality of Processes
How do peopleperceptuallydefineapart of a shape?Hoffman andRichards (1984)havearguedthat a part perceptuallycorrespondsto a segmentbetween two consecutivem- extremaalongthe shape’soutline. To illustrate: In the handshownin Figure 7, the black dots on the outline indicatethe m- extrema. Any segment,whoseendpointsaretwo consecutivedots,is a finger, that is, a perceptualpart of the shape. It shouldnow beobservedthat Hoffman andRichards’theoryis purelya segmentation theoryof parts;that is, partsaredefinedby their endpoints.It will now be arguedthat the process-analysis here givesa more cognitive view of what a part is. Note first, that betweenany two consecutivem- extremain Figure 7, thereis a M+ extremum:the tip of the finger (e.g.,this extremumhasbeen marked on the far left finger). Recall now that the symmetry-curvature duality theoremstatesthat, becausethereis only one extremum(M+) on the segmentbetween’ two consecutivem- extrema,therecan be only one axis on that segment.Furthermore,the theoremstatesthat the axis must terminateat the h4+ extremum.For example,in the caseof a finger,thereis only one axis, and this axis terminatesat the tip of the finger. The secondrule of the process-analysis, the interactionprinciple, then leadsto the crucialconclusionthat the axis is the trace of a process.The importanceof this conclusionis that it implies that a part is not a rigid segment: A part is a process! It is a temporal or carcsal concept. Thus, an important differenceexistsbetweenthe Hoffman & Richards analysisand the process-analysis: The former givesa descriptionof a part simply asa segment, whereasthe latter views a part asa causal explanation. It is arguedherethat causalexplanationis centralto the cognitiverepresentation of shape.That is, one cannothelp seeingan embryo,tumor, cloud, geologicalformation, and so on, as a consequence of historical processes. Thereis anotherimportant differencebetweentheHoffman andRichards segmentationapproachand the process-approach here: accordingto the process-approach, a part is a processterminating at a M+ extremum.By the semanticinterpretationrule (section7), a part must thereforebe a protrusion. However,as Figure 6 shows,a M+ extremumis only one of four typesof extremathat exist. Furthermore,since,by this analysis,processes arehypothesizedto explainextrema,theremust be four timesasmany processesasthereareparts. In fact, by the semanticinterpretationrule, the full
LEVEL
I
M+++ M&M+ P2
LEVEL
1 t v
M+
---M
P3
II
rvl+ hA+ m+ 0
TI
T4
364
M
T3
T2
y+ M’ CT5 m+ t
m+
!A+
gM+ T5
yk%qwJ T6
LEVEL
m
t
Q3
Q4
Q7
Q9
flgure 5. The process-histories inferred by the two inference rules applied to the Richards, Koenderink, ond Hoffmon (1987) drowings of all shopes with up to 8 extremo. 365
LMON
tvl+
Flgun
Figure
6. The
7. The
four
dots
types
show
of curvature
the m-
extrema
extreme
of the hand
INFERRING
CAUSAL
HISTORY
367
collection of processes associated with extrema are protrusions, indentations, squashings,and resistances.Parts correspond to only the first of these categories. It is concluded therefore, that the process-approach to parts is cognitively deeper than a straightforward segmentation approach for two reasons: (1) The process-approach considers parts to be causal explanations. It is claimed here that such explanations are central to the cognition of shape. (2) The process-approach accounts not only for parts, but, since there are four times as many processes as parts, the process-analysis accounts for many more phenomena than parts. 9. The Second Inference Problem Until now the issue of how one infers processes from a singfeshape has been considered. Sections 9-13 will examine a different problem: Suppose one is given two views of an object (e.g., an embryo), at two different stages of development. How can one infer the intervening process-history? The method of solving this problem will be to develop a grammarthat generates the second shape from the first via a sequence of plausible developmental stages. Observe now that, since the later shape is assumed to emerge from the earlier shape, one will wish to explain it, as much as possible, as the outcome of what can be seen in the earlier shape. In other words, one will wish to explain the later shape, as much as possible, asthe extrapolationof what can be seen in the earlier one. As a simple first cut, divide all extrapolations of processes into two types: (1) chltinuatioIls (2) Bifurcations (i.e., forkings).
The only forms that these two alternatives can take will now be elaborated: first, continuations, and then, bifurcations. 10. Continuations Consider any one of the M+ extrema in Figure 8. It is the tip of a protrusion, as predicted by the semantic interpretation rule. What is important to observe is that, if one continued the process creating that protrusion, that is, continued pushing out the boundary in the direction shown, the protrusion would remain a protrusion. The M+ extremum would remain a M+ extremum, the form shown at the far left of Figure 6. This means that continuation at a M+ extremumdoes not structurally alter the boundary. Exactly the same argument applies to any of the m- extrema in Figure 8. Continuation of an indentation remains an indentation, a m- extremum will remain a m- extremum.
LEYTON
,\ / . m’ m> m/y2
i c MC Figure
0. One
of the
shapes
token
M+
from
Figure
5
Now recall that there are four types of extrema, A4,+ m, - m, + M. - It hasbeenseenthat continuationsat the first two do not structurally alter the, shape.However, it will now be shown that continuationsat the secondtwo do causestructural alterations. Considerthesetwo casesin turn. Continuation at m +. A m + extremum occursat the top of the left-hand shapein Figure 9. In accord with the semanticinterpretation rule (Section 7), the processterminating at this extremum is a squashingprocess.Now continue this process;that is, continue pushing the boundaryin the direction shown. At somepoint, an indentation will be created,as shownin the top of the right-hand shapein Figure 9. Observewhat happensto the extremainvolved. Before continuation, as in the left-hand shape,the relevantextremum is m + (at the top). After continuation, as in the right-hand shape,this extremumhaschangedto m- . In fact, observethat a dot hasbeenplacedon eithersideof the m- on the curve itself. Thesetwo dots are points wherethe curve, locally, is completelyflat;
TI Figure
T2 9. Continuation
at m+:
squashing
continues
until
it indents
INFERRING
CAUSAL
(p 10. Continuation
369
M+&M+ t
13 Figure
HISTORY
T4 at M-:
internal
resistance
continues
until
it protrudes
there is 0 (zero) curvature. Therefore the top of the right-hand shape is given by the sequence Om- 0. Thus the transition between the left-hand shape and the right-hand shape can be structurally specified by simply saying that the M + extremum, at the top of the first shape, is replaced by the sequence Om - 0, at the top of the second shape. This transition will be labeled Cm + , meaning Continuation at m+ . That is, Cm+ : m+ - Om-0 Finally, observe that, although this operation is, formally, a rewrite rule on discrete strings of extrema, the rule actually has a highly intuitive meaning. Using the semantic interpretation rule, the meaning is: squashingcon-
tinues until it indents. Continuationat M- . As noted earlier, only one other type of continuation needs to be considered, that at a M- extremum. In order to understand what happens here, consider the left-hand shape in Figure 10. The symmetry analysis of section 3 describes a process-structure for the indentation that is very subtle: There is a flattening of the lowest region of the indentation because the downward arrows, within the indentation, are countered by an upward arrow, which is within the body of the shape and terminates at the M- extremum. This latter process is an example of what the semantic interpretation rule (section 7) calls internal resistance.The overall shape could be that of an island where an inflow of water (into the indentation) has been resisted by a ridge of mountains (in the body of the shape). The consequence is the formation of a bay. Now, recall that the interest here is to see what happens when one continues a process at a M- extremum. Continue the M- process upward, in the left-hand shape of Figure 10; at some point, the process will burst out and create the protrusion shown at the top of the right-hand shape in Figure 10. In terms of the island example, there might have been a volcano in the mountains which erupted and sent lava down into the sea.
LEYTON
t T4
Q6 Figure
11. Bifurcation
at M+:
a nodule
becomes
a lobe
Observenow what happensto the extremainvolved. Before continuation, that is, in the left-hand shape,the relevantextremumis M-, (in the centerof thebay). After continuation,asin the right-handshape,the extremum has changedto M+ (at the top of the protrusion). In fact, observe that, once again, a dot has beenplacedon either side of the M, + on the curveitself. Thesetwo dots represent,asbefore,points wherethe curveis, locally, completelyflat, the curvehas0 curvature.Thereforethe top of the right-handshapeis givenby the sequenceOM+0. Thereforethe transition betweenthe left-hand and right-hand shapes canbe structurallyspecifiedby simply sayingthat theM- extremum,in the first shape,is replacedby the sequenceOM+O, at the top of the second shape.This transitionwill belabeledCM-, meaningcontinuationat M- , or CM- : M’
- OM+O
Finally, observeonce again, that although this operation is a formal rewriterule on discretestringsof extrema,the rule actuallyhas a highly intuitive meaning. Using the semanticinterpretationrule, the meaning is: internal resistancecontinuesuntil it protrudes. 11. Bifurcatiqn
As seenabove,process-continuation cantake only two forms. The ptu$ose now is to elaboratethe only forms which the bifurcation (branching)of a processcantake. Note that, becausethereare four extrema,bifurcation at eachof thesefour must be examined. Bifurcation at Mt. Considerthe M+ extremumat the’top of the lefthandshapein Figure 1I, andthe protrudingprocessterminatingat this extremum. What would’resultif this processbifurcated?Under bifurcation,
INFERRING
CAUSAL
371
HISTORY
P2
T3 Figure
12.
Bifurcation
at t-n-:
an inlet
becomes
a bay
one branchwould go to the left and the other to the right; the branching would createthe upperlobe in the right-handshape. Observewhat happensto the extremainvolved.Beforesplitting, onehas theM+ extremumat thetop of the first shape.In the situationaftersplitting (i.e., in the secondshape),the left-hand branchterminatesat a M+ extremum, and so doesthe right-hand branch.The M+ extremumin the first shapehas split into two copiesof itself in the secondshape.In fact, for mathematicalreasons,a newextremumhasto be introducedbetweenthese two M+ copies.It is them+ shownat the top of the lobeof the secondshape. Therefore,in terms of extrema,the transition betweenthe first and second shapescanbeexpressed thus: The M+ extremumat the top of thefust shape is replacedby the sequenceM+m+M+ alongthe top of the secondshape. The transition is M+ - M+m+M, + and will be labeledBM, + meaning bifurcation at M, + or BM+
: M+
-M+m+M+
Finally, althoughthis transition hasjust beenexpressedas a formal rewrite rule on discretestrings of extrema,the transition has, in fact, the following highly intuitive meaning:a nodule develops into a lobe. Bifurcation at m-. Considernow the m- extremumat thetop of thelefthand shapein Figure I2, and considerthe indentingprocessterminatingat this extremum. If this processbifurcates,one branchwould go to the left and the other to the right. The branchingwould createthe bay in the righthand shape. Observeagain what happensto the extremainvolved. Before splitting, onehasthe singleextremumm- in the indentationof the frostshape.In the situation after splitting (i.e., the secondshape),the left-handbranchterminatesat a m- extremum,and so doesthe right-handbranch. The top min the first shapehasbeensplit into two copiesof itself in the secondshape.
372
LEYTON
Again, for mathematical reasons, a new tween these two m- copies. It is the Mthe second shape. Therefore, in terms of first and second shapes can be expressed shape is replaced by the sequence m-M-m ition is given by the rule m- -m -M-m, ing bifurcation at m, - that is, Bm-
: m-
extremum has to be introduced beshown at the center of the bay in extrema, the transition between the thus: The m- extremum in the first - in the second shape. The trans- and will be labeled Bm, - mean-
- m-M-m-
Finally, although the transition has just been expressed as a formal rewrite rule using discrete strings of extrema, the transition has, in fact, a highly intuitive meaning: an inlet develops into a bay. Bifurcation at m + . The above two types of bifurcation illustrate the general form that bifurcations take: an extremum, E, is split into two copies of itself. Furthermore, for mathematical reasons, an intervening extremum, e, of a different type has to be introduced between the two copies.’ Therefore, bifurcation always takes the form E - EeE. Consider now bifurcation at m. + By the discussion in the previous paragraph, one can see, in advance, that a bifurcation at m+ must change the m + into the sequence m +M+ m + (two copies of m + separated by M+). The bifurcation is given by the rule m+ - m+M+m, + which will be labeled Bm + , meaning bifurcation at m, + that is, Bm+
: m+ - m+M+m+
This rule is illustrated in Figure 13. The m+ extremum, at the top of the left-hand shape, splits and becomes the two m+ extrema on either side of the right-hand shape. Simultaneously, a M+ extremum is necessarily introduced at the top of the right-hand shape. Since a M+ is always a protrusion, one can characterize this bifurcation simply as: a protrusion is introduced. Bifurcation at M-. Consider now bifurcation at M. - Again, from the above considerations, one can deduce, in advance that a bifurcation at Mmust change the M- into the sequence M-m-M(two copies of M- separated by m -). The bifurcation is given by the rule M- - M-m-M, - which will be labeled BM, - meaning bifurcation at M, - that is, BM-
: M-
- M-m-M-
’ It is sufficient to consider cases only where e is the same sign as E. This is because the other casescan be generated from these via the continuation rules given earlier. Thus. for each E, there is a unique e.
INFERRING
CAUSAL
HISTORY
373
h4+ L,+ t fn+/ hA+ 0J m+\M+ t PI
TI Figure
13.
Bifurcation
at m+:
a protrusion
is introduced
This rule is illustrated in Figure 14. The M- extremum,at the centerof the bay in the left-hand shape,splits and becomesthe two M- extremaon either side of the lagoon in the right-hand shape.Siulataneously, a mextremumis necessarilyintroducedat the lowestpoint of the lagoon(in the right-handshape).Sincem- is alwaysan indentation,onecancharacterize this bifurcation simply as: an indentation is introduced. 12. The Complete Grammar
Recall that the problem being examinedin this part of the article is this: Given two views of an object (e.g., a tumor), at two different stagesof development,how is it possibleto infer the intervening process-history? Sinceone wishesto explain the later shapeas an outcomeof the earlier shape,one will try to explainthe later shape,as much as possible,as the extrapolation of the process-structure inferable from the earliershape.It hasnow beenshownthat all possibleprocess-extrapolations are generated by only 6 operations:2 continuationsand4 bifurcations.Therefore,these6
T3
Q3. Figure
14. Bifurcation
at M+:
an indentation
is introduced
LMON
T6
Q5
flguro 15. Two
shapes
ossumed
to
be IWOstages in the history of
the some
object
operations form a grammar that will generate a later shape from an earlier one via processextrapolation. The grammar is as follows: PROCESS GRAMMAR Cm+ CMBM+ BmBm+ BM-
: m+ : M: M+ : m:m+ : M-
-
Oh-0 OM+O M+m+M+ m-M-mm+M+m+ M-m-M-
Recall however that, although these operations are expressed as formal rewrite rules on discrete strings of extrema, they describe six intuitively compelling situations, as follows: SEMANTIC INTERPRETATION OF THE GRAMMAR Cm+ CMBM+ BmBm+ BM-
: squashing continues until it indents. : internal resistance continues until it protrudes : a protrusion bifurcates; for example, a nodule becomes a lobe. : an indentation bifurcates; for example, an inlet becomes a bay.
: a protrusion is introduced. : an indentation is intmduced.
These six situations are illustrated in Figures 9-14. 13. A Compelling Ibtration The intuitive power of the grammar to yield process-explanations will now be illustrated, by considering an extended example. Suppose the two shapes in Figure 15 are two stages in the development of some entity such as a tumor or island. The process-grammar is used to give a complete and compelling account of the intervening history. The grammar provides a number of alternative process-routes from the first shape to the second. In Section 16, a procedure for choosing the most
INFERRING
CAUSAL
HISTORY
375
appropriate route will be described. Suppose that the route chosen by the procedure is the following sequence of operations: CM- followed by BM+ followed by Cm. + This sequence generates the sequence of shapes shown in Figure 16. The first and last shapes in this sequence are the two shapes that one is trying to bridge, and the whole sequence is the conjectured intervening history. The sequence of operations, CM- followed by BM+ followed by Cm, + is, of course, the inferred processexplanation. Given the semantic interpretation rules (sections 7 and 12), the explanation for Figure 16 can be converted into English thus: 1. The entire history is dependent upon a single crucial process. It is the internal resistance represented by the bold upward arrow in the first shape. 2. This process continues upward until it bursts out and creates the protrusion shown in the second shape. 3. This process then bifurcates, creating the upper lobe shown inthe third shape. Involved in this stage is the introduction of a squashing process given by the bold downward arrow at the top of the third shape. 4. Finally, the downward squashing process continues until it causes the new indentation in the top of the fourth shape. 14. Inferring Temporal Order In the previous section, the use of the grammar to provide a history between two shapes was illustrated. However it was noted in passing, that the grammar can provide a number of alternative histories between two shapes. Thus it is necessary to understand how one forms a single most plausible intervening history. To do this, it is necessary to examine how one infers temporal relations between processes along a history. These relations will determine the selection and order of the grammatical operations such that they will generate the optimal history. In the real world, the temporal relations between processes can be sequential, overlapping, or simultaneous. For example, in an embryo, the growth of the torso starts before that of the arms, but later overlaps the growth of the arms. Furthermore, the growth of one arm is exactly simultaneon with the growth of the other arm. Remarkably, it will be found that human beings naturally hypothesize such complex process-relationships. In fact, more remarkably, it will be seen that human beings use a very simple heuristic to give this complex history. In order to understand this heuristic, consider the first shape in Figure 17. It shows a map of Africa as one might come across it in an atlas. The subsequent shapes show successively more blurred versions of the original map. (The blurring sequence is denoted by the numbers 6 down to 1 in the figure.)
T6
Flgure
16. A viable
T5 path,
provided
by the grammar,
between
Q7 the two
shapes
in Figure
15
Q5
6
4
F
3
A 9
E
D
F
D
2
A v
0
cl
4
Figure 17. Six successive blurrings of the outline of Africa. token Mackworth (1986) Scale-based description and recognition of planar
from curves
sional shapes. IEEE Tronsacffons with permission. 01986 IEEE. features on the shapes.
Infelltgence. to indicate
on Pattern Analysis letters have been
and Machine superimposed
Mokhtarian and and two-dimen8, 34-43, particular 377
378
LEVTON
The sequenceof figurescomesfrom Mokhtarian & Mackworth (1986)who discusssmoothing in relation to the problem of recognizingobjects. Considerthe main structural featuresof the top map. Clearly, one of the most salient featuresof the map is the largeindentation shown at A. Look now at the 6 figures in the reversedorder, the order of deblurring. This sequencewill be referredto as Stages1 to 6, correspondingto the numbering in Figure 17.What is observedis that, in the deblurringsequence,the indentationat A emergesearly at Stage2, betweenthe two dots shownat that stage(the dots are flat points, points without curvature).If one tracks this indentation through the subsequentfigures (in the order 1 - 6), one finds that the indentationdeepensprogressively,until it reachesits full sizein the top map, Stage6. Another prominent featureof the top map is the protrusion shownat D. Consideragain its emergencein the deblurring sequence(1 - 6). This protrusion is slightly evidentat Stage1. Furthermore,it alsocontinuesto grow in the subsequentfigures,until it reachesits full sizein the top map (Stage6). Now considera lessprominent featureof the top map. Examination of this map revealsthat, within indentation A, thereis a smallerprotrusion. In the deblurringsequence(1 - 6), this protrusion emergesquite late in Stage 4, at the position marked B, betweentwo new dots. Becauseit is a smaller featurein the top map, it emergeslater in the deblurring sequencethan the more significant features.Subsequently,of courseit continuesto grow until it reachesits full sizein the top map. Observethat not only do theseprocessescontinueto grow in the deblurring direction, but they can also bifurcate. For example, consider once againthe indentation at A, asit emergesin Stage2. It continuesto grow in Stage3. However,in Stage4, it beginsto bifurcate to allow the protrusion, B, to emerge.This makesthe indentation A break into two copiesof itself, one at A and the other at C. Again, considerthe protrusion at D. In the transition from Stage1to Stage2, it bifurcatessubtly, into two protrusions, one at D and the other at F. The above discussionseemstherefore to lead to the following conclusion: Whendeblurring a shape,two things happen,(1)proceaesemergein the order of theirprominence,and (2) theprocessesgrow to theirfullsize by continuing andpossibly bifurcating. Becausedeblurring a contour causesa processto increaseto its full size, it seemsthat deblurring mimics the developmentalhistory of the process. There is, in fact, a deepreasonfor a correspondencebetweendeblurring and process-history.Recall from section6, it was arguedthat processesare conjecturedby human beingsto explainhow the curvaturevariationis introducedinto the boundary. That is, processesare conjecturedto haveshifted the boundary in the direction of increasedcurvaturevariation. What has been found aboveis that deblurring incrementallymovesthe boundary in
INFERRING
CAUSAL
HISTORY
379
the direction of increased curvature variation. Thus deblurring introduces what processes introduce. Another reason why deblurring gives a plausible recapitulation of process-history is that, because deblurring is a uniform means of incrementally introducing curvature variation, it is capable of embodying the following simple heuristic: SIZE-IS-TIME HEURISTIC. In the absence of informution to the contrary, size corresponds to time. That is, the larger the boundary movement, the more likely it is to have taken longer to develop.
This rule is valuable where one does not have any knowledge of the real world situation involved, when all that is available is the shape of the present state of the object. As stated at the beginning of this article, this is exactly the type of situation being investigated here. The concern is with the remarkable human ability to derive a history when no other knowledge is available apart from a shape in the present. One might argue, for example, that geologists now know that the outline of Africa was created by the breaking of continental land masses, rather than by boundary movement. The concern here however, is not with what actually happened in the environment, but with the human ability to see history in the very absence of knowing what actually happened. Nevertheless, one should note that the type of history considered here for Africa does indeed correspond to a standard type of geological scenario for the creation of an island: Inlets are produced by erosion from the sea, bays by the resistance of mountains, promontories by the outward advancement of a coastline, for example by volcanos, and so on. Deblurring mimics the processes that create boundary movement, but to conjecture that deblurring is a technique indeed used by human beings would require the supposition that human beings actually incorporate multiple levels of blurring into their perceptual representations; that human beings maintain sets of representations of the type shown in Figure 17. How valid is it to suppose this? Vision researchers are finding overwhelmingly that human beings represent the environment on multiple levels of blurring (Marr, 1982). In fact, the importance of such multiple-blurring hierarchies has emerged in the last decade as one of the major discoveries in vision (e.g., Koenderink, 1984; Mokhtarian & Mackworth, 1986; Pizer, Oliver, & Bloomberg, 1987; Rosenfeld, 1984; Terzopoulos, 1984; Witkin, 1983; Yuille & Poggio, 1986; Zucker & Hummel, 1986). What has been argued here, is that when such representations are applied to natural objects, one of their uses is to recapitulate process-history. Examine now the role of deblurring in relation to the two inference problems that are of concern in this article: (1) the inference of history from a single shape, and (2) the inference of intervening history between two given
LEYTON
380
shapes. These two problems will be considered in the next two sections, respectively. 15. Temporal Order and Inference from a Single Shape Recall that in order to solve the problem of inference from a single shape, two rules were proposed: (a) the symmetry-curvature duality theorem which assigns, to each extremum, a unique symmetry axis terminating at the extremum; and (b) the interaction principle which implies that each axis is the trace of a process. As a consequence of applying these two rules, diagrams such as those shown in Figure 5 were derived, which will be called process-diagrams. It can now be seen that process-diagrams do not contain a complete reconstruction of the process-history. They do not encode the temporal order of the processes; that is, which of the processes followed each other, which were overlapping, and which were simultaneous. Thus the process-diagrams must be supplemented with extra information. But how can this information be specified? Recall, from the Africa example, that, besides determining the order in which the processes first appeared, deblurring caused processes to continue and bifurcate until they reached their full size in the final shape. The crucial point to observe here is that earlier in this article, a means was developed by which the continuation and bifurcation of processes can be specified: the process-grammar. The grammar was in fact developed to solve the second inference problem: inference of intervening history between two shapes. However, it can now be seen that the grammar is also relevant to the first inference problem: inference from a single shape (e.g., the original map of Africa). It will specify the structural changes that occurred when processes continued and bifurcated in the history leading up to that shape. Thus the extra information that is needed to supplement process-diagrams, such as those in Figure 5, are operations from the process-grammar. To illustrate, return to the Africa example; it conforms to the first of the two inference problems, the inference of history from a single presented shape. The single shape is, of course, the original map (the top map in Figure 17), as might be shown in an atlas. How the grammar encodes the developmental history of this shape will be seen as follows. Consider fist the formation of the indentation at A. As noted earlier, the indentation first emerges in the transition between Stages 1 and 2 in Figure 17. These two stages are reproduced in Figure 18. As shown in the left-hand shape (Stage l), point A is initially a m+ extremum. In its change to an indentation in Stage 2 (right-hand shape), it becomes a m- extremum flanked by two zeros of curvature (the two dots). Therefore, the transition is described thus: m+ - Qm-0
INFERRING
CAUSAL
HISTORY
I
381
2 Figure
18.
Stages
1 and
2 from
Figure
17
However, this is exactly the form of the rule Cm+ in the process-grammar (section 12). Now consider the development of the protrusion D in Figure 17. As stated earlier, this protrusion is slightly evident in Stage 1. At Stage 2, it has grown and bifurcated into two protrusions, D and F. At Stage 3, an indentation, E, has emerged between protrusions D and F. Figure 19 reproduces Stages 1, 2, and 3. As is shown on the far left shape (Stage l), point D is initially visible as a M+ extremum. In the transition to Stage 2, point D bifurcates into two M+ extrema with a m + extremum emerging between them. Therefore, this transition is described thus: A4+ -iWm+M+ However, this is exactly the form of the rule BM+ in the process-grammar (section 12). Finally, in the transition from Stage 2 to Stage 3 (Figure 19), the m+ at Stage 2 (middle shape) becomes the m- extremum in Stage 3 (far right shape), and two zeros of curvature are introduced, flanking the m-. Therefore, this transition is described thus: m+ -Qm-0
This is, of course, the rule Cm+ in the process-grammar (section 12). As can be seen from the above examples, the grammar captures the fate of processes as they emerge over time in the history of a singleshape. Furthermore, deblurring prescribes the order in which the grammatical operations are applied. For example, the operation Cm:: in Figure 18, occurs simultaneously with the first transition, BM, + in Figure 19, but precedes the second transition, Cm,+ in Figure 19.
t
+5 0 382
-
INFERRING
Figure
20.
Two
shapes
assumed
CAUSAL
to be two
16. Temporal Order in Intervening
383
HISTORY
stages
in the history
of the
same
obiect
History
Considernow the secondinferenceproblem of this article: the inferenceof intervening history betweentwo given shapes:for example,two X-rays of the sametumor at different stagesof development.In usingthe grammarto reconstructthe interveninghistory, two problemsareinvolved that did not occur in the situation of inferencefrom a singleshape. Ambiguity. Theremight be alternativewaysto correspondprocesses in the first shapewith processesin the second.For example,in Figure 20, the protrusion, A, in the left shapemight havedevelopedinto any of the protrusionsB, C, or D, in the right shape. Process-Reversal. Processesmight haverecededand disappearedin the transition. The evidencefor this is that curvaturehasdiminishedat certain points. Process-reversalcannot arise in inferencefrom a single shape, becausea singleshapecan offer no evidence of reversal;onehasto assume that the curvaturevariation in that shapeis the resultof processes that grew steadilyover time. Considerthe ambiguity problem first. Thereis, in fact, a natural answer to this problem. One can easily see,that in matching the two shapesin Figure 20, one will automaticallymatch the largefeaturesbeforethe small ones. That is, one first ignores detail while automatically matching the following: (a) the large-scaleprotrusion X in Stage1 with the large-scale protrusionX in Stage2; (b) thelarge-scaleindentationY in Stage1 with the large-scaleindentationY in Stage2; and (c) the large-scaleprotrusion Z in Stage1 with the large-scaleprotrusion Z in Stage2. Thus, becauseoneinitially ignoresdetail, and matchesoverall features,it canbe concludedthat one startsby matchingtwo blurred versionsof the shapes. This is an essentialaspectof a generalmatchingtechniquethat hasbeen elaboratedby Witkin, Terzopoulos,& Kass(1987).They havearguedthat whenthereis possibleambiguity in matchingtwo objectswhich are deformationsof eachother,an appropriatetechniqueis to blur the objectsuntil a nonambiguousmatch can be madebetweenthem, andthen to deblurthem
LEYTON
9
V X
X
Y
Z
yz
I I *
I
I s3
W
. .
s2
I
9 Flgurm
21.
Coarse-to-fine
matching
of the
two
shapes
in Figure
20
slowly so that as the detail gradually appears, it can be matched incrementally across the two shapes. It can be easily seen that human perception does something like this with the two shapes in Figure 20. The matching of large-scale features provides a basis upon which the smaller scale matching can proceed. Although coarse-to-fine matching is generally a valid technique in most matching problems, it gains an extra validity in the probkm of inferring intervening history. This is because it is substantiated by the size-is-time heuristic; that is, debhnring recapitulates time. However, the use of this heuristic is more complex here than in the inference of history from a single shape, as willnowbeseen. To illustrate, a coarse-to-fine matching of the two shapes in Figure 20 wilI be carried out. At the bottom of Figure 21, the two shapes are shown again, and at the top, blurred versions of the figures are shown where the amount of blurring is sufficient to allow a match to be made. It can be seen that because features X, Y, and 2 match in the blurred versions, a feature W that is present in the right-hand blurred shape, but not in the left-hand one, can be identified. This means that, in the transition from left to right, processes caused the appearance of W. Furthermore, the processes must have started early because W is a large-scale feature.
INFERRING
CAUSAL
HISTORY
385
Observe now that, since the process-grammar specifies process-development, the transition from the top-left to the top-right shape is given by a string of grammatical operations which are labeled St, indicated by the horizontal arrow between the two blurred shapes: Observe also that, as one successively introduces more detail downward in Figure 21, additional grammatical operations are SuccessiveIy introduced. At each Ievel downward, an additional new horizontal string is introduced between the two shapes on that level. The entire set of horizontal strings, with its downward order, describes the intervening history. There is, in fact, an easier way of understanding and deriving this history. For any pair of shapes, the intervening history is the set of events that belong to the history of the second shape, minus the set of events that belong to the history of the first shape. The way this is obtained can be illustrated with Figure 21, as follows. As stated before, the history being sought here is that between the two shapes at the bottom. Therefore, the history of the bottom-right shape, minus the history of the bottom-left shape is wanted. The entire history of the bottom-right shape contains the events given by the top horizontal string Sr followed by the right-hand downward string SZ: the combined string St + &. However, some events along this history might have also been part of the history of the bottom-left shape. These latter events are contained in the left-hand downward string Ss. Thus one needs to remove from the combined string SI f SZ, any elements that occurred in &. This gives the intervening history as &+S2-s3
Observe now that a compiete subtraction might not be possible, because some elements in S3 might not actually be in St + &. These are the processes that beionged to the history of the bottom-left shape but receded and disappeared in the transition to the bottom-right shape. These disappearances correspond to the reversals of operations from the process-grammar. The reversed operations appear as negafive operations introduced by the subtraction, - S3. It is concluded therefore, that after all possible cancellations have been made, the string SI + & - S3 gives the intervening history, with any reversals that might have occurred. 17. summary
The use of shape to recover process-history has been examined. The discussion was divided into two parts. In sections l-g, the recovery of processhistory from a singfe shape was examined. It was claimed that this recovery consists of two simpIe and powerful rules. (1) the symmetry-curvature duality theorem, which assigns, to each curvature extremum, a unique sym’ More
strictly, the transition is not a string. but a set of trees.
386
LEYTON
metry axis that terminates at that extremum; and (2) the interaction principle, which converts symmetry axes into traces of processes. These two rules were applied to a large collection of shapes-those shapes with up to 8 extrema-and the resulting process-explanations were found to accord remarkably well with intuition. It was also argued that pwts are processexplanations, and that this view is more powerful than a straightforward segmentation view of parts. In sections 9-13, the recovery of interveningprocess-history was examined, between two successive shapes that are known to be two stages in the development of the same object. It was found that a grammar of only six operations is sufficient to generate the later shape from the earlier one via process-extrapolation. Although the grammar is expressed purely in terms of rewrite rules on discrete strings of extrema, these rules correspond to six intuitively salient developmental situations. Finally, the inference of temporal relations between processes was examined. It was found that the deblurring of a single shape mimics process development because it embodies the size-is-time heuristic, which states that the larger the boundary movement, the more likely it is to have taken longer to develop. The heuristic orders the emergence, continuation, and bifurcation of processes-events that are captured by the process-grammar. Since these events define the history of a single shape, it was observed that the grammar is therefore applicable to the problem of inference from a single shape, even though it was developed for the problem of inference of interveninghistory. Finally, it was found that deblurring also selects and orders the grammatical operations in the intervening history problem. In this case, the two shapes are matched through a deblurring hierarchy and the grammatical operations are allowed to reverse, thus capturing any process-disappearances that might have occurred. n
Original Submission Date: January 13. 1988.
REFERENCES Attneave. F. (1954). Some informational aspects of visual peiception. Psychological 61. 183-193. Blmn. H. (1973). Biological shape and visual science (part 1). Journul of Theoretical
Review, Biology,
38, 205-287.
Brady, H. (1983). Criteria for representation of shapes. In A. Rosenfeld & J. Beck (Eds.). Human and machine virion. Hillsdale, NJ: Erlbaum. Hoffman, D.D., & Richards. W.A. (1984). Parts of recognition. Cognition. 18. 65-%. Koenderink, J.J. (1984). The structure of @ages. Eiologicul Cybernetics, SO, 363-370. Layton. M. (1984). Perceptual organization as nested control. Biological Cybernetics, 51, 141-153. Layton, M. (1985). Generative system of analyzers. Computer Vision, Gruphics und Zmuge Pn3ce5sIilg, 31. 201-241.
INFERRING
CAUSAL
HISTORY
387
Leyton, M. (1986a). Principles of information structure common to six levels of the human cognitive system. Information Sciences, 38, l-129. Leyton, M. (1986b). A theory of information structure I: General principles. Journal of Mathematical Psychology, 30, 103-160. Leyton. M. (1986c). A theory of information structure II: A theory of perceptual organization. Journal of Mathematical Psychology, 30, 257-305. Leyton, M. (1987a). Nested structures of control: An intuitive view. Computer Vbion, Graphics and Image Processing, 37, 2&35. Leyton. M. (1987b). Symmetry-curvature duality. Computer Vision. Graphics and Image Processing, 38, 327-341. Leyton, M. (1987c). A limitation theorem for the differential prototypification of shape. Journal of Mathematical Psychology, 31. 307-320. Leyton, M. (1988). A process-grammar for shape. Artificial Intelligence, 34, 213-247. Marr, b. (1982). V&on: A computational investigation into the human representation and processing of visual information. San Francisco: Freeman. Mokhtarian, F., & Ma&worth. A. (1986). Scale-based description and recognition of planar curves and two-dimensional shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8. 34-43. Pizer. S.M.. Oliver. W., & Bloomberg, S.H. (1987). Hierarchical shape description via the muhiresolution of the symmetric axis transform. IEEE Transactions PAMI, 9,505-5 11. Richards, W., Koenderink, J.J.. & Hoffman, D.D. (1987). Inferring 3D shapes from 2D silhouettes. Journal of the Optical Society of Ameriia, Part A, 4, 1168-l 175. Rosenfeld, A. (Ed.). (1984). Multiresolution image processing and analysti. Berlin: SpringerVerlag. Terzopoulos, D. (1984). Multiresolution computation of visible-surface representations. Unpublished doctoral dissertation, MIT. Cambridge, MA. Witkin. A.P. (1983). Scale-space fdtering. Proceedings of the International Joint Conference on Artificial Intelligence, (pp. 1019-1022). Karlsruhe, Germany. Witkin. A.P.. Terzopoulos, D.. & Kass, M. (1987). Signal matching through scale space. International Journal of Computer Vision, 2, 133-144. Yuille. A.. & Poggio, T.A. (1986). Scaling theorems for zero crossings. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8, 15-25. Zucker, S.W.. & Hummel, R.A. (1986). Receptive fields and the representation of visual information. Human Neurobiology, 5, 121-128.
SCIENCE
13.357487
(1989)
Inferring CausalHistory from Shape MICHAEL LEYTON Rutgers University
The shape
of on object
often
seems
to tell
us something
about
the object’s
that is, the processes of growth, pushing, stretching, resistance, so on, that formed the object. A theory is offered here of how
history;
indentation, ond people are able to
infer the causal history of natural objects such as clouds, tumors, embryos, leoves, geological formations, and the like. Two inference problems are examined: the first is the inference of causal history from a single shape. It is claimed that this inference consists metry and curvature
of two simple and yet powerful structure of the shapes. When
a large collection remarkably well Intervening two stages fested, of the that from
of shapes, it is found that with intuition. The second
a grammar the earlier
Finally, relations detail.
a basic between
when a doctor compares and is able to conjecture
of only six operations one vio psychologically heuristic different
by which
shapes that are known to be This type of inference is mani-
two X-rays, the intervening
taken a month apart, growth. It is found
is sufficient to generate the later meaningful process-extrapolations.
human
processes
based upon the symtwo rules are applied to
the resulting causal histories accord inference problem is the recovery of
causal history between two successive in the development of the same object.
for example, same tumor,
rules, these
beings
in the past
seem
to infer
is proposed,
complex
shape
temporal
and investigated
in
1. Introduction The shapeof an object often seemsto tell us somethingabout the object’s history; that is, the causalprocessesthat formed it. For example,the shape in Figure 1 can be understoodas the result of various processesof growth, pushing, pulling, resistance,and so on. This type of understandingis, in fact, a remarkablephenomenon.A shapeis simply a singlestate, a frozen moment, a stepoutsidethe flow of time; and yet it can beusedas a window into the past. How is this possible? The author gratefully acknowledges the receipt of a Presidential Young Investigator Award (NSF IRI-88%110), part of which supported the writing of this article. Prior preliminary research was supported by NSF grants ET-8418164 and IST-8312240; and AFOSR grant F4%20-83-C-0135. Correspondence and requests for reprints should be sent to Michael Leyton, Department of Psychology, Busch Campus, Rutgers University, New Brunswick, NJ 08903. 357
LEMON
Flguro
1. A natural
shape
It will be argued here that people apply two very simple rules to a shape in order to recover a remarkably rich picture of the history of an object. The objects that will be considered are natural ones, such as leaves, tumors, clouds, and embryos. This is because such objects have much more complicated shapes than manufactured objects, which are usually the simple concatenation of box-like elements. Natural objects grow and evolve; and it is this history that is seen in them. Artificial objects never grow and evolve; and are thus seen as dead, unless they were specifically designed to mimic a sense of development. This article will examine two inference problems. The first is the inference of history from a single shape. As indicated above, this inference is comprised of two simple but powerful rules. The second inference problem is this: When given two shapes which are assumed to be the same object at two different stages of development, people seem able to infer the history that occurred between the two stages. For example, a doctor will examine two X-rays of a tumor, taken a month apart, precisely because he or she is able to infer the intervening history of the tumor. In order to see how this inference problem is solved, a grammar will be elaborated that generates the later shape via a series of developmental stages, from an earlier shape. Finally, it will be seen that in both the inference problems, people are capable not only of inferring causal processes in the past of some object, but also of inferring complex temporal relationships between those processes. The last part of this article will give a detailed analysis of the inference of these relationships.’ The discussion begins with the inference of history from a single shape. The two inference rules to be proposed, are based upon two simple structural features, curvature and symmetry. These two features will first be briefly described before proposing the two inference rules. I The first part of this article gives a cognitively oriented, nonmathematical exposition of work, which can be found in a much less cognitive, and much more technical version in Leyton (1988). The second part of the article consists of new material on the inference of complex temporal relations, and elaborates a new status for the process-grammar with respect to the inference of history from a single shape.
E w INFERRING
Figure point
2. A sequence of lines of increasing E, at which there is greater curvature
CAUSAL
curvature (bend)
HISTORY
359
(left to right). The far right one has a than at the other points on that line.
2. Curvature In what follows, it will be argued that curvature variation is a crucial means by which a person concludes that processes have taken place on a shape. What exactly is curvature variation? Observe first that curvature is the amount of bend. For example, consider the series of lines in Figure 2. The far left line has no bend and thus is regarded as having zero curvature. The amount of bend (curvature) increases in the progression of lines from left to right. Finally, observe that the right-hand curve has a special point E where the curvature on that line is greatest. That is, at any other point on that line, the curve is flatter, and has less bend, than at E. Thus E is called a curvature extremum. Curvature extrema have been regarded as having two important roles in shape perception: Attneave (1954) showed that such extrema are the most frequently used points in creating a visual summary of a shape; and Hoffman and Richards (1984) have proposed that particular kinds of curvature extrema define the endpoints of the perceptual parts of a shape. In this article, it will be argued that curvature extrema have a third psychological role: They are crucial to the inference of processes. 3. Symmetry It will also be argued that process-inference is intimately related to symmetry axes. A symmetry axis is usually defined to be a straight line along which a mirror will reflect one half of a figure onto the other. However, observe that, in complex natural objects, such as Figure 1, a straight axis might not exist. Nevertheless, one might still wish to regard the figure, or part of it, as symmetrical about some curved axis. For example, a branch of a tree tends not to have a straight reflectional axis. Nevertheless, one understands the branch to have an axial core that runs along its center. How can such a generalized axis be constructed? There are good mathematical reasons to proceed in the following way (Blum, 1973; Brady, 1983; Leyton, 1988). Consider Figure 3. It shows two curves cl and cz (the bold curves), which can be understood as two sides of an object. Now introduce
860
LEYTON
Figure
3. A rymmelry
axIs created
by the traiectory
of polntr
Q
a circle that is tangentialsimultaneouslyto the two curves,as shown.The two tangentpoints A and B arethen definedassymmetricalto eachother. Now move the circle continuouslyalongthe two curvescr and cs; while alwaysmaintainingdoubletouching. Onecan then defme a symmetryaxis to be the trajectory of the midpoint Q of the arc AB, asthe circle moves. For example,in Figure 3, the trajectory of Q is representedby the locus of dots shown. This definition was proposedin Leyton (1988),whereit was arguedthat it is more suitedfor the particulartask of process-inference than are other symmetry analyscss, suchas thoseproposedby Blum (1973)and Brady (1983).The intuitively satisfying resultsof this alternativeanalysis will be seenlater. 4. Symmetry-CurvatureDuality Symmetryand curvatureare two very different descriptorsof shape.However,a theoremproposedand provedin Leyton (1987b)showsthat thereis an intimate relationshipbetweenthesetwo descriptors.This relationship will be the basisof the entire article: SYMMETRY-CURVATURE DUALITY THEOREM (Leyton,1987b. p. 329): Any section of curve, that has one and only one curvature ewemum, has one and only one symmetry axis. This axis isforced to terminate at the extremum itself.
To illustrate: Considerthe shapewith which the article began.It is presentedagainin Figure4. The shapehas 14curvatureextrema.Thus, by the abovetheorem,thereare 14uniquesymmetryaxesassociatedwith, andterminating at, the extrema.Thesearethe dottedlines shownin Figure 4. 5. SymmetryAxes and Processes As skited earlier, thi reasonfor involving symmetry axesis that it will be argued&at they arecloselyrelatedto processes. This proposedrelationship is givenby the -followingprinciple:
INFERRING
CAUSAL
HISTORY
361
...* .*.. .. .-1.*.** :: :. .:....*.*. Gx : ‘.. . Figure 4. The natural shape curvature duality theorem
of Figure
1, with
the symmetry
axes
predicted
by the symmetry-
IN’IERACTION PRINCIPLE (Leyton, 1984, p. 149): The symmetry ayes of a perceptual organization are interpreted as the directions along whichpro&s.ses are most likely to act or have acted.
The principle was advancedand extensivelycorroboratedin Leyton (1984,1985,1986a,1986b,1986c,1987a,1987b,1987c)in severalareasof perception,including motion perceptionas well as shapeperception.The argumentusedin Leyton (1984,1986b)to justify the principle,involvesthe following two steps:(1)A processthat actsalonga symmetryaxis tendsto preservethe symmetry,that is, to be structurepreserving.(2)Structure-preservingprocessesare perceivedas the most likely processesto occur or to haveoccurred. 6. The Inference of Processes
The tools requiredto understandhow processesare recoveredfrom shape arenow in hand.In fact, the systemto beproposedconsistsof two inference rulesthat areappliedsuccessively to a shape.The rulescanbeillustratedby returningto Figure4. The first rule is the symmetry-curvatureduality theorem which statesthat, to eachcurvatureextremum,thereis a uniquesymmetry axisterminatingat that extremum.The secondrule is the interaction principle,which statesthat eachof the axesis a directionalongwhich a processhasacted.The implication is that the boundarywasdeformedalongthe axes;for example,eachprotrusionin Figure4 was the resultof pushing out along its axis, and eachindentationwas the result of pushing in along its axis. In fact, eachaxis is the truce or record of boundarymovement! Under this analysis,processesare understoodas creatingthe curvature extrema;for example,they introduceprotrusionsand indentationsinto the shapeboundary.This meansthat, if onewereto go backwardsin time, undoingall theinferredprocesses, onewould eventuallyremoveall theextrema. Observethat thereis only oneclosedcurvewithout extrema:the circle.Thus the implication is that the ultimate startingshapemust havebeena circle, andthis wasdeformedundervariousprocesses,eachof which producedan extremum.
362
LEYTON
7. Corroborating
Examples
To seethat thesetwo rulesconsistentlyyield satisfyingprocess-explanations, the explanationsthat theserules give for a large set of shapeswill be examined.Considerthe setof all possibleshapeswhich have8 extremaor less. Fiiure 5 presentsall theseshapes(asgivenby Richards,Koenderink,& Hoffman, 1987),and superimposes,on eachshape,the processesthat areinferred underthe inferencerules.The arrowsarethe inferredprocesses.As the readercan see,the resultsaccordremarkablywell with intuition. In orderto preparefor the restof the article, two observationsshouldbe made.First, the setof shapesdividesinto threelevelsaccordingto the number,of extremainvolved.LevelI shapeshave4 extrema,LevelII have6, and Level III have8. It hasbeenshownmathematicallythat no other possibilities exist with 8 extremaor less(Leyton, 1988). Second,it will now be seenthat, although only structural aspectsof shapehavebeenconsideredso far, somestrongsemantic constraintsemerge from theseaspects.To seethis, note first that, on eachshape,eachof the curvatureextremais labeledby oneof the four symbolsM, + m, - m, + or M. - Thesesymbolscorrespondto the four typesof extremathat canexist. They areillustratedin Figure 6. Undereachsymbolin the figure, thereis a curve.The dot on eachcurverepresentsthe curve’sextremum.The shading in eachcasemarks the sideof the curveupon which the body of the shape exists.The four extrematypesare characterizedin the following way. The dots on the first and secondcurvesare the points wherethosecurvesbend the most. The dots on the third and fourth curvesare the points wherethe curvesbendthe least; that is, the points wherethosecurvesarethe flattest. The shadingshowsthat the first and secondcurvesare figure-groundreversalsof eachother. Similarly, the third and fourth curvesare figure-ground reversalsof eachother.’ Thesearethe only four kindsof extremawhich can exist. The important point to observenow is that, surveyingthe process-analysesshownin Figure5, one finds that the four typesof extremacorrespond consistentlyto four typesof process: SEUANTIC
INTERPRETATION
RULE
M+ protrusion m- indentation m+ squashing
M- internalresistance 1 The symbols, M,+ m,- m,+ M,- are ch osen because they refer, respectively, to positive maxima, negative minima, positive minima, and negative maxima, of the curvature function. The convention for the specification of curvature on a planar curve is that one goes around the curve such that the inside of the figure is to the left of the curve: and one measures positive curvature as the rate of one’s anticlockwise rotation, and negative curvatwe as the rate of one’s clockwise rotation.
INFERRING
CAUSAL
HISTORY
363
That is, the purely structuralcharacterizations,M, + m, - m, + M, - correspond to semantic terms people useto describethe associatedprocesses. To seehow powerful this effectis, the readershouldlook at severalextrema on the shapesin Figure5, andcomparethem with the abovesemanticcharacterizations. 8. The Generality of Processes
How do peopleperceptuallydefineapart of a shape?Hoffman andRichards (1984)havearguedthat a part perceptuallycorrespondsto a segmentbetween two consecutivem- extremaalongthe shape’soutline. To illustrate: In the handshownin Figure 7, the black dots on the outline indicatethe m- extrema. Any segment,whoseendpointsaretwo consecutivedots,is a finger, that is, a perceptualpart of the shape. It shouldnow beobservedthat Hoffman andRichards’theoryis purelya segmentation theoryof parts;that is, partsaredefinedby their endpoints.It will now be arguedthat the process-analysis here givesa more cognitive view of what a part is. Note first, that betweenany two consecutivem- extremain Figure 7, thereis a M+ extremum:the tip of the finger (e.g.,this extremumhasbeen marked on the far left finger). Recall now that the symmetry-curvature duality theoremstatesthat, becausethereis only one extremum(M+) on the segmentbetween’ two consecutivem- extrema,therecan be only one axis on that segment.Furthermore,the theoremstatesthat the axis must terminateat the h4+ extremum.For example,in the caseof a finger,thereis only one axis, and this axis terminatesat the tip of the finger. The secondrule of the process-analysis, the interactionprinciple, then leadsto the crucialconclusionthat the axis is the trace of a process.The importanceof this conclusionis that it implies that a part is not a rigid segment: A part is a process! It is a temporal or carcsal concept. Thus, an important differenceexistsbetweenthe Hoffman & Richards analysisand the process-analysis: The former givesa descriptionof a part simply asa segment, whereasthe latter views a part asa causal explanation. It is arguedherethat causalexplanationis centralto the cognitiverepresentation of shape.That is, one cannothelp seeingan embryo,tumor, cloud, geologicalformation, and so on, as a consequence of historical processes. Thereis anotherimportant differencebetweentheHoffman andRichards segmentationapproachand the process-approach here: accordingto the process-approach, a part is a processterminating at a M+ extremum.By the semanticinterpretationrule (section7), a part must thereforebe a protrusion. However,as Figure 6 shows,a M+ extremumis only one of four typesof extremathat exist. Furthermore,since,by this analysis,processes arehypothesizedto explainextrema,theremust be four timesasmany processesasthereareparts. In fact, by the semanticinterpretationrule, the full
LEVEL
I
M+++ M&M+ P2
LEVEL
1 t v
M+
---M
P3
II
rvl+ hA+ m+ 0
TI
T4
364
M
T3
T2
y+ M’ CT5 m+ t
m+
!A+
gM+ T5
yk%qwJ T6
LEVEL
m
t
Q3
Q4
Q7
Q9
flgure 5. The process-histories inferred by the two inference rules applied to the Richards, Koenderink, ond Hoffmon (1987) drowings of all shopes with up to 8 extremo. 365
LMON
tvl+
Flgun
Figure
6. The
7. The
four
dots
types
show
of curvature
the m-
extrema
extreme
of the hand
INFERRING
CAUSAL
HISTORY
367
collection of processes associated with extrema are protrusions, indentations, squashings,and resistances.Parts correspond to only the first of these categories. It is concluded therefore, that the process-approach to parts is cognitively deeper than a straightforward segmentation approach for two reasons: (1) The process-approach considers parts to be causal explanations. It is claimed here that such explanations are central to the cognition of shape. (2) The process-approach accounts not only for parts, but, since there are four times as many processes as parts, the process-analysis accounts for many more phenomena than parts. 9. The Second Inference Problem Until now the issue of how one infers processes from a singfeshape has been considered. Sections 9-13 will examine a different problem: Suppose one is given two views of an object (e.g., an embryo), at two different stages of development. How can one infer the intervening process-history? The method of solving this problem will be to develop a grammarthat generates the second shape from the first via a sequence of plausible developmental stages. Observe now that, since the later shape is assumed to emerge from the earlier shape, one will wish to explain it, as much as possible, as the outcome of what can be seen in the earlier shape. In other words, one will wish to explain the later shape, as much as possible, asthe extrapolationof what can be seen in the earlier one. As a simple first cut, divide all extrapolations of processes into two types: (1) chltinuatioIls (2) Bifurcations (i.e., forkings).
The only forms that these two alternatives can take will now be elaborated: first, continuations, and then, bifurcations. 10. Continuations Consider any one of the M+ extrema in Figure 8. It is the tip of a protrusion, as predicted by the semantic interpretation rule. What is important to observe is that, if one continued the process creating that protrusion, that is, continued pushing out the boundary in the direction shown, the protrusion would remain a protrusion. The M+ extremum would remain a M+ extremum, the form shown at the far left of Figure 6. This means that continuation at a M+ extremumdoes not structurally alter the boundary. Exactly the same argument applies to any of the m- extrema in Figure 8. Continuation of an indentation remains an indentation, a m- extremum will remain a m- extremum.
LEYTON
,\ / . m’ m> m/y2
i c MC Figure
0. One
of the
shapes
token
M+
from
Figure
5
Now recall that there are four types of extrema, A4,+ m, - m, + M. - It hasbeenseenthat continuationsat the first two do not structurally alter the, shape.However, it will now be shown that continuationsat the secondtwo do causestructural alterations. Considerthesetwo casesin turn. Continuation at m +. A m + extremum occursat the top of the left-hand shapein Figure 9. In accord with the semanticinterpretation rule (Section 7), the processterminating at this extremum is a squashingprocess.Now continue this process;that is, continue pushing the boundaryin the direction shown. At somepoint, an indentation will be created,as shownin the top of the right-hand shapein Figure 9. Observewhat happensto the extremainvolved. Before continuation, as in the left-hand shape,the relevantextremum is m + (at the top). After continuation, as in the right-hand shape,this extremumhaschangedto m- . In fact, observethat a dot hasbeenplacedon eithersideof the m- on the curve itself. Thesetwo dots are points wherethe curve, locally, is completelyflat;
TI Figure
T2 9. Continuation
at m+:
squashing
continues
until
it indents
INFERRING
CAUSAL
(p 10. Continuation
369
M+&M+ t
13 Figure
HISTORY
T4 at M-:
internal
resistance
continues
until
it protrudes
there is 0 (zero) curvature. Therefore the top of the right-hand shape is given by the sequence Om- 0. Thus the transition between the left-hand shape and the right-hand shape can be structurally specified by simply saying that the M + extremum, at the top of the first shape, is replaced by the sequence Om - 0, at the top of the second shape. This transition will be labeled Cm + , meaning Continuation at m+ . That is, Cm+ : m+ - Om-0 Finally, observe that, although this operation is, formally, a rewrite rule on discrete strings of extrema, the rule actually has a highly intuitive meaning. Using the semantic interpretation rule, the meaning is: squashingcon-
tinues until it indents. Continuationat M- . As noted earlier, only one other type of continuation needs to be considered, that at a M- extremum. In order to understand what happens here, consider the left-hand shape in Figure 10. The symmetry analysis of section 3 describes a process-structure for the indentation that is very subtle: There is a flattening of the lowest region of the indentation because the downward arrows, within the indentation, are countered by an upward arrow, which is within the body of the shape and terminates at the M- extremum. This latter process is an example of what the semantic interpretation rule (section 7) calls internal resistance.The overall shape could be that of an island where an inflow of water (into the indentation) has been resisted by a ridge of mountains (in the body of the shape). The consequence is the formation of a bay. Now, recall that the interest here is to see what happens when one continues a process at a M- extremum. Continue the M- process upward, in the left-hand shape of Figure 10; at some point, the process will burst out and create the protrusion shown at the top of the right-hand shape in Figure 10. In terms of the island example, there might have been a volcano in the mountains which erupted and sent lava down into the sea.
LEYTON
t T4
Q6 Figure
11. Bifurcation
at M+:
a nodule
becomes
a lobe
Observenow what happensto the extremainvolved. Before continuation, that is, in the left-hand shape,the relevantextremumis M-, (in the centerof thebay). After continuation,asin the right-handshape,the extremum has changedto M+ (at the top of the protrusion). In fact, observe that, once again, a dot has beenplacedon either side of the M, + on the curveitself. Thesetwo dots represent,asbefore,points wherethe curveis, locally, completelyflat, the curvehas0 curvature.Thereforethe top of the right-handshapeis givenby the sequenceOM+0. Thereforethe transition betweenthe left-hand and right-hand shapes canbe structurallyspecifiedby simply sayingthat theM- extremum,in the first shape,is replacedby the sequenceOM+O, at the top of the second shape.This transitionwill belabeledCM-, meaningcontinuationat M- , or CM- : M’
- OM+O
Finally, observeonce again, that although this operation is a formal rewriterule on discretestringsof extrema,the rule actuallyhas a highly intuitive meaning. Using the semanticinterpretationrule, the meaning is: internal resistancecontinuesuntil it protrudes. 11. Bifurcatiqn
As seenabove,process-continuation cantake only two forms. The ptu$ose now is to elaboratethe only forms which the bifurcation (branching)of a processcantake. Note that, becausethereare four extrema,bifurcation at eachof thesefour must be examined. Bifurcation at Mt. Considerthe M+ extremumat the’top of the lefthandshapein Figure 1I, andthe protrudingprocessterminatingat this extremum. What would’resultif this processbifurcated?Under bifurcation,
INFERRING
CAUSAL
371
HISTORY
P2
T3 Figure
12.
Bifurcation
at t-n-:
an inlet
becomes
a bay
one branchwould go to the left and the other to the right; the branching would createthe upperlobe in the right-handshape. Observewhat happensto the extremainvolved.Beforesplitting, onehas theM+ extremumat thetop of the first shape.In the situationaftersplitting (i.e., in the secondshape),the left-hand branchterminatesat a M+ extremum, and so doesthe right-hand branch.The M+ extremumin the first shapehas split into two copiesof itself in the secondshape.In fact, for mathematicalreasons,a newextremumhasto be introducedbetweenthese two M+ copies.It is them+ shownat the top of the lobeof the secondshape. Therefore,in terms of extrema,the transition betweenthe first and second shapescanbeexpressed thus: The M+ extremumat the top of thefust shape is replacedby the sequenceM+m+M+ alongthe top of the secondshape. The transition is M+ - M+m+M, + and will be labeledBM, + meaning bifurcation at M, + or BM+
: M+
-M+m+M+
Finally, althoughthis transition hasjust beenexpressedas a formal rewrite rule on discretestrings of extrema,the transition has, in fact, the following highly intuitive meaning:a nodule develops into a lobe. Bifurcation at m-. Considernow the m- extremumat thetop of thelefthand shapein Figure I2, and considerthe indentingprocessterminatingat this extremum. If this processbifurcates,one branchwould go to the left and the other to the right. The branchingwould createthe bay in the righthand shape. Observeagain what happensto the extremainvolved. Before splitting, onehasthe singleextremumm- in the indentationof the frostshape.In the situation after splitting (i.e., the secondshape),the left-handbranchterminatesat a m- extremum,and so doesthe right-handbranch. The top min the first shapehasbeensplit into two copiesof itself in the secondshape.
372
LEYTON
Again, for mathematical reasons, a new tween these two m- copies. It is the Mthe second shape. Therefore, in terms of first and second shapes can be expressed shape is replaced by the sequence m-M-m ition is given by the rule m- -m -M-m, ing bifurcation at m, - that is, Bm-
: m-
extremum has to be introduced beshown at the center of the bay in extrema, the transition between the thus: The m- extremum in the first - in the second shape. The trans- and will be labeled Bm, - mean-
- m-M-m-
Finally, although the transition has just been expressed as a formal rewrite rule using discrete strings of extrema, the transition has, in fact, a highly intuitive meaning: an inlet develops into a bay. Bifurcation at m + . The above two types of bifurcation illustrate the general form that bifurcations take: an extremum, E, is split into two copies of itself. Furthermore, for mathematical reasons, an intervening extremum, e, of a different type has to be introduced between the two copies.’ Therefore, bifurcation always takes the form E - EeE. Consider now bifurcation at m. + By the discussion in the previous paragraph, one can see, in advance, that a bifurcation at m+ must change the m + into the sequence m +M+ m + (two copies of m + separated by M+). The bifurcation is given by the rule m+ - m+M+m, + which will be labeled Bm + , meaning bifurcation at m, + that is, Bm+
: m+ - m+M+m+
This rule is illustrated in Figure 13. The m+ extremum, at the top of the left-hand shape, splits and becomes the two m+ extrema on either side of the right-hand shape. Simultaneously, a M+ extremum is necessarily introduced at the top of the right-hand shape. Since a M+ is always a protrusion, one can characterize this bifurcation simply as: a protrusion is introduced. Bifurcation at M-. Consider now bifurcation at M. - Again, from the above considerations, one can deduce, in advance that a bifurcation at Mmust change the M- into the sequence M-m-M(two copies of M- separated by m -). The bifurcation is given by the rule M- - M-m-M, - which will be labeled BM, - meaning bifurcation at M, - that is, BM-
: M-
- M-m-M-
’ It is sufficient to consider cases only where e is the same sign as E. This is because the other casescan be generated from these via the continuation rules given earlier. Thus. for each E, there is a unique e.
INFERRING
CAUSAL
HISTORY
373
h4+ L,+ t fn+/ hA+ 0J m+\M+ t PI
TI Figure
13.
Bifurcation
at m+:
a protrusion
is introduced
This rule is illustrated in Figure 14. The M- extremum,at the centerof the bay in the left-hand shape,splits and becomesthe two M- extremaon either side of the lagoon in the right-hand shape.Siulataneously, a mextremumis necessarilyintroducedat the lowestpoint of the lagoon(in the right-handshape).Sincem- is alwaysan indentation,onecancharacterize this bifurcation simply as: an indentation is introduced. 12. The Complete Grammar
Recall that the problem being examinedin this part of the article is this: Given two views of an object (e.g., a tumor), at two different stagesof development,how is it possibleto infer the intervening process-history? Sinceone wishesto explain the later shapeas an outcomeof the earlier shape,one will try to explainthe later shape,as much as possible,as the extrapolation of the process-structure inferable from the earliershape.It hasnow beenshownthat all possibleprocess-extrapolations are generated by only 6 operations:2 continuationsand4 bifurcations.Therefore,these6
T3
Q3. Figure
14. Bifurcation
at M+:
an indentation
is introduced
LMON
T6
Q5
flguro 15. Two
shapes
ossumed
to
be IWOstages in the history of
the some
object
operations form a grammar that will generate a later shape from an earlier one via processextrapolation. The grammar is as follows: PROCESS GRAMMAR Cm+ CMBM+ BmBm+ BM-
: m+ : M: M+ : m:m+ : M-
-
Oh-0 OM+O M+m+M+ m-M-mm+M+m+ M-m-M-
Recall however that, although these operations are expressed as formal rewrite rules on discrete strings of extrema, they describe six intuitively compelling situations, as follows: SEMANTIC INTERPRETATION OF THE GRAMMAR Cm+ CMBM+ BmBm+ BM-
: squashing continues until it indents. : internal resistance continues until it protrudes : a protrusion bifurcates; for example, a nodule becomes a lobe. : an indentation bifurcates; for example, an inlet becomes a bay.
: a protrusion is introduced. : an indentation is intmduced.
These six situations are illustrated in Figures 9-14. 13. A Compelling Ibtration The intuitive power of the grammar to yield process-explanations will now be illustrated, by considering an extended example. Suppose the two shapes in Figure 15 are two stages in the development of some entity such as a tumor or island. The process-grammar is used to give a complete and compelling account of the intervening history. The grammar provides a number of alternative process-routes from the first shape to the second. In Section 16, a procedure for choosing the most
INFERRING
CAUSAL
HISTORY
375
appropriate route will be described. Suppose that the route chosen by the procedure is the following sequence of operations: CM- followed by BM+ followed by Cm. + This sequence generates the sequence of shapes shown in Figure 16. The first and last shapes in this sequence are the two shapes that one is trying to bridge, and the whole sequence is the conjectured intervening history. The sequence of operations, CM- followed by BM+ followed by Cm, + is, of course, the inferred processexplanation. Given the semantic interpretation rules (sections 7 and 12), the explanation for Figure 16 can be converted into English thus: 1. The entire history is dependent upon a single crucial process. It is the internal resistance represented by the bold upward arrow in the first shape. 2. This process continues upward until it bursts out and creates the protrusion shown in the second shape. 3. This process then bifurcates, creating the upper lobe shown inthe third shape. Involved in this stage is the introduction of a squashing process given by the bold downward arrow at the top of the third shape. 4. Finally, the downward squashing process continues until it causes the new indentation in the top of the fourth shape. 14. Inferring Temporal Order In the previous section, the use of the grammar to provide a history between two shapes was illustrated. However it was noted in passing, that the grammar can provide a number of alternative histories between two shapes. Thus it is necessary to understand how one forms a single most plausible intervening history. To do this, it is necessary to examine how one infers temporal relations between processes along a history. These relations will determine the selection and order of the grammatical operations such that they will generate the optimal history. In the real world, the temporal relations between processes can be sequential, overlapping, or simultaneous. For example, in an embryo, the growth of the torso starts before that of the arms, but later overlaps the growth of the arms. Furthermore, the growth of one arm is exactly simultaneon with the growth of the other arm. Remarkably, it will be found that human beings naturally hypothesize such complex process-relationships. In fact, more remarkably, it will be seen that human beings use a very simple heuristic to give this complex history. In order to understand this heuristic, consider the first shape in Figure 17. It shows a map of Africa as one might come across it in an atlas. The subsequent shapes show successively more blurred versions of the original map. (The blurring sequence is denoted by the numbers 6 down to 1 in the figure.)
T6
Flgure
16. A viable
T5 path,
provided
by the grammar,
between
Q7 the two
shapes
in Figure
15
Q5
6
4
F
3
A 9
E
D
F
D
2
A v
0
cl
4
Figure 17. Six successive blurrings of the outline of Africa. token Mackworth (1986) Scale-based description and recognition of planar
from curves
sional shapes. IEEE Tronsacffons with permission. 01986 IEEE. features on the shapes.
Infelltgence. to indicate
on Pattern Analysis letters have been
and Machine superimposed
Mokhtarian and and two-dimen8, 34-43, particular 377
378
LEVTON
The sequenceof figurescomesfrom Mokhtarian & Mackworth (1986)who discusssmoothing in relation to the problem of recognizingobjects. Considerthe main structural featuresof the top map. Clearly, one of the most salient featuresof the map is the largeindentation shown at A. Look now at the 6 figures in the reversedorder, the order of deblurring. This sequencewill be referredto as Stages1 to 6, correspondingto the numbering in Figure 17.What is observedis that, in the deblurringsequence,the indentationat A emergesearly at Stage2, betweenthe two dots shownat that stage(the dots are flat points, points without curvature).If one tracks this indentation through the subsequentfigures (in the order 1 - 6), one finds that the indentationdeepensprogressively,until it reachesits full sizein the top map, Stage6. Another prominent featureof the top map is the protrusion shownat D. Consideragain its emergencein the deblurring sequence(1 - 6). This protrusion is slightly evidentat Stage1. Furthermore,it alsocontinuesto grow in the subsequentfigures,until it reachesits full sizein the top map (Stage6). Now considera lessprominent featureof the top map. Examination of this map revealsthat, within indentation A, thereis a smallerprotrusion. In the deblurringsequence(1 - 6), this protrusion emergesquite late in Stage 4, at the position marked B, betweentwo new dots. Becauseit is a smaller featurein the top map, it emergeslater in the deblurring sequencethan the more significant features.Subsequently,of courseit continuesto grow until it reachesits full sizein the top map. Observethat not only do theseprocessescontinueto grow in the deblurring direction, but they can also bifurcate. For example, consider once againthe indentation at A, asit emergesin Stage2. It continuesto grow in Stage3. However,in Stage4, it beginsto bifurcate to allow the protrusion, B, to emerge.This makesthe indentation A break into two copiesof itself, one at A and the other at C. Again, considerthe protrusion at D. In the transition from Stage1to Stage2, it bifurcatessubtly, into two protrusions, one at D and the other at F. The above discussionseemstherefore to lead to the following conclusion: Whendeblurring a shape,two things happen,(1)proceaesemergein the order of theirprominence,and (2) theprocessesgrow to theirfullsize by continuing andpossibly bifurcating. Becausedeblurring a contour causesa processto increaseto its full size, it seemsthat deblurring mimics the developmentalhistory of the process. There is, in fact, a deepreasonfor a correspondencebetweendeblurring and process-history.Recall from section6, it was arguedthat processesare conjecturedby human beingsto explainhow the curvaturevariationis introducedinto the boundary. That is, processesare conjecturedto haveshifted the boundary in the direction of increasedcurvaturevariation. What has been found aboveis that deblurring incrementallymovesthe boundary in
INFERRING
CAUSAL
HISTORY
379
the direction of increased curvature variation. Thus deblurring introduces what processes introduce. Another reason why deblurring gives a plausible recapitulation of process-history is that, because deblurring is a uniform means of incrementally introducing curvature variation, it is capable of embodying the following simple heuristic: SIZE-IS-TIME HEURISTIC. In the absence of informution to the contrary, size corresponds to time. That is, the larger the boundary movement, the more likely it is to have taken longer to develop.
This rule is valuable where one does not have any knowledge of the real world situation involved, when all that is available is the shape of the present state of the object. As stated at the beginning of this article, this is exactly the type of situation being investigated here. The concern is with the remarkable human ability to derive a history when no other knowledge is available apart from a shape in the present. One might argue, for example, that geologists now know that the outline of Africa was created by the breaking of continental land masses, rather than by boundary movement. The concern here however, is not with what actually happened in the environment, but with the human ability to see history in the very absence of knowing what actually happened. Nevertheless, one should note that the type of history considered here for Africa does indeed correspond to a standard type of geological scenario for the creation of an island: Inlets are produced by erosion from the sea, bays by the resistance of mountains, promontories by the outward advancement of a coastline, for example by volcanos, and so on. Deblurring mimics the processes that create boundary movement, but to conjecture that deblurring is a technique indeed used by human beings would require the supposition that human beings actually incorporate multiple levels of blurring into their perceptual representations; that human beings maintain sets of representations of the type shown in Figure 17. How valid is it to suppose this? Vision researchers are finding overwhelmingly that human beings represent the environment on multiple levels of blurring (Marr, 1982). In fact, the importance of such multiple-blurring hierarchies has emerged in the last decade as one of the major discoveries in vision (e.g., Koenderink, 1984; Mokhtarian & Mackworth, 1986; Pizer, Oliver, & Bloomberg, 1987; Rosenfeld, 1984; Terzopoulos, 1984; Witkin, 1983; Yuille & Poggio, 1986; Zucker & Hummel, 1986). What has been argued here, is that when such representations are applied to natural objects, one of their uses is to recapitulate process-history. Examine now the role of deblurring in relation to the two inference problems that are of concern in this article: (1) the inference of history from a single shape, and (2) the inference of intervening history between two given
LEYTON
380
shapes. These two problems will be considered in the next two sections, respectively. 15. Temporal Order and Inference from a Single Shape Recall that in order to solve the problem of inference from a single shape, two rules were proposed: (a) the symmetry-curvature duality theorem which assigns, to each extremum, a unique symmetry axis terminating at the extremum; and (b) the interaction principle which implies that each axis is the trace of a process. As a consequence of applying these two rules, diagrams such as those shown in Figure 5 were derived, which will be called process-diagrams. It can now be seen that process-diagrams do not contain a complete reconstruction of the process-history. They do not encode the temporal order of the processes; that is, which of the processes followed each other, which were overlapping, and which were simultaneous. Thus the process-diagrams must be supplemented with extra information. But how can this information be specified? Recall, from the Africa example, that, besides determining the order in which the processes first appeared, deblurring caused processes to continue and bifurcate until they reached their full size in the final shape. The crucial point to observe here is that earlier in this article, a means was developed by which the continuation and bifurcation of processes can be specified: the process-grammar. The grammar was in fact developed to solve the second inference problem: inference of intervening history between two shapes. However, it can now be seen that the grammar is also relevant to the first inference problem: inference from a single shape (e.g., the original map of Africa). It will specify the structural changes that occurred when processes continued and bifurcated in the history leading up to that shape. Thus the extra information that is needed to supplement process-diagrams, such as those in Figure 5, are operations from the process-grammar. To illustrate, return to the Africa example; it conforms to the first of the two inference problems, the inference of history from a single presented shape. The single shape is, of course, the original map (the top map in Figure 17), as might be shown in an atlas. How the grammar encodes the developmental history of this shape will be seen as follows. Consider fist the formation of the indentation at A. As noted earlier, the indentation first emerges in the transition between Stages 1 and 2 in Figure 17. These two stages are reproduced in Figure 18. As shown in the left-hand shape (Stage l), point A is initially a m+ extremum. In its change to an indentation in Stage 2 (right-hand shape), it becomes a m- extremum flanked by two zeros of curvature (the two dots). Therefore, the transition is described thus: m+ - Qm-0
INFERRING
CAUSAL
HISTORY
I
381
2 Figure
18.
Stages
1 and
2 from
Figure
17
However, this is exactly the form of the rule Cm+ in the process-grammar (section 12). Now consider the development of the protrusion D in Figure 17. As stated earlier, this protrusion is slightly evident in Stage 1. At Stage 2, it has grown and bifurcated into two protrusions, D and F. At Stage 3, an indentation, E, has emerged between protrusions D and F. Figure 19 reproduces Stages 1, 2, and 3. As is shown on the far left shape (Stage l), point D is initially visible as a M+ extremum. In the transition to Stage 2, point D bifurcates into two M+ extrema with a m + extremum emerging between them. Therefore, this transition is described thus: A4+ -iWm+M+ However, this is exactly the form of the rule BM+ in the process-grammar (section 12). Finally, in the transition from Stage 2 to Stage 3 (Figure 19), the m+ at Stage 2 (middle shape) becomes the m- extremum in Stage 3 (far right shape), and two zeros of curvature are introduced, flanking the m-. Therefore, this transition is described thus: m+ -Qm-0
This is, of course, the rule Cm+ in the process-grammar (section 12). As can be seen from the above examples, the grammar captures the fate of processes as they emerge over time in the history of a singleshape. Furthermore, deblurring prescribes the order in which the grammatical operations are applied. For example, the operation Cm:: in Figure 18, occurs simultaneously with the first transition, BM, + in Figure 19, but precedes the second transition, Cm,+ in Figure 19.
t
+5 0 382
-
INFERRING
Figure
20.
Two
shapes
assumed
CAUSAL
to be two
16. Temporal Order in Intervening
383
HISTORY
stages
in the history
of the
same
obiect
History
Considernow the secondinferenceproblem of this article: the inferenceof intervening history betweentwo given shapes:for example,two X-rays of the sametumor at different stagesof development.In usingthe grammarto reconstructthe interveninghistory, two problemsareinvolved that did not occur in the situation of inferencefrom a singleshape. Ambiguity. Theremight be alternativewaysto correspondprocesses in the first shapewith processesin the second.For example,in Figure 20, the protrusion, A, in the left shapemight havedevelopedinto any of the protrusionsB, C, or D, in the right shape. Process-Reversal. Processesmight haverecededand disappearedin the transition. The evidencefor this is that curvaturehasdiminishedat certain points. Process-reversalcannot arise in inferencefrom a single shape, becausea singleshapecan offer no evidence of reversal;onehasto assume that the curvaturevariation in that shapeis the resultof processes that grew steadilyover time. Considerthe ambiguity problem first. Thereis, in fact, a natural answer to this problem. One can easily see,that in matching the two shapesin Figure 20, one will automaticallymatch the largefeaturesbeforethe small ones. That is, one first ignores detail while automatically matching the following: (a) the large-scaleprotrusion X in Stage1 with the large-scale protrusionX in Stage2; (b) thelarge-scaleindentationY in Stage1 with the large-scaleindentationY in Stage2; and (c) the large-scaleprotrusion Z in Stage1 with the large-scaleprotrusion Z in Stage2. Thus, becauseoneinitially ignoresdetail, and matchesoverall features,it canbe concludedthat one startsby matchingtwo blurred versionsof the shapes. This is an essentialaspectof a generalmatchingtechniquethat hasbeen elaboratedby Witkin, Terzopoulos,& Kass(1987).They havearguedthat whenthereis possibleambiguity in matchingtwo objectswhich are deformationsof eachother,an appropriatetechniqueis to blur the objectsuntil a nonambiguousmatch can be madebetweenthem, andthen to deblurthem
LEYTON
9
V X
X
Y
Z
yz
I I *
I
I s3
W
. .
s2
I
9 Flgurm
21.
Coarse-to-fine
matching
of the
two
shapes
in Figure
20
slowly so that as the detail gradually appears, it can be matched incrementally across the two shapes. It can be easily seen that human perception does something like this with the two shapes in Figure 20. The matching of large-scale features provides a basis upon which the smaller scale matching can proceed. Although coarse-to-fine matching is generally a valid technique in most matching problems, it gains an extra validity in the probkm of inferring intervening history. This is because it is substantiated by the size-is-time heuristic; that is, debhnring recapitulates time. However, the use of this heuristic is more complex here than in the inference of history from a single shape, as willnowbeseen. To illustrate, a coarse-to-fine matching of the two shapes in Figure 20 wilI be carried out. At the bottom of Figure 21, the two shapes are shown again, and at the top, blurred versions of the figures are shown where the amount of blurring is sufficient to allow a match to be made. It can be seen that because features X, Y, and 2 match in the blurred versions, a feature W that is present in the right-hand blurred shape, but not in the left-hand one, can be identified. This means that, in the transition from left to right, processes caused the appearance of W. Furthermore, the processes must have started early because W is a large-scale feature.
INFERRING
CAUSAL
HISTORY
385
Observe now that, since the process-grammar specifies process-development, the transition from the top-left to the top-right shape is given by a string of grammatical operations which are labeled St, indicated by the horizontal arrow between the two blurred shapes: Observe also that, as one successively introduces more detail downward in Figure 21, additional grammatical operations are SuccessiveIy introduced. At each Ievel downward, an additional new horizontal string is introduced between the two shapes on that level. The entire set of horizontal strings, with its downward order, describes the intervening history. There is, in fact, an easier way of understanding and deriving this history. For any pair of shapes, the intervening history is the set of events that belong to the history of the second shape, minus the set of events that belong to the history of the first shape. The way this is obtained can be illustrated with Figure 21, as follows. As stated before, the history being sought here is that between the two shapes at the bottom. Therefore, the history of the bottom-right shape, minus the history of the bottom-left shape is wanted. The entire history of the bottom-right shape contains the events given by the top horizontal string Sr followed by the right-hand downward string SZ: the combined string St + &. However, some events along this history might have also been part of the history of the bottom-left shape. These latter events are contained in the left-hand downward string Ss. Thus one needs to remove from the combined string SI f SZ, any elements that occurred in &. This gives the intervening history as &+S2-s3
Observe now that a compiete subtraction might not be possible, because some elements in S3 might not actually be in St + &. These are the processes that beionged to the history of the bottom-left shape but receded and disappeared in the transition to the bottom-right shape. These disappearances correspond to the reversals of operations from the process-grammar. The reversed operations appear as negafive operations introduced by the subtraction, - S3. It is concluded therefore, that after all possible cancellations have been made, the string SI + & - S3 gives the intervening history, with any reversals that might have occurred. 17. summary
The use of shape to recover process-history has been examined. The discussion was divided into two parts. In sections l-g, the recovery of processhistory from a singfe shape was examined. It was claimed that this recovery consists of two simpIe and powerful rules. (1) the symmetry-curvature duality theorem, which assigns, to each curvature extremum, a unique sym’ More
strictly, the transition is not a string. but a set of trees.
386
LEYTON
metry axis that terminates at that extremum; and (2) the interaction principle, which converts symmetry axes into traces of processes. These two rules were applied to a large collection of shapes-those shapes with up to 8 extrema-and the resulting process-explanations were found to accord remarkably well with intuition. It was also argued that pwts are processexplanations, and that this view is more powerful than a straightforward segmentation view of parts. In sections 9-13, the recovery of interveningprocess-history was examined, between two successive shapes that are known to be two stages in the development of the same object. It was found that a grammar of only six operations is sufficient to generate the later shape from the earlier one via process-extrapolation. Although the grammar is expressed purely in terms of rewrite rules on discrete strings of extrema, these rules correspond to six intuitively salient developmental situations. Finally, the inference of temporal relations between processes was examined. It was found that the deblurring of a single shape mimics process development because it embodies the size-is-time heuristic, which states that the larger the boundary movement, the more likely it is to have taken longer to develop. The heuristic orders the emergence, continuation, and bifurcation of processes-events that are captured by the process-grammar. Since these events define the history of a single shape, it was observed that the grammar is therefore applicable to the problem of inference from a single shape, even though it was developed for the problem of inference of interveninghistory. Finally, it was found that deblurring also selects and orders the grammatical operations in the intervening history problem. In this case, the two shapes are matched through a deblurring hierarchy and the grammatical operations are allowed to reverse, thus capturing any process-disappearances that might have occurred. n
Original Submission Date: January 13. 1988.
REFERENCES Attneave. F. (1954). Some informational aspects of visual peiception. Psychological 61. 183-193. Blmn. H. (1973). Biological shape and visual science (part 1). Journul of Theoretical
Review, Biology,
38, 205-287.
Brady, H. (1983). Criteria for representation of shapes. In A. Rosenfeld & J. Beck (Eds.). Human and machine virion. Hillsdale, NJ: Erlbaum. Hoffman, D.D., & Richards. W.A. (1984). Parts of recognition. Cognition. 18. 65-%. Koenderink, J.J. (1984). The structure of @ages. Eiologicul Cybernetics, SO, 363-370. Layton. M. (1984). Perceptual organization as nested control. Biological Cybernetics, 51, 141-153. Layton, M. (1985). Generative system of analyzers. Computer Vision, Gruphics und Zmuge Pn3ce5sIilg, 31. 201-241.
INFERRING
CAUSAL
HISTORY
387
Leyton, M. (1986a). Principles of information structure common to six levels of the human cognitive system. Information Sciences, 38, l-129. Leyton, M. (1986b). A theory of information structure I: General principles. Journal of Mathematical Psychology, 30, 103-160. Leyton. M. (1986c). A theory of information structure II: A theory of perceptual organization. Journal of Mathematical Psychology, 30, 257-305. Leyton, M. (1987a). Nested structures of control: An intuitive view. Computer Vbion, Graphics and Image Processing, 37, 2&35. Leyton. M. (1987b). Symmetry-curvature duality. Computer Vision. Graphics and Image Processing, 38, 327-341. Leyton, M. (1987c). A limitation theorem for the differential prototypification of shape. Journal of Mathematical Psychology, 31. 307-320. Leyton, M. (1988). A process-grammar for shape. Artificial Intelligence, 34, 213-247. Marr, b. (1982). V&on: A computational investigation into the human representation and processing of visual information. San Francisco: Freeman. Mokhtarian, F., & Ma&worth. A. (1986). Scale-based description and recognition of planar curves and two-dimensional shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8. 34-43. Pizer. S.M.. Oliver. W., & Bloomberg, S.H. (1987). Hierarchical shape description via the muhiresolution of the symmetric axis transform. IEEE Transactions PAMI, 9,505-5 11. Richards, W., Koenderink, J.J.. & Hoffman, D.D. (1987). Inferring 3D shapes from 2D silhouettes. Journal of the Optical Society of Ameriia, Part A, 4, 1168-l 175. Rosenfeld, A. (Ed.). (1984). Multiresolution image processing and analysti. Berlin: SpringerVerlag. Terzopoulos, D. (1984). Multiresolution computation of visible-surface representations. Unpublished doctoral dissertation, MIT. Cambridge, MA. Witkin. A.P. (1983). Scale-space fdtering. Proceedings of the International Joint Conference on Artificial Intelligence, (pp. 1019-1022). Karlsruhe, Germany. Witkin. A.P.. Terzopoulos, D.. & Kass, M. (1987). Signal matching through scale space. International Journal of Computer Vision, 2, 133-144. Yuille. A.. & Poggio, T.A. (1986). Scaling theorems for zero crossings. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8, 15-25. Zucker, S.W.. & Hummel, R.A. (1986). Receptive fields and the representation of visual information. Human Neurobiology, 5, 121-128.
