14. Learning Sparse Spaces; Phonology Video Lecture - Artificial Intelligence

PATRICK WINSTON: So today we're
gonna talk about a few
miracles of learning in the
context of the theme that
we're developing here
in the class.
We started off with
a discussion
of some basic methods.
We talked about nearest
neighbors.
And we talked about
identification trees.
And those are kind of basic
things that have been around
for a long time.
Still useful.
Still the right things to do
when you're faced with a
learning problem and you're not
sure what method to try.
Then we went on to talk about
some naive biological mimicry.
We talked about neural nets.
And we talked about genetic
algorithms.
And you look at those things and
you think and reflect back
on what we talked about.
And you have to say
to yourself, are
these nugatory ideas?
Perhaps pistareens?
Or are they supererogatory
ideas that deserve to be
center stage?
Does anybody know what
those words mean?
A pistareen?
Well, a pistareen is
a Spanish coin.
It was so small.
It was of little worth.
These ideas like neural nets,
genetic algorithms, I classify
them as pistareens because
getting them to do something
is rather like getting a dog
to walk on its hind legs.
You can make it happen, but they
never do it very well.
And you have to think it took a
lot of trickery and training
to make it happen.
So not too personally
high on those ideas.
But we teach them to you anyway
because, of course, we
only editorialize part of time
and part of time we like to
cover what's in the field.
Today we're starting a couple
of discussions of mechanisms
or ideas or things
to know about
that are quite different.
Because now we're going to focus
on the problem rather
than on the mechanism.
And then a later on we're
going to talk about deep
theory, FIOS, for
its own sake.
But this week I want
to talk about
mechanisms that were devised.
I want to talk about research
that was done.
Let me not say mechanisms.
Let me say research that was
done to attempt an account of
some of the things that
we humans do well.
Sometimes without even knowing
that we do it.
Now Krishna here tells me his
first language was Telugu.
Telugu.
I once had another student
whose first
language was Telugu.
I said to him, that must
be one of those
obscure Indian languages.
And he said, yes.
It's spoken by 56
million people.
French is spoken by 52.
[LAUGHTER]
PATRICK WINSTON: He's going to
be our experimental subject.
Krishna, if I pluralize words--
you know what it means
to pluralize a word.
So if I say for example, horse,
then if I ask you for
the plural you'll say horses.
So if I say dog, what's
the plural?
STUDENT: Then dogs.
Or in my language?
PATRICK WINSTON: No, no, no.
In English.
STUDENT: Oh, dogs.
PATRICK WINSTON: Well,
what about cat?
STUDENT: Cats.
PATRICK WINSTON: And
he got it right.
Isn't that a miracle?
When did you start
speaking English?
STUDENT: Second grade.
PATRICK WINSTON: Second grade.
But he still got it right.
But he never learned that he's
actually pluralizing those
words differently.
But he is.
So when you pluralize
dog, what's the
sound that comes after?
It's a z sound.
Zzzzzz.
Dogzzz.
If you stick your fingers up
here you can probably feel
your vocal cords vibrating.
If you stick a piece of paper in
front of your mouth you'll
see it vibrate.
But when you say cats,
the pluralizing sound
is sss, like that.
No vocalizing.
No vibration of the
vocal cords.
And old Krishna here learned
that rule, as did all of you
other non-native speakers of
English, effortlessly and
without noticing it.
You learned it.
Buy you always get it right.
How can that possibly be?
Well, be the end of hour you'll
know how that might be.
And you'll experience a case
study in how questions of that
sort can be approached
with a sort of
engineering point of view.
You can say, what if God
were an engineer?
Or alternatively, what if I were
God and I am an engineer?
Think about how it might
happen that way.
So we want to understand how it
might be that the machine
could learn rules like that.
Phonological rules.
Not just that one, but all the
phonological rules you'd
acquire in a course
on phonology.
That part of speaking that deals
with those syllabic and
sub-syllabic sounds.
The phones of the language.
So when Yip and Sussman
undertook to solve this
engineering problem, both being
dedicated engineers, the
first thing they did was
learn the science.
So they went to sit at the
foot of Morris Halle, who
would develop-- was largely
responsible for the
development theories of
so-called distinctive features.
And here's how all that works.
You start off with a person who
wants to say something.
And out that person's mouth
comes some sort of acoustic
pressure wave.
And if I say, hello, George.
And you say hello, George.
Everybody will understand that
we said the same thing.
But that acoustic waveform won't
look anything alike.
It'll be very different
for all of us.
So it's a miracle that words
can be understood.
In any case, it goes
into an ear.
And it's processed.
And out comes a sequence
of distinctive feature.
Vectors.
A distinctive feature is a
binary variable like is the
phone voices or not.
That is to say, are your
vocal cords vibrating
when you say it?
If so, then that's
plus voiced.
If not, it's minus voiced.
So according to the original
distinctive feature theory and
consistent with most of the
theories that have been
derived since the original one,
there are on the order of
14 of these distinctive features
that determine which
phone you're saying.
So if you say ah, that's
one combination of
these binary features.
If you say tuh, that's another
combination of
these binary features.
14 of them.
So how many sounds does that
mean, in principle, there
could be in a language?
SEBASTIAN: 2 to the 14th.
PATRICK WINSTON: And what's
2 the 14th, Sebastian?
Well, it ought to be about
16,000, don't you think?
2 to the 10th is 1,000.
2 the fourth is 16.
So there are about 16,000
possible combination.
But no language on Earth has
more than 100 phones.
That's strange, isn't it?
Because some of those choices
are probably excluded on
physical ground.
But most of them are not.
So we could have a lot more
phones in our language than we
actually do.
English is about 40.
So the sequence of distinctive
features could be viewed as
then producing meaning after,
perhaps, a long series of
operations.
But in the end, those operations
feedback in here
because many of the distinctive
features are
actually hallucinated.
We think we heard them,
but they're not there.
Or they're not even in the
acoustic waveform.
They're there for the
convenience of the phonologist
who make rules out of them.
It's remarkable how much of this
feedback there is, and
even injection from
other modalities.
Many of you may have heard
about the McGurk Effect.
Here's who the McGurk
Effect works.
Look at me while I say ga,
ga, ga, ga, ga, ga.
OK.
I said, g-a.
Now how about ba, ba, ba, ba.
OK.
I said ba like a sheep.
But if I take the sound I make
when I say ba and play it
while you're taking video of
me saying ga, what do you
think you hear?
You don't hear ba.
Some people report that they
hear a d-a sound like da.
When I look at it, I can't
make any sense out of it.
It looks like there's a
disconnection between the
speech and the video.
But it does not sound like ba.
But if I shut my eyes and say
ba, ba, it's absolutely clear
that it's b-a.
So what you see has a large
influence on what you hear.
It's also interesting--
although a side issue-- it's
also interesting to note that
it's very difficult pronounced
things correctly if you don't
see the speaker.
So many people wonder when they
learn foreign languages
why they can't speak
like a native.
And the answer is, they're
not watching the
mouth of the speaker.
I was talking to a German friend
once and said, you
know, I just can't say the
damned umlaut right.
And he said, oh, the trouble
with you Americans is you
don't realize that American
cows say moo but
German cows say muu.
[LAUGHTER]
PATRICK WINSTON: And, of
course, I got instantly
because I could see that the
umlaut sounds are produced
with protruding lips, which we
don't have any sounds an
English that require that.
Ah, but back to what we know
from the phonologists about
all this stuff.
If you talk to Morris
Halle, he will tell
you that over here--
I like to think of it
as a marionette.
There are five pieces
of meat down here.
And the combination of
distinctive features that
you're trying to utter are
like the control of a
marionette on those five
pieces of meat.
So if you want to say an a
sound, the marionette control
goes into a position that
produces that combination.
So let's see.
What does that distinctive
feature sequence look like for
typical word?
Well, here's a word.
A-e-p-l.
Apples.
And we can talk about what
distinctive features are
arrayed in that particular
combination of phones.
So one of the features
that they like to
talk about is syllabic.
Syllabic.
That roughly means, can that
sound form the sort of core of
a syllable?
And the answer is a can,
buy these can't.
So it's plus, minus,
minus, minus.
Down here a little ways you'll
run into the voiced feature.
And for the voiced feature,
well, we can do
the experiment ourselves.
Ahh.
Sounds like it's voices to me.
Pa.
No.
That's not voiced.
Oo.
Yep.
Zzz.
We already said that
was voiced.
So that's the combination you
see when you utter apples for
the voiced feature.
Then another one is the
continuent one.
That roughly says is your
vocal apparatus open?
Is there no obstruction?
And so ahh plus pa
is constricted.
Oo, open.
Zzz, open.
So that one happens to run right
along with voiced in
that particular word.
Oh, and there are
14 altogether.
But let me just write
down one more.
The strident one.
That says, do you use your
tongue to form a
little jet of air?
So you don't on aa, pa, oo.
Buy you do on z.
So that gets a plus.
So that's a glimpse through a
soda straw of what it would
like to represent the word
apples as a set of distinctive
features all arranged
in a sequence.
So it's a matrix of features.
Going down in the columns we
have our distinctive features.
And going across we have time.
So as the first thing Sussman
and Yip did in their effort to
understand how phonological
rules could be learned is to
design a machine that would
interpret words and sounds and
things that you see so
as to produce the
sounds of the language.
So they imagined the following
kind of machine.
The machine has some kind of
mystery apparatus over here
that looks out into the world
and sees what's there.
So I'm looking out in the world
and I see two apples.
So what this machine might do
then is, at some point, decide
that there are two
apples out there.
Then, thinking in terms of
these guys as computer
engineers, they think in terms
of a set of registers that
hold values for concepts like
noun and verb and plural.
And we've not done anything
with the machine yet.
We've provided no input.
So those registers
are all empty.
Then, up in here, we have
a set of words.
And they're all kinds
of words.
Apple is one of them.
And those words up there know
about how the concept is
rendered as a sequence of a
phones, that is to say a
sequence of distinct features.
Then, over here, most
importantly, they have a set
of constraints.
So we'll talk about a particular
constrain, the
plural constraint.
Plural constraint number one.
And it's going to reach around
and connect itself to some
other parts of the machine.
Finally, there's a buffer
of phones to be uttered.
And they're going to flow out
this way to the speaker's
mouth and get translated into
a acoustic wave form.
So those are the elements
of the machine.
Now how are the elements
connected together?
Well, the words are connected,
of course, into the buffer
that is used to generate
the sound over
here on the far left.
The plural register is
connected to what
you see in the world.
What you see in the world is
connected not only to plural
register, but to all of the
objects in the word
repertoire.
This plural constraint here
deserves extra attention
because it's going to be
desirous of actuating itself
in the event but the
thing observed in
the world is plural.
There are lots of them.
So it's going to be connected
then to the plural port.
There's going to be a z sound
port down here connecting to
that file element
in the buffer.
And finally, over here is going
to be a plussed voiced
port, which is going to be
connected to the second
phoneme in the sequence.
That's how the machine is
going to be arranged.
An of course, this is just
one of many constraints.
But it's a constraint that has
a very peculiar property.
Information can flow through
it in multiple ways.
So we think of most programs
as having an
input and an output.
But I try to be careful
to draw circles
here instead of arrows.
Because these are ports and
information can flow in any
direction along them.
What I want to do now is to
show you how this machine
would react if I suddenly
present it with a pair of
apples like so.
So the assumption is that the
vision apparatus comes in and
produces the notion, the
concept, of two apples.
So once that has happened--
that's operation number one--
then information flows from that
meaning register up here
to the apple word.
So that's part of stage
number two.
Another part of stage number two
is information flows along
this wire and marks that
as plus plural.
So operation number one is the
activity of the vision system.
Activity number two is the flow
of information from that
vision system into the
word lexicon and
into this plural register.
So far so good.
Here's activity number three.
This word is also connected
to the registers.
And information flows along
those wires so as to indicate
that it's a noun
but not a verb.
That's part of part
number three.
At the same time, part number
three, information flows down
this wire and writes a-p-l
into those are
elements of the buffer.
Now this constraint up here,
this box, says, well, I can
now see some stuff in
that buffer that
wasn't there before.
So it says, do I see enough
stuff on my ports to get
excited about expressing
values on other ports?
Well, let's see.
What has it got?
It's got the elements
in this buffer.
Also up here in step three
flow the plural thing.
So it know that the
word is plural.
So it says, is this voiced?
P is pa.
That's not voiced.
Is this a z sound.
No, that's not as z sound.
So it sees what it likes on only
one of its three ports.
So it says, I'm not going
to do anything.
I'm [INAUDIBLE].
I'm not in this particular
combat.
So far so good.
What happens next?
What happens next is that
some time passes.
And the elements of the buffer
flow to the left toward the
speaker's mouth.
So we get an a, p, l.
Same as we had before,
but shifted over.
Now what happens?
Now what happens is that
the l is now in
the penultimate position.
So information flows up here.
Item number four-- oh, I guess
that's item number five.
Item number four is the leftward
flow of the word.
So in phase number five, the
p is witnessed by this
constraint.
p is--
sorry, l is witnessed
by this constraint.
We moved it over one.
L is lll.
L is voiced.
So we have some flow
up here like that.
That's number five.
Now we have voiced and
we have plural.
And we have nothing here.
So there's a great desire of
this buffer to have something
written into it.
So now there's a flow
down in there, of z,
as item number six.
So that's how the machine would
work in expressing the
idea that there are apples
in the field of view.
Mmm.
Real apples.
Not plastic imitations.
So that's how the
machine works.
But all those connections
are reversible.
So if I hear apples then I get
the machine running backwards
and my visual apparatus
can imagine that there
are apples out there.
That's how it works.
That's just by way of background
the machine that
they could see it for using the
phonological rules once
they're learned.
All the phonological rules
are expressed in these
constraints.
But since these constraints are
such that information can
flow in any direction, they
deserve to be called
propagators.
And in the good old days when
everyone took 6.001, they
learned about propagators as
a kind of architecture for
building complex systems.
But in any event, there's
the Sussman-Yip machine.
And now comes the
big question.
How do you learn rule
rules like that?
Well, what we need is we need
some positive examples and
some negative examples.
And for the simple classroom
example I've chosen the same
challenge that I presented
to Krishna.
We're gonna have
cats and dogs.
So we're gonna look at the
distinctive features that are
associated with those words.
Syllabic.
Voiced.
Continuent.
And strident.
Just four of the 14 features
that are associated with each
of the sounds on those words.
Could you close the
laptop, please?
Just for the distinctive
features that are arrayed in
those words by way
of illustration.
So here we have k-a-t-z.
Phonetically spelled.
And if we work that
out, let's see.
What is syllabic?
That's not.
That is.
That is.
That's not.
Voiced?
Ka.
Nope.
Ah.
Yep.
T. Nope.
Z. Yes.
That can't be right.
Cats.
I misspelled it.
Because cats.
Sss.
His a hissing sound but
there's no voicing.
So that's not as z sound.
That's an s sound.
So that's not plus voiced.
It's minused voiced.
Continuent.
Let's see.
Is my mouth open when I say k?
No.
Ah?
Yes.
T?
No.
S?
Yes.
And strident.
Minus, minus, minus, plus.
It's only with the s sound that
I have that kind of jet
forming with my tongue.
Now we can look at dogs.
And now we have the z sound
as the pluralization.
We know that because when
we say it, dogzz.
Yep.
There it comes out as a--
we're only gonna look at the
last two columns because
they're the only ones that are
going to matter to us.
So that's plus.
And that's minus.
Gu, gu, gu, gu.
That's plussed.
And that's plussed.
They're both voiced.
Is that right?
Dogu?
Gu.
Gu.
Is g sound voiced?
Yeah, I didn't think so.
G sound is voiced?
Look-- oh.
Oh, it is voiced buy it's
not a continuent.
Just like that.
Yeah.
Cat, dogu zz.
Yeah.
It is voiced.
And it has to be for my
example to work out.
And that's minus, minus,
minus, plus.
So what we're interested in is,
how come one word gets an
s sound and how come the other
words gets a z sound?
Well, it's a pretty sparse
space out there.
We've already decided that
there are 14,000 possible
phonemes and there are only
40 in the language.
So that's one thing
we can consider.
The other thing that we can
think is that, well, maybe
this is a logical problem.
Like the kind of problem
you'd face if you
were designing a computer.
And so Sussman and Yip got
stuck for three months
thinking about the
problem that way.
Couldn't make any progress
whatsoever.
And that happens a lot when
you're doing a search.
You think you've got a way
of approaching it.
Try to make it work.
You stay up all night.
Stay up all night again.
Still can't make it work.
Eventually, you abandon ship
and try something else.
So then they began to say,
well, let's see.
All we care about is the stuff
before the two ending sounds.
We care about that part
of the matrix.
And we care about that
part of the matrix.
And we can ask, in what ways
are those things different?
And they're different
all over the place.
That's why they're
different words.
We can ask the question a
little bit differently.
And we can say, what can
we not care about?
And still retain enough of an
understanding of how the words
are different so as to put the
proper plural ending on them.
And they worried about
that for a long time.
Couldn't find a solution.
The search space was too big.
And then they said, maybe what
we ought to do is we ought to
think about generalizing this
guy here so that we
don't care about it.
So now we don't care
about that guy.
And then he went down through
here saying, well, let's see
when we have to stop
generalizing.
Because we've screwed everything
up and we can no
longer keep the z sound
words separated
from the s sound words.
So that eventually distilled
itself down to
the following algorithm.
First thing they did was
to collect positive
and negative examples.
And there's a positive example
and a negative example.
That's not enough
to do it right.
But that's enough to illustrate
the idea.
So the next thing they did was
something that's extremely
common in learning anything.
And that is to pick a positive
example to start from.
It's actually not a bad idea in
learning anything to start
with a positive example.
So they picked a positive
example and they
called that a seed.
So in our particular case, cats
is going to be our seed.
And the question we're going to
ask is, what are the words
that get pluralized like cat?
So we've got a positive
and negative example.
We've picked a seed.
And now, the next step
is to generalize.
And what I mean by generalize is
you pick some places in the
phoneme matrix that you
just don't care about.
So you may pick a positive
example.
And you don't care about it.
So you change it to an asterisk
or, as demonstrated
in the program I'm about
show you, a ball.
Or you pick one that's
negative and you
turn it to a ball.
Bo.
So cats, this seed,
becomes a pattern.
And in order to pluralize the
word this way, you have to
match all the stuff in here.
But now what we're going to do
is we're going to gradually
turn some of those elements into
don't care symbols until
we get to a point where we've
not cared about so much stuff
that we think that we
pluralize that one
with an s sound too.
So we keep generalizing until
we cover, that is to say we
admit or match, a negative
example.
So that's how it works.
So we generalize like crazy.
And as soon as we cover a
negative example, we quit.
Otherwise, we just go back up
here and generalize some more.
And now we've got to pick a
search technique to decide
which of these guys to actually
generalize when.
We could pick one at random.
And they tried that.
It didn't work.
So what they decided is that the
thing that influences the
pluralization most is the
adjacent phoneme.
And if that isn't the thing that
solves the problem, it'll
be the one next to that.
So in other words, the closer
you are, the more likely you
are to determine the outcome.
So these guys over here are
least likely to matter.
And those are the ones that
are generalized first.
So if we do that,
what happens?
Looks like we're going to come
in here and see that there's a
big difference between the
non-voiced t and the voiced g.
But that's only a guess because
I've only shown you a
fraction of the 14 distinctive
features that are involved.
So I suppose you like to
see a demonstration.
Yeah.
So there's our 14 features.
And that's our seed there,
sitting prominently in the
display with pluses and minuses
indicating the values
of the distinctive features
for all three
of the phones involved.
That funny left bracket
isn't a mistake.
That's just one convention for
rendering the ah sound in cat.
So it's pretty hard to tell from
just that matrix what's
going to be the determining
feature that separates the
positive examples from the
negative examples.
You notice that there
are actually two
examples down here.
There's cat and duck.
Is ducks got an s sound?
Ducks?
Yep.
So dogs and ducks.
They both get pluralized
with an s sound.
And then we have
beach doesn't.
That's beaches.
Dog.
We know that's a z.
Gun.
Gunz.
So that's not in the group.
So we can run this experiment.
Now here we go.
We're generalizing like crazy.
Generalizing, generalizing,
generalizing
from left to right.
So nothing in the first
two columns matters.
Now we get to the t.
Wow.
There it is.
So it looks like you pluralize
with a s sound.
The sss.
If, and only if, you're not
voiced and you're not strident
in the second to the last--
in the last phone
of the word that
you're trying to pluralize.
So that's one phonological
rule that
the system has learned.
And guess what?
It's the same rule
that's found in
phonological textbooks.
So now we can try another
experiment.
So this time we're trying to
deal with dog and gun.
And our negatives are what was
previously positive plus
beach, which is still in there
as a negative example.
So let's see how
that one works.
Nothing matters except for the
last column, the last phone.
And now we find out that if the
last sound is voiced, then
the pluralization gets the z
sound, a voiced determinator.
And finally, just to
deal with beaches.
That's beach in it's funny
phonetic spelling.
So now, if the final sound in
the word is strident, if its
got this jetty sound--
beach.
Beach.
Then it gets the ea sound.
So let's go back to experiment
number one.
Because I want to point out one
small thing about the way
this works.
You'll notice that it talks
about coverage and excluded
down here in the lower
left-hand corner.
Excluded, well, there are three
negative examples, so
they better all be excluded.
You don't want to cover
any of the negatives.
But it says coverage,
two and two.
That's because it actually
is doing--
and now we have the vocabulary
to say it quickly--
it's doing a beam search
through this space.
So it's not just doing
a depth first search.
It's doing a beam search so as
to reduce the possibility of
overlooking a solution.
So it says, oh, the coverage.
Both of the beam search elements
cover both of the
positive examples.
And they, in fact, have
converged to the same solution.
So that's how the Sussman
and Yip thing worked.
And then the next question
to ask is, of
course, why did it work?
And so the answer,
as articulated
by Sussman and Yip--
or rather more by Sussman.
Or rather more by Yip and a
little bit less by Sussman.
Yip thinks that it worked
because it's a sparse space.
And when you have a high
dimensional sparse space, it's
easy to put a hyperplane into
the space to separate one set
of examples for another
set of examples.
So let's consider the
following situation.
Suppose we have a
one-dimensional situation.
And we have two white
examples and we
have two purple examples.
Well, too bad for us you
can't separate them.
Now suppose that this is
actually the projection of a
two-dimensional space that
looks like this.
Here are the white examples
down here.
And here are the purple
examples up here.
Now it's easy to see that you
can separate them with just a
line that goes across
like that.
Now let's take this one more
step and suppose that this is
actually a projection of a
three-dimensional space.
It looks like this.
This will be dimension one.
This'll be two going
back there.
And this will be
three up here.
And suppose that the positive
examples are right
here on this line.
Let's say this is-- well, we're
gonna draw a little old
cube like so.
Those are purple examples
that are up there.
How many ways are there of
partitioning the space along
those axes?
Well, now they're not
even just two.
They're three.
So one way to separate the
purple from the white is to
draw a hyperplane-- or in this
case it's a three dimension,
so a plane--
through here on the
number three axis.
You could also put a plane
in on that axis.
Or you could do both.
So in one case your dividing
line would be--
let's see.
On the first axis that
would be 1/2.
And then the don't care.
Don't care.
Another solution that
would be don't care.
And then we divide on the number
2 axis with a plane at
1/2 and don't care.
Or we could do it with 1/2,
1/2, and don't care.
So the higher the dimension of
the space, the easier it is
sometimes to put in a plane
that separates the data.
That's why Sussman and Yip think
that we use so little of
possible phoneme space.
Because it makes the
thing learnable.
That's one possibility.
So one explanation for sparse
space is learnability.
There's another interesting
possibility, and that is that
if you have a sparse space, high
dimensional space with 14
dimensions, and if the 40 points
of your language are
spread evenly throughout
that space--
now let me say it
the other way.
If they are placed at random in
that space, then according
to the central limit theorem,
then they'll be about equally
distant from each other.
So it ensures that the phonemes
are easily separated
when you speak.
But if you go to ask a linguist
if that's true, they
don't know.
Because they're not looking
at it from a
computational point of view.
Well, we can look at it from a
computational point of view.
So I did that.
After Sussman and Yip published
their paper.
And here's the result.
This is a diagram that shows all
of the phonemes that are
separated by exactly one
distinctive feature.
So if you look over in this
corner here, you'll see that
the constants-- w and x--
are separated by exactly one
distinctive feature.
So they're not exactly distant
from each other in the space.
On the other hand, they are
pretty easy to separate
relative to the vowels.
Which are here in this
part of the diagram.
Which are all tangled up and
the vowels are all close to
each other.
So guess what?
Vowels are much harder to
separate than constants.
Not surprisingly, because there
are many pairs of them
that are different.
And only one distinctive
feature.
All right.
So now you back up and
you say, well, gosh.
That's all been sort
of interesting.
But what does it teach
us about how to
do science and stuff?
And what it teaches us is--
this is an example.
Ow.
This is an example which we can
use to illuminate some of
thoughts of David Marr, who
I spoke of in a previous
lecture, connection
with vision.
But here's Marr's catechism.
I can't spell very well so I
won't try to respell it.
But this is Marr's catechism.
So what Marr said is, when
you're dealing with an AI
problem, first thing to do is
to specify the problem.
Gee, that sounds
awfully normal.
The next thing is to devise
a representation
suited to the problem.
The third thing to do,
vocabulary varies, but it's
something like determine
an approach.
Sometimes thought
of as a method.
And then four, pick a mechanism
or devise an algorithm.
And, finally, five,
experiment.
And of course, it never goes
linearly like that.
You start with the problem and
then you go through a lot of
loops up here.
Sometimes even changing
the problem.
But that's just the scientific
method, right?
You start with the problem
and you end up with the
experiment.
But that's not what people in
AI, over the bulk of its
existence, have tended to do.
What they tended to do is to
fall in love with particular
mechanisms.
And then they attempt
to apply those
mechanisms to every problem.
So you might say, well, gee,
neural nets are so cool.
I think all of human
intelligence can be explained
with a suitable neural net.
That's not the right
way to do it.
Because that's mechanism envy.
You fall in love
with mechanism.
You try to apply it where it
isn't the right thing.
This is example starting with
the problem and bringing to
the problem the right
representations, gosh,
distinctive features.
Once we've got the right
representation, then the
constraints emerge, which
enable us to devise an
approach, write an algorithm,
and do an experiment.
As they did.
So this Sussman-Yip thing is an
example of doing AI stuff
in a way that's congruent with
the Marr's catechism.
Which I highly recommend.
They could have come in here and
said, well, we're devotees
of the idea of neural nets.
Let's see if we can make a
machine that will properly
pluralize words using
a neural net.
That's a loser.
Because it doesn't match the
problem to the mechanism.
It tries to force fit the
mechanism into some
Procrustean bed where
it doesn't
actually work very well.
So what this leaves open, of
course, is the question of,
well, what is a good
representation?
And here's the other half
Marr's catechism.
Characteristic number one
is that it makes the
right things explicit.
So in this particular
case, it makes
distinctive features explicit.
Another thing that Marr was
noted for was stereo vision.
So in that particular world,
discontinuities in the image,
when you go across an
edge with the things
that were made explicit.
Once you've got to a
representation that makes the
right things explicit, you can
say, does it also expose
constraint?
And if you have a representation
that exposes
constraint, then you're
off and running.
Because it's constraint that
you need in order to do the
processing that leads
to a solution.
So don't have the right
representation.
If it doesn't expose
constraints, you're not going
to be able to make a very
good model out of it.
And finally, there's a kind
of localness criteria.
If you have a representation in
which you can see the right
answer by looking at
descriptions through soda
straw, that's probably a better
representation than one
that's all spread out.
It's true with programs,
right?
If you can see how they work
by looking through a soda
straw, you're in much better
situation to understand
something if you have to look
here and there and on the next
page and in the next file.
So all this is basically
common sense.
But this is kind of common sense
that makes you smarter
as an engineer and scientist.
Especially as a scientist
because if you go into a
problem with mechanism envy,
you're apt to study mechanisms
in a naive way and never reach
a solution that will be
satisfactory.