Mega-R4. Neural Nets Video Lecture - Artificial Intelligence A Fundamental

the following content is provided under
a Creative Commons license
your support will help MIT
OpenCourseWare continue to offer high
quality educational resources for free
to make a donation or view additional
materials from hundreds of MIT courses
visit MIT opencourseware at ocw.mit.edu
today we are introducing an exciting new
pledge in 603 for anyone who has already
looked at any of the neural net problems
well have easily been able to see that
even though Patrick only has them back
up to 2006 now there still are well out
of 4 tests perhaps two or three
different ways that the neural nets were
drawn our exciting new pledge is we're
going to draw them in a particular way
this year and I will show you which way
assuming that this works yes we were
going to draw them like the way on the
right the one on the left is the same as
the one on the right at first not having
you had to explain the difference
between the two of them you might think
you want the one on the left but you
really want the one on the right and
I'll explain what the 2007 quiz was
drawn roughly similarly to this also if
you somehow wind up in tutorial or
somewhere else doing one of the older
quizzes a lot of them were drawn exactly
like this in this representation one
thing I really don't like is that the
inputs are called X's and the outputs
are called wise but like there's two x's
so the inputs are not x and y and then
they often correspond to axes of a graph
and then people get confused additional
issues that many people have are the
fact that the summation and
multiplication with the weight is
implied the weights are written on the
edges where outputs and inputs go and
the summation of the two inputs into the
node are also implied but take a look
here this is the same net these um these
WS here would be the W's that are these
WS a would be
views that are written into onto these
lines are here actually the better way
to draw it would be like that a way to
draw it would be like so since each of
these can have their own W like she's
just a friend so each of the W's that
are down here are being explicitly sent
to a multiplier whereas here you just
had to remember to multiply the weight
by the input that was coming by here you
see an input comes to a multiplier you
multiply by the weight then once you've
multiplied all the inputs by the weight
you send them to a sum that's the Sigma
is just a sum you sum them add them all
up together send the result of that into
the sigmoid function our old buddy 1
over 1 plus e to the negative whatever
our input was with a weight for an off
set and then we set the result of that
into more multipliers with more weights
more sums more save lines so this is how
it's going to look like on the quiz and
this is a conversion guide from version
you know 0.9 beta into version 1.0 so if
you see something that looks like this
on one of the old quizzes that you're
doing see if you can convert it and then
solve the problem chances are if you can
convert it you're probably going to do
fine so we'll start off not only with
this conversion guy but also
yeah I'll leave that up here also we're
gonna I'm gonna work out the formulae
for you guys one more time the these are
all the formulae that you're going to
need on the quiz and then we're going to
decide what will change in the formulae
if and this is a very likely if there's
seems to be a good amount of times that
this happens if the sigmoid function in
those neurons up there was ever changed
into some other kind of function it's
change it to a plus already the problem
we're about to do people change it all
the time it's just some bizarro function
I've seen arctangent I think so here we
go let's look at the formula first all
sigmoid well our old buddy sigmoid I
just said it a moment ago sigmoid is 1
over 1 plus e to the minus X also fun
fact about sigmoid um the derivative of
sigmoid of sigmoid is itself the
derivative of sigmoid is let's say that
the sigmoid will just turn sigmoid into
like the letter say why why is the
result right so if you say y equals 1
over 1 plus e to the negative x then the
derivative of sigmoid is y times 1 minus
y you can also write out the whole nasty
thing it's 1 over 1 plus e to the
negative x times 1 minus 1 over 1 plus e
to the negative x it's a nice property
of sigmoid is going to be important for
us in the very near future
and that future begins now so now the
performance function this is a function
we use to tell our neural nets when they
inevitably act up and give us really
crappy results at first we tell them how
just how wrong they are with our
performance function the performance
function can be any sane function that
gives you a that gives you a better
score where better can be decided is
lower or higher if you feel like it that
but gives you a better score if your
answers are closer to the answer you're
looking for
however in this case we have with a very
sneaky for very sneaky reason on chosen
preference function to be 1/2 D which is
the desired output - oh the actual
output squared so we want a small well
it's negative so we want a small
negative or zero that would mean we
perform well so why this well the main
reason is D DX of performance is the -
comes down the o is the variable that
we're actually doing so maybe I should
say D do um that negative comes out we
got a simple D minus L and yeah we're
using derivatives here so not those are
fine these are two assumptions they
could be changed on your test we're
going to figure out what happens if we
change them if we change the performance
if we save the sigmoid that is
we features a sigma into some other
function what's going to happen to the
next three functions which are basically
the only things that you need to know to
do back propagation so let's look at
them first w prime this is the formula
for a new weight after one step of back
propagation a new weight in any of these
positions that you can see up here on
this on this beautiful neural net that W
all each of the WS will have to change
step by step that's in fact how you do
the hill climbing neural nets you change
the weights incrementally you step a
little bit in the direction towards
giving you your desired results until
eventually you hope you have an
intelligent neural net and maybe you
have many different training samples
that you run it on in a cycle I'm hoping
that you don't over fit to your one
sample on a computer but on the test we
won't probably will not do that so let's
take a look at how you calculate the
weights for the next level given that
you have the width for the current level
so first things first new weight weight
prime equals starts with the old weight
that has to go there because otherwise
we're just going to jump off somewhere
at random we want to make a little step
in some direction so we want to start
where we are with the weight and then
we're going to add three things so if
we're talking about the weight between
some I and some J here are some examples
of the names of weights so this is w1 I
that's the way between one and so this
is w1 a so weight between one and a this
is w2 B which is the weight between two
and B makes sense well it makes it so
far but what what if it's just called WB
then it's the way between then these
double use that only have one letter
we'll get to later there's a bias there
the offset they are always attached to
negative one so you can pretty much
treat you
pretty much treat their as being like a
negative one here that is then decided
to a multiplier with this WB if you like
this is implied to be that all the
offsets are implied to be that so W plus
some alpha y is a Greek letter where
does it come from well how do we
calculate it well
alpha is just some value told to you on
the quiz you'll find it somewhere
there's no way you're going to have to
calculate alpha you might be asked to
try to give a say in alpha but probably
not alpha is supposed to give it the
size of our little steps that we take
when we're doing hill climbing very
large alpha take a huge step very small
alpha take tentative steps so alpha is
just there to change basically to change
this answer in make to make the new
value either very close to W or very far
from W depending on our taste so plus
alpha times I so I is the value coming
in the I is the value coming into the
into the node we're changing the weight
here so I is the I is the value for
instance I sub one here I would be that
I a would be the value at W AC I would
be the value coming output of W of of
node a at WBC I would be the output of
node B I is sometimes specifically
labeled as I I is the input coming in to
meet that weight at the multiplier and
then it's multiplied by Delta J where
Delta is the Delta that belongs to these
neural net nodes what is a Delta u say
finding you may ask it is a strange
Greek letter it sort of comes from the
feather we're doing
some partial derivatives and stuff but
the main way you're going to figure out
what the deltas are are these two
formulae that I have not written in yet
so hold off on trying to figure out what
the Delta is until well right now
because I'm about to tell you what the
Delta is so the Delta is basically think
of the Delta as using partial
derivatives to figure out which way
you're going to step when you're doing
hell climbing because you know when
you're doing hill climbing you like look
around you figure out okay this is the
direction of the highest increasing then
you step off in that direction so the
deltas are sort of telling you which way
to step with the weights and the way
they do that is by taking the partial
derivative of basically you try to
figure out how the weight that you're
currently looking at is contributing to
the performance of the net contributing
to either the good performance of the
net or the bad performance of the net so
when you're when you're dealing with
like when you're dealing with the
weights like WBC WACA that pretty much
directly feed into the end of the net
they feed into the last node it then
comes out it's the output okay then
that's that's pretty easy you can tell
exactly how much those weights and the
values coming from them are contributing
to the end and we do that by first we do
that by essentially we'll remember what
the partial derivative so partial
derivative here is in fact the putt is
in fact the way that the final weights
are contributing to the performance is
just the performance function partial
derivative well I've already figured out
the derivative here it's just D - oh
this is for this is for sort of final
weights the weights in the in the last
level do you - how it's that we're not
done yet because we're
do derivatives remember the chain rule
to get from the end to these weights we
pass through well it should be a sigmoid
here it's not we're going to pretend it
is pretend it is for the moment we pass
through the sigmoid and since we passed
through the sigmoid we better take the
derivative of the sigmoid function that
is y times y minus y well what is y what
is the output of this sigmoid it's Oh so
that's also x Oh times 1 - up however
there is a let me see let me see
yes sorry I'm carefully studying the
shape to make sure my nomenclature is
exactly right for our new nomenclature
which so new and brave that we're going
doing it that we only knew for sure we
were going to do it on Wednesday so we
have D minus L times o times 1 minus L
so you say that's fine that can get us
you know these waves here even this w
see how are we going to get the how
we're going to get the deltas for the
two tablets for the new the deltas for
the new weights here oh I realize um
yeah I got it so the Delta this is the
by the way this is a delta at sea
how is neuron sea contributing to the
output wall it's directly contributing
to the output and it's got a sigmoid in
it it doesn't really but we're
pretending it does for now do you - Oh
times
Oh times 1 - ah what about an inner node
node B node a what are we going to have
to do well the way they contribute to
the output is that they contribute to
node C so we can do this problem
recursively so let's do it recursively
first of all as you probably figured out
all of them are going to have it o times
1 - Oh fat ran from the
because they're all sigmoids well
pretending that they're all sequins so
we actually have a dearth of good
problems that are actually sigmoid on on
the web right now there's only 2007 but
here's old times one - oh so what are we
going to do for the rest of it how does
it contribute to our final result well
it contributes to our final results
recursively so we're talking about Delta
I is an inner is sort of an inner node
it's not at it's not a final note it's
somewhere along the way so sum over J of
W going from I to J times Delta J now
sum over all J J such that I leads to J
I need to have a direct path into J so
if I if I in this instance was a
everyone the only possible J in this
would be C that's right we would not sum
over B as one of the J because I does
not lead to two B or a does not lead to
be a only leads to C also note that C
does not need to be here that's going
backwards so you just to figure out
which J you're looking at look directly
forwards at the next one so if there is
another D here or something like that a
does not go to DA goes to C you only
look at the next look at the next levels
children and you sum over all those all
those children to wait between them
multiplied by the child's delta that
makes sense right because the way we
affect if the child's Delta is the way
the child affects the output calling
these children for a moment if the then
if this this one directly affects the
output then the way this one affects it
is you use the it effects it because it
affects this but it's also multiplied by
this wait
so in fact for instance if the weight
between a and C was zero then a doesn't
affect the output at all right
because its weight is zero and when we
do this problem we go this times zero
and then we try to add it in there it
doesn't affect anything its weight is
very high it's going to really dominate
um it's going to really dominate it see
and that is taken into account here and
then multiplied by the Delta for the
right node so I posed the following
question and since I spent a lot of time
with formula and not that many time that
much time starting on the problem I will
not call on someone at random but rather
take a volunteer if no one volunteers
I'll eventually tell you which is we've
got some nice formula for on the bottom
three if we change the sigmoid function
what has to change that's right the only
thing that changes in this crazy-ass
problem right here which by the way
changes the sigmoid functions two adders
is that we take all the Oh times one
minus zero in Delta F and Delta I
equated in and we change it to the new
derivative we then do the exact same
thing that we would have done correct
and on a similar note if we choose the
performance function how many of these
equations that all have to change on the
bottom three yeah
that's right just one just Delta F take
the D minus o make it the new derivative
of the new performance function and in
fact out the I doesn't change at all
does everyone see that because it is
very common for something to be replaced
I think three of the four the quizzes
that we have replaced in something
change something in some way all right
let's go we're going to do 2008 quiz
because it has a part at the end that
screwed up everyone and so let's make
sure we get to that part that's going to
be the part that you probably care about
the most at this point so these are all
adders instead of sigmoids that means
that they
simply add up everything as normal for a
normal normal net and then there's no
sigmoid threshold they just give some
kind of value question they're not
neural net nodes that is one of the
reasons why one of the reasons why that
other form that you can see over there
is elegant it only has the actual nodes
on it it's very compact it's one of the
forms we've used in previous tests the
question is to those multipliers Kaunas
knows however by not putting in the
multipliers we feel it sometimes
confuses people of explicitness the ones
that are nodes will always have a label
like a or here you see there's a sigmoid
and an l1 um the multipliers are there
for your convenience to remind you to
multiply and also those if you look
those Sigma's that are over there you're
there for your convenience to remind you
to add but in fact the only thing that
counts as a node in the neural net
that's a very good question
is usually the sigmoid here it's the
adders we've essentially taken out the
signal which these adders are the oh
here's the way to tell if it's got a
threshold weight associated with it then
it's one of the actual nodes a threshold
weight I guess the multipliers look like
they have a way but this is just the
weight that's being multiplied in to
this is a witness being multiplied in
with the input but if it has a threshold
weight just like waa WBO I promised I
would tell you guys the difference
between the two weights so let's do that
very quickly the kind of weights that
that say w2 b or w1 a our weight that
comes between input 1 and a or between a
and C they're meant to be multiplying
the input by this weight and then
eventually that's added together the
threshold weights they just have a like
WB waw see they are essentially to
decide the threshold for a success or a
failure for a 1 or 0 um
or anything in between at any of the
given nodes so the idea is maybe you at
some node want to have a really high
cutoff you have to have a very high
value coming in or else it's a zero so
you put a high threshold the weight is
multiplied by negative one and in fact
in fact the threshold weight well one
could consider if you wanted to that the
threshold weight times negative one was
also added in it that some instead of
putting it on on the same location as
the node if that works better for you
when you're converting it you can also
think of it that way because the
threshold weight is essentially
multiplied by negative one and added in
it that same sum over there so that's
what that is another way to do it um
there's a lot of ways to visualize these
neural Nets just make sure you have a
way that makes sense to you and that you
can tell pretty much whatever we write
as long as it looks vaguely like that
how to get it in your mind into the
representation that works for you
because once you have the representation
right for you you're more than halfway
to solving these guys they aren't that
bad they just look nasty they don't bite
okay so these are just adders so if it's
just an adder then that means that if we
take all that we take are all the X in
puts coming in let's do x and y for the
moment so we can figure out the
derivative then what comes out after
after we just add up the X what comes
out y equals x right we're just adding
it up I'll do adding up all the input
we're not doing anything to it y equals
x is what this node does do people see
that so the derivative is just 1 so
that's pretty easy because the first
problem says what is the new formula
Delta F so I'll just tell you you guys
probably figure it out it's a times 1
minus o because we replace D minus L
with 1
ok make sense so far please ask
questions along the way because I'm not
going to be asking you guys all
with myself question oh that's a good
question the reason is because I did the
wrong thing so see it's good that you
guys are asking questions it actually
should be replacing Oh times one - all
with one the answer is Delta F equals D
- oh so yes perhaps I did to trick you
know I actually messed up but yes please
ask customs on the way again I don't I
don't have time to call on you guys at
random to figure out if you guys are
following along so I'll do it myself
we're placing the Oh times 1 - whoa with
1 because of the fact that the sigmoid
is gone and we get just Delta F equals D
minus L so great we will we now want to
know what the equation is for Delta I at
the node a so Delta a well let's take a
look
the Oh times what - what was gone now we
just have the sum over J which is we've
guys already told me is only C of W AC
times Delta C and we know that Delta C
is D - oh so the answer is Delta a is
just W a C times D - oh that time I got
right I see the answer here so it's
written in a very different format from
the old quiz any questions on that
well that's part a that we finished out
of C let's go to Part B Part B is doing
one step of back propagation there's
almost always going to be one of these
in here so the first thing it asks is to
figure out what the output oh is for
this neural net if all weights are
initially one except that this guy right
here is negative point five all the
other ones start off is one so let's do
a step Oh what well so let's see what
are the inputs the inputs are also all
one the desired output is also one and
in fact the rate constant alpha is also
one this is the only thing that isn't
one folks
so let's see what happens 1 times 1 is 1
then this is negative 1 times 1 is
negative 1 that's 0 the exact same thing
happens here because it's symmetrical so
these are both 0 0 times 1 is 0 0 times
1 is 0 then this is negative 1 times
negative 0.5 is positive 0.5 so 0 plus 0
plus positive 0.5 the output is positive
point 5 does everyone see that if not um
you can convince yourself that it's
positive point 5 that it would be a good
exercise for you running to one forward
run the output is definitely positive
point 5 first time around okay now we
have to do one step of back propagation
to do that let's calculate all the
deltas so that we can calculate all the
new weights the new weight primes so
Delta see that's easy you guys can tell
me what Delta C is we figured out what
the new Delta C is going to be it's a
simple addition or subtraction problem
every one Delta C is 45 one half yes
alright and we know that Delta a and
Delta B are just W AC times help to see
a WBC times Delta C so they are also
one-half because all the weights were
one easy street ok we've got all the
deltas are one half and all but a few of
the weights are 1 so let's figure out
what the new weights are so oh ma new w
AC ok so yeah so let's see what's going
to be the new W AC so the new W AC is
going to be old W so old WACA which is
one because all of them are one except
for WC plus the rate constant which is
one times
the input coming in here remember that
was zero so actually it's just going to
be the same as the old WC and since it's
this is a symmetrical problem between
being a at the moment this is going to
be the same
all right some things are gonna change
though what about WC that was the one
that was actually not one okay so new WC
WC remember the i4w see the the the I
that we use in this equation is always
negative one because it's a threshold so
we have the old WC which is negative 0.5
plus one x plus one times negative one
times Delta C which is one half so we
have negative point five plus negative
0.5 equals negative 1 w1 a well we've
got w1 a starts as 1 then we also know
that we're w1 a is going to be equal to
1 plus 1 times the input which is 1
times Delta of a which is 1/2 so 1.5 and
since it's symmetrical between a and B
then W to be there's also 1.5 and then
finally W a and W be the offsets here
well they start at 1 plus 1 times
negative 1 times 0.5 so they're they're
both everyone Wow
that's right
that's right because negative one is
there I negative 1 times 1/2 plus
positive 1 is just 1/2 that's one full
step maybe it might easier then you
might be used to seeing but there's a
full step and ask what's going to be the
output after one step of backpropagation
well we can take a look so we have 1
times the new w1 a which is 1.5 we've
got 1 point 5 then the new W a is just
0.5 9 is point 5 that's a 1 coming into
an adder we've got another one coming in
here because it's symmetrical so 1 and a
1 1 times W AC is 1 1 times WBC is 1 so
we have 2 ones coming in here they're
added that's 2 then this has become
negative 1 in fact at this point so
negative 1 times negative 1 that's 3 and
the output is 3 all right cool we've now
finished Part B which is worth over half
of everything
oh no we've not one more thing these are
adders they're not sigmoids what if we
train this entire neural net to try to
learn this data so that it can sort of
draw a line on the graph or draw some
lines or do some kind of learning to
separate off the minuses from all the
pluses you've seen maybe and if not
you're about to in a second because they
it asks you to do this in detail that
neural nets can usually draw one line on
the graph for each of the sort of nodes
in the net because each of the nodes has
some kind of threshold and you can do
some logic between them like ands and
ORS so what do you guys think this net
is going to draw if the anyone can
volunteer I'm not going to ask anyone to
give this answer now a little bit tricky
because usually if you had this many
nodes you could easily draw like a box
and box
off than minuses from the pluses however
it draws this and it asks what is the
error the error is oh yeah it even tells
you there is one-eighth because why
these are all adders you can't actually
do anything logical because this entire
net boils down to just one node because
it just adds up every time and it never
takes a threshold at any point so you
can't turn it into logical ones and
zeros because it's basically it's not
digital at all it's analog it's giving
us some very high number so it all goes
onto one cut off and that's the best one
the one that I drew right here okay did
that not make sense to you that's okay
this problem is much harder and putting
them both on the same quiz is a bit was
a bit brutal but by the time you're done
with this you will understand what a
neural nut can do or not I put these in
simplified form because of the fact that
we don't care about their values or
anything like that
but inside of these little circles is a
sigmoid the multipliers and the summers
are implied just so that I think in the
in the simplified form when we're not
actually doing back propagation it's
easier to view it and see how many nodes
that are for the same reason you asked
your question about how many there are
so all of those big circles are node and
in those node is a sigmoid now not those
crazy adders we have the following
problem
we have to try to match each of ABCDEF
to one two three four five six using
each of them only once that's important
because of the fact that some of the
more powerful networks in here to do a
lot of these so it's like yes the
powerful networks could do some of the
easier problems here but we want to
match each net to a problem it can do
and there is exactly one mapping that
will map but that is one-to-one and maps
exactly uses all six of the nets to
solve all six of these problems here so
some of you may be looking at like what
how am I going to solve these problems I
gave away a hint before which is that
each node in the neural net if sigmoid
node can usually draw one line
on the it can draw one line into the
picture the line can be diagonal if that
node receives both of the inputs which
here I 1 and I 2 you see there's an i1
and i2 axis like X and y axis the node
has to be horizontal or vertical if the
site the line has to be horizontal
vertical if the node only receives one
of the inputs and then if you have a
deeper level these these secondary level
nodes can sort of do a logical boot can
do some kind of boolean thing like and
or or of the first two which can help
you out all right
and so let's try to figure it out so
right off the bat and I'll hope that
people will help and call clothes out
because I know we don't have enough time
that I can force you guys to all get it
right off the bat which one of these
looks like is the easiest one six that's
great
six is definitely the easiest one it's a
single line so this is just how I would
have solved this problem is find the
easiest one now which of these the
crappiest net a a is the crappiest net
fact there's no way in hell that a is
going to be able to get any of these
except for a six so let's right off the
bat say that six is a all right six is a
that's a we don't have to worry about a
okay cool now let's look at some other
ones that are very interesting all the
rest of these draw two lines explode
these three draw two lines these three
draw three lines they draw triangle so
despite the fact that this C is a very
powerful neural net indeed with three
whole levels here of sigmoids it looks
like there's only two that's in our
little stable of nets that are equipped
to handle number one and two and those
aren't a and F because E and F have
three nodes at the first level they can
draw three lines and then they can do
something logical about those lines like
for instance baby if it's inside all of
those lines there's a way to do that
you just basically can give negative and
positive weights as you so choose to
make sure that it's under certain ones
above other ones and then make the
threshold such that it has to follow all
three of your rules
so between E and F which one should be
two and which one should be one anyone
see well let's look at two and one which
one of these is easier to do between two
and one two it's got a horizontal and a
vertical one has all three diagonal and
which one of these is a weaker net
between E and F F F F has one node that
can only do a horizontal and one note
that can only do a vertical line so
which one is f gonna have to do to an e
does one good job guys good job guys you
got this so now let's look at the last
three number three is definitely the
hardest it's an X or those of you who
played around with double to kind of
stuff or even just logic probably know
that there is no way to to make a sort
of simple linear combination of in one
level of logic to create an X or X or is
very difficult to create there are some
interesting problems involving trying to
trying to teach an X or into an oral net
is it knowing that as simple of it just
not going to be able to get the X or
because of the fact that you can tell it
okay I want this one to be high and this
one to be low that's fine you can say
these both have to be high that's fine
it's hard to say like it's pretty much
impossible to say this one or this one
but not the other because of need to be
high in a single node because of the
fact that if you just just play with it
you'll see you need to set a threshold
somewhere it's and it's not going to be
able to distinguish between if the
threshold is set such that the or is
going to work the hole or is going to
work it's going to except when both of
them are positive as well so how are we
going to do an extraordinary
and in orders in a two-level way and to
do that we need the deepest nor let that
we have there's only one that's capable
of that and that is it's C so you there
are many different ways to do it let's
think of a possibility I wanted I to
draw these two lines or so now I went
inside to do these two let's call these
one two three four five of node 1 and
node 2 draw these two lines I'll just
sort of draw it here for you guys then
maybe node 3 gives value to um you let
me see node 3 can give value to perhaps
and nope let's see notes we can give
value to everything that is oh I mean
there are a lot of possibilities here
note 3 could give value to everything
that is up here actually what note they
can give value to everything except for
this bottom part and then node 4 could
give value to say oh no doesn't do it
yet right there's a few there's a
there's a few different ways to do as
you play it around the key the key idea
is that node 3 and node 4 can give value
to some combination of and/or or not and
then node 5 can give value based on
based on being above or below a certain
threshold combination of 3 and 4 you can
build an XOR out of the logic gates I
will I will ponder on that in the back
burner for a moment as we continue
onward but clearly C has to do number 3
ok
now we're left with 4 & 5 I think
intrinsically 5 looks like it may be
more complicated than 4 because of the
fact that it needs to do both different
directions in step 2 of the same
direction
however there's um and so however just
the idea of the one with the fewer lines
being the simpler one may not get us
through here
there's a reason why look what we have
left to use we have to use D or B what
is the property of the two lines of D
can draw D being the simpler one
one horizontal one vertical that's right
so even though it may look simpler to
just have two horizontal lines it
actually requires B B's the only one
that can draw two horizontal lines
because D has to draw one horizontal on
one vertical so that leaves us with B on
this D on this and excellent we have a
question I would have thought it would
be impossible we had no questions or
maybe I just explained this the best I
ever have question all right
so the question is I don't understand
why B has to be two horizontal lines the
answer is it doesn't um B can be
anything but D can't be two horizontal
lines and so by process of elimination
is B well I'll take a look at D right so
D has three nodes 1 2 3 node 1 and no 2
can just draw a line anywhere they want
involving the inputs they receive what
input does no one receive we let's go to
node 1 oh yeah so it can only make a cut
off based on I 1 so therefore it can
only draw by making a cut off above
below I a certain putting I 1 it can
only draw vertical line know two can
only draw a horizontal line because it
can only make a cut off based on where
it is in I 2 therefore they can't both
draw a horizontal that's why this is the
trickiest part this last part because B
is more powerful B does not only have to
do Torre's on the lines it can do two
lines two diagonal lines it can do
anything at once it just happens that
it's stuck doing this somewhat easier
problem because of the fact it's the
only one left that has the power to do
it
so let's see we've done we're done and
we have aced this part of the quiz that
like no one got well not no but very few
people got when we
put it on in 2008 the only thing we have
left to ask is let me see the only yeah
the only thing we have left to ask is
what are we going to do here for this
alright let's see for the XOR let's see
if I can do this XOR so ok um how about
this one ready I'm an idiot this is the
easiest way number one
draws this line number two draws this
line number three and the line the two
lines number three says only if both of
them are true while I accept number 4
knots the two lines and number five ORS
between 3 and 4 thank you no it's not
that hard it I just completely blanked
there's because there's another way that
a lot of people like to do it involves
drawing in a lot of lines or then making
the cutoff be 2 but I can't remember
right at the moment are there any other
question because I be I think that if
you have a question now like for other
people have it and just start raising
their hands so ask any questions about
this drawing thing question why do we do
this drawing thing that's a very good
question the answer is so that you can
see what kinds of nets you might need to
use in these simple problems to to
answer these simple problems so that if
you know Athena forbid you have to draw
using neural net in a job somewhere to
do some actual learning and you see some
sort of quality about the problem you
know not to make a net that's too simple
for instance and you wouldn't want a net
that's more complex than it has to be so
that you can sort of see what the nets
do at each level and more visibly
understand I think a lot of the people
who drew problems like this just won't
make sure people know oh yeah it's not
just these numbers that were mindlessly
back propagating from the other part of
the problem to make them higher or lower
this is what we're doing
each level this is what we're this is
the space that we're looking at each
node is performing logic on the steps
before so that if you actually have to
use and oral that later on down the road
well then you'll be able to you'll be
able to figure out what your next can
need to look like you'll be able to
figure out what it's doing at least as
well as you can figure out what it's
doing for a neural net since it often
will start getting up these really crazy
numbers you'll have all sorts of nodes
in a like a real neural net that's being
used nowadays there will be tons of
nodes and you'll just see the numbers
fluctuating walled-in and suddenly it's
going to start working or not that's a
good question any other questions we
still have a few minutes not many but a
few any other questions on any of the
stuff question okay okay
you're confused why the machine is
learned by what by the pictures on the
right oh okay
machine does not have to learn by
drawing pictures and calling them in let
me give you some real applications my
friend at University of Maryland
recently actually used neural nets
um because yeah he actually did because
of the fact that he was doing a a game
playing competition where the game was
not known when you were designing your
AI it had to be able to there was some
very elegant general games solver thing
that you had to be able to hook up into
and then they made up the rules and you
had a little bit of time and then it
started so some of the AIS what they did
was they trained once they found out
what the rules were on their own with
the rules in his case he had a neural
net because it was so generic you just
have a web of random stuff random goop
he thought it could learn anything and
then um he never did tell me how it went
it probably didn't go well but maybe did
um it basically tried to try a try to
learn some things about the rules some
of the other people who were more
principled game players actually tried
to find out fundamental properties of
the space of the rules by testing a few
different things so they could you know
more knowledge is less search so they
could do less search
when the actual game-playing came on and
then when the actual game-playing came
on pretty much everyone did some kind of
game tree based stuff
he talks telling me that a lot of
Montecarlo based on game Street stuff
that is this very non-deterministic is
what they're doing nowadays rather than
our more deterministic alpha beta
although he said it converges to alpha
beta if you give it enough time that's
what he told me but that's someone I
know who is using neural Nets I've also
in a cognitive science class I took saw
neural nets that tried to attach like
qualities to objects by having just this
huge huge number of nodes in levels in
between and then eventually it was like
a duck flies and you're like how is it
doing this again I'm not sure but it is
so the basic idea is that the way one of
the main reasons rolled upward used so
much back in the day is the people on
many different sides of this problem
cognitive science AI whatever we're all
seeing wait a minute there's networks of
neurons and they can do stuff and we're
seeing it in different places and when
you see it in so many different places
at once must be a genius idea that's
going to revolutionize everything and so
then everyone started using them to try
to connect all these things together
which i think is a noble endeavor but
unfortunately people just stop using it
it didn't work as they had wanted it
turned out that figuring out how our
neurons worked in our head was not the
way to solve all a I heart problems at
once and they fall into disfavor
although are still used for some reasons
like some reasons like that so we
wouldn't use it just to draw these
pictures the reason why we have these
pictures is because we give you simple
nets that you can work out by hand on
the quiz um any net that was really used
nowadays would make it would make you
your head explode if we tried to make
you do something with it on the quiz it
would just be horrible looking so I
think that's a good question if there's
no other questions or even if there are
because we have to head out if there's
any other questions you can see me as
I'm walking out