Training Data - Training a neural network to play a game with TensorFlow and Open AI p.2 Video Lecture | Machine Learning with Python - AI & ML

what's going on welcome to part two of
our deep learning with games open and I
attend shurflo Python all other
buzzwords tutorial in this tutorial what
we're going to be doing is building on
last one which basically we just kind of
showed what what the random games
actually looks like this one you can get
an idea so when we see the real thing
after it's been trained we can be like
impressed so so let's assume that let's
go ahead we're gonna comment that random
games out and now we're going to
actually define our initial kind of
population of data so when we have an
environment that we can generate like
this we can generate training samples so
in this case there are imperfect
training samples because none of these
samples are probably going to beat our
required score but we're going to be
able to make some samples anyway so the
other things you can do like for example
like if you're trying to treat treat it
train in neural network to do math you
actually all those samples could in
theory be perfect you can just generate
random you know just math numbers plus
another number or multiply divide
whatever and save that as well and then
you know so you can generate data really
quick so that's what we're going to do
here so now let me just make some space
and we're going to define initial
population so first I'm gonna have
training data just an empty list for now
we're gonna scores another empty list
and then we're going to have accepted
school training data will be the actual
data that we're interested in training
on this will be the observation and the
move made again the moves will all be
random but we're only going to append a
training data if the score happens to be
above 50 now once we've done that we're
going to say just for underscore in
range of initial gameplay underscores
just there to denote that we don't give
a crap what this is we're just trying to
it
so underscores we're going to say the
score is currently zero we're going to
save some game memory because we don't
work we're not going to know until the
end of the game whether or not we did we
beat the score or not so we're going to
store all the movements and such in game
memory then what we're going to say is
the previous aw servation happen so
previous observation empty list and now
what we're going to do is iterate
through the plausible steps which in
this case are 500 steps so we're going
to say 4 underscore in range of the goal
steps we're going to choose a random
action where there's the action equals
random dot ran range zero to again some
of you wants to change this I'm not
going to change it on the fly but but
this might be better because I'm not
familiar with this function and all the
environments but I'm curious if it would
find like if some of the environments
have like a lot of choices that you
might forget about or something and if
the agent could know but in reality the
agents not going to know that I don't
know I don't know if that's fair anyway
or just good ideas for now so they're
basically a random dot R and range 0 to
2 literally corresponds to being it will
either be a 0 or a 1 this is R and range
to 2 so this will only generate zeros
and ones so we have the action now we're
going to do is basically we're just
going to take this one here I'm just
going to copy and paste it observation
boom step take action awesome
now if Len a previous observation is
greater than 0 ie
we had a previous observation that we
could have made well then we're going to
gain memory pens whatever that previous
observation was please observation and
action so what are we doing here so this
is now normally see I'm not really sure
when the action happens
so but when I look at this code here
this suggests to me that the observation
is occurring after we took the action so
saving the action according to the
actual observation that's returned
doesn't make much sense to me
I think it's smarter to take that
previous observation in the current
action
I think that's smarter and then
basically when you're done here you say
now the new previous observation is
equal to observation I might be totally
wrong here feel free to just go off the
basic observation and not be one frame
behind if I'm incorrect either way we
should do fine so pretty bulb survey
shingles observation okay so now we're
going to say it is score plus equals
whatever that reward is which again will
either be 1 or 0 but 0 only occur if
we've lost if done we break now if all
this occurs because remember so so this
is for the entire game and then we're
storing that games you know happening to
the game memory now if that game is
indeed a winning game if we did good
basically what we want to do is we want
to save that information and we're going
to do that for however many games we're
willing to play which in this case is
going to be 10,000 again you could do a
much smaller number if you if you feel
like that's unfair now let's see what
was I like it's ok so once we once we're
done with the game now we want to
analyze that game and say okay do we do
better than whatever the score is we're
looking for so so this is the actual
game that happened then this is just
iterating through the game so we're
going to come down and just we're going
to be on still under this for loop here
now what we're going to ask is if the
score was greater than or equal to the
score underscore requirement yes score
requirement
if that was acceptable we're going to
accepted under score scores got a pen
score and then we're going to say for
data in game memory what do we want to
do well basically if data one so data in
game memory did we append it yes we did
that oh boy we're not going to get
through this anyway
this contains it's a list of lists of
the observation data which is like four
data points and then the action that we
took which is 0 or 1 but when we
actually go to Train we can't use 0
well we actually could use 0 or 1 as an
output since this is binary
but a lot of these games are not binary
you're going to have like 4 choices at
least like all the arrow keys up down
left right so we actually could get away
with with a single output but we're not
going to do that so we're going to
convert it to a one hot output so if
data 1 is equal to 1 we're going to say
output equals 0 1 LS data 1 equals 0
output equals 1 comma 0
great now over to say is training
underscore data up and set of 0 helpless
great ok we're done and then once that's
done again we're iterating through the
game still so whenever the game is over
we're going to go ahead and end that
reset and then we're going to go ask or
saw the pen for just to keep track of
all of the scores that we saw and then
when that's done we're outside of the
initial games right so we've run through
however many games we wanted to run
through now what do we want to do well
let's go ahead and training underscore
data underscore fav we're just going to
convert this also numpy array training
not us or converting try to train dad to
an umpire right then we're going to NP
dot save and then we're just going to
say save numpy what are we trying to
save the training data save it's kind of
a long name not Kentucky now let's go
ahead and print some things so let's say
average accepted score so these are the
filtered scores that we've already done
so we're going to do mean accepted
scores so what's the the average
acceptance for but then we're also kind
of curious what's like the median so
we're going to say we also do mode will
be median median accepted score median
of course and then while we're at it
we're going to go also just print
counter accepted excess okay this should
be highlighting a data for us to assess
what kind of data were training off of
now let's go ahead and return training
data ok let's save and run that now
we've got to actually run the function
dirt copy come down here paste okay now
let's save and run that well mmm if Len
three I think we call it free Bob
servation actually yeah pre servation
so pre Bob servation if Lane Creve
observation Queen okay please fix no
that's not try again
oh we type out as well right here for
three times each one oh my gosh wow
there's a lot of typos
let's probably quit apologize for all my
typos hopefully bit in yeah okay we
didn't render so as you can see we ran
through ten thousand environments really
fast and that's because we didn't render
them they'd actually you know it took us
a few seconds to run through five and
you can actually use another variable
like remote sequels like you know twelve
or something you could you could run
quite a few at a time but still is not
going to be as fast as running a
headless anyway in these ten thousand
example the average score was sixty one
the median was fifty seven and then here
are basically all of the scores this is
actually a really good sample this is
probably one of the best ones I've ever
seen in most cases maybe one or none of
the scores will be above like a hundred
like I want to run it again as long as
like unfair well actually we will run
this again so it's okay but anyway as
you can see none of these the best one
was a 118 there was only one million to
113 just one and so on and you start
really getting into the meat of the ones
once you get under about 100 anyway
actually wow this guy got a 153 dude
yeah this one is just a little unfair
and we have to run this one again prime
anyway okay so what we're gonna do now
in the next tutorial is create a neural
network model that's going to take this
training data and fit to the training
data and then we're going to use that
network to now to actually play the game
and we'll see how it does playing the
game anyway if you have any questions
comments concerns complaints about my
typos so if we tool even below otherwise
I'll see you in the next video