Intro - Training a neural network to play a game with TensorFlow and Open

what's going on everybody welcome to
another deep learning with Python and
intention for the tutorial in this
tutorial what I'd like to do is kind of
show another interesting example and
kind of a cool characteristic with
specifically neural networks and I guess
it's also it's just it's a statistics
kind of principle that's been known for
a while but anyway I just wanted to show
that it still applies in quiz neural
nets so what we're going to be doing is
what first of all like neural networks
we know neural networks typically I mean
people are working really hard on
getting neural nets and just doing
reinforcement learning and artificial
intelligence much quicker with much less
data but in general you know networks
like a lot of data so a lot of tasks
this can be hard but if we can simulate
or model that task in any way then we
can generate huge datasets to train all
those so things like math and physics we
can just write a script that just
outputs a bunch of examples really
really fast and another kind of modeling
an environment scenario is with
something like opening eyes Jim so
that's what we're going to be using here
is opening eye Jim so if you're not
familiar with opening eye Jim this is
what's one monitor there we go this is
what you go if you go to just open Jim
open a calm and then I'm in the
environments here and these are just a
bunch of environments like these are
just the ones that come with straight
Jim but you can download like for the
Atari version you can play thar game as
if you're on Windows it's not going to
work very well actually it's not going
to work at all if you if you happen to
be on Linux or something you can go in
and play with these games we're going to
actually work with just the court pull
environment so the carpool environment
really simple the idea is that you'll
start the poll you'll have the control
over the cart and your only option is
we've left or right if you move
basically too far to the left or too far
to the right it's game over and then if
the poll itself I think it's what 15 per
so if it moves more than 15 degrees from
vertical you also lose so basically the
task is we're just trying to balance
this poll by moving the cart left or
right just a little bit if if you get a
score above 200 that's considered I'm
sorry this is episode
anyways obviously you want that to be as
low as possible like 239 is pretty hard
well I'll show you that even with this
method we're probably we're not going to
be 239 for being that quick but anyway
we can beat 51,000 now we definitely
can't be 0.0 anyway ah the goal is to
get a score of 200 or greater and that's
considered to be salt so so now we're
actually this is an average of 195 over
100 consecutive trials whatever we will
do that so anyways that's what we're
going to do what you're going to need is
quite few things hopefully if you've
been following along in the deep
learning series you you probably have
quite a few of these but if not
basically what you're going to need is
to install tensorflow
I'm going to be on the GPU version of
center flow that would be - GPU plus
you're gonna have to install a bunch of
dependencies so if you want to use the
GPU version I have tutorials on that you
can come here to Python program at that
just type in 'probably CUDA and you'll
find it from guess 100 times type it
real quick yeah couldn't GPU tensorflow
open so you can search for it up here if
you didn't see that also if you're on
Windows you I have one on YouTube I
don't I never posted that on Python
program Internet but it is this on
youtube if you're on Windows and want to
do the GPU version of tensorflow I've
got an example there also for tutorials
I've got introduction to neural networks
I've got an introduction to actually
deep learning specifically with tensor
flow and then also an intro to TF learn
which is what we're going to be using
here now if you're familiar with a
different high level you know framework
use whatever the heck you want it really
doesn't matter so whatever you want to
use go for it we're just going to like
we're just going to copy and paste from
this tutorial and just as a quick aside
someone we made a comment on one it was
just one person but if you feel like
it's the case that I'm just like copying
and pasting from something and you're
not learning well come to the tutorial
there's already a tutorial on that I'm
not going to repeat myself I'm had exact
same tutorial so anyways we are going to
just take the model from this one that's
why I'm saying like you can take because
we're just going to use a simple
multi-layer perceptrons feed-forward
model
so if you want to do that Terrace or
Queiroz or however you pronounce that
pretty tense or TF slim whatever you
want to use go for it or you can even
use you don't have to use tensor flow at
all so okay let's get started you're
going to need tensor flow GPU you're
going to need just Jim so pip installed
Jim and then we're also like I said I'm
going to need in Kiev learn you can use
whatever the heck you want it's just the
model part not a big deal great so once
you have all that stuff you're ready to
rumble and if I forget to put a link of
the description to the actual this
tutorial which will have links tell the
things I just pointed out so long minute
so so what are we going to show you
today is that with an interesting thing
about what we can do in just statistics
in general is a lot of times you'll find
that something with that generates a
signal so in this case a neural Nets
generating a signal that either we want
to move left or right when when you have
a cluster of really weak signals putting
them together can oftentimes produce a
you know a signal that's stronger than
the sum of all of the parts basically so
it's more accurate than any of the other
signals even combined a lot of times so
anyways that's what we're going to be
kind of illustrating with neural nets so
first we're going to make our import
we're going to import Jim which is the
open a I Jim we're going to import
random because initially we're just
going to let this basically the idea is
you call it like you've got the game
which is the environment and you've got
an agent the agent will initially just
move randomly just so we can get some
starting data and we're going to import
numpy as NP so we can do some numpy
stuff so i guess in theory if you don't
have numpy tickets gone on by imports TF
learn and then we're also going to
import some other stuff I guess we could
type it out as one from TF learned
layers core we're going to import input
underscore data dropout and then fully
connect it so this is just the input
layer dropout information so we're just
going to drop out like 20% basically and
then fully connected just for your
typical fully connected layer
goes to you know convolutional air l SEM
or whatever then we're going to stay
from he has learned a layers estimator
we're going to import regression that's
just for our final layer there and then
from of the test X we're going to import
mean and medium we're going to use both
of these just to illustrate how well
random did we're going to basically
train off of random and we're going to
see what the scores we're learning from
R and then we're going to see what our
score is and you prepare to be amazed
now from collection we're going to
import some counter sorry for my
allergies I've got like scratch it
through anyway learning rate L R equals
one a negative three feel free to tinker
with it this is such a simple task you
really could get away with quite a few
variables still probably have similar
performance well as you make the game
more complex there's more movements and
to flick that you might you might
actually need to start tweaking things
now we're going to find the environment
the environment is going to be Jim dot
make and for us we're going to use not
cartful cat pull a more fun game cat
pull anyway these zero now we're going
to do is m dot reset that just gets the
environment kind of rolling now what
we're going to say is we're just going
to take basically the the environment
we're going to say in theory goal steps
could be two hundred because that's like
basically every frame that we go where
we balanced the pull a price should have
stressed this before but here I am so
again we're trying to balance the pull
on the cart and the way that we get a
score is every basically frame that we
go while the poll is still balanced that
is plus one to the score I'm pretty sure
his frame it could be milliseconds but I
don't I'm pretty sure it's frame anyway
yeah it's definitely frame can be in
time because you can speed it up as
you're going to see so anyway the Gulf
steps will be basically if there is 200
would be enough but let's go ahead make
it 500 just just to do it then we're
going to say score requirement so like I
was saying before we're going to try to
learn from the the Rand
Moo's but obviously we're not just any
of the randoms we're going to go with
all the random games that have a score
of 50 or greater again we could we could
had tweaked this as we go but for now
we're just going to kind of throw some
numbers out pretty quickly and then
we're going to say initial games and for
now let's say 10,000 if I forget to
modify this go ahead and modify it
because in theory if you make this
number too big you might run into like
almost like a brute force every answer
or something but I still I'll try to
explain that hopefully I'll remember all
these things once we get finished but I
just want to address that what query
later so now we're going to do is we're
going to just start off by making random
games so we're going to define some
random games first and then what we're
going to do is for an episode episode
for episode and range let's just do like
5 because basically what I want to do is
just kind of illustrate what the random
games actually look like so for episode
in range of 5 I'm going to do we're
going to reset the environment and three
we actually don't I want to keep that
there but look why don't you again
but anyway 14 what we do in there we
probably get rid of this reset I'm
really anyway 40 in range and for now
we'll do 200 but in theory we could say
goals let's just we'll be good we'll do
those steps 14 a range goal steps tends
not render so if you want to see what's
happening in your game you can render if
you want it to go much faster don't
render but we're going to render because
I just want to show you what it looks
like to move randomly then we're going
to say action is equal to m dot action
underscore spaced samples so what this
does it's a nifty little function that
will take your environment and just take
a random action in your environment just
for you basically this is what we're
going to do later but but it's kind of
nice because you go to any environment
you can generate random actions this way
and then and everything you pipe I'm not
going to change the code that I've kind
of planned out but I you probably could
just go off this and be easier to switch
games later on
food for thought anyway observation
reward done in flow observation will be
basically an array of just data from the
game a lot of times a lot of games it'll
be the actual pixel data so just all the
pixel data in this case it's actually
going to be like pole position court
position and something else I think it's
for values reward basically either
you've got a 1 or 0 the thing was
balanced or not done is the game over
info any other info and then we're going
to say that is equal to M step and a
step takes an action in and then finally
we're just gonna say if done break okay
so let's run some some random games
first and let's just see and make sure
everything's working as we would expect
there we go so as you can see that just
bang through five games and you can see
we've pretty much lost really quickly
so anyway hopefully if yours didn't pop
up or you got an error something like
that leave it below otherwise in the
next tutorial what we're going to do is
actually start building through creating
the sample data based on random moves
and then after that we'll make the
network and run in training all that fun
stuff so if you have questions comments
concerns whatever feel free to leave
them below otherwise I'll see you in the
next video