Using More Data - Deep Learning with Neural Networks and TensorFlow part

what is going on everybody and welcome
to part eight of our deep learning with
neural networks tensorflow and python
tutorial series in this tutorial what
we're going to be talking about is
simply adding more data to a model just
to see what kind of impact that's going
to have so to start we're going to be
using the exact same neural network that
we've been using this since the very
beginning a very basic feed-forward and
back propagation deep neural network and
in the previous tutorial what we did was
we use that on about 10,000 samples of
positive and negative labeled sentiment
data and our accuracy was about 60%
which is pretty atrocious so now the
question is what if we added more data
so if you remember to the very beginning
of the deep learning kind of tutorials I
mentioned that the two major changes for
neural networks that occurred that
brought neural networks back to the
forefront because normal networks are
not new most of the models and concepts
and core of neural networks that you're
going to use or not new but what changed
was the availability of huge data sets
and also the processing power of either
GPUs or maybe even now with tensor
processing units or whatever else people
happen to come up with even Intel's
coming up with CPUs that are more GPU
like so anyway it'll be interesting
future but regardless those are the two
major changes so what happens when we
take that exact same model that we have
been using giving us horrible accuracy
and just throw a bunch more data at it
like what's going to happen so we're
going to use the following data set it's
the sentiment 140 data set sentiment 140
there it is so you can just google that
or if you go to the text-based version
this tutorial I'll have a link to it but
you can come here and download that data
set if you come down here you'll see the
polarity is either 0 2 or 4 0 is
negative 2 is neutral 4 is positive now
if what we're going to ignore this
neutral because our previous training
set didn't have neutral sentiment
labeled information that probably that
can kind of that probably would increase
accuracy significantly to use it and
maybe we will use it in a different
tutorial but for now
we just I just simply want to show the
addition of more data and what kind of
an impact that might have so anyway
you'll grab that and I will just say
that all this code everything will be
available on Python programming net if
you want to grab it I am NOT going to go
through line by line on this because
we're nothing is changing as far as the
concept or the deep neural network so I
don't really see a reason for us to go
line by line again and just waste
everybody's time so we're not going to
do that instead I'm just going to run
through the changes what's different
between what we've written in the past
and currently and then if you want you
can go to Python program internet and
copy and paste that code or whatever if
you'd like so also this data set since
due to its size it's going to run really
slow on a CPU so we're about to make the
transition to using the GPU rather than
a CPU and you'll be able to always
follow along on your CPU if you want but
you're not going to get the kind of
accuracy that we're getting on a CPU or
and if you did you're going to have to
wait like months to train some of these
models so anyway so this is the
pre-processing data if you'll the first
thing I'll note is first of all I wrote
this function before I got to this
function here this convert to vector and
I thought I was going to do the same
thing as before where we converted to
the training set immediately and then we
ran it through the network so that's
that's a faster way to train that way
you just have to convert to vector one
time and that's it but on a much larger
data sets first problem that you're
going to have hit you're not going to
get away with that you're going to get
away with making one huge file I mean
you might because I mean this file
didn't have to be too big for a hard
drive but recall I think it was like was
a hundred times or 400 I can't remember
how many more times large I think we
went from like three megabytes to I
can't remember what it was but if you've
been watching and it's been some time
since I it's been some time since I've
filmed that so I can't remember what
those exact numbers were but anyway
there's a huge increase but anyway so
this one's not necessarily in
important I'm just showing you the code
that I wrote as I went through it anyway
but eventually we decide to kind of do
this in line and when we get to the
actual network part all I'll bring this
up again but for the lexicon it does
kind of make sense to make that
initially and the way that I went about
doing that is first of all you open the
file via buffering so you can kind of
run through and hear this is in bytes so
it's just a hundred thousand bytes and
then we're just going to iterate through
this and then every twenty five hundred
lines we're going to take all of the
vocab from those lines and then that's
how we're making our lexicon so in this
one just this file was labeled twenty
five hundred for that was one out of
every twenty five hundred and then
twenty-six thirty-eight is how many
words are in this lexicon so in this
case it's twenty-six thirty-eight again
this was to keep the model size
reasonable but generally language models
you're gonna see lexicons like
vocabulary sizes more like about a
hundred thousand words and yeah about a
hundred thousand words per language that
you might need to know so keep that in
mind sometimes you can go down you know
I've seen like if you look at tensorflow
I think they're a language translation
example is about forty thousand but it's
based on a model that was like 160
thousand actually was even more than
that was more than two hundred thousand
total on both sides but anyway I digress
so this is a very this is still a very
small model but you have to eat the
bigger this number is the much larger
your network is going to be that has to
be stored into memory so anyway keep
that in mind so then yeah this just gets
commented out really only need to create
a lexicon once but obviously if you have
don't have it you'll have to do that and
then we convert to vector but for the
training set this is absolutely I mean
you're going to make I think the file
size would be somewhere like 20 to 30
gigabytes oops sorry I don't have that
much space on this dual boot so that was
just too big you could save it to a
different drive though like because you
can access a different drive but again I
kind of wanted to give you guys after I
get to this point I was like oh well
this is actually a real problem because
a lot of times you're going to have a
data set that is just simply too big for
you to just store in your memory or
something as you read it in so you might
as well do this in line so we'll get to
that in a little bit shuffle the data
same reason is before you just have to
shuffle it so we're just shuffling that
training data using this right here and
then the test data pickle we just we're
just making a quick pickle so we can
very quickly test this framework so
that's that now we're going to go over
to the actual neural network code here
and try to think of anything major
changes that these are some notes like a
follow up if I only use 200 data it was
52% accurate if I use 2008 it was 62%
both of these were 10 epochs each 2000
data 63 percent which was 50 15 epochs
and then by the time we hit 200,000 data
it was about 74 percent accurate which
you'll see soon enough so batch size you
could make this bigger as well just
depends on what kind of memory you're
working with in this case we I shrunk it
down I guess to two layers I don't
remember at what point I did that but I
seem to recall using two or three layers
didn't really make a huge difference and
for the most part it never will so
depending on how complex your problem is
that's when you add layers but adding
layers is not going to necessarily
improve your accuracy unless you've got
a really complex problem or something
but for the most part two layers is good
enough to because basically one layer
would be for like linear data two layer
would be for nonlinear data three layers
would be for super nonlinear but if you
just keep adding layers you're going to
just over fit most likely your problem
but anyway continuing along simple
neural network again if you want you can
make this three layers and see what
happens if you really want to do it the
saver object so this is the the way that
you can save and restore tensorflow
models so to did it just know that
pretty Google anyway variables creations
initialization and all that so you can
come here and you can check out the
documentation here
I will just note that this acts
differently on the GPU version versus
the CPU version at least as I have found
so far so I'm just paying on which way
you use it just kind of tinker with it
but you can look at their examples but
basically before you call the saver
object here you have to have some tensor
flow variables defined even though in
your head you're thinking to yourself
well maybe you just want it maybe you
want it and you're just calling the
saver object so later you can restore a
variable but that's not good enough
because I guess maybe the variables are
saved but not like the model in the
structure I really don't know I'm not
sure why you necessarily have to do this
after the variables but that is the case
so just remember that you need to do
that and then also you would do this
outside of your session but then to
restore you do that within the session
and then when you do restore you restore
what the session and then from where
where was that checkpoint file and again
we'll run through this in my code as
well but I just want to bring that up
and then obviously when you save it
you're calling the saver object same as
before only this time again within the
session you're doing some stuff and then
when it's time to save you you don't
have to actually assign it to a variable
but you can just say saver dot save what
are we saving the session and then to
where whatever path you want so that's
the saver object also I used this TF log
purely to log the epics
I still haven't quite figured out like
you should be able to store a tensorflow
variable that tracks the epoch number
you are on not happening for me so
anyway I think it's probably because you
know if you try to define it up here
maybe it's replaced I don't know I
couldn't figure it out if you want to
figure it out and post below like how
you're doing it let me know because I
was like I could not figure out why I
couldn't make that happen I tried so
many things so that I was like well I'll
just make a log file that says okay
you're on epoch X you know whatever that
happens to be so yeah anyway
then when we get to the training
everything's pretty much the same don't
forget to initialize all your variables
before you you would do that before you
restore any sessions just for the record
but anyway here I'm just doing a quick
trine except with some lovely code
that's just asked like reading that TF
log file just to see what was the very
last epoch and then if we can't read
that for whatever reason we're just
saying yeah we're on epoch one so if
that file doesn't exist or whatever then
coming down here we begin iterating
through all of our epics so in theory
like sometimes people just run like
infinite loops when they're training and
they don't really care like how many
total epics like the network will just
run forever and then sometimes you
actually want to like limit however many
epics you want to do so I mean in theory
you might not even need the TF log
especially on like much larger data sets
but whatever anyway if the epic is not
equal to one so if we're not starting
over right then we want to load the
model whatever that model is so in this
case where if the epic is not one we're
going to load model dot seek apt and
then we're going to say I'm sorry I
thought that said epic he was one of
suck wait a sec
anyway epic loss one whatever and then
we continue along this just opens up the
lexicon for us so after the lexicon we
begin going through the train set
shuffled via buffering again this file
was not huge necessarily but you could
be working with say a data set that's
like 500 gigabytes or something so you
can buffer through it tensorflow
did have like a feed like I don't think
it's called buffering if I can pull it
up real quick I wasn't really thinking
about it until just now
offering because I do have something for
it but I could not seem to get it to
work so feeding reading from files maybe
it was batching and maybe feeding I
think it was probably actually feeding
now or what was this pipeline I think
I've tried both of these but I just
could not get it to actually do anything
for me so I didn't find it to work out
but probably a better option would be
would be to use this rather than my
method if you can figure it out I just
couldn't figure it out but anyway
reading in the file and then we're just
iterating through the lines as we're
reading them in create working with the
the label is just that and then the
tweet this which is still string and
then we convert it to words we lemma
ties it we do the feature vector the
same thing as we had done before nothing
is different really this is just simply
next this is just simply a Y so actually
we're not even really using batches at
the moment but anyway moving along Oh
savor dot save oh now we save it every
time we're kind of like working through
it and same again we're just receiving
it to model dot checkpoint print out the
epoch whenever we're done so it just
saves per epoch and then also I'm
writing to that epoch log file again I'm
positive there's a way you could do it
with the tensorflow variable and save it
all to your session I just could not
figure that one out it's too too
complicated from our little brain
correct accuracy nothing really changes
there and then processing or doing the
test set again this one just same thing
as before nothing really changes here
either so that pretty much stays the
same and then later on Russian not even
going to use this because this is like
this happens at the end of a very long
process so I ended up actually just
making a new function that is just
simply test the neural network and then
we actually test the neural network and
then finally before we actually test the
neural network I just want to pop on
over to using the neural network
and then that I just added some
functions here at the bottom I think
it's just yet this function here so use
the neural network so obviously when you
come up with a model that you're happy
with you might actually want to like
rather than just training and testing it
all the time use the model right so in
this case um this is a function that
works basically you just say use a
neural network you pass a string like
this one I've got he's an idiot and a
jerk and this was the best store I have
ever seen or I've ever seen and then the
output it's either going to print
positive or negative and all that we're
doing here is we take that input data we
know we need to vectorize it right so
we're loading in the the pickle we load
in the model itself and have it to this
prediction value we run the session
we're restoring via a checkpoint file I
change the name by the way this is just
a checkpoint file that's been chained
trained completely the 74 just means
it's it gives 74 percent accuracy and
then we go let's see where am i yeah so
we load that in start off as epic loss
is equal to zero
I'm not apply is that even there what's
looking at this now and I don't know why
we did that I guess I just shaved it out
and then I didn't think about doing this
there we go or it was a very light night
and I was coding and yeah so anyway so
that's all you need I don't know what we
were doing do with the epics and epic
loss doesn't need to be there either
saver dot restore so we restore that we
take the current words we tokenize them
limit eyes them do the feature vector
for the word the feature vector so we're
just making this feature vector and then
our result is equal this is kind of an
important line here too this doesn't
necessarily need parentheses sets dot
run and then we're running we're asking
what is the TF charge max of prediction
dot eval feed dict and what's the value
that we're passing in here we're just
passing a simple value for X and its
features done and the other thing I
would
I'm gonna have us do is this right I
said this copy that and I do want to
just hate print rather right I do want
to show you all what what the actual
output from this is because I don't
don't think I've actually ever shown
that yet in in the videos so anyway I do
want to show you what that actually
looks like because it's not one zero or
zero one it's like a kajillion and a
lesser kajillion or some of it anyway so
cool so we'll save that and I'm going to
come over here and save this one too
quite sure what we changed there but
anyway so this one I'm actually not
going to run because the actual training
takes forever and I'm actually still on
my VPS but probably in the coming maybe
not the next tutorial but from the here
on out I will be almost certainly
recording on my GPU version so this this
takes a long time to run so yeah anyway
I'm going to open up terminal change
directory into
see desktop and I think this is like
larger data to touch yes yes this where
I want to be so anyway these files by
the way both of these the lexicon and
the model will be up on Python
programming that net so if you want to
just grab those two files so you can
like see it for yourself they are there
go to the text-based version of this
tutorial you can download them as far as
I recall I don't I think I actually
saved them as straight just lexicon dot
pickle looks about see kid trying to
find them just to see if I put them up
there and under what yeah the model will
be modeled at CK PT in lexicon is just
lexicon dot pickle so they don't have
these long notes on them feel free to
add them if you want but just keep in
mind if you use this code the text-based
version of this tutorial has them
correct so if you are copying pasting
from Python program net-net you'll
you'll have the correct the correct
names anyway but just take to keep that
in consideration that that's what they
are anyway so first we'll run a Python 3
TF 8 saving and restoring and what we're
going to do is not train the neural
network that's commented out we are just
simply going to test the neural network
with that processed test set so run that
test about 300 tests exactly 3 hour 59
samples accuracy is seventy four point
six five one eight percent so again all
that really changed one thing I did kind
of note that did also change is we're
kind of training one sample at a time
here rather than training in batches so
that will make a slight difference but
that I I could be completely wrong but
that should not be making this large of
a difference in the accuracy the main
difference here is simply we have 1.6
million samples of data as compared to
10,000 samples of data so next what
we're going to do is we're going to come
over to TF 9 and TF 9 is just use the
neural network and this is going to let
us actually like see how the neural
network actually performs against actual
string data so I'm just using these two
strings here he's an idiot and a jerk
this was the best store I've ever seen
so now we can run that and we can say
Python 3 TF 9 using the network and
there you can see the two outputs here
so the first one is negative and then
the next one is positive and you can see
how close like these actually both of
these were actually very close to each
other it could have easily gone either
way
but you this one is smaller than this
one and that's kind of the point I was
trying to make before was that it's not
going to be a 0 or 1 it's going to be
which neuron is firing the strongest and
in this case it was the negative neuron
right because this one is the positive
neuron this is the negative neuron
they're very close to each other but
this one is firing strongest most likely
because each of these words is
relatively you know you're going to see
this was the store I've
Sene ever might have some weight but it
could go either way too so most of these
words are relatively useless there's
really one word here that means anything
in terms of sentiment and then over here
there's two words but we're still have
all quite a few words that aren't I
don't know how idiot or jerk couldn't
have swayed this a little more but okay
because what if we say he's an idiot
so just he's an idiot is positive right
so yeah anyway there's your 74% accuracy
so yeah you can kind of play around with
that if you want but again savings 74%
accuracy is really nothing that's
special it turns out we need even more
data and a better model so this
traditional just like a straight-up
feed-forward back propagation deep
neural network is not meant for language
data it just doesn't perform well in
language data so there are lots of other
models and stuff that we're going to be
able to possibly look into in the future
that perform much better on language
data if you want to look into it
generally a recurrent neural network and
form of LS TM or something like that is
going to perform better on language data
so yeah anyway that's all for this
tutorial I just wanted to kind of run
through the example of a much larger
data set but also kind of the little
nuances like saving as you go that way
like if your model takes a very long
time to train you don't lose everything
because the power drops or something
also so you can actually use it like as
this thing trains if you wanted to you
know the training is going to take
months but maybe you could just use the
use the model along the way or test it
just to see how it's doing along the way
so so there's that and so obviously
that's useful buffering in data just
working with large data sets is in
general pretty challenging just because
you have to figure out a way to get that
data into your actual model the other
thing that's really challenging
especially when you're working trying to
work on your GPU is input/output and
that's why I think you'd probably rather
be
working with tensors tensor flows
input-output specifically because it
might be and it most likely is more
efficient than straight Python code so
anyway I'll try to figure that one out a
little more if anybody else figures that
one out feel free to post an example of
how to feed in this data using the
tensor flow method of reading and data
because I just I couldn't figure it out
but anyways that's it if you have
questions comments concerns whatever -
feel free to leave them below otherwise
as always thanks for watching thanks for
all the support subscriptions and until
next time