Processing our own Data - Deep Learning with Neural Networks and TensorFlow part 5 Video Lecture | Machine Learning with Python - AI & ML

what is going on everybody and welcome
to part five of our deep learning with
neural networks tensorflow and of course
python tutorial series in this tutorial
we're going to be talking about is
actually taking what we've learned which
is a really simple example of a deep
neural network on some kind of
prepackaged data for us and attempt to
apply that exact same deep neural
network on to data that is not
prepackaged for us so probably the first
hurdle that you're going to come across
after you get through going through like
say a demo or something like that on
tensorflow org or on theano or whatever
you have that first hurdle is just
simply going to be okay now that I know
how this model works I want to apply it
to some data shoot how do I do that like
so so that's we're going to be kind of
doing here is actually applying this to
a realistic data set or at least
applying this in a realistic way to a
data set because the eminence data set
is a realistic data set anyway so the
data that we're going to be working with
I mean you can work with another data
set if you want to go for it but we're
going to be working with positive and
negative sentiment data sets and just
see what kind of if we can get a
positive or negative sentiment kind of
classifier going on here so to begin
there will be the in the text-based
version of this tutorial so if you don't
know I all code and everything it goes a
text-based version of this tutorial so
you can always go to data analysis here
then machine learning and then this is
the machine learning course and if you
scrolled all the way to the bottom
this tutorial will be somewhere down
here I'm filming it
therefore this tutorial is not yet
released so I can't click on it but
anyway if you go there you'll click on
click there and there should be two
buttons there one for the positive file
the negative file but here they are as
well so it's just a two files two text
files that are contain some strings that
are both positive and negative strings
of text so I'm going to go ahead and
just right click save page as and I'm
just saving them to H desktop TF tuts
so that'll be pause text and then this
one I'm going to save as Nursing save as
there we go
neg dot txt good and then we can just
pull one up real quick just to show you
they're about a little over 5,000 lines
total sauce goals a very very bottom
here what there we go
it's about five thousand three hundred
lines long same thing with the negative
file you can take my word for it or look
at it if you want so our plan is to take
the this is our data set right in and
there's really a few things that are
should be kind of curious to you as far
as how we're going to progress and that
is one this is word data these are
strings these aren't vectors so you
can't immediately make a string a tensor
so the first thing the first problem we
have are these are this is a text and we
need to convert it into some sort of
numerical form also these strings are
not of equal length
okay so converting words to numerical
form you would maybe do a back of words
model or something like this and where
you give each word an ID so the first
word we come across is simplistic that's
ID one silly ID two and three
TD is four and so on and just give each
word a unique ID the problem is this a
string here has basically four words but
this string has a you know I don't know
ten or something okay so those are
different length vectors and that's not
acceptable we have to send in the exact
same length of vector every time through
our neural network or at least this
neural network there are neural networks
that we might talk about down the line
where you can actually do kind of
sequence to sequence stuff where there
don't actually have to be the same
length or and use bucketing and all this
kind of stuff but anyways for now it's
kind of a rule that the vectors input
all has to be exactly the same uniform
size and shape so that's our problem
and so that what we're going to do is
we're going to come up with a lexicon of
words and a lexicon is just a dictionary
or your vocabulary of words that you
know and so we will use this bag of
words model or each word basically has
an ID okay in this case it'll be index
and then we're going to kind of
incorporate this not necessarily one hot
but you could call it a hot array or
something so let me go ahead and create
a new document
and we're going to call this create
sentiment feature sets dot pie and I'll
go ahead and open that with spline and
just kind of explain how this is going
to work so first to create your lexicon
you're going to use all the data sets
that we have so like let's say our
entire data set has the word has default
like we only have chair table spoon and
television that's our lexicon array so
after we go through all the documents
and calculate let's say all the unique
words these are all of the unique words
that we found in all of our documents
and that's what we're going to use as
our global lexicon and then we have a
new sentence that comes through and that
sentences I pulled the chair up to the
table okay and that is going to now have
to be converted right and so the first
step that we're going to have is we're
going to say okay we have this lexicon
we'll be uh MGP rows of the Len of our
lexicon whatever that is which is up
here so it'd be four right yeah four and
then we're going to turn on each of
these elements according to which index
it is in this lexicon if it exists so
that's probably really confusing but
initially this will just be a zero zero
zero zero right and then we go through
and we say okay does chair or like I
guess we could say I does I exist in
this Lex Continental polled no the no
chair yes it does so we have chair and
the index of chair in our lexicon is
what zero right so the 0th index down
here is now one up nope we don't have
that to know the no table yeah we do
have that and that index is a 1 so the
first tooth index down here one so now
converted to a vector in this sort of
bag of words model that we're using this
is now our vector for this specific
sentence and then we're going to do that
against all of our sentences and our
lexicon will be created off of our
entire data set not just for random
words I pulled out of my you-know-what
so anyways we will close this out and
we'll just go ahead and save this and
what we're going to do is kind of begin
working through this and to start we are
going to be using NLT K if you do not
have NLT K go ahead ctrl alt T for
Terminal pip3
install and NLT K move my Mike's getting
in my way okay so once you've installed
on TK what you're going to do is go
Python 3 import NLT K and then go in l
TK download and then choose D and then
all and you're just going to want to
download everything you could
specifically download the exact things
that we use but I kind of think it's
kind of silly this might take a while I
might take up to like 20 minutes or
something but anyway well you can pause
it and then continue on once you have
that so I'm going to ctrl C quit cool so
once you have n on CK you're going to be
ready to begin working on this Python
file so I'm going to come back over here
now and I'm gonna go import NLT Kay
we're gonna from NLT K dot tokenize
we're gonna import import import oh my
gosh ok and say it legitimately word
tokenize and just in case you're curious
if you want to know more about NLT K and
working with language and national image
processing you can go to Pilar Burnett
data analysis then down here is natural
language processing so in a moment here
we're talking right now about tokenizing
which is a really simple concept and
then later on we're going to be very
quickly talking about limit izing which
is a little more complex but a very
simple concept still but if my
explanations aren't doing it for you you
can always come back here and you might
want to start with stemming and then go
to limit izing but anyway regardless
just letting you know if you need more
information it's available
so we're tokenize all word tokenize does
is takes a sentence that you know I
pulled the chair up to the table and
just instead it separates all the words
for us so I and in polled and the and
chair and up and so on it just tokenize
--is the words by like it just says okay
this is an element this is an element
this and so on okay so that's word token
then we're going to do is from NLT KDOT
stem we're going to import word net
lemma tiser so it's it's it's a very
limit izing is a very similar operation
to stemming and it's kind of the same
concept it's the ideas that like for
consider for like sentiment reasons or
just consider like word meaning reasons
running ran run these all basically are
the exact same word so you might as well
just take all of those and call it run
okay so what stemming would do is remove
ing Zandi y's and why isn't all this
kind of stuff and edie's and all that
but when you stem something it doesn't
necessarily create like a legitimate
word that you could look up on word net
for example but limit izing will it
makes it an actual word that has a true
meaning that you could look up in a
dictionary so anyways we're going to be
using that obviously again for sentiment
analysis like the question is you might
be like well doesn't tents matter you
know because you can come up with some
things like I like this product or
product versus I used to like this
product okay the one thing is positive
one thing is negative right and so or I
liked this product right so tense does
matter
it's just that generally tense matters
in a different sense like so when you're
just trying to figure out meaning and
something like that you would go through
and limit eyes everything but then when
you're trying to like actually figure
out like what what's being said here for
real you might actually care what the
original word was we're not going to
worry about going that far in depth here
but anyway so we'll move off limit izing
for now now we're going to do is import
numpy as NP NP you jerk we're going to
import random because we're going to
need to shuffle this data at some point
we're going to import pickles so we can
save it at some point from collections
we're going to import counter because we
want to count some stuff at some point
and that should be it
so now we're going to specify a limit
Iser as being a word net limit Iser and
then we're going to say how many lines
and for now I'm just going to say this
but
yeah each document here has about 5,000
lines and if you're like sometimes on
these kind of processing heavy tutorials
people will post like I'm getting this
error it's this memory error if you get
a memory error that looks like that you
ran out of RAM and maybe later on you
ran out of VRAM but for now as long as
we're using CPU tensorflow
uh this means you ran out of RAM so our
model and actually really the training
data but our model will also be put into
RAM here you know like let's say you've
got you know you've got three layers
each layer is a thousand cells that's
three thousand total cells that might
have a vector in them or at least
something is in them plus your entire
data set if that's in your memory and so
on so the larger your data set the
larger or the more ram you're going to
take up so if you want it one thing you
could do is first of all you can use
less layers in the neural network less
nodes and layer okay and then also just
less data and you won't hit this memory
error now accuracy will go down and as
I'm going to hopefully show you and
illustrate to you throughout these
coming videos that the main value in a
neural network or the main the main way
to extract value from a neural network
is to feed it just insane amounts of
data so this last thing you want to do
but if obviously you know it's it's not
reasonable to be running large neural
networks or really running neural
networks on like a CPU or something like
that it's just you can learn but it's
not you're not going to get rid for so
anyways that's that and then now we're
going to actually start doing is
building our functions that are going to
kind of go through this data but I think
I'll go ahead and save that for the next
tutorials simply because we covered a
lot of kind of theory and stuff in this
one but if you have any questions
comments concerns or whatever feel free
to post them below otherwise stay tuned
the next video thanks watching