Preprocessing cont'd - Deep Learning with Neural Networks and TensorFlow

What is going on everybody welcome to part 6 of our deep learning with Neural networks [since] [flo] and python tutorial series in
Last video where we left off we kind of got familiar with our data set familiar with our
Means how we're going to go about actually
passing this and creating a [pre-processing] this [data] to actually fit through a neural network and
Now we're actually gonna write the code [that] does that pre-processing step for us so so far
we've done our imports and those were explained and we've
Defined the limit iser we've explained how many lines and all that and now we're ready [to] begin
[so] the first step that we have to do is we need to
Create a lexicon
So the way that we create this lexicon again is at least at its very most basic level
We have we just it's all of the words that we find in our positive and negative
data sets
so
To start we might say the lexicon equals, and we'll say [just] an empty set here, and then what we're going to say is
with actually let's say for
for file in
pause neg that way don't have to write the same code twice in a row and
Really this [should] have been this price should just be a list
But I'm not whatever hey [poor] [file]. This is not important for that, so then we're just I with
[and] then with open
whatever that file is with the intention to read as
F
We're going to say the contents contents of this are is f dot read lines
and then we're going to say for [L] for like for line in
Contents whoops contents up to however many lines we were going to read
of to honing lines
We're gonna say all words equals word underscore tokenize
L and then we're gonna say Lexicon plus equals list of all
Looks like we type Ode sure did nice
Autocomplete for the save their words there we go
[alright]
so um so let's just going to read our data in and stuff it into a lexicon but of course this lexicon currently contains like
Copies of words and all that and then the other [question] we have is do we actually care about
Every single word, or do we care like about like specific words
Or is there like some sort of range or something and I'm going to kind [of] put forth that there is a range that we
We actually care about so outside of this for loop once we've populated Lexicon with every word that we've come [across]
The first thing we want to do is we're going to limit eyes all of these words [um]
In fact we might actually might be better off to like first do I'm gonna go ahead and just delimit eyes them first
But um you'd probably want to like you can chop away words first
Maybe I don't know yeah because the logic that we're going to use to get rid of words you'd really want [them] limit eyes first
Anyway, we're going to say lexicon equals, and then we're going to do this one line or for loop limit iser limit eyes
[limit] iser to limit eyes eye
For eye in the Lexicon okay, so this is just our way of kind of stemming them into legitimate words
and then we're going to do word counts w underscore or w w underscore counts equals counter and
then we're going [to] apply that counter to our lexicon and
This just gives us a dictionary like element so word counts might look like this like
you know the and these [gonna] occur this many times and then um you [know] and
probably occurs
Let's save [twenty] five thousand. Okay. Something like that. Okay. It's a long dictionary like uh
Object thick it looks like this, okay?
And then what we're going to say is we're going to say and in fact let me comment out
That redefines everything okay, so for w in word?
Counts, what do we want to do? Well? We're going to say if and then we're gonna throw in some
Hard-Coded parameters here. Sorry if 1000 is
Greater than word counts um
W so what we'd like word counts on a word [B]?
so [B]
word counts thee the value would be the actual count like how many occurrences did we have
so word counts w it needs to be less than a thousand and then we're going to say greater than
fifty okay
[you] can kind of tinker with these values
but I'll at least explain why I'm choosing them here in a moment, so
First of all we need l. 2 l 2 l 2 is empty for now and
then L 2 dot append [W] and
Then before I forget this function returns
Return L. 2 okay, so this creates a lexicon [L] 2 is the final Lexicon
[but] what's happening? Here is basically. We're saying like we don't want super common words because super common words are like the and of
Or these kinds of words um look okay, so so we don't get the words aren't going to be valuable
So those are just going to like blow up our model
More than we want right so so the this lexicon is literally going to be that input vector
that input vector is going to be the lexicon in length, [so]
ideally we want the lexicon to be as short as possible so that we can have a
Decent sized model like in terms of like three layers and like I don't know a thousand or something
Nodes or cells in that layer, so we want to be able to make lexicon as efficient as possible
So we're going to say like we don't want super common words because we don't think they're a value
But you might you might say I don't know like because the thousands like [10%] of the data set
So maybe in [5000] or something like that
But we'll go we'll stick with a thousand and then we're going to say like the really rare words we also don't really care about
50 might be again too big of a number for your tastes
[might] be wiser to use a smaller number
But we're going to stick with 50 and we're returning the Lexicon
and
We'll see how that goes, but again. You can kind of tinker with this
I think as you will see soon enough this does not matter much. What's going to matter is something else
so
Once we've got the Lexicon. We're ready to use this lexicon and maybe a you know classify feature sets. So how might we go about?
Actually classifying feature sets first we're going to have some sort of function. It's going to do that for us
so we're going to say define sample underscore handling sample handling is going to take a sample of some kind and
it's going to take the Lexicon and it's going to take a
Classification, so what is this classification supposed to be and then we're going to say feature?
Set equals empty, and then we're going to say with open
whatever the sample is with the intention to read as
[F]
Contents is going to equal f. Dot read lines kind of like we had before and then we're say for L in contents
up to how many lines
So like that, so for each of the lines in our lines
we're going to say the current words and this will be word tokenize and
We're going to tokenize [L]
dot lower
[I] think we'll do a dot low right here [-]
I'm pretty sure it won't matter but I will we definitely want the we want the student be same that's for sure so anyway
We'll keep that there. So those are the current words and we're going to say the current words is now the
limit [iser]
oops, limit eyes our limit eyes [eye]
for [eye] in current words
And then we're gonna say features equals that Mp zeros
and that's going to be the land of whatever the Lexicon is which we passed as a parameter and
Then what we're going [to] do is we're just going to iterate through our words and then set the index in this features array
To one or we're going to use plus equals one because I [think] that makes the most sense
But it really probably does not matter at all, but anyway forward in current
words if that word dot lower is in our
Lexicon well, what do we want to do well?
We're going to say we need [to] find out what the index value is so index value equals
Lexicon no jerk Lex there we go Lexicon dot index and then
word dot lower
So we're going to search in our lexicon for that word dot lower and this will return basically our index value there
And then we're going to take that index value and say okay great, so the features
whoops features
or the feature at Index value
That is going to be either equal to one or you could say plus equals one
So depending on like if you're reading an entire like paragraph or you're using this on like a huge paragraph or something plus equals one
might actually make more sense
[but] since we're using actually pretty small strings this
Probably is going to have absolutely no impact to whether you use plus 1 or plus equals [1] especially on this data
Set size. I don't think it'll matter but anyway you could if you you know use whatever philosophy you think is best
Then we're going to say features equals the list of features and then we're going to say feature set feature
feature-set
dot append
Now we're going to append a list of lists feature set features and ten the classification whatever that classification is
So at the end of the day a feature set up here is going to be this huge thing. That's a list of
Lists where within these lists you're going to have this thing
[that] looks that maybe like this right this is that that a
Hot array, so to speak [that] [has] like each index of the like at the bag of words Model
Basically are some more features, and then you have the label which
soon you will see is like either a 1 in a 0 or a 0 and a 1 right, so the
zeroth element here is the positivity and the first element [here] is the negativity so 1 0 is a
Positive sentiment sample, and then a 0 1 would be a negative sentiment sample
So that's our feature set that's why that's going to look like and then when everything is all said and done
we want to
return to
rent
or turn
That feature set [okay]
so I think we'll cut it here and then in the next tutorial we'll we'll go through actually the
Actually, I think that we only really have like one more function
So I think I'm going to go ahead and have us write this next function. I I don't think you guys will mind, so
Unless we have like a thousand errors
[ok] so what we're going to say next is Define create feature
Sets and labels so so far we've kind of got a high level
Some high level functions that are going to do these things for us
But now we [actually] need a function that's going to pretty much actually like do it and put it all together for us, so
so create
[features] [is] great
so now what we're going to do is we're going to pass a
Positive negative and test size and this will be zero point one, so we're gonna say a 10% test size
Lexicon is going to be able to create Lexicon and we pass pause neg and
then our features will be an empty list and I'm going to say features plus equals the return of sample handling and
then we want to pass in the pause text and
the Lexicon
Lexicon and then our
Classification for positives is 1 0 [I] forget if I see I did say pause in a text
[or] actually save this [ok] and then we're going to the same thing for negatives, so I'm just going to copy this line
paste
neg dot text 0
[&] 1 and
Then what we're going to do is random dot shuffle features features
so
Um we need to shuffle these for sure not just for like testing and stuff but and not just for like statistical reasons
But actually for the neural network because remember a neural network basically you're like. I say everything was positive
I don't really think we've shown it
But hopefully by the end of this tutorial or this like mini-series on implementing this data and testing it
You'll see what a neural network is actually doing and you'll probably will recall before
the question was from the output of the neural network the question was
Does tf dot arg Max of let's say?
the output does that equal the tf dot R max of
our
Expectations right that was the final question of you know did we actually get in get the prediction, right?
Now what does this actually look like?
[so] if we actually
Looked at this like the values of output and expectations
Like let's say let's consider this case whether you just have like these two output nodes
So we're going to ask does the output equal expectations
so let's say we're expecting a [positive] it would look you know tf Organics Here would be 1 0
But the output here is actually probably more like this
Or something. It's probably not this big. It's more like I don't know that and that
okay, and
This is true. Yes, okay? Uh
But this is obviously not a one in this office in a zero
But in those kind of one hot output structure you really just you just want to know what's the maximum argument
That's kind of how strongly we feel that this is positive, right?
[so] close how it works, so if you didn't shuffle, basically you would train the network to
pretty much try its hardest
Like if you were feeding positives, and you said you wanted this you basically train the network to do like this
Right and then like I don't even know like negative 9 mm right because the network [could] be like okay?
Yeah
Basically everything just needs to be like really
Weighted up this way like this you would just try a network to do this so you definitely want to shuffle the data
Not just for statistical reasons, but also just because that's how the neural network models going to work
It's literally going to be shifting the weights to [try] to make these arguments are max functions be true
[and] yeah anyway mmm, so that's why you definitely want to shuffle the features
here
Then we were going to say his features equals an MP [array]
Features okay need this to be an umpire array
And then we're going to say the testing size is just equal to the integer value
We want a whole number here of test size times the len of features, okay?
So [this] is 10% of our features, whatever that whole number is that's what we that's what we want our testing size to be
now we're going to say train x equals the list of
features
: comma 0
This is just some nice numpy array notation that you can actually get away with
When you say : comma 0 basically you'll say you've got an array, and it's like a 5 and a 8 okay
And you've got like a list of these things let's say you got like this
okay, so features : comma 0 and
like let's say you have a 5 a and then a
7 9 okay when you do this : comma 0 stuff basically you're saying I want all of the 0 with elements so this
Here would be
you know 5 7 okay and
In reality these are our actual features so again our feature set is
like this, and then actually in each one is your
Features and label right and then you have a bunch of these
Like this okay, [and] then our features if you recall or actually their own little array, and they're like these little one hot arrays
I'll just do this and then [the] label itself is I don't know 0 and [1] and so on and
so when we say we want the list features 0
We're just taking all of these right all of those like the true feature sets not the labels
[ok] and we want to take those
All the way up to the very end
[software] [updates] on of
Testing size that integer value so basically we want to do this for all of our features up to the last 10 [percent]
Okay, then we take this and I'm just going to copy that paste and we're as I train y equals that only we want actually
You know the first developments there and then when I copy these?
and
Then rather than [train] x it's test x and rather than train why it's test y, and then it's actually negative
testing size to the very end so the last 10% and
Then at the end of the day we want to return
return Train x
Train y test x and
Test why?
Okay, so that's pretty much where everything's going to come together for us
I guess I need to get rid of this before that causes problems actually might not actually got a problem there
I don't [know] anyway uh
And now we're going to go ahead and run this bad boy. We're gonna say if name equals
Main, and this is just so the room
We could reference this later on in our script if we wanted, but if we call this script into action
It's just going to run this instead for us, and then what we're going to say is these values here?
equals and then it's equaling create sentiment feature sets so this function and
Then this function. We pass a pause text
whoops pause, oh text neg text else
We're using all these local um
These local values by the way make sure when you execute this script you're executing it from that directory don't be
Executing yeah, anyway
pause text neg text and
That's it right. We just passed positive
yeah, and the test sighs we're just going to leave [that] because we we wanted that default parameter there and
Now what we're gonna say is with open and then we're going to call this
sentiment set pickle as
WB
or
Yeah, as f rather and what we're going to say is pickle dot dump and we actually want to dump all of
These values and we're going to dump them as a list and we're going to dump them to obviously f
So this creates a pickle of all this information for us because we don't really I mean like we only need to run this function
One time or [this] script one time to have all these elements
And then we can pass them [through] our actual our network, so let's save this and run it and see how many errors we have
So on me try to make this smaller so I can pull this up
cool and
So first we're going to change directory into desktop tf [Tuts]. [I]
Think that's where we want to be right yeah, and then we're going to say python [three] create
Let's see what happens. Okay? It's triple equals
Okay, err number one
Cancel those back over here
Did we print I wonder if we printed the length of that Lexicon? I meant a print len Lexicon, but I don't think I did
Yeah, I [think] for right above L2
It could be useful to just print the line of L2 just so you know like how how big this is
print whoops
print
Len
L2 it just kind of gives you an idea of like how large
But we didn't get any errors and I [promised] to [mning] our pickle is here. Yeah, here's our pickle
That's about 140 megabytes in size so it's 140 megabytes as you can see like this
This is how you can know so they like to run this we're going to use up one hundred forty megabytes for sure of Ram
Just for this and then that's not even including our neural network model
And that's because of the size of our lexicon for the most part so python [3] agree. Yes
We're on that one time hopefully quickly. [we'll] see the Lexicon value here or the length of our lexicon, so
423 so that means every input input vector is going to have
423 Elements to it. Okay
so so that's actually you know quite large as
Opposed to like the initial string Data like opening up pause text for example
Well this first line is actually kind of long. I might it's definitely not 400 characters though
It's you know looks like a tweet almost and then a lot of these are very very short
Short lines so now 423 and really 423 is actually pretty small um overall, but yeah
So okay, so we've got everything now
pre-processed so now we're
basically at the step where [we] were
With the eminence data set there are actually a couple things left changing the script like the batching and all that
Because they had a nice little batch function written for us, but hope that will be a really easy hurdle so anyways
We're pretty much
Well we have we've pre-processed all of our data
And we're ready to feed it through so that's we're going to be doing in the next tutorial if you have any questions comments concerns
Whatever if you're getting an error here anything like that feel free to post below. I'll do my best to help you out otherwise
[until] next time
Maybe I can't seem to get this to stop