Preprocessing data - 3D Convolutional Neural Network w/ Kaggle and 3D medical imaging p.5 Video Lecture | Machine Learning with Python - AI & ML

everybody and welcome to part 5 of our
Kaggle data Science Bowl 2017 first data
run-through tutorial series
in the last video we've we covered
basically how we can at least manipulate
our data now we just want to cut just go
ahead and pre-process the data so what
I'm going to do is like get rid of this
line and I'm just going to say defined
process underscore data it's going to
take patients as a parameter and it
takes a label DF as a parameter so it
can quickly get the label it's going to
take image pick size if it's not
provided we'll just assume default 50
how many slices 20 and then we'll have a
visualize and on default impulse so from
here let's see so we're going to how did
how many slices I must have just deleted
that the Hat tube that hasn't been there
before
I don't know how that got deleted anyway
and then I'm also going to change this
to 50 yeah there's just no way tell me
because I was how many it's like yeah
weird anyway moving along so we've got
processed data cool everything else I
think can just stay the same in that
function for the most part we'll just
say down here if visualize for whatever
reason we'll go ahead this code
otherwise as we get to the end here what
we're going to do is we need this label
basically to be in what's called one hot
format I'll explain that in just a
second but let's start right it out so
if labeled equals 1 so if the label is 1
I'm just going to say label equals and P
dot array 0 1 that's a 0 comma not a
decimal point and then copy/paste
I'm not sure what did I do wrong there l
if I don't think something's wrong
that shouldn't be happening I don't
think I've violated anything by doing
what I did there but apparently I did I
know I think that's valid code to have
it all in one line whenever I have one
line underneath the thing I always do
that I guess oh here's what's doing it I
knew there was something wrong okay so
it's that : up there let me put this
back here I don't think this is pet bait
so I apologize but oh it's still
happening I'm gonna have to run this and
see what happens I don't know
well well we'll come to that soon if
that's a syntax error somewhere we'll
find it just as a quick update do not
follow what I said here on these last
two lines basically if label is one the
array is 0-1 if the label is zero the
array is one zero I just want to make
this correction I catch it in the next
video but I don't want everybody to run
to run this process and then see in the
next video that they've got to redo it
so just make sure that's the case is
just when I copy and paste it I didn't
make the proper changes anyway when
we're all done we're going to return
just for sure again on pi array of the
new slices and label alright now we
already have to find the data directory
way up here so I'm not going to really
worry about it but you want to make sure
you've cut that and then we're just
going to make like one big array and
then we're going to save it to a file
and that's going to be our data set now
couple things just to know one if the
data set was big enough you wouldn't
really necessarily want to do this
because first of all it would just blow
out your memory so you'd like maybe try
to have the whole data set in your
memory but in our case this is a data
set that's just under 1,600 people it's
a 50 by 20 or it's a 50 by 50 by 20
which actually is big for a
convolutional neural network but to have
only 1,500 samples in terms of how much
it's going to take
in our memory just to hold that data set
it's like just you know tiny so it's not
a big deal now but in most cases you
wouldn't get away with doing this you'd
probably have to do the pre-processing
and what's you call online
so as you're feeding the data through
your neural network you would actually
take the data process the data return
this and then feed it through your
neural network you would do this online
but obviously this if you can just do it
once it's going to be quicker to train
the neural network over many epics and
stuff so just if you can break
pre-processing data don't do it online
but if you can't understandable do it
online but in this case we can do it so
I'm going to do it anyway written as
much data what is equal nothing
fantastic variable names if I do say so
for none patients in a noon rain patient
if you're on the cattle Colonel it's not
such a big deal but if you're doing this
against all of the data this can
actually take a while so it's nice to
know where you are so if none module low
100 equals zero print num so basically
for every 100 patients we're going to
it's just going to output that number
just so we know like where we are in the
data set really now we're going to do a
try and accept loop and I'll explain
that I'll fill in the exception soon so
what we're going to say now is image
data label people's process data patient
because we're iterating through the
patient labels which is our label data
frame which actually we try to find way
up here so actually some label DF coming
back down here label underscore DF I
think said dot yes sorry labels DF and
then we're going to say image takes size
equals image size px with this variable
and then hm slices will be equal slice
can't wait did I do appear I'll just say
agent slicer
should be fine great nigga also just
left the D balls anyway much much
underscore data dot append image data
label and then what can happen is
sometimes you won't have labels because
because label DF as it tries to find
that it doesn't exist so for some of the
patients it doesn't exist so except
we're going to handle for a key error
which will happen if I see if that index
if that patient ID doesn't have a label
that we know of we're going to get a key
error so we'll just handle for that and
we're just gonna say print this is
unlabeled data so we don't want to train
on unlabeled data then when all of this
is done NP dot save and we're going to
save and just a simple file much data
- these formatting dot n P Y and then
dot format and then I'm going to do
image size px and then again image size
px and then slice count and then we're
saving much data so numpy that save just
save to this filename and we're just
using string formatting here just so the
filename actually says okay it's much
data it is a you know a fifty by fifty
twenty so this will say much did - -
fifty - fifty - twenty just so we
remember we know okay this data set has
those values like that's the size of the
data that's in there now I'm going to go
ahead and come here
patients let's go just like the first
ten just to make sure it works so I'm
going to run that
sure enough label DF not defined have I
but that's not a labeled yes it does so
it needs to be labeled yes again still
read in any sized mix what we got up
here where's our where's our image pic
sighs all right come on man you're
killing me
image pic size image fixed size image
pic size oh yeah no errors okay sliced
it didn't mean eyes just look at this
good oh okay all right clearly is
there's too many tutorials in a row I'm
really hoping to get through this before
frying my brain but this is just too
much just too much little baby hairs all
right I think we've got everything let
me find this flower real quick and make
sure okay that looks good to me so now
do it for all the patients go ahead and
run this this is going to take a little
young 60 minutes so I'm going to go
ahead and cut the video here because
this is just going to be a little bit of
weight
I'll just see how long it takes there's
1,600 patients or just under so it's
good it'll take quite a while let's see
how long it takes me to get to at least
100 the other thing you probably could
have done is just use like
multi-processing
and I would speed this up significantly
but we really only have to run this one
time such as like start it and then go
make some coffee or something and then
come back maybe maybe more than coffee
maybe coffee in the shower and breakfast
and a walk anyway cool so assuming this
doesn't air or whatever if it does throw
mean air or something all I'll talk
about in the next video so I'm going to
go ahead stop here if you've got
questions comments concerns whatever
feel free to leave them below if this is
helping out give me an upvote on the
colonels and help me out otherwise I
will see you in the next video and we're
actually in the next video we're pretty
much good we're going to take this data
we're going to start running it through
an element so big things happening
alright see you next time