Running the Network - 3D Convolutional Neural Network w/ Kaggle and 3D medical imaging p.6 Video Lecture | Machine Learning with Python - AI & ML

what's going on everybody welcome to
part 6 of the Kaggle data Science Bowl
2017 tutorial series in this video what
we're we talking about is taking our
data and running it through a 3d
convolutional neural network now to do
this we're going to be using tensor flow
if you don't have tensor flow you can go
to really you can just pip install
tensor flow so on pretty much any
operating system you shouldn't have no
problems pip install tensor flow now if
you want help understanding tensor flow
if you need help understanding neural
networks convolutional neural networks
whatever I've got links to pretty much
everything you could need for from
installing tensor flow understanding how
tensor flow works understanding neural
networks understanding convolutional
neural networks all that is linked to in
the actual kernel itself and I'll put a
link to the kernel in the description of
the video and so hopefully you can get
all the information you need there
because what we're going to do here I'm
just basically going to go to the
confident tutorial that I wrote and I'm
going to take the code from that I'm
just a copy and paste and we're just
going to make the necessary edits that
we need to make so with that let's get
started also as always if you have any
questions or whatever
feel free to leave them below I'll do my
best to help you so first of all we're
going to start with like some basic kind
of constants so we're going to import
import hence a new as f important numpy
as NP we're going to start with some
constants here image size px equals 50
so life count equals 20 and the classes
equals 2
we're not going to have a batch size and
probably the rest I think I could just
copy and paste from it honestly if you
want to now to get here you can go to
home data analysis machine learning the
big machine learning series scroll on
down to the basically the very body want
to learn about the actual theory of a
continent you can check here and there
are cat pictures and explanations it's
great or you can go to the next tutorial
that is actually for the actual
continent intention flow that's what
we're going to be working with here
and so I'm actually going to go ahead
and I wanted to take this code and and
I'm sorry actually this is Alice's this
is the continent this is a basic
feed-forward back prop multi-layer
perceptron neural network and somewhere
down here is going to be the full code
to ignore fuzz confident yes this is
your content so I'm going to say take
from X all the way down to the very
bottom copy coming over here
mm after n classes paste and now I'm
going to proceed to edit all of this so
let me just get to this and also let me
talk about n classes real quick one
thing just in case you missed it and if
I forget to edit it I made a mistake in
the previous video where we actually
created the you know the process data
here I said it's label I basically said
the same thing like they believe those
one do this and then also I think it was
actually LS labels one do this it was
really stupid
anyway what's happening here is these
are what's called like one hot raise so
this value here is for you know not
cancer and then this value here is
cancer so with a neural network what
happens is you basically got this output
layer and the output layer just consists
of nodes so let me try to find the
introduction to be learning with
attention flow maybe this will give me a
pretty big note this is for tensor flow
specifically introduction in your nose
let's see there's got to be a good
picture here okay
beautiful handwritten picture by me so
you know you get your input layer which
is your feature set and then you guys
some hidden layers and even output layer
in this case this is basically something
that would have three classifications
dog cat in a boat or something click
that right with our cancer data set we
with our output layer we'll just have
two nodes
they'll just be cancer or not cancer and
each node well you know we're saying in
the training set that's you know full
zero or full one because we know the
answer is in reality your networks
actually going to predict out
an array or I'm sorry not to be arrays
factor was your output layer it's going
to probably say something more like zero
point two one zero point seven nine or
something and in most cases going to
have way more decimals okay and then to
get the actual classification the
Asheville prediction we usually apply an
arc maxis function and I have no idea
what that's all red no clue anyway
that's usually what you would do to get
the you know the actual classification
but in this case what we're doing is
like when I should do your submission
that submission will just be a
percentage likelihood of cancer and then
you're graded on the log loss of that
that value so you know if you're kind of
close or whatever or just because you
know like for example Arg max like say
you had someone who was actually had
cancer and your value was 49 and 51 and
Arg max would say yep we think he's got
cancer but you're still way off like
your actual prediction is you should be
penalized far more than someone who says
like 199 and we should be decimals of
course right so it's kind of how you're
being you're being graded anyway I just
wanted to to bring that to your
attention just in case that's not
totally clear okay coming on down okay
so basically we need to convert this to
coms 3d so we're just going to change
first let's go through and just change
all the twos the threes so calm 2d is
confer any max cool 3d coming down here
calm to D becomes song 3d 3d where's our
max pools here we go 3d 3d and I think
that's it what but wait that's not all
okay so now we got to do is we've got to
fix the strides this is easy boom done
we've done size of the window before it
was a two by two
now it's a two by two by two easy ooh
don't forget these strikes hmm we've got
in trouble
padding same and same padding is going
to compete it's just confusing to me I
don't know what I'm doing wrong let's
kind of point out why I'm confused and
hopefully somebody who's watching is
more intelligent than me and can explain
to me
what the deal is with paddocks the
options you have is same is valid now my
understanding is the same when you've
got this window that is a convolutional
neural network as it's going through
your data basically
should be this one you've got a picture
of a cat let's say and it's split up
into grids and then your convolution
window does this and then it slides over
does it again slides over it does it
again but you have a couple choices and
it basically it condenses all of this
information into just one little feature
right and then it slides it over and
then not another little feature and it
keeps going and you've basically taken
this too now a much smaller kind of
image basically but with padding as
you're doing this you've got valid in
same whereas if it goes off the window
goes off the edge it can either kind of
just repeat and kind of fill in based on
the last minute data or you can only
work off a valid data which is kind of
more like what's happening here so when
you work with valid data you're going to
end up with usually a much smaller kind
of image when you're just doing the
actual moving the window around so
anyway I don't know obviously my ideas
wrong there I'll show you why in a
little bit anyway okay so we're done
there now coming into this here we're
just going to modify this so you've got
a 5x5 patch no it should be a five by
five by five patch one channel 32
features produce a five by five by five
32 channels sixty-four features this
number is what I would think would be
fifty thousand right 50 times fifty
times twenty fifty thousand that's not
going to be that number same thing down
here I think it should be fifty thousand
so long as I mean I guess but it's going
to be a little more because it is in
theory stinking off the edge but I guess
my real question to anyone intelligent
watching that knows about these anyways
how can I calculate this number in
advance like I'm not positive on how I'm
supposed to calculate this number
because the way I'm going to do is I'm
just going to run it I'm gonna get an
error and I'm going to see what it what
tensorflow expects that number to be and
that's how I'm gonna fill in it which
obviously it's not the best way to do it
now we're going to do this reshape here
the tensor should be the images X by Y
and then the image Z which is slice
count and again if this is all real
black box to you I mean feel free to
follow to the very end but really you
should go to the tutorial to learn more
about what's happening here so I'm
really not explaining much of what's
going on I'm just assuming you
understand how a neural net is working
also I'm going to go ahead and change
this five to the Ruiz just because that
5s can just be way too big of a patch
the threes should work it's worked
before so I think it will be fine okay
now that I've done that turn the output
fine so I think our definition of the
comments done that should be all we need
now for training this is going to get a
little more complex so basically what we
want to do here is now we need to load
in the data so let's go ahead and make a
space and I'm just going to say much
data is equal to NP it upload much
detection I think it's just much data
like this much data - 50 - 20 - 20 nty
so that's our data set and remember
that's just more an array I think it's
actually a list of the arrays and then
when I say training data is interesting
much data all the way up to the last 100
what have I done
does that just run I'm sorry that's okay
just ignore that is it wait a minute I'm
on the wrong much it might be because
I'm running from the wrong directory
possibly
I think it's because I'm in the wrong
directory here doing like what second
here
okay so we'll find out that fixed it
later I think I started the notebook
from the wrong directory I think that's
why I wasn't finding it anyway
I didn't mean to run anyways at that
point so train data now valid validation
under stored data will be much data
basically negative 100 onward ok so then
our prediction the same no problem
rather than you know for each epoch here
what I'm going to do instead is just get
rid of this line and instead what I'm
going to say is for data in train data
so for data and train data noting for
this but capital X will be data 0
Capital y will be data 1 this will be
capital X capital y everything else
stays the same this is fine
correct fine this is not this will be X
will be I 0 I 0 for 5 in validation data
and then this will be the same thing
this might be running over my face and
it is nice catch copy-paste
in a little bit easier to see hopefully
so just a quick one liner for loop
basically the zeroth element for I in
validation data same thing for y here
actually it would not be I 0 VI 1 that
would have been a problem and the whole
line is doing that some more so I can
see
I think we're all set I might want to
throw an accuracy right here but I'll
we'll just run through the epics let's
do three epochs um the other thing let's
go ahead and run it we'll hit some
errors we're just going to run through a
bunch of errors probably first hopefully
we'll find the file no such file not
found much data I'm looking at the file
hit the hole I spelt it wrong but am
sure okay let's try again so I guess
that's probably why I didn't find it the
first time
okay so this is getting angry about this
nope not the shapes so it's actually
getting angry about the 2d did I forget
to replace I did so this needs to be max
pool 3d I don't even know why the
dimensions weren't there this needs to
be calm 3d there right max pool I guess
this one didn't have does it just say
max pool and it's assumed that it's 2d
or something that's wrong
let me come down in here max pool 2d uh
I got deleted that's weird okay try
again eventually we're going to this
tensor error so I know at least we have
one more error to go through okay so
here's how I'm getting that 54 so it's a
little over 50,000 understandable
because it gets over and it's knit pads
basically so I know I understand how it
happens I just have no idea how to
calculate it beforehand so this is
literally my method like run it
kits attacking you and then go I don't
know
part of me thinks like what if you maybe
it's a 3-bike because I think what you
could do is like three by three so wait
maybe it's five five five on I don't
know I cuz you could probably could
calculate how much over you're going to
get depending on the size of your window
maybe that's it because it's besides
your window and then your strides and
then your window size maybe that's how
you someone let me know
there's got to be a way anyway fifty 408
so
okay and then we're probably still gonna
have one more issue let me pick set
there so this position the reshapes so
basically here this data needs it and
then then the final reshape for the out
okay which on that one
so expecting one at least one more air
hopefully it's the air I'm expecting
seems like it cuz it's actually running
and then I think what I think probably
one of the problems is one of the data
sets it's not properly resized I guess
I'm not know what's problem okay should
hit it and any anytime now
there we go so at this time it says
input to reshape is a tensor with 40 to
500 values but of course we're
requesting at this time a multiple of
fifty thousand so what that's telling me
is at least one of them is a different
size so my guess is it's one of them it
was maybe a after the re after the
sizing of the depth maybe one of it was
actually 17 total chunks or something I
don't know that's just my guess so what
I'm actually going to do is the
unthinkable in cases in a trying except
forgive me such exception II and I
apologize I am going to I'm just going
to pass here you can feel free to not do
that I'm just I'm gonna do it and I know
that just this just looked so bad but
I'm going to do it say I'm going to do
is I'm going to say success total equals
zero attempt total zero and the FIR data
entrance trained out I'm gonna say
attempt total plus equals one and
nominate down here stay successful x
equals one so if we get to the point
where we've done all this that way we
just know like are we doing most of them
correctly and then this should be string
formatted but I'm just going to continue
with my two way I'm doing it
that's rate : and then we'll just do
what is it's the success don't forget it
common here success underscore L that's
running over my face there we go
success underscore total and then just
divided by what was the other one
I have total just so we can see that
hopefully like 99% of these are actually
running through and not hitting this
this exception but we would not hit this
exception on this number on this no
calculation right so so somewhere here
if we do hidden exception at least will
be made aware that way even though yes
we are passing well no hopefully I think
everything else is hunky-dory we'll see
what happens hopefully I don't remember
II sometimes when I'm filming and doing
this weird things happen I'm not
positive if this I think as I record it
sucks up the memory and this recording
for examples at almost nine hundred
megabytes there's a lot of memory so I'm
not positive on that one but I think
that might be happening on my screen
doing weird things hopefully this
doesn't cut out on me if it does what
I'll do is I'll just pause and then run
the network because this is running on
my GPU it should be totally fine on a
CPU the data set really isn't that big I
will say though we are running at 50 by
50 by 20 that's huge at least we got
through one epic great so success rate
was good yeah Oh perfect pretty much so
it looks like probably one one is the
wrong size I don't know that it the CVT
got resize is pretty infallible I highly
doubt it's there it's the function that
didn't we wrote it's a function that I
wrote I won't blame you guys it's a
function I wrote so I'm going to go
ahead and assume one of them just
doesn't fit so maybe there was 23 chunks
or there was 17 chunks or something and
it's just it's just not resizing right
so okay don't confuse at this point I'm
going to remove success rate because I'm
content but the loss is going down so we
are fitting better that's good but the
real question is how accurate are we
going to be at the end so this accuracy
line is it or real is our real concern
and probably what I'll do
once this is done will read the accuracy
and then I'm going to run it probably
more for like 10 or 15 epics but I'll
pause it as I'm running that so you
don't have to sit through that this is
show you what what the thing is I know
maybe I'll print the accuracy at every
step of the way
okay so we got 60% accuracy the other
thing I want to do is epic
that's not twenty equals one plus one
that's not really that's not very good
but anyway let me take this line copy
so now after every epic will also assess
the accuracy just so we can see like the
loss is going down so I mean we are
fitting something but we're probably
overfitting the data because we don't
have enough data so let me change let me
change epochs to ten I'll run the ten
epochs you can either choose to do this
or not
I'm going to pause the recording so you
don't have to sit through ten epochs if
you don't want to if you do feel free go
for it but anyway I'm going to run this
pause and I'll come back when this is
done I'll ride and we are back this is
just through ten epochs we can see that
we did continue to improve on loss
actually it looks like maybe lost one up
here but for the most part over time you
know where we are improving on the loss
so we probably had room to keep going I
mean I might want to see what this hat
what happens at 30 epochs or something
like that just to see but accuracy wise
me started about sixty percent drop down
to the 50s went all up to about 66 then
drop back down in the 50s so it's
unclear to me what would happen if we
kept going
the problem is continuing to go
is eventually you're going to overfit
the data and especially on a data set
the size that we have you know about you
know truly the training did is like
fifteen hundred or something like that
people so it's just not much so the
likelihood to overfit is pretty high now
this is three dimensional data so at
least it's a little harder to overfit
but I mean it's still if there's a way a
neural network should not find it so
that's one problem it's just letting
this continue to run I mean I'd like to
see something a little more positive in
ten epochs now let's say we stopped in
and it was actually sixty-six percent
accurate you know maybe you're thinking
hey that's you know it's not so bad yes
there are 5050 right because the
question is 50/50 right it's cancer or
not cancer but here's the kicker is I've
actually restarted this so I'm gonna
have to pull back up this labels bf I'm
just going to copy these lines here come
down to the bottom paste killing it
support
pandas as PD I just kept running on
memory sickness decide to keep running
it and if that was you by the way if you
ran out of memory I apologize probably
the best thing to do would be to I don't
know whoever it was when we did the
processing make it smaller just make it
smaller that's a huge even you know it
sounds small like 50 by 50 by twenty to
one but it's that's still huge for a
neural network anyway so what we can do
is we can look at the labels data frame
listen and we can just look at this the
just let's see these labels
underscore D F cancer got value
underscore counts really I swear I said
counts okay so what we're dealing with
is you know basically the real thing is
like the classifier isn't 66% like you
know it's not the cluster is not
competing against 50% accuracy because
that's not quite right what you always
want to compete against is what about a
classifier that always predicted the
most common class so in this case the
classifier that predicts no cancer is
right the vast majority of times for
something like 70% of times now even
further the real question is actually
not so much on the whole data set just
on that on that final testing set that
we did so if we did labels underscore
TFS IX for the index and we're going to
address just the last 100 because that
was our that was our testing set cancer
value underscore counts counts why I
like watches doing it but yeah okay so
this is a perfect hundred if a
classifier that always says not on
cancer is right
72% of time so actually 70 percents the
value you've got to be also this doesn't
even begin to account for the log loss
of your your actual values but really
before I would even bother trying to do
anything an output anything that I would
submit I would like to see an algorithm
that is better than 72% also I mean we
are predicting cancer for people like
it'd be nice if this was 95 percent so
need to be really accurate ideally I
assume that you know even if someone
used this a doctor would first just use
it to quickly go through a bunch of
stuff and then double-check but anyway
or or use the algorithm to see stuff
that maybe a doctor wouldn't see you
know because the algorithms are going to
nice cut like a neural network might
find something totally never before
known that actually helps you spot these
things in advance or something okay so
there's that now I think for now I'm
going to be done with this we've I've
run through a whole lot of information
we went basically from the data to a
model now what okay so you know in a
beautiful world in the ideal world and
the typical world of tutorials we
usually we already know the answer when
we're going through something so in a
tutorial sense when you're going through
it whoever wrote the tutorial knows it's
going to work in this case I pretty much
knew it wasn't going to work just
because I knew we don't have enough data
but I was hoping to see maybe something
a little better than what we got but at
least it worked and this was my first
time running a 3d convolutional neural
networks so cool there could be errors
in here I might have missed something
huge
so maybe I messed something else up
along the way who knows just really a
first pass but regardless let's just
let's say I did everything right and we
still get any numbers what are the next
steps what are the real next steps that
that I would take well first of all my
theory is that we still just don't have
enough data so what do we do well I
showed you the links already you can go
and get another 888 samples of lung
scans pretty sure those are CT scans
saying ish format it's going to be 3d we
as long as you're down sampling them
like we are everything should pretty
much be the same you can also go to that
exterior data sources get it as much
data as possible because that's what
that's the biggest thing I'm not
confident that I've not seen any
exterior data source but like basically
for a neural net I'd like to see at
least one hundred thousand examples
honestly so the next thing that we could
do is create new data from the data that
we have so you can you can add noise to
data set so for example if you followed
the the tutorial series for image
recognition with open CV one of the
things that we did was we made we wrote
a program that would audit I'm not
wearing it oh here it is I have a watch
and it would automatically detect just
this watch but the way that you trained
it was making a bunch of noise images so
you took the image of the watch and it
just slapped it on a bunch of other
images and it rotated the watch zoom
didn't zoomed out did a bunch of stuff
right so you can kind of do that with
these images too but before you do that
I would highly recommend again go to
discussion go that tutorial where that
doctor basically describes what they
look for in a CT scan and make sure you
don't add noise that screws that up or
even worse adenoids that that would if a
doctor saw it would be like oh my gosh
that's cancerous you know so really if
that it's going to be a very tricky
thing but if you can you know move
rotate things a little bit or change
things around especially with you know I
kept it with 2d images it just it works
pretty well with commnets to just do
that and boom you've got some noise
other than those two things you know
maybe a continent isn't a solution I try
to go look it to gradient boosting XG
boost that probably my next step but
yeah there's a lot of models to feed it
through maybe the 3d data isn't the best
way to go about it maybe you feed
through this each slice at a time and
then each slice has a classification you
go by a standard deviation or something
and if someone deviates one way or the
other way that's a label there's so many
ways to approach this
that that's that's the challenge that's
why this is a million dollar competition
so anyway again if you guys found this
notebook useful please do go to the the
notebooks section and give me a vote up
I'll also put a link to this in the
description of the video help me out a
lot that's part of the competition is
the top I think top three vote it top
three voted kernels if you have
suggestions improvements if you take
some of this and you change the model a
little bit you want to share it with us
please do if you've got improvements to
my code I won't be offended this just
wasn't the greatest code I'll be the
first to admit so anyways that's it I
hope that was helpful for some of you I
hope some people learn some new things
hopefully you guys can share some new
things with me definitely try out the
competition it's for definitely for a
good cause so anyway that's it for now
I'll see you another video