Resizing Data - 3D Convolutional Neural Network w/ Kaggle and 3D medical

what is going on everybody welcome to
part 4 of our Kaggle data Science Bowl
2017 tutorial series in this video were
me doing is basically in the last video
we resize this kind of 2d element but
the actual depth part we're having
problem lists now it might be possible
that you can do this with cv2 I just
really don't know that that CBG will
work with 3d images so what I'm going to
do instead is basically my idea is that
well all we really need to do is take
the image or the slices put them into a
list of slices and then chunk that list
of slices into a fixed number of chunks
of slices and then you just average
those chunks okay and then that's how
you could do it so let's just kind of
extract out abstract out the hole that
it's you know chunks of scans just the
question is okay how do we chunk a list
of Lists or something sorry how do we
chunk a list into a list of Lists well I
don't know the answer to that or I
didn't anyway so I went to Google and I
just type you know I have to chunk a
list of Lists a list into a list of
Lists Python so okay first answer let's
check it out scroll on down it's got a
big green check how do you would need to
do this it's a generator that yields the
chunks that you want it takes in L which
I'm going to assume as a list it takes n
which I assume is some sort of number
yes it yields successive n sized chunks
from L so it's not quite what we want we
want a chunk we want to be able to say
hey I want 20 chunks from this list so
it's not quite what we want but that's
alright we can use algebra we can solve
that problem so we're go ahead and take
that Thank You Ned
so we're going to take that we're going
to come over here make some space here
paste cool so and while we're doing that
let's go ahead and source this so it's
Ned I can never get it
emerenc Bachelder I shall - will do I
don't know that shoulder I don't know
I'm looking correctly below tell me how
to say his name for so good that
shoulder
ling-ling boom
alright I said next what we're going to
do is basically when we have these
chunks we're going to need to average
them together so the way that we're I'm
just going to do that
just a mean function to define mean it's
going to be mean of a list return sum L
divided by Ln L really simple stuff okay
we've got image pick size we also need
to have one more and that's going to be
how many slices but now what we're going
to do is before we get too far into this
let's import math so it should actually
go here import import math and now we're
going to come down here to this this
area after we've kind of got our slices
once we have our slice is we're going to
say instead slices actually we'll start
with we'll start with new slices and
that's just going to be an empty list
and then what we're going to say is
we're going to resize the slices so when
SI slices equals and this is just going
to be a list of a one-liner kind of list
of lists here and we're going to say
we're going to copy this copy and paste
for each slice good don't forget you're
on a short each slice in slices so just
a one-liner for loop that's going to do
this resize operation okay so it just
creates these new slices already resized
for us now we're going to do is we've
got to figure out okay so we know we
know we want 20 chunks but we've got
this generator that will just chunk into
n sized chunks so we're just going to
use algebra
so we're going to say basically we need
to know how big of chunks sizes do we
want what would say the size will be
math I feel the Len of the slices
divided by how many slices do we want
that will give us the chunk size where
it produces how many slices you know how
many chunks basically or hm slices
whatever that is that many so so 20 oh
that's our goal
but whether you use masa field map floor
or round or whatever it's not gonna be
perfect it'll be close enough so we can
run that and then what we can do is we
can say we'll come down here and we're
going to say for slice chunk in chunks
that's our generator here and the chunks
takes slices that a list and chunks it
into the chunk sizes that we've
determined here to give us a total of 20
chunks now for the slice chunk and
remember that's a list of slices still
we need to average those together so
what we're going to say slice chunk now
is going to equal list.map if you're in
Python to just map I think this map
still works in Python 2 but don't eat it
we're going to map that mean function to
the zip of argh plus hunger so just you
know any length of slice chunk basically
those are hard now we're going to say
new slices dot append slice chunk now we
should have about 20 it's not going to
be perfect so we're gonna say print Len
new slice and for now I'm going to
comment this out so I'm just going to do
let's see
thanks autocomplete I really I really
appreciate you oh gosh I hate you all
right there we go
no mmm there we go all right today
Centex fights autocomplete so 10 all
right let's run that real quick see what
we got sure enough twenty nineteen
nineteen twenty nine and all the way
down to eighteen even thanks alright so
now i'm going to do is go ahead and
insult the Python uses in the crowd I
really don't know the best way to solve
this problem right I just know of a way
to solve this problem and that's just
that's the reality
okay that's just the way it is if you
know of a better solution or if you know
see me do not resize works on 3d images
feel free to share your your better way
I mean I'm interested to see what better
things people come up with also like
some sort of wow loop would pry D better
than what I'm about to show you but this
is what I did so I'm just this is my
real raw uncut first pass all right so
let's say it's Len new slices if that
length is equal to H M so how many
slices minus 1 beta like do it if we
have 19 slices what do we want to do
well we just need one more slice so
we're going to take new slices dot
append new slices naked one so we're
going to take the last slice and just
spread your stuff it right back in there
like so boom twice is done if if this
happens again like or basically if if
it's not negative one but it's negative
two
well we can just do this done next
question is what we we should not be
more than two if we're more than two
that's really weird either way like that
really shouldn't happen so never say if
when new slices equals hm
- what do we want to do well okay this
is going to be a little longer but the
idea here is like if it's too big what
we're going to do is we're going to take
that last slice that last chunk
basically but now it's just a slice and
we're just going to average it to the
second-to-last slice
such making one big one slightly larger
final slice okay not the greatest but
that's what you get that's what you get
so so now we're Issei is new Val equals
and this will again be list.map and
we're going to map that mean function
again it will be zip again going to be
an Arg and in this case this is going to
be really hard for me not to write wrong
but here we go
new slices is the index of HM full
slices minus one and then we also want
to just do new slices HM slice right so
the final two we're just going to
average them together
that's your new bound now we're going to
say we could just replace it well now we
actually have a kind need to do this so
we're going to del new slices hm pluses
and then we're going to say mix new
slices hm slices oops plus one equals
new move Val okay so HM slices minus one
that's the final one you could also
probably you could say should be able to
get away with this like negative one as
well anyway new Val all right next if
it's plus two what if it's plus one well
we just do the same thing
so there's plus two we still only
removed one slice so plus one is still
going to be the case so boom and if it
was never plus two but still only plus
one right again I apologize profusely
but that's what I wrote the first time
around
feel free to fix it if you'd like now
finally actually hmm I still come on
printed slices what's going to do that
real quick print when new slices all 20s
for everyone okay
now let's go ahead and graph it
now it's 20 so we really just wish me to
do this we're going to say new slices
for numb each slice boom it's a four by
five now and then we're all set
so again so oh we don't need to resize
it right so
[Music]
dress why did you do that
I asked you to run why are you not
running I don't know why I spit out such
an ugly are you really gonna do me like
this let's say Kui
is one of the other data I guess I
shouldn't have stopped it it was there
it just had that really weird looking
fun young dude I will fight you jerk all
right sorry about that
oh it's printing up all of them oh my
gosh how many are we doing here
this is just ten anyway these are our
patients yeah okay so so we've got what
we need we've got what we're looking at
we're content with what we've seen we're
expecting that if there's tumors here we
would still be able to see them even
though you know because in theory our
slices hopefully went from I don't know
a few millimeters to maybe a centimeter
or something like that so if there's
still something there because they're
averaged together it should still be
visible
that's my assertion okay so now that
we've made it to this far what we need
to do next is actually pre-process the
data because our end goal still is to
pass this through some model and the
model of choice that I want to pass it
through is a three dimensional
convolutional neural network so so
basically we need to convert this into
some sort of acceptable data set so
that's what we're going to do in the
next tutorial if you have any questions
comments concerns whatever feel free to
leave them below if this kernel is
helping you out please do give me a vote
in the kernels on Kaggle helped me out a
lot otherwise I will see you in the next
video