Reading Files - 3D Convolutional Neural Network w/ Kaggle and 3D medical

what's going on everybody welcome to the
second video in the kaggle data Science
Bowl 2017 first task video series in
this video we're going to doing is just
talk about handling the data so a lot of
tutorials that you're going to go
through including even some of the ones
that I put out a lot of times the data
that you're dealing with is kind of
pre-processed pre handled already kind
of prepackaged information for you to
just really quickly stuff it through
some sort of classifier in reality data
doesn't just isn't typically in that
form so the first thing that we need to
do and even if it is in that form the
first thing we need to do is we and we
need to look at the data we need to
understand how we can pull this data and
actually begin to look into the data and
then we can start kind of manipulating
it to however we need so the first thing
that we know is the case is at least for
now the data that we have is CT scan
data and what it is is each patient
basically if you go to the file so for
example I can actually bring up the
files so here are the files that I've
got downloaded basically what you get is
this is me I've already pre processed
some data there but basically stage one
and this is the stage 1 labels and
lennie's are all system this is also a
pre process but then this is the sample
submission CSV and all that but
basically stage 1 is what we're
interested in if you're following along
in the kind of embedded kernel on Kaggle
you only have access to sample images so
you'd come here and then as you can see
you've got I don't know 25 patients or
something ominous some amount of
patients so these numbers here these are
your patient IDs so this is a patient
and then BAM this is their scan so you
have some about you know 200 files that
are a scan data for that patient so each
of these is basically like a layer in
that scan but looking at this now you
probably have no clue that's what it was
you've got to do a little bit of
research to figure out ok that's that's
how it's working or if you're already
maybe familiar with medical imaging you
would you just know like oh it's 3d data
but this is all basically one scan
it's just each layer in that scan so and
also just for the records so you could
put these together and someone already
has done that so I'm really not going to
spend time doing it but if you go to the
Colonel's and you go
really oh okay let's go through
competition and kernels so the data
Science Bowl and then go to where is it
there we go
kernels so that's pre-processing
tutorial you can click on that just kind
of see or you can go through this kernel
as well but I'm just going to scroll on
down so right here's the 3d rendering
and then here's another one and another
one and then another one right so so
then these are created with these these
DICOM file scans so anyway that's the
data that we're working with we still we
still need to see it we got to see it so
we know what we're doing so first of all
we need a few things if you're doing
this inside of the casual kernel you
don't have to install anything but if
you're following along on your own
machine you have to do the following up
do it I'm just going to write it into
here but I'm gonna make sure it's big
enough for every bite to see maybe not
that big but that'll do something like
that I wish I could make this I'm sure
there's some way to make this all this
heading go away I'm just not seeing it
would be nice like a district anyway pip
install we're going to need pandas
pandas should automatically give you an
umpire matplotlib but just in case
pandas numpy matplotlib pi DICOM and did
you do that should be it for now I think
the other stuff that we hit I'll kind of
talk about it you could just I just
throw in TV - TV - yeah just make sure
you get at least those before we
continue on okay so I'm gonna clear that
out and now I'm going to go ahead and
begin importing DICOM which will give us
PI daikon this is how we actually need
the daikon files and then we're going to
import Bo s this is just going to let us
kind of do directory operations and then
I'm going to import pandas as PD
for data analysis and really pandas I
only really have one true use for pandas
that you could also probably just import
CSV or something to read this data I'm
just going to use pandas so if you are
comfortable using just the straight-up
import CSV what we're going to do we're
just going to use that to quickly grab
the label for an ID so yeah anyway but
later on pin is like if you're going to
continue doing stuff to this data a pin
is a useful library to use but anyway so
now we're going to do is specify the
directory of everything we're using if
you are in the cable director for the
kaggle lakhan's online version you would
do something like this data dirt equals
[Music]
dot dot slash input slash sample images
if you're not let me write this whole
thing out for people that would be
following along they'll swap it but
basically you wherever your data
directory is wherever you save this
files so data der there we're going to
say patience patience is just going to
equal OS that Lister whatever the data
there is so this will just have all of
whatever in that directory so again
sample images there would be all these
directories and again each of these
directories is that patient's unique ID
so okay so that's our patience and then
we're going to say the labels underscore
DF is equal to P dot read underscore CSV
and we're going to read stage
irrationally started if you're on a
Kangol it would be doc dot slash input
slash stage one underscore labels dot
CSV and then you're going to say index
underscore column equals zero so that
first column now if you're on your own
computer such as me it's pretty to
install these files so or download it
has there's no installation for the data
wherever you download the files to there
the coming
you know zipped up and stuff and you'll
need to be compressed of course but then
it's wherever those are minor and my
x-files so x for me and it's Kegel
underscore data
data science full 2017 in flash stage
one again this isn't necessarily where
your stage one will be it's just where -
and then same thing down here it's
basically the stage one labels I think
actually just put this in the directory
I'm working in so my main drive is
business like a solid-state drive but I
don't want to waste 140 gigabytes in my
solid-state drive so I put this in in
one of my larger hard drives
ok just make sure you whatever these
files are you you you're pointing to
them this should be one of the only
major changes you'll have to make
depending on where you are in what
you're doing and finally we can do
labels on a square da head just to print
the first five rows here sure enough
there we go so patient this first
patient is classified as having cancer
no cancer no cancer cancer cancer and so
on so we know the label is for quite a
few of the patients some of the patients
won't actually aren't contained in here
okay so that's what you're going to
validate again clear so coming down here
once we have yeah we've got the patient
IDs and stuff but we still need to kind
of see what we're dealing with so what
I'm going to do is I'm going to start a
for loop and for now we're just going to
iterate through one thing but later
we're going to we're going to dive a
little deeper and it will actually start
pulling more than just one patient but
for now we're gonna say for patient in
patient to the first two patient
so this will just be one patient and
we're going to say the label label is
equal to labels underscore D F dot get
value and we want to get the value for
basically the index because okay so as
we're iterating through patients just
remember what patient's is that's the
listed directory of that data dirt so
that's all these these patient IDs now
the patient ID is the first column and
the stage one labels so it's here so
we're saying we want to get values for
the index that is the patient so
whatever that unique value is for our
patient that's the index we're
interested in and then we want the value
for the cancer column
so that's the label for the patient and
now we're going to say what's the path
to this patient it's just a big data
dirt and you could you could be a good
little Python Issa and use OSF Lister
right sorry Oh s dot past that joint is
the the way you should be joining
directories but this is my first pass
this is the way I did it initially it's
just easier in my opinion so I'm
cheating a little bit but anyways we'll
do that
whoops I didn't actually mean to run
okay so that's the actual path to this
patient and inside that path is all of
those daikon files okay so I'm gonna use
two of the kind of one-liners that I
grabbed from that top kernel so I'm just
literally going to copy and paste them
into here copy and just for the record
let me bring this I guess so everybody
else can copy looking so it was from
cattle comm competitions there we go
going to competitions cackle you're
going slow man data science Bowl kernels
full pre-processing tutorial and what I
want these lines right here fantastic
copy coming over here paste I couldn't
have said it better myself so we're
going to take those so what's happening
here okay we're using DICOM to read a
file with what file are we leaked
reading we're reading whatever the path
is which we said is the dated early
patient so the whole patient path plus s
so what's s well for s in OSF Lister
path and in the path of the past of the
patient so this is going to get us every
single daikon file awesome now this next
line kind of gives the way one of the
things about these daikon files but
that's okay it's sorting these DICOM
files using this beautiful lambda
function and the way that it's doing is
it's sorting it by X dot image position
patient
- okay so what's happening here is X is
referring to these daikon files and when
you read in these daikon files they're
not just like plain text they actually
have like attributes and stuff to them
so you'll see at the moment but just
know that that's what it's doing it when
the attributes is the image position for
that patient so it tells you where that
slice is supposed to be in order
so if you did do a 3d rendering as they
do and that that that kernel here right
just a little bit below here's that 3d
rendering it would in proper order so
now we're going to do is let's just just
a curiosity let's print the lens of the
slices and whatever the label is we'll
we'll do that also let's print slices 0
so this will tell us how many slices do
we have cancer or not and then also this
will just print out one of the daikon
files for us just so just we can see
what we got here so let's go ahead and
run that just for the record I haven't
said it but to run its Shift + Enter if
you're new to - ipython notebooks enter
is fine you can just keep typing and
won't run anything so so this was the
amount of slices so we have a hundred
ninety five slices and this pin the hey
patient does have cancer and then coming
down here this is all that daikon file
so it's got all this information here so
we have obviously a lot of stuff to work
with so so now what if we instead what
if we just modify this slightly we're
going to say for patient and patients 10
and then let's just do let me think here
we don't need to print the daikon file
let's just print slices the label and
then also one of these scroll down here
where are you
oh right one thing to take note is rows
and columns so 512 512 that's most
likely telling us that you know this
this data is a 512 by 512 image okay so
if you're not familiar with
convolutional neural network ship is a
neural networks in general and working
with image data that's a huge image now
also it's not just a 512 by 512 from
recall this is a 3d scan so this is
actually 512 by 512 by 195 that is huge
so we already know right of the gate hmm
we're going to have to we have to really
resize this information and we're going
to happen to deal with some loss of
information when we do that so the other
thing I want to do is let's get this
information for a few patients so we've
got slices and then also the other thing
we could do is we're pretty sure it's
512 by 512 but we might want to make
sure and what we are going to look for
is there's pixel data and then there's
also somewhere I think is the pixel
array not like totally seeing it
probably blind
because I see the other I seed pixel
pixel data but I think what we're
actually looking for is pixel underscore
or pixel space array probably it's just
not seeing I'm pretty sure we have a
pixel array let me do that real quick so
if I come up here and do let's say
slices 0 dot Hicksville underscore array
and let me just do a shape let's run
that real quick yeah so it does have the
pixel array attribute and I wonder let
me after this is done I'll do pixel data
but this is also bringing up another
issue so as we run through this we see
okay everything is a 512 by 512 but
there are different numbers of slices so
the depth is different opposed these
scans again if you know anything about
any machine learning models you know
that the input data for most models some
models no but let's say for a
convolutional neural network the input
data must be all the same size you can't
have different size vectors so that's
not going to work so again not only do
we have to resize this whole 512 by 512
and also resize 195 I mean these numbers
are way too big too we also have a
problem not all these images are the
same size
so we know right away we've got some
work cut out for us also I just want to
run pixel data real quick I just I'm
curious live testing how you gonna tell
me we not pixel data then I not just see
pixel data
let's do slices zero patient ok clearly
I'm just uh just a big dumb
interesting because you can reference
these other elements because I've
already referenced them I'm not sure
what I'm doing wrong there slices zero
diet and I'm pretty sure it would just
be all underscored but let's do how
about pixel data because that's how we
reference this one so that pixel data oh
we probably made a huge mistake there
it's a lagging out on me have we I think
it's game over this is probably way too
much had to print out or something I'm
totally frozen yeah whoops ha ha ha yeah
you did a science ok so this is a guys
probably take me a little bit to fix I'm
sorry I let you guys down a bad path oh
come on man oh dear I mean ok please let
me edit this I just wanted it it Oh
what I really need to go to do this oh
ok that's fine that's fine
I just wanted to okay so like surprises
what what does it say I was doing the
pixel alright so yeah X let's see no yes
what we could do slices 0.6 tool for
riskless it's a ray dot shape so that's
just an attribute of the daikon file I
suppose alright that was fun
so alright so we know we've got that
problem the other question is okay how
much data do we have well we can quickly
just use Lennon poss Hanson just on our
1,600 so I mean that's a lot of patience
but in terms of a let's see neural
networks that's not a lot so there are a
few things that we can do one we could
you know just quit or we can continue to
at least see what happens you know you
might be the case that this is okay
especially if if it is a really easy
classification type of problem this
might not actually be that that bad I'm
going to wager that's going to be a
problem but one thing to note is like I
was saying before did you'd be doing
yourself a disservice if you don't go to
the forums so or just fully read the
competition so two things I want to
point out is when we go to the data
Science Bowl if I go to tutorial I think
you can find more data e is somewhere
here we go the Luna 16 grand
Challenge so there's another I think
about 800 680 8888 new files I can't
remember for sure ah let's wait um I
don't see it it's it's here somewhere
I'm just going slow comment away so it's
good data first I believe it'll be in
here hmm yeah 888 CT scan so in theory
as long as these are not the same CT
scans maybe you could you could throw
them in but then also if you go to
discussion and we scroll here we go
official external data thread boom so
this is going to be a really common
thing that you're going to be able to
see sure enough you scroll down sources
we okay so you could continue to build
more and more data here now most of
these data sets are not huge so it's
still probably not going to be the case
that you're going to get to a hundred
thousand samples but if you could now
your neural network is going to be
cooking anyway the show must go on so I
think what I'm gonna do is I'm going to
leave it here in the next tutorial we're
going to start talking about using map
Holub to visualize this because right
now we're we're still kind of dealing in
very abstract terms we're iterating
through we're getting single numbers but
we still haven't really seen the data
right we've seen attributes of data so
in the next tutorial we're going to
actually really see the data we're going
to look at it in matplotlib and it
should be absolutely clear to us within
short order that we whether or not we're
actually looking at lung scans or not so
that's what you guys have to look
forward to questions comments concerns
whatever leave them below otherwise I'll
see you the next video