Introduction - 3D Convolutional Neural Network w/ Kaggle Lung Cancer Detection Competiton p.1 Video Lecture | Machine Learning with Python - AI & ML

what's going on everybody welcome to my
first pass through the gaggle data
Science Bowl 2017 challenge so what my
aim here is to do is just just show you
my first pass through dealing with this
data seeing what the data is trying to
model and just seeing kind of what
happens so I think for a lot of people
just jumping into a challenge right of
the gate or any of these competitions
can be really difficult because it's
just there's so much you have to digest
in a really short amount of time also a
lot of times the data is so big and it's
not pretty data so a lot of times if you
follow tutorials even a lot of my
tutorials the people who make them try
to keep the datasets as simple as
possible so we don't spend a whole lot
of time cleaning and processing all that
because that's not exciting but when it
comes to actually do real-world stuff
that's what you got to do so anyway so
that's what this tutorial is going to be
focused on it's just getting through the
data we will apply it we're going to do
a 3d convolutional neural network
against this data and just just do what
happens so this is my real first pass
it's going to be rod there's going to be
errors probably using the code and all
that feel free to correct me feel free
to improve anything you want all that my
goal is that anybody can follow along
we're going to be covering a lot of
concepts a lot of topics here I can't go
into depth in all of them we're going to
be using pandas matplotlib OpenCV
tensorflow and probably a few more than
I'm forgetting if you don't know those
are not familiar with them or whatever
that's totally fine you can ask for help
also I've got tutorial series on all the
things I just listed and for everything
we're going to cover here I'll have I do
have tutorials already out for all those
things so if you're confused or whatever
you can check those out you can go to
Python programming net and just search
for them or something
also as we at least the first times we
hit any of these libraries I'll try to
just bring it up one more time and tell
you where you can find it more
information on them so there we have it
so now if you are not or if you're new
to cattle you can go to Kaggle comm
create an account head to competitions
and the way to these work is pretty much
the same across the board we're going to
be working in data science both
17 but I'm just going to use this one
for an example because most of the
challenges seem to be more like this the
data science poll has one little wrinkle
that's a little different
we actually a couple but so first I'm
just going to reference this one because
this one's a little easier for me to
illustrate to you how cattle typically
works so you can click on the challenge
and like these are the paid challenges
right now that they also have other
competitions going on they're pretty
much always running a digit one always
running Titanic it just it just keeps
you know cycling through basically and
so not all of them are paid to paid the
actual ones that have prizes are just
they're going to be more competitive
they're much more challenging to
actually win okay so but sometimes they
have other things going on so for
example I'll point out everything with
the data science Bowl but first let's
just talk about how these are typically
structured so you've got this nature
conservancy fisheries monitoring
challenge we come down here and I'm not
going to read the whole thing for you
but basically the task here is to
identify the fish that these these
people in the boats are catching that's
the task so cool you can we'll come to
the data last but you can go to kernels
and you can see basically kernel fancy
word for script this is just scripts
that people have written you can see
what the language is I think they just
have are in Python I've never seen like
Java or C or anything here but I'm not
positive on that but it's primarily
Python so you can see that people have
both scripts or ipython notebooks both
of them are acceptable for being
classified as a kernel so so yeah
there's that and so you can check those
out just for examples basically what
we're going to be covering here what I'm
going to be running you through is going
to be a kernel so those are your kernels
also discussion board this has great
information you would be doing yourself
a huge disservice if you don't read the
discussion board there's a lot of really
great things that people just just
openly share off of the discussion board
that will help you out then you've got
the leader board this is how you can see
how people are doing in the competition
the competition is typically scored will
go to overviews and evaluation
a lot of them are scored on logged loss
format so for example this one has eight
different classes of fish but like for
the one that we're going to be doing
it's working with cancer data so it's
basically this either there's cancerous
or not so that's a binary thing but your
your actual model is not necessarily
just all wrong or all right models
working in in like degrees basically
gives a percentage of you know what it
thinks something is the case it's not
just yes this is no it's not the only
time we get that is when we apply an Arg
max function right and then it becomes
that but before that right before we do
the Arg max to get the yes or no answer
we actually have a percentile usually so
anyway log loss so that the person who
is most closely fit to the ash to true
scenario that's generally the most fair
way of going about it there's problems
but that's that's that's the way that
also the other the actual data science
Bowl is is evaluated I think other
prizes and just see so in this case
first place is fifty thousand all the
way down to fifth place gets ten
thousand and then you can see the
timeline and all that ok so we're going
to go and now we're going to look at
actually want to show one more thing the
most important thing the data so most of
the competitions are shaped like this
you've got your basically this is a
sample submission so we'll talk about
that last so actually we'll start with
train so the training data generally
will consist of your data and the
Associated labels so this is what you're
going to feed through supervised machine
learning algorithm in most cases you can
say here's the data this is the class
here's the data here's the class data a
class level one you could you fit and
then you get to test test is just the
data no classes so what you do is your
model you brought through the test data
and you put you create their you predict
the classes and you predict them out to
a CSV file that has usually like an ID
prediction ID prediction ID prediction
and so on and the sample submission is
just an example just to kind of show you
that this is what it should like when
look like when you upload it and then
finally when you are ready when you've
created this file you submit your
predictions and then you're immediately
put to the leaderboard you can see how
you
in general you get like two or three
submissions a day so you can't do too
many because you could just you can
actually make a model that fits the
predictions so as you you submit one
prediction okay you change some things
you submit a second prediction and if
you keep doing that you can actually
just fit that and cheat so that's the
problem so now coming to competitions
let's get talked about the data science
Bowl amount so this one is a little
different
there's a million dollars of prizes and
this one also just has different prizes
so basically you've got you know first
place is five hundred thousand all the
way down to tenth place
you also get five thousand for the top
three most highly voted kernels I'm
going to be submitting this as a kernel
so if you found it useful please do give
me your vote there's also other kernels
that you definitely will want to check
out just like kernels and discussions
same thing I said before go to those and
look through them you're going to find
really useful information in both of
them that people just kind of share
freely and then also ten thousand
dollars in basically sharing the stuff
basically those are just announced as
time goes on right now it's a social
media thing so if you use or hashtags
you're just automatically entered okay
so that's that
let's see I'll get to the data last you
can read the description just kind of
understand what's going on but basically
we're looking at CT scans calf gains for
these low-dose CT scans to basically see
if someone has a cancerous tumor or not
so basically we're looking through this
data that is their CT scans but there's
many of them so basically the CT scans
actually stack on top of each other and
actually create a 3d rendering of
whatever we scan in this case the chest
cavity so that's the data that we have
but we haven't quite yet seen it yet so
we'll get there again the evaluation is
log loss
you can come here for the resources
there's really only two resources there
there's the time line this one's a
little different there's two stages so
the first stage you'll have you do have
testing data that we don't quite yet
have the answer to and then we also
finally will have right so you you'll
you'll create your first submission on
the data that we have right now and then
you've got the test that will be
released after the deadline and the
answer to the validation set which you
have right now are released and then
you'll have the test set so you're
basically you've got to compete in both
stages to be eligible but these two
stages kind of make it probably even
harder to cheat and in the actual
leaderboard now even if you even if you
do cheat the leaderboard you would not
win the competition because you do have
to share how you did it so so you would
be found out and you would lose your
number one position anyways but it can
be demoralizing to other people if if
someone it's just you know way ahead of
everyone else or if a bunch of people
are cheating it'll give you it you know
it's just a wall I think so you think
you didn't win or whatever so I think
we're ready to rumble
that's just my you know quick
introduction to to kaggle what I'm going
to leave you off with is getting the
data there's a bunch of information here
basically you've got the actual data
which you need a password to open the
passwords contained in this file you
need the sample images you need stage 1
the stage 1 labels and just this is a
sample submission so what I found to be
best was to download the torrent because
it just downloaded so much quicker but
you it might not for you but it is 67
gigabytes
it's a big file so and then by the time
you extract everything it's about 140
gigabytes so that's a very long download
so that's going to take a while so go
ahead and get that started if you don't
have space for that data that's also
totally fine I'm going to do my best to
structure this in such a way that you
can follow along
just if you go to Colonels and go new
notebook right here you should be able
to follow along with me through this
entire tutorial your model is not going
to be good but because you're going to
be working with a really small data set
if you do it this way but you can still
follow along and still learn a bunch so
anyway if you don't have space for it or
you don't have a computer that's going
to be happy to process that much
information you can feel free to just
follow through this notebook here you'll
only have to change like two things
basically as we as we go well more than
two things but like like five things or
something not much okay so that's what
this series is going to be all about get
that data downloaded whenever you're
ready I'll see you in the next video