PyTorch Lecture 08: PyTorch DataLoader Video Lecture | Pytorch: A Complete Guide - AI & ML

in this lecture we are going to talk
about data loader which is very powerful
tool to manage our own datasets in our
previous lectures we used this manual
data feed
for example we read all the data from a
file and DVD by them as X data and Y
data and in our chain roof while we did
is they we feed all data to the model
and then using this output will compute
the loss and they will compute the
gradients and they we update our weight
in this example feeding all data
manually is not a big deal because the
data size is small and something 700 but
if you have to deal with thousands or
millions of data set we cannot fit all
of them to the model and then we cannot
compute the gradients for all data
points this is not efficient so usually
what we do is that we divide the entire
data set into small batches with this
beta size and we go through each batch
at once and they recompute the gradient
and then we update our weight more
formally we can define epic and
pesticides in iterations one epic is
that if you can cover entire data that
is one Apple but in order to cover
entire data we have to go through many
batches so each batch covering one batch
is one iteration so for example we have
1,000 training example and then our
backside is 500 then we have a two
batches so we it will take two
iterations to cover one tempo so this is
definition of epic batch exercise and
iterations then how can i implement is
how we can upload our batches for our
training data set so it is possible to
divide all the data set to small batches
and managing these but however if you
can just use data loader from titres we
don't have to worry about the entire
process just this can be deal with the
data border so we don't have to worry
about it in our perspective what we need
is just we want to get iterable data and
then from this iterable data we just get
Edge's for example here let's say this
train loader is one iterable which is
get from data order then we just put
this one in the fall of and each loaf
what we do is that we get the batch data
here and then we divide them as input
and labels and they will wrap with
variables and we can use these inputs
and labels for our training cycle and
the making this custom data order for
our own dataset is extremely simple so
we just need to provide only two things
one is get item for given index which
has written our item and then also here
we need to provide what is the size of
the data and that's it in order to do
that we probably need three steps first
step in in its we need to download and
read error and so on we do some process
in here and then we return one item for
given index and then also we return the
Delta X and then this we can make as a
separate class which extends data set
class so for example in our diabetes
data sets in in its what we need is that
we just bleed this data and then load
into x and y and then we compute this
length using and then we turn our numpy
array to torch tensor and this is the
commenced or start from the numpy and
they will save them as X era and Y data
this is our initialization for other
data and then Kate item basically for
given index we return this item an X
data and then Y data and the lengths we
just written our lengths and then how
can I use this one in our program so we
just call class make instance of the
data diabetes data sets like this data
sets and then we feed this one to the
data loader and then we can decide what
is the exercise and then sapling is
extremely useful when you train the
models we shampoo our data and then if
you if you want to use this loader for
more
two processes we need the number of
workers this is our entire
implementation this spot is to our
custom data loader and then we create
instance for that and the reef is there
to take the rotor and then we create our
training rotor and this train order is
just used in our training roof so here
inside the fall off we just use them and
each iteration will get a one batch so
we divide them as input and labels and
we use them for our training so if you
want we can divide this one as a
separate file and all the other models
that are using this data set can just
import this file and class and use them
without knowing all the details in
similar manner in the title chase you
provide a lot of data set loaders
already implemented for minused cocoa
and many other famous benchmark sets if
we want to use this immunised data set
for your model you don't have to
download this manually or reduce image
files by yourself we can just use
implemented data set same list and then
give them which directory we want to use
and then all the rest will be taken care
of is implementation and then once we
have this data set and they refill this
one to the data loader here and we can
decide what will be the page size that
you want to use until you want to
shuffle and then once you have this data
training order we can just use them
inside of the roof so each iteration
basically we can get one batch size data
and target then we can use them in our
training cycle and once again there are
many data sets available here so you can
play around and see how easy to use this
data set using Python as an exercise you
can check out this existing data set and
also you can fill your own data set with
a very very simple implementation so as
an exercise you can try to build
Titanic data set for your own data order
and then using this data order you can
simple classifier in our last lecture
we're gonna talk about soft max
classifier