Running work on GPU (Cuda) (how-to) Video Lecture - Machine Learning with

hello explorers and welcome to another
video today I'm gonna talk a little bit
about running workloads on GPUs so today
when you are trying to do things that
requires a lot of computation people
usually either have a lot of CPUs or
they use TPUs or they can run a lot of
the workloads on GPUs and GPUs are
really good when you are trying to do
things in either 2d context or a 3d
context because when you're trying to do
that those kind of workloads GPUs are
very efficient because they have already
been optimized to do 2d and 3d graphics
so therefore they are really good to do
matrix manipulations so in order to
start this off I'm gonna use a library
called JCuda could be fetched by
this dependency here and JCuda uses a
library from Nvidia that is called CUDA
and CUDA is a workload library that you
can download from nvidia install on your
machine and windows you just install the
binary package so you download it and
just install it and you're done and
Linux you need to have it installed you
need to have the correct Nvidia drivers
and you also need to have kernel module
called nvidia-uvm so that's a driver
module that you need to have installed
on your system and windows who usually
get these drivers with your standard
Nvidia drivers so if you have an Nvidia
graphics card and you can run this then
you could use CUDA and on the cuda
website you also have a grading for your
graphics card in order to know what kind
of workload you can put
on it you could always build things that
could you be done on all graphics cards
but if the more specific you can be and
the more you can tailor your
implementation to your graphics card the
more performance you will get out of it
so you know when we have put in this
little dependency here we start by
running this very simple hello example
where we create a pointer we actually
take this pointer and put that in the
CUDA so it will actually just create a
pointer on the graphics card and then
free that memory again this is a very
simple example just to see that
everything is up and running and if we
run this it should tell us that we have
a pointer somewhere in native memory and
the interesting thing by this little
hello example is even though you haven't
installed your graphics driver correct
this could actually work you could get a
native pointer to your normal
system so this never failed for me and I
thought that okay everything is set up
and when I ran the correct test I wasn't
running it on my GPU so that was a bit
strange so let's go into something
more interesting let's do some actual
work and we have a little example here
where we actually use a kernel and this
is what we have here this PTX file and
first off to run a cuda example you need
to set that you actually want exceptions
to be thrown from your driver so
everything that is bad in your driver
will throw an exception so that's good
for error handling and so you see what
happens and I'm not sure if it has
performance disadvantage if it goes
faster if you turn it off I have it on
because it goes really fast and either
way then you will init
this flag zero it's what state you want
to set your graphics card into let me
create a device cuda a device and you
get that specific device from your
system and you also create a context and
these are used later and then you will
load this module this is the actual file
this PTX file we talked about earlier
and from that PTX file we will fetch a
function and in this case add so if I
look at my kernel here I this PTX file
is a module and in this module we have
the function add it's a very simple C
like function where I have two floats I
put in and I get summation out and it
also have this in the N value to
actually block that we don't run this
out of the actual space of our array so
these floats here all of these are
arrays with a specific length and this n
tells the length of those arrays so we
don't pass at the end of them
because these calculations where you
have a block of workload and then you
say that this index of this block I want
to work on and this is the dimension of
the block so this is the size of the
blocks and then the current thread I'm
working on and this calculation isn't
always even so in some cases you can
actually put some work on one of your
GPUs cores and it actually falls
outside of your actual workload and then
it should just not do anything so you
don't get any errors because of that and
when we have this file and want to run
it as a PTX file we actually need to
build it so we we can only run PTX or Cubin files and Cubin
files are actually cuda binary files
PTX files or something that looks like
assembler but will be loaded directly
into your memory card so to create PTX
file we can run nvcc and in order to
run nvcc on Windows you need to have it
in your path so the CUDA libraries need
to be in your path so you can run nvcc
because that's a cuda tool and you also
need to have the c linker in your path
and in order to do that on a Windows
system you have visual studio
installed and you will run varargsall
on that bat file in your visual C
directory and that will actually give
you all the path so you can find that
specific exe file an nvcc can take a
few different parameters to it
one of it which is PTX so it will
create a PTX file and then you can
actually just run it with the current
file that you want to compile and then
you get the warning that the SM_20, 21
archetypes are deprecated and I know
that my card is actually in SM 61 and I
know that I run on a 64-bit system so I
could specify this here but when you
create a PTX file it's not actually
interesting because a PTX file is just
some kind of assembler and to optimize
it isn't that super interesting when you
come to binary files it's very
interesting but to do this we put in M
as a machine to say that this is a 64
bits machine and then we can say that it
has an arch of SM
_ 61 because that's my
capabilities of my graphics cards they
have a table with the graphic cards
capabilities and mine are 6.1 I think
there's a 6.2 also if you get an even
newer graphics card and after we have
done that so now we have the file we
have loaded this function we set up that
we want to run through 100,000 elements
and we create some inputs and then we
actually just go through and create
numbers from 1 to 100 thousands as
floats in a huge array and then we will
create a device pointer and this device
pointer input a we will take that device
pointer and actually allocate a buffer
of that size in floats on that device
input and then we copy our data from
this host input that we created up
earlier and over to this device memories
now we have actually copied the memory
into the GPU so we can use it later we
do the same for the input B and then we
will also allocate and result buffer on
the GPU for the device output and then
we just say that we to our kernel we had
four inputs we had the length we had the
A B and the sum so these are the
device input A device input B and the
output and the first one was the actual
number of elements so now we have
created all the parameters to our
functions now we actually want to launch
this
and in order to do that we will give the
GPU a block size it will work on and in
this case we say that every block that
every GPU processing unit will get a
block of 256 and then we create a grid
with blocks of that size in our work so
by creating a block and then a grid of
those blocks we could actually send say
to the different cells in our graphic
card that they will work on different
data units and that's what we
done here will launch a kernel we say
how the the size of the grid size is and
you see here that we have actually grid
dimensions in X Y and Z so we could
work in it 3d space and then we have our
block size also in X Y and Z and then
we tell it how much shared memory they
should have in between so they can
actually talk with each other I'm not
using that at all with my examples and
then we put the kernel parameters in and
those are sent to our function and often
we have run this kernel we wouldn't need
to synchronize there everything shuts
down and that we know that all the work
has been processed and then we copy the
result of the outputs pointer to an
output buffer here so we have the host
output and then we will go through and
actually check the result if we actually
got the result of the two inputs added
together as our result so it goes
through here and checks that it doesn't
have any failures on the graphics card
and then prints that hey the test has
passed or failed and then we need to
free our memory so we don't have any
unfreed (allocated) memory on our graphics card
so let's run this example as well and
test pass and as you see that went
really fast and so they this is the
first example just adding a lot of
numbers together on our GPU but I wanted
to do something a little bit more
interesting so I wanted to do some image
manipulation and in order to see some
kind of comparison it's not that this
comparison will be total it will not
show off exactly how much better it is
you just will get a feel for it so we
don't take any of these numbers for
granted but I have a little filter
example here that you actually make a
blur of an image so this is a dilute or
blur filter and I say that you have a
specific filter size and then this is my
little apply filter function it will go
through and this filter and apply it to
an image it's not that important how I
do this and but it's pretty much what it
does this is the for loop that is a
little bit weird I apply the filter 1
million times because doing this either
way will go ridiculously fast so I had
to actually slow it down in order to see
some differences and then I run my
filter here across the whole image and
this is my main function I take some
test image in I make a grayscale of the
image I printed the before image I get
the pixels of the grayscale image and
then I will run my filter get the result
and then put it back to the actual pixel
set and write the pixel out the pixels
out to a result so let's look at my test
image this is a nice little image of
some landscape and if we run my java
filter I will first off get a grayscale
image let's see with this the before
image it's quite sharp and then after my
dilute filter we can see that there are
some changes some of the fencepost a
little bit more blurry not that huge of
a difference then the interesting part
is here we have a time of 95 so it takes
actually some time to run this we run it
a few times 100 110 97 and so on so and
250 it it actually takes some time to
run this and if we go to running this on
the GPU then then I have pretty much the
same as we had before I will get a
different kernel and down here I will
create my grayscale image still not that
important the implementation details I
have everything on my github repository
if you want to look into it more later
it's a Java cuda example you can find it
on my github I will also put the link
into the video description and I get the
pixels I create a pointer on my graphics
card with the size of the image and I
copy all the pixels over there and then
I will also copy over the filter to the
second argument and then create an
output so I can get the result back and
here we have my kernel parameters width
height pixels filters and output then I
will create a kernel block with block
size 32 times 32 so this is actually a
size
how many large other block I want to
give to the the graphics card and this
is dependent on how much memory you
have on the graphics card actually the
before we had 126 and this 32
times 32 is even larger but I'm doing it
in a grid of a 2d dimension and then I
will create a grid size and I would
supply all those with my kernel
parameters and synchronize then just
copy back the result and write my
grayscale image to the result
PNG here and then free up my memory but
we saw here that I will use the kernel
called
Cubin and to actually compile that we
can use the exact same command we used
earlier the only difference is that
instead of PTX
we say cubin and in this case it's very
important that you have an architecture
and a machine because this will actually
be very much more compromised so we
cannot actually look into the assembly
that we run earlier so this is some kind
of assembly like language that we can
read for the earlier example but this
cubin file is binary so we can't read
that
maybe we can decipher it and read it but
it's not humanly readable from just
looking at the file so if we run this
CUDA filter and another thing to also
know here that I start my counting
before I copy my values to the graphics
card so before I copy an in memory I
start the clock and then I read after I
copied the data back from the GPU so if
we run this cuda filter it takes 13
millisecond
instead of 100 so that's a lot and just
for fun we could move this so we only
look at how much time the calculation on
the graphics card takes so no copying
just the actual execution and that takes
one millisecond so the most of the time
is actually spent copying data to the
graphics card the actual processing
takes no time at all
I wonder if I can actually push this up
to 64 because I have a little bit more
memory on this machine not invalid value
and so it actually gets too large if you
push these sizes too large then you
won't fit it into memory when you work
with it
and so let's look at the cuda result and
it is a bit blurred so it's the exact
same image nothing different here so the
Java and cuda example was exactly the
same we can also look into the cuda
test kernel we have it here we have the
width height image filter result and
here you see that I run my full filter
over the image and then get the result
back so it's pretty much the same as we
did before and yeah so there's a lot of
work being done here and I wonder
did we have the loop for the extra
100,000 wasn't this here it should be
then it's won't be any good comparison
if I have missed this little part that's
a lot of more loops I don't want to fool
any one yeah this was inside of the
apply filter and let's compile
again just to check that we don't do
anything stupid here now it's really
slow hmm that's interesting so it's
actually slow to run on the GPU that's
not what I thought maybe we can see if
it's as slow we run it like this
nope much slower so let's see if if we
remove this from this kernel again just
to be sure that we are looking at the
same result
yeah we need to compile it again
13 and then we run the Java filter
without any extra things 43 so there is
actually a difference it's not that
large of a difference but there is a
difference so may not be that nice if we
run run 1 million on on the java version
or the the actual data on in the kernel
because the kernel will be run over a
lot of things simultaneously but this
will be run inside of the function so
there is a difference there but at least
we can see that they are quicker if we
run them like this and then I wanted to
do one more test because I thought that
maybe I could get it to run fast on the
CPU as well just running a lot of them
on the CPU so I created this little
runner here and that applies the filter
over a set of the image and this runner
gets a part start Y and an end height so
this is actually an end
y and that I so start at one point and
then go to another point I will do some
of the lines but not not all of them and
in my Java filter multi here I have the
same kind of code getting the image and
so on then I will submit a runner that
does a
tenth of the image so a part of the
image - I split up the image in ten
parts and then I run this fixed pool
with ten threads and the reason I took
ten is that I believe my computer
has 12 cores
and as I'm recording and so on I wanted
to use a lot of them CPUs but not all of
them and maybe the recording will need
even more than that I haven't tried it
and then I create this completion
service so I actually will know when it
is completed and a completion service if
we submit some work to this completion
service we can actually take the result
from the completion service and when
it's done it will return so if I take a
value it will be waiting there until it
actually got a result and it will take
the first result it actually gets from
the pool and I when I have gotten ten
result it will shut down the pool and
end it all and right out the time how
much time it took to run this in a
multi-layered settings and it actually
takes 55 milliseconds to run it on 95 so
it's as slow and even perhaps a little
bit slower than running it linearly and
also tried it when I wasn't
recording and I saw the same result so
running on the CUDA graphics driver is
much faster than running it on either
linearly or over multiple cores and
because it's such a small example I
believe it's hard to make it faster just
by creating and doing it on multiple
core
because there is an overhead to actually
start up the threads in the thread pool
and then shutting them down and so on so
it's hard to when you're not having a
long-running process it's hard to
actually make it go faster by just
throwing more threads on it and but you
can run quite small things on GPU and do
it really fast and so yeah that was what
I wanted to talk about today this is how
you set up something and run it on your
GPU you can always extend this and run
in three dimensions if you want to but
much of the workload you do today you do
in two dimensions even when you are
going to machine learning you usually do
that in with a filter over a
two-dimensional matrix and so I hope
that you learned something today I hope
you found this interesting if you have
any comments please leave them down
below like the video and share it with
your colleagues your friends and I
really hope to see you in the next one