New Tutorial: Add Facial Expressions to your Game Characters! Video Lecture | How to make Video Games: Game Editor Tutorial - Game Development

I have some really cool stuff to show
you on how to do facial expressions in
your game our game is getting to the
point where we need to put the
characters in and we don't have enough
time for facial animations because you
know it takes a lot of time and it's
finicky and so we're looking for
software that can help it solve the pain
out of that and make it really easy to
capture facial expressions so there are
a bunch of software out there that can
do that
and the best one for us we found is
called face shift you can download a
trial version on phase-shift comm which
is the one I'm using to show you how it
works and how to get facial expressions
going inside of your game in just a few
minutes
other than the prep work of course now
here's the deal the reason why we chose
phase shift is because it allows us to
track our own face with the depth sensor
like a Microsoft Kinect or other depth
sensors just connect it to your computer
you can get those for 25 bucks in a used
store and connect those to your computer
rather than other tools that only use
the webcam and they don't have any depth
information so the the tracking is not
as accurate then we put those that
tracking information interpolate that on
characters inside of our game and boom
there you have it
we just controlled those characters with
our face so let's take a quick peek on
how this works exactly when you download
phase shift and open it up the software
first asks you to set up your camera now
in my case I've set up my Kinect it's
connected and I putting much plugged in
and all I have to do is download some
drivers as you can see it looks at my
face in a DEP perspective and then it
can add a little point cloud of 3d
information that it sees for my face so
as you can see here which makes it super
super accurate and super awesome so
behind me is a window so I'm just
pulling the cutoff in here so that it
takes the wind nail out and just leaves
the depth information of me sitting in
front of the camera so next I'm going to
go over here to training and let me just
start a scratch here and it's looking at
my face with the webcam inside of the
Kinect and it is looking at my face with
a 3d sensor and then asked me to train
this 3d character guy here in the middle
with all 48 facial expressions that it
supports and so
I'm going to do that right now and it
can take a little bit of time but all I
have to do is hit the captured key and
do exactly what the 3d character does so
in this case it wants me to rotate my
head in a normal expression and while I
do that you can see it is creating 3d
information of my head which is going to
be linked with this character a little
bit later on so now wants me to open my
mouth and fix that then it wants me to
smile and so from there on it wants me
to go through 48 different expressions
which I'll do real quick but I'll save
you some time and fast-forward this
excellent now that I've recorded all of
them I'm going to hit build and what
it's going to do now it's going to
create a 3d guy inside of the 3d
information that it's recorded from the
Kinect and after it's confirmed and
asked me if is this your eye and then
yes I can position it correctly which it
already has and hit OK and then with
that 3d information is going to reshape
this default character to be the shape
of my face and now you will never know
how in symmetrical or asymmetrical your
face is until you've seen this guy being
completed but I trust me it's accurate
now in my case by the end you'll see
that my eyes are kind of off and my nose
is crooked
which is correct you know if I were to
shave my head off or I mean my hair off
then I would look exactly like this
character and here when it's done so
don't get insecure now this is
completely normal so and only a million
people will see this video so I'm just
going to sit here in the corner and cry
my heart out what happens is eventually
I will control this character real time
with my face and then interpolate what
it's doing on top of my 3d characters so
that is really really cool
now what we need to do in order to get
that done is we need to have our own
character and phase-shift comes with a
few of its own so you can try things out
but if you have your own character what
you do is you make those 48 expressions
inside of your 3d modeling application
and those are called blend shapes some
other programs like
my three shoe marks column a little bit
differently they call them morph targets
but all you have to do is you know pick
your model of choice and Google either
blend shapes or more target to find out
how to do it but what you do is you make
those 48 shapes now this is not hard to
do and I'll put some links in the
description but it's just a little bit
time-consuming all done my tracking
profile has been created and I'm going
to hit save here just to save this so
let me just call this Ephrem tutorial
okay save that out so that way we don't
have to Train it again next time I we we
sit here notice that it did not have any
problem with my glasses with my beard or
the microphone I'm wearing which is very
impressive so next I go to tracking and
as you can see there's a default
character loaded and it is already
pretty awesome a little bit shaky here
and there but I mean with the refining
that it's going to be pretty awesome
awesome your okay cool so what I can do
is I can click display here and then
switch which character I want so if I
want to be a pug instead I can be a pug
I mean isn't that cute looks maybe
better than me but I'm going to go and
keep the orc because it reminds me of my
twin brother anyway so what I can do is
I can start recording a take and that's
what we'll do next so I'm going to
pretend to be the orc for a second oh
there's the five me haha okay I'm sure
that scared you there for a second and I
didn't mean to do that but trust me it's
still me so what I can do is I can scrub
through the animation now and it has
each frame saved so it can refine
eventually through the frames because it
knows the future it interpolates between
them and fixes things now first let's
zoom in here and see what's happening as
you can see it's tracking my mouth and
my eyes and if I want if some point the
eyes to be more open wider I could just
pull these things out and it will fix
that for me if I want my mouth to be
closed like right here all I have to do
is hit the C key and it closes the mouth
for me
and that makes it that makes a refining
process really fun to do I mean usually
facial expressions are only done by you
know senior artists but in this case if
it's only refining animations you could
even you know have interns or volunteers
at your studio refine the facial
recordings that come back from your from
your voice-over artist and you know go
from there so as you can see there was a
little blip going on here so I'm going
to just push this a little bit
straighter and I can go in here as
detailed as I want which I don't want
because I don't have the time because we
make games and you probably don't have
either so what I'm going to hit and do
now is I'm going to hit refine and it's
going to go through this frame by frame
back and forth an interplay between the
frames and refine the animation and once
I start playing that back once it's done
you will notice that it is even way more
accurate than it was before
alright there it is so now when I play
it back notice how how much smoother the
character is this time there's the five
me ha ha perfect
so one thing I don't really care about
is the the eyes it tracks the ice pretty
well it's pretty accurate but I always
take the eyes out inside of the game
engine and then make the eyes look at
whatever the character is looking at in
code so I don't care about pupil
tracking as much but you could now I can
click on each of the blend shapes here
and see how it's performing over time
which is great allows us to see which
which they you know shapes it has chosen
so now with that done we can go in and
refine things even more you know we can
we can change settings about the eyes we
can change you know how it refines the
takes all those kinds of things you
could do as many things you like and
refine them all overnight
very powerful features in here but I'll
let you explore that yourself so if you
wanted to load your own character that's
pretty easy your display and then you go
to import
and then you load your character in here
and it will load it here on the right
and then you can match each blend shape
with phase shift name and your name now
if you're smart you set up those blend
shapes with the same name as what as
what phase shift calls them and then
once it's mapped you just hit save
mapping and go back into your game so
that's that now we need to export it so
we can export it as an FBX we can export
it as FS data so I'm going to use the
Unity import for Unity game engine so
I'm going to export the FS binary so let
me just put that here I'm going to call
this tutorial test org for example
awesome okay I can additionally export
the audio wave as well and video and
JPEGs but I'm fine for now so I'm just
going to actually I'm just going to
export the audio as well so it's safe
it's safe as well and that should be
exporting out so next step is adding
this in unity all right now I'm in unity
and have done two things one I've
imported my own character that I have
created with blend shapes and two I have
imported the phase shift script that you
can download from their website I'm
using a slightly newer version that
isn't out yet because the guys from
phase shift are really awesome and
responsive over email so they've sent me
a new version and then it will be out
soon so what I did is I going to my my
character here and I drop that into the
scene I'm going to make him a little bit
bigger so we can see him better and this
character as you can see has all the
blend shapes here on the side of course
I would not import those if this was an
actual game but I just wanted to show
you the different facial expressions
that you can put in so the face is only
going to be as exaggerated as you make
the blend shapes so let's go back to the
main character here and now what I need
to do is I'm going to select that
character and he already has an
animation controller on it so I can use
Beckenham to animate his body and his
mechanism animations will not conflict
with blend shapes because blend shapes
are something completely different which
allows us to trigger different facial
expressions while his animations are
doing different things at the same time
so he can talk while he's walking those
kinds of things pretty cool so then I go
into my phase shift folder into the
script folder and it has a phase shift
script and I can drag that on there and
this allows me to do a whole bunch of
stuff now first I'm going to load this
phase shift clip this is so the binary
file that we saved inside of phase shift
and load that work recording that we did
so this is going to be in documents and
phase shift and performances and there
is the one we just did so I'm going to
open that up and now I need to make sure
that my blend shape makes sense with the
blend shapes of the of the phase shift
default engine so I'm going to click add
blend shape mapping and it asked me to
go through these so the eye blink left
is automatically synced with the eye
blink left on my character because I've
used the same naming convention that
this makes it very easy to map them now
I've already done this mapping before so
I'm not going to go through it instead
I'm just going to go and say import
phase shift retargeting and this is
again in my documents
phase shift and then targets and then
I've saved my retargeting here now as
soon as I load this one it's going to
ask me hey are you sure you want to do
this because this is a different source
rig which usually is not a good idea you
know if I had more time I would set it
all up inside of phase shift and use the
same character and I wouldn't get this
error so there were there will be some
inconsistencies that you see at the
ending animation but for demonstration
purposes I think it's fine and as you
can see everything is mapped now if I go
from 0 to 100 that's pretty much the
range of a blend shape so for 0 the for
example for mouth open then the 0 it's
closed 100 is entirely opened so if
you've noticed that in your modeling
program you've made some blend shapes
too exaggerated you can just put the
maximum - for example 50 and then it
won't you know exaggerate them as much
or if your eyes are closed too much you
can use you know
a larger number there so that's what I'm
going to do now it that I've loaded the
face shift clip and I have loaded to
blend shapes I just have to click attach
as an animation to the object and ask me
where do I want to save this animation
file this is just the unity nm file
which is great so I'm going to do orc
animation save that and boom now what
it's done if I close this script here
it's added another animation component
which is not to be confused with the
animator component and has added the
animation in there and this is purely a
blend shape animation so I could go into
my animation here and actually play that
off but it's already set to play
automatically so as soon as I go into
play mode let me make this window here
just a tad bigger so you can see it it
will start and play the animation that
I've just recorded and there we have it
this is facial animations in game now of
course if I wanted to add the wave file
to it all I have to do is add an audio
source and audio clip here and it will
play from scratch at the same time as
that the animation starts playing I
highly recommend though not triggering
audio files and animations at the same
time from the start because they could
get out of sync instead I recommend
writing a simple script that triggers
the animation as well as the audio file
at the same time and then you'll have
lot less issues inside of your game so
there is another component too as well
to phase shift that you can real-time
change things inside of unity all you
have to do is drag the phase shift live
script in here and this will you know
sync back to phase shift that you have
opened if you put the right variables in
here so that is the live the component
if for people that are interested in
that for me it's much more you know of a
compiled game so it's not for us at the
moment but I highly recommend this this
plugin now I have to add to this that I
really love the company we've been
following phase shift for about three
years when they were first at the Game
Developers Conference if you want to
meet them and see the new plugin
life in action and unity they have a
booth at GDC booth number 25 16 for
phase shift really really cool guys
awesome to work with awesome to talk to
their company has exploded over you know
the last three years that we followed
them and the high-end guys are using
them so it's I really highly recommend
looking them up so thanks for watching
this tutorial and I hope you can make
some facial expressions inside a unity
add a video response to this video so we
can see what you made that would be
really cool and don't forget to
subscribe and also get our newsletter
off of tornado twins calm talk to you
later bye bye