Essential Tools for Machine Learning - MATLAB Video Video Lecture | Introduction to Programming with MATLAB - IT & Software

hello and welcome to this webinar on
machine learning my name is Shanna Patil
and I'm a technical product manager of
networks working on products in the area
of statistics machine learning and deep
learning practice earning math works I
was working on developing algorithms for
human health monitoring by applying
signal processing and machine learning
techniques to data gathered using
variable sensors as the title suggests
in this webinar we will talk about some
essential tools that will enable you to
quickly develop and deploy robust
machine learning models using MATLAB
machine learning has become quite
pervasive today here are some examples
of applications for machine learning as
enabled performance that would have been
very difficult to achieve the
traditional modeling techniques let's
take a look at the characteristics of
these problems that make them such a
good fit for machine learning techniques
problems such as speech recognition
object recognition and engine health
monitoring are too complex for
handwritten rules or equations machine
learning algorithms are designed to
learn such complex nonlinear
relationships weather forecasting energy
load forecasting and stock market
predictions are examples of applications
where the underlying system is
constantly changing so the solution also
needs to change and adapt machine
learning algorithms can quickly learn
from new data and thus keep up with such
dynamic systems applications like IOT
analytics taxi availability analysis and
airline slide dealer predictions
typically involve learning from large
amounts of data machine learning
algorithms are designed to learn
efficiently from such large data sets in
this webinar we will focus on one such
application we are applying machine
learning to develop a solution is a good
fit using a real-world data set we will
use the machine learning workflow to go
from raw sensor data to real-time
classification and hard sounds as shown
in this video along the way we will
identify several essential tools that
enable us to quick
develop and deploy robust machine
learning solutions here is the agenda
for this webinar we will start with a
quick overview of machine learning and
talk about typical elements of machine
learning workflow we will then work to
an example in which we will develop a
solution based on machine learning
techniques to classify heart sounds
finally we will review some of the key
challenges and talk about how tools
available in MATLAB help address them in
simple terms machine learning uses data
and produces a program to perform a task
this ability is especially useful for
complex problems the closed form
solutions might be too difficult or even
impossible to derive let's take the
example of human activity detection
using a smart phone our goal here is to
use the internal tri-axial accelerometer
to determine when the person is
performing activities like sitting
standing and walking if it is the
traditional approach we might be able to
come up with some handwritten rules
based on empirical observations or we
might try to develop a formula in either
case the solution will be suboptimal as
the problem is quite complex and the
solution space has many combinatorial
possibilities on the other hand a
machine learning solution would involve
collection of large amounts of sensor
data with corresponding activity levels
and then allowing the algorithm to learn
that these complex nonlinear
relationships between inputs and outputs
depending on the complexity of the
problem we can choose to use a simple
algorithm for example an IU base or go
with something more complex like support
vector machines our task here is to
select the most appropriate technique
and avoid pitfalls such as overfitting
machine learning can be broadly divided
into supervised and unsupervised
learning in supervised learning the data
set includes input values as well as
corresponding outputs and our task here
is to train a model to predict or
estimate the output on new data if the
output has discrete values we refer to
it as a classification problem whereas
if the output is continuous we refer to
it as a regression problem for example
if you are trying to predict tumor size
into small medium or large then it is a
classification problem whereas if you
are trying to predict electricity demand
from a grid say in kilowatt hours then
it is a regression problem in case of
unsupervised learning our task is to
group the data based on some measure of
similarity input data set does not have
corresponding output or labels so many
different solutions are possible and our
goal here is to discover naturally
occurring patterns in the data
clustering techniques fall under the
category of unsupervised learning an
example of unsupervised learning is a
problem of determining optimal placement
of antennas so that all cellular
customers receive good service here are
some examples of techniques under each
of these categories support vector
machines and discriminant analysis are
examples of classification techniques
linear regression and regression trees
are examples of regression techniques
whereas k-means and Gaussian mistreat
mixture models are examples of
clustering techniques
let's talk about the steps involved in a
typical machine learning workflow
accessing and exploring the data is
almost always the first step this step
involves reading data from different
sources like various file formats
databases or even streaming data from
sensors the first step is usually
followed by the pre-processing step in
this step raw data is manipulated and
transformed for consumption than machine
learning algorithms this involves
cleaning of messy data example outliers
or missing values extracting features or
predictors by using domain knowledge for
example image features and applying
various data reduction and
transformation techniques to keep only
the most relevant information once the
data is pre processed it is ready for
the third step this step involves the
development of predictive models in this
step we have a lot of flexibility with
respect to the learning techniques we
want to use as well as the greatest
parameters of that technique typical
tasks include training and comparing
multiple models optimizing model
parameters and validating model
performance to ensure robustness the
final step is to share and deploy train
models by integrating them into analytic
pipelines models may run on embedded
systems shared as standalone
applications or even deploy on the cloud
the goal here is to rapidly build and
deploy accurate models
now let's work through an example that
will help us identify some of the
pitfalls in this workflow as well as
tools that can enable us to deal with
them effectively this example is based
on hard sums data set that was used in
the 2016 fishnet and computing and
cardiology challenge five sounds contain
two edge information that can enable
early diagnosis of serious pathologies
of the cardiac system because listening
to heart sounds requires a simple device
it is an ideal screening tool that can
be used in the field to identify and
defer at-risk individuals for further
testing however the heart sound signal
is highly complex and reliable diagnosis
often requires a trained clinician in
this example our objective is to develop
an algorithm that can be used for
automatic classification of heart sounds
with the goal of deploying the final
solution on an embedded system for
real-time diagnosis the data set
consists of a training and validation
set the training set set consists of
3,240 recordings and the validation set
consists of about 301 recordings the
recordings are very variable ends
ranging from 5 seconds to more than 120
seconds each recording comes with a
label of either a normal or abnormal in
the heart sounds example we will focus
on a subset of elements from the
workflow that we reviewed earlier our
data set consists of files with
recordings of sensor data in the
pre-processing step we will focus on
feature extraction and feature selection
to identify the most relevant features
to develop our predictive model we will
train and compare multiple models
perform parameter optimization and
validate our model on new data finally
we will take our machine learning
solution and prepare it for deployment
to an embedded system throughout this
example we will use MATLAB as a
programming language in the
pre-processing step we will rely on the
capabilities provided by the signal
processing
box and the wavelet tool box for feature
extraction and the statistics on machine
learning to the box for feature
selection we will use apps and
algorithms from the statistics and
machine learning toolbox for developing
predictive models and use the MATLAB
coder to generate C code from MATLAB
code for deployment we will use the live
editor to work through this example for
those of you who are not familiar with
this feature live editor is an
interactive way to write code and
generate computational narratives that
are easy to share and communicate in the
hard science data set each recording is
an audio file with a normal or abnormal
label let's start by plotting and
listening to what an abnormal heart
sounds like MATLAB provides tools like
audio read and audio player which make
it easy to work with such audio files
now let's listen to what a normal heart
sounds like okay that was a bit quick so
let's let's play them again so here is
the abnormal heart and here is a normal
heart while they are not experts we
certainly notice some differences the
abnormal heart seems to have higher
frequencies and a noise like quality in
between beats the normal heart on the
other hand is more regular with silence
between beats now that we have some idea
of what the signals look like and sound
like let's take a look at the signal in
the frequency domain the signal
processing toolbox why is an app called
the signal analyzer that enables us to
do this without writing any code so
let's open up the app
in the signal analyzer app we can take a
look at the two signals side-by-side so
on the left side we have the normal
heart sound signal and on the right side
we have that normal heart sound signal
we can clearly notice some differences
as I saw earlier in the live editor now
we can take a look at the the power
spectrum for both of these these signals
we can immediately notice that there is
there is a presence of some higher
frequencies in the case of that normal
heart sounds we can also use the banner
feature which allows us to take a look
at a short segment of the signal at a
time and essentially kind of go across
the signal and see what changes if there
are any changes across the power
spectrum as we as we pan across the
signal again we kind of observed this
presence of some higher frequencies for
for that normal heart sound signal so
now that we have some idea of what the
signal looks like in the frequency
domain let's go back to the live editor
and work on one feature extraction next
let's start preparing for the
pre-processing step
let's take a look at how the files are
organized for this data set the data set
that we are using in this example
includes more than three thousand files
spread across multiple folders the two
main folders in this data set are
training and validation under the
training folder we have multiple folders
from A to F each of which contains
several hundred files corresponding to
two individual hard sound recordings
this folder also contains the reference
file which includes the labels
associated with each of these recordings
so rather than writing code to navigate
through these folders for accessing the
data files we will create a file data
store object
a-five datastore creates a collection of
files and uses a read function to load
the file sequentially into memory for
processing this enables us to work with
large collections of files without
worrying about the number organization
or naming of the files next we will read
the reference files can create a table
with file names and corresponding tables
we will use this reference table during
feature extraction to build the feature
table so we now move from the access and
explore stage to the pre-processing
stage in this step we will process and
extract a set of features from the raw
heart sound signal using five-second
non-overlapping videos bulk of the
computation here happens in this extract
features function let's take a look at
it in order to capture as much relevant
information as possible we extract a
large set of about 71 features from each
window the extracted features include
summary statistics like mean median and
standard deviation features from the
frequency domain like dominant frequency
and spectral entropy as well as features
extracted using wavelet analysis we also
extract features from the speech
processing domain called the mel
frequency sub sub coefficients all these
features capture information from
different perspectives and they may or
may not be relevant to our problem so
let's go back to the live editor and
extract its features from our data set
depending on how many features you are
extracting and the size of your data set
each extraction can be a time-consuming
step in order to speed up feature
extraction we can use power 4 from the
parallel computing toolbox to distribute
computations across all available course
in the locally or on a computer cluster
for this example I have calculated the
features earlier so a skip feature
extraction and load the pre calculated
feature set let's take a look at let's
take a look at the features here we can
see the first five observations from the
from the features table so the table is
organized as the the columns
corresponding to different features or
predictors and rows corresponding to
individual observations for each
observation the final column corresponds
to the class which could be abnormal or
normal our next step is to quickly train
and compare multiple classification
algorithms the classification learner
app has been designed exactly for this
purpose so let's launch the app and take
a look in the classification learn app
we can start a new session the first
step is to select the table or matrix to
work with so in this case we are going
to use the feature stable the second
step is to assign the variables with as
either predictors responses or ignore
them by assigning them as to model board
in this example we are going to use all
the variables and assign the last
variable which is class as response the
next step is to select a validation
strategy because we have lots of
observations in this example we are
going to use the holdout validation with
50% of the data for training and 50% for
testing so let's start the session the
first view that we get from the
classification learner is a scatter plot
the scatter plot has one feature on the
x-axis the other feature on the y-axis
and the points in the scatter plot are
labeled by the different classes so this
is a good view for us to quickly look at
our features and and see which features
might carry some useful information when
you have lots of features like Indic in
this case it is
it is hard to actually look at all the
possible combinations this is a very
high dimensional data set so instead of
exploring this data set in a squat
scatter plot let's start training some
classifiers so in the classification our
app we have access to a number of
different classifiers so you have access
to decision trees logistic regression
support vector machines nearest neighbor
classifiers as well as in samples of of
different classifiers so the other nice
feature of the classification of our app
is that you can select to again select
these classifiers individually or you
can choose to train multiple classifier
sama traces for example I can select all
support vector machines and use the
ability of the classification or I've
trained these classifiers in parallel to
quickly train and compare their
performance so I have access to 12 cores
on a MathWorks cluster and I can use all
of them to train train my classifier
class wise pretty quickly so here the
training results show that the Kubica
SVM as well as the Gaussian and
quadratic ESRI and seem to give good
performance so let's take a look at what
this performance numbers look like so if
we take a look at the confusion matrix
this tells us that the MIS
classification break for the abnormal
class is significantly higher compared
to the normal class and this is most
likely because of the imbalance between
the number of observations so we can see
that the number of observations for
normal class is significantly higher
than the normal class which increases
the likelihood that our classifier is is
biased the other aspect to consider here
is that in the problem that we are
working on it might be better to miss
classify normals instead of
misclassifying abnormals that is it's
better to to be more accurate at
classifying abnormals at the cost of
misclassifying normals as abnormals
because this is a screening tool and
consequence of not screening someone
with abnormal is significantly higher
compared to screening some Lewis normal
and then figuring out that they don't
have they don't have medical condition
the ROC curve is another useful tool for
processing performance so here we get
get an idea of the trade-off between
true positives and false positives in
this example as I said earlier we might
be ok with with a higher false false
positive rate for the normal toss in the
fit in favor of increasing the true
positive rate so let's let's go back to
the library and dig deeper into some of
these issues in order to train the
classifier programmatically
let's first split the data set into
training and testing with a 50/50 ratio
function CV partition makes it easy to
create such splits if we take a look at
the number of observations for normal
and abnormal classes we can clearly see
the imbalance that could lead to a bias
classifier this is a common challenge
when it comes to supervised learning
so let's now train a classifier and try
to really take into account this data
imbalance issue so to come for data
imbalance we can assign higher miss
classification cost associated with the
class with fewer observations so in our
examples we are interested in maximizing
the detection of true positives for the
abnormal class so we will assign a 10x
cost with its classification in other
words we are going to tell the
classifier that it is okay if someone
who is normal gets miss classified as
abnormal rather than the other way
around however if the false positive
rate is too high the examination will
become useless so we must try to achieve
a good balance between the higher
accuracy for the abnormal class and
tolerance for for false positives for
the normal class another challenge
associated with classifier training is
with hyper open
returning most classifiers have one or
more parameters that can be tuned to
achieve a better fit in the case of
support vector machines one can tune box
constrained kernel scale and kernel
function tuning parameters manually can
be a tedious process and lead to
suboptimal values on the other hand
exhausting click search approaches can
be time-consuming and slow down the
iterative process of machine learning
MATLAB provides bayesian optimization
based method for for hyper parameter
tuning that is tightly coupled with the
fit function API so by specifying name
value parameters in the training
function the method efficiently searches
through the parameter space and finds
the optimal values for the hyper
parameters so let's take a look at how
hyper parameter tuning works
Bayesian optimization the method works
by building a Bayesian model of the
function that it is trying to minimize
and and uses it to select the next point
for evaluating evaluating the function
this approach allows the method to
quickly search the parameter space to
identify optimal parameter values this
method is specifically designed to
minimize the number of function
evaluations and is very efficient
compared to grid search or other
standard optimization techniques so as
you can see here the method has
completed all the iterations and has it
has found optimal values for box
constrain and kernel scale which it has
automatically selected for for the
classifier so let's go back to the live
editor and look at the performance of a
classifier that has been trained with
the misclassification cost and hyper
parameter tuning I have saved previously
in classifier so instead of training
training it here I'm just going to load
that classifier and look at its
performance so that's let's see so so
here is a confusion matrix as a heat map
and we can see that
by taking into account the miss
classification because we have we have
been able to significantly increase the
accuracy for the abnormal class however
this has come at a cost for for for the
normal class and we are now
significantly misclassifying normals as
as it mammals let's see what we can now
we can we can do about this so a
potential reason for poor performance of
your classifier could be overfitting one
approach for dealing with overfitting is
to reduce the number of parameters or
features by performing feature selection
feature selection is an important step
in the machine learning workflow and it
results not only in savings in terms of
computational cost and storage
requirements but also results in simpler
models that are very less likely to over
fit in this example we extract 71
features so there is a potential that
there might be features which are either
redundant or or or not getting any any
useful information so let's go back to
the pre-processing step and use the
feature selection technique called
neighborhood component analysis or nca
for short nca is a powerful technique
for feature selection as it is able to
handle very high dimensional data sets
significantly higher higher dimensions
than what we are dealing with as well as
datasets that are extremely large with
with lots and lots of observations the
other factor here is that you have as a
user you have control over the lambda
parameter which is the tag utilization
parameter and it allows you to control
the sparsity and as well as minimize the
redundancy in the selected features by
increasing the value value of lambda so
let's let's run through the feature
selection on our data set and see which
which features are I selected
so here the nca method went through our
features and identified out of the core
of the 71 features that identified only
11 features that are they're carrying
carrying useful useful information so
now let's go ahead and clean our
classification model with only the 11
selected features so we'll use the
selected features and also perform hyper
parameter optimization to train a new
classifier so here again as before I
have saved a previously claimed
classification model trained on the
selected features and I'm going to
simply load and look at the performance
of this of this classifier so if you
take a look at the performance the
performance on that normal class has
improved slightly but there is a
significant improvement in terms of type
of performance on the on the normal
class this indicates that there was a
potential for overfitting which was
addressed by the feature selection step
so at this point in our analysis we
might be happy with the performance that
we're getting from the trained
classifier our trained classifier is
able to accurately detect more than 98%
of their normal cases and almost 75% of
the normal cases the 25%
misclassification rate for normal cases
might be okay as this is just a
screening test and these cases will be
further evaluated by experts the next
step for us is to deploy our analytics
because the goal is to use hard sounds
for diagnosis in the field one
deployment solution might be to deploy
analytics to an embedded system which
can process data and provide results
immediately
Madeline provides support for code
generation that makes it easy to
automatically generate code for
deployment of various embedded targets
let's take a look at the process the
process consists of three steps first we
save the Train model using save compact
model method next we create an entry
point function in this case called
classify hard sounds click classify hard
sounds is a function that takes your
sensor data as input and produces the
classification label of normal and
abnormal after processing the raw sensor
data data to feature extraction and the
train classification model the final
step is to use the coder app to
automatically generate the C code so
let's take a look at what are the coder
app looks like and the steps involved
the bat encoder app provides a guided
step-by-step process to generate
stand-alone C and C++ code from MATLAB
code the first step is to select the
entry point function the next step is to
define input types that can do this
automatically by evaluating the entry
point function for this example we don't
want the signal to be a fixed length
next the app checks for any runtime
issues and prompts you to fix parts and
replace incompatible functions
looks like we did not have any issues
the final step is to generate C or C++
code
the app is successfully generated C code
which can now be deployed to various
embedded targets the final step is to
validate the generated C code by
processing files from the validation set
this can be done using the Mex file as
an interface between MATLAB and C code
in this plot we can see the various hard
sound signals as well as their actual
and predicted labels similar to the
performance on our testing set most of
the errors are associated with miss
classification of normal as abnormal
let's review some of the challenges that
we encounter during the workflow for
example data access and exploration
often involves working with data that
comes in all shapes and sizes these
lower data sets are messy and not always
tabular pre-processing data requires
domain-specific algorithms and tools for
example signal or image processing
algorithms are required to extract
useful features from signal and image
data when dealing with high dimensional
data sets
selecting the right set of features is
often important to avoid overfitting
issues developing and selecting
predicting models that generalize well
requires training and comparing multiple
algorithms algorithms often require
careful parameter tuning which can be a
time-consuming process integration and
deploy deployment of analytics often
requires translating or interfacing with
different hardware and software
platforms this can be a bottleneck for
applications that require frequent model
updates finally machine learning
workflows are never in a convenient
linear fashion we have constantly to go
back and forth and iterate and try
different ideas
for the first challenge of data
diversity we saw how MATLAB can work
with different types of data MATLAB can
also work directly with financial data
feeds text geospatial data and several
other data formats MATLAB is high
quality libraries of industry standard
algorithms and functions that can enable
feature engineering without requiring
domain expertise MATLAB also provides
tools for evaluating and selecting the
right set of features for both tall and
wide data sets with interactive abdill
and workflows we can quickly train and
compare models so the focus can be on
machine learning and not programming and
debugging tight integration of parameter
tuning techniques enables development of
models with optimal parameter values
MATLAB enables deployment of analytics
on a broad range of platforms ranging
from standalone apps to enterprise scale
systems code generation makes it easy to
produce stand-alone code that can be
rapidly deployed to embedded systems
finally MATLAB is an inherently flexible
modeling environment and a complete
programming language with no
restrictions to the customizations you
can make to your analysis this makes
MATLAB an excellent platform for machine
learning to summarize errors and key
takeaways about how tools available in
MATLAB and it will you to rapidly
develop and deploy machine learning
models if you are interested in learning
more please take a look at the product
documentation there are plenty of
examples and gossip pages that can help
you master the tools to learn more about
algorithms application areas examples
and webinars predicted to machine
learning please feel free to visit the
machine learning page that brings us to
the end of the webinar thanks for
joining us today in closing I'm going to
pause for a few minutes while we gather
questions if you have any questions you
want to ask please enter them in the Q&A
panel now