Tutorial 11 for Arduino: SD Cards and Datalogging Video Lecture | Arduino Programming: Step by Step (to become an Expert) - Electronics and Communication Engineering (ECE)

everybody welcome back for Arduino
tutorial number 11 I know it's been a
few weeks since my last tutorial I had
to catch up on some schoolwork and stuff
like that and I'm still working hard on
that so the next few tutorials won't
come out consecutive weeks I'm going to
get them out as fast as I can but I need
to make sure I don't fill any of my
classes to do these videos but I'm I'm
going to do this one right now and it's
going to be on SD cards and data logging
so we're going to make a nice little
data logger I've already put them
together and run some tests with it this
is the data logger that we're going to
end up with today we're going to using
an Arduino Uno an SD card shield from
cooking - hacks comm with a micro SD
card we'll have a temperature sensor a
light sensor and I also put two infrared
distant sensors in the back and aside
from the SD card these are all things
that we've worked with before so I won't
go too heavy into the code for these or
into the descriptions of these parts
because we all know how they work
already
so what I'll focus on is on actually
implementing some of the code testing it
out and getting the data off the SD card
and doing cool stuff with it and I'll
cover both reading and writing with the
SD card with the Arduino and yeah it's a
pretty extensive set of knowledge
required to put this together so if you
haven't watched my first 10 audio
tutorials now is the time to do that
specifically the SPI tutorial will be
useful so that's how we communicate with
the SD card the i2c Tory will be useful
because that's how we'll communicate
with temperature sensor and the analog
input tutorial will be useful because
we'll use that to grab data from our
infrared distance sensor as well as our
photo sensor for detecting the light
levels all right so that's what we're
working with let's take a look at the
circuit that we're going to put together
we'll do a little block diagram see what
we need and I will get started
programming it and testing out the data
logger the first step with any slightly
larger project like this one is going to
be to draw a block diagram so that we
know we're working with and how we have
to hug everything up now to start with
I'm just going to hook up the SD card
and we'll learn how to program at first
and then we'll go back and revise this
block diagram to add additional sensors
that we can log onto the SD card so to
start off with all we're going to have
very soon
communicating via SPI is going to be an
SD card that's it and we'll debug to a
computer using simple serial connection
as we've done many times in the past and
that's basically that's how we're going
to start just an SD card hooked up via
an SPI interface to the Arduino and
we'll debug it with sterile to the
computer it's important to note that SD
cards operate at 3.3 volts as opposed to
5 volts like the Arduino so they'll use
an on-board regulator that's already on
board the SD card shield that we're
using to regulate the 5 volts the
Arduino down the 3.3 volts to power the
SD card so let's get this set up so we
can start testing the SD card here we
have our Arduino Uno as well as the SD
card shield that we'll be using
I've already put the micro SD card into
the slot here and we're ready to put on
the Arduino this particular SD card
shield can be utilized in two ways you
can connect the pins over here to
digital pins 8 through 13 or you can
attach the in circuit serial programmer
to the connector right here both of them
provide the appropriate SPI pin
connections the only difference really
is that if you decide to use these it
makes a little easier to put shields on
or different shields on in addition to
this one and it also means that you have
to set the power pin high for a pin 8
because we'll be using that to power the
device directly off of one of the
digital i/o pins there's also a jumper
they used to select which interface
you'll be using I'm going to choose to
put it on the digital pins 8 through 13
to utilize it great the very first thing
that we're going to do to test out the
SD card is we're just going to write try
writing it some very simple data over
and over again about every 5 seconds and
then we'll read the data with the
computer to confirm that the Arduino
wrote data successfully to the Arduino
then we can start doing some more
complex stuff with it
I've already started our program out
here first off you have to include the
default SDH library it comes with the
Arduino programming environment so make
sure you include that next thing is we
have to
at any pins or that we're going to need
to set as inputs or outputs so you need
to change the chip select pin that's
going to be pin 10 you always need to
set that whenever using SPI although 10
is generally the default if you remember
from our SPI tutorial and we're going to
find the power pin and remember as I
said before I have the SD card shield
plugged into digital pins 8 through 13
and 8 is just gonna be passing 5 volts
of power to it by sending in a digital
high and that will give it 5 volts so
we'll define the pin that we'll use for
power in this set function we'll start a
serial process so that we can send
serial data back to the computer to see
what's going on so we'll start the
serial communication and just let the
user know that we're initializing the
card next up we'll set the chip select
pin is an output because we're going to
toggle that higher loan we're writing
reading or writing data to the SD card
because you could potentially have other
SPN devices on the same vessel that we
don't at the moment and next thing we're
going to draw power from pin 8 so that
the SD card has power to read and write
and now let's fill in the rest of this
program so the first thing is we're
gonna have to check if the card is
reading remember you can figure out all
these functions for SD card if you just
go to the Arduino website and a look at
the SD - library they explain all these
but I'll show you how to get started
here so the first thing we have to do is
check to see if we can actually write to
the SD card by running SD begin and in
one fell swoop we're going to run an SD
dot begin and check to see if it was
successful with this command so what
this is going to do is it's going to
begin an SD card process and while it's
doing that if that for whatever reason
returns value of false which it will if
for some reason you can't succeed I
can't connect to the card oops we
plugged it in wrong or some like that
this is we return it for false and we
can check for that to show the user that
there's been a problem so if this is the
case and it returns false we'll do
serial dot print line
card filled that way the user knows that
something is going on and by putting a
return statement right here this will
just end the program right here so if
the card fails there's no reason to go
around the loop or anything like that
it's not going to work so we're going to
return and the entire program is going
to terminate that's going to happen
whenever you put a return statement okay
and then we're just going to if that
didn't give an error and it didn't
return then it means the card is ready
so let's just let the user now serial
dot print line card ready great okay
now let's fill in the loop all I'm going
to do is print the word hello you can
pick any word you want maybe you want to
write
Jeremy is so cool
and I'm just going to print that over
and over into a text file every five
seconds so the first thing that we need
to do here is we need to make something
of a structure type file so we'll do
file and what's called data file going
to equal SD dot open you'll specify the
file you want to write to it does not
have to exist yet if it doesn't exist it
a lot of math you're creating so I'm
going to call it log dot txt make sure
you put it in quotes because filename
and the second parameter is optional if
you don't put a second primary and it'll
only be able to read from it but if you
do put the second parameter in you can
specify that you want to be able to
write to this file so we'll specify that
okay we now have something called data
file which is a type file that opens on
the SD card log text you can only have
one file open a time for writing so once
we're done with this we'll close it so
we can use another file if we so desire
so let's first see if we actually
managed to successfully open it it's
important to do error checking like this
so that when you're running this in the
field you can account for potential
issues that might arise
okay so data file dot print line if if
we are able to connect to it of course
we'll just print the data string to it
so this works very similar to the way
you print stuff over serial we have the
print line command just like we can do
serial dot print line we can add a new
data file dot print line so we'll print
data string which you defined earlier
just onto the next available line in the
text problem it's as simple as that
and then once you're done we want to
make sure that we close data file so
that at some point later we can open a
different file so close it and now for
debugging purposes I'm just going to
print to the display window the exact
same thing that we just edited the text
file so we can keep track of our program
okay and now let's handle the other case
so this is assuming that data file was
created
if data file couldn't be created for
some reason we need to handle that case
it will make an else statement and in
there
will tell the user that we couldn't open
or create the log file so we'll do
serial dot print line so remember serial
is sending into the computer data file
is sending it to the file on the SD card
will do serial dot print line couldn't
access file and that should be just
about it there's only one more thing
that we have to do here which is to put
our delayed five which I've already done
so remember we were working in
milliseconds here so five thousand is a
five-second delay and what this means is
that it will wait for five seconds we
can go back and then write it another
line into the SD card so let's see how
this works so the file has been uploaded
to our SD card enabled Arduino let's
open up the serial monitor great it says
that it's initializing the card card
ready and Jeremy is cool and a few
seconds it will print it again and I'll
keep doing this every five seconds now
I'll take the SD card out of the Arduino
plug into the computer and show you that
we can actually read the data right off
the text file
that we just saved onto the micro SD
card I'm now taking the SD card out of
the Arduino and plugged it into my
computer and we can see that did indeed
create the log file so let's open it up
and sure enough there's the data that we
wrote to it all right awesome so next up
let's delete that we can recreate it
later through the program and now I'm
going to add a text file to my SD card
and we're going to use this to read data
from the SD card so I'm gonna make a
file called commands dot txt and in here
I'm just going to put one simple value a
1000 let's say and what we'll do is
we'll read this out and we'll use this
in our program as a delay between when
it writes data to the SD card and this
will allow us to dynamically change the
program without ever actually having to
plug the Arduino in and reprogram it you
can change the way the Arduino works by
simply adding a bunch of commands here
and then interpreting them with the
Arduino to do something different ok
so now let's put this card back in the
Arduino and we'll see how to read the
data off the card alright now I've gone
through and started to modify our
original program to read the data off
the SD card first thing I did was define
a new float float just meaning that it
has a decimal value potentially and
you'll see in a second when I do the
math why I have to do that that's going
to define their fresh rate so before we
just hard-coded it as five thousand or
five seconds and I will be defined by
the value in text file which if you
remember we set to 1000 so the beginning
of the setup is all the same we're just
going to connect to the SD card etc but
now we have to add another thing to
setup this part here that reads from the
configuration information on the SD card
and sets the variables appropriately now
the way spi or any serial form of
communication works is one byte is sent
back and forth at a time so we can't
just read the number 5,000 off we
actually need to read it byte by byte
and then reassemble it into the number
five thousand 1000 whatever it is so the
first thing we'll do is just like we did
before we're going to write a new file
type called command file and it'll be
commands up text and before where we had
to put on read file we don't have to do
that or a write file because
only going to be reading from it again
we'll have an if statement to check if
the command file was actually found and
if it was um we'll let the user know
that we are now reading the command file
by printing it to the serial terminal
now we have to do is actually figure out
what the number is that's printed there
and that's actually a little bit more
challenging than it seems you have to
come up with some clever math to do that
and that's what this is doing right here
so the first thing I'm doing is I'm
defining the decade the decade being
just essentially the number of digits
that are in the number so for a value of
a thousand the decade that it starts
with is going to be a thousand and then
if it was 500 the decade would be a
hundred it's basically just the the
number of it's basically representing
the size of the number in decades of ten
so if we do that to find that value um
all I'm doing is I'm saying the power so
this is doing the power so 10 raised to
this value which is the second argument
of the power function which is a math
function that's included with Arduino
and what this is doing is this is
looking at the command file and getting
the available number of bytes in it so
the available number of bytes if we have
a thousand the value of a thousand in
there there's going to be total of four
bytes because there's four digits in the
number 1000 so it'll return the number
four and then we'll subtract one from
that so that'll be 10 to the 3 and 10 to
the 3 is a thousand so that's giving us
the decade that we want but it could
very well be five thousand or three
thousand all we know is that it's 1,000
times a singular number between one and
nine basically we know that it's in that
that decade order so the next thing
we're going to do is while the command
file has bytes available so we're going
to step through it byte by byte we're
going to take the float we're going to
take a float that we're defining as a
temporary float and set it equal to that
current byte and if you remember from
the serial communication tutorial that I
did a while back we can get that byte as
a number
but region in as a char but we want as a
number by just subtracting the character
equivalent of zero this is going to read
the first byte the first byte will be a
one because we use the value a thousand
so it's going to be one minus that byte
value so that we actually get a float
out that's going to be one point zero
basically
now the refresh rate which when you
define up at the top to start as zero
we're going to update that refresh rate
so we remember we started a decade at a
thousand so what's going to happen is
this temp value that we just got out
when you got a one so this is the first
digit is one temperature is giving you
that digit we're going to multiply one
by the current decade so we're gonna
multiply it by a thousand and then we're
going to add it to whatever the first
rate current is so it starts at zero so
right now we're fresh rating equal to a
thousand and then we actually already
have the number we need but we don't
know that we could have entered a number
you're like 1324 so we need to
accommodate for that so the next thing
we'll do is we'll divide the decade by
ten so if it was a thousand and now goes
down to a hundred will read the next
byte the next byte is a zero and now the
refresh rate will equal what it
previously was so a thousand plus 0
times decade which is a hundred so it's
just going to be zero again and you're
going to end up with a thousand after
you finish going through this loop so
this is just a clever way to read out a
number easily from a sequence of bytes
and you can imagine how this would work
if we had a more complex number like
1300 let's say the next number that came
down would be a three so we would get
the three is the temp value we'd
multiply it by 300 because we just
decrement the decade from a thousand to
100 by dividing by ten so we would have
1,000 plus 300 and we would have 1300
the number that we want so that's how we
get the integer out of it I know it's a
little confusing if you have any
questions feel free to ask me so now we
can print to the user this is the
refresh rate awesome otherwise we have
to say we did not read the command file
and just like before we'll return
completely ending the function if it
can't find that command file the data
string can be exactly the same as before
and all this is actually pretty much the
same the only thing that we're going to
change now is we're going to change the
delay instead of being a hard-coded
5,000
to refresh rate it okay and that should
do it let's compile it and see if it
refreshes the data just a little bit
faster okay so we can see that it's
loaded up it's read the command file
properly it knows there the refresh rate
is now 1000 milliseconds keeping in mind
that I had to use a float here because
we're dividing by 10 and you can't be
sure that it's number that's divisible
by 10 although technically it should be
you could potentially run into issues
and it's definitely refreshing once
every one second now instead of five
seconds great we can now read and write
from text files or ready to make the
final code that will implement our data
logger great we've managed to
successfully get the Arduino to
communicate with an SD card over an SPI
protocol while simultaneously sending
debug information over serial to the
computer we're ready for the next step
which is to add some sensors whose
information we can then record to the SD
card
we'll use a light sensor just a standard
photo detector a temperature sensor on
the i2c communication bus what you've
learned about in previous tutorial and
to end fret distance sensors and we'll
feed all those into the Arduino as
analog values and record them in comma
separated value format to the SD card
which we can then read with Excel MATLAB
just when anything you want to visualize
all the awesome data you've acquired so
let's get that circuit set up and then
we'll program it on the computer to get
it recording some data I've gone ahead
and wired up the circuit for us we've
got the same Arduino as before with an
SD card reader mounted on top of it I
fastened the Arduino to the breadboard
by actually running some wires through
the holes that hold it in place and just
popping in with some breadboard holes
just to hold everything together on the
breadboard here I have the light sensor
connected through a 10k resistor voltage
divider into the Arduino don't worry
I'll have a somatic for this available
for download so you can see it in
schematic form here's our eye to see
temperature sensor we have the pull-up
resistors that are working on the i2c
bus and then - I do see lines that that
feed into those inputs on the Arduino if
you remember correctly they are
log inputs four and five on the Arduino
then we have the two info just ancestors
which I actually have mounted to the
back of this plate you can see the wires
going to them and those feed into to the
analog inputs as well so we're using 5v
analog input pins two for the info
distance sensors we're using one for the
light sensor and two for the
communication lines of the i2c bus
because those along input lines are
multiplexed as I to see communication
lines so that's basically it the only
other thing to note here is that this
system is going to require a fair amount
of power because of all stuff running on
it so when you actually start loud and
in it you're going to need to use a nine
to 12 volt power adapter like this one
you can also use a nine volt battery the
reason being is the USB port is not
gonna be able supply enough current
Falls to run correctly and I actually
found that trying to rather serve a USB
port I was reported incorrect values
from the IHF sea temperature sensor due
to a lack of a required current to run
the entire circuit plugging this in
fixes it totally and no one did anyway
because the whole purpose of this logger
is to run it without a computer what's
the fun if you have it plugged in by USB
all right now that we've got this wired
up
we'll go write the code for it and
remember you can grab the schematics for
this from my website so now that we've
actually managed to find a way to read
and write from the SD card that was
basically alone new material so I'll
just walk you through the code that I've
already written that grabs the data from
the various sensors that we've hooked up
concatenates into a string and then
spits it onto a file on the SD card now
instead of using a txt file I'm going to
be using a comma separated value file or
a dot CSV file you guys it can be easily
read by a program like Excel to make
pretty graphs and stuff like that so
here's the changes that we had to make
first off i'm now importing the i2c
library because you're using i2c
temperature sensor i'm defining the pins
that i'm using it for various analog
sensors including the ir pins and light
pin i'm defining the temperature address
of the i2c temperature sensor because if
you remember on high to see
communication buses you identify devices
by their ID and I'm starting a value
called ID equals one that will put at
the beginning of each piece of data that
we read so read data once every second
is specified by our command file and
assign each set of data and ID and we'll
increment that ID as we go so you can
see how many pieces of data we've
recorded in total let me look at the
outputted file most of the setup is
basically the same the only thing I've
added here is at the end after we've
written after we've read from the
command file and stuff like that we're
going to write to the top of the log
file with header information so instead
of just writing raw data at the top of
it we'll provide information about what
each column represents so we're going to
write each of columns in this order ID
then light data temperature data and
then each of the infrared distant
sensors so this basically just makes a
heading for each column that tells us
what data below is corresponding to ok
so that's basically it for the setup and
now all the loop is going to do is every
time that we hit our refresh rate we're
going to go through and read the data
from all the sensors and save it to a
new line in data file so reading the
light level is easy that's just a simple
analog read reading the temperature is
copy pot copy pasted directly from my
tutorial on I to see communication so
this is just taken right from there and
all this does is it reads the current
temperature value and then converts it
to degrees Fahrenheit from the high
juicy temperature sensor and then lastly
we're going to read the to distance
values from the infrared distance
sensors now that we have all the values
we're catenate them into one string then
we can write to one line of the data
file each piece of data will be
separated by comma that's why it's
called a comma separated values file and
we're going to concatenate them together
using simple plus signs we can write
them two strings first with this command
string because we want to concatenate
them all together into one string that
we can then write easily whereas right
now there are integers so it can write
the ID to any string write it and then
put a comma and then we'll put the light
level then comma then temperature in
Fahrenheit another comma then the
infrared distance value and then another
comma and then we'll finish up the line
with the second in for a distance value
so now we have a piece of data that we
can write to it
this is equivalent to before just
writing Jeremy is cool all right now
we're going to open the log file which
I've changed from log text to log dot
CSV cuz it's a comma separated values
file and we're just gonna write that
data string that we just created so data
string is that concatenation of data
that we just put together I'm just going
to write it to a new line on the SD file
and that's it last thing we have to do
is increment the ID number so that next
time after the first time it'll print
one here the second time I'll print two
etc so you can identify each of the
readings and then they'll just wait for
the guarantee the set refresh time so if
we set it as a thousand in the command
file then it will take new samples
approximately every one second it's not
exactly every one second because it
takes a little bit time to run all of
this here but basically we'll be taking
a sample about every second and that's
it so we'll upload that now and we'll
set up our data logger to grab some data
over a 24-hour period and it'll come
back and check on and see what kind of
stuff got okay now that we have it all
program I've gone ahead and wedged it
into my window over here and I have two
infrared distance sensors facing in
towards the room and then facing out
towards the window is a light sensor and
a temperature sensor so my hope is I'll
run this for 24 hours now and see how my
movements within the room correspond to
the change in temperature inside
basically inside the room I guess I want
to take it outside too much although
it's next to window so it'll probably
fluctuate a little bit but more
importantly the time of day outside now
I'm not sure if a one-second sampling
rate is actually sufficient to capture a
whole lot of movement in the room so we
might not see a whole lot of data from
the infrared distance sensors it'll
depend how much I move in front of them
to be honest I'll probably be in class
all day so who knows if we'll get any
data from them but more importantly
you'll definitely be able to see the
changing light levels outside over the
course of a day so let's go ahead and
plug this in I've already gone and
removed the the log file so we're
starting fresh with it so all we have to
do now is plug our power in like that
and we're all ready to go it's now
recording and if you look at the back of
it you'll be able to see that the the
transfer LED on pin 13 is blinking
once every second indicate that it is
grabing data so flip that run then we'll
come back and see what kind of stuff we
got I've gone ahead and run the system
for almost exactly 24 hours now I
started it at 11:30 at night and turned
it back off the next night at 11:30 so
there's 24 hours of data here and we can
open up the CSV file on the computer I
suggest you save it off the SD card onto
the computer in Excel and you can see we
have our nice row headers here and the
data that's taken any data point as well
as an incrementing ID we can scroll
through and there's a ton of data here
and this is only taking up a tiny amount
of memory - that's a 2 gigabyte SD card
I could have probably taken samples once
every 5 milliseconds and stun more than
enough space on here but anyway once
every second is fine and this is the
data we have so let's go ahead and graph
it to see what kind of stuff we we
actually got out of here and I'm sure
there'll be a few anomalies nothings
that were quite perfect but we can
average those out using some moving
filters so we'll go ahead to insert a
link scatterplot and generate welcome
data ok cool so we can see right off the
bat that the blue line here is marking
daylight and that's obviously during day
hours and I started at nighttime you can
see when went up here from a light in my
room because I hadn't gone to sleep yet
and then when it's down the bottom is
when I was sleeping so first off is a
ton of data points here so let's grab
each of these and on data series to
remove the actual points on them because
they're kind of in the way and it's a
little annoying all right that's a
little bit better right all right so
we've removed all data points now the
next thing you want to do is temperature
is on a different scale than all these
so temperatures and actual degree value
whereas light ir-1 and ir-2 are just
analog values from 0 to 1023
so let's adjust temperature to put it on
a separate y-axis value format the data
series put it on a secondary axis okay
cool so now the right axis just
corresponds to the red line or the
temperature and so we can see how that
fluctuates it obviously got warm at this
point during the day right when the Sun
was as peak so
that makes perfect sense and there's a
bunch of anomalies in here that's just
due to random bad readings from the i2c
temperature sensor it's going to happen
occasionally luckily we can filter those
out by putting a simple trendline in
here and using a moving average filter
so we will add a trendline on a moving
average and you can see it's already
little bit better but the period of two
is way too small let's do more like a
hundred and that'll filter out all the
weird values that we've got going on
okay that's looking better but it's
obviously too much try 10 that's looking
pretty good maybe a little bit more than
10 okay that looks pretty solid so now
why don't we just go ahead and turn off
the red line entirely because it's not
really helping us very much we want the
best fitted line anyway so we'll go to
line color and turn it to no line okay
great and now black is representing a
moving average of temperature infrared
distant sensors are measuring basically
movement in the room again there's
probably some anomalies in there too and
we can put a moving average filter for
those to check for them but I don't know
I was in my room a couple times during
the day this could very well represent
me moving in or out it could represent
the blinds over the window belong back
and forth things along those lines
it could even represent things like
shadows moving across the room okay but
the key point here is that we can see
all the information very clearly this
started at around midnight and ended at
around midnight and it's very clearly in
the middle of the day it was sunny out
because this is the light sensor there
was also a temperature peak in the
middle of the day that makes sense
because someone's at its highest and it
was gloomy in the morning and got sunny
in the afternoon so it makes sense to be
a little warmer and then stay a little
bit warmer until I turned it off that
could also have something to do with the
heating in my apartment turning on the
infrared distant sensors as I kind of
expected didn't really do very much
because I didn't move in front of them a
whole lot so they're mostly just picking
up some random noise from various things
fluttering around in the room and stuff
like that nothing too interesting but
it's pretty clear that we can do some
really neat stuff with this and now we
can track this over time and in future
tutorials I'll show you how you can
real-time from the internet or from GPS
satellites and things like that
all right great so we have our data
logger now you can hook up any kind of
sensors you want to this and get really
cool data out of this and then plot it
and do whatever you want so awesome I
love to see what you guys managed to put
together with this data logger
functionality thanks guys for watching
this episode of my arm user toro series
I hope you enjoyed it as always if you
have any comments suggestions questions
etc please feel free to post them on
youtube on my blog Jeremy Blum comm our
element14 and you can find all the
schematics source code and parts list on
both my website and on element14.com
thanks again to cooking - hacks that con
for supplying shield that I used to make
this video and you guys can look forward
to at least four more videos before May
31st on a variety of cool topics that
should be pretty fun and I'm looking
forward to making them so yeah if you
have any questions or anything let me
know otherwise I'll see you as soon as I
can get the next video app thanks for
watching thanks to element14 for helping
me to sponsor this video series they
were kind enough to provide a lot of the
materials that I'll be using to create
these tutorials
feel free to go visit their website at
element14.com check out their community
which is a great place talk to you about
electronics the Arduino and basically
anything else engineering related and
they also have a store where you can buy
a lot of the parts that we'll be using
in these videos