How to Get Ideas and How to Measure - Stewart Butterfield & Adam D'Angelo Video Lecture | Startup School Online by Y Combinator - Entrepreneurship

>> Good afternoon.
Today we have two different speakers.
We have Stewart the founder and
CEO of Slack and
then we have Adam,
the Founder and CEO of Quora.
I'm going to interview Stewart.
We're going to talk about ideas and how to
find good ones, and then Adam is going to
talk about metrics why they are important,
and what to measure.
Stewart, thank you for coming and
spending some time with us.
Stewart is the founder of Flickr and
Slack, and what I didn't know until very
recently is that he tried to be
the founder twice of massively multiplayer
online games, and they turned
into these two iconic companies.
So I wanted to talk about just the process
of coming up with the idea for
the game and then the ideas for
Flickr, and Slack.
>> So yes, twice we set out trying to
build a web-based massively multiplayer
game and the origin of that was all the
way back in 1992, I arrive at college and
I got my account on the school's
Unix machine and was exposed to
the internet for the first time and
it was kind of a mind blowing experience.
So, this is about a year
before the web became popular.
So it was like IRC, HughesNet, talk, email
and I was really fascinated with the idea
that people could find their community
irrespective of what that community was.
So if you were a breast cancer survivor or
a model train enthusiast or whatever,
if you were interested in some as I
was obscure 19th century biologists,
then you can find all that stuff.
And the idea of a game that lived online
like that and had the game like aspects or
play as a pretext for social
interactions seemed really interesting.
And I'm not going to go into to
long of an explanation, because it
turns out it's like a terrible, terrible
idea at least as a commercial business.
Like it might have been fun as a hobby,
I think it could have been the kind of
business that sustained the number
of employees required to operate it.
So, like a restaurant or
something like that.
But when in 2009, we started
the company that ended up making Slack,
we raised $17.5 million in extra
capital and it's just never
going to become the kind of business
that would justify that investment.
The first time was a little bit different.
So, 2002 was after the Dot.com crash.
It was after WorldCom and
Enron accounting scandals.
It was after 9/11 and
the NASDAQ was down 80%.
S&P 500 was down, 65% from its peak.
No one wanted to invest
in internet facing stuff.
So it was very hard to tell whether
this was a good idea or not,
because there was no signal
from the market that said,
this is an investable idea, because
nothing was investable at that time.
We raised a little bit of friends and
family money.
We spent about a year or
so building a prototype.
The prototype was well-received, but
it was going to take us another year and
a half to finish it and
we couldn't raise any money at all.
We got to the point where the one person
on the team who had kids against was
the only person that's getting paid and
then we decided to try to find something
else we could do that would make use of
the technology that we already developed,
but that will be quicker to get to market.
And that thing was Flicker, but
it was also a bad idea at first.
So because we had developed this game
client and we had a messaging server on
the back-end, we just re-purposed the game
client as a photo sharing client.
So in the game you had a bunch of slots
along the bottom of the screen for
inventory, those became slots
into which you can fit photos.
And because the game was
all real time interaction,
the photo sharing was all real-time
interaction which meant that you have to
be online at the same time as someone
you cared about in order to use it.
And while it demoed really well,
it was amazing technology, especially for
at the time we started and
that maybe like late 2003.
It was a terrible product,
because you didn't care about any of
the people who were actually online.
That morphed really quickly and
the experience making Slack
was a little bit different.
We had all of the resources
we could ever ask for.
I mean, we had more than enough money,
because we had money leftover and
we decided to abandon the idea and
we had AWS which we didn't have before.
Hardware was effectively free internet
connections were much better.
There were many more people online.
Hardware was better.
So, none of like those external
conditions was the reason this wasn't.
This massively multiplayer game
on the web wasn't a good idea.
It just wasn't a good idea, because
there wasn't a big enough market for it.
>> And why was Slack a good idea and
how did you decide that
Slack was a thing to build?
>> Well,
we started developing the system for
internal communication that was base IRC
and those of you who had IRC kind of,
and also use Slack can recognize
that Slack is IRC with all of things
that were missing when it was
originally designed in 1989.
I think this is, I don't know,
it's like it's post facto analysis.
So it's hard to know if this
is really true or not, but
I think that the reason this lock has been
so successful is because we had a 3.5-year
long design process where we weren't
conscious that we were designing it.
So, we started using IRC.
IRC has many deficiencies.
Among them, you can't store and
forward the messages.
So if I wanted to send you a message and
you're not connected to the server that
the moment I want to send it,
I just can't send you a message.
So, the first thing we built was
about to log all the messages and
put them in our archive.
And once we had that,
we had messages at a database and
we thought search would be useful, so
we built search for it and this was 2009.
There were no good IRC clients for
iPhones.
And so, we built an HTML5 front-end for
the archives and Safari.
And once we had that, we wanted to be able
to post and just like on and on and on.
But in the normal process
of software development,
there's a huge amount of ego and
speculation.
So when you're arguing with someone about
whether this feature is a good idea or
not and if you can get to the point
of enlightenment and a Buddha sense,
then maybe you can do
that without any ego.
But the practical reality is people are
very ego laden in those discussions and
there's a lot of speculation like, I think
this could be really valuable for someone.
Whereas when we were building the system
that was kind of the proto Slack,
that was the inspiration for Slack, it was
whatever problem is the most irritating.
Like something that we just absolutely
can't stand, we would address that and
then with a minimum amount of effort and
then go back to what we're supposed to be
doing and then kind of let cook for
several months.
So we had 45 people working on it,
working on the game.
So, we had a lot of real world feedback or
when there was a problem.
I mean, when an opportunity that was so
obvious that we couldn't help, but
take advantage of it, we would spend
the minimum number of minutes on that and
go back to it.
So the end of that process, we had this
fully designed product that we were using
that was a terrible implementation not
what you would do if you started from
scratch, but it was obvious that this
was something that had enormous value.
Everyone who worked at the company agreed
they would never work without something
like this again.
And so just on that basis,
it seemed like it was a great idea and
we had years of evidence.
>> Were there other things
you considered pivoting to or
was it just like there's this thing
we're going to do it, it's working.
>> I wish that I had better access to the
proposals that I personally had written,
ciz I had a whole bunch of **** terrible,
terrible ideas and this was the one that
had the broader consensus among the team
as being something that would be valuable.
What's interesting at the time though,
so we had $5 million left in the bank.
None of our investors
wanted their money back.
They want us to just go
try something else and
Jason Horowitz is one of our investors,
so we went down and
did the partner meeting and
kind of presented what our big idea was.
And at that time, this is before we
actually started development of Slack.
But when we had this proto version,
we said that one day,
this could be like a hundred million
dollar a year revenue business.
It could be valued at a billion dollars
on that basis and we thought that,
that was like a decade out and
that would be the culmination.
That would be We have 100% of the ten and
we went past that in year two,
kind of blew past it.
We'll do 200 million in
gap revenue this year and
still be growing at a very fast rate.
And now we think that number is
more like 10 billion in revenue.
Like a hundred billion dollar company.
So we didn't realize how
good of an idea it was, but
it's also there's an interesting
story about deciding to shut it down.
Because we still had money, because we had
a relatively big user base who were very
engaged, but we had a really like a leaky
bucket, it was very difficult to add
new users to the game that we're
working on, which is called Glitch.
I felt very stuck in this position,
because on the one hand you have narrative
of good entrepeneurs are resilient.
When everyone else thinks it's a bad idea,
it's probably a really good idea.
You have to dig in, prove people wrong.
Keep going, even when times are dark.
One the other hand,
there's a morning when I woke up and
it's just like,
I can't do this anymore, I'm the CEO,
I'm the chairman of the board,
I'm sure that this is not going to work.
And it's actually hard to
convince everyone else.
>> I think that's one of the hardest
things about evaluating ideas.
Is saying when this
one's not going to work.
So, do you remember what it was about
that morning when you woke up and said,
you know what, it's time to move on?
>> I don't.
I mean, I think people have very little
interest in access to their own
cognitive processes to begin with.
So anything say would be
kind of making it up.
But I think it was I had exhausted all
the ideas we had to fix problems, so
we focused on new user experience.
And we really wanted to teach
people what this was and
kind of what the possibilities were and
get them excited about it.
Because we were okay at attracting
people into the top of the funnel but
had a hard time retaining them.
And.
>> That's always when I tell people to
give up an idea when you're out of ideas
about how to make it work.
>> Yeah exactly, so we'd have an idea,
try it didn't work, have another idea and
at first you can fill up post it notes,
like those giant boards and you can fill
up white boards with ideas and you argue
about which ones are the highest value.
And then one by one you eliminate them.
Months go by, a year goes by,
another six months goes by.
And it's like ****, these are all of the
ideas we thought were the worst ones to
begin with and we only have three left so
this is probably not going to work.
>> You mentioned at one point
you had a bunch of proposals,
how do you come up with ideas?
Or how did you even think that you know,
slack is maybe this thing
is a really good idea.
How do you come up with new ideas and
how do you evaluate them?
>> Coming up with new ideas is easy.
Like I just I feel like it's hard
almost to not come up with ideas.
I mean,
they're not necessarily good ideas so
the recognition of which ones
are good is actually much harder.
I recently finally got around
to reading thinking fast and
Slow and there's this story there of
a psychologist named Herbert Simon
who studied in Chess Grand Masters
because Chess Grand Masters
are one of the category of people
like really experienced doctors.
That people regard as sort of miraculous
in their ability to intuit a situation,
like glance at a board and say,
white checkmate in three moves.
And people thought that this was like
some super well developed intuition.
And Professor Simon argues that it's just
a recognition in the same way that like
a 18 month old can look at a dog and
say doggie.
Or within the first 500 milliseconds of
someone speaking you can tell if they're
angry at you, like you just develop
the ability to recognize situation and
ques which.
Are hard to disambiguate, they're hard to
articulate and enumerate in your mind and
say exactly why you think this way.
But over time, you get better and
better doing it.
I've been making software
professionally for about 20 years, so
the specific micro-ideas, or sub-ideas,
about feature implementation, or the order
of actions in the user experience.
Or whether a certain feature is more
important than another feature.
Those ones,
I feel like I have an intuitive sense.
Which is actually just the ability
to recognize, over time.
So it's really just
a process of filtering.
Like you just have constant ideas.
Everyone has ideas.
We at Slack have developed a lot
of organizational design and
practices to try to
absorb as many as we can.
So we get about, somewhere between 10 and
20,000 tweets at us a month.
And all of those are responded
to by a actual human being.
We get a similar number of customer
support tickets, and we have Processes for
trying to filter up and abstract all
of the feedback that we're getting.
But even down to the level
of all the tweets at
Slack the company are pumped into our
Slack instance and people check them, and
they can add an emoji reaction and
that filters it to another channel.
For the ones that seem above some
threshold of this seems like a good idea,
and it also seems practical.
I could imagine that we could do it.
And then there's some debate at
that level, but, I mean, every
Well, because almost every person I know
in the technology industry also use Slack.
Every time I talk to any of
them they have ideas for me.
I mean they have complains or proposals
>> So
the big ideas or
the little ideas I get the intuition on.
What about the big idea, did you spend
a lot of time thinking before you
decided to go ahead with Slack, did you
spend a lot of time thinking about this
would be really strategically valuable and
there would be a deep network effect?
>> No not at all.
I mean so we thought that this would
be valuable for individual teams and
we could see that value
demonstrated in our own experience.
We definitely didn't see
how applicable it would be.
Because right now, we have 38,000
paid customers around the world, so
that's a lot.
We have a sales team but we've never
talked to 90 plus percent of our customers
because we just couldn't
talk to that many people.
So again, this is already farther than
I thought we would get at the very end.
And so we under recognize it.
Those big ideas are, I don't know,
we talked briefly about this topic
before and one of my favorite lines is,
I have a great idea for a novel all I
need is for someone to write it for me.
Because obviously preposterous.
Right?
Like no one would really think,
I mean only a really deluded person
would think that that's true.
But people definitely have ideas for
apps all the time, and
those can be harder to assess.
And it's so rare that I hear one where
someone proposes an idea where it just,
it's like prima facie that is a good
idea as opposed to [SOUND] I don't know,
I mean it really depends on the execution.
Actually just last night,
I was watching something on YouTube, and
the ad that came up before was for an app
called MileIQ, and it's you turn it on,
ask for location permissions, and
then every time you go on a car ride,
it just looks to see the speed
that you're traveling.
And then at the point you stop,
it stops it.
So kind of like an automatic Strava.
Like it just records all your trips.
And then you can swipe right to say that
this is a trip that my company's supposed
to reimburse me for and swipe left, not.
Anyway, it's a very simple idea.
It's actually a really great idea because
people have trouble recording those,
they have trouble in audits with the IRS
to prove that they used the card for
this purpose and stuff like that.
And that's one that actually seems like.
Hm, the execution doesn't
seem very difficult.
I think that would be a good idea.
If you could market it effectively, it
would probably be a successful business.
But like 95% of the ideas I hear,
I have no idea whether it'd be successful.
>> Where are you generally on
the spectrum of ideas matter and
it's worth the effort to
think of a good one and
I think it's just all that execution,
and ideas don't matter at all.
And you just have to keep
trying things until someday
one of your internal
tools turns into slack.
>> So there's,
I'm There's
a lot of people in this class and then
there's a lot of people at Stanford, and
there's a lot of people in South Bay and
there's a lot of people in the Bay area
and many many people want to start tech
companies and all hope to be successful.
So to a certain extent there is like 100
million monkeys banging on typewriters and
someone's going to start a Shakespeare
sonnet so, there is a little bit of that.
Like a little bit of survivorship bias and
other cognitive biases
that come into play.
On the other hand, I don't think
either end of the spectrums is right.
I would definitely lean
more towards execution.
There's a good Steve Jobs elaboration
of John Sculley got hired and
started hiring all these people at Apple
who thought that only the ideas matter.
The idea absolutely matters, but
the idea when you start hitting execution
has to accommodate so many things.
In Steve Jobs case, it's about what
it's possible to get glass to do and
how small you can make batteries and what
kind of tensile strength plastics have,
and stuff like that.
because your idea has to stand up to all
these things and it'll transmogrify and
alter and may even be unrecognizable
at the end, but if you're really
good at execution and the core of the idea
can make it through all those steps.
So I absolutely favor execution,
but you can execute
a lot on a terrible idea and then not
get anywhere so you kind of need both.
>> If you were starting
a new company today,
you know April 2017,
where would you look for ideas?
What sorts of things would
you be thinking about?
>> I don't know if this works for
other people, but
I would definitely look at my own
experiences as a consumer, generally.
Because it's very easy for
me to see things that are frustrating,
I complain about stuff a lot like,
especially given how good my life is.
My life is really good and yet
every time I have to fly somewhere I have
at least ten significant complaints and
50 minor complaints.
And every one of those is an opportunity,
right?
Every one of those things
that don't go as well
as you would like are opportunities for
improvement and some of those seem like
someone else should see them and
it should be really obvious.
But they're often not.
There's this story I tell in our
internal on boarding process of me and
our head of product design going for
a walk in Vancouver.
And our office in Vancouver is in
a neighborhood that has really narrow
sidewalks.
The sidewalks have sandwich
boards from vendors out so
it's very crowded so
you kind of wind around.
And it starts raining
while we're on this walk.
And maybe two thirds of the people had
umbrellas with them and we didn't and
people are walking towards
us on the sidewalk.
And almost no one would move
their umbrella out of the way so
that the pokey things
wouldn't get us in the eye.
And like I said, the sidewalks are very
narrow, so we had to keep on ducking.
And I can come up with many
explanations for why this would be.
But don't explain with malice that
which can be explained by ignorance.
So it probably wasn't just that
these people have few avenues for
exercising power in their lives,
and they were choosing like this
moment to exercise some power.
But then there's only
really two explanations.
One is,
they just walk through the world and
they don't see that we're going like
this to get away from their umbrella,
even though inevitably they've
had that experience themselves.
Or they see that this is happening and
they're like,
I just can't think of anything I
can do to ameliorate the situation,
despite the fact that it's this tiny,
like a hundredth of a calorie,
worth of effort and
a tiny amount of consideration.
And the point of telling that story is
that that's the way, and this is a sad way
of looking at the world, but that's the
way that most people go through the world.
They're oblivious to the problems that
other people have and If they notice
the problems, they're unable to
come up with any kind of solution.
So like two thirds of the people
just didn't tilt their umbrellas.
Which means that if you're the kind
of person who's willing to tilt your
umbrella, there's a whole world
of opportunity out there.
>> Do you have any tips on how to notice
that you should tilt your umbrella?
>> Pay attention.
I mean, that is the biggest thing.
But pay attention
especially to other people.
So you can pay attention
to your own signals and
maybe if you get practiced-
>> This has actually worked
at the [INAUDIBLE] history when people
have needed to come up with a new idea
on the fly, it doesn't usually work,
but when they really just commit to
paying attention to other people and sort
of what seems broken, that has worked.
And then so if you do it this way you
come up with like thousands of ideas.
Anything you've noticed as a good way to
filter those besides just building up
the experience of seeing a lot and
the recognition?
>> Yeah there is the kind of
dialectic between execution and
idea because there's definitely
something to be said
when things are above some threshold
of good to sticking with the same idea.
Not so much because that idea's
the best one, but because the wild and
erratic chasing of the next idea
when you have something that has a little
bit of traction can be really dangerous.
So, Slack is a good example of that.
We conceived of Slack as a tool for
business.
We developed it that way,
we priced it that way, it was successful.
But, there's also this huge
amount of social usage, like now,
maybe a third of the Slack teams that get
created are for some kind of social usage.
And a lot of people had the idea
that we should pursue that,
we should make Slack for groups, or even
just pivot to being a consumer company,
like kind of a private version of
Twitter or something like that.
And we've been pretty steadfast in not
doing that because you can only do so
many things well.
And there's a lot of shiny objects
like that, that are very distracted.
>> Are you the one that says no
to most of those new ideas or
you have you had to build
that into the company?
When something's going as well as Slack,
there's so many new ideas.
>> I'm chief no sayer but
there's definitely a whole bunch of
people who are equally committed to no.
>> Great.
We can probably take one question for
Stewart if someone has
a really good one before-
>> Only a really, really good one.
>> Maybe that was so
much pressure now no one's going to ask.
>> [LAUGH]
>> So
you already had a successful
company before Slack.
I was wondering what the differences
between as a first time founder and
a second time founder?
>> What are the differences
as a second time founder?
>> So Everyone of the external
factors was much easier.
So the first time I mentioned that
we ended up making Flickr because we
couldn't raise any money and we're
desperate to try to finish something.
Fast forward seven years, and
we could raise as much money as we wanted.
So that was easy.
We didn't even make a DAC, we just said,
we would like some money and
then people gave us money.
So a lot of those things were easier,
it's always easier to get press,
it's easier get attention,
easier to recruit, easier to raise money.
It's not any easier to be successful
certainly because we started the company
that made Slack in 2009 and failed
pretty hard for three and a half years.
Kind of like pushing into
the wind of something that wasn't
ultimately going to work.
And we could have just stopped there.
It was great for
investors that we didn't recap and
we just kept the same company and so
that company has now been successful.
We could've also just walked
away from that company and
incorporated a new company.
And then it would've been my second
company was a failure instead of
my second company was a great pivot.
I'm not sure which things get harder,
I'm not sure than any do other than
the degree of self criticism and
like awareness of your own limitations.
Great, thank you very much.
>> No problem.
>> Appreciate you coming in.
>> [APPLAUSE]
>> Now,
we'll have Adam talk about metrics.
>> Great, so thanks for having me.
I'm happy to be here.
I'm going to talk about metrics and
measurement and
this whole group of issues that come
up as you're trying to start a company.
And, one of the things, I think is really
interesting about this topic that is so
much of what you have to do in
a startup is specific to the startup.
You're going to have to get to
be an expert at some domain.
You're going to have to get to be
better than anyone else at that domain.
And a lot of what it takes to do that is,
hard to teach in a class because it's
just different for every startup.
But, metrics is this domain
that's kind of general,
general enough to be useful
to almost all startups.
So, I'm going to start
off with this example.
So, this is the first version of
amazon.com from the early 90s.
[COUGH] And,
I really like looking at these.
You can look at the first versions
of lots of different products.
It's a good reminder that it's much
easier to get these things started than
you'd think.
Even these companies that are massive
today, if you just go and
look at how they started,
this is a good way
to keep yourself in touch with
the reality of how companies get started.
So, does anyone know why Amazon
decided to start with books?
Yeah?
>> Choice.
>> Choice, yeah, yeah, yeah, so
that's actually, I think what you mean is
that you have a lot of, you've got to
choose out from a lot of different books.
Yeah, yeah, so
it wasn't just that they chose books,
it was that-
>> [LAUGH]
>> It was books give you, so,
Jeff Bezos wanted to a start a store and
one of the most important
things when you're getting
started is differentiation.
So, you don't have to make a product
that's going to appeal to everyone
immediately, but, you need to make
a product that is going to appeal to some
people more than anything else.
So, you want to find some way
to differentiate, and usually,
you want to do that by doing something
that's going to be easy for you but
that's really hard for
all the existing companies.
And, the Internet enabled this in
Amazon's case because brick and
mortar stores have limited
space to hold their inventory.
And so,
you realize that as on the Internet,
you can have a big warehouse somewhere,
you can have items shipped from all over.
This is something that enables you
to offer much wider selection
than a normal store.
So then, you go and
run through the thought experiment.
You say, well, if what we can do is
provide better selection, then which
category, what kind of store is going to
benefit the most from a greater selection?
And it turns out that books is
a category where you have the most,
there are more books than there
are other products, right?
So if you look at like movies,
it's very expensive to make a movie.
That's not a industry that
would benefit immediately from
the Internet in the way that books would.
So, I think there's a general
point here that measurement can
turn a vague idea into a good idea.
And, if you do a lot of these kind of
like back of the envelope calculations,
it can really helps sharpen your ideas and
it can help you iterate and
move through the space of ideas a lot
faster than if you have to go and like,
build the prototype.
You can measure what Amazon started and
they started to focus on movies and
after a few years, they realized
that it wasn't working that well,
then they're set back by a few years,
now then there's competition.
These kind of metrics on,
even in the realm of ideas can
really help you get started.
And the one other thing I'd say about this
is that, I think using measurement here
is very good for helping you figure out
what products are going to be possible or
where you're going to be able to
provide a lot of value to a user?
I actually don't think metrics works
that well for stuff like market size.
So you just saw this example that Stewart,
they thought that the market size for
chat was 100 million a year in revenue.
It turns out to be 100 times bigger.
It's just very hard to
figure those things out.
I wouldn't spend a lot of time on that.
I think you want to know
that whatever you're doing,
someday there's some path for
it to eventually get really big.
But, you want to really apply your metrics
to how can you differentiate your product,
and what's going to be possible
that wasn't previously possible.
So, okay so next, now you're building
your product and what should you measure?
Just in any guesses, if you guys have
products, what are you measuring,
what would you recommend, yeah?
>> [INAUDIBLE]
>> The amount of users, yeah, yeah, so
what exactly would you
mean when you say that?
>> [INAUDIBLE]
>> Yeah,
so how many people use a total and
how often they use it and go ahead.
>> I think [INAUDIBLE] people share it but
it's viral, [INAUDIBLE]
>> Like whether it's growing or
whether they do something so
that it would grow, yeah?
>> Total addressable market like
when you're getting started.
>> Total addressable market, yep.
Yeah?
>> User attention.
>> Yeah, user attention.
So these are all great ideas.
And, there's lots of things
that you can measure.
As a startup, you really need to focus.
So the core concept I would try to focus
on is, users that are getting value today.
So, there's different indicators of this.
You might say,
you want to focus on active users.
People are going to be active,
if they're getting value.
You might want to focus on revenue.
That's a sign that people
are getting value.
In the case of a marketplace, where you
have things like buyers and sellers,
that's usually a case where it would
make sense to measure transactions.
Any time you have these like
different groups, if you say, okay,
we're going to measure buyers and
then we're going to measure sellers then,
it's actually lot of stuff you can do that
will make things better for the buyers and
worse for the sellers and
some of those things are good.
And so measuring transactions
is a way to unify them and
align your work on things that
are going to benefit both sides.
So, if you have a two-sided market or
anything like that then,
you want to measure transactions or
transaction values.
So things like eBay,
they'll measure total value of all
the goods that are sold on eBay.
So to start,
this is kind of a quick thing.
And you go a long way
just by measuring this.
If you're not measuring anything,
you're susceptible to all these cognitive
biases that Stewart talked about.
There's this all these kinds of problems,
you're just not going to
know what's going on.
So next was most important thing
to think about is retention.
So, first, I want to make sure
everyone understands what a cohort is.
So, a cohort is all the users who first
use your product in a certain time window.
So for a startup,
you might make the time window a week.
So you could say, there's a cohort of
all the users who first use your product
in the first week that it launched, and
you can track those users over time.
There's a cohort of all the users that
first use the product a week later.
And, you measure all these cohorts and
you can track them and
this tells you what's
happening to users over time.
And this is a really
important way to balance,
you're just measuring the total value
that all the users are getting so
This is a graph of total
of total cohort usage for
a sort of relevantly old Internet product.
So the blue line is users who
joined this product in 2004.
And that's as you can see, and so
this the y axis here is what
percent of them are still using it.
And the x axis is time.
So overtime you just follow the blue line.
You can see usually over
month there's fewer and
fewer people that joined the product
in 2004 that are still using it.
And this is usually the same story for
all these different Cohort lines.
So what do you think is going to
happen to this product in ten.
This is about,
let's say this is about six years.
The scale of this graph.
What do you think is going
to happen six years later?
Any guesses?
>> So kind of the opposite.
>> It's going to turn around.
>> Yeah.
>> Yeah, it could.
I think something would need,
something different would have
to happen to change that.
>> No one using.
>> No one will be using it.
Yeah, so this is,
it's hard to know what would happen.
I think probably everyone is
going to stop using their product.
I think some companies will
manage to turn these around.
But it's very hard too
because you get fewer and
fewer, you get less and
less usage from your old users.
It's also usually the old users
are what caused you to get new users.
And so
if you have this problem it's pretty bad.
This is a pretty scary place to be.
Even if you can have a ton of users
you can have total usage growing.
But if usage from every
cohort is declining
then you're going to
run into big problems.
So there's this idea called
the ring of fire which
is a way to kind of visualize this.
So unless there's some kind of lower bound
you're going to burn out all your users.
And so here's the analogy,
imagine there's a big field of let's
say of dead grass and
someone lights a match in the middle.
You're going to get a fire and
the fire's going to grow and grow.
But then in the middle of the fire you're
going to start to run out of fuel and
it's going to burn out.
And so you get this ring.
And the ring is going to get bigger and
bigger.
And in the center you're going to
burn out more and more of the fuel.
And so the fire will get bigger and
bigger but
eventually you're going to run out of the
whole field and the fires going to stop.
You don't want this to happen.
Here's an animation I made
to try to illustrate that.
>> [LAUGH]
>> So here's what you notice.
So it looks great at first.
There's more fire every frame
than the previous frame.
But in the end there's nothing, right.
So you don't want this to happen.
And it's really important to
internalize this concept.
Because if you don't you can fool yourself
and you can do a lot of things that
are going to cause you to get more and
more usage over time.
But at the expense of
the existing users and
if you don't do this eventually
your company is going to die.
I think some recent examples are I think
Groupon was a good example of this.
They got a huge tons of usage,
all this merchants but
it wasn't a good experience for
the merchants in the long term.
And so they all stopped using it.
And then I think more recently
Pokeman GO is another example.
It got really big, lots of users but
it just kind of faded away.
And so you don't want to build a fad.
You want to build something
that's going to last.
And so you need to make
sure that you're measuring
the usage from the existing users.
It's not just that you're
getting more users.
So you need to make sure the existing
users keep using your product.
Actually if this can go in reverse you're
in an extremely powerful position.
There's a small number of products
where the cohort usage actually will
increase over time.
One example is Whatsapp.
This is I think most messaging apps,
as you get more of your friends on the app
there is more people to send messages to.
And so they are able to get this
increase in cohorts usage overtime.
Uber on the rider side.
This is another good example as people
get used to Uber they use it more.
As the pickup times come
down they use it more.
As prices come down they use it more.
And so
Uber's in a really strong position there.
It's not true,
actually on the driver side.
So Uber has problems on
driver side because they
don't have this property there.
And then Facebook you can see
the numbers they report publicly.
They keep on their total usage is growing
faster than their number of users.
So you know that they must have this
increase in usage per user overtime.
So if you get to this,
this is very hard to get to this.
But if you get to this point then
you're in a really strong position and
one way to actually think
an idea is to whether.
As the product gets bigger
as you get more users.
As you develop the market is it
the kind of product where people
would use it more over time.
I think that's another lens you can
apply as you're considering ideas.
Just one more data point, this venture
capitalist named Tom Tunguz did
a study on what predicts valuations
that startups raise money at.
And it's just a simple correlation study.
And these are some different
factors in how much
they correlate to the valuation
that a startup's going to get.
First is growth of revenue,
and that's 0.18 correlation.
Second is just how much
revenue they have in total.
That actually Carl is a little
bit more strongly than growth.
But by far the strongest correlation
is account expansion which means people
spending more overtime.
It's the same thing as
retention increasing overtime.
So and
this correlation is extremely strong.
So for a complex number like what
the valuation of a company is going to be.
The fact that you can explain point
five four out of one of the variation
in that number by the single variable.
About whether the company is
increasing revenue per user over time.
That's a sign, it's almost like
there's just two classes of companies.
One class where there's
increasing revenue over time and
another class that gets totally different
evaluations where this is not happening.
So the main thing I just want you to
remember from today is you should measure
your retention.
That's going to lead you to do a lot
of things that will help you in all
kinds of ways.
It's going to help you,
it's going to help you grow.
It's going to help you retain your users.
It's going to help you build
a product that users really love.
So next I just want to talk
about exponential growth.
So retention is the most important
thing but it's not sufficient.
You need to have a product that's
going to retain your users.
But that's not necessarily enough to make
sure that you're going to be able to
build a big product and reach everyone.
You also need to make sure
that you are going to grow and
by far the most powerful way to
grow is through exponential growth.
This is just a few functions spotted
the red line is linear growth.
And blue line is I think
is cubic growth and
then the green line is exponential growth.
And the linear growth path this
you could do these with few things
like continually doing PR.
You can get more and more users.
Eventually, there's a limit to how
big you're going to be able to get.
It gets expensive to run advertising,
it get's expensive to do these kinds
of things that attract
a fixed number of users.
But if you can do things that cause you
to get users in proportion to the number
of users you already have, then that's
going to get you to exponential growth.
So there are other places to learn
about what to do to grow faster.
But as you're measuring your growth, you
want to make sure that the way you measure
it is aligned with what you want, and
if what you want is exponential growth,
then what you need to measure
is the rate of growth.
So a good way to measure this is
just the percent increase per week
in how much usage you have,
or how much revenue you have.
This is a tweet from Paul Graham
that I especially agreed with.
He said, a suggestion for really tough
founders is instead of graphing your
revenue, graph the percent
growth per week.
I don't actually know why he said that
this is for really tough founders.
I actually think this is what's going to
make you really tough if you measure this.
I think the causality
can go the other way.
So this is a very
important thing to measure.
The reason he says you
need to be tough is that,
it's going to be hard to keep
up exponential growth, and
it might not feel good to just see
this line that doesn't increase.
But I mean you can also
measure your total revenue.
There's no problem with doing that, but
just make sure that you're measuring
the percent growth per week.
Because a lot of the time,
there's going to be things you can do
that don't cause exponential growth,
that would actually help you
maybe more in the short term.
And you want to focus on the stuff that's
going to cause the product to grow,
in the short-term and in the long term.
Which is basically things
like viral growth,
getting users to invite more of their
friends, getting word of mouth spreading.
Getting some kind of dynamic where
the bigger your product gets,
the easier it is to attract
the next user to it.
So next I want to talk about iteration.
So one of the most important things, one
of the most important advantages you have
as a start-up, is that you can
iterate faster than a big company.
There's all kinds of things that cause
it to be hard for companies to change
direction, and you just don't have these
forces when you're a small company.
And so if you can iterate much
faster than other companies,
then you can go through ideas faster.
You can go through changes
to your product faster.
You can go through changes
to improve retention faster.
You can go through changes
to increase growth faster.
It's going to add up to a really,
really strong position.
But at the same time you have to be
iterating in the right direction.
And so one of the reasons metrics, even
just simple metrics, are so important,
is that they let you know whether
you're going in the right direction.
They let you know whether what you're
doing is working, and they let you out
of these different ways you can go
they kind of help you stay aligned.
One other point I want to make,
I think this isn't mentioned too often,
but you can measure how
fast you're iterating,
and you can measure all these
components of iteration.
So whether it's fully quantified or not,
you should have in your head, okay,
how long did this cycle take us, and what
could we do to make the next cycle faster?
At Quora, one of the metrics we track is,
how long does it take from an engineer
hitting commits on their code,
to us being able to run the unit test, and
then and
then get that code out into production.
And we try to keep that in like
between 10 and 15 minutes,
from a change in the code base to it
being totally out to all the users,
because that allows people to
iterate just much much faster.
And if we didn't track that metric,
there's just a lot of forces that
naturally would cause it to get slower.
I think a lot of companies end up,
they maybe start out like that,
then they end up releasing daily, or
they go to weekly, or even monthly.
I've heard of teams that are on like
every three months they do a release.
And you just imagine,
how could you possibly iterate if
your releases are only going monthly,
or even weekly?
So next, there's a spectrum of
how seriously to take metrics,
that I see a lot of companies stumble on.
So on the one hand, there's not
taking your metrics seriously enough.
And this might mean things like,
you don't have metrics.
It might mean that you have metrics,
they're on some page, but
no one ever looks at that.
It might mean that you have metrics and
one person in the company looks at it.
But no one else looks at it, and
the issues that happen on
this end of the spectrum are,
you kind of like, you can fool yourself,
you go around in circles,
you just like lose sense of what
the right things to do are.
And then there's another
end of the spectrum,
where you can actually take
metrics too seriously.
So you've gotta remember that
metrics are an abstraction.
There is where you're actually
care about underneath is,
let's say the value to the users, and
maybe the long-term value to the users.
And so a good metric will be
strongly correlated with that,
but it won't be perfect.
And so you don't want to forget that
there's a reality under the metric that's
not exactly the metric itself.
And so
if you focus too much on the metrics,
then you start to get into things like,
there's these ways to game the metrics.
And we can make the metrics go up, but
actually we're doing some things that
are going to hurt us long-term, and
those things are going to
outweigh the short-term benefits.
So I would really make sure
to be careful about that.
I'd say, personally, I think the ideal
is to be somewhere in the middle,
and I'd say you want to be very close to
taking the metrics extremely seriously.
Not too seriously, but
I think most companies bias toward not
being serious enough about their metrics.
If you're in the right, if you're middle
in the right spot then you're learning,
you're making good decisions,
and you're informed.
But you're still aware that there
is a product underneath, and
there is a strategy and there's more
to the world than just these metrics.
I see a lot of times companies
will oscillate between these,
like they'll say,
they'll get really focused on metrics and
then that causes them to do some
things that they think are bad.
And they're like these
metrics are terrible, and
we're going to throw away our metrics,
not focus on them at all.
And then you get into this world where you
don't actually know what you're doing, and
there's no accountability.
And so oscillation is not very productive,
you want to just make sure you find
the right point and try to stick there.
So finally, I want to just talk about some
of these psychological aspects that come
with metrics.
I think one of the big things that
come when you start measuring things,
is you just get closer to
being in touch with reality.
And that can be painful,
because sometimes the metrics are going to
be worse than you thought they were.
So sometimes they can be better than
you thought, and that's great, and
everyone feels great.
But a lot of times if you weren't
measuring something chances
are you weren't even aware of it, it was
hard for you to make it good, and so
it won't be good.
And so
that means that as you set up metrics,
there's going to be some pain,
and that can cause problems for
companies, I even think at the leadership
level, this can get even worse.
Actually, I think the majority of leaders
out there in the world rely on some
amount of delusion they're
creating among their followers.
To get them excited, to have them working
on this project in the first place, and
metrics can get in the way of that.
And so people get into this, then you get
into this question of like, okay, well,
what should you do as a leader?
And this is where I come out,
after thinking about this for a long time.
I think you want to be learning from the
past, so you want everyone fully aware of
what has happened, what has worked,
what has not worked.
You want to be totally in touch with
the reality of where you are today.
So if things are good,
you should know they're good.
If things are not good, you should know
they're not good, you should know exactly
where there are problems, and
where there are not problems.
But then, those things are not in tension
with being optimistic about the future,
you should be optimistic about the future.
Because there's so much potential, and
the reason you're doing a startup
is because of this potential.
And if you don't feel
confident that the path
you're on is able to get you to a place
that you feel optimistic about,
then that's a good sign that you
might want to change the path.
And you should do something about that,
because it's totally in your control.
So there's not really a tension
between acknowledging that things
are not in a great spot, and having people
not be optimistic about the future.
So as long as you just make sure that
the reality is that, and I don't think you
should make people optimistic about the
future in a way that you don't believe.
I think you should do the things you need
to do, to get yourself to be confident,
and optimistic about the future.
And then it's easy to get
people to follow through that.
So that's all I have, and
happy to take questions.
>> [INAUDIBLE] the question
with the IT's desk
>> [LAUGH]
>> Thank you very much.
>> [APPLAUSE]
>> All right, see you on Thursday.