How to Build a Product I, Michael Seibel, Steve Huffman, Emmett Shear Video

So this is going to be the first of four
lectures on how to build a great product.
I apologize because none of the people
here today can speak to that topic.
>> [LAUGH]
>> [LAUGH] My name is Michael.
I'm the CEO of YC and
help run the batch and
program as most of you
understand YC to be.
This here is Steve who's
the co-founder and CEO of Reddit.
And there's Emmet who's cofounder and
CEO of Twitch.
Can you guys introduce yourselves
in a couple of seconds?
>> You just did but I can do it again.
>> [LAUGH]
>> And Steve, yeah co-founder and
CEO of Reddit.
I study computer science.
I graduated from UVA in 2005.
I was at Reddit for five years,
I left for five years.
I've been back for
about a year and a half.
>> My name is Emmet I graduated
from Yale computer science in 2005.
I started my first company
right out of college.
We made a bad copy of Google calendar
before Google calendar existed.
And-
>> Google calendar
>> Yeah, that's right, and
then we started Justin TV which
is the company that one turn and
so I've been doing,
basically continuously employed
working on that since 2006
>> And
then can you just, each of you guys say
whatever the KPI you guys track for
the business and approximately what it is?
Just to give folks a sense
of why you guys might be
experts at building great products.
>> [COUGH] One of our more shareable
ones is monthly active users and
we hit 300 million last month.
>> Nice.
>> [LAUGH]
>> That's a good number.
>> Our primary metric that we care about
is a metric called minutes watched,
the number of minutes of video that
are consumed every month on Twitch.
And we just crossed 30 billion minutes.
This year for the first timers.
This is very exciting.
>> Total?
>> No, per month.
>> Jesus.
>> [INAUDIBLE]
>> So
that's the main thing we
look at rather than uniques.
>> So, what we're going to do for
you guys today is I have about 12
questions I'm going to ask these guys.
We're going to see how many
of them we get through.
And then we're going to leave
the last ten minutes for
questions about start up stuff, YC, or
whatever else you'd like to talk about.
My goal is to kind of focus this
advice on early, early stage.
So, these guys do a lot
of stuff now that really
is completely irrelevant to you
>> But hopefully they'll be talking to you
about the things they did, bad or
good, when they were starting.
So, I want to start with
one of the major tenants of
YC is how you talking to your users.
And we repeat that a bunch of times, and
then we don't really go into any detail.
So, can you guys talk a little bit about
how you talk to talk to your users?
Maybe the good and maybe some of
the mistakes you've made as well.
>> Sure, so, I, most of the advice I
give today will have two perspectives,
one at Reddit and one at Hipmunk.
because the companies are completely
different and had different paths and
I think, kind of the ground rules for
this discussion,
the first piece of advice I would give
is every company is really different,
and there are lots of
different paths you can take.
And what works in one place,
very likely won't work in another.
So at Reddit, I mean the product
is talking to users, right?
That was built in.
In the early days we didn't have
comments and we didn't have communities.
So actually,
we got a lot of emails, was our,
for the first probably six months was
the way we communicated to our users.
But when we added the comment feature,
there was basically no option but
to talk to our users.
And the challenge then becomes which
users do you want to listen to?
And what are they actually saying?
And what do they actually mean?
Because they're often saying one thing and
they mean something else.
At Hipmunk, because it was selling
plane tickets and hotel reservations.
It was completely different.
We didn't really have a good forum for
connecting with our users.
We used a product called Olark which
was basically this imbedded Java Script
chat thing.
And that was indispensable,
we still use it.
It was users, if they got stuck,
they could ask us for help.
If they wanted to give us compliments or
criticisms, they could do that.
And we can often, both over email at
Reddit and over Olark at Hipmunk,
turn Like emotional users, angry users
will often become your most loyal
users if you can flip them around, there's
like this absolute value of emotion.
And so, if you can find those angry
users that are having a bad time and
treat them well they will likely.
Quickly turn into some of
your most loyal supporters.
So, we found that whole
cycle to be very valuable.
>> The interesting thing for me,
between Twitch and Justin TV,
which are two very similar
product right there.
With live video products with chat.
So for Justin TV, we were our own users.
We were building a product for
us to build a TV show.
And after that,
we really sort of declined to actually
speak to users in any significant way.
And, unsurprisingly, all of the good
ideas we had were things we came up with
when we were building it for ourselves.
Which sort of brings me to my first point,
of talking to users, is it's possible,
especially very early,
to build a product yourself.
And, if you truly are building
something that is for
you to use, that can be super effective.
You don't necessarily need to
talk to talk to anyone else.
Eventually, you'll need to, because you'll
learn about people other than yourself but
for the first three months it's to develop
something, just something that you want.
But you have to actually really want it.
It can't be, you can't be like
lying to yourself about that,
it can't be something you think is cool.
It has to be something you, yourself
actually desperately want to use everyday.
And then we had went through a long
period in the desert where we were
wondering about.
Not talking to users,
not really using our own product, and
not building anything of value.
And the thing we did with Twitch
that I thought was really good for
production users is we
talked to them in person.
I had done some amount of surveys and
talking to users over email.
And I found that getting people on,
at least on Skype, or if not,
literally in person changed my depth
of understanding of who they were.
The other thing I would say about about
it is that we focused down really hard on
which of our users we cared about.
And we decided it was the broadcasters and
not only was it the broadcasters,
it was the successful broadcasters.
We didn't want to talk to
people who had two viewers.
We wanted to talk to
people who had 200 views.
And so, we went and talked to a bunch
of broadcasters on the platform who
decided they were the important ones.
The other thing we did that was I think
really valuable is we decided it wasn't
just people who were currently using our
service it was people who we wanted to use
our service.
And so, we talked to a bunch of people who
were streaming on other platforms rather
than just already on Twitch.
Because the people who have
chosen to not use your product,
especially if they have deliberately
looked at your product,
decided it sucked and
went somewhere else, are like,
some of the best people to talk to,
because they know what's wrong with it.
And so, I actually think the most valuable
things I learned, especially early on,
were from users who had tried Justin TV,
they had tried UStream.
They tried own 3D, they tried YouTube, and
they had a coherent opinion as to
why ours was not the one they chose.
And that was super valuable,
because they really pointed at
the details of where we were wrong.
>> And often those details
are smaller than you think.
>> It's like memorably one of
the biggest issues was we were
not listed on
the Team Liquid Starcraft fan website.
We lost maybe 3,000
broadcasters over that.
We had to email them and
ask them to re list us,
because they just didn't have us there.
Tiny issue, but
like really mattered to our users.
>> It's easy to run a company for
years and then be like,
I should have fixed this stupid one day
fix if you don't talk to your users.
>> So a lot of people talk about MVPs,
minimum viable products.
How many of you have heard that term,
MVPs?
So one of the challenges I have is
when folks like you apply to YC and
get into YC even though you know the idea
that you should be releasing something
that is crappy, you don't do it.
And as a result, users slow down.
And oftentimes,
that prevents you from getting into YC or
prevents you from building and
talking to customers.
And so, I think that what's
interesting is that just TV and
Reddit are pretty good
examples of crappy MVP's.
So, can you guys describe how
the products worked in the beginning or
maybe how they only kind of worked?
>> Yeah, we started, I think I wrote
my first line of code on June 3rd or
4th, 2005 for Reddit and
we launched on June 22nd.
I didn't even launch it.
Paul Graham linked to us from
his blog without telling me.
>> [LAUGH]
>> And from then,
it was on and
the good thing about that for
us in particular is we didn't
actually have a vision for Reddit for
better or for worse which would affect us
in a lot of different ways going forward.
But as soon as we launched, that's when we
started to find the path and we just would
kind of follow the users everyday as
what do we think is best for the users?
What do we think is best for us?
And just build towards that and
we've built a lot of loyalty
through those actions and
that would not have happened, if we were
sitting in our apartment not launching.
Because if we launched and then for the
next six months, we just added a ton of
new features, only about 25% of which
lasted more than a day or two, but
there was stuff we
though was a great idea.
Like we had these categories,
I remember in July at some point.
I made Alexis my co-founder stay up all
night one night categorizing every link
that'd ever been submitted to Reddit.
And the next morning, I was like we lost.
It's not working turn it off, but
we could've probably, if we had launched,
we could have had that feature until
November and then launched and
then wondered why it didn't work.
>> I think that Amazon actually has
a really good saying about mineral and
viral products.
They like to call them,
what they're aiming for
a minimum remarkable product which I
think gets more at what you're trying to
accomplish which is a viral product bring
into mind this whole ecosystem of support
like a viable organism that's got
all the things it needs to live and
you don't actually want that.
Like if you're launching something that's
actually self-sufficient, that's viable in
a this thing could live on its own sense,
you definitely waited too long.
This thing is going to be on live
support when you launch it and
you are the life support system.
What you actually want is
you want something about
the product when you launch
it to be actually good.
You want to get to the part where you've
built something that maybe at least one
person in the world finds remarkably
useful or remarkably good and you don't
know really if you've gotten there
until you put it in front of someone,
which is why you need to launch it so
quickly.
I think it's really important to
remember Dropbox, for example,
one of the big YC successes did not launch
an immediate product, because they were
building a backup system and backup
systems aren't allowed to lose your data.
[LAUGH] And so, the last thing you want to
do if you're building backup software is
go rush something out that's
going to lose your customer's data,
because you will never
get that trust back.
And so, the minimum remarkable product for
Dropbox was a video.
Basically, saying, look at this awesome
thing we were going to build and
getting all the senate for a newsletter
and that worked, because the idea like
the visual idea of what Dropbox is
going to do was super compelling.
That would not have worked for just in TV.
We really need to get something out
in front of customers Immediately.
Most importantly, because we all need to
push the show in this idea we are going to
build the show and
we basically did the minimum possible work
may be a little less than that
to get our show on the internet.
And when we launch it,
it like could only have one channel.
The way that broadcasting worked,
the only way we could
actually get it to work
consistently is we had a Mac Mini
running Parallels which is
a virtualization environment for
Windows XP where we had this
piece of a broadcasting
software called,
it had got bought by Adobe later.
Onflicks?
OnLot?
I can't remember.
Anyway, that we were automating
with Windows shell scripting
to pick up the right camera input and
then auto reboot if it went down,
because it broke all the time because
it was really buggy software and
that extremely jury-rigged thing worked.
It was like put a video stream on
the internet and it was not a scale.
We could not scale that up
to thousands of streams.
It was impossible, but we got something
that at least was compelling to one person
onto the internet, that was compelling to
us and that was a critical step for us.
because before, we did that.
We basically weren't learning anything.
>> At HitMonk, we again,
started in June 2010 and
I'd given my co-founder, basically
yhis ultimatum of what our MVP was.
We have to have legit data,
because we could scrape anybody.
We needed legit data, real data.
And when a customer buys a flight,
we get revenue.
And I'll say,
if we can do that in three months,
we will continue with this business.
And if we can't launch that in
three months, we're calling it.
And that for me, that MVP,
that was that minimum useful thing.
Like if we had a customer
who was giving us money,
that meant we did something right.
And unfortunately, we actually
achieved that in three months, so
I worked on it for five years.
>> [LAUGH]
>> But
I think it's good, sometimes to set
yourself deadlines and specific
milestones to keep you focused and
keep you pointed in the right direction.
>> So another thing that comes up
a lot and this is actually one of
the things that hits the, when a company
gets into YC is we will often ask them,
what are they doing for analytics?
What are they tracking?
And I think that this is another one of
those MVP like things, everyone knows they
should be doing it and it's the first
thing that gets caught off of any list.
So, can you guys talk about
how an early stage startup
should think about measuring?
What analytics tools should they be using?
And generally, like how you incorporate
tracking into building a product?
>> I have two very different answers,
so reality is somewhere in between.
At Reddit, we just didn't.
We didn't know our traffic for years.
We didn't measure anything.
We did the product development by
intuition for a very long time.
The company ended up working out.
I'm still now quite sure how many
users we have and it's a huge,
huge pain in the **** now.
>> How many users you have?
When you get to 300 million, it's like-
>> [LAUGH]
>> No, in all seriousness,
we have no idea how many unique
people use the site every month.
>> It's probably somewhere between 250 and
320.
Yeah.
>> Anyway, at Hipmunk we were actually
really good about it from day one.
And we were really because we didn't
have that kind of instant feedback with
the users we actually got really
disciplined about testing, and analytics,
and measuring, and watching.
And that company lasted honestly,
quite a bit longer than I think
it would have without doing that.
And so if I can do it again,
there are a lot of corners you can
cut when you're building your MVP.
You can create all kinds of tech sins and
corporate sins and I don't know,
be completely dysfunctional,
your users don't care.
But data debt can haunt you.
because you can't make up missing data,
and
eventually your intuition's
going to fail you.
So I would think very carefully about
what's the minimum viable data.
You don't even have to look at it.
Just log it somewhere so
it's there when you need it.
>> To reinforce that point,
having historical baselines for
important user actions,
you will thank yourself later.
Every time I've ever been like, right,
how many people are using that feature?
And we don't know, and we can't find out.
And now even if we find out how much it is
now, we don't know if it's up a lot, or
down a lot.
And we have no idea what the trends are.
It's awful and
you can't make a decision for
another three to six months
as you build that baseline.
Actually I got the best advice about
this from Sue Hale, who runs Mixpanel,
when he was convincing us to run
Mixpanel for the first time.
Which, by the way, didn't work, because
Mixpanel at the time was only six month
old, and
it broke when we tried to everything.
But, we went back to it-
>> Thanks for
convincing me to use Mixpanel
before you came to that conclusion.
[LAUGH]
>> You're welcome.
>> It works better now.
>> It works a lot better now, we went back
to it, we're still on Mixpanel today.
We use a bunch of other stuff to, because
we're a little too big for just Mixpanel.
But Mixpanel,
the advice Sue Hale gave me that was,
I think, more important that
the software we were using.
Mixpanel in specific, even though
I do think it's good software was,
pick your top five to seven most important
user actions, whatever those are, and
just log those.
You don't need to log everything,
you really don't.
Most of the things people are going to
do with your product aren't important.
There's maybe five things people can do.
Like for Reddit, Reddit might be vote
on something, comment on something,
submit a story, load a page,
I don't know some short list of things.
And those are important and you can
kind of just ignore everything else.
If they're not instrumented who cares.
And so we instrumented,
I watched a minute of video,
I sent a chat message,
I followed a stream,
and I bought a subscription.
And that turned out to just give, those
four actions gave us a huge amount of
insight into what was going on with our
customers, that we hadn't had before.
>> This is one of those areas where, if I
could go back and give myself just advice
for 30 minutes, that would dramatically
affect the trajectory of the company,
it would be around this topic.
And so if I were in your shoes,
I would just take half a day, and
just read best practices for
collecting events and storing them.
>> But and also, I know another thing I'd
recommend is use a third party service.
Use, collect logs, by all means,
put them into your own system.
because I guarantee you,
you'll want to do something a third party
service doesn't allow at some point and
you'll be super annoyed if your data's
only in the third party service.
So collect logs or whatever,
but just in the beginning,
you don't have time to mess around
with building analytics tools.
So just use Mixpanel or
use Google Analytics or use whatever.
Anything, they're all better
than what you need right now.
Later on you'll need
something more complicated.
You don't need it today.
>> So one of the challenges that
happens usually after a company,
in the early stages,
starts talking to their customers.
Is they realize maybe there's a big
disconnect between what they thought their
customers wanted and
what they actually want.
And at YC this usually comes to
us in office hours in terms of,
we call the big redesign.
So everything about our site's incorrect.
We just basically need to
rebuild it from scratch.
Or rebuild huge components of it from
scratch in order to start serving our
customers.
Needless to say, this is a very,
how do I phrase this?
Distracting and
oftentimes not fruitful exercise.
But how do you see that type of challenge
of, man, we got this feedback and
we think we need to be going
in a different direction now?
But how do you deal with
that kind of challenge?
>> That's a big one.
And I think you need to be clear whether
you're talking about tech or the product.
I'll repeat what I said before,
is your users don't care about your tech.
So if you can move forward,
leave that for another day.
When you're talking about
redesigning your products.
You're probably in trouble,
because whatever habits led you to build
a product your customers don't like.
If you haven't addressed those habits,
you're probably going to build another
product your customers don't like.
So I would make sure that, because
I've been through lots of rewrites.
At Hipmunk we rewrote out flight and
hotels probably half a dozen times each.
At Reddit,
we re-wrote the tech a bunch of times.
And we're redoing the UI and
UX now with better habits and better data.
And so you want to make sure
that you know something new.
But honestly, if you're in that situation,
I would also go through
another mental exercise of okay,
what's the new MVP, right?
It comes up in tech a lot, right?
The second system syndrome.
The same thing happens in the product
where you're like, all right,
I've got this vision.
I'm going to fix all of the products,
all of the mistakes at once, and
it's going to be perfect.
That product never ships.
It never ships.
So can you iterate your way there, can you
just start from scratch with an MVP and
build it back up?
Those are much more likely to
actually get you somewhere.
But they're also harder to do, and
requires a little bit more discipline.
>> So we've also gone
through a bunch of redesigns.
Big redesigns where you have
to rebuild the whole product.
They've been mixed actually.
We've had some that were super successful.
Twitch did a big redesign of our
mobile app, maybe three years ago,
from the first version that had
basically been a cloned version of
the Justin.tv app to
a completely new version.
It's basically the version
you can use today.
And that increased engagement by 35,
40% per customer when we launched it.
It's huge, changes that make that big
of an impact are actually really rare.
>> Usually data errors.
>> Yeah, yeah, [LAUGH] yeah,
they're usually data errors.
And in this case it wasn't.
In this case,
it actually was a huge increase in usage.
And so absolutely,
redesigns can make a big difference.
But we weren't doing the big Twitch,
we didn't do that redesign because we'd
discovered we couldn't move
forward on our existing product.
Since we got some insights is
the what what things should
be presented at the top level of a nav.
And we re-did the navigation to present
those things at the top level instead of
burying them three clicks deep.
Three touches deep, I should say.
Most of the time when I see people doing
a big redesign it's because they fall into
this trap that engineers and
project managers often fall into.
Which is,
we have to fix this eight things.
Let's just fix them all
at once it'll be easier.
That's wrong, you should just fix
them each one at a time individually.
If you have eight things that are wrong
with your product that's awesome,
you have this huge great list of things.
Just fix them one at a time,
fix everything one at a time.
Crank them out quickly,
you'll get there way faster one change at
a time than you will in a big rewrite.
That'd be great for
a really early stage startup is almost
equivalent to pivoting the company.
And we generally advise against it.
>> It's this weird sort of
programmer product procrastination.
If I don't know what to do,
therefore I'm going to invent
this big project to work on.
>> [LAUGH]
>> We don't need to use that library.
I'm going to build it myself.
>> [LAUGH]
>> Very common, use the damn library.
>> [LAUGH]
>> So I'd like to talk
a little bit about decision making.
So maybe after you talk to your users but
before you decide what to build, usually
there has to be some type of consensus
building process either function or
dysfunctional between the co-founders.
And what are some tips or
tricks on how to come to a consensus?
And then maybe more importantly,
how do you deal with the contrarian,
how do you deal with
having to make a decision where you
know there's going to be a disagreement?
>> I think the first thing you
said is actually super important.
It's something I want to reinforce.
You talk to your users first, and
then you have the ideas about the product.
Almost everyone does it
in the opposite order.
It's called product validation.
If you find yourself thinking, I'm going
to go talk to my users to validate my
product idea,
you've gone horribly off the rails.
You do it in the other order, you don't
talk to users to validate your product
ideas, you talk to your users
to have your product ideas.
And so
I would just have a rule in general for
people who want to have
a voice in product design,
is that they better talk to the users and
look at the data too.
If you haven't talked to users and
you haven't looked at data, you don't get
to have an opinion about the product.
I'm sorry, the person who's actually
done the work gets to have the opinion.
You can have ideas but
they get to make the call.
But in case you wind up where both
people really have talked to the users.
And both people really
have looked at the data.
And people still disagree
as to what we should build.
Which by the way is
actually much more rare.
Once both people have
actually talked to users and
actually looked at the data, it's usually
relatively easy to get to consensus.
But it will come up where
people still disagree.
I've found it's better when you
just have someone who's in charge.
And at the end of the day,
you have someone who's the CEO, and
everyone agrees this person's
judgement will be trusted.
And you let them make the call.
Don't avoid conflict.
Talk about it.
Argue about it,
but then you have someone who just gets
to make the call, because anything else
leads to just decision making
that's far too slow for a startup.
>> Yeah, maybe it's because
I'm usually the one in charge,
I just feel like I don't
have a lot of disagreements.
>> [LAUGH]
>> [LAUGH]
>> There are-
>> Meet the CEO.
>> Yeah.
Largely, I find the situation I'm in,
I say one of two things a lot,
which is I don't want to argue
about it if we can just test it,
and I'm very willing to
be surprised on this.
I'm probably right, but
this isn't getting into my top three
important things to worry about today.
So let's just see.
It's easy to do that when you have
more resources, to let's just see.
But I think about,
like Ciss and I over the years.
And Adam and I over the years at Hipmunk,
I think as it panned out,
we've very rarely had these philosophical
differences over with the product,
sometimes it was timing.
Should we build this thing now?
What's the prioritization of these things?
But we always agreed on general
mechanics of what we were doing and
why we were doing it.
And so the strategy might change but the
details, we're very rarely not emotional.
So those are getting emotional,
I think there's probably
something else structurally wrong.
You might be swimming upstream.
That happens a lot.
The arguments have gotten that rid over
the last year like when we try to promote,
we do cross promotion from mobile web
to the app, it just doesn't feel great.
It works.
It works and
we like the numbers going up but
it's not really with the user's best
intention in mind all the time.
Sometimes it drifts into this like
it's good for the company but
maybe not for the users.
And when you're in those situations,
that's usually when
the disagreements come out.
And you're not really arguing
about the future anymore.
You're arguing about short term or long
term, or best for the business or best for
the users.
And then when you beat at it,
and beat at it, and
beat at it, eventually you come to a
solution that actually feels right, right.
That meets your intuition, you don't feel
like you're fighting the technology,
you're not fighting the users anymore.
Those are usually
situations I look out for.
>> Sir, I want to get to you
guys telling product story but
I want to cover one topic first which
is that lets take people passed lunch.
Well now you need to have some type
of process around talking to users,
looking at data, making a decision,
building a product, releasing a product.
Checking your success and repeating,
product development cycle.
What are some advice you can give to early
stage startups around how to set that up,
what that cadence should be,
maybe some pitfalls to avoid?
>> The first, just to take one
little step back is, I see a lot
of startups, they get too emotionally
high when the numbers are going up.
And they're not investing
in this sort of process.
Because growth masks all problems.
And then at some point
that kind of stops and
you don't have any of these mannerisms.
I also see startups get a little too low
when things not going right at first.
And they kind of psych themselves out.
So you want to live in that in between,
right?
If the data's really good,
well, check the data.
Right, if the data's really bad,
check the data.
[LAUGH]
>> [LAUGH]
>> Okay, so then in terms
of the actual process, first of all it's
going to be different at every company and
it's going to change, as you scale.
I think it's just most important
to have a way of working,
to have if you just want to start
out really, really specific of.
The way I like to think about it is
where do we want to be in a year?
Okay, let's draw a straight
line from that to here,
what do we need to be doing
today to get there in a year?
So that's why I like our hypothesis.
Do we need to validate
anything about that?
Probably a few things here or there.
Do that, go to check out.
Yes, and
then it's just like straight line.
And I like to work on two week cycles.
Check in every week on
basically that kind of thing.
Where do we want to be in a year?
Are we still on track,
are we still doing the right thing?
Does it still make sense?
Are our assumptions still
holding to be true?
If not, pop it up a stack.
And then you give people whatever it is.
If you're early startup,
you give them 13 days to work.
And if you're a more mature startup,
you give them nine days to work.
[LAUGH]
>> [LAUGH]
>> And then keep going on at that.
I found that that's worked really well for
me.
>> One thing I found actually is that
if your goal of talking to users
shouldn't be course corrections
on a week to week basis.
I did a whole crap on a talking
users at the very start of
trying to figure out what the users
wanted, how to think about them.
But then actually, we didn't because we
had other reasons to go talk to users,
but we could have gone six months
without talking to a user,
because actually we
knew what users wanted.
They wanted to make money on the service.
Streamers wanted to be able to make money,
they wanted to reach more viewers and
they wanted positive social feedback for
what they were doing.
And you kind of just check
anything you were doing,
is this going to deliver this
to our streamers, yes or no?
And if so how much?
Great, you don't have to talk to anybody.
You already know what they want.
And you need to go back and
revisit it and sort of talk to users
periodically because you need to
understand more of the nuances.
The things about your service that are
bugging them [INAUDIBLE] doing something
really cool.
But the goal of talking to users is not
to get them to tell you what features to
build because users
are really bad at that.
They actually have no idea
what features to build.
[INAUDIBLE] users is to get to
understand them really well.
And users don't change that fast.
So I personally prefer more
in-depth time With my customers,
talking to them for a while.
And then, depending on the size of
the thing we need to do or we're trying to
move, we might go just work and build and
look at metrics for the next six months.
And not actually talk to
users that much for a while.
Because if you know that what
people want is to make more money,
you can spend six months optimizing
the sell through rate on an object or
on how much advertising you have.
And you just can have faith that it's
going to be good and you will be rewarded.
>> I think the classic advice
that's very difficult to follow is,
you need to have the courage
of your convictions.
So, you decide what you're going to build,
it might take six months to build it.
And at Reddit we just kind of
do it in front of everybody and
get flamed by users the whole time.
Which is fine, I'm used to it, but I have
to kind of tell our product people like,
just keep pushing, keep pushing.
Just ship a little bit every week, and
then in six months, they'll get it.
And at the same time,
the completely contradictory advice is
you have to know when you're wrong.
You have to be able to
identify when you're wrong.
You have to be able to say, all of
you people are wrong, and I'm right.
And then at some point you have to decide,
wait no, you're right.
>> [LAUGH]
>> And if you can walk that line, you need
to have this like good combination of high
ego and high humility and balance those.
But I always try to just kind
of run this process in my head.
Wait, am I wrong?
Nope, they're still wrong.
>> [LAUGH]
>> As much as I said don't talk to users,
you don't have to talk to users every
week, it's totally unnecessary.
You do need to look at your numbers
every week, you really need to know.
And you don't need to look at them every
day, it's actually like pointless and
almost just ego gratification to
look at your numbers every day.
But you should look at your
numbers every single week, and
you should confront, are they up or
down and ask yourself why.
And the holy grail of product design
is for your numbers to move and
for you to truly
understand why they moved.
We still struggle with that.
Like, there have been points where
I feel like I understood it, and
then it slips away again as
the business gets more complicated.
But you're always trying to be able to
explain fully down to like atomic ground
truth, we are up 20% this week because
we got this many more new devices in.
And our retention rate is
this on those devices, and
this is the conversation rates.
And if you do the math and you can explain
it all the way down to this new feature
moved this conversation rate on this page,
by this amount, and therefore we are up.
And if you can really do that,
it's super powerful.
It's also really hard.
But you'll never get there if you don't
build with the intention of looking at it
every week.
>> So in the last five minutes
before we open for questions,
can you guys walk through a feature
that is visible on your sites now?
And I'll let you choose,
that's either bad or good.
And talk about the process
of how it came to be.
>> I have a lot.
>> [LAUGH]
>> I'll talk about
one of my favorite ones.
This is from the early days of Reddit.
So the early days of Reddit,
it was just links.
Everything was external.
And eventually we added comments,
probably about six months in.
The first comment was,
this is the end of Reddit.
>> [LAUGH]
>> The first of,
that would be the first comment on
every product we've ever released.
And users started doing this
curious thing where they would, so
on any post you could go,
you could click on the link or
you could click on the comments page
to go to the comments for that page.
Our IDs for posts are just
a counter that's in the URL.
So users would submit the URL that
would represent the comment page for
the URL they're submitting, right?
So that would just increment,
they would take the newest post and
increment by one and submit that URL.
So that you had a post that would
link to its own comments page.
And it was really cool, so
we called those self posts.
And users were doing this, and
I remember watching them do it thinking,
this is really cool.
I would have never thought to do this.
Sometimes they'd guess wrong, right?
And so they'd submit a post to
somebody else's comment page.
>> [LAUGH]
>> Which was funny.
And so because our user base at the time
was fairly technical, the way we made this
work is when you're submitting a link, you
would just type in in the URL area, self.
And it would automatically do that
little circular thing for you.
And that was the beginning
of self posts on Reddit.
Self posts are now 60% of our content.
>> [LAUGH]
>> I don't know, a few million a day.
And we would've never built that feature
if we hadn't been watching our users.
I don't think I, I mean I'm
explaining it now, but if I explained
that feature without it existing I don't
know if it would make a ton of sense.
And users, most of them didn't
get it at first either.
They were just like,
comments are destroying the site and
there's not even a link here,
it's just comments.
And it just kind of goes to show, I think
the best product ideas are the ones
where your users doing it anyway,
and you just grease it a little bit.
That's when I talk about before where
you're either swimming upstream, or
just everything just feels right.
That was a classic example
of it just felt right.
And it completely changed the site.
>> I think my favorite story here
is an advertising product actually,
which is funny.
So all of our users told us
really really clearly, we want to
show advertising and
make money off of our streams.
It was a clear desire,
everybody wanted it.
And they also told us their
least favorite thing about
our competitors' products was that they
would show these ads over the stream, and
it would be disruptive to their
user's experience, and they hated it.
And it was very interesting getting
these two pieces of feedback because
they both really wanted us to put ads on.
And sometimes, literally,
sometimes different people,
sometimes it was literally
the same people.
Their biggest complaint about
the opposing service was,
competitive service is annoying
my viewers with all these ads.
And this was like,
this is a hard nut to crack.
Because you kind of need to solve for
both,
you're not allowed to just give up
as a product manager and pick one.
And I'm really proud of our solution,
which was, we gave them a button.
We inverted it, and
we gave the streamer the ability to hit
a button to run ads on their own channel.
And, like it's funny because
normally you'd think
of that as something that will, if you're
uploading content to like Facebook,
you would never press a button on that
to show more ads to your friends.
But because we were cutting
them in on the revenue,
they were actually heavily
incentivized to hit this button.
And we basically got this paid workforce
of people placing advertising
in the downtime of the stream.
And I'm really proud of
that clever solution and
I think it points at something that I
think is a really important principle of
product design in general.
Which is, that you want to just try
inverting all of your assumptions.
We got stuck on that
advertising problem for
a long time because we didn't want to make
it disruptive to the customer experience.
And we couldn't figure out how to automate
putting it on the stream in a good way.
And at the same time we really wanted
to show ads on the video stream.
And we had
this implicit assumption in our heads that
we had to figure out where to put the ad.
And once we gave that assumption up,
it was much more clear
what to do going forward.
And I think that's a, and
it's one of the most, I don't know,
I'm just really proud of that product
today because it was one of the things
that made our broadcasters super happy.
And no one, like you can't complain,
you hit the button.
>> How many times is that
button clicked every day now?
You know, that's a really good question,
I don't know the numbers off hand, a lot,
it's clicked very often.
People really like that
button because it's a,
click the button make $15 button and
like, you know.
>> [LAUGH]
>> But
they're also aware of the trade off.
Which is that their viewers don't like
it when they hit the button too much.
And so by putting that in their court,
they get to make the optimal trade-off for
their stream and their situation.
>> All right, those are my questions,
I want to hook it up to you guys.
Happy to talk about product,
happy to talk about YC stuff,
whatever you'd like to chat about,
go ahead.
>> What about the process of discovering
you've had product market fit.
Getting past that launch pathway and
discovering there's something sustainable,
beyond this real core group of users
that are really committed to you?
>> Just to repeat the question, how did
you know when you had product market fit?
So at Reddit, this is the nice scenario,
we were growing.
Despite not knowing anything
about our business,
knowing little about our users,
not really knowing how to run a business,
not having a ton of product sense,
being down a lot.
We were growing.
So that's the nice scenario,
that's a good one.
At Hipmunk,
our natural state was zero users.
I'm not sure we ever
got product market fit.
The users who came to Hipmunk liked it,
but
we had to fight for
every single one of them.
It's possible to do that, but we were,
again, I think that entire company,
we were swimming upstream.
And I remember the day we
kind of realized that.
We ran the survey of like,
why do you use Hipmunk?
And we put a lot of
effort into the product,
we thought it was a really good product.
Legitimately you can find a flight a lot
faster on us than our competitors.
35% of people said it's
because of the logo, and
50% of the people said because we
had lower prices, which we didn't.
>> [LAUGH]
>> So, [LAUGH] I mean it's like,
we made it.
The company ran for five years.
And I ended up selling,
which is fine, but yeah.
I think we were really fighting that one.
>> I think this is a huge misconception,
by the way.
Most of the companies that apply to
YC don't have product market fit.
Most of the companies on demo day
don't have product market fit.
Most companies die before
product market fit.
A lot of people talk about product market
fit as this organic step in the process.
You raise angel,
then you get an office, and
you hire some people, and
you have product market fit.
And it's just like, you would be
surprised at how few companies.
Product market fit is typically described
as, you are overwhelmed with demand for
your product.
>> It's really obvious when
you've hit product market fit.
So your product is bad, you can point at
seven things that are broken about it.
And you have all these features
you know you need to build.
And weirdly,
it's growing really fast anyways.
It's really obvious when you
have product market fit,
your product is growing like crazy.
And you are spending, you don't get to
fix any of the things that are broken or
bad about your product,
because you're spending all of
your time fixing it as it scales.
And you are, so the metaphor I use for
startups is kind of Sisyphean,
you're pushing this boulder up a hill.
And you're pushing the boulder and
you're pushing the boulder,
you're pushing the boulder.
Product market fit is
when you crest the hill.
And your job doesn't get easier, but
it does change a lot, because now you
are chasing the boulder as it tries to
roll away from you down the other side.
And now you're springing flat out, trying
to catch up, which is hard in its own way.
But it feels totally different, because
as whereas before you have this, God,
where every inch is like,
as Steve was describing with Hipmunk,
every inch is you're pushing this boulder
and you're fighting for every inch.
Now it's like, crap, if we don't keep up
with this, it's going to run away from us.
And that's a totally different feeling,
and you'll know.
>> And here's the problem.
Eventually you'll catch the boulder on
the way down, and then you're like ****,
I'm actually pushing it uphill again.
>> [LAUGH] Yep, absolutely.
>> [LAUGH]
>> That's what happens.
>> [LAUGH]
>> And I guess it's like asking,
how do you know if you're in love?
Trust me, you know.
>> It's funny, because I think one of the
major problems that YC companies have is,
how do they stay small and lean and
don't spend a lot of money.
Don't have a management overhead,
don't have a lot of bills,
until they reach product market.fit.
One of the typical mistakes is
confusing an angel round or
a couple million dollars in the bank for
product market fit.
It's extremely common.
All right, other questions?
>> How do you recognize
your first group of user?
>> How do you recognize
your first groups of users?
>> Reddit was easy,
they were just like Alexis and I.
Emmet was kind of talking about
the intuition phase of building products.
We were in that for a long time.
Our users, they read the same
Paul Graham blog that we did.
They were programmers like I was,
so it was really, really easy.
Hipmunk, we were actually completely
wrong about who our users were.
And I don't know if we ever
quite solved this problem.
They solved it through acquisition.
Which is, we built a product for
ourselves, like tech savvy
people who travel a fair amount and
buy their own travel.
But in reality, the only way to make
money in travel is to sell to business
travelers, because they're the only one
that travels multiple times a year.
And business travelers don't get use
things like Hipmunk or Kayak, right?
They use American Express and Concur.
So it took us actually a couple
years to synthesize that, and
we were just like, ****.
>> [LAUGH]
>> I think there's two kinds of startups.
There are startups that
are building things for themselves,
where your customer is easy to identify.
It's you and people like you.
And there are startups building things for
third party group users.
Where you should arrive at which
customers are your customers through
an analytic mode, right?
Through like thinking about the problem.
So Twitch was an analytic start up,
because I was a huge fan
of watching game streaming.
But actually I wasn't much of a streamer,
I've never been much of a streamer.
And I basically dug in on Justin TV and
realized that 80% of
the minutes watched were coming
from our top 200 streamers.
And I was like they're
the most important people.
We lose them, we lose everything,
and if we can get 200 more of them,
we'll be twice as big.
And so it was sort of
identified analytically that
that was the right group of users,
and those are the right customers.
And I think that you should
just know which one you are.
And the most important thing there is,
if it's not for you, if you're not
legitimately building something for you
and for you to use, and you aren't a big
group of people in this case, you should
absolutely use the analytic mode.
I don't know, it's different for
every single company.
But think hard about it, I don't know.
>> But if it's not for you,
your intuition is lying to you.
>> Yeah, yes, yes.
>> Every time, and you have to supplement
with data or talking to users or whatever.
>> So shockingly, we were 100% on time.
If everyone can thank these guys for
coming out today, appreciate it.
>> [APPLAUSE]