‫One of the most important steps
‫in building data intensive apps is to actually model
‫all this data in MongoDB.
‫And so that's what we're gonna talk about in this lecture.
‫So it's really crucial that you follow it
‫through even at first its a lot to take in. All right.
‫Anyway, lets now get started.
‫Now, data modeling is probably a very new concept to you.
‫So before we start;
‫lets make clear what we're actually gonna talk about.
‫So, data modeling is the process of taking unstructured data
‫generated by a real world scenario
‫and then structure it into a logical data model
‫in a database.
‫And we do that according to a set of criteria
‫which we're gonna learn about in this video.
‫For example; lets say that we want to design
‫an online shop data model.
‫There will be initially a ton of unstructured data
‫that we know we need.
‫Right.
‫Stuff like products, categories,
‫customer's orders, shopping carts, suppliers.
‫And so on and so forth.
‫Our goal with data modeling is to then structure
‫this data into a logical way.
‫Reflecting the real-world relationships
‫that exists between some of these data sets.
‫A bit like you can see in this example.
‫And this is of course just a kind of imaginary situation
‫but you get the idea.
‫Right.
‫Now, many backend developers say that data modeling
‫is where we have to think the most.
‫That its the most demanding part of building
‫an entire application.
‫Because it really is not always straight-forward.
‫And sometimes there are simply no right answers.
‫So not just one unique correct way of structuring the data.
‫But anyway I will do my best to lay down the process
‫in this video.
‫And for that we're gonna go through four steps.
‫So in the first step; we learned about how to identify
‫different types of relationships between data.
‫Then we're gonna understand the difference
‫between referencing or normalization
‫and embedding or denormalization.
‫In the next and most important step;
‫I will show you my framework for deciding
‫whether we should embed documents
‫or reference to other documents
‫based on a couple of different factors.
‫Also, we have to quickly talk about
‫different types of referencing.
‫Because that's important if that is the type of design
‫that we choose for our data.
‫So this is gonna be in fact a quite theoretical lecture.
‫But also an absolutely essential one for your progress
‫as a back-end developer.
‫Because the way we design data so the way we model our data
‫can make or break our entire application.
‫And there will be a lot of examples along the way
‫to make this process easier.
‫All right.
‫And the first thing that we are gonna talk about
‫is the different types of relationships
‫that can exist between data.
‫So there are three big types of relationships.
‫One to one, one to many, and many to many.
‫And I'm gonna use a movie application
‫as an example in this slide.
‫Okay?
‫So first a one to one relationship
‫between data is basically
‫when one field can only have one value.
‫So in our movie application example; one movie only ever
‫have one name.
‫And so this is a simple example
‫of a one to one relationship.
‫But these relationships are not really that important
‫in terms of data modeling.
‫Now the most important relationships
‫are the one to many relationships.
‫And they are so important that in MongoDB
‫we actually distinguish between three types
‫of one to many relationships.
‫One to a few, one to many, and one to a ton or to a million
‫or something like that. So the difference here is based
‫on the relative amount of the many. All right.
‫So an example to a one to a few relationship is that
‫one movie can win many awards but actually just a few.
‫So movie is not gonna win a thousand awards
‫but it can win some.
‫And so this is a typical one to few relationship.
‫So you see that in general a one to many relationship
‫means that one document can relate to many other documents.
‫Now this might look a bit abstract without the JSON data
‫but that's actually the purpose here.
‫I just wanna show you a conceptual overview
‫of these different types of relationships.
‫Anyway, any one to many relationship
‫one document can relate to hundreds or thousands
‫of other documents.
‫For example; one movie can have thousands of reviews
‫in our application.
‫And so this not really a one to few
‫but one to many relationship. Okay?
‫And finally we have the one to ton relationship.
‫Imagine we wanted to implement some logging functionality
‫in our app. So basically to know exactly what's going on
‫on our server.
‫This logs can then easily grow to millions of documents.
‫And so this is a very typical example
‫of a one to tons a relationship.
‫And the difference between many and a ton is of course
‫a bit fuzzy but just think that if something can grow
‫almost to infinity then its definitely
‫a one to a ton relationship.
‫So again the one to many relationships
‫are the most important ones to know.
‫By the way; in relational databases
‫there is just one to many without quantifying
‫how much that many actually is.
‫In MongoDB databases though
‫it is an extremely important difference.
‫Because its one of the factors that we're gonna use
‫to decide if we should denormalize or normalize data
‫as you will learn a bit later.
‫Anyway, the less type of relationship is the many to many
‫where one movie can have many actors.
‫But at the same time one actor can play in many movies.
‫And so here the relationship basically
‫goes in both directions.
‫Where before in the other types
‫it was only in one direction.
‫For example one movie can have many reviews
‫but one specific is only for that one movie. Right.
‫And the same goes for the awards.
‫So one specific award like for the best actor
‫goes to only one movie not multiple ones.
‫But with movies and actors it is indeed different.
‫So again one movie stars many actors
‫but one actor plays many movies
‫and so its a many to many relationship.
‫Okay.
‫So keep all this in mind as we now move forward
‫in this lecture.
‫And probably the most important aspect that we need to learn
‫about MongoDB databases is referencing
‫and embedding two datasets.
‫And we actually already talked a little bit
‫about this before but lets review it here
‫and go a bit deeper also.
‫So each time we have two related datasets
‫we can either represent that related data in a reference
‫or normalized form or in an embedded or denormalized form.
‫And I keep using the two related terms together
‫like referencing and normalizing
‫because you will see them both being used
‫and so its important that you know all of them.
‫Anyway, in the referenced form we keep two related
‫datasets and all the documents separated.
‫So again all the data is nicely separated
‫which is exactly what normalized means.
‫So continuing, the movie database example from before
‫we would have one movie document and one actor document
‫for each actor. Now how would we then make the connection
‫between movie and the actors so that later in our app
‫we can show which actors played in a particular movie.
‫Because if they are all completely different document
‫the movie has no way of knowing about the actors. Right.
‫Well that's where the IDs come in.
‫So we use the actor IDs in order to create references
‫on the movie document.
‫Effectively connecting movies with actors.
‫So you see that in a movie document we have an array
‫where we stored the IDs of all the actors
‫so that when we request data about a certain a movie
‫we can easily identify its actors. Does that make sense?
‫Now this type of referencing is called child referencing
‫because its the parent in this case the movie
‫who references its children. In this case the actors.
‫So we're really creating some sort of hierarchy here. Right.
‫Now there is also parent referencing
‫and we are gonna talk about that a bit later.
‫And by the way in relational databases; all data is always
‫represented in normalized form like this.
‫But in a no sequel database like MongoDB
‫we can denormalize data into a denormalized form
‫simply by embedding the related document
‫right into the main document.
‫So now we have all the relevant data about actors
‫right inside in one main movie document without the need
‫for separate documents, collections, and IDs.
‫So again, if we choose to denormalize or to embed our data
‫we will have one main document containing all the main data
‫as well as the related data. All right.
‫And the result of this is that our application
‫will need to fewer queries to the database.
‫Because we can get all the data
‫about movies and actors all at the same time
‫which will of course increase our performance.
‫Now the downside here is of course
‫that we can't really query the embedded data on its own.
‫And so if that's a requirement for the application
‫you would have to choose a normalized design
‫and since we're talking about pros and cons
‫of the denormalized form; lets do the same
‫about the normalized design.
‫And basically its kind of the opposite
‫of what we just talked about.
‫So there is an improvement in performance
‫when we often need to query the related data on it's own
‫because we then can just query the data that we need
‫and not always movies and actors together.
‫But on the other hand; when we need to actually query
‫movies and actors together we then are gonna need
‫many queries to the database.
‫So first the query for the movie and then from there
‫we will also need a query for the actor
‫and that is of course works for performance.
‫So when designing your database; this is the kind of stuff
‫that you need to keep in mind. All right.
‫And now just as a side note; we could of course begin
‫our thought process with denormlized data
‫and then come to the conclusion
‫that its best to actually normalize the data.
‫So when thinking about our data model
‫this way of organizing data works of course in both ways.
‫Now, how do we actually decide if we should
‫normalize or denormalize the data?
‫Well that's exactly what we're gonna learn next.
‫So when we have two related datasets; we have to decide
‫if we're gonna embed the datasets or if we're gonna
‫keep them separated and reference them
‫from one dataset to the other.
‫And I kind of developed this decision framework
‫which I'm gonna show you where we use three criteria
‫to take that decision.
‫First we look at the type of relationships
‫that exists between datasets.
‫Second we try to determine the data access pattern
‫of the dataset that we want to either embed or reference.
‫And this just means to analyze how often data is read
‫and written in that dataset.
‫Then we also look at something that I call data closeness.
‫And data closeness is term that I actually just made up
‫but what it means is how much the data is really related
‫and how we want to query the data from the database.
‫And this will make more sense
‫when we talk about it in a moment.
‫Now to actually take the decision; we need to combine
‫all of these three criteria
‫and not just use one of them in isolation.
‫So for example; just because criteria number one
‫says to embed it doesn't mean that we don't need to look
‫at the other two criteria.
‫All right and lets start with the relationship type.
‫So usually when we have one to few relationship
‫we will always embed the related dataset
‫into the main dataset just like we learned
‫in the last slide. Right.
‫Now in a one to many relationship; things are a bit
‫more fuzzy so its okay to either embed or reference.
‫In that case we will have to decide
‫according to the other two criteria.
‫Now on the other hand, on a one to a ton
‫or a many to many relationship we usually always reference
‫the data. That's because if we actually did embed
‫in this case we could quickly create way too large document.
‫Even potentially surpassing the maximum of 16 megabytes.
‫And so the solution for that is of course referencing
‫or normalizing the data. And as a quick example;
‫lets say that in our movie database example
‫we have around 100 images associated to each movie.
‫So we could say its a one to many relationship
‫but are we gonna embed the dataset or should we rather
‫reference them here. Well we don't really know.
‫So lets take a look at the other two criteria.
‫So the second one is about data access patterns
‫where its just a fancy description
‫for evaluating whether a certain dataset
‫is mostly written to or mostly read from.
‫So if the dataset that we're deciding about is mostly read
‫and the data is not updated a lot
‫then we should probably embed that dataset.
‫So a high read/write ratio just means
‫that there is a lot more reading than writing.
‫And a again, a dataset like that is a good candidate
‫for embedding.
‫The reason for this is that by embedding we only need
‫one trip to the database per query.
‫While for referencing we need two trips. Right.
‫So if we embed data that is read a lot;
‫in each query we save one trip to the database
‫making the entire process way more performant.
‫So I think that our movie image example
‫would actually be a good candidate for embedding.
‫Because once the 100 image are saved to the database
‫they are not really updated anymore
‫because there is not really anything to update
‫about an image. Right, so its all about reading
‫and therefore based on this criteria
‫we would embed the imaged documents.
‫Now on the other hand, if our data is updated a lot
‫then we should consider referencing or normalizing the data.
‫That's because its more work for the database engine
‫to update and embed a document
‫than a more simple standalone document.
‫And since our main goal is performance;
‫we just normalize the dataset.
‫In our example lets say each movie has many reviews
‫and each review can be marked as helpful by the user.
‫So each time someone clicks on this review was helpful
‫in our application. We need to update
‫the corresponding document.
‫And this means that the data can change all the time
‫and so this is a great candidate for normalizing.
‫Again because we don't want to be querying the movies
‫all the time if all we really wanna update is the reviews
‫by marking them as helpful.
‫Okay, does that make sense?
‫And finally the last criteria I call data closeness;
‫which is just like a measure for how much the data
‫is related. So if the two datasets really
‫intrinsically belong together then they should
‫probably be embedded into one another.
‫In our example; all users can have many email addresses
‫on their account and since they are so intrinsically
‫connected to the user, there is no doubt emails
‫should be embedded into the document.
‫Now if we frequently need to query both of datasets
‫on their own then that's a very good reason
‫to normalize the data into two separate datasets.
‫Even if they are closely related.
‫So imagine that in our app we have a quiz
‫where users have to identify a movie based on images.
‫This means that we're gonna query a lot of images
‫on their own.
‫So without necessarily querying for the movies themselves.
‫And so if we apply this third criteria;
‫we come to the conclusion that we should actually normalize
‫the image dataset. All right.
‫Because again if we implement this quiz functionality;
‫images are gonna be queried on their own all the time.
‫So, all of this shows that we should really look
‫all the three criteria together
‫rather than just one of them in isolation.
‫Because that might lead to less optimal decisions.
‫And I say less optimal instead of wrong
‫because they are not really completely right
‫or completely wrong ways of modeling our data.
‫There are no hard rules; these are just like guidelines
‫that you can follow to find the probably
‫most correct way of structuring your data.
‫But again, it's hard to be really really wrong.
‫Okay?
‫Now, lets say that we have chosen to normalize
‫our datasets.
‫So in other words to reference data.
‫Then after that we still have to choose
‫between three different types of referencing.
‫Child referencing, parent referencing
‫and two-way referencing.
‫So the first type is child referencing.
‫Which is the referencing type I actually showed you before.
‫Okay?
‫And lets not take the error logging example
‫that I mentioned earlier. Where we could potentially
‫have millions of locked documents.
‫So in child referencing; we basically keep references
‫to the related child documents in a parent document.
‫And they are usually stored in an array.
‫So you see that each log has an ID
‫and then in the app document there is that array
‫with all of these IDs. Right?
‫However, the problem here is that this array
‫of IDs can become very large if there are lots of children.
‫And this is an anti-pattern in MongoDB.
‫So something that we should avoid at all costs.
‫Also, child referencing makes it
‫so that parents and children are very tightly coupled.
‫Which is not always ideal. But that's exactly
‫why we have parent referencing.
‫So in parent referencing; it actually works
‫the other way around.
‫So here in each child document we keep a reference
‫to the parent element.
‫Therefore the name parent referencing.
‫In this example the app ID is 23
‫and so in each log there is the app field
‫with the 23 ID in it.
‫So that the child always knows its parent.
‫And so in this case the parent actually knows nothing
‫about the children.
‫Not who they are and not how many they are.
‫So, they are way more isolated and more standalone.
‫In that, it can sometimes be beneficial.
‫So which of these two types is actually better
‫for this data relationship.
‫And remember how I said that there
‫could be millions of logs and so lets suppose
‫there is two million logged documents.
‫In a case of child referencing, that would mean
‫that there are two million ID references
‫in the app document.
‫Right? Now also remember how I said that
‫there is 16 megabyte limit on documents.
‫So if we kept adding and adding these child IDs
‫into the array on the parent; then we would
‫pretty quickly hit that 16 megabytes limit
‫that each Bson document can hold.
‫Simply because that array will grow so much.
‫So that's not really gonna work.
‫Is it?
‫On the other hand with parent referencing
‫that problem is not gonna happen.
‫We will simply have two million locked documents
‫just like before but each of them holds ID of its parent.
‫But there is no array that will grow indefinitely
‫and therefore parent referencing
‫would be best solution here.
‫So the conclusion of all this is that in general
‫child referencing is best used
‫for one to a few relationships. Where we know before hand
‫that the array of child documents won't grow that much.
‫On the other hand, parent referencing is best used
‫for one to many and one to a ton relationships
‫like this one. Okay?
‫So again always keep in mind that one of the most
‫important principals of MongoDB data modeling
‫is that array should never be allowed to grow indefinitely.
‫In order to never break that 16 megabyte limit.
‫We also don't want to send our users an array
‫with thousands of IDs each time
‫they request a parent dataset. Okay?
‫So did this logic make sense to you?
‫Then lets move on to third type of referencing
‫which is two-way referencing.
‫And this time with the movie and actor example
‫I showed you when we talked about
‫many to many relationships. Remember that?
‫So again, each movie has many actors
‫and each actor plays in many movies.
‫And so that's a typical many to many relationship.
‫And we usually use this two-way referencing to design
‫many to many relationships. And it works like this;
‫in each movie we will keep references to all the actors
‫that star in that movie. So a bit like in child referencing.
‫However and at the same time in each actor
‫we also keep references to all the movies
‫that the actor played in.
‫So movies and actors are connected in both directions.
‫In therefore the name two-way referencing.
‫And this makes it really easy to search
‫for both movies and actors completely independently.
‫While also making it easy to find the actors
‫associated to each movie and the movies associated
‫to each actor.
‫(deep breath)
‫This was quite a long lecture indeed.
‫With a lot of new concepts and principals
‫and guidelines to remember.
‫So in order to help you with that; here goes a quick
‫summary and some more general guidelines that you can
‫take a look at when you need it.
‫So the most important principal is: structure your data
‫to match the ways that your application queries
‫and updates data.
‫Or in other words: identify the questions that arise
‫from your application's use cases first, and then model
‫your data so that the questions can get answered
‫in the most efficient way.
‫For example; when I need to query movies and actors
‫always together or are there scenarios where I only
‫query movies or only actors.
‫That kind of questions is what your data model
‫will be based on.
‫In general, always favor embedding unless there is a good
‫reason not to embed.
‫Especially on one to a few and one to many relationships.
‫Next up, a one to a ton or a many to many relationship
‫is usually a good reason to reference instead of embedding.
‫Also, favor referencing when data is updated a lot
‫and if you need to frequently access a dataset on its own.
‫Use embedding when data is mostly read but rarely updated
‫and when two dataset belong intrinsically together.
‫Don't allow arrays to grow indefinitely.
‫Therefore, if you want to normalize; use child referencing
‫for one to many relationships and parent referencing
‫for one to a ton relationships.
‫And finally use two-way referencing
‫for many to many relationships.
‫All right?
‫And that pretty much sums it up.
‫I would actually recommend you watching this video
‫twice if you can, just because of how important
‫this material really is. All right?
‫Anyway, see you in the next video.