Let's carry on exactly where we left off. Remember what we've discussed so far, that text has to be converted...
into binary data in order for your machine to understand it.
And we need to do that whenever we transport data over the Web...
for example, when we send data to a server, when a user submits a form. That's why we're talking about...
this.
Don't get lost in all the detail.
Sometimes we stand too close to the wall...
we can't even see the painting.
So that's why we have to take a step back sometimes to remind ourselves what this is all about.
We discussed ISO, that it's very old school.
We then discussed the ASCII character set, which only has 128 unique values.
And that is just not enough.
Let's be realistic.
We need more characters.
And this brings me onto the UTF-8 character encoding set.
As we've already seen, to create a table that maps characters to letters for a language that...
uses more than 256 characters, one byte or 8 bits, simply isn't enough.
If you had two bytes, which is 16 bits, you can store 65,536
unique values.
And over the years, many encoding types and many organizations, many people have tried to invent different...
encoding types, to deal with the ASCII limitations.
But finally someone got fed up, and they set out to create one encoding standard to unify all encoding...
standards.
And this standard, my dear students, is called Unicode.
And it was invented by the Unicode Consortium, which is just an organization, a group of people that...
came up with a standard, as I mentioned, to rule the roost 🐔, not that kind of roost...
and this Unicode set basically defines a ginormous table, a massive table of one million, one hundred...
and fourteen thousand, one hundred and twelve code points that can be used for all sorts of letters and symbols.
Just so you know, this is plenty enough to encode all European, Middle Eastern, Far Eastern, and Southern ...
Northern, Western, prehistoric and even futuristic robotic languages in the future that we don't even...
know about yet.
It is plenty, plenty enough.
But lemme ask you this question.
How many...
bits do you think Unicode uses to encode all of these characters?
It's a trick question because the answer is ... none.
What! This is because Unicode is not an encoding type.
Are you confused 😕?
Many people are,
so don't worry.
Unicode, first and foremost, defines a table of code points for characters.
Let me repeat that. Unicode defines a table of code points for characters.
And again, don't let all these fancy jargons intimidate you.
All this means is that it's a fancy way of saying, for example, big A it's just assigned the code...
point 0041. And little "a" is the code point 0061.
And what about the coffee ☕ image? Well that's assigned a code point of 2615.
And you might think I'm actually joking, but I'm not.
This is actually defined as a coffee cup.
Let me jump over to Word quickly, and I'll show you. As I said in the lecture,
I'm not making these things up.
We do literally have a coffee symbol in UTF, and its code is 2615.
I'm in Microsoft Word and all we have to do to convert that into the character, is to press "Alt + x" on the...
keyboard, "alt + x". And BOOMSHAKALAKA 💥. There is our coffee symbol.
So there we go.
I'm not just making these things up.
These are literally the different codes assigned to the different variables, different characters,
the different emojis in the UTF encoding type.
Cool, let's hop back into the lecture. Okay, starting to make sense.
I'm not just making these things up.
Alright, so now we have a whole bunch of code points.
We know that all of these code points have to be converted into bits in the background. And to convert...
these code points into bits...
we need an encoding type and this is what UTF is all about. To represent...
1.1m values, 2 bytes...
is just not enough.
Interestingly, 3 bytes would be enough, but 3 bytes often just awkward to work with for computer
programmers...
so 4 bits is just a lot more comfortable. But, 4 bits also pose some problems and is overkill...
in many cases.
Why do I say that?
Well, unless you're actually using Chinese or some of the other characters with big numbers that take...
lots of bits to encode, you're never going to use a huge chunk of those four bytes.
So, for example, if the letter A was encoded into four bytes every single time...
it would like like this.
And if this was the case, I'm sure you can see how bloated a document would become.
In fact, it would be 4 times the necessary size.
"OK, Clyde, stop giving me problems." Well, okay. To fix this...
there are several ways to convert Unicode characters into bits. The most common is UTF-32, UTF-16 and UTF-8.
Well, let's first look at UTF-8 and UTF-16.
These are known as variable-length encodings.
And all I mean by this, is that if a character can be represented using a single byte, for example,
because it's code point is a very small number like the letter A, then UTF-8 will encode it with...
a single byte. If it requires 2 bytes, it will use 2 bytes, so on and so forth.
For example, if you look at the below, we can see the letter A could be stored in one byte.
But when you get some cool letter in Chinese for example, here it's stored in 3 bytes. Whew, that...
seems a bit elaborate though, doesn't it?
And it is. It's very elaborate.
And the system uses very elaborate ways to use the highest bits in a byte to signal how many bytes a...
character consists of.
So this can save space, but it also may waste space if these signal bits need to be used often.
And that's why UTF-16 is there. It kind of sits in the middle...
but instead of using a single byte, it always uses at least 2 bytes, growing up to 4 bytes as necessary...
same as UTF-8. Whew, we've learnt a lot.
The other thing I want to say about UTF-8 is that it is binary compatible with ASCII, which is...
the de facto baseline for all encodings.
All characters available in ASCII encoding only take up a single byte in UTF-8 and they are the exact...
same bytes as are used in ASCII.
So in other words, ASCII maps 1:1 with UTF-8.
Any characters that are not an ASCII takes up two or more bytes in UTF-8.
So, what this means is that for most programming languages that expect to parse ASCII, you can include UTF-8...
directly into your programs and you won't have any issues.
How awesome is that?
Whew,
I don't know about you, but I just feel like my mind has been blown 🤯.
We've learned an incredible amount about this simple looking attribute, "accept-charset=utf-8".
Let me just give you a quick, quick summary, because you really have learnt a lot in a very quick...
space of time.
Just remember, a computer cannot store letters, numbers, pictures or anything else like this.
The only thing it can store and work with are bits. And bits are binary data. And remember a bit can only...
have two values.
Yes or no, true or false, 1 or 0, or whatever else you want to call these two values.
Just a bit of FYI...
since a computer works with electricity, an actual bit is a blip of electricity that's either there...
or isn't. For humans...
we just like representing these as 1s and 0s,
so of course I've been sticking with this convention. But moving on, to use bits to represent anything
at all besides actual bits or 1 and 0s,
we need to convert a sequence of characters into bits and then we need to later decode those bits back...
into characters.
And the rules governing all of this, is what's referred to as the encoding type.
Don't get confused or get overwhelmed.
It's actually quite interesting once you understand all of this, and it's very practical.
In fact, all text is already encoded in some encoding.
When you type text into source code, it has some encoding, specifically whatever...
you saved it as in your text editor.
If you get it from a database, it's already in some encoding.
If you read text from a file, it's already in some encoding.
So just remember, text is either encoded or it's not.
And when you specify the accept-charset attribute, you're telling the server basically to encode the...
text from its existing encoding type and to use, for example, UTF-8.
And that's all there is to it.
So I hope it's starting to make a bit more sense.
Don't worry if this was very hard for you.
It is a very advanced topic.
I warned you at the beginning - I'm throwing you into the deep end. But I hope you're having a ton of fun.
And there are a few more things I want to harp on about this accept-charset, because it is fascinating.
For example, you're probably wondering what happens if you type a symbol or character that's not defined...
by the character set.
That's exactly what I want to jump into, in the next lecture.
See you now 👋.