Welcome back to yet another awesome lecture on encoding. And I don't want to hop on too much about URL
encoding for the reason that this course is about forms.
But it's good to know because sometimes in forms, especially with a GET request, characters are appended
to the URL and it's going to be doing funky things, sometimes.
And that's why we've kind of taken this tangent to learn about encoding.
And this lecture in particular is quite advanced.
There's a ton of information I want to get through, so please forgive me if I'm going too fast.
If anything is unclear, please ask on Q&A.
But let me ask you this.
What is a Web address?
What is it used for?
Well, that's right.
A Web address is used to point to a resource on the Web, such as a Web page.
You can think of a web address as directions.
It tells your browser where to go to fetch a resource.
And currently web addresses are expressed,
they're defined, they're written using Uniform Resource Identifiers or URIs.
And we've already seen that these URIs are governed by certain rules and these rules are defined in
a document called the RFC 3986.
And the long and the short of it, is that a URI is defined as a sequence of characters chosen from the
US ASCII character set.
And the key word here is the ASCII character set.
This essentially restricts web addresses to a small number of characters, basically just upper and
lower case letters of the English alphabet, European numerals, and a small number of symbols.
Well, as I'm sure you can agree with me, my students times have changed and users expectations and
the use of the Internet have moved on since then, and there's now a growing need to enable use of characters
from any language in web addresses.
Why?
Well, a web address in your own language and alphabet is just easier to create, memorize, transcribe,
interpret, guess, and relate to.
So it doesn't really make sense to restrict web addresses to the ASCII character set, does it?
Well, no, it doesn't.
But I wish things were simple.
But unfortunately, when it comes to coding and development, things are complex and there is not one
unified spec for URLs or URIs.
They are found in many different places.
Different organizations have tried to attempt to write rules on how they should be governed.
I don't want to get into all these different specs and what these organizations say.
The point I'm trying to make is that over the years there have been lots of changes.
"What kind of changes, Clyde?"
Well, that's a good question.
Originally, and I'm talking way back now in the 90s, everything was defined as a URL. Everything
that you wrote in an address bar was a Uniform Resource Locator.
But in the term was later changed to become a URI in 2005.
Later, the RFC 3987, not 6, remember 3986
that defines a URL,
but the RFC 3987 defined an IRI, and said that IRIs can be used instead of URIs.
But do you notice still that we've got separate definitions - a URI and an IRI?
Don't worry, I'm going to be talking more about Iris shortly.
But for now, note that this RFC 3987 is important.
It's important because the W3C has accepted this spec, which means all browsers need to conform to
it.
And remember, I said there were all the other organizations that have attempted to define URLs.
One of them is this WHATWG consortium.
They've produced their own URL spec, basically mixing ideas from URIs and URLs and IRIs with a strong
focus on browsers.
And it kind of makes sense, right?
Because how confusing is it ...
there are so many different specs around with different definitions.
It doesn't make sense.
It's actually just confusing.
So what else can I say about this
WHATWG consortium?
What did they try and achieve?
Well, one of the goals was to align RFC 3986,
remember, that defines the URL and RFC 3987, which defines IRIs.
And what's cool about the WHATWG is that
it's very liberal in what a URL can accept.
In fact, they say that a URL should be able to handle non-ASCII characters, which makes sense.
And I guess unsurprisingly, they say that URLs should be specified as UTF-8, which you and I know
can contain more than enough characters.
So it really would be ideal if the spec became mandatory.
But as I mentioned, it's RFC 3987, which rules the roost at the moment.
Okay, cool.
That's fine.
But if you're anything like me, you love seeing examples, so let's look at a URL.
Don't worry about what the international characters mean there.
It's just some Japanese characters I put there.
I just want us to talk about how this URL will be encoded.
I want to talk about what this means.
Well, firstly, this is not a URL,
strictly speaking.
This is known as an International Resource Identifier.
An IRI.
And why is this important?
Well, this is important because, as I mentioned, a URI supports only ASCII character encoding.
Remember, that's defined in RFC 3986.
An IRI, on the other hand, fully supports international characters and good news for us is that
UTF-8 is the most popular encoding used for IRIs.
We're going to be talking a lot more about this example URL that I've put up there shortly.
But for now, and I've mentioned this before, important for us is that the W3C URI spec basically
the World Wide Web Consortium, have accepted the RFC 3987, which defines an IRI.
And why is this important?
Well, it's important because various document formats, specifications and browsers support IRIs.
Okay.
So various document specs and browsers already support IRIs.
But the problem is that not many protocols allow IRIs to pass through unchanged.
And the protocol that we familiar with when it comes to building sites and apps is HTTP or HTTPS.
So if an IRI can't pass through a protocol, as is, what do we do?
Well, that's a great question.
And how an IRI works depends on where the non-ASCII character is located.
Is it located in the domain name, the path?
And this, my dear students, has created a lot of confusion, even amongst developers.
Trust me.
So what I'm about to share with you is super, super interesting, and it's going to put you ahead of
the pack.
But let me not get ahead of myself.
Let's look at the URL again.
It's exactly the same one we had before.
And let's break this up.
We already know that this HTTP is what?
That's right.
It's known as the scheme or the schema.
It contains information about the scheme to be used.
And this is what's important.
Non-ASCII characters are not allowed in the scheme.
So that's step one, and it's pretty obvious we don't want funky characters there.
We've just got to keep it plain and simple.
The next part is known as the ...
that's right,
it's known as the domain name.
And the remainder of the URL is known as the path.
And the path indicates the actual location of the resource you're trying to point to from the server
route.
Okay.
Have you got it?
Memorize this picture.
I want us to now talk about what happens with those international characters in the domain name versus
what happens to them in the path.
Let's first and very briefly, discuss the domain name.
Remember, that's that middle portion, the domain name.
What happens to domain names?
Well, what's interesting about domain names is that they are managed by domain name registration companies
that are spread across around the world. And the Internet Engineering Task Force,
back in early 2000, they produced a spec that governs how multilingual domain names should be dealt
with.
And if you're very interested, you can read all of those specs.
But the long and the short of it, is that the domain name register defines the list of characters that
people can request to be used in their country for top level domains.
And what's really cool is that these organizations have agreed to certain kind of formats.
And if a person requests a domain name using non-ASCII characters like those Japanese symbols you just
saw, then these symbols, these characters, will get converted over to "punycode". (record scratch sound)
Wait a second!
What is punycode?
Well, don't stress my dear students. I don't want to get too much into it, but it just allows for the encoding
of characters in the host name - the domain name - that should in theory only support ASCII characters.
That's all that punycode allows for.
And there's certain rules around punycode. There are certain rules that define how that conversion should
take place and the format of it.
And all these domain registrar companies around the world have agreed to this format.
And in theory, what's cool is that punycode could be used to allow for host names that use emojis.
How cool would that be?
But emojis are not a widely supported standard as of yet, so there's only a limited subset of top level
domains that support emojis currently.
But you never know.
Things do change. Anyway,
let's hop back into the lecture.
So we've discussed domain names ... that, kind-of you can view them as having their own set of rules.
And any Non-ASCII character in a domain name has to be converted over to punycode.
Have you got it?
We really are doing a lot, so please stick with me.
We are almost, almost done.
So I've spoken about domain names, but now I quickly want to talk about how the path is dealt with,
because the path is what we care about when building forms, right?
Remember with the GET request, data of the form is appended in the path of the URL.
So this is really what concerns us, not really the domain name.
And to remind you, we've got our URL and I only want to now talk about the path.
Remember what we just said when dealing with domain names, that there are
domain registration companies spread all over the world and they've all agreed to accept domain names
in a particular form with a particular encoding.
And that encoding was ASCII based
punycode.
But path names are more complicated.
Why?
Well, just because path names can identify resources located on many different kinds of platforms whose
file systems do and will continue to use many different encodings.
And this makes the path much more difficult to handle than a domain name.
But we don't have to stress because the good news is that the IETF standard 3987 deals with non-ASCII
characters in the path.
And at the crux of it, it's actually pretty simple.
The spec says that browsers need to represent all characters using percent escaping aka URL encoding.
So what does this mean for our URL?
Well, let's just forget about the domain section.
Let's just look at our path.
Right.
"dir1" and the Japanese symbol.
That's the path.
Let's assume the page the characters are on are encoded in UTF-8 because that's pretty much every
single site we visit today.
How will this work?
Well, the IRI spec says that the IRI should be converted to UTF-8.
First, the user agent, aka the browser, then needs to convert every non-ASCII character to percent
escapes.
Remember, this is just URL encoding. So it starts with a percentage symbol followed by two hexadecimal
values.
So what does this mean for our URL?
Well, firstly, the dir1 stays as dir1.
They are all ASCII character based, right?
A "d" and "i" and "r" and a "1" all form part of the ASCII character set, so nothing needs to happen.
The only thing that needs to be % encoded is that symbol.
And there we have it.
It'll look something like this.
Isn't that pretty cool?
It just means the URL path section is now in URI form.
It's kind of being converted from an IRI to a URI.
And why is this important?
Well, remember a few slides back.
Actually, if I go back here, let me try and show you.
Here.
On the slide here, remember I said there are many different documents,
specs and browsers that support IRIs, but not so many protocols allow IRIs to pass through,
unchanged.
Right.
And the protocol we're dealing with is HTTP.
So HTTP needs a URL, a valid URL in order to transport that over the wire.
Whew, here
we are, back here again.
That's why it's important, right, to get from and IRI to a valid URI.
It's just going to allow protocols such as HTTP to send that request.
And just note how that dir1 did not change.
They were unreserved characters in the ASCII set, which we spoke about earlier.
So at this point, the user agent, the browser can now send the request for the page.
Whew, so this is all good and well, and we are almost done.
But you might be thinking, okay, cool.
So anything in the path name has to be kind of percent encoded.
I've got it.
But when we looked at an example of a form that submitted Japanese symbols, why did we see those actual Japanese
symbols in the address bar?
In other words, why don't we see all the percentage hex values in the address bar?
Well, that's a very, very good question.
And just remember, the address bar on the browser is not the actual URL that's sent over the wire.
The address bar is a UI component that allows users to enter all kinds of fun strings that will get
converted over to URL at some point.
Basically, it just makes the web experience nicer for a user, so you can kind of think of the URL
address bar as is just being a visual help to us users.
It's not necessarily the actual true form of the URL. And modern clients,
I'm just meaning web browsers,
they are able to transform back and forth between percent encoding and Unicode,
so the URL is transferred as ASCII, but it looks pretty for us as the user because the browser understands
both - the browser understands IRIs, it understands URIs, so it makes sense that it should just
display us the correct symbols. And it can do its thing in the background.
I know, I know this lecture is getting very long and I'm just about to finish off.
What is the bottom line of this whole lecture?
Well, it is that IRIs are basically URIs that allow non-ASCII characters to be used. It make sense.
And most browsers today will allow you to see international characters in the address bar.
But in the background they are using techniques to convert these characters to ASCII so it can be transported
over the HTTP protocol.
They're using techniques.
What kind of techniques?
Well, it depends, right?
It depends if those international characters are in the hostname or in the path.
If it's in the hostname, the browsers use Punycode.
If it's in the path, the browsers use percent URL encoding which is defined in the RFC 39
86 and 3987.
So there we have it.
I told you this was going to blow your mind ðŸ¤¯.
And seriously, this is very, very advanced stuff.
It took me a long time to actually wrap my head around how your URL encoding works, so I hope you appreciate
it.
And yes, you might not need to know as much as we've discussed here, but you know what?
It's just another feather in your cap.
It's going to make you a better programmer.
And when you start dealing with forms, GET requests and you start seeing some percentage URL encoding,
sometimes you'll see that numerical character reference,
remember that? When you see those things, at least now you'll be able to appreciate what's happening.
So I think let me end the lecture here.
There's still a few more things I want to talk about when it comes to URL encoding, things that are
perhaps a bit more practical.
Stay motivated, grab a coffee and I'll see you in the next lecture.