Hey guys,
welcome to Day 48 of 100 Days of Code. Today
we're going to be introducing a new technology for advanced web scraping,
and that is Selenium WebDriver.
Now you might have heard of Selenium WebDriver before
because it's probably one of the most well-known automation and testing tools
for web developers out there. But you might be wondering, well,
we already have Beautiful Soup. So why do we need to learn a new technology?
Well,
one of the things that we've really been limited by is we can't actually use all
the capabilities that browsers can do.
So when we load up a website with beautiful soup, we can't, for example,
type something into the website and then click on something. And to create these
chains of continuous actions
where we basically automate the entire flow of a particular job or a particular
task. To do that,
we're going to need to use Selenium WebDriver.
Now, this is a free tool and it basically allows us to automate the browser,
get the browser to do things automatically
depending on a script or a piece of code that we write. Now,
this is going to enable us to type as well as click as well as scroll.
Basically anything that a human pretty much can do on a website, you can do using
a Selenium driven browser.
It's kind of like we're building a robot and telling it what to do on a browser.
And selenium is the tool that allows the robot to interact and communicate with
the browser. And once we've leveled up,
we'll be able to do all sorts of things like, for example,
play web-based games like cookie clicker automatically to click on this cookie
all by itself using the code instead of us having to manually do it and then to
select upgrades and to purchase them, again, completely automatically.
If you ever wanted to be the best at cookie-clicker or any other web-based
game and you don't want to invest the time in scaling up and getting good at it
yourself, well then this is where automation might help you.
But on a more serious note,
you might use something like selenium to automate
filling in forms or transferring information from an Excel spreadsheet to a
online Google form,
or basically doing anything that is repetitive and tedious,
and you don't want to do it yourself.
So once you are ready, head over to the next lesson
and let's start learning about this new superpower.