Automated Dapps Scrapping with Selenium and Metamask

One of the newest trends in web development is the rise of decentralized applications, also known as Dapps. These applications are built leveraging decentralized networks in order to provide trustless interactions between users, using predefined interactions built as smart contracts (if you want to know more about dapps head over here).

To access Dapps users need to use a crypto wallet to connect with, this creates a new challenge for developers who want to scrap and/or test Dapps using tools like Selenium. In this post we will cover the basics on how to solve this using Python and Chromium, however, the principles described here can be applied using any programming language and web browser automation tool.

Most current Dapps depend on a crypto wallet being present in the user browser as an extension, injecting in the web page information about the user wallet and the network it's connected with. One of the most popular browser crypto wallet is Metamask. In order to successfully scrape a Dapp we need to interact not only with the target website but also with the Metamask extension simultaneously, to approve the app connection with our wallet and other possible transactions.

Compressing the extension

In order to load the extension on our automated browser we will first need to compress the Metamask extension to a .crx file, here are the steps:

Install Metamask on your regular chrome

Navigate to chrome://extensions/

Click 'Pack extension' and enter the local path to the Metamask extension This will generate a .crx file that you can use to load as an extension on Chromium. Save the name of the folder where the extension is installed, this will be the 'Extension ID' that we will use later.

Loading the extension

To load Chromium with Metamask installed run:

from selenium import webdriver

EXTENSION_PATH = 'ENTER THE PATH TO YOUR CRX FILE'
opt = webdriver.ChromeOptions()
opt.add_extension(EXTENSION_PATH)

driver = webdriver.Chrome(chrome_options=opt)

Interacting with Metamask

In order to interact with the dapp and Metamask simultaneously, we will need to have multiple tabs in our Chromium, one for the target Dapp and another one for Metamask itself.

When Chromium starts it will have a welcome screen for the Metamask extension, which will prompt you to setup your wallet, this is an example code to import an existing wallet (You may need to update some steps depending on your Metamask version):

driver.find_element_by_xpath('//button[text()="Get Started"]').click()
driver.find_element_by_xpath('//button[text()="Import wallet"]').click()
driver.find_element_by_xpath('//button[text()="No Thanks"]').click()

# After this you will need to enter you wallet details

inputs = driver.find_elements_by_xpath('//input')
inputs[0].send_keys(SECRET_RECOVERY_PHRASE)
inputs[1].send_keys(NEW_PASSWORD)
inputs[2].send_keys(NEW_PASSWORD)
driver.find_element_by_css_selector('.first-time-flow__terms').click()
driver.find_element_by_xpath('//button[text()="Import"]').click()
driver.find_element_by_xpath('//button[text()="All Done"]').click()

After this Metamask will be successfully set up in Chromium, ready to connect to a Dapp.
When you need to interact with Metamask again you will need to use it in a different tab, like so:

EXTENSION_ID = 'ENTER HERE THE EXTENSION ID THAT YOU SAVED EARLIER'

driver.execute_script("window.open('');")
driver.switch_to.window(driver.window_handles[1])
driver.get('chrome-extension://{}/popup.html'.format(EXTENSION_ID))

With this Metamask will be opened in this new tab, ready to be interacted with.

That's it, now you're ready to start automating interactions with Dapps.
Follow me for more articles like this and if you have any questions feel free to leave them in the comments below.