Problem: How do I use my Selenium tests to run load tests?

Answer: You can run a 5,000 user load test for $10 per hour with RedLine13.com.

We distribute your Webdriver code to any number of AWS instances to scale to any size. We handle the distribution of the load agents running PhantomJS to emulate real user processes hitting your application/website at heavy load within minutes.

YES, IT’S REAL

The Past

The previous answer had been “It’s too expensive” or go create a proxy to capture the web requests and build your own JMeter tests.

Sauce Labs and others have said this for a while – Ask a Selenium Expert: Should I Test Load with Selenium?  Other blogs talk about using a proxy to capture your web request, such as the BlazeMeter’s proxy recorder.  RedLine13 even built a Firefox plugin to capture and convert your Selenium tests into JMeter tests during playback.

These were all ways to avoid using webdriver because scaling up real browsers was hard, expensive, and lacking in performance data.

The Future

A few things have since changed:

A) Amazon Web Services, specifically spot instances.  

EC2 M4.16xLarge is a 64 vCPU and 256 GB of Memory. Spot prices in the Ohio region are around $0.42 per hour.

B) PhantomJS 

PhantomJS has less to offer than a real browser but enough power to drive the majority of load testing needs to simulate real user behavior.

C) Performance Data

We can gather stats in real time from PhantomJS. The performance data available has more metrics available than JMeter or Gatling can provide (though today we can extract out the same metrics as JMeter collects and generate JMeter data files).

D) Serverless Orchestration with RedLine13

Simplified and (almost) free for dropping in your webdriver test it provides the quickest way to scale a load test. No management of infrastructure is required.  We spin up and down the resources needed on your own AWS account. Your only cost per test is the EC2 cost of running the load agents (emulate ~250 users per instance)

Key Take Aways

  • Run a 5000 user Selenium test for 1 hour costs $10.0954
  • No changes in your webdriver code, only how you access the RedLine/PhantomJS driver
  • Beta version supports Selenium WebDriver via NodeJS
  • Your test will generate a JMeter JTL output file
  • Building load tests is not the hard part, scaling your solution is hard
  • Step-up scaling your load tests – 10, 100, 250, 500, 1000, and so on.

Run your first WebDriver Performance Test Locally

https://www.npmjs.com/package/redline13-webdriver

npm install redline13-webdriver

Then inside your webdriver source, you only need to access the driver from a single call.

// include redline
var redline = require( 'redline13-webdriver' );
 
// load your driver via redline13
// redline13 library uses standard mechanism but presets some capabilities
var browser = redline.loadBrowser('chrome');
 
// You can require webdriver yourself or use redline13.webdriver
var By = redline.webdriver.By;
var until = redline.webdriver.until;
 
browser.get( "http://example.com" );

Run your performance test

node test.js

And there are two ways to see the performance data

  • Command Line Output 
    • $ node test.js 

Record Load Time for (http://example.com/) in (2.256513999949675)
  • JMeter Output File 
    • cat output/runLoadTest.jtl 

1485891318,2228,http://example.com/,200,OK,Thread Group 1-0,text/html,true,,0,1,1,http://example.com/,2204

Run your first WebDriver Performance Test at Scale (on RedLine13.com)

  • https://www.redline13.com/StartTest start test
  • Select ‘Custom Test’
  • Set Number of Users to 50
  • Upload Custom – drop in your test.js file
  • start-webdriver-node-test
  • In Advanced Custom Test Options Select ‘WebDriver NodeJS’ plugin
  • Select ‘PhantomJS’ as the browser type
  • setup-webdriver-nodejs-with-phantomjs
    • Domains – You can provide a space separated list of domains which will track full URLs from that URL
      • Other domains will be truncated to their domain name and recorded, lumping together all metrics from a domain.  For example, all google analytics calls.
    • Hard Filter – If checked will only track from the domains specified, other metrics will not be recorded.
    • Capture ScreenShot on Error –  If checked, when the script throws an error an image will automically be captured if possible and be available at end of test.
  • Start Test
  • Watch Resultswebdriver-results-summary
  • Other outputs include
    • Real-time Performance Graphs
    • Error screenshots if generated
    • Full JMeter output file (jtl)
    • Full JMeter dashboard report

webdriver-apache-jmeter-output

And with that, you have run a 50 user selenium test.

Having any issues, just email info@redline13.com

Architecture

RedLine13 architecture

  • Test scheduling which can be done through the web UI, API, or via Jenkins.
  • Test coordination, this is the pain part that redline13 makes easy.
    • Coordinate EC2 instance in your environment, spot or on-demand making all the API requests
  • Test execution once the coordination is in a ready state.
    • Each test instance accepts the test to execute and calls pre-test, test-execute, and post-test tasks.
    • These include spinning up phantom or Chrome
  • Test collection during the test run.
    • During your test we are extracting performance data from phantom or post-test from chrome, summarizing and sending back to redline13.com in increments to display
  • Test reporting primarily happens during the test where the collected performance data is being graphed
    • Some post-test reporting is included as well – percentile reporting or Apache JMeter Dashboard report.
  • Test finalization includes knowing when to shutdown test agents.

architecture-webdriver-sideways

 

Example

Building a demo site to test scaling to 5,000 current users is not a simple task.  Systems like wordpress require tuning and configuration.  We found a simple static content management system (cmsmadesimple) was the easiest project to scale up with just a simple forward facing ELB.

Our load test wanted to simulate a fairly realistic scenario

  • Hit website
  • Search for content that is not there
  • Search for content that is there
  • Click through to content
  • Go Home
  • 20% of the time add some content
    • Login
    • Add News
    • Logout
    • Search for content we just created
    • Go Home

https://gist.github.com/richardfriedman/3df9be3ae82e24386a7dd171c1d5fb38

Running our test locally was simple, you can just use – node cmsmadesimple.js

Thanks

  • Selenium
  • WebDriver
  • PhantomJs
  • Apache JMeter
  • Amazon WebServices

 

 

 

Comments are closed.