It is not unusual for an application to require hitting a third party endpoint from your own server-side requests. However, hitting that end point during a load test can lead to serious problems.
- Vendor limits: Bing free has a 5,000 / month limit and unpublished per minute and daily limits.
- Vendor policy: Determined to be hitting request limits in high bursts can get an account locked
If you are determined your load test will access all backend resource you are best to contact the provider and work with them on a thorough load testing plan.
If your existing code requires results from third party backend you are best to work out mock data and service strategies for your particular development stack.
Here is a walkthrough of a Load Test Plan accessing Bing Search API and the rate limit errors we received.
Building the Bing Load Test required
- HTTP Authorization Manager for the Azure MarketPlace API
- You need to have your API Key and enter that in the password field.
- The Base URL should be (For web) https://api.datamarket.azure.com/Bing/Search/v1/Web
- Setup HTTP Request Defaults
- Data from CSV
- This was to alternate our query strings. We used the 10,000 most common words list from http://www.mit.edu/~ecprice/wordlist.10000
- Variable name word
- HTTP Request
- Path : /Bing/Search/v1/Web
- Send Parameters with request
- Query : ‘${word}’ (the ‘ are required) – encode this field
- $format : json
To demonstrate hitting the limits, we executed the Load Plan with 5 servers, 100 Users per server, 11 iterations.
Performance results, Wordlist, and JMX file available in this example Load Plan. You can
Instead of hitting the 5,000 maximum limit for the free tier, we quickly bumped into an unpublished per minute rate limit. This generated the following errors and locked us out from testing. If you were testing something like this against your production API key, you could have locked out real user behavior.
Code(503) Message(The number of requests per minute for the subscription has reached the maximum threshold that is allowed. Please try again after 2/11/2015 5:15:57 AM. Activity Id 9c4fface-5b2b-498d-b3c5-53eb98caead7) Url(HTTP Request)
On this test run, it caused about 60% of our requests to fail.
