The recorder was built by using Firefox docs, examples, and some open projects that demonstrate how to listen to HTTP traffic within a Firefox extension.

Overview of source files

We built this separating the work into 3 prototype objects.
  • HttpRecorder.js – setup listening for requests/response
  • HttpVisitor.js – when a request or response is made we need to inspect the headers and data
  • HttpPostParser.js – take a deeper look into the POST data stream

Getting access to the Observer Service

  • This is how we access the service.
HttpRecorder.prototype.observerService = Components.classes["@mozilla.org/observer-service;1"].getService(Components.interfaces.nsIObserverService);

Observing the HTTP Traffic

  • Using the Observer service we sign up for certain topics (events)
  • this – means our object provides the methods for callback
HttpRecorder.prototype.start = function() {
    this.observerService.addObserver(this, "http-on-modify-request", false);
    this.observerService.addObserver(this, "http-on-examine-response", false);
};
This is a pattern provided for Firefox extensions which allows actual modifications of the requests, though our plugin only needs to watch.
Using ‘this’ as the first parameter means our javascript object implement a function called ‘observe’ which will invoked for each web request.
This will capture all web requests not just for the tab you are on.  There are other mechanism to watch firefox behavior, like monitoring the DOM that can be page specific, read more about intercepting Web Requests.  Don’t be surprised when you start seeing requests that you were not expecting.

The Observer callback function

The observe function is the callback for all events you are going to capture.
  • Subject: A notification specific interface pointer, so you must know the type based on the event
  • Topic:  the string representing the event
  • Data: optional data object depending on event
The biggest learning curve was understanding the ‘QueryInterface’ function.  This looks into the current event (Subject) and sets its interface, to the Interface you pass in.   I grokked it as casting to a type.
  • Process the observe callback looking for the topics we care about
  • http-on-modify-request is HTTP requests,
  • http-on-examine-response  is the HTTP Response
HttpRecorder.prototype.observe = function(subject, topic, data) {
    var chan = subject.QueryInterface(Components.interfaces.nsIHttpChannel);

    switch (topic) {
        case 'http-on-modify-request':
            this.onRequest(subject);
            break;
        case 'http-on-examine-response':
            this.onResponse(subject);
            break;
        default:
            break;
    }
};

Stop Observing the HTTP Traffic

  • Simply remove the observers
HttpRecorder.prototype.stop = function() {
    try {
        this.observerService.removeObserver(this, "http-on-examine-response");
        this.observerService.removeObserver(this, "http-on-modify-request");
    } catch (e) {
        console.log("Failed to remove observer", e);
    }
};

Parsing the HTTP Request

  • Validate we want to parse this request
  • Parse the basics
  • Parse the headers
  • if POST parse the POST
HttpRecorder.prototype.onRequest = function(http) {

   var uri = http.URI.asciiSpec;

   // Our own function to limit what we capture.
   if (this.shouldInclude(uri) && !this.shouldExclude(uri)) {

        var request = {};
        request.timestamp = Date.now();
        request.uri = uri;
        request.method = http.requestMethod;
        // ... Missing - but here we pick apart the request 

        // The headers of the request are not directly available this wraps the work 
        var visitor = new HttpVisitor(http);
        request.headers = visitor.walkRequest();
       
        // Parsing an HTTP Post requires a different interface so the work is separated out.
        if (http.requestMethod == 'POST') {
            var post = visitor.parsePost();
            if ( post ) {
                request.postBody = post.body;
                request.postHeaders = post.headers;
                request.postLines = post.lines;
                request.postBinary = post.binary;
            }
        }
        this.requests.push(request);
    } else {
        this.ignoredRequests++;
    }
};

Parsing the Response and matching to the request

  • On response based on URI match to request
  • Get request meta
  • Get request headers
HttpRecorder.prototype.onResponse = function(http) {
  var uri = http.URI.asciiSpec;
  try {
    if (this.shouldInclude(uri) && !this.shouldExclude(uri)) {

      // Match to request. 
      var theRequest = null;
      for (var i in this.requests) {
        if (this.requests[i].uri == uri) {
          theRequest = this.requests[i];
          break;
        }
      }
      if (theRequest != null) {
        var response = {};
        response.uri = uri;
        response.timestamp = Date.now();
        response.method = http.requestMethod;
        // .... missing but parses more of the response 

        // Parse the response headers
        var visitor = new HttpVisitor(http);
        response.headers = visitor.walkResponse();
        theRequest.response = response;
      } else {
        this.missedResponses++;
      }

    } else {
      this.ignoredResponses++;
    }
  } catch (e) {
    console.log("Exception", e);
  }

};

Parsing the Request or Response Headers

From above we can see we pass the request to HttpVisitor then call .walkResponse() or .walkRequest()
HttpVisitor.prototype.walkRequest = function() {
  this.headers = {};
  // Tell the request to
  this.http.visitRequestHeaders(this);
  return this.headers;
};

HttpVisitor.prototype.walkResponse = function() {
  this.headers = {};
        // Tell the response to
  this.http.visitResponseHeaders(this);
  return this.headers;
};

// The callback 
HttpVisitor.prototype.visitHeader = function(name, value) {
  this.headers[name] = value;
};