node http.get() -- how do I get at the XML returned so I can do stuff with it?

Problem

I am trying to make an HTTP GET request to an XML API, wait for the returned XML string, and convert it into a JS object and eventually to JSON. The xml2js module should work fine for the end of that process, but I'm trying to use node's native http.request / http.get methods to make the request, and I don't understand how it works exactly even after reading the docs several times.

Here's a gist of what I'm trying to do, and how I'm receiving a whole lot of etc that I don't know how to parse.

https://gist.github.com/jasonrhodes/6942015

How does the http.get() method work in this case?

Problem courtesy of: rhodesjason

Solution

I am assuming you are using version 0.10.*. What happens is that HTTP can send requests in chunks, which means the response may come in many different pieces.

You should have a String object and concatenate the chunks as you receive them to this object, after there are no more chunks a end event is emitted by the response object.

The code follows:

var http = require('http');
var express = require('express');
var app = express();

// app.use(express.static(__dirname + '/public'));

app.get("/", function (req, res) {
    var options = {
        hostname: "search.mysite.com",
        path: '/search?site=hub&client=hub_frontend&output=xml_no_dtd&q=cats'
    };

    var gsaReq = http.get(options, function (response) {
        var completeResponse = '';
        response.on('data', function (chunk) {
            completeResponse += chunk;
        });
        response.on('end', function() {
            console.log(completeResponse);
        })
    }).on('error', function (e) {
        console.log('problem with request: ' + e.message);
    });

});

app.listen(3000);
Solution courtesy of: Augusto Pascutti

Discussion

There is currently no discussion for this recipe.

This recipe can be found in it's original form on Stack Over Flow.