Can I perform many asynchronous database requests without deeply nesting my code?

Problem

I'm sort of new to programming asynchronously. I've run into a situation where I need to perform 8 database lookups in a loop. I'm not sure how to accomplish this-- my database library returns the data in a callback function, and I cannot continue with my code until I have all 8 rows, so I need to halt until all 8 lookups have completed.

This is sort of what I picture right now:

db.world.Queue.find(@user.kingdom.troops_queue).on 'success', (troops_queue) ->
    db.world.Queue.find(@user.kingdom.tanks_queue).on 'success', (tanks_queue) ->
        #etc etc

This is horrible and gross of course, but I can't think of a way to roll it up into a loop that will allow my code to pause and only continue when the last item has been filled. I was looking into things like jQuery's .each() function, but what is the behavior of that function? Does the code after it immediately continue, or does it wait for the loop to finish?

Problem courtesy of: Nicholas Flynt

Solution

There are two commonly used ways. The first one is using a library like caolans async:

async.parallel
  a: (cb) -> doTaskA cb
  b: (cb) -> doTaskB cb
, (err, {a, b}) ->
  # you can use err, a and b here now

The second approach is streamlinejs:

troops_queue = db.world.Queue.find(@user.kingdom.troops_queue).on 'success', _
tanks_queue = db.world.Queue.find(@user.kingdom.tanks_queue).on 'success', _
# go on here

However, both solutions assume that the first argument of the callback is the error - if it isn't that way, you should bug the author of the library you're using to change it.

Solution courtesy of: thejh

Discussion

I always use Step for my projects, it's very clean and nice.

Here's an example of some callbacks in a row:

Step(
  function readSelf() {
    fs.readFile(__filename, this);
  },
  function capitalize(err, text) {
    if (err) throw err;
    return text.toUpperCase();
  },
  function showIt(err, newText) {
    if (err) throw err;
    console.log(newText);
  }
);

You may also want to do some parallel stuff and only then trigger the callback, fortunately this.parallel() is just the thing for that:

Step(
  // Loads two files in parallel
  function loadStuff() {
    fs.readFile(__filename, this.parallel());
    fs.readFile("/etc/passwd", this.parallel());
  },
  // Show the result when done
  function showStuff(err, code, users) {
    if (err) throw err;
    console.log(code);
    console.log(users);
  }
)
Discussion courtesy of: alessioalex

Use flow control like after.js

var data = {};
    cb = after(8, function ( ) {
        // handle all 8 results
    }),
    queue = db.world.Queue,
    kingdom = this.user.kingdom;

queue.find(kingdom.troops_queue).on('success', function (result) {
    data["troops_queue"] = result;
    cb();
});

// etc

P.S. I fixed that coffeescript for you.

Discussion courtesy of: Raynos

Since you tagged with "coffescript" some links you can check (long threads!): issue241, issue287, issue350, issue1710 (and probably some more...)

In a nutshell:

Discussion courtesy of: tokland

You could organize your queries in a 'successful progression' sequence; this sequence can be used by a continuation-style loop, kinda like this in Javascript:

var lookups = [
   function(){ Q.find(@user.kingdom.troops_queue; },
   function(troops_queue){ Q.find(...tanks_queue; },
   function(tanks_queue){ Q.find(...next_queu; },
   ....
];

And this 'stratified' sequence can be drewn through the asynchronous mill like

function proceedOn( sequence, previous_result ) {       
   var first = sequence[0];
   first( previous_result ).onSuccess( function(result){
     proceedOn( sequence[1..], result );
   } );
}

proceedOn( lookups );

(Where I took some poetic liberties on array syntactic sugar)

Discussion courtesy of: xtofl

This recipe can be found in it's original form on Stack Over Flow.