Alrighty, like most of life, I’m not aloud to do just one thing at a time. Let’s see how well I can juggle two blog entries at the same time!

The problem statement is simple: there are assets recording in our database but appear to be missing. Right now I’m trying to obtain an inventory of which assests are missing. Querying the database was rather simple in NodeJS however the streaming interface was a little backwards. I found myself with something like the following:

async function db_query( logger, config, handler ) {
  let pg = require('pg');
  let connection = new pg.Client( config )
  let promise = new Promise( ( accept, reject ) => {
    let query = new pg.Query( "SELECT * FROM s3_assests" )
    query.on( 'row', ( row ) => {
      handler( row )
    query.on( 'end', () => {
      accept() "Query complete" )
    } )
    query.on( 'error', ( problem ) => {
      logger.error( "Error encountered while querying", problem )
      reject( problem )
    } )

    connection.query( query )

  await connection.connect();
  return promise

This allows me to convert the the streaming intreface into a function but still await for the entire stream to close before exiting. Not too thrilling. Next up is decomposing the row for the asset and ensuring I don’t saturate my network connection. Looks like the node package bottleneck will do the trick!

So bottleneck is an interesting package. I overlooked that they inject a parameter for the callback, which lead to a bit of a confusion. As a result I decided it would be cleaner to use the Promise API. Effectively I slapped async onto the handler and called schedule instead of submit. When there task is ready to be run it will invoke the given function to generate the promise and we’re off.