Sunday, October 23, 2016

nodejs postgres pg.client returning partial results in json

i had a very weird problem with a nodejs and postgre app i'm building:
i'm running a sql query with join in dbeaver:
select matches.match_date, t1.team_long, t1.team_short, t1.id, 
t1.logo, t2.team_long, t2.team_short, t2.id, t2.logo 
from teams t1, teams t2, matches 
where t1.id=matches.host_team_id 
and t2.id=matches.guest_team_id
and in dbeaver it is returning the expected result
match_date team_long team_short 
2016-11-22 Juventus JUVE 
id logo team_long team_short id logo
6 1juventus.png Everton EVE 7 1everton.png
which includes all expected columns.
but when run in my node app with pg client, i'm only receiving a portion of the columns in the resulting json:
{"match_date":"2016-11-22","team_long":"Everton",
"team_short":"EVE","id":7,"logo":"everton-badge-2014-90x90.png"}
in the above json, the data for Juventus is missing.
for some crazy reason, i thought.

here is the js code i'm using to read the data in node:
// SQL Query > Select Data
const query = client.query(' select matches.match_date, 
t1.team_long as host_team_long, t1.team_short 
as host_team_short, 
t1.id as host_id, t1.logo as host_logo, 
t2.team_long as guest_team_long, 
t2.team_short as guest_team_short, t2.id guest_id, 
t2.logo as guest_logo from teams t1, 
teams t2, matches where t1.id=matches.host_team_id 
and t2.id=matches.guest_team_id');

// Stream results back one row at a time
query.on('row', (row) => {
    console.log(row);
    resultData.push(row);
});
so, somewhere here hides the key to the weird behavior.
but what could it be???
is it something in the postgre db 'driver'?
or nodejs itself?

about an hour of guessing later, i turned to write a question about this problem on stackoverflow.com.

copy/paste the sql query, copy paste the results...should i include the column names as well?
let's do it for readibility...

wait a second!

the column names for the same fields of different records are the same!
but, but, but - how will this project in the json results?

apparently you cannot have two properties with the same name so....SOOOOOOOO...here lies the reason for my 'trimmed' results!

whatever json lib the pg client is using it doesn't care (perhaps rightly so, perhaps not) that we may be overwriting values in result of a sql query


 schoooop

quickly changing the sql query with custom result column names (using 'as') solved the problem! here are the correct and beautiful results:
{ match_date: '2016-10-22',
  host_team_long: 'Atlético de Madrid',
  host_team_short: 'ATL',
  host_id: 3,
  host_logo: 'Atletico_Madrid.png',
  guest_team_long: 'Internazionale',
  guest_team_short: 'INT',
  guest_id: 8,
  guest_logo: 'Inter_Milan.png' }


on to the next problem!
cya!

Saturday, February 06, 2016

Let me tell you about Hapi - building a REST API engine for Nodejs

(i will be showing something a built with hapi - check it out here: https://cloudy-api.herokuapp.com/documentation)

 i would admit again and again that i missed the javascript revolution, but i am trying to catch up!
first i had an idea for a small site where i could just drop some video links and be able to play videos off the site - instaplaylist.co is the result of this experiment.

the only thing that made it possible for me to create this site was the magnificent MeteorJS full stack framework.
at the time i started playing with Meteor i had barely an idea about fullstack, nodejs and what you can do with javascript (both server and client side).

well, Meteor opened my eyes to a wonderful new world indeed - i learned quite a lot, and my next web app will be created with it for sure.

but now i had an idea for a mobile app.
Meteor actually also works with mobile as well - thanks to integration with Cordova and this seems to work great, but i wanted something more flexible on the frontend/clientside and found out about Ionic - which seems like a great choice for hybrid cross platform apps - i was instantly sold.
yet for the backend i wanted to try something new - i wanted to play and get to know the mighty Nodejs.

i had heard about Express, which seems like the best engine for node, but i wanted something a little higher level then that i quickly found out about Sails.
Sailjs seemed a great project, very promissing and with a lot of features. but one thing started bothering me while i was reading the web about it - and that was that the project seemed on the decline in terms of support and progress.

so i kept looking around and then there was Hapi.

i played a bit for some example projects and then started creating my own stuff.
and this turned out extremely easy to do!
first you create your server:
const server = new Hapi.Server();
server.connection({
  port: process.env.PORT
});
and then, since we're building an API - start adding endpoints. as simple as:
  server.route({
    method: 'POST',
    path: '/dosomething',
we just said the server that we're exposing some functionality to react to HTTP POST requests at "/dosomething". let's continue with the "config" section
    config: {
      tags: ['api'],
      description: 'create user account with email and password',
i'm using swagger to document my api and you should too! read more about swagger and what else it offers for testing the api and much more
      auth: false,
"auth" defines how and if we want to limit access to this endpoint - to this resource. more on this later
      validate: {
        payload: {
          email: Joi.string().email().lowercase().required(),
          username: Joi.string().alphanum().min(3).max(30).optional(),
          password: Joi.string().required()
        }
      },
behold one of the beauties of hapi: it allows you in a declarative way to define validation of the expected input it receives from connected clients. this is done with the help of the excellent Joi validation package. so here for our POST request we say that the body should be a json conforming to a certain schema, with fields which Joi will later to ensure are valid email, username and password. and we can even declare which fields we require and which are optional!
      handler: function(request, reply) {
        // Create mongodb user object to save it into database
        var user = new UserModel(request.payload);

        // Call save methods to save data into database
        // and pass callback methods to handle error
        user.save(function(error) {
          if (error) {
            return reply(Boom.badRequest('user.save failed: ' + error.message));
          } else {
            return reply().code(201);
          }
        });
      }
    }
  });
the handler is where we process the request and return response to the client. i have a few models created by the superb mongoose. my data is not that complex, so i just created a few mongodb collections. at some time i might go back to some relational magic with postgre. note also the line mentioning Boom. Boom is a quite handy package, perhaps especially for REST APIs where you communicate through HTTP verbs and notations and responding with meaningful HTTP codes is quite important. well, Boom helps you with that.

Let's get back to that "auth" thing.

there are different ways to handle security and nodejs and related technologies being based around web programming are well all into that. hapi also offers - or should i say, there are many authentication packages build for hapi, which help you select your approach and build a secure site. i my case i decided to use json web tokens with the hapi-auth-jwt2 package.

this is how it works:

server.register([require('hapi-auth-jwt2')], function(err) {
  server.auth.strategy('jwt', 'jwt', {
    key: process.env.JWT_SECRET,
    validateFunc: validate,
    verifyOptions: {
      ignoreExpiration: false
    }
  });

  server.auth.default('jwt');
first we need to register the jwt2 package with the server and tell it that by default any endpoint is secured with jwt tokens. then for every route we need to secure we just add this in the config section:
    config: {
      auth: 'jwt',
and thats really it - from now on, if any access to the endpoint doesn't include a jwt token, access will be denied! if we want an endpoint or other resource to not be restricted, just put
auth: false 
as in an earlier example. of course you may provide a function that is ensuring the token complies with some additional logic, that's all up to you, for a serious app you'd likely have to do this, the function signature is
var validate = function(decoded, request, callback)

so, add more routes, read more, follow best practices and don't forget to start the server!
  server.start((err) => {

    if (err) {
      throw err;
    }

    console.log('Server running at:', server.info.uri);
  });


so this is basically it from me.
i'm quite new to hapijs, and not really familiar with many things/approaches etc, but one learns and has fun, so let the party continue

the github repo with the hapi api is at https://github.com/esdee-git/rest-api-with-hapi

hope that this may help someone!

Tuesday, January 05, 2016

emulator: ERROR: unknown skin name 'nexus_5'

while playing with ionic everything worked fine with the ios emulator, but i got this error when starting the android emulator:
emulator: ERROR: unknown skin name 'nexus_5'
all articles on stackoverflow advised to remove the default created avd image and create a new one. this is fine, but how do i find and run the avd manager on mac?
well there is a quick tip for this too:
go to where android is installed, for me it was
/Users/username/Library/Android/sdk
i cound't find anything named 'avd' there though, but it turned out the correct way to invoke this is
./android avd
now, the avd manager should run and you can create a new virtual device.
but why did we receive the error in the first place?
i my setup there was a default created device for nexus_5, so i decided to run it and...got the same error. so the virtual device itself was bad? turned out an easy fix - avd itself tells you if there is a problem with the device and it was easy to just set the skin to a valid one - probably on osx the android sdk somehow doesn't provide the nexus_5 skin?

after this running
ionic emulate android
ran flawlessly and the sample app appeared.

i'm very excited about ionic, let's see if something will come out of this too. one point of bother for me is that i only yesterday found that that the current ionic is ionic framework 1, and there is a v2 in the works, to be released around Q3. so should i even start with v1, will it not be obsolete to invest in it, when v2 finally arrives?

just realised this is my first post for 2016 :) Happy New Year everyone!

Saturday, January 02, 2016

fall in love with meteorjs + deploy an app to heroku

yes, fall in love indeed - because for me, a mostly c++ person, the recent developments in web, specifically the rise of javascript as a powerful tool to build apps, went past too quickly to observe and have time to play with. but meteor provides an excellent fundament, a basis, to build javascript/nodejs apps with really not that much frontend knowledge. and i could see this pretty much the first few hours after i started playing with meteor.

why? i had some basic knowledge about javascript, and the knowledge that nodejs is a huge callbacks mess which i don't want to have anything to do with. but here came meteor, and it wraps nodejs so nicely, that i never need to bother about it - and instead with the help of familiar paradigms - RPC, client/server - allows to easily jump and play with things like bootstrap, jquery etc without worrying about servers setup etc

not surprisingly i started with meteor tutorial at http://meteortips.com/first-meteor-tutorial and i must say it is great in laying out the basic principles of building apps with meteor - Spacebars, publications, methods etc. the tempo is well paced and examples quite nice too.

actually i wanted this post to document the steps for deploying the app i built to heroku.com. heroku uses something called buildpacks, to help deploying apps requiring certain development tools in the hosting enviroment. so once you find out that the recommended build pack for meteor is meteor-buildpack-horse the rest is easy, because the github page has detailed info how to deploy. except for one step - if you're using some configuration, provided in a (json) file - e.g. settings.json - in this case the tip provided by geekforbrains at https://gist.github.com/geekforbrains/c8a11d128f13d038b119 is invaluable:
heroku config:add METEOR_SETTINGS="$(cat settings.json)"
mmm...it is good (actually priceless) to have good logging for your app, otherwise it would be quite head-breaking discovering deployment errors - the case "but it works fine on my machine, why doesn't it run on the server? :(", so a simple logging message, 'handling' the situation with the missing settings file, helped me resolve the problem within minutes, instead of, potentially, hours - and that trying to find the error remotely, with potentially multiple deployments and code changes..blah, blah,,,you get the idea

the work on the app is till under progress - it is deployed and available at http://instaplaylist.co. the idea is to drag and drop youtube-hosted videos from various sites and view later on this site (instead of on youtube, which also provides this, and better, and nicer - but hey :)) still under progress - misses a landing page with info etc, but it is fully functional, but lots of ideas to expand it, given time. i'll do my best to post the source to github as well - maybe in a different blog i'll also make a more elaborate post on my experience with the awesome meteorjs

update 18.09.2016: for a new app i had a problem with deploying to  heroku:
- first deploy works fine, logs display meteor 1.2.1
- next day a new deploy - fails, logs display meteor 1.4.1

the solution was to run
heroku config:set BUILDPACK_CLEAR_CACHE=1
(as mentioned here: http://stackoverflow.com/questions/38672782/heroku-rejecting-push-of-a-meteor-app)