rogerz notes
octave vectorizer tips

Get the index of minimum element

a = rand (6);
idx = (1:1:size(a)) * (a==min(a));

Slice a matrix

idx = [1 3 2 3]';  // the second and fourth row equals 3
idx == 3;  // [0 1 0 1]
find(idx==3);   // [2 4]
X(find(idx==3), :);  // get the second and fourth row of X
implementation of select() in lwip

lwip has its own select() implementation named lwip_select(). There are two key points in the implementation of select().

First, the file descriptor needs to provide an interface to check the readiness of reading or writing.

In lwip socket, it is implemented in lwip_selscan() which will check the data and event binding to a socket.

if (sock != NULL) {
  lastdata = sock->lastdata;
  rcvevent = sock->rcvevent;
  sendevent = sock->sendevent;
  errevent = sock->errevent;
}

Second, the select() function should provide a mechanism to suspend current thread and wake it up when it is ready for read or write.

lwip uses a global list select_cb_list to hold all the thread pending on select().

/* Put this select_cb on top of list */
select_cb.next = select_cb_list;
if (select_cb_list != NULL) {
  select_cb_list->prev = &select_cb;
}
select_cb_list = &select_cb;
/* Increasing this counter tells even_callback that the list has changed. */
select_cb_ctr++;

Here select_cb is a local struct in select(), which stores the readset, writeset and exceptset passed by select(). A semaphore is created to let the thread wait on and be waken up later.

  waitres = sys_arch_sem_wait(&select_cb.sem, msectimeout);

Then how do we wake the threads up? It is inside event_callback() which is triggered whenever there is something new on socket. It traverse the select_cb_list and wake the corresponding threads up by signal the semaphore.

while (1) {
    sys_sem_wait(selectsem);
    for (scb = select_cb_list; scb; scb = scb->next) {
        if (scb->sem_signalled == 0) {
        /* Test this select call for our socket */
        if (scb->readset && FD_ISSET(s, scb->readset))
            if (sock->rcvevent > 0)
                break;
        if (scb->writeset && FD_ISSET(s, scb->writeset))
            if (sock->sendevent)
                break;
        }
    }
    if (scb) {
        scb->sem_signalled = 1;
        sys_sem_signal(scb->sem);
        sys_sem_signal(selectsem);
    } else {
        sys_sem_signal(selectsem);
        break;
    }
}
using jquery-validation without a form

jquery-validation offers client-side validation to html form. There is a limitation that the elements to be validated must be children of form. A lot of people has been asking about the possibility of not using a form on stackoverflow.

It could be achieved with a small patch.

The original jquery-validation plugin stores the rules in the form node and referencing the form by element.form. So all we need to to is to replace all occurrence of element.form with $(element).data('form')

What makes a good JSON Editor

I’m seeking for a json editor. After trying every result in the first four pages of google search result, the one named JSON Editor Online looks the best.

Why do I make the choice? What makes a good JSON Editor。

Besides the clean design and style, this one also has some attracting features.

  1. Add node or delete existing one by one click

  2. Duplicate button makes array construction easy

  3. You can drag the node

But still there could be some improvements

  1. Import json from URL

  2. Indicate the error position. JSONLint does a good job on it.

Another special editor is the JSON editor with schema

Inspecting hello world of flatironjs

flatiron is yet another framework for building web application based on nodejs. If you feel tired of connect and express, give it a try. Follow the instruction on Getting started with flatiron to create the hello world project.

var flatiron = require('flatiron'),
    path = require('path'),
    app = flatiron.app;

app.config.file({ file: path.join(__dirname, 'config', 'config.json') });

app.use(flatiron.plugins.http);

app.router.get('/', function () {
  this.res.json({ 'hello': 'world' })
});

app.start(3000);

It looks simple, but there’s a lot behind it. By going deep into the hello world example created by flatiron, we can get a knowledge of how it works. The powerful node-inspector will help us to achieve it.

Launch node-inspector and start the app in debug mode.

$ node-inspector&
$ node --debug-brk app.js

Open localhost:8080 in Chrome, and the app will break at first line. You may now step into each function to know what exactly happens behind the page.

broadway for plugins glue

app.js:7, (anonymous function)()
     app.use(flatiron.plugins.http);

broadway/app.js:160,    App.prototype.use()
     this.plugins[name].attach.call(this, options);

     this: app
     name: 'http'
     this.plugins[name]: flatiron.plugins.http

flatiron/plugins/http.js:50,     exports.attach()
     app.router = new director.http.Router().configure({async: true});

The flatiron app inherits use method from [broadway] (http://github.com/flatiron/broadway), which is a layer to manage the plugins. broadway call the attach method of each plugin, in this example, flatiron.plugin.http. The plugin attach a new director.http.Router to the app as app.router.

director for router

registering methods

director/http/index.js:32, exports.Router()
     this.extend(exports.methods.concat(['before', 'after']));

     exports.methods = ['options', 'get', 'post', 'put', ...]

director/router.js:627, Router.extend()
     (function(method){...}(methods[i]);

     methods = ['option', 'get', 'post', ...];

director/router.js:619, Router.extend()
     self[method] = function () {...; self.on.apply(...);}

     self: app.router
     method: 'get'

app.router comes with standard http request methods such as get, post and so on. The methods e.g. app.router.get is extended by director.Router.extend(), see the bottom of call sequence. It also use self.on.apply(...) to register the callback on the method get.

creating routes

app.js:9, (anonymous function)()
     app.router.get('/', function () {...});

director/router.js:625, (anonymous function)
     self.on.apply(self, extra.concat(Array.prototype.slice.call(arguments)));

     extra: 'get'
     arguments: ['/', function]

director/http/index.js:59, Router.prototype.on = function () {}
     director.Router.prototype.on.call(this, method, path, route);

     method: 'get'
     path: '/'
     route: function

director/router.js:216, Router.prototype.on = Router.prototype.route = function (...){}
     this.insert(method, this.scope.concat(path.split(new RegExp(this.delimiter))), route);

     method: 'get'
     this: app.router
     this.scope: []
     route: function

director/router.js:565, Router.prototype.insert = function (method, path, route, parent)
     parent[method] = route; // app.routes[method] = route;

     method: 'get'
     path: ['', '']
     parent: this.routes
     this: app.routes
     route: function

By calling get with a path and a callback, director insert a new route internally, it will be used by dispatch when the server get a request.

dispatching

stream.js:80 Stream.prototype.pipe = function () {...; function onend() {...} }
     dest.end()

events.js:88, EventEmitter.prototype.emit
     listeners[i].apply(this, arg);

events.js:156, EventEmitter.prototype.once
     listener.apply(this, arguments);

director/http/index.js:124, Router.prototype.dispatch = function () {...;function parseAndInvode() {...}}
     self.invoke(runlist, thisArg, callback);

     runlist: [function]
     thisArg: {req: req, res:res}
     callback: function error(){}

director/http/index.js:341, Router.prototype.invoke
     _asyncEverySeries(fns, function apply(fn, next) {...}), function() {//callback});

     fns: [function]

director/http/index.js:346, Router.prototype.invoke
     fn.apply(thisArg, fns.captures.concat(next));

     fn: function
     thisArg: {req: req, res: res}

app.js:10, app.router.get('/', function() {...});
     this.res.json({ 'hello': 'world' })

A notable fact is that the dispatch will parse the route and invoke a run list. One request will result in a sequence of calls. The director router is actually quite complicated. See the document on project page for details.

union for middleware

Please ignore the first few calls in broadway. It just give us the information that broadway will initialize all plugins before start.

app.js: 13, (anonymous function)
     app.start(3000);

flatiron/plugins/http.js:60, exports.attach:app.start
     app.init(function(){});

flatiron/app.js,31, App.prototype.init
     broadway.App.prototype.init.apply(this, Array.prototype.slice.call(arguments));

broadway/app.js:127, App.prototype.init
     this.bootstrapper.init(this, initPlugins);

broadway/bootstrapper.js:53, exports.init
     broadway.plugins.config.init.call(app, function(err){});

broadway/plugins/config.js:44, exports.init = function(done);
     return err ? done(err) : done();

     done: function(err){...;initPlugins();}

broadway/app.js:111 App.prototype.init:function initPlugins()
     async.forEach(Object.keys(self.initializers), initPlugin, ensureFeatures);

broadway/app.js:107 App.prototype.init:function initPlugin(plugin, next)
     self.emit(['plugin', plugin, 'init']);
     next();

broadway/app.js:91, App.prototype.init:function ensureFeatures()
     features.ensure()

broadway/features/index.js:10, exports.ensure
     return callback();

broadway/app.js:85 App.prototype.init:function onComplete()
     callback();

flatiron/plugins/http.js:69 export.attach:app.start()
     app.listen(port, host, callback)

flatiron/plugins/http.js:79 export.attach:app.listen()
     app.server = union.createServer({
          after: app.http.after,
          before: app.http.before.concat(function (req, res) {
               if (!app.router.dispatch(req, res, ...){...}
          }),
          ...
     });

union/lib/core.js:52, core.createServer
     return http.createServer(requestHandler);

The last two calls show that union insert some callbacks(app.http.after is also an array) when creating the native http server. There’s a comparison between union and connect on the project page.

nconf for config

broadway/plugins/config.js:46, exports.init
     this.config.load();

nconf/provider.js:372, Provider.prototype.load
     loadBatch(getStores(), callback);

     getStores(): [{/*file*/},{/*literal*/}]

nconf/provider.js:328, Provider.prototype.load:function:loadBatch(targets, done)
     async.map(targets, loadStores, function (err, objs) {}

nconf/provider.js:319, Provider.prototype.load:function:loadStore(store, next)
     next(null, store.loadSync())

nconf/store/file.js:126, File.prototype.loadSync
     this.store = data;     // data: file content of config.json

It will load the content of configuration file to this.store.

NOT in hello world

There are more component in flatiron framework, which is not mentioned in the hello world example, e.g. plate for template, resourceful for data managment, Winston for logging.

Best practice of periodical task in RTOS

It is very common to have some periodical task in RTOS, e.g. checking for updates every second.

Not so good

A common mistake for the beginners is using delay in the thread.

while(1) {
    // do something
    delay(1000);
}

Obviously, do something will cost some time and the period is actually longer than 1000ms you set. delay is for delay, it is to ensure not doing something in a short time rather than do something immediately after 1000ms. When the delay is up, the task could still be blocked by other high priority tasks.

An immediate idea would be put the action in timer handler. It will be much preciser. But don’t forget that the timer handler usually runs under interrupt context. It is function limited no priority management (always the highest as it is in interrupt. So you can’t put a complex task in timer handler.

So how should we solve this?

Recommended implementation

My idea is to create a periodical timer but put only a signal generator in it. And create a separate task to receive the signal and do the things. The signal could be message queue, event flags or semaphore, depending on your application.

Take the message queue for example, the timer handler should do nothing but send a message periodically.

timer_handler(){
    send_message(&queue, msg);
}

In the task, it receive message and do the job

while(receive_message(&queue, &msg)) {
    // do something
}

The interval between each execution may not be precise due to task racing. But the average execution rate will be satisfied. Even when there’s a temporary congestion that delayed the execution more than a whole cycle. The system could still catch up later because the message is buffered in the queue.

Think carefully what is your actual requirements. Maybe there’s no need to catch up with the normal cycle when there’s an overrun (new signal is generated before the old one is processed). In this case, you could consider using event flags. Which can not be set repeatedly before last one is cleared.

Semaphore is also an economic choice if you don’t need to pass any data to the execution.

Summary

Separating the trigger and execution in timer and task will make the system more predictable, the most important feature for a RTOS application.

x11 forwarding with putty and Xming on Amazon EC2 instance

It won’t work with default settings. Besides the normal steps:

  1. setup Xming and start
  2. create session in putty and enable Connection->SSH->X11->Enable X11 forwarding and set X display location to :0.0

You need to install xorg-x11-xauth in addition

[ec2-user@ip-10...]$ yum install xorg-x11-xauth
git based configuration mangagement

It will be a wrapping of git commands. The common use cases are

  • Fetch script or bundle from remote host
    • specifying the branch and name
  • Committing the script
    • self-containing of git url
  • List all scripts and bundles available
  • Compare with old version
  • Branching for local environment
    • windows / linux
    • different host
Understanding twitter oauth

The following analyse is based on the nodejs module everyauth. But during the study, I found the oauth site has already given a clear description on the authentication flow.

When using oauth for third party authentication, there are three parties of course.

  • user: has an account in provider and want to use it in another service
  • service: is what the user want to access, a.k.a consumer to the provider, e.g. cc.jit.su
  • provider: provides account and authorization e.g. twitter

The steps and what happens behind is as following

  1. user<=>service: user access the entry path on service, url: http://cc.jit.su/auth/twitter
  2. service<=>provider: service host (not user host) will ask the provider for request token and request token secret (a.k.a. oauth token and oauth token secret), url: https://api.twitter.com/oauth/request_token
  3. user<=>provider: service host redirect user to the provider and ask user to authorize the request token. url: https://api.twitter.com/oauth/authorize?request_token=xxx
  4. user<=>service:, provider redirect user to callback path to trigger next step, url: http://cc.jit.su/auth/twitter/callback?request_token=xxx&request_verifier=yyy
  5. service<=>provider:, service host contact the provider to exchange request token for access token and access token secret. url: https://api.twitter.com/oauth/access_token
  6. user<=>service<=>provider: After that service could use the access token to access user’s resources on provider

There are three pairs of key/secret.

  1. consumer key / consumer secret, stored on service host. The request for oauth token will include consumer key as a parameter and signed with the consumer secret. This is the way for provider to identify a valid consumer. It is setup in the provider site and configured in the service host.

  2. request token / request secret, used for exchange the access token. It is generated randomly by the provider and valid for only once.

  3. access token / access token secret, saved in the user session on service host. Service host will use access token and sign the request with access token secret to access resources on provider. It is also setup in the provider site: https://dev.twitter.com/apps

The request verifier returned by the provider is usually quite short to allow user entering manually on out of browser applications. Whatever, the consumer should use request token along with verifier to exchange for access token and secret.

Twitter oauth API reference

How to evaluate a flash file system

Digest of a chat with prife

  1. node identification stored in flash pages (parent, serial, path, data offset, version…)
  2. tree structure in memory (how to locate the header page for given path)
  3. file writing, appending, directory entry (file and folder) creating and removing (extension for topic 1)
  4. traverse of directory (extension for topic 2)
  5. read writing performance under different occupy rate
  6. memory cost, both fixed (capacity independent) and variable (capacity dependent)
  7. read writing performance under different occupy rate and garbage collection
  8. features such as multi-thread and cross device support