N2O: Erlang Web Framework

Introduction

This post implies a good intro into N2O.

In order to learn about Erlang/OTP Web Framework N2O main features you should visit the github page or the official SynRC website. You will find there all the charts and presentations which you like so much.

In this article I will cover the principles of framework operation. Version under consideration: N2O: 1.1.0. It’s always more interesting to see the result than to talk about it. I suggest that you install N2O to your PC first and then you will go deeply into its insides. It’s clearer that way. You can find answers to the arisen questions and recommendations at the official channel IRC #n2o at FreeNode.net.

Installation

Install Erlang if hasn’t been done yet.

Download N2O, compile and run it.

git clone git://github.com/5HT/n2o.git
cd n2o/samples
make && make console

Look at: 127.0.0.1:8000/

Having opened a few windows you can chat with yourself.

A Lot of Theory

Niche

It’s intended, most of all, for development of a horizontally scaled solutions with low delay for client requests processing. Online games, chats, instagrams, twitters – that’s not exactly what Erlang was developed for, but it’s where N2O can be used.

Under the Hood

N2O is a redeveloped Nitrogen project. There is the fastest web-server (Cowboy), operation through the WebSockets, binary data exchange wherever is possible, HTML and DTL templates, minimum JavaScript quantity without using exterior libraries.

You can find the comparison of Erlang web frameworks here.

Using Features:

• NoSQL solutions, such as Mnesia, RIAK, KAI via a simple abstract pattern in the form of KVS application.
• Authorization libraries AVZ (Facebook, Google, Twitter, Github, Microsoft).
• MQS libraries of MQ-queues for RabbitMQ.
• Erlang code interpreter in JavaScript with check capacity during compilation — Shen.
• Any other Erlang decisions, just adding into the application.

N20 compiles source files on-the-fly, even without open sessions corruption. It is based on Sync.

Code without friction.

Server data exchange with clients is implemented via WebSockets and has a minimal overhead. The following data formats are possible:

• BLOB (RAW Binary), perfect for images
• Bert Encoded, for Erlang-terms
• Pickled, coded data in Base64, or AES/RIPEMD160

Clusterization and fault-tolerance are available for Web as never before, with max design simplicity.

Behavior Model

N2O is built as Erlang itself on the basis of message transfer. Not from one process to another, but from client to server. I will call it event-driven model.

Let’s describe an operation timeline of the built-in n2o_sample.

1. During the /index page call the HTML markup is passed to the browser. It’s described in index:body()function.
2. JavaScript does initialize WebSocket connection.
3. Then the payload in the form of JavaScript code and initialization of the page elements by data through index:event(init) are transferred via WebSocket. One of JavaScript roles here is, for example, creating an event for the button at the client in the form of JS-function, that will send to the server information about clicking.
4. After clicking the button, Bert-Encoded Base64 message arrives to the server and index:event(Term) function is performed. Term — is a term, described in the record field of the button: #button.postback. For instance, button{postback=sasay} after being pressed will make index:event(sasay) function perform.

5. The server, in its turn, can also send data to the browser at any time. If the data is sent from another X process, the following conditions should be met:

   • In index:event(init) the main process serving the client (browser’s tab) should be registered under some name with wf:reg/1 call.
   • After completion of generating a code for page reload (wf:insert/2, wf:wire/1 etc.) wf:flush/1 function should be called in X process in order to send the code renew to the browser, where it will be performed by JS-machine.

The first thing comes to mind is that the page dynamically changes not on the basis of functions call from the file with JavaScript, but being based on the obtained JavaScript from the server in real time.

It’s a native interactive mode for a web application without using such workarounds as AJAX and LongPooling Comet. The future is here, darlings. And its name is WebSockets.

Support chart of WS browsers support can be found here. You can test your own browser here.

Application Structure in Details

If you got acquainted with Erlang recently, you will most likely have questions like: what is where located and where can I apply my creative ideas.

Highlighted are the folders and files that will be affected by editing. Let’s consider the key parts.

n2o_sample

N2O already contains an example of user application — n2o_sample. n2o_sample is a separate Erlang application, which works at N2O. As you can see from the picture above that it’s located in apps/ folder. It’s a collection of user applications; we can add ours there if we need to divide n2o_sample into two or more independent applications.

n2o_sample, can be renamed or replaced by something else. But I wouldn’t recommend you to do that in the beginning. Just use the available code as a starting point of your application.

Application list

At n2o_sample startup, it runs all applications it depends on, as is proper for Erlang application (relations from deps/, and user ones, if available, from apps/). This code is located in n2o_sample/src/web_app.erl:

-module(web_app).
-behaviour(application).
-export([start/2, stop/1]).

start(_StartType, _StartArgs) -> 
    application:start(crypto),
    application:start(sasl),
    application:start(ranch),
    application:start(cowboy),
    application:start(gproc),
    application:start(mimetypes),
    application:start(syntax_tools),
    application:start(compiler),
    application:start(erlydtl),
    application:start(rest),
    application:start(n2o),
    web_sup:start_link().

stop(_State) -> ok.

Function web_sup:start_link() runs n2o_sample itself.

But if we want to widen the relation list – we should know that it’s not the only place with application list. We should also fix two more files: reltool.config and .applist. The latter is created after the first make command in the console. Keeping the same things in three places is a temporary workaround which is promised to be fixed.

sys.config

Parameters for all applications are saved here. Into one file. It’s handy. Then from the code we call application:get_env(App,Key) and that’s it.

vm.args

Here you can set virtual machine keys and environment variables (instead of $ export SOME_PARAM=value).

priv/static/

The folder which has been passed to Cowboy web server as a filestore of “statics”. JavaScript code can be placed here. You can also store pictures and hentai here. n2o_sample/src/web_sup.erl:

dispatch_rules() ->
    cowboy_router:compile(
        [{'_', [
            {"/static/[...]", cowboy_static,
                {priv_dir, ?APP, >,
                [{mimetypes,cow_mimetypes,all}]}},
                ****

Pages

For dynamic pages creation N2O allows including HTML files to the project as DTL templates. In deps/erlydtl/ ErlyDTL application folder is located. It’s intended for DTL templates compilation in the Erlang byte-code.

Templates are located in n2o_sample/priv/templates/ folder and look like HTML files with declarations of external data sources via the words that are enclosed in double braces {{ }}. Thus, we can create HTML layout with static information and extract the dynamic information to Erlang by using {{ }}.

Let’s consider login:main/0 function body as an example. The function gives the browser the initial page state when visiting 127.0.0.1/login/:

#dtl{ file = "login",
      app=n2o_sample,
      bindings=[
                {title,title()},
                {body,body()}
                ]}.

• login — is the template name: /priv/static/template/index.html;
• title and body — named enclosures in HTML template {{title}} and {{body}};
• title() and body() — functions, the result of which will be placed into HTML template.

Events

Client-side data exchange is implemented via JavaScript, with the help of /deps/n2o_scripts/n2o/bullet.js (initialization of WebSocket connection) and n2o.js (processing the findings). From the server side the endpoints are: /n2o/src/endpoints/cowboy/bullet_handler.erl (initialization of WebSocket connection) and n2o_bullet.erl (data exchange).

API

Let’s take a look at the key API functions, which lead to most of the questions when getting acquainted with N2O.

wf:comet/1, wf:async/1, wf:async/2

Register the process under a unique name (“comet” for comet/1 and async/1) within the limits of the node via global:register_name/2. If it’s already registered, its Pid is returned.

wf:flush/1

Withdraws with the help of wf_context:actions/0 saved in the current process state changes for the page and sends them via wf:send/2 to the process (within the limits of node), the name of which is given with the parameter.

wf:reg/1, wf:reg/2 (?REGISTRATOR = n2o_mq)

Register the process as Property under a non unique name within the limits of node via GProc. You won’t be able to register the process under the same name one more time. Registration state in stored in the process state (get/1, put/1) and skip (n2o_mq.erl) term will be returned.

By default the registrar is n2o_mq module, but it can be redefined. For example it can e done in order to register processes within the cluster limits.

wf:send/2 (?REGISTRATOR = n2o_mq)

Sends the message, passed to the function by the second argument, to the process, which is registered via wf:reg/1 or wf:reg/2.

wf:q/1

Retrieves data, transferred from the client through the so-called postback elements. For example:

body() ->
    [ #textbox{ id=message },
    #button{ postback={button_pressed}, source=[message] } ].
event({button_pressed}) ->
    wf:info("Message: ~p",[wf:q(message)]);

The text from the textbox at the moment of key press will be printed to the console.

wf:qs/1

Extracts parameters from HTTP forms, for example: wf:qs(>) will extract > if URL would be localhost:8000/index?x=ABC.

wf:wire/1

Can accept as an argument both the JavaScript text and event recoding, declared in “Actions” section of /n2o/include/wf.hrl file, for example: wf:wire(#alert{text=«Hello world!»}). JavaScript will also be wrapped with #wire{actions=JS}, that will lead to construction wf:wire(#wire{sctions=JS}).

wf:update/2, wf:insert_top/2, wf:remove/1 and others

These are particular cases of wf:wire/1, including a ready JavaScript code for DOM changing in the client’s browser.

wf:info, wf:warning, wf:error

These are the functions recommended to be used instead of error_logger:info_msg/1 and others.

wf:f, wf:to_list, wf:to_binary, wf:html_encode, wf:url_encode, wf:hex_encode and others

…are located in the “Convert and Utils API” section of /n2o/src/wf.erl file. All of them are add-ons for standard Erlang functions. But they are cleverer. wf:f — the analog is io_lib:format, further on the roll converter-functions accepting any term, then more web specific functions. There’s no use in describing them in details, I just wanted to show that they exist.

wf:pickle/1, wf:depickle/1 (?PICKLER = n2o_pickle)

Terms coder and decoder for system calls communication between the client and server. n2o_pickle encodes to Base64, while n2o_secret uses AES/RIPEMD160 encryption with random key.

That’s it about API; you can find more information at N2O API.

Worth Paying Attention

Below are listed free software programs of SynRC, which can speed up your production Erlang project in tens of times.

KVS

KVS — an abstract template of K-V noSQL database, feeds charts via double linked lists and secondary indexes (kvs:index/3). For the moment the use is possible with Mnesia, RIAK and KAI.

AVZ

AVZ — an authorization system via Twitter, Google, Facebook, Github and Microsoft.

Shen

Shen — Erlang to JavaScript code interpreter. It allows to use Erlang compiler to validate JavaScript.

MQS

MQS — MQ library for RabbitMQ.

Feeds

Feeds — the handler pool of commands for data correspondence maintenance at all cluster nodes, also the cache server.

SkyLine

SkyLine — an example of an online store built with N2O.

COUNTACH

Countach — a social system and an application store. Production-ready. Uses KVS, AVZ и Feeds. Based on VOXOZ.

VOXOZ

VOXOZ — an open Erlang cloud platform (PaaS). Uses Docker, ErlangOnXen. More information at blog.docker.io.

In Conclusion

In order to let information sink in properly, that’s all for now.