Archive for the ‘erlang’ Category

New version of ec2nodefinder

ec2nodefinder is an application that enables remote erlang node discovery when hosted on EC2.

This new version uses the EC2 query interface instead of os:cmd’ing the amazon api tools.

It has no external dependencies now when deploying the release. Previous needed Java and the API tools.

Finally removed the need for the cert and pk files. It only uses AMAZONACCESSKEYID and AMAZONSECRETACCESSKEY.

Also new is an implementation of V2 signature code for AWS.
Given that V1 is deprecating at the end of the year, that’s a head start.

I did not see it anywhere in erlang yet, so HTH (as they say).

TODO : the secret key tend to come up in the logs. It will be removed in an upcoming release.

Advertisements

Building OTP applications with Rake for EC2

I uploaded a mashup of other’s code this morning to github. Namely charpi’s Rakefile, used to build the Duke of Erl’s ec2nodefinder.

Rakefile

It’s way more than a Rakefile, actually. It’s a whole project organization.

Your OTP applications go into lib/, and you can have as many as you want in there.

Just typing rake will build the release files, doing all the chores (creating app files and rel files … BORING !)

Here are all the defined tasks.


rake clean                     # Remove any temporary products.
rake clobber                   # Remove any generated file.
rake compile                   # Compile all project
rake dialyzer                  # Run dialyzer
rake edoc[name]                # Buid Application documentation
rake edocs                     # Buid all application documentation
rake erlang_applications       # Build application resource file
rake erlang_modules            # Compile Erlang sources
rake erlang_release_files      # Build erlang boot files
rake erlang_releases           # Build release tarball
rake erlang_target_systems[n]  # Build release tarball with erts
rake run[name,node]            # Runs application
rake tests                     # Runs EUnit tests
rake upload                    # Sends release to S3 (presumably for deployment on EC2)

ec2nodefinder

In a nutshell, makes all erlang nodes in a single EC2 security group discover each other automatically.

Your applications on those instances can have processes joining pg2 groups, and you get scalability with very little efforts. However pg2 is quite crude in choosing a process as its only strategy is closest process or random.

ec2nodefinder is described here. Works well and very handy. I however would like to remove the dependency on ec2-describe-instances (which in turn requires a JRE on the deployment server). It’s working as advertised but JVM startup slows down discovery.

AMIs

Check out Kevin Smith’s blog. He is building erlang centric AMI based on Eric Hammond’s Ubuntu AMIs

Next cool things I’d like to do or see

Run OTP releases on instances, with just one command :


ec2-run-instances ami-XXXXX -n  -d "release=/"

It won’t be that easy, because the AMIs will need credentials to access S3 (and EC2 admin commands)

Actually that’s my 10€ question. Where are AWS credentials best stored ?

  • have the cert / private key on S3 and pass the S3 AWSACCESS* values as parameters :

ec2-run-instances ami-XXXXX -n  -d "release=/&AWS_ACCESS_KEY=XXXXX&AWS_SECRET_ACCESS_KEY=XXXX++XXXX"
  • pass everything in a tar.gz (-f flag of ec2-run-instances)

Not decided yet.

relup, appup … hot code upgrade

No rake magic yet for those. Watch this space or charpi’s !

Code

Get the code on github.

Validating Atom entries as PubSub payloads in ejabberd

Small bit of code to validate Atom when in xmlelement tuples.

Written for the pubsub component of ejabberd, integration to follow shortly.

Not perfect yet, but should still be useful. Patches appreciated.

Find it on github !

Should have full EUnit coverage too.

mod_couch : embedding ecouch client for CouchDB in ejabberd

Quick and dirty :

  • checkout, compile and copy the ecouch directory somewhere erlang will find it. Mine is /usr/local/lib/erlang/lib

  • download this, compile, copy the beam file in the ejabberd ebin directory.

  • in ejabberd.cfg :


{modules, [
.
.
.
.
  {mod_couch,     [{server,{"127.0.0.1", "5984"}}]},
.
.
.
]}
  • restart ejabberd

  • You now have access to your CouchDB server within ejabberd.

Atompub interface for distributed microblogging over XMPP

atompubsub has been updated and simplified and renamed to atommicroblog. Some might argue that it’s useful now.

The scope of the plugin is reduced to one node : the “urn:xmpp:microblog” node of the upcoming Microblogging over XMPP XEP.

Also improved : removed dependency on Yaws and implemented caching like they want it.

The code is on the ejabberd_modules svn.

For full context, these mails tell about production use.

Once this patch is implemented, it’s going to be even better. I will move entry payload processing into a nodemacroblog. And maybe I’ll find a more generic interface for atommicroblog.erl.

To the broader subject of distributed microblogging over XMPP, you might want to read Jack Moffit’s blog post. Follow the links !

PS : Yup, very pompous title.

Writing ejabberd modules, good places to find documentation [Updated]

Just found on the ejabberd mailing list Anders Conbere’s blog, with quite a lot of information to get you started, and code for the http server, write a bot.

[U] And Jack Moffit in the comment directed me to his weblog. Very good read about ejabberd deployment and administration.

Both of these live well in my NetNewsWire subscriptions.

Other locations

The Process One wiki is also a location to bookmark. All the standard hooks are listed.

Last but not least, the source code ! The simple features of erlang make it quite easy to navigate through the source code.

Modules you’ll use on a frequent basis are jlib, ejabberdrouter, xml and xmlstream along with mnesia and lists.

  • jlib : for manipulating JIDs, iq stanzas

  • ejabberd_router : send your stanzas elsewhere

  • xml_stream and xml : for parsing xml into the internal tuple representation or the other way round.

[Updated 2] Atom-PubSub module for ejabberd

As requested, the Atom PubSub bridge

This module offersr an AtomPub interface to ejabberd PubSub data. Currently in two unfinished flavors, one for use with yaws embedded. One for use with ejabberd_http server

Howto

You need to have Yaws available. It will start in embedded mode, with the mod_yaws module (included).
To build, edit the Makefile to match your erlang install and make Put the resulting beams in some place where ejabberd will find them.

Also you’ll need to set the BASEURL macro in atom_pubsub.erl to your webserver hostname.

You’ll also need to add the module to your ejabberd.cfg in the mmodules section:


{mod_yaws,[{logdir, "/tmp/"},
    {servers, [
    {"localhost", 5224, "/opt/var/yaws/www", [
     {dir_listing, true},
     {appmods, {"/atom",     atom_pubsub}}
     ]}
   ]}
  ]}

What you get

The AtomPub interface passes the Atom Protocol Exerciser (though some warnings remain).

It means that any AtomPub clients will be able to post to a specific node in your PubSub tree.

It also means that your PubSub tree will also be available as an AtomFeed.

Of course, each time an item is posted through AtomPub or PubSub on a node you are subscribed to, you’ll get the notification.

Can I have it with OpenFire and Epeios ?

That’s not possible. At some point, there’s no way around hitting directly the PubSub mnesia tables. So you can’t extract the code as a component.

Moreover, it only works with PubSub nodes derived from the default node type. (because of the mnesia tables stuff)

What’s next ?

I’ll update the code soon.
A few of things I’d like to implement :

  • remove all calls to mnesia and work through mod_pubsub API.
  • add HEAD, etag and slug support (that’s a patch for ejabberd though)
  • remove that baseurl horrible macro
  • add node subscription through REST
  • as soon as ejabberd 2.1 is published remove dependency from yaws
  • add binary collections support

Mickaël Rémond from Process-One kindly offered to host atom-pubsub on the ejabberd_modules svn.


svn co https://svn.process-one.net/ejabberd-modules/atom_pubsub/trunk/

There’s a quick port to the ejabberd_http server at this location :
You need to be running ejabberd 2.1 or current trunk to have it work.


svn co https://svn.process-one.net/ejabberd-modules/atom_pubsub/branches/ejabberd_http_branch/

Check out the README for installation.

Shoot your questions in the comment or via email (anything on this weblog domain goes to my inbox)

CookieStore on Yaws

I implemented a session cookie store, just like the one in Ruby on Rails 2.0.

Available with the same caveats :
Session data is encoded in base64 and sent in the cookie with a SHA MAC of this data.
This means that the user can see what’s inside, but will not be able to tamper with it.

Moreover session data should stay in small amount as the encoded and signed data may not exceed 4096 bytes.

This being said, that should give us Yaws clustering for free 🙂
And no more sessions to expire, just set the cookie expiration date.

One small thing, make sure crypto is started.

session1.yaws has been rewritten to make use of this code.

Download here.

mod_rpc : Jabber-RPC within ejabberd

What is mod_rpc ?

mod_rpc is an ejabberd module which will handle rpc queries … in a modular way.

It is is easily extensible, and is designed to access the mnesia database from XMPP clients.

It plugs in the access control list to allow or prevent access to the rpc modules.

Installing mod_rpc

Download the mod_rpc.erl file and copy it over your ejabberd src/ directory.
make ejabberd.

Do not restart yet, we have some configuration to do !

Using mod_rpc

Let’s say you want to publish two functions, echo and mult. The code would go as follow :

rpc_test.erl :


-module(rpc_test).
-export([handle/2]).

handle(_State, {call, echo, [A]}) -> {response, [A]};
handle(_State, {call, mult, [A, B]}) -> {response, [A*B]};

Copy into your ejabberd source directory and make.

Now you need to configure access to your functions.
In

ejabberd.cfg


% 2 groups : admins, and the rest.
{access, rpc_admin, [{allow, admin}]}.
{access, rpc_all, [{allow, all}]}.
%...

%... in modules configuration
%...
  {mod_register,   [{access, register}]},
  {mod_rpc, [{access, [{rpc_test, rpc_admin}] }]}, % only admins can call echo and mult
  {mod_roster,     []},
%....

Now start ejabberd. (of course you could do the hot code stuff if you want)

From now on I have rpc_test@rpc.localhost answering to my rpc queries.

Let’s test from ruby using xmpp4r. Get the SVN version for jabber-RPC support.


require 'xmpp4r'
require 'xmpp4r/rpc/helper/client'
require 'xmpp4r/rpc/helper/server'
include Jabber
jid = JID::new('cstar@localhost') #this one is admin !
cl = Client::new(jid)
cl.connect
cl.auth("PASS")
rpc= RPC::Client.new(cl, 'rpc_test@rpc.localhost')
puts rpc.call("echo", 'Test string') # outputs Test string
puts rpc.call("mult", 2,4) # outputs ... 8

If you try with a non-admin user, you’d get

Jabber::AuthenticationFailure: not-authorized

About Groovy Jabber-RPC

I have been playing with it, and it does not work directly out of the box. The groovy lib will try to see if rpc_test@rpc.localhost is in the user roster.

Patching to making it work is quite simple ;
In file xmlrpc-groovy/src/main/java/groovy/net/xmlrpc/JabberRPCServerProxy.java

Just replace : request.setTo(getId(connection.getRoster(), this.to)); (line 102)

with : request.setTo(this.to);

And the following will work :


import groovy.net.xmlrpc.*
import org.jivesoftware.smack.XMPPConnection

def clientConnection = new XMPPConnection("localhost")
clientConnection.connect()
clientConnection.login("cstar", "PASS")
def serverProxy = new JabberRPCServerProxy(clientConnection, "rpc_test@rpc.localhost")
serverProxy.echo("test")
clientConnection.disconnect()

Necessary caveats

This is my first foray in developping a module in ejabberd, I still have to check how this actually scales. I only have one process handling all queries, which is not very concurrency oriented programming 🙂

Thanks

The guys from ejabberd, for making software really easy to use and extend 😉

Download :

mod_rpc.erl

Feedback

I really welcome enhancements and fixes (especially regarding the concurrency stuff!)

License

Don’t sue me, don’t remove copyright/name kind of license.

From ejabberd xml to xmerl and back – XEP-009

Those last days I was busing implementing an ejabberd module responding to Jabber-RPC (a.k.a XEP-009).

I used the xmlrpc module shipped with erlang to do the parsing. But that was tricky. xmlrpc uses xmerl for parsing and representing XML but ejabberd does not. It a simpler xml module handling the parsing.

To make a long story short, to get ejabberd’s xml understood by xmlrpc, here’s the thing :

% ....
xmlrpc_decode:payload(xmerl_ucs:to_utf8(
            xml:element_to_string(xml:get_subtag(SubEl, "methodCall")))),
% ....

SubEl being the sub_el field of an iq packet.

The complex part is that you need to call xmerl_ucs:to_utf8/1 else the module will crash.

For returning the resulting XML to the client, you need to call xml_stream:parse_element (ejabberd’s internal XML parser) to get the XML structure right for handling by ejabberd.

case xmlrpc_encode:payload(handle(State,Decoded)) of % handle is the xmlrpc method
 {ok, EncodedPayload} ->
    Res = IQ#iq{type = result,
        sub_el = [{xmlelement, "query", [{"xmlns", ?NS_RPC}],
        [ xml_stream:parse_element(EncodedPayload) ]
               }]
    },
%....

That’s not very optimal of course, as data is converted at nearly every step :

XML -> ejabberd -> XML -> xmerl -> XML -> ejabberd -> XML.

But developper time is more expensive than machine time, right ?

So now I can call methods on the server directly from Groovy using Groovy Jabber-RPC (using XMPPPool, of course).

If there is any interest, I’ll release the code, but for the time being it is very crude. I’ll like to “componentize” a bit more, and hook up ACLs for specific RPC handlers.