erls3 : OTP application for accessing S3

Just committed erls3 over to github.

Enables access to S3. Tailored for highly concurrent access with small items rather than sending multigigabyte items. Everything you get/send from/to S3 is stored in the VM.

Usage examples will come shortly.

The API is however very straightforward.

Re: XMPP a 10 ans

Je reviens tout juste de la Cité des Sciences et de l’Industrie pour une journée 50% familial,e 50% XMPP et 100% réussie.

Je ne m’attendais pas à découvrir XMPP, mon objectif primaire était de faire une bijection tête <-> JID (c’est toujours enrichissant).

Les présentations étaient assez peu techniques. Certainement un choix assumé – mais le public était composé surtout de techos, et on aurait pu rentrer dans les détails, avec du XML et des XEP.

A noter que la présentation “Introduction à Jabber facile” a du être annulée, et c’était peut-être celle-là qui aurait donné les billes.

J’aurais aussi aimé qu’elle soit plus positive. C’est-à-dire qu’il y avait beaucoup d’arguments “XMPP >> MSN parce que MSN c’est le mal” (donc c’est en négatif). Ce en quoi je suis d’accord, mais il y aurait fallu beaucoup plus insister sur les points forts de XMPP dans l’absolu : l’extensibilité, la facilité de développement d’applicatif au-delà de l’IM.

Avec trois présentations qui ont être annulées, et ayant manqué la présentation de Laurent Lathieyre sur BuddyMob et celle de Jan Torben Heuer, ma perception était probablement biaisée.

De toute manière, faut pas se leurrer, XMPP va finir par gagner tout comme HTTP et SMTP ont réussi dans leur domaine respectif. C’est juste une question de temps, de code et d’évangélisme !

Un gros succès était la présentation de Kael et de Jehan qui nous ont présentés des bots qui présentaient le programme télé, et les téléchargeaient en pilotant VLC. Kael va publier son code soon, me dit-il.

Pour finir, merci beaucoup, Jehan, d’avoir pris le temps et eu l’énergie d’organiser cette journée.

PS : Et si vous voulez fêter l’anniversaire à nouveau, je préparerai peut-être quelque chose. (Ca fait quelques temps que j’ai pas donné de cours, et ça me manque de faire le clown devant des slides).

XMPP a 10 ans !

Pour fêter ça :

Une journée de présentation sur XMPP à la Villette.

Les infos sont ici et ici.

Merci à Laurent de me l’avoir appris !

Je vais y aller pour rencontrer les gens et écouter certaines conf. Et aussi pour emmener le nain aux expositions permanentes de la Cité des Sciences.

Le communiqué complet :

La communauté francophone des utilisateurs de Jabber/XMPP organise un
évènement anniversaire pour les 10 ans du protocole, samedi 28 février
au Carrefour Numérique de la Cité des Sciences et de l’Industrie.

Cet événement a pour but de faire découvrir au grand public comme aux
professionnels les possibilités de ce protocole de messagerie instantané,
ses usages actuels et futurs, ainsi que de faire un état des arts de la
communication instantanée.

L’évènement vous proposera des conférences d’une part, où vous seront
présentés divers projets ainsi que des thématiques générales par
divers acteurs communautaires (Ludovic Gilbon, Jan Torben Heuer, Jehan,
Kael, Thierry Stœhr, Nicolas Vérité) ainsi que des acteurs
professionnels sponsorisant l’évènement (Process One, Ubikod et Violet),
et des ateliers d’autres part.

Nous remercions en particulier (et dans un ordre aléatoire ne reflétant
aucune forme de préférence) l’ensemble de la communauté qui apporte aide
et soutien, nos sponsors, la XSF pour la gestion du développement du
protocole XMPP, ainsi que les associations AFUL, APRIL, FSF France et
Parinux qui nous ont énormément aidés à organiser l’évènement.

Le programme ci-dessous n’est pas entièrement fixé et est donc
susceptible de changer, mais donne néanmoins une idée des sujets
présentés et des horaires.

Conférences:

10H15: « présentation succinte de la journée et du protocole Jabber pour
la messagerie instantanée et la présence sur Internet » (intervenant:
Jehan — durée: 15 minutes)

10H45: « historique, situation et perspectives de Jabber/XMPP »
(intervenant: Nicolas Vérité — durée 45 minutes)

11H45: « L’utilisation de bots pour automatiser la récupération
répétitive d’informations contextuelles » (intervenant: Kael et Jehan –
durée: 15 minutes)

13h15: « BuddyMob : un réseau social mobile basé sur xmpp »
(intervenant: Laurent Lathieyre, Ubikod — durée: 30 minutes)

14h: « Violet et Violet OOOS : La plate forme Violet Open Object Operating
System » (intervenant: Olivier Mével, Violet — durée: 30 minutes)

14h45: « Social networks based on Jabber/XMPP – The power of decentrality
and privacy » (intervenant: Jan Torben Heuer, en anglais)

15h30: « Introduction à Jabber facile » (intervenant: Ludovic Gilbon –
durée 30 minutes)

16h15: « Les formats ouverts, protocoles ouverts et la messagerie
instantanée » (intervenant: Thierry Stœhr — durée: 30 minutes)

17h15: (intervenant: Process One — durée 30 minutes)

Erlang SimpleDB application

SimpleDB

SimpleDB is a the cloud database by Amazon Web Services.

Still in beta, SimpleDB provides you with metered access to fat storage designed for an internet scale database. Compared to MySQL or other RDBMS, it has few features (no transactions, no joins …), but using it is a no-brainer.

Still having a library wrapping around the HTTP calls to SimpleDB is good.

erlsdb

Hence the erlsdb OTP app. However development seemed to have stopped. So I took it to github and hack it.

It went surprisingly quickly (most certainly due to erlang’s power than my own skills) as I managed to add async http calls, multiple workers and finished implementing the API in a few hours.

Still needs a bit of polish but it’s already waiting for feedback !

Get it here !

examples

(if your eyes don’t burn from the syntax coloring)

New version of ec2nodefinder

ec2nodefinder is an application that enables remote erlang node discovery when hosted on EC2.

This new version uses the EC2 query interface instead of os:cmd’ing the amazon api tools.

It has no external dependencies now when deploying the release. Previous needed Java and the API tools.

Finally removed the need for the cert and pk files. It only uses AMAZONACCESSKEYID and AMAZONSECRETACCESSKEY.

Also new is an implementation of V2 signature code for AWS.
Given that V1 is deprecating at the end of the year, that’s a head start.

I did not see it anywhere in erlang yet, so HTH (as they say).

TODO : the secret key tend to come up in the logs. It will be removed in an upcoming release.

Why choose (XMPP versus HTTP or HTTP versus XMPP) ?

Morning reading

With the aftermath of the XMPP Summit, I’ve seen a few posts about XMPP as a potential replacement for HTTP.

HTTP

I love HTTP. That protocol has so many great features, I wouldn’t know where to start.

And it’s only a couple of years that the world (and me) really understood what this HTTP thing was really about. (Before that, it was only inside the brain of Roy Fielding and a few others) – I thank DHH for raising awareness and Richardsson and Ruby for writing that great O’Reilly book.

The bits and pieces one describes as REST (the VERBS, mime-types, caching strategies) build a very clever and stable architecture.

Yet it’s polling based and every 30 minutes my feed reader polls a load of feed, most of them returning 304 Not Modified.

XMPP

XMPP is a decade old, it’s only becoming very popular recently. But it still has that “new frontier” smell I like. Still a lot of work to do. And I hope I can be part of that.

It’s a place where a lot of work has been done, but that protocol is still working to expand itself beyond instant messaging.

Enters PubSub. The solution to polling endlessly Atom feeds. Machines sending events to machines in a distributed, decentralized way. Bandwidth saved.

Yet XMPP has limits :

• message size : Sending more than a few kilobytes of data per stanza can fill up your server queues (especially when you have thousand of messages to route)

• binary transfer : encode your data in base64, split it up in approprietly sized stanzas, send it over. It’s slow as hell (but reliable).

• connected socket : one connects usually on an XMPP server on TCP port 5222. Loose the socket, loose the connection. Hence on iPhone, task switching means loosing the connection. And each time I chat from the iPhone, I have the need to switch between OneTeam XMPP client and my calendar or mail.

There are other nitpicks I have : discovery is cool, but without paging and caching your bandwidth and CPU bills will go through the roof. I started implementing paging in ejabberd’s MUC, SHIM – for caching – is an itch I may scratch one day.

Overcoming those limits

Every time HTTP has a solution for “fixing” XMPP.

The first two limits can be fixed by running a WebDAV server. Upload to the WebDAV server, share the link. That’s a solution everyone can do without XMPP client support. Of course, having a way to do that transparently with client and server support, with signed URLs (à la S3) would greatly improve the process.

For the connected socket problem, there’s BOSH. That’s basically running XMPP over HTTP. With the added bonus of having the server retaining the “connection” for a couple of minutes – that fixes my iPhone problem. Once I relaunch the client in the two minutes window, all the pending messages are delivered.

There’s also a nice side effect : HTTP tools (load balancers, proxys) can be used in front of the server.

XMPP and HTTP are here to stay.

In my opinion XMPP needs more HTTP than HTTP needs XMPP.

I wouldn’t mind if my XMPP avatar was accessed by clients through the HTTP component of the XMPP server (they all have one embedded, that’s a sign, right ?), as opposed to fetching the VCard base64 encoded database stored version of it through XMPP.

I wouldn’t mind if my filetransfers never fail because the binary files are uploaded first on my XMPP server via HTTP, itself notifying the receiving client I am sending the file to that the payload is ready.

As a final note, writing my Atompub-PubSub bridge was quite gratifying, I could leverage MarsEdit to publish on my pubsub nodes.

The French catch-22

The catch-22

• The right to strike is in the Constition.
• When you’re on strike, the law says you are supposed to be at the office. Not working but in the right location.
• Given that public transportation is on strike, it is impossible to go to your office.
• Hence you are not on strike.
• That means that you should go to your office for working.

Right ?

The solution

Take a day off. You’re French, you have many of them.

OK, that whole post is stupid.

Yes.

Also this is cute

“How Ohmicide Changed My Life”

Thank you too mate. Thank you.

SSH to all your EC2 instances automatically

This small Mac OS X ruby script opens a terminal window and connect to each instance you are currently running.

Update : You’ll need the amazon-ec2 gem installed.

Building OTP applications with Rake for EC2

I uploaded a mashup of other’s code this morning to github. Namely charpi’s Rakefile, used to build the Duke of Erl’s ec2nodefinder.

Rakefile

It’s way more than a Rakefile, actually. It’s a whole project organization.

Your OTP applications go into lib/, and you can have as many as you want in there.

Just typing rake will build the release files, doing all the chores (creating app files and rel files … BORING !)

Here are all the defined tasks.

rake clean                     # Remove any temporary products.
rake clobber                   # Remove any generated file.
rake compile                   # Compile all project
rake dialyzer                  # Run dialyzer
rake edoc[name]                # Buid Application documentation
rake edocs                     # Buid all application documentation
rake erlang_applications       # Build application resource file
rake erlang_modules            # Compile Erlang sources
rake erlang_release_files      # Build erlang boot files
rake erlang_releases           # Build release tarball
rake erlang_target_systems[n]  # Build release tarball with erts
rake run[name,node]            # Runs application
rake tests                     # Runs EUnit tests
rake upload                    # Sends release to S3 (presumably for deployment on EC2)

ec2nodefinder

In a nutshell, makes all erlang nodes in a single EC2 security group discover each other automatically.

Your applications on those instances can have processes joining pg2 groups, and you get scalability with very little efforts. However pg2 is quite crude in choosing a process as its only strategy is closest process or random.

ec2nodefinder is described here. Works well and very handy. I however would like to remove the dependency on ec2-describe-instances (which in turn requires a JRE on the deployment server). It’s working as advertised but JVM startup slows down discovery.

AMIs

Check out Kevin Smith’s blog. He is building erlang centric AMI based on Eric Hammond’s Ubuntu AMIs

Next cool things I’d like to do or see

Run OTP releases on instances, with just one command :

ec2-run-instances ami-XXXXX -n  -d "release=/"

It won’t be that easy, because the AMIs will need credentials to access S3 (and EC2 admin commands)

Actually that’s my 10€ question. Where are AWS credentials best stored ?

• have the cert / private key on S3 and pass the S3 AWSACCESS* values as parameters :

ec2-run-instances ami-XXXXX -n  -d "release=/&AWS_ACCESS_KEY=XXXXX&AWS_SECRET_ACCESS_KEY=XXXX++XXXX"

• pass everything in a tar.gz (-f flag of ec2-run-instances)

Not decided yet.

relup, appup … hot code upgrade

No rake magic yet for those. Watch this space or charpi’s !

Code

Get the code on github.

Validating Atom entries as PubSub payloads in ejabberd

Small bit of code to validate Atom when in xmlelement tuples.

Written for the pubsub component of ejabberd, integration to follow shortly.

Not perfect yet, but should still be useful. Patches appreciated.

Find it on github !

Should have full EUnit coverage too.