erlsdb and erls3 use ibrowse

I had some issues with inets under heavy load with erlsdb and erls3.

And when you are talking to Amazon Web Services, you’d want to write in parallel as much as possible. You also want to pipeline requests in one single socket, especially while using SSL encryption (even more costly to establish).

ibrowse seemed very interesting, especially since the CouchDB project started using it !

Got it out of jungerl, which is always a bit of a pain. You can also find it on github, I figured later.

Porting my code to ibrowse was quite easy. Though I had to change a bit of the async code. Instead of sending one message once the inets process received the HTTP response, it sends a message upon receiving headers then a slew of messages for each chunk it receives.

Had a few Too Many Open Files errors while loadtesting. As it appears, I had over 500 connections opened to Amazon AWS. Got more sensible defaults and the problem went away.

Configuration is by host, that forced me to change the naming of the S3 buckets from http://bucket.s3.amazonaws.com/ to http://s3.amazonaws.com/bucket/

One caveat : accessing SimpleDB using SSL gives InvalidSignature errors for the time being. Will squash that soon.

Using ibrowse will also unable me to write a client to S3 that will stream files to and from disk.

The ibrowse version are in the ibrowse branch for both projects.

%d bloggers like this: