erlsdb and erls3 use ibrowse
And when you are talking to Amazon Web Services, you’d want to write in parallel as much as possible. You also want to pipeline requests in one single socket, especially while using SSL encryption (even more costly to establish).
Porting my code to ibrowse was quite easy. Though I had to change a bit of the async code. Instead of sending one message once the inets process received the HTTP response, it sends a message upon receiving headers then a slew of messages for each chunk it receives.
Had a few Too Many Open Files errors while loadtesting. As it appears, I had over 500 connections opened to Amazon AWS. Got more sensible defaults and the problem went away.
One caveat : accessing SimpleDB using SSL gives InvalidSignature errors for the time being. Will squash that soon.
Using ibrowse will also unable me to write a client to S3 that will stream files to and from disk.
The ibrowse version are in the ibrowse branch for both projects.