LibreS3: Open Source Amazon S3 Implementation

The first version of LibreS3 project has been announced. LibreS3 is a robust open source implementation of the Amazon S3 service, supporting a subset of the S3 REST API. It’s compatible with Amazon S3 client libraries and built on their basis tools, such as python-boto, s3cmd and DragonDisk. The source codes of LibreS3 are written in С and OCaml languages, the code is allocated under the GPLv2 license (libraries under LGPL). The platform can be used in Linux, OS X and different BSD-systems.

It’s worth noting that the project is being developed by the design team which had left ClamAV project in order to develop their new ideas. You can come across the name of Tomasz Kojm among the developers, he’s the founder of the free antivirus package ClamAV; and three more developers standing at the origins of the project. For the data storage management of the group of servers the dispersed cluster repository Skylable Sx is applied. The initial release of Skylable Sx is presented at the same time with LibreS3. Skylable Sx is also a free project and takes over the tasks on data replication between the nodes and deduplication of the identical data at different cluster nodes. Skylable Sx allows to bind into the entire virtual store the footprint from several servers. In order to provide fault-tolerance and information integrity the date sets are replicated on several different servers. The amount of copies that will be kept on different servers is determined by the administrator during the system configuration. In order to save the data encryption technique on the client’s side is applied, wherein information comes to system servers in encrypted form, which excludes the data capture in case if special services or attackers get access to the storage server. Also, sections encryption on the servers can be optionally used, but in this case efficiency from deduplication using is weakening. Data exchange between the client and the server and between the nodes is encrypted with HTTPS. In order to spare the disc space synchronous data compression is used. Skylable Sx consists of the following components: - sxhttpd – demon that serves requests for HTTP, which is used for both clients calls and communication of the storage nodes. - sx.fcgi – operating with FastCGI process using, processing requests and allocating them between child handling processes: - Manager of operations execution – plans the data blocks transfer and operating management commands; - Manager of data blocks transfer – transfers the data between the storage nodes and gives the data to clients; - Garbage collector – carries out searching and peeling data blocks, which lost their significance or aren’t linked with files.