Transparent compression for ext2 filesystem =========================================== What this document is. ---------------------- This document is intended for explaining how e2compress has been implented/ported in kernel 2.4. It also give a status of current work. You need to have e2compress knowledge (i.e. to know how e2compress works, from a general point of view) What this document is not. -------------------------- This document is not a full explaination of how e2compress work. For this, there are other documents such as fs/ext2/Readme.e2compr file for the technical point of view and user manual can be found at . This site is also a place were you will find many information about e2compress development for kernel 2.2, tools, manuals and so on. Introduction ============ This is a first adaptation of e2compress for kernel 2.4. The work has been done by Alcatel (Alcatel Business Systems - R&D) at Illkirch. It has been started from the latest patch provided by Peter Moulder for kernel 2.2, i.e. e2compr-0.4.39-patch-2.2.18. It is full compatible with previous version. Here after you will first find some explainations about the choices mades for the development, and then the status of current work from functionnal point of view. Development =========== As for previous patches, most interesting happens when reading in ext2_readpage and when writing in ext2_writepage and ext2_file_write. In fact, in 2.2 kernel, compression occures on cluster of blocks. So when reading or writing a part of a file, we first have to compute the cluster on which I/O occures, then we have to get every buffers of the cluster, uncompress the data if needed, then reading/writing happens "as for normal files". In 2.4 kernels, I/O occures through page cache: i.e. when reading/writing to a part of the file, first the corresponding page is get, we then get the needed buffers, which point to the page; this means that for keeping same work as for 2.2, we have to use the notion of cluster of page. For getting every buffers of a cluster, we first get every pages of the cluster, then get buffers of every pages... So, things happens as follow: ext2_readpage ------------- If data corresponding to the page are in a compressed cluster, this functions perfoms more works: instead of reading one page, it reads the whole "cluster of pages". In fact, anyway, we have to read all compressed buffer. Once we have got all buffers of the cluster, uncompressed (at least a part of) the data, and located the part of the uncompressed data which correspond to the requested page, there is not any more lot of work for also reading (i.e. doing some memcpy) other pages belonging to this cluster. So, the first reading of the first page of the cluster is quite longer, but then, every pages of the cluster are uptodate in the cache. ext2_writepage -------------- An overhead has been added for pages belonging to a compressed cluster. In fact, if cluster is still compressed on the disk, we can't directly write the page (which contains uncompressed data) in the middle of a compressed cluster. So, we first have to uncompress the whole cluster on the disk, then we can write the new data of the dirty page(s). ext2_file_write --------------- This replaces `generic_file_write' when e2compress option is activated. It is a copy of `generic_file_write'. The main difference is that instead of looping page by page in `generic_file_write', we loops on cluster of page. In each loop: * we compute the cluster on which beginning of data (to be written) belongs to. * then, we get all pages of the cluster. * If cluster is a compressed one, we read all pages, and uncompress it. Otherwise, we perfoms a `prepare_write' (as in generic_file_write). * We copy the data on each page from user space, * Call `commit_write' on dirty pages. * When reaching end of cluster, we compress it. (As in 2.2) Note: Another implentation could have been to keep generic_file_write, and add an overhead to `ext2_prepare_write' and `ext2_commit_write'; on the first access to a page of a compressed cluster, whole cluster will be uncompressed (i.e. all pages of the cluster will be read and uncompressed in `ext2_prepare_write') and when commiting the last page of the cluster, compression occures... ext2_open_file -------------- In 2.4.16 kernel, this function has been added for treating the case of files opened for "direct IO". Direct IO is not supported on compressed file. So opening a file by this way is forbidden. Other places in ext2 -------------------- Other changes occures as in 2.2 for managing the compression flags of files and specific `COMPRESSED_BLK_ADDR' address for compressed blocks. So please, refer to existing documentation for 2.2 about this topic. Status ====== Today (middle of december 2001), e2compress on kernel 2.4.16 has been tested on i386 architecture, is used with success by tens of people in the department from some weeks. It is full fonctionnal on ix86, full compatible with 2.2 version of e2compress. It should work on other architecture, but has NOT been tested. Please, note the following: * No performance tests have been done. * I don't proclaim that code is optimized (and it is probably not, but I hope that "gurus" will not find it too bad) So, I think I can say that there is no known "big" bug or "blocking" bug. Some strange things has been observed in very limit case, i.e. when memory is overloaded. As usual, this e2compress comes without warranty, use it at your won risk, etc...