CompoundFileWriter.javaAPI DocApache Lucene 1.4.37844Tue Mar 30 00:48:02 BST 2004org.apache.lucene.index


public final class CompoundFileWriter extends Object
Combines multiple files into a single compound file. The file format:
  • VInt fileCount
  • {Directory} fileCount entries with the following structure:
    • long dataOffset
    • UTFString extension
  • {File Data} fileCount entries with the raw data of the corresponding file
The fileCount integer indicates how many files are contained in this compound file. The {directory} that follows has that many entries. Each directory entry contains an encoding identifier, an long pointer to the start of this file's data section, and a UTF String with that file's extension.
Dmitry Serebrennikov
$Id:,v 1.3 2004/03/29 22:48:02 cutting Exp $

Fields Summary
private Directory
private String
private HashSet
private LinkedList
private boolean
Constructors Summary
public CompoundFileWriter(Directory dir, String name)
Create the compound stream in the specified file. The file name is the entire name (no extensions are added).

        if (dir == null)
            throw new IllegalArgumentException("Missing directory");
        if (name == null)
            throw new IllegalArgumentException("Missing name");

        directory = dir;
        fileName = name;
        ids = new HashSet();
        entries = new LinkedList();
Methods Summary
public voidaddFile(java.lang.String file)
Add a source stream. If sourceDir is null, it is set to the same value as the directory where this compound stream exists. The id is the string by which the sub-stream will be know in the compound stream. The caller must ensure that the ID is unique. If the id is null, it is set to the name of the source file.

        if (merged)
            throw new IllegalStateException(
                "Can't add extensions after merge has been called");

        if (file == null)
            throw new IllegalArgumentException(
                "Missing source file");

        if (! ids.add(file))
            throw new IllegalArgumentException(
                "File " + file + " already added");

        FileEntry entry = new FileEntry();
        entry.file = file;
public voidclose()
Merge files with the extensions added up to now. All files with these extensions are combined sequentially into the compound stream. After successful merge, the source files are deleted.

        if (merged)
            throw new IllegalStateException(
                "Merge already performed");

        if (entries.isEmpty())
            throw new IllegalStateException(
                "No entries to merge have been defined");

        merged = true;

        // open the compound stream
        OutputStream os = null;
        try {
            os = directory.createFile(fileName);

            // Write the number of entries

            // Write the directory with all offsets at 0.
            // Remember the positions of directory entries so that we can
            // adjust the offsets later
            Iterator it = entries.iterator();
            while(it.hasNext()) {
                FileEntry fe = (FileEntry);
                fe.directoryOffset = os.getFilePointer();
                os.writeLong(0);    // for now

            // Open the files and copy their data into the stream.
            // Remeber the locations of each file's data section.
            byte buffer[] = new byte[1024];
            it = entries.iterator();
            while(it.hasNext()) {
                FileEntry fe = (FileEntry);
                fe.dataOffset = os.getFilePointer();
                copyFile(fe, os, buffer);

            // Write the data offsets into the directory of the compound stream
            it = entries.iterator();
            while(it.hasNext()) {
                FileEntry fe = (FileEntry);

            // Close the output stream. Set the os to null before trying to
            // close so that if an exception occurs during the close, the
            // finally clause below will not attempt to close the stream
            // the second time.
            OutputStream tmp = os;
            os = null;

        } finally {
            if (os != null) try { os.close(); } catch (IOException e) { }
private voidcopyFile(org.apache.lucene.index.CompoundFileWriter$FileEntry source, os, byte[] buffer)
Copy the contents of the file with specified extension into the provided output stream. Use the provided buffer for moving data to reduce memory allocation.

        InputStream is = null;
        try {
            long startPtr = os.getFilePointer();

            is = directory.openFile(source.file);
            long length = is.length();
            long remainder = length;
            int chunk = buffer.length;

            while(remainder > 0) {
                int len = (int) Math.min(chunk, remainder);
                is.readBytes(buffer, 0, len);
                os.writeBytes(buffer, len);
                remainder -= len;

            // Verify that remainder is 0
            if (remainder != 0)
                throw new IOException(
                    "Non-zero remainder length after copying: " + remainder
                    + " (id: " + source.file + ", length: " + length
                    + ", buffer size: " + chunk + ")");

            // Verify that the output length diff is equal to original file
            long endPtr = os.getFilePointer();
            long diff = endPtr - startPtr;
            if (diff != length)
                throw new IOException(
                    "Difference in the output file offsets " + diff
                    + " does not match the original file length " + length);

        } finally {
            if (is != null) is.close();
Returns the directory of the compound file.

        return directory;
public java.lang.StringgetName()
Returns the name of the compound file.

        return fileName;