Programming against Crail

The best way to program against Crail is to use Maven. Make sure you have the Crail dependency specified in your application pom.xml file with the latest Crail version (e.g. 1.1-incubating):

<dependency>
  <groupId>org.apache.crail</groupId>
  <artifactId>crail-client</artifactId>
  <version>X.Y</version>
</dependency>

Then, create a Crail client as follows:

CrailConfiguration conf = new CrailConfiguration();
CrailStore store = CrailStore.newInstance(conf);

Make sure the $CRAIL_HOME/conf directory is part of the classpath.

Crail supports different file types. The simplest way to create a file in Crail is as follows:

CrailFile file = store.create(filename, CrailNodeType.DATAFILE, CrailStorageClass.DEFAULT, CrailLocationClass.DEFAULT).get().syncDir();

Aside from the actual filename, the create() call takes as input the storage and location classes which are preferences for the storage tier and physical location that this file should be created in. Crail tries to satisfy these preferences later when the file is written. In the example we do not request any particular storage or location affinity.

This create() command is non-blocking, calling get() on the returning future object awaits the completion of the call. At that time, the file has been created, but its directory entry may not be visible. Therefore, the file may not yet show up in a file enumeration of the given parent directory. Calling syncDir() waits to for the directory entry to be completed. Both the get() and the syncDir() operation can be deffered to a later time at which they may become non-blocking operations.

Once the file is created, a file stream can be obtained for writing:

CrailBufferedOutputStream outstream = file.getBufferedOutputStream(1024);

Here, we create a buffered stream so that we can pass heap byte arrays as well. We could also create a non-buffered stream using

CrailOutputStream outstream = file.getDirectOutputStream(1024);

In both cases, we pass a write hint (1024 in the example) that indicates to Crail how much data we are intending to write. This allows Crail to optimize metadatanode lookups. Crail never prefetches data, but it may fetch the metadata of the very next operation concurrently with the current data operation if the write hint allows to do so.

Once the stream has been obtained, there exist various ways to write a file. The code snippet below shows the use of the asynchronous interface:

CrailBuffer dataBuf = fs.allocateBuffer();
Future<DataResult> future = outputStream.write(dataBuf);
...
future.get();

Reading files works very similar to writing. There exist various examples in org.apache.crail.tools.CrailBenchmark.