org.apache.hadoop.fs
public class: LocalDirAllocator [javadoc |
source]
java.lang.Object
org.apache.hadoop.fs.LocalDirAllocator
An implementation of a round-robin scheme for disk allocation for creating
files. The way it works is that it is kept track what disk was last
allocated for a file write. For the current request, the next disk from
the set of disks would be allocated if the free space on the disk is
sufficient enough to accomodate the file that is being considered for
creation. If the space requirements cannot be met, the next disk in order
would be tried and so on till a disk is found with sufficient capacity.
Once a disk with sufficient space is identified, a check is done to make
sure that the disk is writable. Also, there is an API provided that doesn't
take the space requirements into consideration but just checks whether the
disk under consideration is writable (this should be used for cases where
the file size is not known apriori). An API is provided to read a path that
was created earlier. That API works by doing a scan of all the disks for the
input pathname.
This implementation also provides the functionality of having multiple
allocators per JVM (one for each unique functionality or context, like
mapred, dfs-client, etc.). It ensures that there is only one instance of
an allocator per context per JVM.
Note:
1. The contexts referred above are actually the configuration items defined
in the Configuration class like "mapred.local.dir" (for which we want to
control the dir allocations). The context-strings are exactly those
configuration items.
2. This implementation does not take into consideration cases where
a disk becomes read-only or goes out of space while a file is being written
to (disks are shared between multiple processes, and so the latter situation
is probable).
3. In the class implementation, "Disk" is referred to as "Dir", which
actually points to the configured directory on the Disk which will be the
parent for all file write/read allocations.
| Constructor: |
|---|
 public LocalDirAllocator(String contextCfgItemName) {
this.contextCfgItemName = contextCfgItemName;
}
Create an allocator object Parameters:
contextCfgItemName -
|
| Method from org.apache.hadoop.fs.LocalDirAllocator Detail: |
|---|
public File createTmpFileForWrite(String pathStr,
long size,
Configuration conf) throws IOException {
AllocatorPerContext context = obtainContext(contextCfgItemName);
return context.createTmpFileForWrite(pathStr, size, conf);
} Creates a temporary file in the local FS. Pass size as -1 if not known
apriori. We round-robin over the set of disks (via the configured dirs)
and select the first complete path which has enough space. A file is
created on this directory. The file is guaranteed to go away when the
JVM exits. |
public Path getLocalPathForWrite(String pathStr,
Configuration conf) throws IOException {
return getLocalPathForWrite(pathStr, -1, conf);
} Get a path from the local FS. This method should be used if the size of
the file is not known apriori. We go round-robin over the set of disks
(via the configured dirs) and return the first complete path where
we could create the parent directory of the passed path. |
public Path getLocalPathForWrite(String pathStr,
long size,
Configuration conf) throws IOException {
AllocatorPerContext context = obtainContext(contextCfgItemName);
return context.getLocalPathForWrite(pathStr, size, conf);
} Get a path from the local FS. Pass size as -1 if not known apriori. We
round-robin over the set of disks (via the configured dirs) and return
the first complete path which has enough space |
public Path getLocalPathToRead(String pathStr,
Configuration conf) throws IOException {
AllocatorPerContext context = obtainContext(contextCfgItemName);
return context.getLocalPathToRead(pathStr, conf);
} Get a path from the local FS for reading. We search through all the
configured dirs for the file's existence and return the complete
path to the file when we find one |
public static boolean isContextValid(String contextCfgItemName) {
synchronized (contexts) {
return contexts.containsKey(contextCfgItemName);
}
} Method to check whether a context is valid |
Save This Page