hdfs-site
Contents
| name | value | description |
|---|---|---|
| dfs.blocksize | 134217728 | The default block size for new files, in bytes. You can use the following suffix (case insensitive): k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.), Or provide complete size in bytes (such as 134217728 for 128 MB). |
| dfs.client.read.shortcircuit | FALSE | This configuration parameter turns on short-circuit local reads. |
| dfs.cluster.administrators | ACL for the admins, this configuration is used to control who can access the default servlets in the namenode, etc. The value should be a comma separated list of users and groups. The user list comes first and is separated by a space followed by the group list, e.g. “user1,user2 group1,group2”. Both users and groups are optional, so “user1”, “ group1”, “”, “user1 group1”, “user1,user2 group1,group2” are all valid (note the leading space in “ group1”). ‘‘ grants access to all users and groups, e.g. ‘‘, ‘* ‘ and ‘ *’ are all valid. | |
| dfs.datanode.data.dir | file://${hadoop.tmp.dir}/dfs/data | Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. The directories should be tagged with corresponding storage types ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for HDFS storage policies. The default storage type will be DISK if the directory does not have a storage type tagged explicitly. Directories that do not exist will be created if local filesystem permission allows. |
| dfs.datanode.data.dir.perm | 700 | Permissions for the directories on on the local filesystem where the DFS data node store its blocks. The permissions can either be octal or symbolic. |
| dfs.datanode.du.reserved | 0 | Reserved space in bytes per volume. Always leave this much space free for non dfs use. Specific storage type based reservation is also supported. The property can be followed with corresponding storage types ([ssd]/[disk]/[archive]/[ram_disk]) for cluster with heterogeneous storage. For example, reserved space for RAM_DISK storage can be configured using property ‘dfs.datanode.du.reserved.ram_disk’. If specific storage type reservation is not configured then dfs.datanode.du.reserved will be used. |
| dfs.datanode.du.reserved.pct | 0 | Reserved space in percentage. Read dfs.datanode.du.reserved.calculator to see when this takes effect. The actual number of bytes reserved will be calculated by using the total capacity of the data directory in question. Specific storage type based reservation is also supported. The property can be followed with corresponding storage types ([ssd]/[disk]/[archive]/[ram_disk]) for cluster with heterogeneous storage. For example, reserved percentage space for RAM_DISK storage can be configured using property ‘dfs.datanode.du.reserved.pct.ram_disk’. If specific storage type reservation is not configured then dfs.datanode.du.reserved.pct will be used. |
| dfs.datanode.hostname | Optional. The hostname for the Datanode containing this configuration file. Will be different for each machine. Defaults to current hostname. | |
| dfs.ha.automatic-failover.enabled | FALSE | Whether automatic failover is enabled. See the HDFS High Availability documentation for details on automatic HA configuration. |
| dfs.ha.fencing.methods | A list of scripts or Java classes which will be used to fence the Active NameNode during a failover. See the HDFS High Availability documentation for details on automatic HA configuration. | |
| dfs.ha.namenode.id | The ID of this namenode. If the namenode ID is not configured it is determined automatically by matching the local node’s address with the configured address. | |
| dfs.ha.namenodes.EXAMPLENAMESERVICE | The prefix for a given nameservice, contains a comma-separated list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE). Unique identifiers for each NameNode in the nameservice, delimited by commas. This will be used by DataNodes to determine all the NameNodes in the cluster. For example, if you used “mycluster” as the nameservice ID previously, and you wanted to use “nn1” and “nn2” as the individual IDs of the NameNodes, you would configure a property dfs.ha.namenodes.mycluster, and its value “nn1,nn2”. | |
| dfs.hosts | Names a file that contains a list of hosts that are permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, all hosts are permitted. | |
| dfs.hosts.exclude | Names a file that contains a list of hosts that are not permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, no hosts are excluded. | |
| dfs.internal.nameservices | Comma-separated list of nameservices that belong to this cluster. Datanode will report to all the nameservices in this list. By default this is set to the value of dfs.nameservices. | |
| dfs.ls.limit | 1000 | Limit the number of files printed by ls. If less or equal to zero, at most DFS_LIST_LIMIT_DEFAULT (= 1000) will be printed. |
| dfs.namenode.acls.enabled | FALSE | Set to true to enable support for HDFS ACLs (Access Control Lists). By default, ACLs are disabled. When ACLs are disabled, the NameNode rejects all RPCs related to setting or getting ACLs. |
| dfs.namenode.name.dir | file://${hadoop.tmp.dir}/dfs/name | Determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. |
| dfs.namenode.resource.du.reserved | 104857600 | The amount of space to reserve/require for a NameNode storage directory in bytes. The default is 100MB. |
| dfs.nameservice.id | The ID of this nameservice. If the nameservice ID is not configured or more than one nameservice is configured for dfs.nameservices it is determined automatically by matching the local node’s address with the configured address. | |
| dfs.nameservices | Comma-separated list of nameservices. | |
| dfs.net.topology.impl | org.apache.hadoop.hdfs.net.DFSNetworkTopology | The implementation class of NetworkTopology used in HDFS. By default, the class org.apache.hadoop.hdfs.net.DFSNetworkTopology is specified and used in block placement. This property only works when dfs.use.dfs.network.topology is true. |
| dfs.permissions.enabled | TRUE | If “true”, enable permission checking in HDFS. If “false”, permission checking is turned off, but all other behavior is unchanged. Switching from one parameter value to the other does not change the mode, owner or group of files or directories. |
| dfs.permissions.superusergroup | supergroup | The name of the group of super-users. The value should be a single group name. |
| dfs.replication | 3 | Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time. |
| dfs.replication.max | 512 | Maximal block replication. |
| dfs.support.append | TRUE | Enables append support on the NameNode. |
| dfs.use.dfs.network.topology | TRUE | Enables DFSNetworkTopology to choose nodes for placing replicas. When enabled, NetworkTopology will be instantiated as class defined in property dfs.net.topology.impl, otherwise NetworkTopology will be instantiated as class defined in property net.topology.impl. |
| dfs.web.authentication.filter | org.apache.hadoop.hdfs.web.AuthFilter | Authentication filter class used for WebHDFS. |
| dfs.web.authentication.simple.anonymous.allowed | If true, allow anonymous user to access WebHDFS. Set to false to disable anonymous authentication. | |
| dfs.webhdfs.enabled | TRUE | Enable WebHDFS (REST API) in Namenodes and Datanodes. |
