Apache Hadoop

wikipedia:Hadoop (2006)

• HDFS
• MapReduce
• brew install hadoop

hadoop fs

There are currently several monitoring platforms to track HDFS performance, including Hortonworks, Cloudera, and Datadog.

Example

hadoop
Usage: hadoop [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
 or    hadoop [OPTIONS] CLASSNAME [CLASSNAME OPTIONS]
  where CLASSNAME is a user-provided Java class

  OPTIONS is none or any of:

--config dir                     Hadoop config directory
--debug                          turn on shell script debug mode
--help                           usage information
buildpaths                       attempt to add class files from
                                 build tree
hostnames list[,of,host,names]   hosts to use in worker mode
hosts filename                   list of hosts to use in worker
                                 mode
loglevel level                   set the log4j level for this
                                 command
workers                          turn on worker mode

  SUBCOMMAND is one of:


    Admin Commands:

daemonlog     get/set the log level for each daemon

    Client Commands:

archive       create a Hadoop archive
checknative   check native Hadoop and compression libraries
              availability
classpath     prints the class path needed to get the Hadoop jar
              and the required libraries
conftest      validate configuration XML files
credential    interact with credential providers
distch        distributed metadata changer
distcp        copy file or directories recursively
dtutil        operations related to delegation tokens
envvars       display computed Hadoop environment variables
fedbalance    balance data between sub-clusters
fs            run a generic filesystem user client
gridmix       submit a mix of synthetic job, modeling a profiled
              from production load
jar <jar>     run a jar file. NOTE: please use "yarn jar" to
              launch YARN applications, not this command.
jnipath       prints the java.library.path
kdiag         Diagnose Kerberos Problems
kerbname      show auth_to_local principal conversion
key           manage keys via the KeyProvider
rbfbalance    move directories and files across router-based
              federation namespaces
rumenfolder   scale a rumen input trace
rumentrace    convert logs into a rumen trace
s3guard       S3 Commands
version       print the version

    Daemon Commands:

kms           run KMS, the Key Management Server
registrydns   run the registry DNS server

SUBCOMMAND may print help when invoked w/o parameters or with -h.

Related terms

• Apache Hive (2010)
• Apache Zookeeper

See also

• Apache Hadoop, HDFS, MapReduce
• Cloudera, Hadoop, Cloudera Data Platform (CDP)
• Apache: Apache Mesos, Apache Cassandra, Hadoop YARN, Apache CloudStack, Apache Hadoop, Apache Maven, Apache Ivy, Apache Ant, Apache Ignite, Apache Flume, Apache HTTP server, Apache Archiva, Apache Karaf, Apache Flink, Apache Beam, Apache Solr, Apache JMeter, Apache Groovy, Apache NiFi, RewriteEngine, Apache CouchDB, Apache Parquet, Apache Chainsaw, Apache Spark, Apache Lucene, MXNet