hadoop的部署以及應用

阿新 • • 發佈：2017-07-09

hadoop

1.基礎環境

[[email protected] ~]$ cat  /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core) 
[[email protected] ~]$ 
[[email protected] ~]$ getenforce 
Disabled
[[email protected] ~]$ systemctl  status  firewalld 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
[[email protected]

/* */ ~]$

2.IP以及對應節點

IP	主機名	hadoop node	hadoop 進程名稱
192.168.56.100	master	master	namenode,jobtracker
192.168.56.101	slave1	slave	datanode,tasktracker
192.168.56.102	slave2	slave	datanode,tasktracker
192.168.56.103	slave3	slave	datanode,tasktracker

[[email protected] ~]# cat  /etc/hosts
192.168.56.100  Master
192.168.56.101  slave1
192.168.56.102  slave2
192.168.56.103  slave3
[[email protected]

/* */ ~]#

3.增加hadoop用戶，所有節點

useradd  hadoop
echo hadoop|passwd  --stdin  hadoop

4.jdk

[[email protected] application]# ll
total 4
lrwxrwxrwx 1 root root   24 Jul 10 01:35 jdk -> /application/jdk1.8.0_60
drwxr-xr-x 8 root root 4096 Aug  5  2015 jdk1.8.0_60
[[email protected] application]# pwd
/application
[[email protected]

/* */ application]# [[email protected] ~]# java -version java version "1.8.0_60" Java(TM) SE Runtime Environment (build 1.8.0_60-b27) Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode) [[email protected] ~]#

5.master(192.168.56.100)上的hadoop用戶可以ssh所有slave節點的hadoop用戶下

6.設置hadoop安裝路徑以及環境變量（所有節點）

su  -  hadoop
tar xf  hadoop-2.7.0tar.gz
/home/hadoop/hadoop-2.7.0
vi /etc/profile  添加hadoop環境變量
export HADOOP_HOME=/home/hadoop/hadoop-2.7.0
export PATH=$PATH:$HADOOP_HOME/bin
source /etc/profile

7.修改hadoop的環境的Java環境變量

/home/hadoop/hadoop-2.7.0/etc/hadoop
vi hadoop-env.sh 添加
###JAVA_HOME
export JAVA_HOME=/application/jdk/

8.修改hadoop的配置文件

cd  /home/hadoop/hadoop-2.7.0/etc/hadoop
1.##############################
[[email protected] hadoop]$ cat core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
 <name>fs.default.name</name>
  <value>hdfs://master:9000</value>
</property>
<property>
  <name>hadoop.tmp.dir</name>
 <value>/home/hadoop/tmp</value>
</property>
</configuration>
[[email protected] hadoop]$ 
2.###################################(默認不存在 拷貝個模板即可)
[[email protected] hadoop]$ cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>master:9001</value>
</property>
<property>
  <name>mapred.local.dir</name>
 <value>/home/hadoop/tmp</value>
</property>
</configuration>
[[email protected] hadoop]$ 
3.#########################################
[[email protected] hadoop]$ cat  hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/home/hadoop/name1,/home/hadoop/name2,/home/hadoop/name3</value> 
<description>  </description>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/hadoop/data1,/home/hadoop/data2,/home/hadoop/data3</value>
<description> </description>
</property>
<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>
</configuration>
[[email protected] hadoop]$ 
[[email protected] hadoop]$ cat masters 
master
[[email protected] hadoop]$ cat slaves 
slave1
slave2
slave3
[[email protected] hadoop]$

9.分發到slave節點

scp   -r  /home/hadoop/hadoop-2.7.0  slave1:/home/hadoop/
scp   -r  /home/hadoop/hadoop-2.7.0  slave2:/home/hadoop/
scp   -r  /home/hadoop/hadoop-2.7.0  slave3:/home/hadoop/

10.master 節點測試

/home/hadoop/name1 /home/hadoop/name2  /home/hadoop/name3  這三個目錄不要創建，如果創建會提示
重新reload

cd  /home/hadoop/hadoop-2.7.0
[[email protected] hadoop-2.7.0]$ ./bin/hadoop namenode -format
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

17/07/10 02:57:34 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = Master/192.168.56.100
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.7.0
STARTUP_MSG:   classpath = /home/hadoop/hadoop-2.7.0/etc/hadoop:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-auth-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/asm-3.2.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-annotations-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/junit-4.11.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jettison-1.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-6.1.26.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/avro-1.7.4.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/activation-1.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xz-1.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/gson-2.2.4.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-nfs-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-tests.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-tests.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/activation-1.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/xz-1.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-registry-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-common-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-common-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-api-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-client-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.0-tests.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.0.jar:/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.0.jar:/home/hadoop/hadoop-2.7.0/contrib/capacity-scheduler/*.jar:/home/hadoop/hadoop-2.7.0/contrib/capacity-scheduler/*.jar
STARTUP_MSG:   build = Unknown -r Unknown; compiled by ‘root‘ on 2015-05-27T13:56Z
STARTUP_MSG:   java = 1.8.0_60
************************************************************/
17/07/10 02:57:34 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
17/07/10 02:57:34 INFO namenode.NameNode: createNameNode [-format]
17/07/10 02:57:35 WARN common.Util: Path /home/hadoop/name1 should be specified as a URI in configuration files. Please update hdfs configuration.
17/07/10 02:57:35 WARN common.Util: Path /home/hadoop/name2 should be specified as a URI in configuration files. Please update hdfs configuration.
17/07/10 02:57:35 WARN common.Util: Path /home/hadoop/name3 should be specified as a URI in configuration files. Please update hdfs configuration.
17/07/10 02:57:35 WARN common.Util: Path /home/hadoop/name1 should be specified as a URI in configuration files. Please update hdfs configuration.
17/07/10 02:57:35 WARN common.Util: Path /home/hadoop/name2 should be specified as a URI in configuration files. Please update hdfs configuration.
17/07/10 02:57:35 WARN common.Util: Path /home/hadoop/name3 should be specified as a URI in configuration files. Please update hdfs configuration.
Formatting using clusterid: CID-77e0896d-bda2-49f1-8127-c5343f1c52c9
17/07/10 02:57:35 INFO namenode.FSNamesystem: No KeyProvider found.
17/07/10 02:57:35 INFO namenode.FSNamesystem: fsLock is fair:true
17/07/10 02:57:35 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
17/07/10 02:57:35 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
17/07/10 02:57:35 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
17/07/10 02:57:36 INFO blockmanagement.BlockManager: The block deletion will start around 2017 Jul 10 02:57:36
17/07/10 02:57:36 INFO util.GSet: Computing capacity for map BlocksMap
17/07/10 02:57:36 INFO util.GSet: VM type       = 64-bit
17/07/10 02:57:36 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
17/07/10 02:57:36 INFO util.GSet: capacity      = 2^21 = 2097152 entries
17/07/10 02:57:36 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
17/07/10 02:57:36 INFO blockmanagement.BlockManager: defaultReplication         = 3
17/07/10 02:57:36 INFO blockmanagement.BlockManager: maxReplication             = 512
17/07/10 02:57:36 INFO blockmanagement.BlockManager: minReplication             = 1
17/07/10 02:57:36 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
17/07/10 02:57:36 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false
17/07/10 02:57:36 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
17/07/10 02:57:36 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
17/07/10 02:57:36 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
17/07/10 02:57:36 INFO namenode.FSNamesystem: fsOwner             = hadoop (auth:SIMPLE)
17/07/10 02:57:36 INFO namenode.FSNamesystem: supergroup          = supergroup
17/07/10 02:57:36 INFO namenode.FSNamesystem: isPermissionEnabled = true
17/07/10 02:57:36 INFO namenode.FSNamesystem: HA Enabled: false
17/07/10 02:57:36 INFO namenode.FSNamesystem: Append Enabled: true
17/07/10 02:57:36 INFO util.GSet: Computing capacity for map INodeMap
17/07/10 02:57:36 INFO util.GSet: VM type       = 64-bit
17/07/10 02:57:36 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
17/07/10 02:57:36 INFO util.GSet: capacity      = 2^20 = 1048576 entries
17/07/10 02:57:36 INFO namenode.FSDirectory: ACLs enabled? false
17/07/10 02:57:36 INFO namenode.FSDirectory: XAttrs enabled? true
17/07/10 02:57:36 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
17/07/10 02:57:36 INFO namenode.NameNode: Caching file names occuring more than 10 times
17/07/10 02:57:36 INFO util.GSet: Computing capacity for map cachedBlocks
17/07/10 02:57:36 INFO util.GSet: VM type       = 64-bit
17/07/10 02:57:36 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
17/07/10 02:57:36 INFO util.GSet: capacity      = 2^18 = 262144 entries
17/07/10 02:57:36 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
17/07/10 02:57:36 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
17/07/10 02:57:36 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
17/07/10 02:57:36 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
17/07/10 02:57:36 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
17/07/10 02:57:36 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
17/07/10 02:57:36 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
17/07/10 02:57:36 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
17/07/10 02:57:36 INFO util.GSet: Computing capacity for map NameNodeRetryCache
17/07/10 02:57:36 INFO util.GSet: VM type       = 64-bit
17/07/10 02:57:36 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
17/07/10 02:57:36 INFO util.GSet: capacity      = 2^15 = 32768 entries
17/07/10 02:57:36 INFO namenode.FSImage: Allocated new BlockPoolId: BP-467031090-192.168.56.100-1499626656612
17/07/10 02:57:36 INFO common.Storage: Storage directory /home/hadoop/name1 has been successfully formatted.
17/07/10 02:57:36 INFO common.Storage: Storage directory /home/hadoop/name2 has been successfully formatted.
17/07/10 02:57:36 INFO common.Storage: Storage directory /home/hadoop/name3 has been successfully formatted.
17/07/10 02:57:36 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
17/07/10 02:57:36 INFO util.ExitUtil: Exiting with status 0
17/07/10 02:57:37 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Master/192.168.56.100
************************************************************/
[[email protected] hadoop-2.7.0]$

11.啟動服務

[[email protected] sbin]$ pwd
/home/hadoop/hadoop-2.7.0/sbin
[[email protected] sbin]$ 
[[email protected] sbin]$ ./start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /home/hadoop/hadoop-2.7.0/logs/hadoop-hadoop-namenode-master.out
slave3: starting datanode, logging to /home/hadoop/hadoop-2.7.0/logs/hadoop-hadoop-datanode-slave3.out
slave2: starting datanode, logging to /home/hadoop/hadoop-2.7.0/logs/hadoop-hadoop-datanode-slave2.out
slave1: starting datanode, logging to /home/hadoop/hadoop-2.7.0/logs/hadoop-hadoop-datanode-slave1.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop-2.7.0/logs/hadoop-hadoop-secondarynamenode-master.out
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/hadoop-2.7.0/logs/yarn-hadoop-resourcemanager-master.out
slave3: starting nodemanager, logging to /home/hadoop/hadoop-2.7.0/logs/yarn-hadoop-nodemanager-slave3.out
slave2: starting nodemanager, logging to /home/hadoop/hadoop-2.7.0/logs/yarn-hadoop-nodemanager-slave2.out
slave1: starting nodemanager, logging to /home/hadoop/hadoop-2.7.0/logs/yarn-hadoop-nodemanager-slave1.out

[[email protected] sbin]$ netstat  -lntup 
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 192.168.56.100:9000     0.0.0.0:*               LISTEN      4405/java           
tcp        0      0 0.0.0.0:50090           0.0.0.0:*               LISTEN      4606/java           
tcp        0      0 0.0.0.0:50070           0.0.0.0:*               LISTEN      4405/java           
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::22                   :::*                    LISTEN      -                   
tcp6       0      0 :::8088                 :::*                    LISTEN      4757/java           
tcp6       0      0 ::1:25                  :::*                    LISTEN      -                   
tcp6       0      0 :::8030                 :::*                    LISTEN      4757/java           
tcp6       0      0 :::8031                 :::*                    LISTEN      4757/java           
tcp6       0      0 :::8032                 :::*                    LISTEN      4757/java           
tcp6       0      0 :::8033                 :::*                    LISTEN      4757/java           
[[email protected] sbin]$

http://192.168.56.100:50070/dfshealth.html#tab-overview

http://192.168.56.103:8042/node/allApplications

http://192.168.56.100:50090/status.html

本文出自 “磚家博客” 博客，請務必保留此出處http://wsxxsl.blog.51cto.com/9085838/1945709

hadoop的部署以及應用

hadoop1.基礎環境[[email protected]/* */ ~]$ cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [[email protected]/* */ ~]$ [[email

基於Zookeeper的TbSchedule任務排程服務部署以及應用

一、前言前面已經說過zookeeper叢集的部署，接下來我們在zookeeper叢集的基礎上部署tbSchedule任務排程，開始部署之前我們先來簡要看看tbSchedule是什麼？ TBSchedule：分散式任務排程框架，主要是為了協調多節點處理相同任務的避免資

cdh版本的hue安裝配置部署以及集成hadoop hbase hive mysql等權威指南

service sources ces 兩個配置語句基金會 pan 創建 hue下載地址：https://github.com/cloudera/hue hue學習文檔地址：http://archive.cloudera.com/cdh5/cdh/5/hue-3.7

Gitlab的部署、應用以及502報錯解決-centos

gitlab安裝 gitlab漢化 gitlab502錯誤　　　　　　 Gitlab的部署以及應用GitLab簡介GitLab是一個用於倉庫管理系統的開源項目，使用Git作為代碼管理工具，它也是一個自

spark-2.2.0 叢集安裝部署以及hadoop叢集部署

Spark在生產環境中，主要部署在安裝Linux系統的叢集中。在linux系統中安裝Spark需要預先安裝JDK、Scala等所需要的依賴。由於Spark是計算框架，所以需要預先在叢集內有搭建好儲存資料的持久化層，如HDFS、Hive、Cassandra等，最後可以通過啟動指令碼執行應用。

apache伺服器安裝以及使用passenger外掛部署rails應用

小例子可以部署在rails自帶的WEBrick上，逐漸往後走還得上Apache。安裝apache伺服器命令是sudo apt-get install apache2 安裝passenger外掛安裝完畢還不能立刻用，因為想執行rails應用的話，還要為apache伺服器安裝外掛passenge

PPTP/L2TP協議詳解以及應用部署

PPTP/L2TP協議講述以及應用部署一、基本概念 PPTP PPTP是對端對端協議（PPP）的一種擴充套件，它採用了PPP所提供的身份驗證、壓縮與加密機制。PPTP能夠隨TCP/IP協議一道自動進行安裝。PPTP與Microsoft端對端加密（MPPE）技術提供

jenkins的容器化部署以及k8s應用的CI/CD實現

-name aced share spec csr .sql tin else 2.3 1. 使用Helm安裝Mysql: ??上一篇博文談到了如何使用Helm安裝Redis和RabbitMQ，下來我們來聊聊如何用Helm安裝mysql. ??本人對於Mysql數據庫不是非

hadoop記錄-[Flink]Flink三種運行模式安裝部署以及實現WordCount（轉載）

bject def ast atm jar int args 獲取 wordcount [Flink]Flink三種運行模式安裝部署以及實現WordCount 前言 Flink三種運行方式：Local、Standalone、On Yarn。成功部署後分別用Scala和J

泥鰍山老司機教你部署tomcat應用服務器

tomcat的安裝部署tomcat的安裝一、查看操作系統環境操作系統查看 linux 版本信息的方法：#uname –aLinux idc010vm010 2.6.32-220.4.2.el6.x86_64 #1 SMP Tue Feb 14 06:00:16 GMT 2012 x86_64 x86_64 x

HTML5 格式化方式以及應用

文字 nbsp con set html5 art char spa pre <b>加粗字體 <big>定義大號字體 <em>定義著重文字 <i>定義斜體字 <small>定義小號字體 <strong&g

線程機制、CLR線程池以及應用程序域

生死 stack 並不會 post 提交利用 tac 總結 window 線程機制、CLR線程池以及應用程序域最近在總結多線程、CLR線程池以及TPL編程實踐，重讀一遍CLR via C#，比剛上班的時候收獲還是很大的。還得要多讀書，讀好書，同時要多總結，多實踐

關於docker部署javaweb應用的問題

ava target docker部署問題是什麽 link javaweb 什麽 docker 我做了兩個鏡像，一個mysql，一個tomcat。建完mysql容器之後，在建tomcat的時候用--link把他們鏈接起來了進tomcat的容器裏面 /etc/hosts

使用Jenkins部署.Net應用程序

ons 認證們的 asp alt 驗證 files build路徑 file 首先從 https://jenkins.io/download/ 下載所需的版本這裏選擇Windows版本來測試。直接安裝jenkins.msi，安裝完後使用Win+R輸入services

Tomcat環境部署以及tomcat多實例搭建（同一臺機器）

java 軟件服務器 local profile 先搭建部署一臺Tomcat服務器：yum install -y gcc*本次需要的軟件安裝包：apache-tomcat-7.0.42.tar.gzjdk-7u25-linux-x64.tar.gznginx-1.2.6.tar.gz（安

SAPUI5教程——框架簡介以及應用實踐

sapui5 fiori 前言SAPUI5是SAP公司推出的一款前端UI技術框架，基於HTML5技術，開發語言為JavaScript, 誕生於2011年，此款移動框架和SAP 系列產品貼合緊密，開發迅速，符合SAP系統的整體風格，SAPUI5是一款封閉框架（收費），如果擁有SAP Netweaver

在jboss上部署web應用

jboss sss arc ips gin 包括 art edi src 1、JBoss介紹 JBoss完全實現了J2EE的服務棧： EJB (Enterprise JavaBeans) JMS (Java Message Service) JTS/JTA (J

GlusterFS 部署和應用

glusterfs 關於GlusterFS的原理理論介紹，請點擊這裏查看一、GlusterFS 安裝本博文講簡要介紹GlusterFS在CentOS 7 中的部署和應用。由於GlusterFS需要使用網絡，因此還必須事先根據環境設置防火墻規則，關閉SELinux。因為Glu

ELK部署logstash安裝部署及應用（二）

日誌 elk elkstack Logstash 安裝部署註意事項： Logstash基本概念:logstash收集日誌基本流程: input-->codec-->filter-->codec-->outputinput:從哪裏收集日誌。filter:發出去前進行過濾out

Tomcat中部署web應用 ---- Dubbo服務消費者Web應用war包的部署

目錄虛擬 iptables 設置部署 inpu 端口 www 所有樣例視頻：http://www.roncoo.com/course/view/f614343765bc4aac8597c6d8b38f06fd IP: 192.168.2.61 部署容器：apach

hadoop的部署以及應用

相關推薦