Skip to content

Commit 16c09a8

Browse files
judasnjudasn
committed
🚧 hadoop
1 parent a252c14 commit 16c09a8

File tree

4 files changed

+55
-23
lines changed

4 files changed

+55
-23
lines changed

‎README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,7 @@
9494
- [CI 整套服务安装和使用](markdown-file/CI-Install-And-Usage.md)
9595
- [YApi 安装和配置](markdown-file/YApi-Install-And-Settings.md)
9696
- [Kafka 安装和配置](markdown-file/Kafka-Install-And-Settings.md)
97+
- [Hadoop 安装和配置](markdown-file/Hadoop-Install-And-Settings.md)
9798
- [Showdoc 安装和配置](markdown-file/Showdoc-Install-And-Settings.md)
9899
- [WordPress 安装和配置](markdown-file/WordPress-Install-And-Settings.md)
99100
- [GoAccess 安装和配置](markdown-file/GoAccess-Install-And-Settings.md)

‎SUMMARY.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,7 @@
8080
* [CI 整套服务安装和使用](markdown-file/CI-Install-And-Usage.md)
8181
* [YApi 安装和配置](markdown-file/YApi-Install-And-Settings.md)
8282
* [Kafka 安装和配置](markdown-file/Kafka-Install-And-Settings.md)
83+
* [Hadoop 安装和配置](markdown-file/Hadoop-Install-And-Settings.md)
8384
* [Showdoc 安装和配置](markdown-file/Showdoc-Install-And-Settings.md)
8485
* [WordPress 安装和配置](markdown-file/WordPress-Install-And-Settings.md)
8586
* [GoAccess 安装和配置](markdown-file/GoAccess-Install-And-Settings.md)

‎TOC.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,7 @@
7777
- [Node.js 安装和使用](markdown-file/Node-Install-And-Usage.md)
7878
- [YApi 安装和配置](markdown-file/YApi-Install-And-Settings.md)
7979
- [Kafka 安装和配置](markdown-file/Kafka-Install-And-Settings.md)
80+
- [Hadoop 安装和配置](markdown-file/Hadoop-Install-And-Settings.md)
8081
- [Showdoc 安装和配置](markdown-file/Showdoc-Install-And-Settings.md)
8182
- [WordPress 安装和配置](markdown-file/WordPress-Install-And-Settings.md)
8283
- [GoAccess 安装和配置](markdown-file/GoAccess-Install-And-Settings.md)

‎markdown-file/Hadoop-Install-And-Settings.md

Lines changed: 52 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ ssh localhost
6161
```
6262

6363
- 将公钥复制到两台 slave
64+
- 如果你是采用 pem 登录的，可以看这个：[SSH 免密登录](SSH-login-without-password.md)
6465

6566
```
6667
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 root@172.16.0.43，根据提示输入 hadoop-node1 机器的 root 密码，成功会有相应提示
@@ -95,6 +96,7 @@ tar zxvf hadoop-2.6.5.tar.gz，有 191M 左右
9596
```
9697

9798
- **给三台机子都先设置 HADOOP_HOME**
99+
- 会 ansible playbook 会方便点：[Ansible 安装和配置](Ansible-Install-And-Settings.md)
98100

99101
```
100102
vim /etc/profile
@@ -338,29 +340,31 @@ SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
338340
339341
```
340342

341-
- 启动
343+
## HDFS 启动
344+
345+
- 启动：start-dfs.sh，根据提示一路 yes
342346

343347
```
344-
启动：start-dfs.sh，根据提示一路 yes
345-
hadoop-master 会启动：NameNode 和 SecondaryNameNode
346-
从节点启动：DataNode
348+
这个命令效果：
349+
主节点会启动任务：NameNode 和 SecondaryNameNode
350+
从节点会启动任务：DataNode
351+
347352
348-
查看：jps，可以看到：
353+
主节点查看：jps，可以看到：
349354
21922 Jps
350355
21603 NameNode
351356
21787 SecondaryNameNode
352357
353358
354-
然后再从节点可以 jps 可以看到：
359+
从节点查看：jps 可以看到：
355360
19728 DataNode
356361
19819 Jps
357-
358362
```
359363

360-
```
361364

362-
查看运行更多情况：hdfs dfsadmin -report
365+
- 查看运行更多情况：`hdfs dfsadmin -report`
363366

367+
```
364368
Configured Capacity: 0 (0 B)
365369
Present Capacity: 0 (0 B)
366370
DFS Remaining: 0 (0 B)
@@ -371,15 +375,9 @@ Blocks with corrupt replicas: 0
371375
Missing blocks: 0
372376
```
373377

378+
- 如果需要停止：`stop-dfs.sh`
379+
- 查看 log：`cd $HADOOP_HOME/logs`
374380

375-
```
376-
377-
如果需要停止：stop-dfs.sh
378-
379-
查看 log：cd $HADOOP_HOME/logs
380-
381-
382-
```
383381

384382
## YARN 运行
385383

@@ -391,22 +389,53 @@ start-yarn.sh
391389
392390
停止：stop-yarn.sh
393391
392+
```
393+
394+
## 端口情况
394395

396+
- 主节点当前运行的所有端口：`netstat -tpnl | grep java`
397+
- 会用到端口（为了方便展示，整理下顺序）：
398+
399+
```
400+
tcp 0 0 172.16.0.17:9000 0.0.0.0:* LISTEN 22932/java >> NameNode
401+
tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 22932/java >> NameNode
402+
tcp 0 0 0.0.0.0:50090 0.0.0.0:* LISTEN 23125/java >> SecondaryNameNode
403+
tcp6 0 0 172.16.0.17:8030 :::* LISTEN 23462/java >> ResourceManager
404+
tcp6 0 0 172.16.0.17:8031 :::* LISTEN 23462/java >> ResourceManager
405+
tcp6 0 0 172.16.0.17:8032 :::* LISTEN 23462/java >> ResourceManager
406+
tcp6 0 0 172.16.0.17:8033 :::* LISTEN 23462/java >> ResourceManager
407+
tcp6 0 0 172.16.0.17:8088 :::* LISTEN 23462/java >> ResourceManager
408+
```
409+
410+
- 从节点当前运行的所有端口：`netstat -tpnl | grep java`
411+
- 会用到端口（为了方便展示，整理下顺序）：
412+
413+
```
414+
tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 14545/java >> DataNode
415+
tcp 0 0 0.0.0.0:50020 0.0.0.0:* LISTEN 14545/java >> DataNode
416+
tcp 0 0 0.0.0.0:50075 0.0.0.0:* LISTEN 14545/java >> DataNode
417+
tcp6 0 0 :::8040 :::* LISTEN 14698/java >> NodeManager
418+
tcp6 0 0 :::8042 :::* LISTEN 14698/java >> NodeManager
419+
tcp6 0 0 :::13562 :::* LISTEN 14698/java >> NodeManager
420+
tcp6 0 0 :::37481 :::* LISTEN 14698/java >> NodeManager
395421
```
396422

397-
- 可以看到当前运行的所有端口：`netstat -tpnl | grep java`
423+
-------------------------------------------------------------------
398424

425+
## 管理界面
399426

427+
- 查看 HDFS 管理界面：<http://hadoop-master:50070>
428+
- 访问 YARN 管理界面：<http://hadoop-master:8088>
400429

401-
查看HDFS管理界面：http://hadoop-master:50070
402-
访问YARN管理界面：http://hadoop-master:8088
403430

431+
-------------------------------------------------------------------
404432

433+
## 运行作业
405434

406-
搭建完成之后，我们运行一个Mapreduce作业感受一下：
407-
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar pi 5 10
408-
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar wordcount /data/input /data/output/result
435+
- 运行一个 Mapreduce 作业试试：
436+
- `hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar pi 5 10`
409437

438+
-------------------------------------------------------------------
410439

411440
## 资料
412441

0 commit comments

Comments
 (0)