> For the complete documentation index, see [llms.txt](https://vitalino.gitbook.io/index-of/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://vitalino.gitbook.io/index-of/bigdata/subindo-um-cluster-hadoop.md).

# Subindo um cluster Hadoop

Crie a rede virtual "hadoop":

```bash
docker network create --driver=bridge hadoop
```

Instancie os containers docker hadoop-slave1 e hadoop-slave2:

```bash
docker run -d --name hadoop-slave1 \
 --network hadoop \
 --hostname hadoop-slave1 \
 vitalino/hadoop:1.3
 
docker run -d --name hadoop-slave2 \
 --network hadoop \
 --hostname hadoop-slave2 \
 vitalino/hadoop:1.3
```

Suba o hadoop-master:

```bash
docker run -it \
 --name hadoop-master \
 --network hadoop \
 --hostname hadoop-master \
 vitalino/hadoop:1.3 bash
```

Formate /root/hdfs:

```bash
/usr/local/hadoop/bin/hdfs namenode -format
```

Inicie os serviços:

```bash
service ssh start
$HADOOP_HOME/sbin/start-dfs.sh
$HADOOP_HOME/sbin/start-yarn.sh
```

Liste os nodes:

```bash
hdfs dfsadmin -report
yarn node -list -all
```

No shell da máquina hospedeira, logue como root e ajuste o arquivo /etc/hosts:

```bash
sudo su -
```

```bash
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}} {{.Name}}' $(docker ps -q) | tr -d / >> /etc/hosts
```

Abra o navegador, e acesse:

* <http://hadoop-master:50070/explorer.html>
* [http://hadoop-master:8088](http://hadoop-master:8088/)

Calcule o número Pi usando o método estatístico de Monte Carlo:

```bash
hadoop \
 jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.10.2.jar \
 pi 16 16
```

Conte as ocorrências da palavra "azul":

```bash
wget https://blog.vitalino.com.br/azul.txt
hadoop dfs -put azul.txt /
hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.10.2.jar wordcount /azul.txt /palavras
```

Veja a saída em:

```bash
hdfs dfs -cat /palavras/part-r-00000
```

Crie e ajuste permissões de diretórios para o Apache Hive:

```bash
hadoop fs -mkdir -p /user/hive/warehouse
hadoop fs -chmod g+w /user/hive/warehouse
hadoop fs -mkdir -p /tmp/hive
hadoop fs -chmod a+w /tmp/hive
hadoop fs -chmod g+w /tmp
```

Inicialize a base de metadados do Hive:

```bash
$HIVE_HOME/bin/schematool -dbType derby -initSchema
```

Faça a importação das tabelas wordpress do blog no HDFS e Hive:

```bash
hadoop fs -ls /user/hive/warehouse
sqoop import-all-tables \
  --connect jdbc:mysql://blog.vitalino.com.br/wordpress \
  --username wp_user \
  --password senha123456 \
  --hive-import \
  --hive-database default \
  --hive-overwrite \
  --m 4
hadoop fs -ls /user/hive/warehouse
```

Liste os arquivos em: <http://hadoop-master:50070/explorer.html>

Acesse o Hive e execute algumas querys:

```bash
hive
```

```sql

SHOW DATABASES;
USE default;
SHOW TABLES;
DESCRIBE FORMATTED wp_posts;
SELECT post_title FROM wp_posts WHERE post_status = 'publish';
SELECT * FROM wp_posts;
SELECT post_author, COUNT(*)  FROM wp_posts GROUP BY post_author;
SELECT 
    u.ID AS author_id,
    u.display_name AS author_name,
    COUNT(p.ID) AS total_posts
FROM wp_posts p
INNER JOIN wp_users u
    ON p.post_author = u.ID
GROUP BY 
    u.ID,
    u.display_name
ORDER BY 
    total_posts DESC;
```