> For the complete documentation index, see [llms.txt](https://vitalino.gitbook.io/index-of/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://vitalino.gitbook.io/index-of/bigdata/brincando-com-o-spark.md).

# Brincando com o Spark

Calculando o Pi:

```bash
spark-submit --class org.apache.spark.examples.SparkPi \
    --master yarn \
    --deploy-mode cluster \
    $SPARK_HOME/examples/jars/spark-examples_*.jar 10
```

Contando palavras:

```bash
spark-submit --class org.apache.spark.examples.JavaWordCount \
    --master yarn \
    --deploy-mode cluster \
    $SPARK_HOME/examples/jars/spark-examples_*.jar hdfs:///azul.txt
```

Execute o Spark Shell:

```bash
pyspark
```

Ler arquivo do HDFS e fazer operações sobre ele:

```python
df = spark.read.text("hdfs:///azul.txt")
df.count()
df.show(51)
df.show(51, False)
df.show(10, False)
df.show(1, False)
```

Outro formato para manipulação dos dados:

```python
rdd = sc.textFile("hdfs:///azul.txt")
rdd.take(20)
```

Filtrando linhas que contenham ocorrências da palavra "azul":

```bash
df.filter(df.value.contains("azul")).show(truncate=False)

```

Contador de palavras:

```bash
from pyspark.sql.functions import explode, split

words = df.select(explode(split(df.value, "\\s+")).alias("word"))
words.groupBy("word").count().orderBy("count", ascending=False).show(50)
```

Acessando dump de bancos no HDFS:

```python
from pyspark.sql import SparkSession

# Inicia sessão do Spark
spark = SparkSession.builder \
    .appName("ReadWordpressSqoopText") \
    .getOrCreate()

# Caminho no HDFS onde o Sqoop salvou os arquivos
hdfs_path = "hdfs:///user/hive/warehouse/wp_posts"

# Leitura no formato texto
df = spark.read.text(hdfs_path)

# Mostra as primeiras linhas
df.show(20, truncate=False)

# Mostra o schema (sempre será: value STRING)
df.printSchema()

df.createOrReplaceTempView("wp_posts_text")

# Exemplo de consulta SQL filtrando por linhas que contêm a string "post_title"
result = spark.sql("""
    SELECT value
    FROM wp_posts_text
    WHERE value LIKE '%post%'
""")

result.show(truncate=False)

spark.stop()
```

Conectando ao banco e executando um SELECT:

```python
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("MySQL") \
    .config("spark.jars", "/usr/local/lib/sqoop/lib/mysql-connector-java-5.1.49-bin.jar") \
    .getOrCreate()

df = spark.read.format("jdbc").options(
    url="jdbc:mysql://banco-mysql/wordpress",
    driver="com.mysql.jdbc.Driver",
    dbtable="wp_posts",
    user="wp_user",
    password="senha123456"
).load()

df.groupBy("post_author").count().show()
```