써니 노트 써니 노트

simply spearking, JSON is a tree-based model While CSV represents flat and homogeneous dataset, Json and XML has a power to express nested, heterogeneous dataset. 1) flat dataset 2) nested dataset 4) nested and heterogeneous dataset

Spark 책 추천 : Learning Spark https://pages.databricks.com/rs/094-YMS-629/images/LearningSpark2.0.pdf Spark transformation and action : https://spark.apache.org/docs/2.0.0/programming-guide.html#transformations

https://colab.research.google.com/github/RumbleDB/rumble/blob/master/RumbleSandbox.ipynb#scrollTo=nkuOUsukRBQb RumbleSandbox.ipynb Run, share, and edit Python notebooks colab.research.google.com official docs for JSONiq : https://www.jsoniq.org/docs/JSONiq/webhelp/index.html

word counting example 1) first, text file should be converted to key-value type 2) map function can map each word to value one 3) same key will be grouped and passed to reducer. then reduce function can just sum values over same key how selection works how projection works

Possible storage layer Hadoop infrastructure version 1

1. Download Hadoop files 2. Update necessary config files 3. Download Hive files 4. Update Hive config file 5. Install Hive metastore /* Update the system and install Java */ sudo apt update sudo apt install openjdk-8-jdk -y java -version; javac -version /* Install open SSH */ sudo apt install openssh-server openssh-client -y ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub >> ~/.s..

1. Creating Azure Resource Group Create Azure account and go to Resource Groups and hit the review&create button in the below *Resource group - A container that holds related resources for an Azure solution. The resource group can include all the resources for the solution, or only those resources that you want to manage as a group. You decide how you want to allocate resources to resource group..

Flume is a service for rapidly gathering, aggregating, and transporting massive amounts of log data that is distributed, reliable, and available. Its architecture is simple and adaptable, based on streaming data flows. It has configurable reliability techniques as well as several failovers and recovery mechanisms, making it resilient and fault tolerant. It employs a straightforward extensible da..