[SPARK] Spark的編譯環境: 以SparkPi為例

作者： -
在上一篇文章中, 我們介紹了如何安裝單機版的Spark,
接下來, 為了要撰寫Spark程式, 我們要先建立Spark的編譯環境,
此時, 就要用到SBT這個套件, SBT (Simple Build Tool) 是Scala的編譯器,
有興趣可以參考: https://www.scala-sbt.org/

接下來, 我們就要重新創一個專案,
重新執行之前範例中的SparkPi, 我們主要參考以下連結:
https://console.bluemix.net/docs/services/AnalyticsforApacheSpark/spark_app_example.html
因為有一些修改, 就把指令紀錄如下:

一開始, 先建立專案的資料夾, 把SparkPi移入,
mkdir -p ~/spark-submit/project
mkdir -p ~/spark-submit/src/main/scal
cp /usr/lib/spark/examples/src/main/scala/org/apache/spark/examples/SparkPi.scala ~/spark-submit/src/main/scala/SparkPi.scala

接著, 設定SBT的環境:
vim ~/spark-submit-example/build.sbt
貼上以下內容 (空行必須保留, Scala的版本號記得修改):
name := "SparkPi"

version := "1.0"

scalaVersion := "2.11.6"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.2"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.2"

resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

輸入SBT的對應版本號, 這裡是寫0.13.2,
截至目前為止2018/2/19, 最新版應該是1.1.1, 但是此處先照舊,
cat <<EOT > ~/spark-submit/project/build.properties
sbt.version=0.13.2
EOT

以下三行指令進行編譯, 必須先進行編譯後才能執行,
如果是第一次編譯會下載很多關聯檔案 (dependency), 大約要花1小時的時間
cd ~/spark-submit
sbt compile //compile the project
sbt run //run the project
sbt package //package the project

關於SBT編譯的說明可以參考這一篇文章:
https://alvinalexander.com/scala/sbt-how-to-compile-run-package-scala-project
該篇文章以Scala為例, 比較容易了解SBT和Scala之間的關係,
編譯好的Jar檔案在 ~/spark-submit-example/target/scala-2.10/sparkpi_2.10-1.0.jar
接著, 只要執行Jar檔就可以了, 指令如下:
spark-submit ~/spark-submit-example/target/scala-2.10/sparkpi_2.10-1.0.jar
在spark-submit時, 可以給予不同參數, 請參考:
https://console.bluemix.net/docs/services/AnalyticsforApacheSpark/spark_submit_example.html#example-running-a-spark-application-with-optional-parameters

如果出現 "ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your configuration"
是因為在原本宣告中, 不帶有本機執行的資訊,
請修改原始程式的這個部分, 並重新執行:
val spark = SparkSession
   .builder
   .appName("Spark Pi")
   .config("spark.master", "local")
   .getOrCreate()
結果, 顯示如下:
~/submit$ sbt compile
[info] Loading project definition from /home/ubuntu/submit/project
[info] Loading settings for project submit from build.sbt ...
[info] Set current project to SparkPi (in build file:/home/ubuntu/submit/)
[info] Executing in batch mode. For better performance use sbt's shell
[info] Compiling 1 Scala source to /home/ubuntu/submit/target/scala-2.11/classes ...
[info] Done compiling.
[success] Total time: 5 s, completed Oct 29, 2018 10:20:54 AM
ubuntu@testspark:~/submit$ sbt run
[info] Loading project definition from /home/ubuntu/submit/project
[info] Loading settings for project submit from build.sbt ...
[info] Set current project to SparkPi (in build file:/home/ubuntu/submit/)
[info] Packaging /home/ubuntu/submit/target/scala-2.11/sparkpi_2.11-1.0.jar ...
[info] Done packaging.
[info] Running org.apache.spark.examples.SparkPi
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
18/10/29 10:21:17 INFO SparkContext: Running Spark version 2.1.2
18/10/29 10:21:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/10/29 10:21:18 INFO SecurityManager: Changing view acls to: ubuntu
18/10/29 10:21:18 INFO SecurityManager: Changing modify acls to: ubuntu
18/10/29 10:21:18 INFO SecurityManager: Changing view acls groups to:
18/10/29 10:21:18 INFO SecurityManager: Changing modify acls groups to:
18/10/29 10:21:18 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(ubuntu); groups with view permissions: Set(); users  with modify permissions: Set(ubuntu); groups with modify permissions: Set()
18/10/29 10:21:18 INFO Utils: Successfully started service 'sparkDriver' on port 44564.
18/10/29 10:21:18 INFO SparkEnv: Registering MapOutputTracker
18/10/29 10:21:18 INFO SparkEnv: Registering BlockManagerMaster
18/10/29 10:21:18 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/10/29 10:21:18 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/10/29 10:21:18 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-bb299189-1de7-4ea8-a92a-a50e75b33c70
18/10/29 10:21:18 INFO MemoryStore: MemoryStore started with capacity 408.9 MB
18/10/29 10:21:18 INFO SparkEnv: Registering OutputCommitCoordinator
18/10/29 10:21:18 INFO Utils: Successfully started service 'SparkUI' on port 4040.
18/10/29 10:21:18 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://172.16.0.222:4040
18/10/29 10:21:19 INFO Executor: Starting executor ID driver on host localhost
18/10/29 10:21:19 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 45212.
18/10/29 10:21:19 INFO NettyBlockTransferService: Server created on 172.16.0.222:45212
18/10/29 10:21:19 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
18/10/29 10:21:19 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 172.16.0.222, 45212, None)
18/10/29 10:21:19 INFO BlockManagerMasterEndpoint: Registering block manager 172.16.0.222:45212 with 408.9 MB RAM, BlockManagerId(driver, 172.16.0.222, 45212, None)
18/10/29 10:21:19 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 172.16.0.222, 45212, None)
18/10/29 10:21:19 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 172.16.0.222, 45212, None)
18/10/29 10:21:19 INFO SparkContext: Starting job: reduce at SparkPi.scala:39
18/10/29 10:21:19 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:39) with 2 output partitions
18/10/29 10:21:19 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:39)
18/10/29 10:21:19 INFO DAGScheduler: Parents of final stage: List()
18/10/29 10:21:19 INFO DAGScheduler: Missing parents: List()
18/10/29 10:21:19 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:35), which has no missing parents
18/10/29 10:21:19 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1832.0 B, free 408.9 MB)
18/10/29 10:21:20 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1167.0 B, free 408.9 MB)
18/10/29 10:21:20 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.16.0.222:45212 (size: 1167.0 B, free: 408.9 MB)
18/10/29 10:21:20 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:996
18/10/29 10:21:20 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:35)
18/10/29 10:21:20 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
18/10/29 10:21:20 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 5953 bytes)
18/10/29 10:21:20 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
18/10/29 10:21:20 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1041 bytes result sent to driver
18/10/29 10:21:20 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL, 5953 bytes)
18/10/29 10:21:20 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
18/10/29 10:21:20 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 123 ms on localhost (executor driver) (1/2)
18/10/29 10:21:20 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 1041 bytes result sent to driver
18/10/29 10:21:20 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 37 ms on localhost (executor driver) (2/2)
18/10/29 10:21:20 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
18/10/29 10:21:20 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:39) finished in 0.170 s
18/10/29 10:21:20 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:39, took 0.410026 s
Pi is roughly 3.142555712778564
18/10/29 10:21:20 INFO SparkUI: Stopped Spark web UI at http://172.16.0.222:4040
18/10/29 10:21:20 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
18/10/29 10:21:20 INFO MemoryStore: MemoryStore cleared
18/10/29 10:21:20 INFO BlockManager: BlockManager stopped
18/10/29 10:21:20 INFO BlockManagerMaster: BlockManagerMaster stopped
18/10/29 10:21:20 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
18/10/29 10:21:20 INFO SparkContext: Successfully stopped SparkContext
[success] Total time: 5 s, completed Oct 29, 2018 10:21:20 AM
18/10/29 10:21:20 INFO ShutdownHookManager: Shutdown hook called
18/10/29 10:21:20 INFO ShutdownHookManager: Deleting directory /tmp/spark-ed77a1be-cb46-41c9-834d-09d07a18ebbf
基本上和之前範例一致,
不過, 由於我們沒有指定由幾個tasker執行 (之前指定為10),
因此, 按照預設值分成2份平行執行



留言

張貼留言

熱門文章

LTE筆記: RSRP, RSSI and RSRQ

作者： -
圖片
在一般通訊系統中, 我們常用 RSSI (Received Signal Strength Indication) 來表示訊號強度,  其單位為 (dBm), 可以轉換成功率大小 (mW): https://en.wikipedia.org/wiki/DBm [註] dBm 的原意即是: (dB) reference to (m)W, 因此, 1 W = 10*log10(1000mW) = 30dBm. 在 LTE 通訊系統中, 除了 RSSI 之外, 又額外定義了兩個用以標示訊號強度的數值: RSRP 和 RSRQ 以下, 我們將介紹 RSRP, RSRQ 和 RSSI 的不同： Reference Symbol Received Power (RSRP) RSRP 為 UE 量測 Reference Signals (RS) 的平均訊號強度, DRS 為基地台在下行訊號中, 於固定 RB (Resource Block) 上傳送的訊號, 由於這些訊號具有特定的傳送格式, 接收端便可以解出這些控制訊號的強度, 對於不同的相鄰基地台, 甚至是同一基地台不同的天線, 可以透過不同的 RS 配置, 讓使用者得知不同目標的量測數值, 如下圖所示: 紅色為 RS, 不同基地台或天線使用不同的 RS 排列方法, 可分開量測 來自: https://api.semanticscholar.org/CorpusID:22030582 RSRP 的回報由 UE 進行 Measurement Report, 回傳給基地台, 其中, 關於 Measurement Report 的設置定義於 RRC 的規範中, 我們之後有機會再提, RSRP 的量測, 通常在換手 (handover) 時進行量測, 用以指出目標基地台的訊號強度, 其中, UE 所量測的數值, 將藉由下表進行轉換 (訊號強度 (dBm) - (-140)), 並回報給 eNB 作為換手的決策使用: (3GPP Reference: TS 36.133) RSSI (Received Signal Strength Indication)  RSSI 是 UE 接收到的 wide-band 訊號強度, 包含 DRS 以及其他...
閱讀完整內容

[WiFi] WiFi 網路的識別: BSS, ESS, SSID, ESSID, BSSID

作者： -
圖片
在 WiFi 網路中, 一個基本的識別方式就是 SSID (Service Set Identifier), 也就是我們口語中的 "WiFi 網路名稱", 然而, 此網路事實上是一個服務集合 (Service Set) 的概念, 而一個服務群集, 也可以對應到一個基本的無線網路, 包含: 一個 WiFi AP 以及多個 WiFi 使用者, 對於剛剛這種簡單的網路架構, 又稱為 BSS (Basic Service Set), 若是多個 BSS 都使用同一個 SSID, 則稱為 ESS (Extend Service Set), 如下圖所示: 來自: https://ppt.cc/fXIvex 對於 ESS 來說, 有兩個假設, 第一, 底下的 BSS 必須是相鄰 (有交疊範圍) 第二, 這些 BSS 有網路連線, (有線或是無線都可以) 這兩個假設是為了完成 ESS 的主要功能, 也就是換手 (handoff), 當裝置在同一個 ESS 下不同 BSS 範圍中移動時, 可不必重新連線就可以繼續傳輸. 這樣的流程事實上是由原本服務的 AP (BSS 1), 把使用者的資料轉傳到換手目標的 AP (BSS 2), 資料流程為: DS -> BSS 1 -> BSS 2 -> Sation, 也因此, 兩個 AP 之間必須連線, 同時切換時間也不能太久 (相鄰條件), 那麼在上述例子中, 如何知道 BSS 1 和 BSS 2 的差別呢? 在 WiFi 網路中, 為了完成這件事情, 所以就有了 ESSID 和 BSSID, ESSID 如上所述, 對應於 ESS 所使用的 SSID, 至於 BSSID (通常是 WiFi AP 的 MAC 位址) 來區分同個 SSID 下的網路. 反過來說, 同一個 WiFi AP 也可以擁有多個 SSID, 此功能稱為 VAP (Virtual AP), 此時, 一個 WiFi AP 會虛擬出多個 BSSID, 通常作法就是從原有的 MAC 位址往上加一, 作為多個不同的 BSSID. 這些 VAP 會對應到 WiFi AP 中的不同 VLAN 中, 因此, 我們可以給他們不同的網路設定, 例如: 流量, 安全性, 頻寬等, ...
閱讀完整內容

LTE筆記: 5G NR Measurement Events

作者： -
圖片
在通訊系統中, 量測訊號 (RSRP, RSRQ, SINR) 主要的用途就是用來進行換手, 此功能從 2G 系統就被實作, 並隨著更多通訊技術的發展而增加, 在 TS 3GPP TS 38.331 中定義了 5G 網路中, 以下和換手與量測報告的相關驅動事件: Event A1 (Serving becomes better than threshold) Event A2 (Serving becomes worse than threshold) Event A3 (Neighbor becomes offset better than SpCell) Event A4 (Neighbor becomes better than threshold) Event A5 (SpCell becomes worse than threshold1 and neighbor becomes better than threshold2) Event A6 (Neighbour becomes offset better than SCell) Event B1 (Inter RAT neighbour becomes better than threshold) Event B2 (SpCell becomes worse than threshold1 and inter RAT neighbor becomes better than threshold2) 其中, A 系列的事件對應於相同的 RAT (例如: 4G, 5G, etc), B 系列的事件, 則對應於 RAT 之間 (inter-RAT) 的換手,  在各事件的描述中, 以下為一些簡寫的說明: special cell (SpCell): 主要的服務基地台 secondary cell (SCell): 次要的服務基地台, 提供 CA (Carrier Aggregation) 功能 在各項驅動事件中, threshold 為針對事件的一致性定義, 針對每一事件, 我們都可以定義其對應的數值, offset 則是針對每一個基地台定義, 其數值可以為正或為負, 各項數值的物理意義與定義範圍顯示如下表: 在所有的事件中, 我們可以依照其驅動條件分成兩類: 絕對數值事件: A1, A2...
閱讀完整內容