Created
September 18, 2021 03:56
-
-
Save Kontinuation/056ae64d3ecfd5a03a1dd2b45a6dd12f to your computer and use it in GitHub Desktop.
Spatial join fails after merging commit a0ed49 ([SEDONA-28] shape&wkb ser de)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 21/09/18 11:51:31 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable | |
| Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties | |
| 21/09/18 11:51:31 INFO SparkContext: Running Spark version 3.1.2 | |
| 21/09/18 11:51:31 INFO ResourceUtils: ============================================================== | |
| 21/09/18 11:51:31 INFO ResourceUtils: No custom resources configured for spark.driver. | |
| 21/09/18 11:51:31 INFO ResourceUtils: ============================================================== | |
| 21/09/18 11:51:31 INFO SparkContext: Submitted application: SedonaSQL-demo | |
| 21/09/18 11:51:31 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) | |
| 21/09/18 11:51:31 INFO ResourceProfile: Limiting resource is cpu | |
| 21/09/18 11:51:31 INFO ResourceProfileManager: Added ResourceProfile id: 0 | |
| 21/09/18 11:51:31 INFO SecurityManager: Changing view acls to: kontinuation | |
| 21/09/18 11:51:31 INFO SecurityManager: Changing modify acls to: kontinuation | |
| 21/09/18 11:51:31 INFO SecurityManager: Changing view acls groups to: | |
| 21/09/18 11:51:31 INFO SecurityManager: Changing modify acls groups to: | |
| 21/09/18 11:51:31 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(kontinuation); groups with view permissions: Set(); users with modify permissions: Set(kontinuation); groups with modify permissions: Set() | |
| 21/09/18 11:51:31 INFO Utils: Successfully started service 'sparkDriver' on port 35501. | |
| 21/09/18 11:51:31 INFO SparkEnv: Registering MapOutputTracker | |
| 21/09/18 11:51:31 INFO SparkEnv: Registering BlockManagerMaster | |
| 21/09/18 11:51:31 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information | |
| 21/09/18 11:51:31 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up | |
| 21/09/18 11:51:31 INFO SparkEnv: Registering BlockManagerMasterHeartbeat | |
| 21/09/18 11:51:31 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-46091ea7-98e3-405b-b461-93b295962c72 | |
| 21/09/18 11:51:31 INFO MemoryStore: MemoryStore started with capacity 366.3 MiB | |
| 21/09/18 11:51:32 INFO SparkEnv: Registering OutputCommitCoordinator | |
| 21/09/18 11:51:32 INFO Utils: Successfully started service 'SparkUI' on port 4040. | |
| 21/09/18 11:51:32 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://kontinuation:4040 | |
| 21/09/18 11:51:32 INFO SparkContext: Added JAR file:/home/kontinuation/documents/geospatial/experi/sedona-test/target/sedona-test_2.12-1.0.0-SNAPSHOT-shaded.jar at spark://kontinuation:35501/jars/sedona-test_2.12-1.0.0-SNAPSHOT-shaded.jar with timestamp 1631937091411 | |
| 21/09/18 11:51:32 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://kontinuation:7077... | |
| 21/09/18 11:51:32 INFO TransportClientFactory: Successfully created connection to kontinuation/172.25.87.59:7077 after 29 ms (0 ms spent in bootstraps) | |
| 21/09/18 11:51:32 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20210918115132-0003 | |
| 21/09/18 11:51:32 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20210918115132-0003/0 on worker-20210918105325-172.25.87.59-40855 (172.25.87.59:40855) with 4 core(s) | |
| 21/09/18 11:51:32 INFO StandaloneSchedulerBackend: Granted executor ID app-20210918115132-0003/0 on hostPort 172.25.87.59:40855 with 4 core(s), 1024.0 MiB RAM | |
| 21/09/18 11:51:32 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44191. | |
| 21/09/18 11:51:32 INFO NettyBlockTransferService: Server created on kontinuation:44191 | |
| 21/09/18 11:51:32 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy | |
| 21/09/18 11:51:32 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, kontinuation, 44191, None) | |
| 21/09/18 11:51:32 INFO BlockManagerMasterEndpoint: Registering block manager kontinuation:44191 with 366.3 MiB RAM, BlockManagerId(driver, kontinuation, 44191, None) | |
| 21/09/18 11:51:32 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, kontinuation, 44191, None) | |
| 21/09/18 11:51:32 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, kontinuation, 44191, None) | |
| 21/09/18 11:51:32 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20210918115132-0003/0 is now RUNNING | |
| 21/09/18 11:51:33 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 | |
| 21/09/18 11:51:33 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/home/kontinuation/documents/geospatial/experi/sedona-test/spark-warehouse'). | |
| 21/09/18 11:51:33 INFO SharedState: Warehouse path is 'file:/home/kontinuation/documents/geospatial/experi/sedona-test/spark-warehouse'. | |
| 21/09/18 11:51:35 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.25.87.59:53362) with ID 0, ResourceProfileId 0 | |
| 21/09/18 11:51:35 INFO BlockManagerMasterEndpoint: Registering block manager 172.25.87.59:44483 with 366.3 MiB RAM, BlockManagerId(0, 172.25.87.59, 44483, None) | |
| Sedona Configuration: | |
| useIndex: false | |
| indexType: QUADTREE | |
| serializerType: SHAPE | |
| joinSparitionDominantSide: LEFT | |
| joinBuildSide: LEFT | |
| joinApproximateTotalCount: -1 | |
| datasetBoundary: Env[0.0 : 0.0, 0.0 : 0.0] | |
| fallbackPartitionNum: -1 | |
| joinGridType: KDBTREE | |
| 21/09/18 11:51:36 INFO InMemoryFileIndex: It took 34 ms to list leaf files for 1 paths. | |
| 21/09/18 11:51:36 INFO InMemoryFileIndex: It took 2 ms to list leaf files for 1 paths. | |
| 21/09/18 11:51:37 INFO FileSourceStrategy: Pushed Filters: | |
| 21/09/18 11:51:37 INFO FileSourceStrategy: Post-Scan Filters: (length(trim(value#6, None)) > 0) | |
| 21/09/18 11:51:37 INFO FileSourceStrategy: Output Data Schema: struct<value: string> | |
| 21/09/18 11:51:38 INFO CodeGenerator: Code generated in 255.211895 ms | |
| 21/09/18 11:51:38 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 286.9 KiB, free 366.0 MiB) | |
| 21/09/18 11:51:38 INFO SedonaWKBKryoRegistrator: Registering custom serializers for geometry types | |
| 21/09/18 11:51:38 INFO SedonaVizKryoRegistrator: Registering custom serializers for visualization related types | |
| 21/09/18 11:51:38 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 24.2 KiB, free 366.0 MiB) | |
| 21/09/18 11:51:38 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on kontinuation:44191 (size: 24.2 KiB, free: 366.3 MiB) | |
| 21/09/18 11:51:38 INFO SparkContext: Created broadcast 0 from load at ExperiSedona.scala:53 | |
| 21/09/18 11:51:38 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. | |
| 21/09/18 11:51:38 INFO SparkContext: Starting job: load at ExperiSedona.scala:53 | |
| 21/09/18 11:51:38 INFO DAGScheduler: Got job 0 (load at ExperiSedona.scala:53) with 1 output partitions | |
| 21/09/18 11:51:38 INFO DAGScheduler: Final stage: ResultStage 0 (load at ExperiSedona.scala:53) | |
| 21/09/18 11:51:38 INFO DAGScheduler: Parents of final stage: List() | |
| 21/09/18 11:51:38 INFO DAGScheduler: Missing parents: List() | |
| 21/09/18 11:51:38 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at load at ExperiSedona.scala:53), which has no missing parents | |
| 21/09/18 11:51:39 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 10.8 KiB, free 366.0 MiB) | |
| 21/09/18 11:51:39 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 5.4 KiB, free 366.0 MiB) | |
| 21/09/18 11:51:39 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on kontinuation:44191 (size: 5.4 KiB, free: 366.3 MiB) | |
| 21/09/18 11:51:39 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1388 | |
| 21/09/18 11:51:39 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at load at ExperiSedona.scala:53) (first 15 tasks are for partitions Vector(0)) | |
| 21/09/18 11:51:39 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks resource profile 0 | |
| 21/09/18 11:51:39 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (172.25.87.59, executor 0, partition 0, PROCESS_LOCAL, 4925 bytes) taskResourceAssignments Map() | |
| 21/09/18 11:51:39 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 172.25.87.59:44483 (size: 5.4 KiB, free: 366.3 MiB) | |
| 21/09/18 11:51:40 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.25.87.59:44483 (size: 24.2 KiB, free: 366.3 MiB) | |
| 21/09/18 11:51:41 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 2236 ms on 172.25.87.59 (executor 0) (1/1) | |
| 21/09/18 11:51:41 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool | |
| 21/09/18 11:51:41 INFO DAGScheduler: ResultStage 0 (load at ExperiSedona.scala:53) finished in 2.364 s | |
| 21/09/18 11:51:41 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job | |
| 21/09/18 11:51:41 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished | |
| 21/09/18 11:51:41 INFO DAGScheduler: Job 0 finished: load at ExperiSedona.scala:53, took 2.417185 s | |
| 21/09/18 11:51:41 INFO CodeGenerator: Code generated in 13.549909 ms | |
| 21/09/18 11:51:41 INFO FileSourceStrategy: Pushed Filters: | |
| 21/09/18 11:51:41 INFO FileSourceStrategy: Post-Scan Filters: | |
| 21/09/18 11:51:41 INFO FileSourceStrategy: Output Data Schema: struct<value: string> | |
| 21/09/18 11:51:41 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 286.9 KiB, free 365.7 MiB) | |
| 21/09/18 11:51:41 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 24.2 KiB, free 365.7 MiB) | |
| 21/09/18 11:51:41 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on kontinuation:44191 (size: 24.2 KiB, free: 366.2 MiB) | |
| 21/09/18 11:51:41 INFO SparkContext: Created broadcast 2 from load at ExperiSedona.scala:53 | |
| 21/09/18 11:51:41 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. | |
| 21/09/18 11:51:41 INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. | |
| 21/09/18 11:51:41 INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. | |
| 21/09/18 11:51:41 INFO FileSourceStrategy: Pushed Filters: | |
| 21/09/18 11:51:41 INFO FileSourceStrategy: Post-Scan Filters: (length(trim(value#30, None)) > 0) | |
| 21/09/18 11:51:41 INFO FileSourceStrategy: Output Data Schema: struct<value: string> | |
| 21/09/18 11:51:41 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 286.9 KiB, free 365.4 MiB) | |
| 21/09/18 11:51:41 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 24.2 KiB, free 365.4 MiB) | |
| 21/09/18 11:51:41 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on kontinuation:44191 (size: 24.2 KiB, free: 366.2 MiB) | |
| 21/09/18 11:51:41 INFO SparkContext: Created broadcast 3 from load at ExperiSedona.scala:61 | |
| 21/09/18 11:51:41 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. | |
| 21/09/18 11:51:41 INFO SparkContext: Starting job: load at ExperiSedona.scala:61 | |
| 21/09/18 11:51:41 INFO DAGScheduler: Got job 1 (load at ExperiSedona.scala:61) with 1 output partitions | |
| 21/09/18 11:51:41 INFO DAGScheduler: Final stage: ResultStage 1 (load at ExperiSedona.scala:61) | |
| 21/09/18 11:51:41 INFO DAGScheduler: Parents of final stage: List() | |
| 21/09/18 11:51:41 INFO DAGScheduler: Missing parents: List() | |
| 21/09/18 11:51:41 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[13] at load at ExperiSedona.scala:61), which has no missing parents | |
| 21/09/18 11:51:41 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 10.8 KiB, free 365.4 MiB) | |
| 21/09/18 11:51:41 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 5.4 KiB, free 365.4 MiB) | |
| 21/09/18 11:51:41 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on kontinuation:44191 (size: 5.4 KiB, free: 366.2 MiB) | |
| 21/09/18 11:51:41 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1388 | |
| 21/09/18 11:51:41 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[13] at load at ExperiSedona.scala:61) (first 15 tasks are for partitions Vector(0)) | |
| 21/09/18 11:51:41 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks resource profile 0 | |
| 21/09/18 11:51:42 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1) (172.25.87.59, executor 0, partition 0, PROCESS_LOCAL, 4921 bytes) taskResourceAssignments Map() | |
| 21/09/18 11:51:42 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 172.25.87.59:44483 (size: 5.4 KiB, free: 366.3 MiB) | |
| 21/09/18 11:51:42 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 172.25.87.59:44483 (size: 24.2 KiB, free: 366.2 MiB) | |
| 21/09/18 11:51:42 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 135 ms on 172.25.87.59 (executor 0) (1/1) | |
| 21/09/18 11:51:42 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool | |
| 21/09/18 11:51:42 INFO DAGScheduler: ResultStage 1 (load at ExperiSedona.scala:61) finished in 0.144 s | |
| 21/09/18 11:51:42 INFO DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job | |
| 21/09/18 11:51:42 INFO TaskSchedulerImpl: Killing all running tasks in stage 1: Stage finished | |
| 21/09/18 11:51:42 INFO DAGScheduler: Job 1 finished: load at ExperiSedona.scala:61, took 0.148971 s | |
| 21/09/18 11:51:42 INFO FileSourceStrategy: Pushed Filters: | |
| 21/09/18 11:51:42 INFO FileSourceStrategy: Post-Scan Filters: | |
| 21/09/18 11:51:42 INFO FileSourceStrategy: Output Data Schema: struct<value: string> | |
| 21/09/18 11:51:42 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 286.9 KiB, free 365.1 MiB) | |
| 21/09/18 11:51:42 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 24.2 KiB, free 365.1 MiB) | |
| 21/09/18 11:51:42 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on kontinuation:44191 (size: 24.2 KiB, free: 366.2 MiB) | |
| 21/09/18 11:51:42 INFO SparkContext: Created broadcast 5 from load at ExperiSedona.scala:61 | |
| 21/09/18 11:51:42 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. | |
| 21/09/18 11:51:42 INFO JoinQueryDetector: Planning spatial join for ST_Contains relationship | |
| 21/09/18 11:51:42 INFO FileSourceStrategy: Pushed Filters: | |
| 21/09/18 11:51:42 INFO FileSourceStrategy: Post-Scan Filters: | |
| 21/09/18 11:51:42 INFO FileSourceStrategy: Output Data Schema: struct<_c0: string, _c1: string> | |
| 21/09/18 11:51:42 INFO FileSourceStrategy: Pushed Filters: | |
| 21/09/18 11:51:42 INFO FileSourceStrategy: Post-Scan Filters: | |
| 21/09/18 11:51:42 INFO FileSourceStrategy: Output Data Schema: struct<_c1: string> | |
| == Physical Plan == | |
| *(3) Sort [cnt#54L DESC NULLS LAST], true, 0 | |
| +- Exchange rangepartitioning(cnt#54L DESC NULLS LAST, 200), ENSURE_REQUIREMENTS, [id=#93] | |
| +- *(2) HashAggregate(keys=[name#26], functions=[count(1)]) | |
| +- Exchange hashpartitioning(name#26, 200), ENSURE_REQUIREMENTS, [id=#89] | |
| +- *(1) HashAggregate(keys=[name#26], functions=[partial_count(1)]) | |
| +- *(1) Project [name#26] | |
| +- RangeJoin geom#27: geometry, geom#51: geometry, false | |
| :- Project [_c0#22 AS name#26, st_geomfromwkt(_c1#23) AS geom#27] | |
| : +- FileScan csv [_c0#22,_c1#23] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex[file:/home/kontinuation/documents/geospatial/experi/sedona-test/data/china_prov..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<_c0:string,_c1:string> | |
| +- Project [st_geomfromwkt(_c1#47) AS geom#51] | |
| +- FileScan csv [_c1#47] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex[file:/home/kontinuation/documents/geospatial/experi/sedona-test/data/ubm_trip_r..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<_c1:string> | |
| 21/09/18 11:51:42 INFO JoinQueryDetector: Planning spatial join for ST_Contains relationship | |
| 21/09/18 11:51:42 INFO FileSourceStrategy: Pushed Filters: | |
| 21/09/18 11:51:42 INFO FileSourceStrategy: Post-Scan Filters: | |
| 21/09/18 11:51:42 INFO FileSourceStrategy: Output Data Schema: struct<_c0: string, _c1: string> | |
| 21/09/18 11:51:42 INFO FileSourceStrategy: Pushed Filters: | |
| 21/09/18 11:51:42 INFO FileSourceStrategy: Post-Scan Filters: | |
| 21/09/18 11:51:42 INFO FileSourceStrategy: Output Data Schema: struct<_c1: string> | |
| 21/09/18 11:51:42 INFO CodeGenerator: Code generated in 14.883032 ms | |
| 21/09/18 11:51:42 INFO CodeGenerator: Code generated in 30.195449 ms | |
| 21/09/18 11:51:42 INFO CodeGenerator: Code generated in 44.536571 ms | |
| 21/09/18 11:51:42 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 286.8 KiB, free 364.8 MiB) | |
| 21/09/18 11:51:42 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 24.3 KiB, free 364.7 MiB) | |
| 21/09/18 11:51:42 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on kontinuation:44191 (size: 24.3 KiB, free: 366.2 MiB) | |
| 21/09/18 11:51:42 INFO SparkContext: Created broadcast 6 from show at ExperiSedona.scala:73 | |
| 21/09/18 11:51:42 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. | |
| 21/09/18 11:51:42 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 286.8 KiB, free 364.5 MiB) | |
| 21/09/18 11:51:42 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 24.3 KiB, free 364.4 MiB) | |
| 21/09/18 11:51:42 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on kontinuation:44191 (size: 24.3 KiB, free: 366.1 MiB) | |
| 21/09/18 11:51:42 INFO SparkContext: Created broadcast 7 from show at ExperiSedona.scala:73 | |
| 21/09/18 11:51:42 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. | |
| 21/09/18 11:51:42 INFO SparkContext: Starting job: aggregate at SpatialRDD.java:481 | |
| 21/09/18 11:51:42 INFO DAGScheduler: Got job 2 (aggregate at SpatialRDD.java:481) with 2 output partitions | |
| 21/09/18 11:51:42 INFO DAGScheduler: Final stage: ResultStage 2 (aggregate at SpatialRDD.java:481) | |
| 21/09/18 11:51:42 INFO DAGScheduler: Parents of final stage: List() | |
| 21/09/18 11:51:42 INFO DAGScheduler: Missing parents: List() | |
| 21/09/18 11:51:42 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[26] at show at ExperiSedona.scala:73), which has no missing parents | |
| 21/09/18 11:51:42 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 14.5 KiB, free 364.4 MiB) | |
| 21/09/18 11:51:42 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 7.6 KiB, free 364.4 MiB) | |
| 21/09/18 11:51:42 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on kontinuation:44191 (size: 7.6 KiB, free: 366.1 MiB) | |
| 21/09/18 11:51:42 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1388 | |
| 21/09/18 11:51:42 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 2 (MapPartitionsRDD[26] at show at ExperiSedona.scala:73) (first 15 tasks are for partitions Vector(0, 1)) | |
| 21/09/18 11:51:42 INFO TaskSchedulerImpl: Adding task set 2.0 with 2 tasks resource profile 0 | |
| 21/09/18 11:51:42 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2) (172.25.87.59, executor 0, partition 0, PROCESS_LOCAL, 4925 bytes) taskResourceAssignments Map() | |
| 21/09/18 11:51:42 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 3) (172.25.87.59, executor 0, partition 1, PROCESS_LOCAL, 4925 bytes) taskResourceAssignments Map() | |
| 21/09/18 11:51:42 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on 172.25.87.59:44483 (size: 7.6 KiB, free: 366.2 MiB) | |
| 21/09/18 11:51:43 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on 172.25.87.59:44483 (size: 24.3 KiB, free: 366.2 MiB) | |
| 21/09/18 11:51:43 WARN TaskSetManager: Lost task 0.0 in stage 2.0 (TID 2) (172.25.87.59 executor 0): java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.sql.sedona_sql.package$ | |
| at org.apache.spark.sql.sedona_sql.expressions.ST_GeomFromWKT.eval(Constructors.scala:131) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) | |
| at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) | |
| at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) | |
| at scala.collection.Iterator.foreach(Iterator.scala:941) | |
| at scala.collection.Iterator.foreach$(Iterator.scala:941) | |
| at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) | |
| at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:162) | |
| at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:160) | |
| at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1429) | |
| at scala.collection.TraversableOnce.aggregate(TraversableOnce.scala:219) | |
| at scala.collection.TraversableOnce.aggregate$(TraversableOnce.scala:219) | |
| at scala.collection.AbstractIterator.aggregate(Iterator.scala:1429) | |
| at org.apache.spark.rdd.RDD.$anonfun$aggregate$2(RDD.scala:1207) | |
| at org.apache.spark.SparkContext.$anonfun$runJob$6(SparkContext.scala:2290) | |
| at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) | |
| at org.apache.spark.scheduler.Task.run(Task.scala:131) | |
| at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497) | |
| at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439) | |
| at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) | |
| at java.lang.Thread.run(Thread.java:748) | |
| 21/09/18 11:51:43 WARN TaskSetManager: Lost task 1.0 in stage 2.0 (TID 3) (172.25.87.59 executor 0): java.lang.ExceptionInInitializerError | |
| at org.apache.spark.sql.sedona_sql.expressions.ST_GeomFromWKT.eval(Constructors.scala:131) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) | |
| at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) | |
| at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) | |
| at scala.collection.Iterator.foreach(Iterator.scala:941) | |
| at scala.collection.Iterator.foreach$(Iterator.scala:941) | |
| at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) | |
| at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:162) | |
| at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:160) | |
| at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1429) | |
| at scala.collection.TraversableOnce.aggregate(TraversableOnce.scala:219) | |
| at scala.collection.TraversableOnce.aggregate$(TraversableOnce.scala:219) | |
| at scala.collection.AbstractIterator.aggregate(Iterator.scala:1429) | |
| at org.apache.spark.rdd.RDD.$anonfun$aggregate$2(RDD.scala:1207) | |
| at org.apache.spark.SparkContext.$anonfun$runJob$6(SparkContext.scala:2290) | |
| at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) | |
| at org.apache.spark.scheduler.Task.run(Task.scala:131) | |
| at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497) | |
| at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439) | |
| at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Caused by: org.apache.spark.SparkException: There is no active SparkContext. Hence, cannot create SedonaSerializer | |
| at org.apache.spark.sql.sedona_sql.package$.<init>(package.scala:31) | |
| at org.apache.spark.sql.sedona_sql.package$.<clinit>(package.scala) | |
| ... 24 more | |
| 21/09/18 11:51:43 INFO TaskSetManager: Starting task 1.1 in stage 2.0 (TID 4) (172.25.87.59, executor 0, partition 1, PROCESS_LOCAL, 4925 bytes) taskResourceAssignments Map() | |
| 21/09/18 11:51:43 INFO TaskSetManager: Starting task 0.1 in stage 2.0 (TID 5) (172.25.87.59, executor 0, partition 0, PROCESS_LOCAL, 4925 bytes) taskResourceAssignments Map() | |
| 21/09/18 11:51:43 INFO TaskSetManager: Lost task 0.1 in stage 2.0 (TID 5) on 172.25.87.59, executor 0: java.lang.NoClassDefFoundError (Could not initialize class org.apache.spark.sql.sedona_sql.package$) [duplicate 1] | |
| 21/09/18 11:51:43 INFO TaskSetManager: Starting task 0.2 in stage 2.0 (TID 6) (172.25.87.59, executor 0, partition 0, PROCESS_LOCAL, 4925 bytes) taskResourceAssignments Map() | |
| 21/09/18 11:51:43 INFO TaskSetManager: Lost task 1.1 in stage 2.0 (TID 4) on 172.25.87.59, executor 0: java.lang.NoClassDefFoundError (Could not initialize class org.apache.spark.sql.sedona_sql.package$) [duplicate 2] | |
| 21/09/18 11:51:43 INFO TaskSetManager: Starting task 1.2 in stage 2.0 (TID 7) (172.25.87.59, executor 0, partition 1, PROCESS_LOCAL, 4925 bytes) taskResourceAssignments Map() | |
| 21/09/18 11:51:43 INFO TaskSetManager: Lost task 0.2 in stage 2.0 (TID 6) on 172.25.87.59, executor 0: java.lang.NoClassDefFoundError (Could not initialize class org.apache.spark.sql.sedona_sql.package$) [duplicate 3] | |
| 21/09/18 11:51:43 INFO TaskSetManager: Starting task 0.3 in stage 2.0 (TID 8) (172.25.87.59, executor 0, partition 0, PROCESS_LOCAL, 4925 bytes) taskResourceAssignments Map() | |
| 21/09/18 11:51:43 INFO TaskSetManager: Lost task 1.2 in stage 2.0 (TID 7) on 172.25.87.59, executor 0: java.lang.NoClassDefFoundError (Could not initialize class org.apache.spark.sql.sedona_sql.package$) [duplicate 4] | |
| 21/09/18 11:51:43 INFO TaskSetManager: Starting task 1.3 in stage 2.0 (TID 9) (172.25.87.59, executor 0, partition 1, PROCESS_LOCAL, 4925 bytes) taskResourceAssignments Map() | |
| 21/09/18 11:51:43 INFO TaskSetManager: Lost task 0.3 in stage 2.0 (TID 8) on 172.25.87.59, executor 0: java.lang.NoClassDefFoundError (Could not initialize class org.apache.spark.sql.sedona_sql.package$) [duplicate 5] | |
| 21/09/18 11:51:43 ERROR TaskSetManager: Task 0 in stage 2.0 failed 4 times; aborting job | |
| 21/09/18 11:51:43 INFO TaskSchedulerImpl: Cancelling stage 2 | |
| 21/09/18 11:51:43 INFO TaskSchedulerImpl: Killing all running tasks in stage 2: Stage cancelled | |
| 21/09/18 11:51:43 INFO TaskSchedulerImpl: Stage 2 was cancelled | |
| 21/09/18 11:51:43 INFO DAGScheduler: ResultStage 2 (aggregate at SpatialRDD.java:481) failed in 0.967 s due to Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 8) (172.25.87.59 executor 0): java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.sql.sedona_sql.package$ | |
| at org.apache.spark.sql.sedona_sql.expressions.ST_GeomFromWKT.eval(Constructors.scala:131) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) | |
| at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) | |
| at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) | |
| at scala.collection.Iterator.foreach(Iterator.scala:941) | |
| at scala.collection.Iterator.foreach$(Iterator.scala:941) | |
| at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) | |
| at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:162) | |
| at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:160) | |
| at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1429) | |
| at scala.collection.TraversableOnce.aggregate(TraversableOnce.scala:219) | |
| at scala.collection.TraversableOnce.aggregate$(TraversableOnce.scala:219) | |
| at scala.collection.AbstractIterator.aggregate(Iterator.scala:1429) | |
| at org.apache.spark.rdd.RDD.$anonfun$aggregate$2(RDD.scala:1207) | |
| at org.apache.spark.SparkContext.$anonfun$runJob$6(SparkContext.scala:2290) | |
| at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) | |
| at org.apache.spark.scheduler.Task.run(Task.scala:131) | |
| at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497) | |
| at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439) | |
| at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Driver stacktrace: | |
| 21/09/18 11:51:43 INFO DAGScheduler: Job 2 failed: aggregate at SpatialRDD.java:481, took 0.973420 s | |
| 21/09/18 11:51:43 INFO TaskSetManager: Lost task 1.3 in stage 2.0 (TID 9) on 172.25.87.59, executor 0: java.lang.NoClassDefFoundError (Could not initialize class org.apache.spark.sql.sedona_sql.package$) [duplicate 6] | |
| 21/09/18 11:51:43 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool | |
| 21/09/18 11:51:43 INFO SparkUI: Stopped Spark web UI at http://kontinuation:4040 | |
| 21/09/18 11:51:43 INFO StandaloneSchedulerBackend: Shutting down all executors | |
| 21/09/18 11:51:43 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down | |
| 21/09/18 11:51:43 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! | |
| 21/09/18 11:51:43 INFO MemoryStore: MemoryStore cleared | |
| 21/09/18 11:51:43 INFO BlockManager: BlockManager stopped | |
| 21/09/18 11:51:43 INFO BlockManagerMaster: BlockManagerMaster stopped | |
| 21/09/18 11:51:43 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! | |
| 21/09/18 11:51:43 INFO SparkContext: Successfully stopped SparkContext | |
| Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 8) (172.25.87.59 executor 0): java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.sql.sedona_sql.package$ | |
| at org.apache.spark.sql.sedona_sql.expressions.ST_GeomFromWKT.eval(Constructors.scala:131) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) | |
| at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) | |
| at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) | |
| at scala.collection.Iterator.foreach(Iterator.scala:941) | |
| at scala.collection.Iterator.foreach$(Iterator.scala:941) | |
| at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) | |
| at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:162) | |
| at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:160) | |
| at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1429) | |
| at scala.collection.TraversableOnce.aggregate(TraversableOnce.scala:219) | |
| at scala.collection.TraversableOnce.aggregate$(TraversableOnce.scala:219) | |
| at scala.collection.AbstractIterator.aggregate(Iterator.scala:1429) | |
| at org.apache.spark.rdd.RDD.$anonfun$aggregate$2(RDD.scala:1207) | |
| at org.apache.spark.SparkContext.$anonfun$runJob$6(SparkContext.scala:2290) | |
| at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) | |
| at org.apache.spark.scheduler.Task.run(Task.scala:131) | |
| at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497) | |
| at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439) | |
| at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) | |
| at java.lang.Thread.run(Thread.java:748) | |
| Driver stacktrace: | |
| at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2258) | |
| at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2207) | |
| at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2206) | |
| at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) | |
| at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) | |
| at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) | |
| at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2206) | |
| at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1079) | |
| at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1079) | |
| at scala.Option.foreach(Option.scala:407) | |
| at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1079) | |
| at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2445) | |
| at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2387) | |
| at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2376) | |
| at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) | |
| at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:868) | |
| at org.apache.spark.SparkContext.runJob(SparkContext.scala:2196) | |
| at org.apache.spark.SparkContext.runJob(SparkContext.scala:2291) | |
| at org.apache.spark.rdd.RDD.$anonfun$aggregate$1(RDD.scala:1209) | |
| at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) | |
| at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) | |
| at org.apache.spark.rdd.RDD.withScope(RDD.scala:414) | |
| at org.apache.spark.rdd.RDD.aggregate(RDD.scala:1202) | |
| at org.apache.spark.api.java.JavaRDDLike.aggregate(JavaRDDLike.scala:427) | |
| at org.apache.spark.api.java.JavaRDDLike.aggregate$(JavaRDDLike.scala:425) | |
| at org.apache.spark.api.java.AbstractJavaRDDLike.aggregate(JavaRDDLike.scala:45) | |
| at org.apache.sedona.core.spatialRDD.SpatialRDD.analyze(SpatialRDD.java:481) | |
| at org.apache.spark.sql.sedona_sql.strategy.join.TraitJoinQueryExec.doExecute(TraitJoinQueryExec.scala:67) | |
| at org.apache.spark.sql.sedona_sql.strategy.join.TraitJoinQueryExec.doExecute$(TraitJoinQueryExec.scala:52) | |
| at org.apache.spark.sql.sedona_sql.strategy.join.RangeJoinExec.doExecute(RangeJoinExec.scala:37) | |
| at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180) | |
| at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218) | |
| at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) | |
| at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215) | |
| at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176) | |
| at org.apache.spark.sql.execution.InputAdapter.inputRDD(WholeStageCodegenExec.scala:525) | |
| at org.apache.spark.sql.execution.InputRDDCodegen.inputRDDs(WholeStageCodegenExec.scala:453) | |
| at org.apache.spark.sql.execution.InputRDDCodegen.inputRDDs$(WholeStageCodegenExec.scala:452) | |
| at org.apache.spark.sql.execution.InputAdapter.inputRDDs(WholeStageCodegenExec.scala:496) | |
| at org.apache.spark.sql.execution.ProjectExec.inputRDDs(basicPhysicalOperators.scala:50) | |
| at org.apache.spark.sql.execution.aggregate.HashAggregateExec.inputRDDs(HashAggregateExec.scala:141) | |
| at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:746) | |
| at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180) | |
| at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218) | |
| at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) | |
| at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215) | |
| at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176) | |
| at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.inputRDD$lzycompute(ShuffleExchangeExec.scala:118) | |
| at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.inputRDD(ShuffleExchangeExec.scala:118) | |
| at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.shuffleDependency$lzycompute(ShuffleExchangeExec.scala:151) | |
| at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.shuffleDependency(ShuffleExchangeExec.scala:149) | |
| at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.$anonfun$doExecute$1(ShuffleExchangeExec.scala:166) | |
| at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52) | |
| at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.doExecute(ShuffleExchangeExec.scala:163) | |
| at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180) | |
| at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218) | |
| at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) | |
| at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215) | |
| at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176) | |
| at org.apache.spark.sql.execution.InputAdapter.inputRDD(WholeStageCodegenExec.scala:525) | |
| at org.apache.spark.sql.execution.InputRDDCodegen.inputRDDs(WholeStageCodegenExec.scala:453) | |
| at org.apache.spark.sql.execution.InputRDDCodegen.inputRDDs$(WholeStageCodegenExec.scala:452) | |
| at org.apache.spark.sql.execution.InputAdapter.inputRDDs(WholeStageCodegenExec.scala:496) | |
| at org.apache.spark.sql.execution.aggregate.HashAggregateExec.inputRDDs(HashAggregateExec.scala:141) | |
| at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:746) | |
| at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180) | |
| at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218) | |
| at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) | |
| at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215) | |
| at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176) | |
| at org.apache.spark.sql.execution.TakeOrderedAndProjectExec.executeCollect(limit.scala:187) | |
| at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3696) | |
| at org.apache.spark.sql.Dataset.$anonfun$head$1(Dataset.scala:2722) | |
| at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687) | |
| at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103) | |
| at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163) | |
| at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90) | |
| at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) | |
| at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) | |
| at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3685) | |
| at org.apache.spark.sql.Dataset.head(Dataset.scala:2722) | |
| at org.apache.spark.sql.Dataset.take(Dataset.scala:2929) | |
| at org.apache.spark.sql.Dataset.getRows(Dataset.scala:301) | |
| at org.apache.spark.sql.Dataset.showString(Dataset.scala:338) | |
| at org.apache.spark.sql.Dataset.show(Dataset.scala:825) | |
| at org.apache.spark.sql.Dataset.show(Dataset.scala:784) | |
| at org.apache.spark.sql.Dataset.show(Dataset.scala:793) | |
| at ExperiSedona$.main(ExperiSedona.scala:73) | |
| at ExperiSedona.main(ExperiSedona.scala) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) | |
| at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951) | |
| at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) | |
| at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) | |
| at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) | |
| at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039) | |
| at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048) | |
| at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) | |
| Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.sql.sedona_sql.package$ | |
| at org.apache.spark.sql.sedona_sql.expressions.ST_GeomFromWKT.eval(Constructors.scala:131) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) | |
| at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) | |
| at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) | |
| at scala.collection.Iterator.foreach(Iterator.scala:941) | |
| at scala.collection.Iterator.foreach$(Iterator.scala:941) | |
| at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) | |
| at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:162) | |
| at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:160) | |
| at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1429) | |
| at scala.collection.TraversableOnce.aggregate(TraversableOnce.scala:219) | |
| at scala.collection.TraversableOnce.aggregate$(TraversableOnce.scala:219) | |
| at scala.collection.AbstractIterator.aggregate(Iterator.scala:1429) | |
| at org.apache.spark.rdd.RDD.$anonfun$aggregate$2(RDD.scala:1207) | |
| at org.apache.spark.SparkContext.$anonfun$runJob$6(SparkContext.scala:2290) | |
| at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) | |
| at org.apache.spark.scheduler.Task.run(Task.scala:131) | |
| at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497) | |
| at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439) | |
| at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) | |
| at java.lang.Thread.run(Thread.java:748) | |
| 21/09/18 11:51:43 INFO ShutdownHookManager: Shutdown hook called | |
| 21/09/18 11:51:43 INFO ShutdownHookManager: Deleting directory /tmp/spark-a846237b-773e-40e1-828b-76f646b1f247 | |
| 21/09/18 11:51:43 INFO ShutdownHookManager: Deleting directory /tmp/spark-330d3500-8f92-463c-bd01-d27af27f4b71 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment