Created
September 1, 2015 02:08
-
-
Save nrshrivatsan/ef47064eca68b790c4af to your computer and use it in GitHub Desktop.
A simple hack to load CSV contents into apache spark dataframes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| //Start spark using https://github.com/databricks/spark-csv#spark-compiled-with-scala-211 | |
| //$SPARK_HOME/bin/spark-shell --packages com.databricks:spark-csv_2.11:1.2.0 | |
| import org.apache.spark.sql.SQLContext | |
| val sqlContext = new SQLContext(sc) | |
| //Download Google Stock info CSV from Quandl | |
| val df = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").load("YAHOO-GOOG.csv") | |
| df.columns |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment