Spark DataFrame基础操作
创建SparkSession和SparkContext
val spark = SparkSession.builder.master("local").getOrCreate()
val sc = spark.sparkContext
从数组创建DataFrame
spark.range(1000).toDF("number").show()
指定Schema创建DataFrame
val data = Seq( Row("A", 10, 112233), Row("B", 20, 223311), Row("C", 30, 331122)) val schema = StructType(List( StructField("name", StringType), StructField("age", IntegerType), StructField("phone", IntegerType))) spark.createDataFrame(sc.makeRDD(data), schema).show()
从JSON文件加载DataFrame
/* data.json {"name":"A","age":10,"phone":112233} {"name":"B", "age":20,"phone":223311} {"name":"C", "age":30,"phone":331122} */ spark.read.format("json").load("/Users/tobe/temp2/data.json").show()
从CSV文件加载DataFrame
/* data.csv name,age,phone A,10,112233 B,20,223311 C,30,331122 */ spark.read.option("header", true).csv("/Users/tobe/temp2/data.csv").show()
读取MySQL数据库加载DataFrame
/* data.csv name,age,phone A,10,112233 B,20,223311 C,30,331122 */ spark.read.option("header", true).csv("/Users/tobe/temp2/data.csv").show()
RDD转DataFrame
/* data.csv name,age,phone A,10,112233 B,20,223311 C,30,331122 */ spark.read.option("header", true).csv("/Users/tobe/temp2/data.csv").show()
创建Timestamp数据
Spark的TimestampType类型与Java的java.sql.Timestamp对应，
/* data.csv name,age,phone A,10,112233 B,20,223311 C,30,331122 */ spark.read.option("header", true).csv("/Users/tobe/temp2/data.csv").show()
创建DateType数据
Spark的DateType类型与Java的java.sql.Date对应，
/* data.csv name,age,phone A,10,112233 B,20,223311 C,30,331122 */ spark.read.option("header", true).csv("/Users/tobe/temp2/data.csv").show()
版权声明：本文内容由互联网用户自发贡献，该文观点仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请联系我们举报，一经查实，本站将立刻删除。
发布者：全栈程序员-站长，转载请注明出处：https://javaforall.net/119645.html原文链接：https://javaforall.net
