pyspark.sql.HiveContext

class pyspark.sql.HiveContext(sparkContext, sparkSession=None, jhiveContext=None)

A variant of Spark SQL that integrates with data stored in Hive.

Configuration for Hive is read from hive-site.xml on the classpath. It supports running both SQL and HiveQL commands.

Deprecated since version 2.0.0: Use SparkSession.builder.enableHiveSupport().getOrCreate().

Parameters:

sparkContextSparkContext

The SparkContext to wrap.

jhiveContextoptional

An optional JVM Scala HiveContext. If set, we do not instantiate a new HiveContext in the JVM, instead we make all calls to this object. This is only for internal use.

Methods

cacheTable(tableName)	Caches the specified table in-memory.
clearCache()	Removes all cached tables from the in-memory cache.
createDataFrame(data[, schema, ...])	Creates a DataFrame from an RDD, a list, a pandas.DataFrame, or a pyarrow.Table.
createExternalTable(tableName[, path, ...])	Creates an external table based on the dataset in a data source.
dropTempTable(tableName)	Remove the temporary table from catalog.
getConf(key[, defaultValue])	Returns the value of Spark SQL configuration property for the given key.
getOrCreate([sc])	Get the existing SQLContext or create a new one with given SparkContext.
newSession()	Returns a new SQLContext as new session, that has separate SQLConf, registered temporary views and UDFs, but shared SparkContext and table cache.
range(start[, end, step, numPartitions])	Create a DataFrame with single pyspark.sql.types.LongType column named id, containing elements in a range from start to end (exclusive) with step value step.
refreshTable(tableName)	Invalidate and refresh all the cached metadata of the given table.
registerDataFrameAsTable(df, tableName)	Registers the given DataFrame as a temporary table in the catalog.
registerFunction(name, f[, returnType])	An alias for spark.udf.register().
registerJavaFunction(name, javaClassName[, ...])	An alias for spark.udf.registerJavaFunction().
setConf(key, value)	Sets the given Spark SQL configuration property.
sql(sqlQuery)	Returns a DataFrame representing the result of the given query.
table(tableName)	Returns the specified table or view as a DataFrame.
tableNames([dbName])	Returns a list of names of tables in the database dbName.
tables([dbName])	Returns a DataFrame containing names of tables in the given database.
uncacheTable(tableName)	Removes the specified table from the in-memory cache.

Attributes

read	Returns a DataFrameReader that can be used to read data in as a DataFrame.
readStream	Returns a DataStreamReader that can be used to read data streams as a streaming DataFrame.
streams	Returns a StreamingQueryManager that allows managing all the StreamingQuery StreamingQueries active on this context.
udf	Returns a UDFRegistration for UDF registration.
udtf	Returns a UDTFRegistration for UDTF registration.