org.apache.spark.sql.sources

Class HadoopFsRelation

java.lang.Object

org.apache.spark.sql.sources.BaseRelation

org.apache.spark.sql.sources.HadoopFsRelation

All Implemented Interfaces:

Logging, org.apache.spark.sql.execution.FileRelation

public abstract class HadoopFsRelation
extends BaseRelation
implements org.apache.spark.sql.execution.FileRelation, Logging

::Experimental:: A BaseRelation that provides much of the common code required for relations that store their data to an HDFS compatible filesystem.

For the read path, similar to PrunedFilteredScan, it can eliminate unneeded columns and filter using selected predicates before producing an RDD containing all matching tuples as Row objects. In addition, when reading from Hive style partitioned tables stored in file systems, it's able to discover partitioning information from the paths of input directories, and perform partition pruning before start reading the data. Subclasses of HadoopFsRelation() must override one of the four buildScan methods to implement the read path.

For the write path, it provides the ability to write to both non-partitioned and partitioned tables. Directory layout of the partitioned tables is compatible with Hive.

Since:

1.4.0

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HadoopFsRelation.FakeFileStatus
static class	HadoopFsRelation.FakeFileStatus$

Constructor Summary

Constructors

Constructor and Description
HadoopFsRelation()
HadoopFsRelation(scala.collection.immutable.Map<java.lang.String,java.lang.String> parameters)

Method Summary

Methods

Modifier and Type	Method and Description
RDD<Row>	buildScan(org.apache.hadoop.fs.FileStatus[] inputFiles) For a non-partitioned relation, this method builds an RDD[Row] containing all rows within this relation.
RDD<Row>	buildScan(java.lang.String[] requiredColumns, org.apache.hadoop.fs.FileStatus[] inputFiles) For a non-partitioned relation, this method builds an RDD[Row] containing all rows within this relation.
RDD<Row>	buildScan(java.lang.String[] requiredColumns, Filter[] filters, org.apache.hadoop.fs.FileStatus[] inputFiles) For a non-partitioned relation, this method builds an RDD[Row] containing all rows within this relation.
protected scala.collection.mutable.LinkedHashSet<org.apache.hadoop.fs.FileStatus>	cachedLeafStatuses()
abstract StructType	dataSchema() Specifies schema of actual data files.
java.lang.String[]	inputFiles()
static org.apache.hadoop.fs.FileStatus[]	listLeafFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus status)
static scala.collection.mutable.LinkedHashSet<org.apache.hadoop.fs.FileStatus>	listLeafFilesInParallel(java.lang.String[] paths, org.apache.hadoop.conf.Configuration hadoopConf, SparkContext sparkContext)
StructType	partitionColumns() Partition columns.
abstract java.lang.String[]	paths() Paths of this relation.
abstract OutputWriterFactory	prepareJobForWrite(org.apache.hadoop.mapreduce.Job job) Prepares a write job and returns an OutputWriterFactory.
StructType	schema() Schema of this relation.
static boolean	shouldFilterOut(java.lang.String pathName) Checks if we should filter out this path name.
long	sizeInBytes() Returns an estimated size of this relation in bytes.
java.lang.String	toString()
scala.Option<StructType>	userDefinedPartitionColumns() Optional user defined partition columns.

Methods inherited from class org.apache.spark.sql.sources.BaseRelation

needConversion, sqlContext, unhandledFilters

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

HadoopFsRelation

public HadoopFsRelation()

HadoopFsRelation

public HadoopFsRelation(scala.collection.immutable.Map<java.lang.String,java.lang.String> parameters)

Method Detail

shouldFilterOut

public static boolean shouldFilterOut(java.lang.String pathName)

Checks if we should filter out this path name.

listLeafFiles

public static org.apache.hadoop.fs.FileStatus[] listLeafFiles(org.apache.hadoop.fs.FileSystem fs,
                                              org.apache.hadoop.fs.FileStatus status)

listLeafFilesInParallel

public static scala.collection.mutable.LinkedHashSet<org.apache.hadoop.fs.FileStatus> listLeafFilesInParallel(java.lang.String[] paths,
                                                                                              org.apache.hadoop.conf.Configuration hadoopConf,
                                                                                              SparkContext sparkContext)

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

cachedLeafStatuses

protected scala.collection.mutable.LinkedHashSet<org.apache.hadoop.fs.FileStatus> cachedLeafStatuses()

paths

public abstract java.lang.String[] paths()

Paths of this relation. For partitioned relations, it should be root directories of all partition directories.

Returns:

(undocumented)

Since:

1.4.0

inputFiles

public java.lang.String[] inputFiles()

Specified by:

inputFiles in interface org.apache.spark.sql.execution.FileRelation

sizeInBytes

public long sizeInBytes()

Description copied from class: BaseRelation

Returns an estimated size of this relation in bytes. This information is used by the planner to decide when it is safe to broadcast a relation and can be overridden by sources that know the size ahead of time. By default, the system will assume that tables are too large to broadcast. This method will be called multiple times during query planning and thus should not perform expensive operations for each invocation.

Note that it is always better to overestimate size than underestimate, because underestimation could lead to execution plans that are suboptimal (i.e. broadcasting a very large table).

Overrides:

sizeInBytes in class BaseRelation

Returns:

(undocumented)

partitionColumns

public final StructType partitionColumns()

Partition columns. Can be either defined by userDefinedPartitionColumns or automatically discovered. Note that they should always be nullable.

Returns:

(undocumented)

Since:

1.4.0

userDefinedPartitionColumns

public scala.Option<StructType> userDefinedPartitionColumns()

Optional user defined partition columns.

Returns:

(undocumented)

Since:

1.4.0

schema

public StructType schema()

Schema of this relation. It consists of columns appearing in dataSchema and all partition columns not appearing in dataSchema.

Specified by:

schema in class BaseRelation

Returns:

(undocumented)

Since:

1.4.0

dataSchema

public abstract StructType dataSchema()

Specifies schema of actual data files. For partitioned relations, if one or more partitioned columns are contained in the data files, they should also appear in dataSchema.

Returns:

(undocumented)

Since:

1.4.0

buildScan

public RDD<Row> buildScan(org.apache.hadoop.fs.FileStatus[] inputFiles)

For a non-partitioned relation, this method builds an RDD[Row] containing all rows within this relation. For partitioned relations, this method is called for each selected partition, and builds an RDD[Row] containing all rows within that single partition.

Parameters:

inputFiles - For a non-partitioned relation, it contains paths of all data files in the relation. For a partitioned relation, it contains paths of all data files in a single selected partition.

Returns:

(undocumented)

Since:

1.4.0

buildScan

public RDD<Row> buildScan(java.lang.String[] requiredColumns,
                 org.apache.hadoop.fs.FileStatus[] inputFiles)

For a non-partitioned relation, this method builds an RDD[Row] containing all rows within this relation. For partitioned relations, this method is called for each selected partition, and builds an RDD[Row] containing all rows within that single partition.

Parameters:

requiredColumns - Required columns.

inputFiles - For a non-partitioned relation, it contains paths of all data files in the relation. For a partitioned relation, it contains paths of all data files in a single selected partition.

Returns:

(undocumented)

Since:

1.4.0

buildScan

public RDD<Row> buildScan(java.lang.String[] requiredColumns,
                 Filter[] filters,
                 org.apache.hadoop.fs.FileStatus[] inputFiles)

For a non-partitioned relation, this method builds an RDD[Row] containing all rows within this relation. For partitioned relations, this method is called for each selected partition, and builds an RDD[Row] containing all rows within that single partition.

Parameters:

requiredColumns - Required columns.

filters - Candidate filters to be pushed down. The actual filter should be the conjunction of all filters. The pushed down filters are currently purely an optimization as they will all be evaluated again. This means it is safe to use them with methods that produce false positives such as filtering partitions based on a bloom filter.

inputFiles - For a non-partitioned relation, it contains paths of all data files in the relation. For a partitioned relation, it contains paths of all data files in a single selected partition.

Returns:

(undocumented)

Since:

1.4.0

prepareJobForWrite

public abstract OutputWriterFactory prepareJobForWrite(org.apache.hadoop.mapreduce.Job job)

Prepares a write job and returns an OutputWriterFactory. Client side job preparation can be put here. For example, user defined output committer can be configured here by setting the output committer class in the conf of spark.sql.sources.outputCommitterClass.

Note that the only side effect expected here is mutating job via its setters. Especially, Spark SQL caches BaseRelation instances for performance, mutating relation internal states may cause unexpected behaviors.

Parameters:

job - (undocumented)

Returns:

(undocumented)

Since:

1.4.0