org.apache.spark.rdd

PairRDDFunctions

class PairRDDFunctions[K, V] extends Logging with SparkHadoopMapReduceUtil with Serializable

Extra functions available on RDDs of (key, value) pairs through an implicit conversion. Import org.apache.spark.SparkContext._ at the top of your program to use these functions.

Linear Supertypes
Serializable, Serializable, SparkHadoopMapReduceUtil, Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. PairRDDFunctions
  2. Serializable
  3. Serializable
  4. SparkHadoopMapReduceUtil
  5. Logging
  6. AnyRef
  7. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new PairRDDFunctions(self: RDD[(K, V)])(implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null)

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  7. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  8. def cogroup[W1, W2](other1: RDD[(K, W1)], other2: RDD[(K, W2)], numPartitions: Int): RDD[(K, (Iterable[V], Iterable[W1], Iterable[W2]))]

    For each key k in this or other1 or other2, return a resulting RDD that contains a tuple with the list of values for that key in this, other1 and other2.

  9. def cogroup[W](other: RDD[(K, W)], numPartitions: Int): RDD[(K, (Iterable[V], Iterable[W]))]

    For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.

  10. def cogroup[W1, W2](other1: RDD[(K, W1)], other2: RDD[(K, W2)]): RDD[(K, (Iterable[V], Iterable[W1], Iterable[W2]))]

    For each key k in this or other1 or other2, return a resulting RDD that contains a tuple with the list of values for that key in this, other1 and other2.

  11. def cogroup[W](other: RDD[(K, W)]): RDD[(K, (Iterable[V], Iterable[W]))]

    For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.

  12. def cogroup[W1, W2](other1: RDD[(K, W1)], other2: RDD[(K, W2)], partitioner: Partitioner): RDD[(K, (Iterable[V], Iterable[W1], Iterable[W2]))]

    For each key k in this or other1 or other2, return a resulting RDD that contains a tuple with the list of values for that key in this, other1 and other2.

  13. def cogroup[W](other: RDD[(K, W)], partitioner: Partitioner): RDD[(K, (Iterable[V], Iterable[W]))]

    For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.

  14. def collectAsMap(): Map[K, V]

    Return the key-value pairs in this RDD to the master as a Map.

  15. def combineByKey[C](createCombiner: (V) ⇒ C, mergeValue: (C, V) ⇒ C, mergeCombiners: (C, C) ⇒ C): RDD[(K, C)]

    Simplified version of combineByKey that hash-partitions the resulting RDD using the existing partitioner/parallelism level.

  16. def combineByKey[C](createCombiner: (V) ⇒ C, mergeValue: (C, V) ⇒ C, mergeCombiners: (C, C) ⇒ C, numPartitions: Int): RDD[(K, C)]

    Simplified version of combineByKey that hash-partitions the output RDD.

  17. def combineByKey[C](createCombiner: (V) ⇒ C, mergeValue: (C, V) ⇒ C, mergeCombiners: (C, C) ⇒ C, partitioner: Partitioner, mapSideCombine: Boolean = true, serializer: Serializer = null): RDD[(K, C)]

    Generic function to combine the elements for each key using a custom set of aggregation functions.

    Generic function to combine the elements for each key using a custom set of aggregation functions. Turns an RDD[(K, V)] into a result of type RDD[(K, C)], for a "combined type" C Note that V and C can be different -- for example, one might group an RDD of type (Int, Int) into an RDD of type (Int, Seq[Int]). Users provide three functions:

    - createCombiner, which turns a V into a C (e.g., creates a one-element list) - mergeValue, to merge a V into a C (e.g., adds it to the end of a list) - mergeCombiners, to combine two C's into a single one.

    In addition, users can control the partitioning of the output RDD, and whether to perform map-side aggregation (if a mapper can produce multiple items with the same key).

  18. def countApproxDistinctByKey(relativeSD: Double = 0.05): RDD[(K, Long)]

    Return approximate number of distinct values for each key this RDD.

    Return approximate number of distinct values for each key this RDD. The accuracy of approximation can be controlled through the relative standard deviation (relativeSD) parameter, which also controls the amount of memory used. Lower values result in more accurate counts but increase the memory footprint and vice versa. The default value of relativeSD is 0.05. Hash-partitions the output RDD using the existing partitioner/parallelism level.

  19. def countApproxDistinctByKey(relativeSD: Double, numPartitions: Int): RDD[(K, Long)]

    Return approximate number of distinct values for each key in this RDD.

    Return approximate number of distinct values for each key in this RDD. The accuracy of approximation can be controlled through the relative standard deviation (relativeSD) parameter, which also controls the amount of memory used. Lower values result in more accurate counts but increase the memory footprint and vice versa. HashPartitions the output RDD into numPartitions.

  20. def countApproxDistinctByKey(relativeSD: Double, partitioner: Partitioner): RDD[(K, Long)]

    Return approximate number of distinct values for each key in this RDD.

    Return approximate number of distinct values for each key in this RDD. The accuracy of approximation can be controlled through the relative standard deviation (relativeSD) parameter, which also controls the amount of memory used. Lower values result in more accurate counts but increase the memory footprint and vice versa. Uses the provided Partitioner to partition the output RDD.

  21. def countByKey(): Map[K, Long]

    Count the number of elements for each key, and return the result to the master as a Map.

  22. def countByKeyApprox(timeout: Long, confidence: Double = 0.95): PartialResult[Map[K, BoundedDouble]]

    :: Experimental :: Approximate version of countByKey that can return a partial result if it does not finish within a timeout.

    :: Experimental :: Approximate version of countByKey that can return a partial result if it does not finish within a timeout.

    Annotations
    @Experimental()

  23. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  24. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  25. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  26. def flatMapValues[U](f: (V) ⇒ TraversableOnce[U]): RDD[(K, U)]

    Pass each value in the key-value pair RDD through a flatMap function without changing the keys; this also retains the original RDD's partitioning.

  27. def foldByKey(zeroValue: V)(func: (V, V) ⇒ V): RDD[(K, V)]

    Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.

    Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.g., Nil for list concatenation, 0 for addition, or 1 for multiplication.).

  28. def foldByKey(zeroValue: V, numPartitions: Int)(func: (V, V) ⇒ V): RDD[(K, V)]

    Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.

    Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.g., Nil for list concatenation, 0 for addition, or 1 for multiplication.).

  29. def foldByKey(zeroValue: V, partitioner: Partitioner)(func: (V, V) ⇒ V): RDD[(K, V)]

    Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.

    Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.g., Nil for list concatenation, 0 for addition, or 1 for multiplication.).

  30. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  31. def groupByKey(): RDD[(K, Iterable[V])]

    Group the values for each key in the RDD into a single sequence.

    Group the values for each key in the RDD into a single sequence. Hash-partitions the resulting RDD with the existing partitioner/parallelism level.

    Note: If you are grouping in order to perform an aggregation (such as a sum or average) over each key, using PairRDDFunctions.reduceByKey or PairRDDFunctions.combineByKey will provide much better performance,

  32. def groupByKey(numPartitions: Int): RDD[(K, Iterable[V])]

    Group the values for each key in the RDD into a single sequence.

    Group the values for each key in the RDD into a single sequence. Hash-partitions the resulting RDD with into numPartitions partitions.

    Note: If you are grouping in order to perform an aggregation (such as a sum or average) over each key, using PairRDDFunctions.reduceByKey or PairRDDFunctions.combineByKey will provide much better performance.

  33. def groupByKey(partitioner: Partitioner): RDD[(K, Iterable[V])]

    Group the values for each key in the RDD into a single sequence.

    Group the values for each key in the RDD into a single sequence. Allows controlling the partitioning of the resulting key-value pair RDD by passing a Partitioner.

    Note: If you are grouping in order to perform an aggregation (such as a sum or average) over each key, using PairRDDFunctions.reduceByKey or PairRDDFunctions.combineByKey will provide much better performance.

  34. def groupWith[W1, W2](other1: RDD[(K, W1)], other2: RDD[(K, W2)]): RDD[(K, (Iterable[V], Iterable[W1], Iterable[W2]))]

    Alias for cogroup.

  35. def groupWith[W](other: RDD[(K, W)]): RDD[(K, (Iterable[V], Iterable[W]))]

    Alias for cogroup.

  36. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  37. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  38. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging

  39. def join[W](other: RDD[(K, W)], numPartitions: Int): RDD[(K, (V, W))]

    Return an RDD containing all pairs of elements with matching keys in this and other.

    Return an RDD containing all pairs of elements with matching keys in this and other. Each pair of elements will be returned as a (k, (v1, v2)) tuple, where (k, v1) is in this and (k, v2) is in other. Performs a hash join across the cluster.

  40. def join[W](other: RDD[(K, W)]): RDD[(K, (V, W))]

    Return an RDD containing all pairs of elements with matching keys in this and other.

    Return an RDD containing all pairs of elements with matching keys in this and other. Each pair of elements will be returned as a (k, (v1, v2)) tuple, where (k, v1) is in this and (k, v2) is in other. Performs a hash join across the cluster.

  41. def join[W](other: RDD[(K, W)], partitioner: Partitioner): RDD[(K, (V, W))]

    Return an RDD containing all pairs of elements with matching keys in this and other.

    Return an RDD containing all pairs of elements with matching keys in this and other. Each pair of elements will be returned as a (k, (v1, v2)) tuple, where (k, v1) is in this and (k, v2) is in other. Uses the given Partitioner to partition the output RDD.

  42. def keys: RDD[K]

    Return an RDD with the keys of each tuple.

  43. def leftOuterJoin[W](other: RDD[(K, W)], numPartitions: Int): RDD[(K, (V, Option[W]))]

    Perform a left outer join of this and other.

    Perform a left outer join of this and other. For each element (k, v) in this, the resulting RDD will either contain all pairs (k, (v, Some(w))) for w in other, or the pair (k, (v, None)) if no elements in other have key k. Hash-partitions the output into numPartitions partitions.

  44. def leftOuterJoin[W](other: RDD[(K, W)]): RDD[(K, (V, Option[W]))]

    Perform a left outer join of this and other.

    Perform a left outer join of this and other. For each element (k, v) in this, the resulting RDD will either contain all pairs (k, (v, Some(w))) for w in other, or the pair (k, (v, None)) if no elements in other have key k. Hash-partitions the output using the existing partitioner/parallelism level.

  45. def leftOuterJoin[W](other: RDD[(K, W)], partitioner: Partitioner): RDD[(K, (V, Option[W]))]

    Perform a left outer join of this and other.

    Perform a left outer join of this and other. For each element (k, v) in this, the resulting RDD will either contain all pairs (k, (v, Some(w))) for w in other, or the pair (k, (v, None)) if no elements in other have key k. Uses the given Partitioner to partition the output RDD.

  46. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging

  47. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  48. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  49. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  50. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  51. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  52. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  53. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  54. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  55. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  56. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  57. def lookup(key: K): Seq[V]

    Return the list of values in the RDD for key key.

    Return the list of values in the RDD for key key. This operation is done efficiently if the RDD has a known partitioner by only searching the partition that the key maps to.

  58. def mapValues[U](f: (V) ⇒ U): RDD[(K, U)]

    Pass each value in the key-value pair RDD through a map function without changing the keys; this also retains the original RDD's partitioning.

  59. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  60. def newJobContext(conf: Configuration, jobId: JobID): JobContext

    Definition Classes
    SparkHadoopMapReduceUtil

  61. def newTaskAttemptContext(conf: Configuration, attemptId: TaskAttemptID): TaskAttemptContext

    Definition Classes
    SparkHadoopMapReduceUtil

  62. def newTaskAttemptID(jtIdentifier: String, jobId: Int, isMap: Boolean, taskId: Int, attemptId: Int): TaskAttemptID

    Definition Classes
    SparkHadoopMapReduceUtil

  63. final def notify(): Unit

    Definition Classes
    AnyRef

  64. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  65. def partitionBy(partitioner: Partitioner): RDD[(K, V)]

    Return a copy of the RDD partitioned using the specified partitioner.

  66. def reduceByKey(func: (V, V) ⇒ V): RDD[(K, V)]

    Merge the values for each key using an associative reduce function.

    Merge the values for each key using an associative reduce function. This will also perform the merging locally on each mapper before sending results to a reducer, similarly to a "combiner" in MapReduce. Output will be hash-partitioned with the existing partitioner/ parallelism level.

  67. def reduceByKey(func: (V, V) ⇒ V, numPartitions: Int): RDD[(K, V)]

    Merge the values for each key using an associative reduce function.

    Merge the values for each key using an associative reduce function. This will also perform the merging locally on each mapper before sending results to a reducer, similarly to a "combiner" in MapReduce. Output will be hash-partitioned with numPartitions partitions.

  68. def reduceByKey(partitioner: Partitioner, func: (V, V) ⇒ V): RDD[(K, V)]

    Merge the values for each key using an associative reduce function.

    Merge the values for each key using an associative reduce function. This will also perform the merging locally on each mapper before sending results to a reducer, similarly to a "combiner" in MapReduce.

  69. def reduceByKeyLocally(func: (V, V) ⇒ V): Map[K, V]

    Merge the values for each key using an associative reduce function, but return the results immediately to the master as a Map.

    Merge the values for each key using an associative reduce function, but return the results immediately to the master as a Map. This will also perform the merging locally on each mapper before sending results to a reducer, similarly to a "combiner" in MapReduce.

  70. def rightOuterJoin[W](other: RDD[(K, W)], numPartitions: Int): RDD[(K, (Option[V], W))]

    Perform a right outer join of this and other.

    Perform a right outer join of this and other. For each element (k, w) in other, the resulting RDD will either contain all pairs (k, (Some(v), w)) for v in this, or the pair (k, (None, w)) if no elements in this have key k. Hash-partitions the resulting RDD into the given number of partitions.

  71. def rightOuterJoin[W](other: RDD[(K, W)]): RDD[(K, (Option[V], W))]

    Perform a right outer join of this and other.

    Perform a right outer join of this and other. For each element (k, w) in other, the resulting RDD will either contain all pairs (k, (Some(v), w)) for v in this, or the pair (k, (None, w)) if no elements in this have key k. Hash-partitions the resulting RDD using the existing partitioner/parallelism level.

  72. def rightOuterJoin[W](other: RDD[(K, W)], partitioner: Partitioner): RDD[(K, (Option[V], W))]

    Perform a right outer join of this and other.

    Perform a right outer join of this and other. For each element (k, w) in other, the resulting RDD will either contain all pairs (k, (Some(v), w)) for v in this, or the pair (k, (None, w)) if no elements in this have key k. Uses the given Partitioner to partition the output RDD.

  73. def saveAsHadoopDataset(conf: JobConf): Unit

    Output the RDD to any Hadoop-supported storage system, using a Hadoop JobConf object for that storage system.

    Output the RDD to any Hadoop-supported storage system, using a Hadoop JobConf object for that storage system. The JobConf should set an OutputFormat and any output paths required (e.g. a table name to write to) in the same way as it would be configured for a Hadoop MapReduce job.

  74. def saveAsHadoopFile(path: String, keyClass: Class[_], valueClass: Class[_], outputFormatClass: Class[_ <: OutputFormat[_, _]], conf: JobConf = ..., codec: Option[Class[_ <: CompressionCodec]] = None): Unit

    Output the RDD to any Hadoop-supported file system, using a Hadoop OutputFormat class supporting the key and value types K and V in this RDD.

  75. def saveAsHadoopFile(path: String, keyClass: Class[_], valueClass: Class[_], outputFormatClass: Class[_ <: OutputFormat[_, _]], codec: Class[_ <: CompressionCodec]): Unit

    Output the RDD to any Hadoop-supported file system, using a Hadoop OutputFormat class supporting the key and value types K and V in this RDD.

    Output the RDD to any Hadoop-supported file system, using a Hadoop OutputFormat class supporting the key and value types K and V in this RDD. Compress with the supplied codec.

  76. def saveAsHadoopFile[F <: OutputFormat[K, V]](path: String, codec: Class[_ <: CompressionCodec])(implicit fm: ClassTag[F]): Unit

    Output the RDD to any Hadoop-supported file system, using a Hadoop OutputFormat class supporting the key and value types K and V in this RDD.

    Output the RDD to any Hadoop-supported file system, using a Hadoop OutputFormat class supporting the key and value types K and V in this RDD. Compress the result with the supplied codec.

  77. def saveAsHadoopFile[F <: OutputFormat[K, V]](path: String)(implicit fm: ClassTag[F]): Unit

    Output the RDD to any Hadoop-supported file system, using a Hadoop OutputFormat class supporting the key and value types K and V in this RDD.

  78. def saveAsNewAPIHadoopDataset(conf: Configuration): Unit

    Output the RDD to any Hadoop-supported storage system with new Hadoop API, using a Hadoop Configuration object for that storage system.

    Output the RDD to any Hadoop-supported storage system with new Hadoop API, using a Hadoop Configuration object for that storage system. The Conf should set an OutputFormat and any output paths required (e.g. a table name to write to) in the same way as it would be configured for a Hadoop MapReduce job.

  79. def saveAsNewAPIHadoopFile(path: String, keyClass: Class[_], valueClass: Class[_], outputFormatClass: Class[_ <: OutputFormat[_, _]], conf: Configuration = self.context.hadoopConfiguration): Unit

    Output the RDD to any Hadoop-supported file system, using a new Hadoop API OutputFormat (mapreduce.

    Output the RDD to any Hadoop-supported file system, using a new Hadoop API OutputFormat (mapreduce.OutputFormat) object supporting the key and value types K and V in this RDD.

  80. def saveAsNewAPIHadoopFile[F <: OutputFormat[K, V]](path: String)(implicit fm: ClassTag[F]): Unit

    Output the RDD to any Hadoop-supported file system, using a new Hadoop API OutputFormat (mapreduce.

    Output the RDD to any Hadoop-supported file system, using a new Hadoop API OutputFormat (mapreduce.OutputFormat) object supporting the key and value types K and V in this RDD.

  81. def subtractByKey[W](other: RDD[(K, W)], p: Partitioner)(implicit arg0: ClassTag[W]): RDD[(K, V)]

    Return an RDD with the pairs from this whose keys are not in other.

  82. def subtractByKey[W](other: RDD[(K, W)], numPartitions: Int)(implicit arg0: ClassTag[W]): RDD[(K, V)]

    Return an RDD with the pairs from this whose keys are not in other.

  83. def subtractByKey[W](other: RDD[(K, W)])(implicit arg0: ClassTag[W]): RDD[(K, V)]

    Return an RDD with the pairs from this whose keys are not in other.

    Return an RDD with the pairs from this whose keys are not in other.

    Uses this partitioner/partition size, because even if other is huge, the resulting RDD will be <= us.

  84. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  85. def toString(): String

    Definition Classes
    AnyRef → Any

  86. def values: RDD[V]

    Return an RDD with the values of each tuple.

  87. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  88. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  89. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def reduceByKeyToDriver(func: (V, V) ⇒ V): Map[K, V]

    Alias for reduceByKeyLocally

    Alias for reduceByKeyLocally

    Annotations
    @deprecated
    Deprecated

    (Since version 1.0.0) Use reduceByKeyLocally

Inherited from Serializable

Inherited from Serializable

Inherited from SparkHadoopMapReduceUtil

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped