QuantileDiscretizer (Spark 2.1.0 JavaDoc)

Object
- org.apache.spark.ml.PipelineStage
- - org.apache.spark.ml.Estimator<Bucketizer>
  - - org.apache.spark.ml.feature.QuantileDiscretizer

All Implemented Interfaces:

java.io.Serializable, Params, DefaultParamsWritable, Identifiable, MLWritable
```
public final class QuantileDiscretizer
extends Estimator<Bucketizer>
implements DefaultParamsWritable
```
QuantileDiscretizer takes a column with continuous features and outputs a column with binned categorical features. The number of bins can be set using the numBuckets parameter. It is possible that the number of buckets used will be smaller than this value, for example, if there are too few distinct values of the input to create enough distinct quantiles.
NaN handling: NaN values will be removed from the column during QuantileDiscretizer fitting. This will produce a Bucketizer model for making predictions. During the transformation, Bucketizer will raise an error when it finds NaN values in the dataset, but the user can also choose to either keep or remove NaN values within the dataset by setting handleInvalid. If the user chooses to keep NaN values, they will be handled specially and placed into their own bucket, for example, if 4 buckets are used, then non-NaN data will be put into buckets[0-3], but NaNs will be counted in a special bucket[4].
Algorithm: The bin ranges are chosen using an approximate algorithm (see the documentation for org.apache.spark.sql.DataFrameStatFunctions.approxQuantile for a detailed description). The precision of the approximation can be controlled with the relativeError parameter. The lower and upper bin bounds will be -Infinity and +Infinity, covering all real values.

See Also:
Serialized Form

Constructor Summary

Constructors
Constructor and Description

QuantileDiscretizer()

QuantileDiscretizer(String uid)

Constructors
Constructor and Description
`QuantileDiscretizer()`
`QuantileDiscretizer(String uid)`

Method Summary

Methods
Modifier and Type	Method and Description
`static Params`	`clear(Param<?> param)`
`QuantileDiscretizer`	`copy(ParamMap extra)` Creates a copy of this instance with the same UID and some extra params.
`static String`	`explainParam(Param<?> param)`
`static String`	`explainParams()`
`static ParamMap`	`extractParamMap()`
`static ParamMap`	`extractParamMap(ParamMap extra)`
`Bucketizer`	`fit(Dataset<?> dataset)` Fits a model to the input data.
`static <T> scala.Option<T>`	`get(Param<T> param)`
`static <T> scala.Option<T>`	`getDefault(Param<T> param)`
`static String`	`getHandleInvalid()`
`String`	`getHandleInvalid()`
`static String`	`getInputCol()`
`static int`	`getNumBuckets()`
`int`	`getNumBuckets()`
`static <T> T`	`getOrDefault(Param<T> param)`
`static String`	`getOutputCol()`
`static Param<Object>`	`getParam(String paramName)`
`static double`	`getRelativeError()`
`double`	`getRelativeError()`
`static Param<String>`	`handleInvalid()`
`Param<String>`	`handleInvalid()` Param for how to handle invalid entries.
`static <T> boolean`	`hasDefault(Param<T> param)`
`static boolean`	`hasParam(String paramName)`
`static Param<String>`	`inputCol()`
`static boolean`	`isDefined(Param<?> param)`
`static boolean`	`isSet(Param<?> param)`
`static QuantileDiscretizer`	`load(String path)`
`static IntParam`	`numBuckets()`
`IntParam`	`numBuckets()` Number of buckets (quantiles, or categories) into which data points are grouped.
`static Param<String>`	`outputCol()`
`static Param<?>[]`	`params()`
`static DoubleParam`	`relativeError()`
`DoubleParam`	`relativeError()` Relative error (see documentation for `org.apache.spark.sql.DataFrameStatFunctions.approxQuantile` for description) Must be in the range [0, 1].
`static void`	`save(String path)`
`static <T> Params`	`set(Param<T> param, T value)`
`QuantileDiscretizer`	`setHandleInvalid(String value)`
`QuantileDiscretizer`	`setInputCol(String value)`
`QuantileDiscretizer`	`setNumBuckets(int value)`
`QuantileDiscretizer`	`setOutputCol(String value)`
`QuantileDiscretizer`	`setRelativeError(double value)`
`static String`	`toString()`
`StructType`	`transformSchema(StructType schema)` :: DeveloperApi ::
`String`	`uid()` An immutable unique ID for the object and its derivatives.
`static MLWriter`	`write()`

Methods inherited from class org.apache.spark.ml.Estimator
fit, fit, fit, fit

Methods inherited from class Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.param.Params
clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn

Methods inherited from interface org.apache.spark.ml.util.Identifiable
toString

Methods inherited from interface org.apache.spark.ml.util.DefaultParamsWritable
write

Methods inherited from interface org.apache.spark.ml.util.MLWritable
save

- Constructor Detail
  - QuantileDiscretizer
```
public QuantileDiscretizer(String uid)
```
  - QuantileDiscretizer
```
public QuantileDiscretizer()
```
- Method Detail
  - load
```
public static QuantileDiscretizer load(String path)
```
  - toString
```
public static String toString()
```
  - params
```
public static Param<?>[] params()
```
  - explainParam
```
public static String explainParam(Param<?> param)
```
  - explainParams
```
public static String explainParams()
```
  - isSet
```
public static final boolean isSet(Param<?> param)
```
  - isDefined
```
public static final boolean isDefined(Param<?> param)
```
  - hasParam
```
public static boolean hasParam(String paramName)
```
  - getParam
```
public static Param<Object> getParam(String paramName)
```
  - set
```
public static final <T> Params set(Param<T> param,
             T value)
```
  - get
```
public static final <T> scala.Option<T> get(Param<T> param)
```
  - clear
```
public static final Params clear(Param<?> param)
```
  - getOrDefault
```
public static final <T> T getOrDefault(Param<T> param)
```
  - getDefault
```
public static final <T> scala.Option<T> getDefault(Param<T> param)
```
  - hasDefault
```
public static final <T> boolean hasDefault(Param<T> param)
```
  - extractParamMap
```
public static final ParamMap extractParamMap(ParamMap extra)
```
  - extractParamMap
```
public static final ParamMap extractParamMap()
```
  - inputCol
```
public static final Param<String> inputCol()
```
  - getInputCol
```
public static final String getInputCol()
```
  - outputCol
```
public static final Param<String> outputCol()
```
  - getOutputCol
```
public static final String getOutputCol()
```
  - numBuckets
```
public static IntParam numBuckets()
```
  - getNumBuckets
```
public static int getNumBuckets()
```
  - relativeError
```
public static DoubleParam relativeError()
```
  - getRelativeError
```
public static double getRelativeError()
```
  - handleInvalid
```
public static Param<String> handleInvalid()
```
  - getHandleInvalid
```
public static String getHandleInvalid()
```
  - save
```
public static void save(String path)
                 throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - write
```
public static MLWriter write()
```
  - uid
```
public String uid()
```
    Description copied from interface: Identifiable
    
    An immutable unique ID for the object and its derivatives.
    
    Specified by:
    
    uid in interface Identifiable
    
    Returns:
    (undocumented)
  - setRelativeError
```
public QuantileDiscretizer setRelativeError(double value)
```
  - setNumBuckets
```
public QuantileDiscretizer setNumBuckets(int value)
```
  - setInputCol
```
public QuantileDiscretizer setInputCol(String value)
```
  - setOutputCol
```
public QuantileDiscretizer setOutputCol(String value)
```
  - setHandleInvalid
```
public QuantileDiscretizer setHandleInvalid(String value)
```
  - transformSchema
```
public StructType transformSchema(StructType schema)
```
    Description copied from class: PipelineStage
    
    :: DeveloperApi ::
    Check transform validity and derive the output schema from the input schema.
    We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().
    Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
    
    Specified by:
    
    transformSchema in class PipelineStage
    
    Parameters:
    schema - (undocumented)
    
    Returns:
    (undocumented)
  - fit
```
public Bucketizer fit(Dataset<?> dataset)
```
    Description copied from class: Estimator
    
    Fits a model to the input data.
    
    Specified by:
    
    fit in class Estimator<Bucketizer>
    
    Parameters:
    dataset - (undocumented)
    
    Returns:
    (undocumented)
  - copy
```
public QuantileDiscretizer copy(ParamMap extra)
```
    Description copied from interface: Params
    
    Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().
    
    Specified by:
    
    copy in interface Params
    
    Specified by:
    
    copy in class Estimator<Bucketizer>
    
    Parameters:
    extra - (undocumented)
    
    Returns:
    (undocumented)
  - numBuckets
```
public IntParam numBuckets()
```
    Number of buckets (quantiles, or categories) into which data points are grouped. Must be greater than or equal to 2.
    See also handleInvalid, which can optionally create an additional bucket for NaN values.
    default: 2
    
    Returns:
    (undocumented)
  - getNumBuckets
```
public int getNumBuckets()
```
  - relativeError
```
public DoubleParam relativeError()
```
    Relative error (see documentation for org.apache.spark.sql.DataFrameStatFunctions.approxQuantile for description) Must be in the range [0, 1]. default: 0.001
    
    Returns:
    (undocumented)
  - getRelativeError
```
public double getRelativeError()
```
  - handleInvalid
```
public Param<String> handleInvalid()
```
    Param for how to handle invalid entries. Options are 'skip' (filter out rows with invalid values), 'error' (throw an error), or 'keep' (keep invalid values in a special additional bucket). Default: "error"
    
    Returns:
    (undocumented)
  - getHandleInvalid
```
public String getHandleInvalid()
```

Class QuantileDiscretizer

Constructor Summary

Method Summary

Methods inherited from class org.apache.spark.ml.Estimator

Methods inherited from class Object

Methods inherited from interface org.apache.spark.ml.param.Params

Methods inherited from interface org.apache.spark.ml.util.Identifiable

Methods inherited from interface org.apache.spark.ml.util.DefaultParamsWritable

Methods inherited from interface org.apache.spark.ml.util.MLWritable

Constructor Detail

QuantileDiscretizer

QuantileDiscretizer

Method Detail

load

toString

params

explainParam

explainParams

isSet

isDefined

hasParam

getParam

set

get

clear

getOrDefault

getDefault

hasDefault

extractParamMap

extractParamMap

inputCol

getInputCol

outputCol

getOutputCol

numBuckets

getNumBuckets

relativeError

getRelativeError

handleInvalid

getHandleInvalid

save

write

uid

setRelativeError

setNumBuckets

setInputCol

setOutputCol

setHandleInvalid

transformSchema

fit

copy

numBuckets

getNumBuckets

relativeError

getRelativeError

handleInvalid

getHandleInvalid