Hashing
- class category_encoders.hashing.HashingEncoder(max_process=0, max_sample=0, verbose=0, n_components=8, cols=None, drop_invariant=False, return_df=True, hash_method='md5')[source]
A multivariate hashing implementation with configurable dimensionality/precision.
The advantage of this encoder is that it does not maintain a dictionary of observed categories. Consequently, the encoder does not grow in size and accepts new values during data scoring by design.
It’s important to read about how max_process & max_sample work before setting them manually, inappropriate setting slows down encoding.
Default value of ‘max_process’ is 1 on Windows because multiprocessing might cause issues, see in : https://github.com/scikit-learn-contrib/categorical-encoding/issues/215 https://docs.python.org/2/library/multiprocessing.html?highlight=process#windows
- Parameters
- verbose: int
integer indicating verbosity of the output. 0 for none.
- cols: list
a list of columns to encode, if None, all string columns will be encoded.
- drop_invariant: bool
boolean for whether or not to drop columns with 0 variance.
- return_df: bool
boolean for whether to return a pandas DataFrame from transform (otherwise it will be a numpy array).
- hash_method: str
which hashing method to use. Any method from hashlib works.
- max_process: int
how many processes to use in transform(). Limited in range(1, 64). By default, it uses half of the logical CPUs. For example, 4C4T makes max_process=2, 4C8T makes max_process=4. Set it larger if you have a strong CPU. It is not recommended to set it larger than is the count of the logical CPUs as it will actually slow down the encoding.
- max_sample: int
how many samples to encode by each process at a time. This setting is useful on low memory machines. By default, max_sample=(all samples num)/(max_process). For example, 4C8T CPU with 100,000 samples makes max_sample=25,000, 6C12T CPU with 100,000 samples makes max_sample=16,666. It is not recommended to set it larger than the default value.
- n_components: int
how many bits to use to represent the feature. By default we use 8 bits. For high-cardinality features, consider using up-to 32 bits.
References
- 1
Feature Hashing for Large Scale Multitask Learning, from
https://alex.smola.org/papers/2009/Weinbergeretal09.pdf .. [R8dde675226a2-2] Don’t be tricked by the Hashing Trick, from https://booking.ai/dont-be-tricked-by-the-hashing-trick-192a6aae3087
- Attributes
- feature_names
Methods
fit
(X[, y])Fits the encoder according to X and y.
fit_transform
(X[, y])Fit to data, then transform it.
Returns the names of all transformed / added columns.
get_params
([deep])Get parameters for this estimator.
hashing_trick
(X_in[, hashing_method, N, ...])A basic hashing implementation with configurable dimensionality/precision
set_params
(**params)Set the parameters of this estimator.
transform
(X[, override_return_df])Perform the transformation to new categorical data.
require_data
- Parameters
- verbose: int
integer indicating verbosity of output. 0 for none.
- cols: list
a list of columns to encode, if None, all string and categorical columns will be encoded.
- drop_invariant: bool
boolean for whether or not to drop columns with 0 variance.
- return_df: bool
boolean for whether to return a pandas DataFrame from transform and inverse transform (otherwise it will be a numpy array).
- handle_missing: str
how to handle missing values at fit time. Options are ‘error’, ‘return_nan’, and ‘value’. Default ‘value’, which treat NaNs as a countable category at fit time.
- handle_unknown: str, int or dict of {columnoption, …}.
how to handle unknown labels at transform time. Options are ‘error’ ‘return_nan’, ‘value’ and int. Defaults to None which uses NaN behaviour specified at fit time. Passing an int will fill with this int value.
- kwargs: dict.
additional encoder specific parameters like regularisation.
- Attributes
- feature_names
Methods
fit
(X[, y])Fits the encoder according to X and y.
fit_transform
(X[, y])Fit to data, then transform it.
Returns the names of all transformed / added columns.
get_params
([deep])Get parameters for this estimator.
hashing_trick
(X_in[, hashing_method, N, ...])A basic hashing implementation with configurable dimensionality/precision
set_params
(**params)Set the parameters of this estimator.
transform
(X[, override_return_df])Perform the transformation to new categorical data.
require_data
- fit(X, y=None, **kwargs)
Fits the encoder according to X and y.
- Parameters
- Xarray-like, shape = [n_samples, n_features]
Training vectors, where n_samples is the number of samples and n_features is the number of features.
- yarray-like, shape = [n_samples]
Target values.
- Returns
- selfencoder
Returns self.
- fit_transform(X, y=None, **fit_params)
Fit to data, then transform it.
Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.
- Parameters
- Xarray-like of shape (n_samples, n_features)
Input samples.
- yarray-like of shape (n_samples,) or (n_samples, n_outputs), default=None
Target values (None for unsupervised transformations).
- **fit_paramsdict
Additional fit parameters.
- Returns
- X_newndarray array of shape (n_samples, n_features_new)
Transformed array.
- get_feature_names() List[str]
Returns the names of all transformed / added columns.
- Returns
- feature_names: list
A list with all feature names transformed or added. Note: potentially dropped features (because the feature is constant/invariant) are not included!
- get_params(deep=True)
Get parameters for this estimator.
- Parameters
- deepbool, default=True
If True, will return the parameters for this estimator and contained subobjects that are estimators.
- Returns
- paramsdict
Parameter names mapped to their values.
- static hashing_trick(X_in, hashing_method='md5', N=2, cols=None, make_copy=False)[source]
A basic hashing implementation with configurable dimensionality/precision
Performs the hashing trick on a pandas dataframe, X, using the hashing method from hashlib identified by hashing_method. The number of output dimensions (N), and columns to hash (cols) are also configurable.
- Parameters
- X_in: pandas dataframe
description text
- hashing_method: string, optional
description text
- N: int, optional
description text
- cols: list, optional
description text
- make_copy: bool, optional
description text
- Returns
- outdataframe
A hashing encoded dataframe.
References
Cite the relevant literature, e.g. [R6b702480991a-1]. You may also cite these references in the notes section above. .. [R6b702480991a-1] Kilian Weinberger; Anirban Dasgupta; John Langford; Alex Smola; Josh Attenberg (2009). Feature Hashing for Large Scale Multitask Learning. Proc. ICML.
- set_params(**params)
Set the parameters of this estimator.
The method works on simple estimators as well as on nested objects (such as
Pipeline
). The latter have parameters of the form<component>__<parameter>
so that it’s possible to update each component of a nested object.- Parameters
- **paramsdict
Estimator parameters.
- Returns
- selfestimator instance
Estimator instance.
- transform(X, override_return_df=False)
Perform the transformation to new categorical data.
- Parameters
- Xarray-like, shape = [n_samples, n_features]
- override_return_dfbool
override self.return_df to force to return a data frame
- Returns
- parray or DataFrame, shape = [n_samples, n_features_out]
Transformed values with encoding applied.