"Automated machine learning (AutoML) takes a higher-level approach to machine learning than most practitioners are used to, so we've gathered a handful of guidelines on what to expect when running AutoML software such as TPOT.\n",
"\n",
"#### AUTOML ALGORITHMS AREN'T INTENDED TO RUN FOR ONLY A FEW MINUTES\n",
"Of course, you can run TPOT for only a few minutes, and it will find a reasonably good pipeline for your dataset. However, if you don't run TPOT for long enough, it may not find the best possible pipeline for your dataset. It may not even find any suitable pipeline at all, in which case a RuntimeError('A pipeline has not yet been optimized. Please call fit() first.') will be raised. Often it is worthwhile to run multiple instances of TPOT in parallel for a long time (hours to days) to allow TPOT to thoroughly search the pipeline space for your dataset.\n",
"AutoML algorithms aren't as simple as fitting one model on the dataset; they consider multiple machine learning algorithms (random forests, linear models, SVMs, etc.) in a pipeline with multiple preprocessing steps (missing value imputation, scaling, PCA, feature selection, etc.), the hyperparameters for all of the models and preprocessing steps, and multiple ways to ensemble or stack the algorithms within the pipeline.\n",
"As such, TPOT will take a while to run on larger datasets, but it's important to realize why. With the default TPOT settings (100 generations with 100 population size), TPOT will evaluate 10,000 pipeline configurations before finishing. To put this number into context, think about a grid search of 10,000 hyperparameter combinations for a machine learning algorithm and how long that grid search will take. That is 10,000 model configurations to evaluate with 10-fold cross-validation, which means that roughly 100,000 models are fit and evaluated on the training data in one grid search. That's a time-consuming procedure, even for simpler models like decision trees.\n",
"Typical TPOT runs will take hours to days to finish (unless it's a small dataset), but you can always interrupt the run partway through and see the best results so far. TPOT also provides a warm_start and a periodic_checkpoint_folder parameter that lets you restart a TPOT run from where it left off.\n",
"If you're working with a reasonably complex dataset or run TPOT for a short amount of time, different TPOT runs may result in different pipeline recommendations. TPOT's optimization algorithm is stochastic, which means that it uses randomness (in part) to search the possible pipeline space. When two TPOT runs recommend different pipelines, this means that the TPOT runs didn't converge due to lack of time or that multiple pipelines perform more-or-less the same on your dataset.\n",
"This is actually an advantage over fixed grid search techniques: TPOT is meant to be an assistant that gives you ideas on how to solve a particular machine learning problem by exploring pipeline configurations that you might have never considered, then leaves the fine-tuning to more constrained parameter tuning techniques such as grid search or bayesian optimization."
"Matplotlib is building the font cache; this may take a moment.\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n"
"It's also possible to use TPOT for regression problems with the TPOTRegressor class. Other than the class name, a TPOTRegressor is used the same way as a TPOTClassifier. You can read more about the TPOTClassifier and TPOTRegressor classes in the API documentation."
"1. `scorers`: Scorers are functions that have the signature (estimator, X_test, y_test) and take in estimators that are expected to be fitted to training data. These can be produced with the [sklearn.metrics.make_scorer](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html) function. This function is used to evaluate the test folds during cross validation (defined in the `cv` parameter). These are passed into TPOT via the scorers parameter. This can take in the scorer itself or the string corresponding to a scoring function ([as listed here](https://scikit-learn.org/stable/modules/model_evaluation.html)). TPOT also supports passing in a list of several scorers for multi-objective optimization. For each fold of CV, TPOT only fits the estimator once, then evaluates all provided scorers in a loop.\n",
"2. `other_objective_functions` : Other objective functions in TPOT have the signature (estimator) and returns a float or list of floats. These get passed a single unfitted estimator once, outside of cross validation. The user may choose to fit the pipeline within this objective function as well.\n",
"Each scorer and objective function must be accompanied by a list of weights corresponding to the list of objectives, these are `scorers_weights` and `other_objective_function_weights`, respectively. By default, TPOT maximizes objective functions (this can be changed by `bigger_is_better=False`). Positive weights means that TPOT will seek to maximize that objective, and negative weights correspond to minimization. For most selectors (and the default), only the sign matters. The scale of the weight may matter if using a custom selection function for the optimization algorithm. A zero weight means that the score will not have an impact on the selection algorithm.\n",
"TPOT will always automatically name the scorers based on the function name for the columns in the final results dataframe. TPOT will use the function name as the column name for `other_objective_functions`. However, if you would like to specify custom column names, you can set the `objective_function_names` to be a list of names (str) for each value returned by the function in `other_objective_functions`. This can be useful if your additional functions return more than one value per function.\n",
"It is possible to have either the scorer or other_objective_function to return multiple values. In that case, just make sure that the `scorers_weights` and `other_objective_function_weights` are the same length as the number of returned scores.\n",
"TPOT comes with a few additional built in objective functions you can use. The first table are objectives applied to fitted pipelines, and thus are passee into the `scorers` parameter. The second table are objective functions for the `other_objective_functions` param.\n",
"| tpot.objectives.complexity_scorer | Estimates the number of learned parameters across all classifiers and regressors in the pipelines. Additionally, currently transformers add 1 point and selectors add 0 points (since they don't affect the complexity of the \"final\" predictive pipeline.) |\n",
"| tpot.objectives.average_path_length | Computes the average shortest path from all nodes to the root/final estimator (only supported for GraphPipeline) |\n",
"| tpot.objectives.number_of_leaves_objective | Calculates the number of leaves (input nodes) in a GraphPipeline |\n",
"| tpot.objectives.number_of_nodes_objective | Calculates the number of nodes in a pipeline (whether it is an scikit-learn Pipeline, GraphPipeline, Feature Union, or the previous nested within each other) |"
"When running TPOT, including a secondary objective that measures model complexity can sometimes be beneficial. More complex models can yield higher performance, but this comes at the cost of interpretability. Simpler models may be more interpretable but often have lower predictive performance. Sometimes, however, vast increases in complexity only marginally improve predictive performance. There may be other simpler and more interpretable pipelines with marginal performance decreases that could be acceptable for the increased interpretability. However, these pipelines are often missed when optimizing purely for performance. By including both performance and complexity as objective functions, TPOT will attempt to optimize the best pipeline for all complexity levels simultaneously. After optimization, the user will be able to see the complexity vs performance tradeoff and decide which pipeline best suits their needs. \n",
"Two methods of measuring complexity to consider would be `tpot.objectives.number_of_nodes_objective` or `tpot.objectives.complexity_scorer`. The number of nodes objective simply calculates the number of steps within a pipeline. This is a simple metric, however it does not differentiate between the complexity of different model types. For example, a simple LogisticRegression counts the same as the much more complex XGBoost. The complexity scorer tries to estimate the number of learned parameters included in the classifiers and regressors of the pipeline. It is challenging and potentially subjective how to exactly quantify and compare complexity between different classes of models. However, this function provides a reasonable heuristic for the evolutionary algorithm that at least separates out qualitatively more or less complex algorithms from one another. While it may be hard to compare the relative complexities of LogisticRegression and XGBoost exactly, for example, both will always be on opposite ends of the complexity values returned by this function. This allows for pareto fronts with LogisticRegression on one side, and XGBoost on the other.\n",
"TPOT can be used to optimize hyperparameters, select models, and optimize pipelines of models including determining the sequence of steps. **Tutorial 2** goes into more detail on how to customize search spaces with custom hyperparameter ranges, model types, and possible pipeline configurations. TPOT also comes with a handful of default operators and parameter configurations that we believe work well for optimizing machine learning pipelines. Below is a list of the current built-in configurations that come with TPOT. These can be passed in as strings to the `search space` parameter of any of the TPOT estimators.\n",
"| linear | A linear pipeline with the structure of \"Selector->(transformers+Passthrough)->(classifiers/regressors+Passthrough)->final classifier/regressor.\" For both the transformer and inner estimator layers, TPOT may choose one or more transformers/classifiers, or it may choose none. The inner classifier/regressor layer is optional. |\n",
"| linear-light | Same search space as linear, but without the inner classifier/regressor layer and with a reduced set of faster running estimators. |\n",
"| graph | TPOT will optimize a pipeline in the shape of a directed acyclic graph. The nodes of the graph can include selectors, scalers, transformers, or classifiers/regressors (inner classifiers/regressors can optionally be not included). This will return a custom GraphPipeline rather than an sklearn Pipeline. More details in Tutorial 6. |\n",
"| mdr |TPOT will search over a series of feature selectors and Multifactor Dimensionality Reduction models to find a series of operators that maximize prediction accuracy. The TPOT MDR configuration is specialized for genome-wide association studies (GWAS), and is described in detail online here.\n",
"\n",
"Note that TPOT MDR may be slow to run because the feature selection routines are computationally expensive, especially on large datasets. |\n",
"The `linear` and `graph` configurations by default allow for additional stacked classifiers/regressors within the pipeline in addition to the final classifier/regressor. If you would like to disable this, you can manually get the search space without inner classifier/regressors through the function `tpot.config.template_search_spaces.get_template_search_spaces` with `inner_predictios=False`. You can pass the resulting search space into the `search space` param. "
"The specific hyperparameter ranges used by TPOT can be found in files in the tpot/config folder. The template search spaces listed above are defined in tpot/config/template_search_spaces.py. Search spaces for individual models can be acquired in the tpot/config/get_configspace.py file (`tpot.config.get_search_space`). More details on customizing search spaces can be found in Tutorial 2.\n",
" Returns a search space which can be optimized by TPOT.\n",
"\n",
" Parameters\n",
" ----------\n",
" search_space: str or SearchSpace\n",
" The default search space to use. If a string, it should be one of the following:\n",
" - 'linear': A search space for linear pipelines\n",
" - 'linear-light': A search space for linear pipelines with a smaller, faster search space\n",
" - 'graph': A search space for graph pipelines\n",
" - 'graph-light': A search space for graph pipelines with a smaller, faster search space\n",
" - 'mdr': A search space for MDR pipelines\n",
" If a SearchSpace object, it should be a valid search space object for TPOT.\n",
" \n",
" classification: bool, default=True\n",
" Whether the problem is a classification problem or a regression problem.\n",
"\n",
" inner_predictors: bool, default=None\n",
" Whether to include additional classifiers/regressors before the final classifier/regressor (allowing for ensembles). \n",
" Defaults to False for 'linear-light' and 'graph-light' search spaces, and True otherwise. (Not used for 'mdr' search space)\n",
" \n",
" cross_val_predict_cv: int, default=None\n",
" The number of folds to use for cross_val_predict. \n",
" Defaults to 0 for 'linear-light' and 'graph-light' search spaces, and 5 otherwise. (Not used for 'mdr' search space)\n",
"\n",
" get_search_space_params: dict\n",
" Additional parameters to pass to the get_search_space function.\n",
"\n",
"### cross_val_predict_cv\n",
"\n",
"Additionally, utilizing `cross_val_predict_cv` may increase performance when training models with inner classifiers/regressors. If this parameter is set, during model training any classifiers or regressors that is not the final predictor will use `sklearn.model_selection.cross_val_predict` to pass out of sample predictions into the following steps of the model. The model will still be fit to the full data which will be used for predictions after training. Training downstream models on out of sample predictions can often prevent overfitting and increase performance. The reason is that this gives downstream models a estimate of how upstream models compare on unseen data. Otherwise, if an upsteam model heavily overfits the data, downsteam models may simply learn to blindly trust the seemingly well-predicting model, propagating the over-fitting through to the end result.\n",
"\n",
"The downside is that cross_val_predict_cv is significantly more computationally demanding, and may not be necessary for your given dataset. \n"
"Note that we use a short time duration for a quick example, but in practice, you may need to run TPOT for a longer duration. By default, TPOT sets a time limit of 1 hour with a max limit of 5 minutes per pipeline. In practice, you may want to increase these values.\n",
"There are three methods of terminating a TPOT run and ending the optimization process. TPOT will terminate as soon as one of the conditions is met.\n",
"* `max_time_mins` : (Default, 60 minutes) After this many minutes, TPOT will terminate and return the best pipeline it found so far.\n",
"* `early_stop` : The number of generations without seeing an improvement in performance, after which TPOT terminates. Generally, a value of around 5 to 20 is sufficient to be reasonably sure that performance has converged.\n",
"* `generations`: The total number of generations of the evolutionary algorithm to run.\n",
"* When running tpot from an .py script, it is important to protect code with `if __name__==\"__main__\":` . This is because of how TPOT handles parallelization with Python and Dask."
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 0 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 1 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 2 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 3 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 4 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 5 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 6 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 7 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 8 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 9 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 10 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 11 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 12 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 13 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 14 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 15 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 16 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 17 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 18 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 19 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 20 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 21 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 22 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 23 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 24 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 25 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 26 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 27 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 28 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 29 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 30 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 31 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 32 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 33 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 34 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 35 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 36 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 37 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 38 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 39 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 40 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 41 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 42 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 43 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 44 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 45 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 46 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 47 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 48 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 49 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 50 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 51 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 52 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 53 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 54 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 55 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 56 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 57 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 58 are removed. Consider decreasing the number of bins.\n",
" warnings.warn(\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_discretization.py:307: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 59 are removed. Consider decreasing the number of bins.\n",
"Here we use a toy example dataset included in scikit-learn. We will use the `light` configuration and the `complexity_scorer` to estimate complexity.\n",
"Note, for this toy example, we set a relatively short run time. In practice, we would recommend running TPOT for a longer duration with an `early_stop` value of around 5 to 20 (more details below)."
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/neural_network/_multilayer_perceptron.py:690: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
"You can access the best pipeline selected by TPOT with the `fitted_pipeline_` attribute. This is the pipeline with the highest cross validation score (on the first scorer, or first objective function if no scorer is provided.)"
" n_iter_no_change=32))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\"> Pipeline<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.pipeline.Pipeline.html\">?<span>Documentation for Pipeline</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>Pipeline(steps=[('minmaxscaler', MinMaxScaler()),\n",
"The `evaluated_individuals` attribute of the tpot estimator object is a Pandas Dataframe containing information about a run. Each row corresponds to an individual pipeline explored by tpot. The dataframe contains the following columns:\n",
"| \\<n objective function columns\\> | The first set of columns will correspond to each objective function. These can either be automatically named by TPOT, or passed in by the user. |\n",
"| Parents | This contains a tuple that contains the indexes of the 'parents' of the current pipeline. For example, (29, 42) means that the pipelines in indexes 29 and 42 were utilized to generate that pipeline. |\n",
"| Variation_Function | The function applied to the parents to generate the new pipeline |\n",
"| Individual | The individual class that represents a specific pipeline and hyperparameter configuration. This class also contains functions for mutation and crossover. To get the sklearn estimator/pipeline object from the individual you can call the `export_pipeline()` function. (as in, `pipe = ind.export_pipeline()`) |\n",
"| Generation | The generation where the individual was created. (Note that the higher performing pipelines from previous generations may still be present in the current \"population\" of a given generation if selected.) |\n",
"| Submitted Timestamp | Timestamp, in seconds, at which the pipeline was sent to be evaluated. This is the output of time.time(), which is \"Return the time in seconds since the epoch as a floating-point number. \" |\n",
"| Completed Timestamp | Timestamp at which the pipeline evaluation completed in the same units as Submitted Timestamp |\n",
"| Pareto_Front\t | If you have multiple parameters, this column is True if the pipeline performance fall on the pareto front line. This is the set of pipelines with scores that are strictly better than pipelines not on the line, but not strictly better than one another. |\n",
"| Instance | This contains the unfitted pipeline evaluated for this row. (This is the pipeline returned by calling the export_pipeline() function of the individual class) |\n"
"Plotting the performance of multiple objectives in a scatterplot is a helpful way to visualize the tradeoff between model complexity and predictive performance. This is best visualized when plotting the Pareto front pipelines, which present the best-performing pipeline along the spectrum of complexity. Generally, higher complexity models may yield higher performance but be more difficult to interpret. "
" KNeighborsClassifier(n_jobs=1, n_neighbors=2))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-11\" type=\"checkbox\" ><label for=\"sk-estimator-id-11\" class=\"sk-toggleable__label sk-toggleable__label-arrow \"> Pipeline<a class=\"sk-estimator-doc-link \" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.pipeline.Pipeline.html\">?<span>Documentation for Pipeline</span></a><span class=\"sk-estimator-doc-link \">i<span>Not fitted</span></span></label><div class=\"sk-toggleable__content \"><pre>Pipeline(steps=[('maxabsscaler', MaxAbsScaler()),\n",
"Plotting performance over time is a good way to assess whether or not the TPOT model has converged. If performance asymptotes over time, there may not be much more performance to be gained by running for a longer period. If the plot looks like it is still improving, it may be worth running TPOT for a longer duration. \n",
"* If the `warm_start` parameter is set to True, subsequent calls to `fit` will continue training where it left off (The conventional scikit-learn default is to retrain from scratch on subsequent calls to fit). \n",
"* If `periodic_checkpoint_folder` is set, TPOT will periodically save its current state to disk. If TPOT is interrupted (job canceled, PC shut off, crashes), you can resume training from where it left off. The checkpoint folder stores a data frame of all evaluated pipelines. This data frame can be loaded and inspected to help diagnose problems when debugging.\n",
"\n",
"\n",
"**Note: TPOT does not clean up the checkpoint files. If the `periodic_checkpoint_folder` parameter is set, training from the last saved point will always continue, even if the input data has changed. A common issue is forgetting to change this folder between experiments and TPOT continuing training from pipelines optimized for another dataset. If you intend to start a run from scratch, you must either remove the parameter, supply an empty folder, or delete the original checkpoint folder.**"
"Here is a subset of the most common parameters to customize and what they do. See the docs for `TPOTEstimator` or `TPOTEstimatorSteadyState` full documentation of all parameters. \n",
"| memory | str, memory object | If supplied, pipeline will cache each transformer after calling fit with joblib.Memory. |\n",
"| periodic_checkpoint_folder | str | Folder to save the population to periodically. If None, no periodic saving will be done. If provided, training will resume from this checkpoint.|\n",
"On small datasets, it is not impossible for TPOT to overfit the cross-validation score itself. This can lead to lower-than-expected performance on held-out datasets. TPOT will always return the model with the highest CV score as its final fitted_pipeline. However, if the highest performing model, as evaluated by cross-validation, actually was just overfit to the CV score, it may actually be worse performing compared to other models on the Pareto front.\n",
" * Using a secondary complexity objective and evaluating the entire pareto front may be beneficial. In some cases a lower performing pipeline with lower complexity can actually perform better on held out sets. These can either be evaluated and compared on a held out validation set, or sometimes, if very data limited, simply using a different seed of splitting the CV folds can work as well.\n",
" * TPOT can do this automatically. The `validation_strategy` parameter can be set to re-test the final pareto front on either a held-out validation set (percent of data set by `validation_fraction`) or a different seed for splitting the CV folds. These can be selected by setting `validation_strategy` to \"split\" or \"reshuffled\", respectively.\n",
" * Increasing the number of folds of cross-validation can mitigate this. \n",
" * Nested cross-validation can also be used to estimate the performance of the TPOT optimization algorithm itself.\n",
" * Removing more complex methods from the search space can reduce the chances of overfitting"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Tips and tricks for speeding up TPOT\n",
"\n",
"TPOT can be a computationally demanding algorithm as it fits thousands of complex machine learning pipelines on potentially large datasets. There are several strategies available for improving run time by reducing the compute needed. \n",
"\n",
"There are three main strategies implemented in TPOT to reduce redundant work and/or prevent wasting compute on poorly performing pipelines.\n",
"\n",
"1. TPOT pipelines will often have the exact same components doing the exact same computation (e.g. the first steps of the pipeline remain the same and only the parameters of the final classifier changed.) In these cases, that The first strategy is to simply cache these repeat computations so that they only happen once. More info in the next subsection.\n",
"2. Successive Halving. This idea was first tested with TPOT by Parmentier et al. in [\"TPOT-SH: a Faster Optimization Algorithm to Solve the AutoML Problem on Large Datasets\"](https://www.researchgate.net/profile/Laurent-Parmentier-4/publication/339263193_TPOT-SH_A_Faster_Optimization_Algorithm_to_Solve_the_AutoML_Problem_on_Large_Datasets/links/5e5fd8b8a6fdccbeba1c6a56/TPOT-SH-A-Faster-Optimization-Algorithm-to-Solve-the-AutoML-Problem-on-Large-Datasets.pdf). The algorithm operates in two stages. Initially, it trains early generations using a small data subset and a large population size. Later generations then evaluate a smaller set of promising pipelines on larger, or even full, data portions. This approach rapidly identifies top-performing pipeline configurations through initial rough evaluations, followed by more comprehensive assessments. More information on this strategy in Tutorial 8.\n",
"3. Most often, we will be evaluating pipelines using cross validation. However, we can often tell within the first few folds whether or not the pipeline is going have a reasonable change of outperforming the previous best pipelines. For example, if the best score so far is .92 AUROC and the average score of the first five folds of our current pipeline is only around .61, we can be reasonably confident that the next five folds are unlikely to this pipeline ahead of the others. We can save a significant amount of compute by not computing the rest of the folds. There are two strategies that TPOT can use to accomplish this (More information on these strategies in Tutorial 8).\n",
" 1. Threshold Pruning: Pipelines must achieve a score above a predefined percentile threshold (based on previous pipeline scores) to proceed in each cross-validation (CV) fold.\n",
" 2. Selection Pruning: Within each population, only the top N% of pipelines (ranked by performance in the previous CV fold) are selected to evaluate in the next fold.\"\n",
"With the memory parameter, pipelines can cache the results of each transformer after fitting them. This feature is used to avoid repeated computation by transformers within a pipeline if the parameters and input data are identical to another fitted pipeline during the optimization process. TPOT allows users to specify a custom directory path or joblib.Memory in case they want to re-use the memory cache in future TPOT runs (or a warm_start run).\n",
"**Note: TPOT does NOT clean up memory caches if users set a custom directory path or Memory object. We recommend that you clean up the memory caches when you don't need it anymore.**"
" * Individual pipelines may need more time to complete fitting; increase `max_eval_time_seconds.`\n",
" * The configuration may not include the optimal model types or hyperparameter ranges, explore other included templates, or customize your own search space (see Tutorial 2!)\n",
" * Check that `periodic_checkpoint_folder` is set correctly. A common issue is forgetting to change this folder between experiments and TPOT continuing training from pipelines optimized for another dataset.\n",
" * Check that at least one of the three termination conditions is set to a reasonable level. These are `max_time_mins`, `early_stop`, or `generations`. Additionally, check that `max_eval_time_seconds` gives enough time for most models to train without being overly long. (Some estimators may take an unreasonably long time to fit; this parameter is intended to prevent them from slowing everything to a halt. In my experience, SVC and SVR tend to be the culprits, so removing them from the search space may also improve run time).\n",
" * Increase n_jobs to use more processes/CPU power. See Tutorial 7 for advanced Dask usage, including parallelizing across multiple nodes on an HPC.\n",
" * Use feature selection, either the build in configuration of sklearn methods (see Tutorial 2), or genetic feature selection (see Tutorials 3 and 5 for two different strategies).\n",
" * Use successive halving to reduce computational load (See tutorial 8).\n",
"* Many pipelines in the evaluated_individuals data frame have crashed or turned up invalid!\n",
" * This is normal and is expected behavior for TPOT. In some cases, TPOT may attempt an invalid hyperparameter combination, resulting in the pipeline not working. Other times, the pipeline configuration itself may be invalid. For example, a selector may not select any features due to its hyperparameter. Another common example is `MultinomialNB` throwing an error because it expects positive values, but a prior transformation yielded a negative value. \n",
" * If you used custom search spaces, you can use `ConfigSpace` conditionals to prevent invalid hyperparameters (this may still occur due to how TPOT uses crossover).\n",
" * Setting `verbose=5` will print out the full error message for all failed pipelines. This can be useful for debugging whether or not there is something misconfigured in your pipeline, custom search space modules, or something else.\n",
" * Set the `memory_limit` parameter so that n_jobs*memorylimit is less than the available RAM on your machine, plus some wiggle room. This should prevent crashing due to memory concerns.\n",
" * Using feature selection may also improve memory usage, as described above.\n",
"* Why are my TPOT runs not reproducible when random_state is set?\n",
" * Check that `periodic_checkpoint_folder` is set correctly. If this is set to a non-empty folder, TPOT will continue training from the checkpoint rather than start a new run from scratch. For TPOT runs to be reproducible, they have to have the same starting points.\n",
" * If using custom search spaces, pass in a fixed `random_state` value into the configspace of the scikit-learn modules that utilize them. TPOT does not check whether estimators do or do not take in a random state value (See Tutorial 2).\n",
" * If using the pre-built search spaces provided by TPOT, make sure to pass in `random_state` to `tpot.config.get_configspace` or `tpot.config.template_search_spaces.get_template_search_spaces`. This ensures all estimators that support it get a fixed random_state value. (See Tutorial 2).\n",
" * If using custom Node and Pipeline types, ensure all random decisions utilize the rng parameter passed into the mutation/crossover functions.\n",
" * If `max_eval_time_mins` is set, TPOT will terminate pipelines that exceed this time limit. If the pipeline evaluation happens to be very similar to the time limit, small random fluctuations in CPU allocation may cause a given pipeline to be evaluated in one run but not another. This slightly different result would throw off the random number generator throughout the rest of the run. Setting `max_eval_time_mins` to None or a higher value may prevent this edge case.\n",
" * If using `TPOTEstimatorSteadyState` with `n_jobs`>1, it is also possible that random fluctuations in CPU allocation slightly change the order in which pipelines are evaluated, which will affect the downstream results. `TPOTEstimatorSteadyState` is more reliably reproducible when `n_jobs=1` (This is not an issue for the default `TPOTEstimator`, `TPOTClassifier`, `TPOTRegressor` as they used a batched generational approach where execution order does not impact results).\n",
"* TPOT is not using all the CPU cores I expected, given my `n_jobs` setting.\n",
" * The default TPOT algorithm uses a generational approach. This means the TPOT will need to evaluate `population_size` (default 50) pipelines before starting the next batch. At the end of each generation, TPOT may leave threads unused while it waits for the last few pipelines to finish evaluating. Some estimators or pipelines can be significantly slower to evaluate than others. This can be addressed in a few ways:\n",
" * Decrease `max_eval_time_mins` to cut long-running pipeline evaluations early.\n",
" * Remove estimators or hyperparameter configurations that are prone to very slow convergence (which is very often `SVC` or `SVR`).\n",
" * Alternatively, `TPOTEstimatorSteadyState` uses a slightly different backend for the evolutionary algorithm that does not utilize the generational approach. Instead, new pipelines are generated and evaluated as soon as the previous one finishes. With this estimator, all cores should be utilized at all times. \n",
" * Sometimes, setting n_jobs to a multiple of the number of threads can help minimize the chances of threads being idle while waiting for others to finish"
"`tpot.TPOTClassifier` and `tpot.TPOTRegressor` have a simplified set of hyperparameters with default values set for classification and regression problems. Currently, both of these use the standard evolutionary algorithm in the `tpot.TPOTEstimator` class. If you want more control, you can look into either the `tpot.TPOTEstimator` or `tpot.TPOTEstimatorSteadyState` class.\n",
"1. The `tpot.TPOTEstimator` uses a standard evolutionary algorithm that evaluates exactly population_size individuals each generation. This is similar to the algorithm in TPOT1. The next generation does not start until the previous is completely finished evaluating. This leads to underutilized CPU time as the cores are waiting for the last individuals to finish training, but may preserve diversity in the population. \n",
"2. The `tpot.TPOTEstimatorSteadyState` differs in that it will generate and evaluate the next individual as soon as an individual finishes the evaluation. The number of individuals being evaluated is determined by the n_jobs parameter. There is no longer a concept of generations. The population_size parameter now refers to the size of the list of evaluated parents. When an individual is evaluated, the selection method updates the list of parents. This allows more efficient utilization when using multiple cores.\n"
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/preprocessing/_data.py:2785: UserWarning: n_quantiles (688) is greater than the total number of samples (426). n_quantiles is set to n_samples.\n",
"/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/linear_model/_sag.py:349: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
" scorers=['roc_auc_ovr'], #scorers can be a list of strings or a list of scorers. These get evaluated during cross validation. \n",
" scorers_weights=[1],\n",
" classification=True,\n",
" n_jobs=1, \n",
" early_stop=5, #how many generations with no improvement to stop after\n",
" \n",
" #List of other objective functions. All objective functions take in an untrained GraphPipeline and return a score or a list of scores\n",
" other_objective_functions= [ ],\n",
" \n",
" #List of weights for the other objective functions. Must be the same length as other_objective_functions. By default, bigger is better is set to True. \n",
