MORPH
SIGN IN SIGN UP
EpistasisLab / tpot UNCLAIMED

A Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming.

0 0 1 Jupyter Notebook
Normal View History Raw
edits
2024-09-20 14:48:56 -07:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# What to expect from AutoML software\n",
"Automated machine learning (AutoML) takes a higher-level approach to machine learning than most practitioners are used to, so we've gathered a handful of guidelines on what to expect when running AutoML software such as TPOT.\n",
"\n",
"#### AUTOML ALGORITHMS AREN'T INTENDED TO RUN FOR ONLY A FEW MINUTES\n",
"Of course, you can run TPOT for only a few minutes and it will find a reasonably good pipeline for your dataset. However, if you don't run TPOT for long enough, it may not find the best possible pipeline for your dataset. It may even not find any suitable pipeline at all, in which case a RuntimeError('A pipeline has not yet been optimized. Please call fit() first.') will be raised. Often it is worthwhile to run multiple instances of TPOT in parallel for a long time (hours to days) to allow TPOT to thoroughly search the pipeline space for your dataset.\n",
"\n",
"#### AUTOML ALGORITHMS CAN TAKE A LONG TIME TO FINISH THEIR SEARCH\n",
"AutoML algorithms aren't as simple as fitting one model on the dataset; they are considering multiple machine learning algorithms (random forests, linear models, SVMs, etc.) in a pipeline with multiple preprocessing steps (missing value imputation, scaling, PCA, feature selection, etc.), the hyperparameters for all of the models and preprocessing steps, as well as multiple ways to ensemble or stack the algorithms within the pipeline.\n",
"\n",
"As such, TPOT will take a while to run on larger datasets, but it's important to realize why. With the default TPOT settings (100 generations with 100 population size), TPOT will evaluate 10,000 pipeline configurations before finishing. To put this number into context, think about a grid search of 10,000 hyperparameter combinations for a machine learning algorithm and how long that grid search will take. That is 10,000 model configurations to evaluate with 10-fold cross-validation, which means that roughly 100,000 models are fit and evaluated on the training data in one grid search. That's a time-consuming procedure, even for simpler models like decision trees.\n",
"\n",
"Typical TPOT runs will take hours to days to finish (unless it's a small dataset), but you can always interrupt the run partway through and see the best results so far. TPOT also provides a warm_start parameter that lets you restart a TPOT run from where it left off.\n",
"\n",
"#### AUTOML ALGORITHMS CAN RECOMMEND DIFFERENT SOLUTIONS FOR THE SAME DATASET\n",
"If you're working with a reasonably complex dataset or run TPOT for a short amount of time, different TPOT runs may result in different pipeline recommendations. TPOT's optimization algorithm is stochastic in nature, which means that it uses randomness (in part) to search the possible pipeline space. When two TPOT runs recommend different pipelines, this means that the TPOT runs didn't converge due to lack of time or that multiple pipelines perform more-or-less the same on your dataset.\n",
"\n",
"This is actually an advantage over fixed grid search techniques: TPOT is meant to be an assistant that gives you ideas on how to solve a particular machine learning problem by exploring pipeline configurations that you might have never considered, then leaves the fine-tuning to more constrained parameter tuning techniques such as grid search."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# TPOT with code\n",
"\n",
"We've taken care to design the TPOT interface to be as similar as possible to scikit-learn.\n",
"\n",
"TPOT can be imported just like any regular Python module. To import TPOT, type:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from tpot2 import TPOTClassifier"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"then create an instance of TPOT as follows:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"classification_optimizer = TPOTClassifier()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It's also possible to use TPOT for regression problems with the TPOTRegressor class. Other than the class name, a TPOTRegressor is used the same way as a TPOTClassifier. You can read more about the TPOTClassifier and TPOTRegressor classes in the API documentation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tpot2 import TPOTRegressor\n",
"regression_optimizer = TPOTRegressor()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Fitting a TPOT model works exactly like any other sklearn estimator. Some example code with custom TPOT parameters might look like:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Generation: : 3it [00:33, 11.04s/it]\n",
"/home/perib/miniconda3/envs/myenv/lib/python3.10/site-packages/sklearn/preprocessing/_data.py:2762: UserWarning: n_quantiles (1895) is greater than the total number of samples (455). n_quantiles is set to n_samples.\n",
" warnings.warn(\n",
"/home/perib/miniconda3/envs/myenv/lib/python3.10/site-packages/sklearn/linear_model/_sag.py:350: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
" warnings.warn(\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"auroc_score: 0.9904100529100529\n"
]
}
],
"source": [
"import sklearn\n",
"import sklearn.datasets\n",
"import sklearn.metrics\n",
"\n",
"classification_optimizer = TPOTClassifier(search_space=\"light\", max_time_mins=30/60, n_jobs=30, cv=5)\n",
"\n",
"X, y = sklearn.datasets.load_breast_cancer(return_X_y=True)\n",
"X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, random_state=1, test_size=0.2)\n",
"\n",
"classification_optimizer.fit(X_train, y_train)\n",
"\n",
"auroc_score = sklearn.metrics.roc_auc_score(y_test, classification_optimizer.predict_proba(X_test)[:,1])\n",
"print(\"auroc_score: \", auroc_score)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Scorers, Objective Functions, and multi objective optimization.\n",
"\n",
"There are two ways of passing objectives into TPOT2. \n",
"\n",
"1. `scorers`: Scorers are functions that have the signature (estimator, X, y). These can be produced with the [sklearn.metrics.make_scorer](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html) function. This function is used to evaluate the test folds during cross validation. These are passed into TPOT2 via the scorers parameter. This can take in the scorer itself or the string corresponding to a scoring function ([as listed here](https://scikit-learn.org/stable/modules/model_evaluation.html)). TPOT2 also supports passing in a list of several scorers for multiobjective optimization. \n",
"\n",
"2. `other_objective_functions` : Other objective functions in TPOT2 have the signature (estimator) and returns a float or list of floats. These get passed an unfitted estimator (in the case of TPOT2, a `tpot2.GraphPipeline`). \n",
"\n",
"\n",
"\n",
"Each scorer and objective function must be accompanied by a list of weights corresponding to the list of objectives. By default, TPOT2 maximizes objective functions (this can be changed by `bigger_is_better=False`). Positive weights means that TPOT2 will seek to maximize that objective, and negative weights correspond to minimization.\n",
"\n",
"Here is an example of using two scorers\n",
"\n",
" scorers=['roc_auc_ovr',tpot2.objectives.complexity_scorer],\n",
" scorers_weights=[1,-1],\n",
"\n",
"\n",
"Here is an example with a scorer and a secondary objective function\n",
"\n",
" scorers=['roc_auc_ovr'],\n",
" scorers_weights=[1],\n",
" other_objective_functions=[tpot2.objectives.number_of_leaves_objective],\n",
" other_objective_functions_weights=[-1],\n",
"\n",
"\n",
"TPOT will automatically name the scores based on the function name for the columns in the final results dataframe. If you would like to specify custom function names, you can set the `objective_function_names` to be a list of names (str) for each score. The order of the names are scorers first, and other objective functions second. (e.g. `objective_function_names=['scorer1','scorer2', 'objective1','objective2'])`.\n",
"\n",
"It is possible to have either the scorer or other_objective_function to return multiple values. In that case, just make sure that the `scorer_weights` and `other_objective_function_weights` are the same length as the number of returned scores.\n",
"\n",
"\n",
"TPOT comes with a few additional built in objective functions you can use. The first table are objectives applied to fitted pipelines, and thus are passee into the `scorers` parameter. The second table are objective functions for the `other_objective_functions` param.\n",
"\n",
"Scorers:\n",
"| Function | Description |\n",
"| :--- | :----: |\n",
"| tpot2.objectives.complexity_scorer | Estimates the number of learned parameters across all classifiers and regressors in the pipelines. Additionally, currently transformers add 1 point and selectors add 0 points (since they don't affect the complexity of the \"final\" predictive pipeline.) |\n",
"\n",
"Other Objective Functions.\n",
"\n",
"| Function | Description |\n",
"| :--- | :----: |\n",
"| tpot2.objectives.average_path_length | Computes the average shortest path from all nodes to the root/final estimator (only supported for GraphPipeline) |\n",
"| tpot2.objectives.number_of_leaves_objective | Calculates the number of leaves (input nodes) in a GraphPipeline |\n",
"| tpot2.objectives.number_of_nodes_objective | Calculates the number of nodes in a pipeline (whether it is an scikit-learn Pipeline, GraphPipeline, Feature Union, or the previous nested within each other) |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Built In Configurations\n",
"TPOT can be used to optimize hyperparameters, select models, and optimize pipelines of models including determining the sequence of steps. Tutorial 2 goes into more detail on how to customize search spaces with custom hyperparameter ranges, model types, and possible pipeline configurations. TPOT also comes with a handful of default operators and parameter configurations that we believe work well for optimizing machine learning pipelines. Below is a list of the current built-in configurations that come with TPOT. These can be passed in as strings to the `search space` parameter of any of the TPOT estimators.\n",
"\n",
"| String | Description |\n",
"| :--- | :----: |\n",
"| linear | A linear pipeline with the structure of \"Selector->(transformers+Passthrough)->(classifiers/regressors+Passthrough)->final classifier/regressor.\" For both the transformer and inner estimator layers, TPOT may choose one or more transformers/classifiers, or it may choose none. The inner classifier/regressor layer is optional. |\n",
"| light | Same search space as linear, but without the inner classifier/regressor layer and with a reduced set of faster running estimators. |\n",
"| graph | TPOT will optimize a pipeline in the shape of a directed acyclic graph. The nodes of the graph can include selectors, scalers, transformers, or classifiers/regressors (inner classifiers/regressors can optionally be not included). This will return a custom GraphPipeline rather than an sklearn Pipeline. More details in Tutorial 6. |\n",
"| mdr |TPOT will search over a series of feature selectors and Multifactor Dimensionality Reduction models to find a series of operators that maximize prediction accuracy. The TPOT MDR configuration is specialized for genome-wide association studies (GWAS), and is described in detail online here.\n",
"\n",
"Note that TPOT MDR may be slow to run because the feature selection routines are computationally expensive, especially on large datasets. |\n",
"\n",
"Note: the `linear` and `graph` configurations by default allow for additional stacked classifiers/regressors within the pipeline in addition to the final classifier/regressor. If you would like to disable this, you can manually get the search space without inner classifier/regressors through the function `tpot2.config.template_search_spaces.get_template_search_spaces` with `inner_predictios=False`. You can pass the resulting search space into the `search space` param.\n",
"\n",
"The specific hyperparameter ranges used by TPOT can be found in files in the tpot2/config folder. The template search spaces listed above are defined in tpot2/config/template_search_spaces.py. Search spaces for individual models can be acquired in the tpot2/config/get_configspace.py file (`tpot2.config.get_search_space`). More details in Tutorial 2."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Example analysis "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Best Practices\n",
"\n",
"When running tpot from an .py script, it is important to protect code with `if __name__==\"__main__\":` . This is because of how TPOT handles parallelization with Python and Dask."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#my_analysis.py\n",
"\n",
"from dask.distributed import Client, LocalCluster\n",
"import tpot2\n",
"import sklearn\n",
"import sklearn.datasets\n",
"import numpy as np\n",
"\n",
"if __name__==\"__main__\":\n",
" scorer = sklearn.metrics.get_scorer('roc_auc_ovo')\n",
" X, y = sklearn.datasets.load_digits(return_X_y=True)\n",
" X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, train_size=0.75, test_size=0.25)\n",
"\n",
"\n",
" est = tpot2.TPOTClassifier(n_jobs=4, max_time_mins=60, verbose=2)\n",
" est.fit(X_train, y_train)\n",
"\n",
"\n",
" print(scorer(est, X_test, y_test))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Common parameters\n",
"\n",
" scorers : (list, scorer)\n",
" A scorer or list of scorers to be used in the cross-validation process. \n",
" see https://scikit-learn.org/stable/modules/model_evaluation.html\n",
" \n",
" scorers_weights : list\n",
" A list of weights to be applied to the scorers during the optimization process.\n",
" \n",
" classification : bool\n",
" If True, the problem is treated as a classification problem. If False, the problem is treated as a regression problem.\n",
" Used to determine the CV strategy.\n",
" \n",
" cv : int, cross-validator\n",
" - (int): Number of folds to use in the cross-validation process. By uses the sklearn.model_selection.KFold cross-validator for regression and StratifiedKFold for classification. In both cases, shuffled is set to True.\n",
" - (sklearn.model_selection.BaseCrossValidator): A cross-validator to use in the cross-validation process.\n",
" - max_depth (int): The maximum depth from any node to the root of the pipelines to be generated.\n",
" \n",
" other_objective_functions : list, default=[tpot2.objectives.estimator_objective_functions.average_path_length_objective]\n",
" A list of other objective functions to apply to the pipeline.\n",
" \n",
" other_objective_functions_weights : list, default=[-1]\n",
" A list of weights to be applied to the other objective functions.\n",
" \n",
" objective_function_names : list, default=None\n",
" A list of names to be applied to the objective functions. If None, will use the names of the objective functions.\n",
" \n",
" bigger_is_better : bool, default=True\n",
" If True, the objective function is maximized. If False, the objective function is minimized. Use negative weights to reverse the direction.\n",
" \n",
" generations : int, default=50\n",
" Number of generations to run\n",
" \n",
" max_time_mins : float, default=float(\"inf\")\n",
" Maximum time to run the optimization. If none or inf, will run until the end of the generations.\n",
" \n",
" max_eval_time_mins : float, default=60*5\n",
" Maximum time to evaluate a single individual. If none or inf, there will be no time limit per evaluation.\n",
"\n",
" n_jobs : int, default=1\n",
" Number of processes to run in parallel.\n",
" \n",
" memory_limit : str, default=\"4GB\"\n",
" Memory limit for each job. See Dask [LocalCluster documentation](https://distributed.dask.org/en/stable/api.html#distributed.Client) for more information.\n",
"\n",
" \n",
" verbose : int, default=1 \n",
" How much information to print during the optimization process. Higher values include the information from lower values.\n",
" 0. nothing\n",
" 1. progress bar\n",
" \n",
" 3. best individual\n",
" 4. warnings\n",
" >=5. full warnings trace\n",
" 6. evaluations progress bar. (Temporary: This used to be 2. Currently, using evaluation progress bar may prevent some instances were we terminate a generation early due to it reaching max_time_mins in the middle of a generation OR a pipeline failed to be terminated normally and we need to manually terminate it.)\n",
" \n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# More Options\n",
"\n",
"`tpot2.TPOTClassifier` and `tpot2.TPOTRegressor` have a simplified set of hyperparameters with default values set for classification and regression problems. Currently, both of these use the standard evolutionary algorithm in the `tpot2.TPOTEstimator` class. If you want more control you can look into either the `tpot2.TPOTEstimator` or `tpot2.TPOTEstimatorSteadyState` class.\n",
"\n",
"There are two evolutionary algorithms built into TPOT2, which corresponds to two different estimator classes.\n",
"\n",
"1. The `tpot2.TPOTEstimator` uses a standard evolutionary algorithm that evaluates exactly population_size individuals each generation. This is similar to the algorithm in TPOT1. The next generation does not start until the previous is completely finished evaluating. This leads to underutilized CPU time as the cores are waiting for the last individuals to finish training, but may preserve diversity in the population. \n",
"\n",
"2. The `tpot2.TPOTEstimatorSteadyState` differs in that it will generate and evaluate the next individual as soon as an individual finishes evaluation. The number of individuals being evaluated is determined by the n_jobs parameter. There is no longer a concept of generations. The population_size parameter now refers to the size of the list of evaluated parents. When an individual is evaluated, the selection method updates the list of parents. This allows more efficient utilization when using multiple cores.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import tpot2\n",
"import sklearn\n",
"import sklearn.datasets\n",
"\n",
"scorer = sklearn.metrics.get_scorer('roc_auc_ovo')\n",
"X, y = sklearn.datasets.load_iris(return_X_y=True)\n",
"X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, train_size=0.75, test_size=0.25)\n",
"\n",
"\n",
"est = tpot2.TPOTClassifier(n_jobs=40, max_time_mins=30, verbose=5, generations=1, population_size=5)\n",
"est.fit(X_train, y_train)\n",
"\n",
"\n",
"print(scorer(est, X_test, y_test))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"est._evolver_instance.population.evaluated_individuals.iloc[0]['Individual'].export_pipeline()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import tpot2\n",
"import sklearn\n",
"import sklearn.metrics\n",
"import sklearn.datasets\n",
"\n",
"scorer = sklearn.metrics.get_scorer('neg_mean_squared_error')\n",
"X, y = sklearn.datasets.load_diabetes(return_X_y=True)\n",
"X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, train_size=0.75, test_size=0.25)\n",
"\n",
"est = tpot2.tpot_estimator.templates.TPOTRegressor(n_jobs=4, max_time_mins=30, verbose=2, cv=5)\n",
"est.fit(X_train, y_train)\n",
"\n",
"print(scorer(est, X_test, y_test))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### tpot2.TPOTEstimatorSteadyState"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import tpot2\n",
"import sklearn\n",
"import sklearn.datasets\n",
"\n",
"\n",
"graph_search_space = tpot2.search_spaces.pipelines.GraphPipeline(\n",
" root_search_space= tpot2.config.get_search_space([\"KNeighborsClassifier\", \"LogisticRegression\", \"DecisionTreeClassifier\"]),\n",
" leaf_search_space = tpot2.config.get_search_space(\"selectors\"), \n",
" inner_search_space = tpot2.config.get_search_space([\"transformers\"]),\n",
" max_size = 10,\n",
")\n",
"\n",
"est = tpot2.TPOTEstimatorSteadyState( \n",
" search_space = graph_search_space,\n",
" scorers=['roc_auc_ovr'], #scorers can be a list of strings or a list of scorers. These get evaluated during cross validation. \n",
" scorers_weights=[1],\n",
"\n",
" classification=True,\n",
"\n",
" max_eval_time_mins=15,\n",
" max_time_mins=30,\n",
" verbose=2)\n",
"\n",
"\n",
"scorer = sklearn.metrics.get_scorer('roc_auc_ovo')\n",
"X, y = sklearn.datasets.load_iris(return_X_y=True)\n",
"X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, train_size=0.75, test_size=0.25)\n",
"est.fit(X_train, y_train)\n",
"print(scorer(est, X_test, y_test))\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fitted_pipeline = est.fitted_pipeline_ # access best pipeline directly\n",
"fitted_pipeline.plot()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#view the summary of all evaluated individuals as a pandas dataframe\n",
"est.evaluated_individuals"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import tpot2\n",
"import sklearn\n",
"import sklearn.datasets\n",
"\n",
"est = tpot2.TPOTEstimatorSteadyState( \n",
" search_space = graph_search_space,\n",
" scorers=['roc_auc_ovr',tpot2.objectives.complexity_scorer],\n",
" scorers_weights=[1,-1],\n",
"\n",
" classification=True,\n",
"\n",
" max_eval_time_mins=15,\n",
" max_time_mins=30,\n",
" verbose=2)\n",
"\n",
"\n",
"scorer = sklearn.metrics.get_scorer('roc_auc_ovo')\n",
"X, y = sklearn.datasets.load_iris(return_X_y=True)\n",
"X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, train_size=0.75, test_size=0.25)\n",
"est.fit(X_train, y_train)\n",
"print(scorer(est, X_test, y_test))\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fitted_pipeline = est.fitted_pipeline_ # access best pipeline directly\n",
"fitted_pipeline.plot() #plot the best pipeline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"view the results of all evaluated individuals as a pandas dataframe"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"est.evaluated_individuals"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"view pareto front as a pandas dataframe"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"est.pareto_front"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pareto_front = est.pareto_front\n",
"\n",
"#plot the pareto front of number_of_leaves_objective vs roc_auc_score\n",
"\n",
"import matplotlib.pyplot as plt\n",
"plt.scatter(pareto_front['complexity_scorer'], pareto_front['roc_auc_score'])\n",
"plt.xlabel('complexity_scorer')\n",
"plt.ylabel('roc_auc_score')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### tpot2.TPOTEstimator"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import tpot2\n",
"import sklearn\n",
"import sklearn.datasets\n",
"\n",
"est = tpot2.TPOTEstimator( \n",
" search_space = graph_search_space,\n",
" population_size=30,\n",
" generations=5,\n",
" scorers=['roc_auc_ovr'], #scorers can be a list of strings or a list of scorers. These get evaluated during cross validation. \n",
" scorers_weights=[1],\n",
" classification=True,\n",
" n_jobs=1, \n",
" early_stop=5, #how many generations with no improvement to stop after\n",
" \n",
" #List of other objective functions. All objective functions take in an untrained GraphPipeline and return a score or a list of scores\n",
" other_objective_functions= [ ],\n",
" \n",
" #List of weights for the other objective functions. Must be the same length as other_objective_functions. By default, bigger is better is set to True. \n",
" other_objective_functions_weights=[],\n",
" verbose=2)\n",
"\n",
"scorer = sklearn.metrics.get_scorer('roc_auc_ovo')\n",
"X, y = sklearn.datasets.load_iris(return_X_y=True)\n",
"X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, train_size=0.75, test_size=0.25)\n",
"est.fit(X_train, y_train)\n",
"print(scorer(est, X_test, y_test))"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "tpot_dev",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.14"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "7fe1fe9ef32cd5efd76326a08046147513534f0dd2318301a1a96ae9071c1c4e"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Reference in New Issue Copy Permalink