Skip to contents

Extreme Gradient Boosting Classification Learner

Source: R/LearnerClassifXgboost.R
mlr_learners_classif.xgboost.Rd

eXtreme Gradient Boosting classification. Calls xgboost::xgb.train() from package xgboost.

Note that using the evals parameter directly will lead to problems when wrapping this mlr3::Learner in a mlr3pipelines GraphLearner as the preprocessing steps will not be applied to the data in evals. See the section Early Stopping and Validation on how to do this.

Note

To compute on GPUs, you first need to compile xgboost yourself and link against CUDA. See https://xgboost.readthedocs.io/en/stable/build.html#building-with-gpu-support.

The outputmargin, predcontrib, predinteraction, and predleaf parameters are not supported. You can still call e.g. predict(learner$model, newdata = newdata, outputmargin = TRUE) to get these predictions.

Initial parameter values

  • nrounds:

    • Actual default: no default.

    • Adjusted default: 1000.

    • Reason for change: Without a default construction of the learner would error. The lightgbm learner has a default of 1000, so we use the same here.

  • nthread:

    • Actual value: Undefined, triggering auto-detection of the number of CPUs.

    • Adjusted value: 1.

    • Reason for change: Conflicting with parallelization via future.

  • verbose:

    • Actual default: 1.

    • Adjusted default: 0.

    • Reason for change: Reduce verbosity.

  • verbosity:

    • Actual default: 1.

    • Adjusted default: 0.

    • Reason for change: Reduce verbosity.

Early Stopping and Validation

In order to monitor the validation performance during the training, you can set the $validate field of the Learner. For information on how to configure the validation set, see the Validation section of mlr3::Learner. This validation data can also be used for early stopping, which can be enabled by setting the early_stopping_rounds parameter. The final (or in the case of early stopping best) validation scores can be accessed via $internal_valid_scores, and the optimal nrounds via $internal_tuned_values. The internal validation measure can be set via the custom_metric parameter that can be a mlr3::Measure, a function, or a character string for the internal xgboost measures. Using an mlr3::Measure is slower than the internal xgboost measures, but allows to use the same measure for tuning and validation.

Dictionary

This mlr3::Learner can be instantiated via the dictionary mlr3::mlr_learners or with the associated sugar function mlr3::lrn():

mlr_learners$get("classif.xgboost")
lrn("classif.xgboost")

Meta Information

  • Task type: “classif”

  • Predict Types: “response”, “prob”

  • Feature Types: “logical”, “integer”, “numeric”

  • Required Packages: mlr3, mlr3learners, xgboost

Parameters

IdTypeDefaultLevelsRange
alphanumeric0\([0, \infty)\)
approxcontriblogicalFALSETRUE, FALSE-
base_scorenumeric-\((-\infty, \infty)\)
boostercharactergbtreegbtree, gblinear, dart-
callbacksuntypedlist()-
colsample_bylevelnumeric1\([0, 1]\)
colsample_bynodenumeric1\([0, 1]\)
colsample_bytreenumeric1\([0, 1]\)
deviceuntyped"cpu"-
disable_default_eval_metriclogicalFALSETRUE, FALSE-
early_stopping_roundsintegerNULL\([1, \infty)\)
etanumeric0.3\([0, 1]\)
evalsuntypedNULL-
eval_metricuntyped--
custom_metricuntyped--
extmem_single_pagelogicalFALSETRUE, FALSE-
feature_selectorcharactercycliccyclic, shuffle, random, greedy, thrifty-
gammanumeric0\([0, \infty)\)
grow_policycharacterdepthwisedepthwise, lossguide-
interaction_constraintsuntyped--
iterationrangeuntyped--
lambdanumeric1\([0, \infty)\)
max_bininteger256\([2, \infty)\)
max_cached_hist_nodeinteger65536\((-\infty, \infty)\)
max_cat_to_onehotinteger-\((-\infty, \infty)\)
max_cat_thresholdnumeric-\((-\infty, \infty)\)
max_delta_stepnumeric0\([0, \infty)\)
max_depthinteger6\([0, \infty)\)
max_leavesinteger0\([0, \infty)\)
maximizelogicalNULLTRUE, FALSE-
min_child_weightnumeric1\([0, \infty)\)
missingnumericNA\((-\infty, \infty)\)
monotone_constraintsuntyped0-
nroundsinteger-\([1, \infty)\)
normalize_typecharactertreetree, forest-
nthreadinteger-\([1, \infty)\)
num_parallel_treeinteger1\([1, \infty)\)
objectiveuntyped"binary:logistic"-
one_droplogicalFALSETRUE, FALSE-
print_every_ninteger1\([1, \infty)\)
rate_dropnumeric0\([0, 1]\)
refresh_leaflogicalTRUETRUE, FALSE-
seedinteger-\((-\infty, \infty)\)
seed_per_iterationlogicalFALSETRUE, FALSE-
sampling_methodcharacteruniformuniform, gradient_based-
sample_typecharacteruniformuniform, weighted-
save_nameuntypedNULL-
save_periodintegerNULL\([0, \infty)\)
scale_pos_weightnumeric1\((-\infty, \infty)\)
skip_dropnumeric0\([0, 1]\)
subsamplenumeric1\([0, 1]\)
top_kinteger0\([0, \infty)\)
traininglogicalFALSETRUE, FALSE-
tree_methodcharacterautoauto, exact, approx, hist, gpu_hist-
tweedie_variance_powernumeric1.5\([1, 2]\)
updateruntyped--
use_rmmlogical-TRUE, FALSE-
validate_featureslogicalTRUETRUE, FALSE-
verboseinteger-\([0, 2]\)
verbosityinteger-\([0, 2]\)
xgb_modeluntypedNULL-
use_pred_offsetlogical-TRUE, FALSE-

Offset

If a Task has a column with the role offset, it will automatically be used during training. The offset is incorporated through the xgboost::xgb.DMatrix interface, using the base_margin field. During prediction, the offset column from the test set is used only if use_pred_offset = TRUE (default) and the Task has a column with the role offset. The test set offsets are passed via the base_margin argument in xgboost::predict.xgb.Booster(). Otherwise, if the user sets use_pred_offset = FALSE (or the Task doesn't have a column with the offset role), the (possibly estimated) global intercept from the train set is applied. See https://xgboost.readthedocs.io/en/stable/tutorials/intercept.html.

References

Chen, Tianqi, Guestrin, Carlos (2016). “Xgboost: A scalable tree boosting system.” In Proceedings of the 22nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 785–794. ACM. doi:10.1145/2939672.2939785 .

See also

Other Learner: mlr_learners_classif.cv_glmnet, mlr_learners_classif.glmnet, mlr_learners_classif.kknn, mlr_learners_classif.lda, mlr_learners_classif.log_reg, mlr_learners_classif.multinom, mlr_learners_classif.naive_bayes, mlr_learners_classif.nnet, mlr_learners_classif.qda, mlr_learners_classif.ranger, mlr_learners_classif.svm, mlr_learners_regr.cv_glmnet, mlr_learners_regr.glmnet, mlr_learners_regr.kknn, mlr_learners_regr.km, mlr_learners_regr.lm, mlr_learners_regr.nnet, mlr_learners_regr.ranger, mlr_learners_regr.svm, mlr_learners_regr.xgboost

Super classes

mlr3::Learner -> mlr3::LearnerClassif -> LearnerClassifXgboost

Active bindings

internal_valid_scores

(named list() or NULL) The validation scores extracted from model$evaluation_log. If early stopping is activated, this contains the validation scores of the model for the optimal nrounds, otherwise the scores are taken from the final boosting round nrounds.

internal_tuned_values

(named list() or NULL) If early stopping is activated, this returns a list with nrounds, which is extracted from $best_iteration of the model and otherwise NULL.

validate

(numeric(1) or character(1) or NULL) How to construct the internal validation data. This parameter can be either NULL, a ratio, "test", or "predefined".

model

(any)
The fitted model. Only available after $train() has been called.

Methods

Public methods

Inherited methods

Method new()

Creates a new instance of this R6 class.

Usage

LearnerClassifXgboost$new()

Method importance()

The importance scores are calculated with xgboost::xgb.importance().

Usage

LearnerClassifXgboost$importance()

Returns

Named numeric().

Method clone()

The objects of this class are cloneable with this method.

Usage

LearnerClassifXgboost$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

Examples

# Define the Learner and set parameter values
learner = lrn("classif.xgboost")
print(learner)
#> 
#> ── <LearnerClassifXgboost> (classif.xgboost): Extreme Gradient Boosting ────────
#> • Model: -
#> • Parameters: nrounds=1000, nthread=1, verbose=0, verbosity=0,
#> use_pred_offset=TRUE
#> • Validate: NULL
#> • Packages: mlr3, mlr3learners, and xgboost
#> • Predict Types: [response] and prob
#> • Feature Types: logical, integer, and numeric
#> • Encapsulation: none (fallback: -)
#> • Properties: hotstart_forward, importance, internal_tuning, missings,
#> multiclass, offset, twoclass, validation, and weights
#> • Other settings: use_weights = 'use', predict_raw = 'FALSE'

# Define a Task
task = tsk("sonar")

# Create train and test set
ids = partition(task)

# Train the learner on the training ids
learner$train(task, row_ids = ids$train)

# Print the model
print(learner$model)
#> ##### xgb.Booster
#> call:
#>   xgboost::xgb.train(params = pv[names(pv) %in% formalArgs(xgboost::xgb.params)], 
#>     data = xgb_data, nrounds = pv$nrounds, evals = pv$evals, 
#>     custom_metric = pv$custom_metric, verbose = pv$verbose, print_every_n = pv$print_every_n, 
#>     early_stopping_rounds = pv$early_stopping_rounds, maximize = pv$maximize, 
#>     save_period = pv$save_period, save_name = pv$save_name, callbacks = pv$callbacks %??% 
#>         list())
#> # of features: 60 
#> # of rounds:  1000 

# Importance method
if ("importance" %in% learner$properties) print(learner$importance())
#>          V11          V44          V36          V27          V39          V31 
#> 0.2695612384 0.0767602955 0.0534629061 0.0495598404 0.0439474966 0.0436734388 
#>          V12           V9          V51          V45          V20          V47 
#> 0.0418860897 0.0403469550 0.0328710617 0.0315765982 0.0256193337 0.0242189464 
#>          V23          V28          V17          V52           V3           V4 
#> 0.0223276583 0.0220107694 0.0218727454 0.0180908980 0.0170899016 0.0168644617 
#>          V58          V13          V35          V55          V34          V41 
#> 0.0150293607 0.0138347531 0.0127656222 0.0127229704 0.0126051090 0.0106864222 
#>          V48          V50          V10          V38          V43           V2 
#> 0.0077612826 0.0071840864 0.0065475359 0.0057306463 0.0056585461 0.0049971443 
#>          V21          V57          V18          V40           V6          V32 
#> 0.0047707806 0.0046292754 0.0045257171 0.0041487932 0.0035976166 0.0029803511 
#>          V19          V54           V7          V60          V53           V5 
#> 0.0016330666 0.0013435950 0.0012746084 0.0011752298 0.0011161542 0.0005219524 
#>          V59          V37          V15 
#> 0.0004627309 0.0003563353 0.0001996791 

# Make predictions for the test rows
predictions = learner$predict(task, row_ids = ids$test)

# Score the predictions
predictions$score()
#> classif.ce 
#>  0.1449275 

# Early stopping
learner = lrn("classif.xgboost", nrounds = 100, early_stopping_rounds = 10, validate = 0.3)

# Train learner with early stopping
learner$train(task)

# Inspect optimal nrounds and validation performance
learner$internal_tuned_values
#> $nrounds
#> [1] 66
#> 
learner$internal_valid_scores
#> $logloss
#> [1] 0.3486057
#>