mlr3resampling: Resampling Algorithms for 'mlr3' Framework

A supervised learning algorithm inputs a train set, and outputs a prediction function, which can be used on a test set. If each data point belongs to a group (such as geographic region, year, etc), then how do we know if it is possible to train on one group, and predict accurately on another group? Cross-validation can be used to determine the extent to which this is possible, by first assigning fold IDs from 1 to K to all data (possibly using stratification, usually by group and label). Then we loop over test sets (group/fold combinations), train sets (same group, other groups, all groups), and compute test/prediction accuracy for each combination. Comparing test/prediction accuracy between same and other, we can determine the extent to which it is possible (perfect if same/other have similar test accuracy for each group; other is usually somewhat less accurate than same; other can be just as bad as featureless baseline when the groups have different patterns). For more information, <https://tdhock.github.io/blog/2023/R-gen-new-subsets/> describes the method in depth. How many train samples are required to get accurate predictions on a test set? Cross-validation can be used to answer this question, with variable size train sets.

Version: 2024.9.6
Imports: data.table, R6, checkmate, paradox, mlr3, mlr3misc
Suggests: ggplot2, animint2, mlr3tuning, lgr, future, testthat, knitr, markdown, nc, rpart, directlabels
Published: 2024-09-11
DOI: 10.32614/CRAN.package.mlr3resampling
Author: Toby Hocking ORCID iD [aut, cre] (<https://orcid.org/0000-0002-3146-0865>), Michel Lang ORCID iD [ctb] (<https://orcid.org/0000-0001-9754-0393>, Author of mlr3 when Resampling/ResamplingCV was copied/modified), Bernd Bischl ORCID iD [ctb] (<https://orcid.org/0000-0001-6002-6980>, Author of mlr3 when Resampling/ResamplingCV was copied/modified), Jakob Richter ORCID iD [ctb] (<https://orcid.org/0000-0003-4481-5554>, Author of mlr3 when Resampling/ResamplingCV was copied/modified), Patrick Schratz ORCID iD [ctb] (<https://orcid.org/0000-0003-0748-6624>, Author of mlr3 when Resampling/ResamplingCV was copied/modified), Giuseppe Casalicchio ORCID iD [ctb] (<https://orcid.org/0000-0001-5324-5966>, Author of mlr3 when Resampling/ResamplingCV was copied/modified), Stefan Coors ORCID iD [ctb] (<https://orcid.org/0000-0002-7465-2146>, Author of mlr3 when Resampling/ResamplingCV was copied/modified), Quay Au ORCID iD [ctb] (<https://orcid.org/0000-0002-5252-8902>, Author of mlr3 when Resampling/ResamplingCV was copied/modified), Martin Binder [ctb], Florian Pfisterer ORCID iD [ctb] (<https://orcid.org/0000-0001-8867-762X>, Author of mlr3 when Resampling/ResamplingCV was copied/modified), Raphael Sonabend ORCID iD [ctb] (<https://orcid.org/0000-0001-9225-4654>, Author of mlr3 when Resampling/ResamplingCV was copied/modified), Lennart Schneider ORCID iD [ctb] (<https://orcid.org/0000-0003-4152-5308>, Author of mlr3 when Resampling/ResamplingCV was copied/modified), Marc Becker ORCID iD [ctb] (<https://orcid.org/0000-0002-8115-0400>, Author of mlr3 when Resampling/ResamplingCV was copied/modified), Sebastian Fischer ORCID iD [ctb] (<https://orcid.org/0000-0002-9609-3197>, Author of mlr3 when Resampling/ResamplingCV was copied/modified)
Maintainer: Toby Hocking <toby.hocking at r-project.org>
BugReports: https://github.com/tdhock/mlr3resampling/issues
License: GPL-3
URL: https://github.com/tdhock/mlr3resampling
NeedsCompilation: no
Materials: NEWS
CRAN checks: mlr3resampling results

Documentation:

Reference manual: mlr3resampling.pdf
Vignettes: Comparing sizes when training on same or other groups (source, R code)
Older resamplers (source, R code)

Downloads:

Package source: mlr3resampling_2024.9.6.tar.gz
Windows binaries: r-devel: mlr3resampling_2024.9.6.zip, r-release: mlr3resampling_2024.9.6.zip, r-oldrel: mlr3resampling_2024.9.6.zip
macOS binaries: r-release (arm64): mlr3resampling_2024.9.6.tgz, r-oldrel (arm64): mlr3resampling_2024.9.6.tgz, r-release (x86_64): mlr3resampling_2024.9.6.tgz, r-oldrel (x86_64): mlr3resampling_2024.9.6.tgz
Old sources: mlr3resampling archive

Linking:

Please use the canonical form https://CRAN.R-project.org/package=mlr3resampling to link to this page.