expect_s7_class() tests if an object is an S7 class
(#1580).
expect_no_failure(),
expect_no_success() and
expect_snapshot_failure() provide more options for testing
expectations.
testthat now requires waldo 0.6.0 or later to access the latest features (#1955).
expect_condition() and related functions now include
the class of the expected condition in the failure message,
if provided (#1987).
expect_error() and friends now error if you supply
... but not pattern (#1932). They no longer
give an uninformative error if they fail inside a magrittr pipe
(#1994).
expect_no_*() expectations no longer incorrectly
emit a passing test result if they in fact fail (#1997).
expect_setequal() correctly identifies what is
missing where (#1962).
expect_snapshot() now strips line breaks in test
descriptions (@LDSamson, #1900), and errors when called
from a test_that() that has an empty description (2,
#1980).
expect_true() and expect_false() give
better errors if actual isn’t a vector (#1996).
expect_visible() and expect_invisible()
have clearer failure messages (#1966).
local_reproducible_output() (used in
test_that() blocks) now sets LANGUAGE to
"C" instead of "en" to disable translations,
avoiding warnings on some platforms (#1925).
skip_if_not_installed() generates a clearer message
that sorts better (@MichaelChirico, #1959).
with_mock() and local_mock() have been
unconditionally deprecated as they will no longer work in future
versions of R (#1999).
Fix incorrect format string detected by latest R-devel. Fix thanks to Tomas Kalibera.
expect_snapshot() handles unexpected errors like
errors outside of snapshots, i.e. they terminate the entire test and get
a traceback (#1906).
JunitReporter() now uses ensures numeric values are
saved the xml file with . as decimal separator. (@maksymiuks,
#1660)
local_mocked_bindings() can now mock any object, not
just functions (#1896).
skip_if_offline() now uses
captive.apple.com by default. This is the hostname that
Apple devices use to check that they’re online so it should have a
higher reliability than r-project.org (@jdblischak,
#1890).
test_file(desc = ) will now find
describe() tests as well as test_that() tests
(#1903).
is_informative_error() and the wrap
argument to test_dir() and friends are now
defunct.
expect_no_error(), expect_no_warning(),
expect_no_message(), expect_no_condition(),
local_mocked_bindings(), and
with_mocked_bindings() are now stable, not
experimental.
All packages, regardless of whether or not they use rlang 1.0.0, now use the new snapshot display for errors, warnings, and messages (#1856). This no longer shows the class name, instead focussing on a display that more closely mimics what you’ll see interactively, including showing the error call.
testthat uses an improved algorithm for finding the srcref
associated with an expectation/error/warning/skip. It now looks for the
most recent call that has known source and is found inside the
test_that() call. This generally gives more specific
locations than the previous approach and gives much better locations if
an error occurs in an exit handler.
Helpers are no longer run twice.
expect_setequal() correctly displays results when
only one of actual and expected is missing elements (#1835).
expect_snapshot() and friends no longer create a
temporary file on every invocation.
expect_snapshot_file() now generates clickable links
to review changes (#1821).
expect_snapshot_value() has an improved error if the
object can’t be safely serialized using the specified style
(#1771).
options(rlang_interactive = TRUE) no longer causes
skip_on_cran() to not run on CRAN (#1868).
skip_if_offline() now errors if you don’t have curl
installed (#1854).
StopReporter gains the ability to suppress praise
when a test passes.
ProgressReporter now uses is a two characters wide
skip column in order to have a consistent width when 10 or more tests
are skipped in a single file (@mgirlich, #1844).
test_file() gains a desc argument which
allows you to run a single test from a file (#1776).
Fix for upcoming R-devel release.
testthat now sets the
_R_CHECK_BROWSER_NONINTERACTIVE_ environment variable when
running tests. This should ensure that left-over browser()
statements will trigger an error if encountered while running tests.
This functionality is only enabled with R (>= 4.3.0).
(#1825)
New expect_contains() and expect_in()
that works similarly to
expect_true(all(expected %in% object)) or
expect_true(all(object %in% expected)) but give more
informative failure messages (#1346).
New is_snapshot() returns TRUE if code
is running inside a snapshot test (#1796) and is_checking()
returns TRUE if test is running inside of
R CMD check (#1795)
ProgressReporter only reports the run time of test
files that take longer than 1s, rather than 0.1s. (#1806) and
re-displays all failures at the end of the results. Skips are now only
shown at the end of reporter summaries, not as tests are run. This makes
them less intrusive in interactive tests while still allowing you to
verify that the correct tests are skipped (#1801). When using parallel
tests, links to failed tests (#1787) and links to accept/review snapshot
(#1802) now work.
set_state_inspector() allows to to register a
function that’s called before and after every test, reporting on any
differences. This is very useful for detecting if any of your tests have
made changes to global state (like options, env vars, or connections)
(#1674). This function was inspired by renv’s testing
infrastructure.
skip_on_cran() no longer skips (errors) when run
interactively.
teardown_env() works in more cases.
testthat no longer truncates tracebacks and uses rlang’s default tree display.
expect_snapshot() differences no longer use
quotes.
expect_error(), expect_warning(), and
expect_message() now correctly enforce that the condition
is of the expected base class (e.g. error, warning, message) even when
the class argument is used (#1168).
it() now calls local_test_context() so
that it behaves more similarly to test_that() (#1731), and
is now exported so that you can more easily run BDD tests interactively
(#1587)
skip_on_bioc() now uses the documented environment
variable (IS_BIOC_BUILD_MACHINE) (#1712).
source_file(), which is used by various parts of the
helper and setup/teardown machinery, now reports the file name in the
case of errors (#1704).
test_path() now works when called within helper
files (#1562).
New vignette("special-files") describes the various
special files that testthat uses (#1638).
with_mocked_bindings() and
local_mocked_bindings() now also bind in the imports
namespace and can mock S3 methods. These changes make them good
substitutes for the deprecated functions with_mock() and
local_mock(), so those older functions now recommend
switching to the newer equivalents instead of using the mockr or mockery
packages.
expect_setequal() gives more actionable feedback
(#1657).
expect_snapshot() no longer elides new lines when
run interactively (#1726).
Experimental new with_mocked_bindings() and
local_mocked_bindings() (#1739).
sprintf().Deprecation warnings are no longer captured by
expect_warning(code, NA),
expect_no_warning(code), or
expect_silent(code). This ensures that they bubble up to
the top level so that you can address them (#1680). If you want to
assert that code does not throw a deprecation warning, use
expect_no_condition(code(), class = "lifecycle_warning_deprecation").
New experimental expect_no_error(),
expect_no_warning(), expect_no_message(), and
expect_no_condition() for asserting the code runs without
an error, warning, message, or condition (#1679).
Fixed a warning in R >=4.2.0 on Windows that occurred when using the C++ testing infrastructure that testthat provides (#1672).
Fixed an issue that could prevent compilation of Catch unit tests
with LLVM 15. In the interim, packages needing a local workaround can
set PKG_CPPFLAGS = -DCATCH_CONFIG_CPP11_NO_SHUFFLE in their
src/Makevars. (2, #1687)
Improve way capture_output() handles encoding thanks
to suggestion from Kurt Hornik (#1693). This means that snapshots using
UTF-8 encoded text on windows work once again.
local_reproducible_output() will no longer attempt
to set the local language when LANG='C' is set or an R
version is used that was not compiled with natural language support
(NLS), which would previously emit non-test-related warnings during
testing (@dgkf, #1662;
@heavywatal,
#1689).
test_check() now suppresses hyperlinks since they’ll
take you to the wrong places (#1648).
New set_max_fails() helper to make it easier to set
the maximum number of failures before stopping the test suite. And the
advice to set to Inf is now clickable (#1628).
You can now configure the behaviour of the implicit
devtools::load_all() call performed by
devtools::test() in your package DESCRIPTION file (#1636).
To disable exports of internal functions and of testthat helpers,
use:
Config/testthat/load-all: list(export_all = FALSE, helpers = FALSE)Helpers are now attached on the search path by default after calling
devtools::test().
Package that explicitly depend on rlang in their description file are now opting into a new snapshot display for errors, warnings, and messages. Previously this only concerned packages that explicitly depended on rlang >= 1.0.0. This display will eventually become the default for all packages.
Changes include:
Condition classes are no longer included in the snapshot by default. This is to avoid snapshot noise when upstream code adds or changes a class. For instance, r-devel has added classes to base errors.
Warnings and errors are now printed with rlang, including the
call field. This makes it easy to monitor the full
appearance of warning and error messages as they are displayed to
users.
This change is part of a push towards mentioning the useful context of an error as part of messages, see the release notes of rlang 1.0.0 for more about this.
Test results show hyperlinks to failed expectation when supported (#1544).
testthat now uses brio for all reading and writing (#1120). This ensures that snapshots always use “” to separate lines (#1516).
expect_snapshot() no longer inadvertently trims
trailing new lines off of errors and messages (#1509).
If expect_snapshot() generates a snapshot with
different value but still compares as equal (e.g. because you’ve set a
numeric tolerance), the saved values no longer update if another
snapshot in the same file changes.
expect_snapshot() now only adds a .new
file for the variants that actually changed, not all variants, while
expect_snapshot_file() with variant with no longer
immediately deletes .new files (#1468).
expect_snapshot_file() gains a
transform argument to match expect_snapshot()
(#1474). compare now defaults to NULL,
automatically guessing the comparison type based on the
extension.
expect_snapshot_file() now errors if the file being
snapshot does not exist; SnapshotReporter also now treats
the file directory as an absolute path (#1476, @malcolmbarrett)
New expect_snapshot_warning() to match
expect_snapshot_error() (#1532).
JUnitReporter now includes skip messages/reasons
(@rfineman,
#1507)
local_reproducible_output() gains a
lang argument so that you can optionally override the
language used to translate error messages (#1483). It also sets the
global option cli.num_colors in addition to
crayon.enabled.
test_that() no longer inappropriately skips when
calling expect_equal() when you’ve temporarily set the
locale to non-UTF-8 (#1285).
skip_if_offline() now automatically calls
skip_on_cran() (#1479).
snapshot_accept() and snapshot_review()
now work with exactly the same file specification which can be a
snapshot name, a file name, or a directory (#1546). They both work
better with variants (#1508). Snapshot cleanup also removes all empty
directories (#1457).
When a snapshot changes the hint also mentions that you can use
snapshot_review() (#1500, @DanChaltiel) and the message tells you
what variant is active (#1540).
JUnit reporter now includes skip messages/reasons (@rfineman, #1507).
Condition expectations like expect_error() now match
across the ancestry of chained errors (#1493). You can disable this by
setting the new inherit argument to
FALSE.
Added preliminary support for rlang 1.0 errors. It is disabled by
default for the time being. To activate it, specify
rlang (>= 1.0.0) in your DESCRIPTION file
(or >= 0.99.0.9001 if you’re using the dev version).
Once activated, snapshots will now use rlang to print error and
warning messages, including the Error: and
Warning: prefixes. This means the call field
of conditions is now displayed in snapshots if present. Parent error
messages are also displayed. Following this change, all snapshots
including error and warning messages need to be revalidated.
We will enable the new rlang 1.0 output unconditionally in a future release.
expect_snapshot() gains a new argument
cnd_class to control whether to show the class of errors,
warnings, and messages.
The default is currently unchanged so that condition classes keep
being included in snapshots. However, we plan to change the default to
FALSE in an upcoming release to prevent distracting
snapshot diffing as upstream packages add error classes. For instance,
the development version of R is currently adding classes to basic
errors, which causes spurious snapshot changes when testing against
R-devel on CI.
If you depend on rlang 1.0 (see above), the default is already set to
FALSE.
expect_snapshot() no longer processes rlang
injection operators like !!.
Fixed bug in expectations with long inputs that use
:: (#1472).
expect_snapshot() is no longer
experimental.
expect_snapshot() and friends gets an experimental
new variant argument which causes the snapshot to be saved
in _snaps/{variant}/{test}.md instead of
_snaps/{test}.md. This allows you to generate (and compare)
unique snapshots for different scenarios like operating system or R
version (#1143).
expect_snapshot() gains a transform
argument, which should be a function that takes a character vector of
lines and returns a modified character vector of lines. This makes it
easy to remove sensitive (e.g. API keys) or stochastic (e.g. random
temporary directory names) from snapshot output (#1345).
expect_snapshot_file() now replaces previous
.new snapshot if code fails again with a different
value.
expect_snapshot_value() now has an explicit
tolerance argument which uses the testthat default, thus
making it more like expect_equal() rather than
expect_identical(). Set it to NULL if you want
precise comparisons (#1309).
expect_snapshot_value(style = "deparse") now works with
negative values (#1342).
If a test containing multiple snapshots fails (or skips) in between snapshots, the later snapshots are now silently restored. (Previously this warned and reset all snapshots, not just later snapshots).
If you have multiple tests with the same name that use snapshots (not a good idea), you will no longer get a warning. Instead the snapshots will be aggregated across the tests.
Condition expectations now consistently return the expected
condition instead of the return value (#1371). Previously, they would
only return the condition if the return value was NULL,
leading to inconsistent behaviour.
This is a breaking change to the 3rd edition. Where you could previously do:
expect_equal(expect_warning(f(), "warning"), "value")You must now use condition expectations on the outside:
expect_warning(expect_equal(f(), "value"), "warning")
# Equivalently, save the value before inspection
expect_warning(value <- f(), "warning")
expect_equal(value, "value")This breaking change makes testthat more consistent. It also makes it possible to inspect both the value and the warning, which would otherwise require additional tools.
Errors in test blocks now display the call if stored in the condition object (#1418). Uncaught errors now show their class (#1426).
Multi-line skips only show the first line in the skip summary.
expr_label(), which is used to concisely describe
expressions used in expectations, now does a better job of summarising
infix function (#1442).
local_reproducible_output() now sets the
max.print option to 99999 (the default), so your tests are
unaffected by any changes you might’ve made in your
.Rprofile (1367).
ProgressReporter (the default only) now stops at the
end of a file; this ensures that you see the results of all related
tests, and ensures that snapshots are handled consistently
(#1402).
ProgressReporter now uses an env var to adjust the
maximum number of failures. This makes it easier to adjust when the
tests are run in a subprocess, as is common when using RStudio
(#1450).
skip_on_os() gains an arch argument so
you can also choose to skip selected architectures (#1421).
test_that() now correctly errors when an expectation
fails when run interactively (#1430).
test_that() now automatically and correctly generate
an “empty test” skip if it only generates warnings or messages (and
doesn’t contain any expectations).
testthat_tolerance() no longer has an unused
argument.
The vendored Catch code used for use_catch() now
uses a constant value for the stack size rather than relying on
SIGSTKSZ. This fixes compatibility for recent glibc versions where
SIGSTKSZ is no longer a constant.
Fixed an issue that caused errors and early termination of tests on R <= 3.6 when a failing condition expectation was signalled inside a snapshot.
expect_snapshot_file() gains a compare
argument (#1378, @nbenn). This is a customisation point for how
to compare old and new snapshot files.
The functions compare_file_binary() and
compare_file_text() are now exported from testthat to be
supplied as compare argument. These implement the same
behaviour as the old binary argument which is now
deprecated.
expect_snapshot() no longer deletes snapshots when
an unexpected error occurs.
New announce_snapshot_file() function for developers
of testthat extensions. Announcing a snapshot file allows testthat to
preserve files that were not generated because of an unexpected error or
a skip() (#1393). Unannounced files are automatically
deleted during cleanup if the generating code isn’t called.
New expectation: expect_no_match(). It complements
expect_match() by checking if a string doesn’t
match a regular expression (@michaelquinn32, #1381).
Support setting the testthat edition via an environment variable
(TESTTHAT_EDITION) as well (@michaelquinn32, #1386).
Failing expectations now include a backtrace when they’re not
called directly from within test_that() but are instead
wrapped in some helper function (#1307).
CheckReporter now only records warnings when not on
CRAN. Otherwise failed CRAN revdep checks tend to be cluttered up with
warnings (#1300). It automatically cleans up
testthat-problems.rds left over from previous runs if the
latest run is successful (#1314).
expect_s3_class() and expect_s4_class()
can now check that an object isn’t an S3 or S4 object by
supplying NA to the second argument (#1321).
expect_s3_class() and expect_s4_class()
format class names in a less confusing way (#1322).
expect_snapshot() collapses multiple adjacent
headings of the same, so that, e.g., if you have multiple lines of code
in a row, you’ll only see one “Code:” heading (#1311).
New testthat.progress.verbose_skips option. Set to
FALSE to stop reporting skips as they occur; they will
still appear in the summary (#1209, @krlmlr).
CheckReporter results have been tweaked based on
experiences from running R CMD check on many packages. Hopefully it
should now be easier to see the biggest problems (i.e. failures and
errors) while still having skips and warnings available to check if
needed (#1274). And now the full test name is always shown, no matter
how long (#1268).
Catch C++ tests are no longer reported multiple times (#1237) and
are automatically skipped on Solaris since Catch is not supported
(#1257). use_catch() makes it more clear that your package
needs to suggest xml2 (#1235).
auto_test_package() works once again (@mbojan, #1211,
#1214).
expect_snapshot() gains new error
argument which controls whether or not an error is expected. If an
unexpected error is thrown, or an expected error is not thrown,
expect_snapshot() will fail (even on CRAN)
(#1200).
expect_snapshot_value(style = "deparse") handles
more common R data structures.
expect_snapshot_value() now passes ...
on to waldo::compare() (#1222).
expect_snapshot_file() gives a hint as to next steps
when a failure occurs in non-interactive environments (with help from
@maelle, #1179).
expect_snapshot_*() gives a more informative hint when
you’re running tests interactively (#1226).
expect_snapshot_*() automatically removes the
_snaps directory if it’s empty (#1180). It also warns if
snapshots are discarded because tests have duplicated names (#1278,
@krlmlr).
local_reproducible_output() now sets the LANGUAGE
env var to “en”. This matches the behaviour of R CMD check in
interactive settings (#1213). It also now unsets RSTUDIO envvar, instead
of setting it to 0 (#1225).
RstudioReporter has been renamed to
RStudioReporter.
skip_if_not() no longer appends “is not TRUE” to
custom messages (@dpprdan, #1247).
test_that() now warns (3e only) if code doesn’t have
braces, since that makes it hard to track the source of an error (#1280,
@krlmlr).
testhat 3.0.0 brings with it a 3rd edition that makes a number of
breaking changes in order to clean up the interface and help you use our
latest recommendations. To opt-in to the 3rd edition for your package,
set Config/testthat/edition: 3 in your
DESCRIPTION or use local_edition(3) in
individual tests. You can retrieve the active edition with
edition_get(). Learn more in
vignette("third-edition").
context() is deprecated.
expect_identical() and expect_equal()
use waldo::compare() to compare actual and expected
results. This mostly yields much more informative output when the actual
and expected values are different, but while writing it uncovered some
bugs in the existing comparison code.
expect_error(), expect_warning(),
expect_message(), and expect_condition() now
all use the same underlying logic: they capture the first condition that
matches class/regexp and allow anything else
to bubble up (#998/#1052). They also warn if there are unexpected
arguments that are never used.
The all argument to expect_message()
and expect_warning() is now deprecated. It was never a
particularly good idea or well documented, and is now superseded by the
new condition capturing behaviour.
expect_equivalent(),
expect_reference(), expect_is() and
expect_that() are deprecated.
Messages are no longer automatically silenced. Either use
suppressMessages() to hide unimportant messages, or
expect_message() to catch important messages
(#1095).
setup() and teardown() are deprecated
in favour of test fixtures. See vignette("test-fixtures")
for more details.
expect_known_output(),
expect_known_value(), expect_known_hash(), and
expect_equal_to_reference() are all deprecated in favour of
expect_snapshot_output() and
expect_snapshot_value().
test_that() now sets a number of options and env
vars to make output as reproducible as possible (#1044). Many of these
options were previously set in various places (in
devtools::test(), test_dir(),
test_file(), or verify_output()) but they have
now been centralised. You can use in your own code, or when debugging
tests interactively with local_test_context().
with_mock() and local_mock() are
deprecated; please use the mockr or mockery packages instead
(#1099).
New family of snapshot expectations (expect_snapshot(),
expect_snapshot_output(),
expect_snapshot_error(), and
expect_snapshot_value()) provide “snapshot” tests, where
the expected results are stored in separate files in
test/testthat/_snaps. They’re useful whenever it’s painful
to store expected results directly in the test files.
expect_snapshot_file() along with
snapshot_review() help snapshot more complex data, with
initial support for text files, images, and data frames (#1050).
See vignette("snapshotting") for more details.
CheckReporter (used inside R CMD check) now prints
out all problems (i.e. errors, failures, warnings and skips; and not
just the first 10), lists skips types, and records problems in machine
readable format in tests/testthat-problems.rds
(#1075).
New CompactProgressReporter tweaks the output of
ProgressReporter for use with a single file, as in
devtools::test_file(). You can pick a different default by
setting testthat.default_compact_reporter to the name of a
reporter.
ProgressReporter (the default reporter) now keeps
the stack traces of an errors that happen before the before test, making
problems substantially easier to track down (#1004). It checks if you’ve
exceeded the maximum number of failures (from option
testthat.progress.max_fails) after each expectation, rather
than at the end of each file (#967). It also gains new random praise
options that use emoji, and lists skipped tests by type
(#1028).
StopReporter adds random praise emoji when a single
test passes (#1094). It has more refined display of failures, now using
the same style as CompactProgressReporter and
ProgressReporter.
SummaryReporter now records file start, not just
context start. This makes it more compatible with modern style which
does not use context() (#1089).
All reporters now use exactly the same format when reporting the location of an expectation.
Warnings now include a backtrace, making it easier to figure out where they came from.
Catch C++ tests now provide detailed results for each test. To
upgrade existing code, re-run testthat::use_catch()
(#1008).
Many reporters (e.g. the check reporter) no longer raise an error
when any tests fail. Use the stop_on_failure argument to
devtools::test() and testthat::test_dir() if
your code relies on this. Alternatively, use
reporter = c("check", "fail") to e.g. create a failing
check reporter.
New vignette("test-fixtures") describes test
fixtures; i.e. how to temporarily and cleanly change global state in
order to test parts of your code that otherwise would be hard to run
(#1042). setup() and teardown() are superseded
in favour of test fixtures.
New teardown_env() for use with
withr::defer(). This allows you to run code after all other
tests have been run.
New vignette("skipping") gives more general
information on skipping tests, include some basics on testing skipping
helpers (#1060).
ProgressReporter() and CheckReporter()
list the number of skipped tests by reason at the end of the reporter.
This makes it easier to check that you’re not skipping the wrong tests,
particularly on CI services (#1028).
test_that() no longer triggers an error when run
outside of tests; instead it produces a more informative summary of all
failures, errors, warnings, and skips that occurred inside the
test.
test_that() now errors if desc is not a
string (#1161).
test_file() now runs helper, setup, and teardown
code, and has the same arguments as test_dir() (#968). Long
deprecated encoding argument has been removed.
test_dir() now defaults stop_on_failure
to TRUE for consistency with other test_
functions. The wrap argument has been deprecated; it’s not
clear that it should ever have been exposed.
New test_local() tests a local source package
directory. It’s equivalent to devtools::test() but doesn’t
require devtools and all its dependencies to be installed
(#1030).
testthat no longer supports tests stored in
inst/tests. This has been deprecated since testthat 0.11.0
(released in 2015). test_package() (previously used for
running tests in R CMD check) will fail silently if no tests are found
to avoid breaking old packages on CRAN (#1149).
capture_output() and verify_output()
use a new testthat_print() generic. This allows you to
control the printed representation of your object specifically for tests
(i.e. if your usual print method shows data that varies in a way that
you don’t care about for tests) (#1056).
context_start_file() is now exported for external
reporters (#983, #1082). It now only strips first instance of
prefix/suffix (#1041, @stufield).
expect_error() no longer encourages you to use
class. This advice removes one type of fragility at the
expense of creating a different type (#1013).
expect_known_failure() has been removed. As far as I
can tell it was only ever used by testthat, and is rather
fragile.
expect_true(), expect_false(), and
expect_null() now use waldo to produce more informative
failures.
verify_output() no longer always fails if output
contains a carriage return character (“) (#1048). It uses the
pdf() device instead of png() so it works on
systems without X11 (#1011). And it uses waldo::compare()
to give more informative failures.
The last version of testthat introduced a performance regression in error assertions (#963). To fix it, you need to install rlang 0.4.2.
Fixed error assertions with rJava errors (#964).
Fixed issue where error and warning messages were not retrieved
with conditionMessage() under certain
circumstances.
This release mostly focusses on an overhaul of how testthat works with conditions (i.e. errors, warnings and messages). There are relatively few user-facing changes, although you should now see more informative backtraces from errors and failures.
Unexpected errors are now printed with a simplified backtrace.
expect_error() and expect_condition()
now display a backtrace when the error doesn’t conform to expectations
(#729).
expect_error(), expect_warning() and
expect_message() now call conditionMessage()
to get the condition message. This generic makes it possible to generate
messages at print-time rather than signal-time.
expect_error() gets a better warning message when
you test for a custom error class with regexp.
New exp_signal() function is a condition signaller
that implements the testthat protocol (signal with stop()
if the expectation is broken, with a continue_test
restart).
Existence of restarts is first checked before invocation. This makes it possible to signal warnings or messages with a different condition signaller (#874).
ListReporter now tracks expectations and errors,
even when they occur outside of tests. This ensures that
stop_on_failure matches the results displayed by the
reporter (#936).
You can silence warnings about untested error classes by
implementing a method for is_uninformative_warning(). This
method should be lazily registered, e.g. with
vctrs::s3_register(). This is useful for introducing an
experimental error class without encouraging users to depend on the
class in their tests.
Respect options(warn = -1) to ignore all warnings (@jeroen #958).
Expectations can now be explicitly subclassed with
new_expectation(). This constructor follows our new
conventions for S3 classes and takes an optional subclass and optional
attributes.
Unquoted inputs no longer potentially generate multiple test messages (#929).
verify_output() no longer uses quasiquotation, which
fixes issues when verifying the output of tidy eval functions
(#945).
verify_output() gains a unicode
parameter to turn on or off the use of Unicode characters by the cli
package. It is disabled by default to prevent the tests from failing on
platforms like Windows that don’t support UTF-8 (which could be your
contributors’ or your CI machines).
verify_output() now correctly handles multi-line
condition messages.
verify_output() now adds spacing after condition
messages, consistent with the spacing added after normal
output.
verify_output() has a new syntax for inserting
headers in output files: insert a "# Header" string
(starting with # as in Markdown) to add a header to a set
of outputs.
compare.numeric() uses a more sophisticated default
tolerance that will automatically skip tests that rely on numeric
tolerance if long doubles are not available (#940).
JunitReporter now reports tests in ISO 8601 in the
UTC timezone and uses the maximum precision of 3 decimal places
(#923).
test_rd() and add a couple of
tests to hopefully detect the problem earlier in the future.verify_output() is designed for testing output
aimed at humans (most commonly print methods and error messages). It is
a regression test that saves output in a way that makes it easy to
review. It is automatically skipped on CRAN (#782, #834).as.data.frame.testthat_results() now always returns
a data frame with 13 columns (@jozefhajnala, #887).
auto_test_package() now correctly handles helper
files (tests/testthat/helper-*.R), automatically reloading
all code and rerunning all tests (@CorradoLanera, #376, #896).
expect_match() now displays info even
when match length is 0 (#867).
expect_s3_class() gains new exact
argument that allows you to check for an exact class match, not just
inheritance (#885).
fail() and succeed() gain
info argument, which is passed along to
expect().
test_examples() gets some minor fixes: it now
returns the results invisibly, doesn’t assume that examples should
contain tests, and documents that you shouldn’t be using it routinely
(#841).
test_file() only calls
Reporter$end_context() if a context was started, fixing an
error in TeamcityReporter (@atheriel, #883).
skip() now reports reason for skipping as:
Reason: {skip condition} (@patr1ckm, #868).
skip_if() and skip_if_not() now report
Reason: {skip condition} is TRUE and
Reason: {skip condition} is not TRUE respectively (@
patr1ckm, #868).
skip_if_translated() now tests for translation of a
specific message. This is more robust than the previous approach because
translation happens message-by-message, not necessarily for the entire
session (#879) (and in general, it’s impossible to determine what
language R is currently using).
skip_on_covr() allows you to skip tests when covr is
running. (@ianmcook,
#895)
expect_known_value() gains a new serialisation
version argument, defaulting to 2. Prevents the
.rds files created to hold reference objects from making a
package appear to require R >= 3.5 (#888 @jennybc).
New expect_visible() and
expect_invisible() make it easier to check if a function
call returns its result visibly or invisibly (#719).
New expect_mapequal(x, y) checks that x
and y have the same names, and the same value associated
with each name (i.e. they compare the values of the vector standardising
the order of the names) (#863).
New expect_vector() is a wrapper around
vctrs::vec_assert() making it easy to test against the
vctrs definitions of prototype and size (#846). (Currently requires
development version of vctrs.)
All expectations give clearer error messages if you forget the
object or expected arguments (#743).
expect_equal() now correctly compares infinite
values (#789).
In expect_equal_to_reference(), the default value
for update is now FALSE (@BrodieG, #683).
expect_error() now returns the error object as
documentated (#724). It also now warns if you’re using a classed
expectation and you’re not using the class argument. This
is good practice as it decouples the error object (which tends to be
stable) from its rendering to the user (which tends to be fragile)
(#816).
expect_identical() gains a ... argument
to pass additional arguments down to identical()
(#714).
expect_lt(), expect_lte(),
expect_gt() expect_gte() now handle
Inf and NA arguments appropriately (#732), and
no longer require the inputs to be numeric.
expect_output() gains a width argument,
allowing you to control the output width. This does not inherit from
getOption("width"), ensuring that tests return the same
results regardless of environment (#805).
expect_setequal() now works with more vector types
(including lists), because it uses %in%, rather than
sort(). It also warns if the inputs are named, as this
suggests that your mental model of how expect_setequal()
works is wrong (#750).
is_true() and is_false() have been
deprecated because they conflict with other functions in the
tidyverse.
Reporter documentation has been considerably improved (#657).
CheckReporter, used by R CMD check, now includes a
count of warnings.
JUnitReporter no longer replaces . in
class names (#753), and creates output that should be more compatible
with Jenkins (#806, @comicfans).
ListReporter now records number of passed tests and
original results in new columns (#675).
ProgressReporter, the default reporter, now:
Automatically generates a context from the file name. We no
longer recommend the use of context() and instead encourage
you to delete it, allowing the context to be autogenerated from the file
name.
This also eliminates the error that occurred if tests can before the
first context() (#700, #705).
Gains a update_interval parameter to control how
often updates are printed (default 0.1 s). This prevents large printing
overhead for very fast tests. (#701, @jimhester)
Uses a 3 character wide column to display test successes, so up to 999 successful tests can be displayed without changing the alignment (#712).
reporter$end_reporter() is now only called when
testing completes successfully. This ensures that you don’t get
unnecessary output when the test fails partway through (#727).
skip_if_offline() skips tests if an internet
connection is not available (#685).
skip_on_ci() skips tests on continuous integration
systems (@mbjoseph,
#825) by looking for a CI env var..
New testthat_examples() and
testthat_example() make it easy to access new test files
bundled with the package. These are used in various examples to make it
easier to understand how to use the package.
New local_mock() which allows you to mock a function
without having to add an additional layer of indentation as with
with_mock() (#856).
auto_test_package() works better with recent
devtools and also watches src/ for changes (#809).
expect_s3_class() now works with unquoting (@jalsalam, #771).
expectation objects now contain the failure message,
even when successful (#836)
devtools::test() no longer fails if run multiple
times within the same R session for a package containing Catch tests.
(devtools
#1832)
New testing_package() retrieves the name of the
package currently being tested (#699).
run_testthat_tests C entrypoint is registered more
robustly.
skip() now always produces a message of
length 1, as expected elsewhere in testthat (#791).
Warnings are passed through even when
options(warn = 2) is set (@yutannihilation, #721).
“Can’t mock functions in base packages”: You can no longer use
with_mock() to mock functions in base packages, because
this no longer works in R-devel due to changes with the byte code
compiler. I recommend using mockery or mockr instead.
The order of arguments to expect_equivalent() and
expect_error() has changed slightly as both now pass
... on another function. This reveals itself with a number
of different errors, like:
If you see one of these errors, check the number, order, and names of arguments to the expectation.
“Failure: (unknown)”. The last release mistakenly failed to test
bare expectations not wrapped inside test_that(). If you
see “(unknown)” in a failure message, this is a failing expectation that
you previously weren’t seeing. As well as fixing the failure, please
also wrap inside a test_that() with an informative
name.
“Error: the argument has already been evaluated”: the way in
which expectations now need create labels has changed, which caused a
couple of failures with unusual usage when combined with
Reduce, lapply(), and Map().
Avoid these functions in favour of for loops. I also recommend reading
the section below on quasiquotation support in order to create more
informative failure messages.
expect_condition() works like
expect_error() but captures any condition, not just error
conditions (#621).
expect_error() gains a class argument
that allows you to make an assertion about the class of the error object
(#530).
expect_reference() checks if two names point to the
same object (#622).
expect_setequal() compares two sets (stored in
vectors), ignoring duplicates and differences in order (#528).
skip_if() makes it easy to skip a test when a
condition is true (#571). For example, use
skip_if(getRversion() <= 3.1) to skip a test in older R
versions.
skip_if_translated() skips tests if you’re running
in an locale where translations are likely to occur (#565). Use this to
avoid spurious failures when checking the text of error messages in
non-English locales.
skip_if_not_installed() gains new
minimum_version argument (#487, #499).
We have identified a useful family of expectations that compares the results of an expression to a known good value stored in a file. They are designed to be use in conjunction with git so that you can see what precisely has changed, and revert it if needed.
expect_known_output() replaces
expect_output_file(), which has been soft-deprecated. It
now defaults to update = TRUE and warn, rather than failing
on the first run. It gains a print argument to
automatically print the input (#627). It also sets the width option to
80 to ensure consistent output across environments (#514)
expect_known_value() replaces
expect_equal_to_reference(), which has been
soft-deprecated. It gains an update argument defaulting to
TRUE. This changes behaviour from the previous version, and
soft-deprecated expect_equal_to_reference() gets
update = FALSE.
expect_known_failure() stored and compares the
failure message from an expectation. It’s a useful regression test when
developing informative failure messages for your own
expectations.
All expectations can now use unquoting (#626). This makes it much easier to generate informative failure messages when running tests in a for loop.
For example take this test:
f <- function(i) if (i > 3) i * 9 else i * 10
for (i in 1:5) {
expect_equal(f(i), i * 10)
}When it fails, you’ll see the message
Error: `f(i)` not equal to `i * 10`. That’s hard to
diagnose because you don’t know which iteration caused the problem!
for (i in 1:5) {
expect_equal(f(!!i), !!(i * 10))
}If you unquote the values using !!, you get the failure
message `f(4L)` not equal to 40.. This is much easier to
diagnose! See ?quasi_label() for more details.
(Note that this is not tidy evaluation per se, but is closely related. At this time you can not unquote quosures.)
New setup() and teardown() functions
allow you to run at the start and end of each test file. This is useful
if you want to pair cleanup code with the code that messes up state
(#536).
Two new prefixes are recognised in the test/
directory. Files starting with setup are run before tests
(but unlike helpers are not run in
devtools::load_all()). Files starting with
teardown are run after all tests are completed
(#589).
All files are now read and written as UTF-8 (#510, #605).
is_testing() allows you to tell if your code is
being run inside a testing environment (#631). Rather than taking a
run-time dependency on testthat you may want to inline the function into
your own package:
is_testing <- function() {
identical(Sys.getenv("TESTTHAT"), "true")
}It’s frequently useful to combine with
interactive().
A new default reporter, ReporterProgress, produces more
aesthetically pleasing output and makes the most important information
available upfront (#529). You can return to the previous default by
setting options(testthat.default_reporter = "summary").
Output colours have been tweaked to be consistent with clang: warnings are now in magenta, and skips in blue.
New default_reporter() and
check_reporter() which returns the default reporters for
interactive and check environments (#504).
New DebugReporter that calls a better version of
recover() in case of failures, errors, or warnings (#360,
#470).
New JunitReporter generates reports in JUnit
compatible format. (#481, @lbartnik; #640, @nealrichardson; #575)
New LocationReporter which just prints the location
of every expectation. This is useful for locating segfaults and C/C++
breakpoints (#551).
SummaryReporter received a number of smaller
tweaks
Aborts testing as soon the limit given by the option
testthat.summary.max_reports (default 10) is reached
(#520).
New option testthat.summary.omit_dots = TRUE hides
the progress dots speeding up tests by a small amount (#502).
Bring back random praise and encouragement which I accidentally dropped (#478).
New option testthat.default_check_reporter, defaults
to "check". Continuous Integration system can set this
option before evaluating package test sources in order to direct test
result details to known location.
All reporters now accept a file argument on
initialization. If provided, reporters will write the test results to
that path. This output destination can also be controlled with the
option testthat.output_file (#635, @nealrichardson).
is_null() and matches() have been
deprecated because they conflict with other functions in the tidyverse
(#523).Updated Catch to 1.9.6. testthat now understands and
makes use of the package routine registration mechanism required by CRAN
with R >= 3.4.0.
Better reporting for deeply nested failures, limiting the stack trace to the first and last 10 entries (#474).
Bare expectations notify the reporter once again. This is
achieved by running all tests inside test_code() by default
(#427, #498). This behaviour can be overridden by setting
wrap = FALSE in test_dir() and friends
(#586).
auto_test() and auto_test_package()
provide hash parameter to enable switching to faster,
time-stamp-based modification detection (#598, @katrinleinweber).
auto_test_package() works correctly on windows
(#465).
capture_output_lines() is now exported
(#504).
compare.character() works correctly for vectors of
length > 5 (#513, @brodieG)
compare.default() gains a max_diffs
argument and defaults to printing out only the first 9 differences
(#538).
compare.numeric() respects
check.attributes() so expect_equivalent()
correctly ignores attributes of numeric vectors (#485).
Output expectations (expect_output(),
expect_message(), expect_warning(), and
expect_silent()) all invisibly return the first argument to
be consistent with the other expectations (#615).
expect_length() works with any object that has a
length method, not just vectors (#564, @nealrichardson)
expect_match() now accepts explicit
perl and fixed arguments, and adapts the
failure message to the value of fixed. This also affects
other expectations that forward to expect_match(), like
expect_output(), expect_message(),
expect_warning(), and expect_error().
expect_match() escapes special regular expression
characters when printing (#522, @jimhester).
expect_message(), expect_warning() and
expect_error() produce clearer failure messages.
find_test_scripts() only looks for
\.[rR] in the extension (#492, @brodieG)
test_dir(), test_package(),
test_check() unset the R_TESTS env var
(#603)
test_examples() now works with installed packages as
well as source packages (@jimhester, #532).
test_dir(), test_package(), and
test_check() gain stop_on_failure and
stop_on_waring arguments that control whether or not an
error is signalled if any tests fail or generate warnings (#609,
#619).
test_file() now triggers a gc() after
tests are run. This helps to ensure that finalisers are run earlier
(#535).
test_path() now generates correct path when called
from within tools::testInstalledPackage() (#542).
test_path() no longer assumes that the path exists
(#448).
test_that() calls without any expectations generate
a default skip() (#413).
test_dir() gains load_helpers argument
(#505).
show_failures() simply prints a failure if it
occurs. This makes it easier to show failures in examples.
with_mock() disallows mocking of functions in base
packages, because this doesn’t work with the current development version
of R (#553).
std::logic_error() constructed with
std::string() argument, to avoid build errors on
Solaris.New expect_output_file() to compare output of a
function with a text file, and optionally update it (#443, @krlmlr).
Properly scoped use + compilation of C++ unit testing code using
Catch to gcc and clang only, as Catch includes
code that does not strictly conform to the C++98 standard. (2)
Fixed an out-of-bounds memory access when routing Catch output
through Rprintf(). (2)
Ensure that unit tests run on R-oldrel (remove use of
dir.exists()).
Improved overriding of calls to exit() within Catch,
to ensure compatibility with GCC 6.0. (@krlmlr)
Hardened formatting of difference messages, previously the
presence of % characters could affect the output (#446,
@krlmlr).
Fixed errors in expect_equal() when comparing
numeric vectors with and without attributes (#453, @krlmlr).
auto_test() and auto_test_package()
show only the results of the current test run and not of previously
failed runs (#456, @krlmlr).
The expectation() function now expects an expectation
type (one of “success”, “failure”, “error”, “skip”, “warning”) as first
argument. If you’re creating your own expectations, you’ll need to use
expect() instead (#437).
The expectation system got a thorough overhaul (#217). This primarily makes it easier to add new expectations in the future, but also included a thorough review of the documentation, ensuring that related expectations are documented together, and have evocative names.
One useful change is that most expectations invisibly return the
input object. This makes it possible to chain together
expectations with magrittr:
factor("a") %>%
expect_type("integer") %>%
expect_s3_class("factor") %>%
expect_length(1)(And to make this style even easier, testthat now re-exports the pipe, #412).
The exception to this rule are the expectations that evaluate (i.e.
for messages, warnings, errors, output etc), which invisibly return
NULL. These functions are now more consistent: using
NA will cause a failure if there is a
errors/warnings/messages/output (i.e. they’re not missing), and will
NULL fail if there aren’t any
errors/warnings/messages/output. This previously didn’t work for
expect_output() (#323), and the error messages were
confusing with expect_error(..., NA) (#342, @nealrichardson +
@krlmlr, #317).
Another change is that expect_output() now requires you
to explicitly print the output if you want to test a print method:
expect_output("a", "a") will fail,
expect_output(print("a"), "a") will succeed.
There are six new expectations:
expect_type() checks the type of the object
(#316), expect_s3_class() tests that an object is S3 with
given class, expect_s4_class() tests that an object is S4
with given class (#373). I recommend using these more specific
expectations instead of the more general
expect_is().
expect_length() checks that an object has expected
length.
expect_success() and expect_failure()
are new expectations designed specifically for testing other
expectations (#368).
A number of older features have been deprecated:
expect_more_than() and
expect_less_than() have been deprecated. Please use
expect_gt() and expect_lt() instead.
takes_less_than() has been deprecated.
not() has been deprecated. Please use the explicit
individual forms expect_error(..., NA) ,
expect_warning(.., NA) and so on.
Now all expectations are also conditions, and R’s condition system is
used to signal failures and successes (#360, @krlmlr). All known conditions (currently,
“error”, “warning”, “message”, “failure”, and “success”) are converted
to expectations using the new as.expectation(). This allows
third-party test packages (such as assertthat,
testit, ensurer, checkmate,
assertive) to seamlessly establish testthat
compatibility by issuing custom error conditions (e.g.,
structure(list(message = "Error message"), class = c("customError", "error", "condition")))
and then implementing as.expectation.customError(). The
assertthat package contains an example.
The reporters system class has been considerably refactored to make existing reporters simpler and to make it easier to write new reporters. There are two main changes:
Reporters classes are now R6 classes instead of Reference Classes.
Each callbacks receive the full context:
add_results() is passed context and test as well as the
expectation.test_start() and test_end() both get the
context and test.context_start() and context_end() get the
context.Warnings are now captured and reported in most reporters.
The reporter output goes to the original standard output and is
not affected by sink() and expect_output()
(#420, @krlmlr).
The default summary reporter lists all warnings (#310), and all
skipped tests (@krlmlr, #343). New option
testthat.summary.max_reports limits the number of reports
printed by the summary reporter. The default is 15 (@krlmlr, #354).
MinimalReporter correct labels errors with E and
failures with F (#311).
New FailReporter to stop in case of failures or
errors after all tests (#308, @krlmlr).
New functions capture_output(),
capture_message(), and capture_warnings()
selectively capture function output. These are used in
expect_output(), expect_message() and
expect_warning() to allow other types out output to
percolate up (#410).
try_again() allows you to retry code multiple times
until it succeeds (#240).
test_file(), test_check(), and
test_package() now attach testthat so all testing functions
are available.
source_test_helpers() gets a useful default path:
the testthat tests directory. It defaults to the test_env()
to be consistent with the other source functions (#415).
test_file() now loads helpers in the test directory
before running the tests (#350).
test_path() makes it possible to create paths to
files in tests/testthat that work interactively and when
called from tests (#345).
Add skip_if_not() helper.
Add skip_on_bioc() helper (@thomasp85).
make_expectation() uses
expect_equal().
setup_test_dir() has been removed. If you used it
previously, instead use source_test_helpers() and
find_test_scripts().
source_file() exports the function testthat uses to
load files from disk.
test_that() returns a logical that
indicates if all tests were successful (#360, @krlmlr).
find_reporter() (and also all high-level testing
functions) support a vector of reporters. For more than one reporter, a
MultiReporter is created (#307, @krlmlr).
with_reporter() is used internally and gains new
argument start_end_reporter = TRUE (@krlmlr, 355).
set_reporter() returns old reporter invisibly (#358,
@krlmlr).
Comparing integers to non-numbers doesn’t raise errors anymore, and falls back to string comparison if objects have different lengths. Complex numbers are compared using the same routine (#309, @krlmlr).
compare.numeric() and
compare.character() received another overhaul. This should
improve behaviour of edge cases, and provides a strong foundation for
further work. Added compare.POSIXt() for better reporting
of datetime differences.
expect_identical() and
is_identical_to() now use compare() for more
detailed output of differences (#319, @krlmlr).
Added Catch
v1.2.1 for unit testing of C++ code. See ?use_catch() for
more details. (2)
Handle skipped tests in the TAP reporter (#262).
New expect_silent() ensures that code produces no
output, messages, or warnings (#261).
New expect_lt(), expect_lte(),
expect_gt() and expect_gte() for comparison
with or without equality (#305, @krlmlr).
expect_output(), expect_message(),
expect_warning(), and expect_error() now
accept NA as the second argument to indicate that output,
messages, warnings, and errors should be absent (#219).
Praise gets more diverse thanks to the praise package, and you’ll now get random encouragement if your tests don’t pass.
testthat no longer muffles warning messages. If you don’t want to
see them in your output, you need to explicitly quiet them, or use an
expectation that captures them (e.g. expect_warning()).
(#254)
Use tests in inst/tests is formally deprecated.
Please move them into tests/testthat instead
(#231).
expect_match() now encodes the match, as well as the
output, in the expectation message (#232).
expect_is() gives better failure message when
testing multiple inheritance,
e.g. expect_is(1:10, c("glm", "lm")) (#293).
Corrected argument order in compare.numeric()
(#294).
comparison() constructure now checks its arguments
are the correct type and length. This bugs a bug where tests failed with
an error like “values must be length 1, but FUN(X[[1]]) result is length
2” (#279).
Added skip_on_os(), to skip tests on specified
operating systems (2).
Skip test that depends on devtools if it is not
installed (#247, @krlmlr)
Added skip_on_appveyor() to skip tests on Appveyor
(@lmullen).
compare() shows detailed output of differences for
character vectors of different length (#274, @krlmlr).
Detailed output from expect_equal() doesn’t confuse
expected and actual values anymore (#274, @krlmlr).
Failure locations are now formatted as R error locations.
Add an ‘invert’ argument to find_tests_scripts().
This allows one to select only tests which do not match a
pattern. (#239, @jimhester).
Deprecated library_if_available() has been
removed.
test (test_dir(), test_file(),
test_package(), test_check()) functions now
return a testthat_results object that contains all results,
and can be printed or converted to data frame.
test_dir(), test_package(), and
test_check() have an added ... argument that
allows filtering of test files using, e.g., Perl-style regular
expressions,or fixed character filtering. Arguments in
... are passed to grepl() (@leeper).
test_check() uses a new reporter specifically
designed for R CMD check. It displays a summary at the end
of the tests, designed to be <13 lines long so test failures in
R CMD check display something more useful. This will
hopefully stop BDR from calling testthat a “test obfuscation suite”
(#201).
compare() is now documented and exported. Added a
numeric method so when long numeric vectors don’t match you’ll see some
examples of where the problem is (#177). The line spacing in
compare.character() was tweaked.
skip_if_not_installed() skips tests if a package
isn’t installed (#192).
expect_that(a, equals(b)) style of testing has been
soft-deprecated. It will keep working, but it’s no longer demonstrated
any where, and new expectations will only be available in
expect_equal(a, b) style. (#172)
Once again, testthat suppresses messages and warnings in tests (#189)
New test_examples() lets you run package examples as
tests. Each example counts as one expectation and it succeeds if the
code runs without errors (#204).
New succeed() expectation always succeeds.
skip_on_travis() allows you to skip tests when run
on Travis CI. (Thanks to @mllg)
colourise() was removed. (Colour is still supported,
via the crayon package.)
Mocks can now access values local to the call of
with_mock (#193, @krlmlr).
All equality expectations are now documented together (#173); all matching expectations are also documented together.
BDD: testhat now comes with an initial behaviour driven
development (BDD) interface. The language is similar to RSpec for Ruby
or Mocha for JavaScript. BDD tests read like sentences, so they should
make it easier to understand the specification of a function. See
?describe() for further information and examples.
It’s now possible to skip() a test with an
informative message - this is useful when tests are only available under
certain conditions, as when not on CRAN, or when an internet connection
is available (#141).
skip_on_cran() allows you to skip tests when run on
CRAN. To take advantage of this code, you’ll need either to use
devtools, or run Sys.setenv(NOT_CRAN = "true"))
Simple mocking: with_mock() makes it easy to
temporarily replace functions defined in packages. This is useful for
testing code that relies on functions that are slow, have unintended
side effects or access resources that may not be available when testing
(#159, @krlmlr).
A new expectation, expect_equal_to_reference() has
been added. It tests for equality to a reference value stored in a file
(#148, @jonclayden).
auto_test_package() works once more, and now uses
devtools::load_all() for higher fidelity loading (#138,
#151).
Bug in compare.character() fixed, as reported by
Georgi Boshnakov.
colourise() now uses option
testthat.use_colours (default: TRUE). If it is
FALSE, output is not colourised (#153, @mbojan).
is_identical_to() only calls
all.equal() to generate an informative error message if the
two objects are not identical (#165).
safe_digest() uses a better strategy, and returns NA
for directories (#138, #146).
Random praise is renabled by default (again!) (#164).
Teamcity reporter now correctly escapes output messages (#150, @windelinckx). It also uses nested suites to include test names.
library_if_available() has been deprecated.Better default environment for test_check() and
test_package() which allows S4 class creation in
tests
compare.character() no longer fails when one value
is missing.
testthat 0.8 comes with a new recommended structure for storing your
tests. To better meet CRAN recommended practices, testthat now recommend
that you to put your tests in tests/testthat, instead of
inst/tests (this makes it possible for users to choose
whether or not to install tests). With this new structure, you’ll need
to use test_check() instead of test_packages()
in the test file (usually tests/testthat.R) that runs all
testthat unit tests.
The other big improvement to usability comes from @kforner, who contributed
code to allow the default results (i.e. those produced by
SummaryReporter) to include source references so you can
see exactly where failures occurred.
MultiReporter, which combines several reporters into
one. (Thanks to @kforner)
ListReporter, which captures all test results with
their file, context, test and elapsed time. test_dir,
test_file, test_package and
test_check now use the ListReporter to
invisibly return a summary of the tests as a data frame. (Thanks to
@kforner)
TeamCityReporter to produce output compatible with
the TeamCity continuous integration environment. (Thanks to @windelinckx)
SilentReporter so that testthat can
test calls to test_that. (Thanks to @craigcitro, #83)
expect_null() and is_null to check if
an object is NULL (#78)
expect_named() and has_names() to check
the names of a vector (#79)
expect_more_than(), is_more_than(),
expect_less_than(), is_less_than() to check
values above or below a threshold. (#77, thanks to @jknowles)
expect_that() (and thus all expect_*
functions) now invisibly return the expectation result, and stops if
info or label arguments have length > 1 (thanks to @kforner)
fixed two bugs with source_dir(): it did not look for the source
scripts at the right place, and it did not use its chdir
argument.
When using expect_equal() to compare strings, the
default output for failure provides a lot more information, which should
hopefully help make finding string mismatches easier.
SummaryReporter has a max_reports
option to limit the number of detailed failure reports to show. (Thanks
to @crowding)
Tracebacks will now also contain information about where the functions came from (where that information is available).
matches and expect_match now pass
additional arguments on to grepl so that you can use
fixed = TRUE, perl = TRUE or
ignore.case = TRUE to control details of the match.
expect_match now correctly fails to match NULL.
(#100)
expect_output, expect_message,
expect_warning and expect_error also pass … on
to grepl, so that you can use fixed = TRUE,
perl = TRUE or ignore.case = TRUE
Removed stringr and evaluate
dependencies.
The not() function makes it possible to negate
tests. For example, expect_that(f(), not(throws_error()))
asserts that f() does not throw an error.
Make dir_state less race-y. (Thanks to @craigcitro,
#80)
auto_test now pays attention to its ‘reporter’
argument (Thanks to @crowding, #81)
get_reporter(), set_reporter() and
with_reporter() are now exported (#102)
Ignore attributes in is_true and
is_false (#49)
make_expectation works for more types of input
(#52)
Now works better with evaluate 0.4.3.
new fail() function always forces a failure in a
test. Suggested by Richie Cotton (#47)
Added TapReporter to produce output compatible with
the “test anything protocol”. Contributed by Dan Keshet.
Fixed where auto_test would identify the wrong files
as having changed. (Thanks to Peter Meilstrup)
SummaryReporter: still return informative messages
even if no tests defined (just bare expectations). (Fixes #31)
Improvements to reference classes (Thanks to John Chambers)
Bug fixes for when nothing was generated in
gives_warning / shows_message. (Thanks to
Bernd Bischl)
New make_expectation function to programmatically
generate an equality expectation. (Fixes #24)
SummaryReporter: You don’t get praise until you have
some tests.
Depend on methods rather than requiring it so that
testthat works when run from Rscript
auto_test now normalises paths to enable better
identification of file changes, and fixes bug in instantiating new
reporter object.
All mutatr classes have been replaced with
ReferenceClasses.
Better documentation for short-hand expectations.
test_dir and test_package gain new
filter argument which allows you to restrict which tests
are run.
autotest correctly loads code and executes tests in same environment
contexts are never closed before they are opened, and always closed at the end of file
fixed small bug in test_dir where each test was not
given its own environment
all expect_* short cut functions gain a label
argument, thanks to Steve Lianoglou
all expectations now have a shortcut form, so instead of expect_that(a, is_identical_to(b)) you can do expect_identical(a, b)
new shows_message and gives_warning expectations to test warnings and messages
expect_that, equals, is_identical_to and is_equivalent to now have additional label argument which allows you to control the appearance of the text used for the expected object (for expect_that) and actual object (for all other functions) in failure messages. This is useful when you have loops that run tests as otherwise all the variable names are identical, and it’s difficult to tell which iteration caused the failure.
executing bare tests gives nicer output
all expectations now give more information on failure to make it easier to track down the problem.
test_file and test_dir now run in code in separate environment to avoid pollution of global environment. They also temporary change the working directory so tests can use relative paths.
test_package makes it easier to run all tests in an installed package. Code run in this manner has access to non-exported functions and objects. If any errors or failures occur, test_package will throw an error, making it suitable for use with R CMD check.
colourise also works in screen terminal
equals expectation provides more information about failure
expect_that has extra info argument to allow you to pass in any extra information you’d like included in the message - this is very helpful if you’re using a loop to run tests
is_equivalent_to: new expectation that tests for equality ignoring attributes
library_if_available now works! (thanks to report and fix from Felix Andrews)
specify larger width and join pieces back together whenever deparse used (thanks to report and fix from Felix Andrews)
test_dir now looks for any files starting with test (not test- as before)