scoreContributions() returns, for any fit, each response's contribution
to the score of the marginal likelihood in its item's parameters (B, every
free loading, C, D), its observed information, its leave-one-out
predictive probability, and each person's score in the ability mean. They
are posterior expectations of the complete-data score (the Fisher
identity), by adaptive Gauss-Hermite quadrature at each person's Laplace
mode -- the terms the quadrature kernel already formed for the fit's
gradient, now written out per row (row_out in src/agq_backend.cpp).
Checked person by person against finite differences of each person's
marginal (agreement 1e-10, where plug-in scores miss by 1e-2), on a 4PL
with covariates and a 2-D 3PL with some abilities fixed.
scoreTest() turns them into tests of whether an item parameter depends on
a covariate (DIF, uniform or not, in any of B, A, C, D) or whether ability
does, without refitting: score (LM) tests with linear, spline or factor
bases and a one-step estimate of each coefficient; nonparametric
fluctuation tests (double max, Cramer-von Mises, maxLM) with p-values
simulated from the process's exact covariance, so that information
accruing unevenly along the covariate (impact does this) does not
invalidate them; covariates per person or per response; one test per
item, per item group, or shared. Each item's other parameters are
projected out (Neyman's C(alpha)), so the small pull of the item priors is
handled. purify = TRUE anchors on unflagged items -- person posteriors
and ability coefficients re-estimated from those alone, items held -- and
retests, against DIF contamination, without a refit.
The item tests use each response's expected information by default: a response's marginal score is exactly the score of the Bernoulli for that response given the person's other responses, and that Bernoulli's Fisher information is the score's conditional variance, never negative. The observed (Louis) information of a single response can be negative, and on thin items it made some discrimination tests explode (one statistic of 1257 on 35 responses). On a simulated thin bank without DIF (50 to 200 responses an item) expected information held 3.8 to 5.5 per cent at a nominal 5, and the outer product of the scores 7 to 12.
A covariate related to ability must be in the ability model for its DIF
test to be valid, but it need not have been in the fit: given the fit,
scoreTest(adjust = "auto") (the default) adds a covariate the ability
model lacks after the fact -- a centred natural spline, or indicators,
with coefficients and the ability SD re-estimated and the item parameters
held -- and tests under that model. A covariate given per response enters
per response, so ability may also move with it within person. On the
2025 score-test paper's simulation grid the adjusted tests matched a refit
with the covariate item by item (median correlation of p values .999
under impact, the same decision for 99.3 per cent of items) and held 5.0
to 5.2 per cent under impact (the refit 4.9 to 5.1), where tests that
ignore it reached 37 to 43; with a covariate varying within person, 5.4
per cent against 70. The result's adjustment records the coefficients,
the re-estimated ability SD, and the iterations taken, and a warning is
given if they did not converge. Both the SD's EM and the outer-product
steps used when predictors vary within person converge linearly, so the
whole parameter vector is extrapolated jointly (SQUAREM): on the
Mindsteps mathematics fit 19 and 35 passes where plain iteration took 35
and 93, to the same point.
scoreTest(adjustMethod = "fast"), the default, holds the fit's own
ability coefficients and estimates only the added covariate's and the
residual SD, by a joint Newton step (each person's Louis information for
the coefficients, a normal posterior's moments for the SD's curvature;
exact gradients, so the same maximum). Regressing the fit's posterior
means on the covariate instead recovers only about the reliability's share
of the impact. On the 2025 paper's grid (14,400 replications) it agreed
with "full" to 5.5e-5 in every item's p value, with the same decisions,
in a median 4 passes against 8; on the Mindsteps fit 93 s against 232 s,
the same 34 items flagged. Covariates given per response (no person-level
Newton step) and multidimensional models use "full".
adjust = "none" tests the fit as it is, and then warns when a
person-level covariate is related to ability but absent from it. See
vignette("score-tests"). A person-level covariate already in the fit's
ability model under another name is recognised (it lies in the span of
the fit's person predictors) and not added again.
scoreTest(effects = TRUE) reports DIF effect sizes. A response's score
is the score of a Bernoulli given the person's other responses, so the
derivative of its predicted probability in the item's parameters is
exact; applied to the one-step DIF estimate it gives the response's
change in the probability of a correct answer at its own ability.
effects averages it per item and contrast; dtf (with purify, for a
person-level covariate) sums it per person over the flagged items, as a
change in expected number correct and its ability equivalent. Test-level
effects need anchors: with the covariate's ability effect estimated
freely, the DIF of all items sums to about zero per group. Simulated
(50 replications, three DIF items of .8 logits): item effects within
.006 of the truth, the expected-score change 93 to 95 per cent of it.
sandwich::estfun() on a bigIRT_scores object returns the per-person
(or per-response) score matrix by item and parameter, so
strucchange::gefp() and sctest() run on bigIRT fits. With no DIF and
impact modelled their outer-product decorrelation held 2.8 to 3.9 per cent
where scoreTest() held 4.5 to 5.0.
modificationTests() tests candidate changes to a fitted model at the
fit: covariates of ability, DIF (purified, with effect sizes), per-item
lower and upper asymptotes (exact from each response's leave-one-out
probability, P being linear in c and d), and group dimensions for item
sets (the variance-component score test, from each person's posterior
expectation of (sum g)^2 + sum h over the set). Dimensions need no item
covariate: M = sum E[g g' + diag(h)] is the score for residual covariance
among items, so its leading eigenvector, found without forming M (M v is
one quadrature pass), proposes the dimension most worth adding, against a
simulated null. improveModel() refits each kind of change as a batch and
keeps it only if the held-out persons' marginal log-likelihood rises,
paired by person (heldoutLogLik(), which adds the prior's normalising
constant the quadrature value leaves out). Every score is checked against
finite differences of brute-force integration; on a simulation with one of
each it added the two ability covariates and not the null one, DIF on
exactly the three DIF items, and the testlet, with or without the topic
that defines it.
improveModel() now tries the kinds of change in turn within a round --
ability, DIF, asymptotes, group dimensions -- each against the model as it
stands after the batches kept before it. Before, every kind was judged
against the round's starting model and the kept batches combined, so DIF
along a covariate could be judged without that covariate's effect on
ability, which DIF then absorbs; the combined refit is gone too.
improveModel(checkpoint = dir) keeps each completed step (base fit, each
round's screens, each batch, the final fit) and a rerun resumes from them,
reusing a step only under the same data size, holdout, settings and
specification; on large data a run is hours. The result records seconds
per batch and timing for the fits and screens.
difCovariates (in modificationTests() and improveModel()) chooses
which covariates get the purified DIF screen; by default all of them.
improveModel() tries a kind again in a later round only if it, or a kind
after it, was kept in the round before; a rejected kind was already judged
against everything kept before it, and retrying refitted the same batch.
fitIRT(dropPerfectScores = FALSE) is now the default. Persons (and
items) whose responses are all correct or all wrong are kept. Neither
estimator needs them dropped: the ability prior keeps such a person's
ability finite and the item priors such an item's parameters. Dropping them
biased what depends on them. With them dropped, the upper-asymptote screen
of modificationTests() rejected 15 per cent of true nulls on a simulated
adaptive bank (7 with random selection), because all-correct persons are
exactly the responses that pull the upper score down; kept, it holds 5.
Fits that relied on the old default now include these persons: pass
dropPerfectScores = TRUE to reproduce them. With priors = FALSE an item
with all-same responses has no finite estimate (reported as diverged), so
drop it or keep the priors.
Named item sets now take precedence over overlapping sets found from the scores alone: a labelled set was specified in advance and says what the dimension is; the exploratory route is for structure the labels miss.
The group-dimension screen's one-step size (tau2) now divides the score by
its null information, from two data sets simulated under the fitted model,
instead of by the observed outer product. Under the alternative the outer
product carries each person's squared mean, which grows with their
responses in the set, so the old estimate collapsed exactly where persons
answer many items of a set: with true SD 0.5 it gave 0.52, 0.38, 0.20 and
0.09 at 5, 15, 40 and 100 items a person, while the new one holds steady.
On the Mindsteps bank the old estimate put the three subdomains at SD 0.06,
below the 0.3 floor, while a correlated three-subdomain model has residual
correlations of .70 to .78 and predicts held-out students far better. The
test statistic (z) is unchanged.
improveModel() adds group dimensions one per round, the strongest set
first. Until a real dimension is modelled the general ability leans towards
its items and the other items share the leftover covariance; tried in one
batch with a real testlet, a 15-item set with no structure of its own rode
in with it in 10 per cent of simulated runs once the size estimate above
stopped shrinking.
Asymptote tests: an item whose other asymptote is free now uses its fitted
value (it used 0 or 1, so a second round after freeing c tested d with the
wrong derivative); a fit with every item fixed no longer errors; and each
test reports identified, the share of the asymptote's information left
once the item's own parameters are projected out.
scoreTest(contributions = ) takes scoreContributions(fit) already
computed, for when the fit itself is passed (for purify or adjust)
and the pass over the data would otherwise be repeated; contributions from
another fit are refused. modificationTests() now computes them once for
all its DIF screens, where it computed them again for each covariate.
modificationTests(shape = TRUE) tests each item's response curve for
shape the model lacks: the Hermite terms He2 and He3 of the item's
composite, entered in its linear predictor, a 2-df score test per item with
one-step coefficients and the largest change they make to the curve
(dP). The scores are posterior expectations from the quadrature kernel
(shape_out in src/agq_backend.cpp), checked against finite differences
of brute-force integration. A curve that rises, flattens and rises again,
or flattens early, shows here when no asymptote or DIF covariate captures
it.
The exploratory dimension screen now proposes its sets on one random half of the persons and tests and sizes them on the other. Found and tested on the same persons, the leading eigenvector's set was tested against a null that had not been chosen to fit it. Set membership still comes from the direction found on all persons.
improveModel() judges DIF one covariate at a time, and before a
covariate's DIF adds its effect on ability if the model lacks one
(difImpact = TRUE): the covariate's basis columns the ability model does
not already span. A DIF covariate without impact modelled has its impact
absorbed by DIF on every item. What is missing is judged when the stage is
reached, after the ability batch, and a batch that would add nothing is
not refitted.
improveModel() tries two ways of adding asymptotes and keeps the better
if it passes: the items the screen selects, or every item free under the
asymptote prior, whose mean across items is estimated
(asymptotePooled = TRUE). That needs no selection, and it collects many
small asymptotes that are each too weakly identified to pass a screen.
With item sets that partition the items (partitions = TRUE),
improveModel() also tries the partition as correlated dimensions, one
per set, as an alternative to a group dimension for its strongest set,
and keeps the better if it passes. Group dimensions on a general ability
add sets one at a time and orthogonally; on Mindsteps the three subdomains
as correlated dimensions predicted held-out students far better.
fitIRT(startFrom = fit) starts the optimisation from a previous fit's
estimates, matched by label (items, dimensions, persons, predictors);
anything the previous fit lacks keeps its default start. Only the starting
point changes: from its own estimates a fit stays at its optimum in a
fraction of the evaluations, and a model with DIF terms added, asymptotes
freed or one dimension split into correlated ones reaches the optimum its
cold start reaches (test-start-from.R). improveModel() starts every
batch, and the final refit, from the current model, so a batch begins at
the model it extends and can only climb. On Mindsteps one batch (a 3-D
4PL with four DIF terms added) came back 16,551 held-out nats worse than
the model it extended, in a run at its memory limit; refitted alone from
the priors it was 2.7 nats worse, as it should be. What went wrong in the
run was not established.
improveModel(checkpoint = ) no longer holds the base fit in memory
through the rounds (it is read back at the end), and holds only the best
passing fit of a stage. On Mindsteps a refit held three fits at once and
ran out of memory at 67 of 128 GB.
No compatibility shims; each fails loudly on old calls.
fitIRT()'s tuning and prior arguments are grouped. 39 arguments went:
control = irtControl(...):
refine, certify, maxIter, tol, gradTol, tolScale, polish,
personTol, jitter, corrParam, optimizer, agqNodes, agqPoints,
keepCovariance, diagnostics, plot, plotEvery and the
empirical-Bayes settings (ebayesMethod, ebayesIter, ebayesCoarse,
ebayesMinSD, ebayesMultiplier). These were laplaceRefine,
laplaceCertify, laplaceOuterIter and so on.priors = irtPriors(...): Amean, Asd, AmeanSD,
Bmean, ..., abilityMean, abilitySD, abilityMeanSD, betaSD.
These were invspAMeandat, BSD, logitCSD, AbilitySD,
betaScale and so on. priors = TRUE or FALSE still works.normalise takes TRUE or list(scale = , mean = ) in place of
normaliseScale and normaliseMean.Each constructor keeps only the settings given, so lists merged with
modifyList() (as improveModel(fitArgs = ) does) do not reset each
other. A call with an old name stops and says where it went.
Other changes to the arguments:
NULL for personDat, itemDat, loadings,
loadingsFixed, AbilityCorr and trainingRows.dat in the signature.improveModel()'s default fitArgs is
list(control = irtControl(refine = "agq")).marginalApprox takes "laplace" or "map". The aliases "none",
"laplace_fast" and "laplace_direct" are gone.
laplaceCorrParam = "stan_corsqrt" is now "cpc" (canonical partial
correlations, as Stan's cholesky_corr and ctsem's sdpcor2cov). The old
name suggested ctsem's corsqrt, a different map that does not reach every
correlation matrix; this one does, to 1e-16 on round trips.
vcov(fit) (what = "item") is a sparse block-diagonal matrix over the
free item parameters, named param[item], not a P x P x Nitems array.
itemInformation() (so vcov, tidyIRT(conf.int = TRUE), summary SEs)
refuses a fit made with normalise = TRUE: its errors were the
unnormalised fit's.
testInformation() needs scale when the fit has more than one dimension.
estMeans rejects unknown entries, and the ability mean together with the
B mean unless fixed difficulties or abilities pin the location.
improveModel() refuses covariates with missing values before fitting, and
its checkpoints record a content digest of the data and the installed
build, so existing checkpoints are not reused.
Removed: the options bigIRT.debug.direct, bigIRT.agq_switch_gain,
bigIRT.fisher_refresh and bigIRT.agq_fixed_nodes (now constants), the
environment variable BIGIRT_LOCALITY_ORDER, and the always-NA columns of
laplaceDiagnostics (personStepSec, refreshStepSec, the strict and
stability criteria and streaks, recent*).
vignettes/laplace-objective.Rmd, written for the withdrawn alternating
backend, moved to dev/.
laplaceStatus$initialized_from says what the fit started from,
"prior" or "startFrom" (it read "prior_anchored" even under
startFrom). direct_objective (always TRUE) and the aliases
strict_convergence and frozen_effects are gone; use converged and
beta_frozen.
The tests run under testthat's third edition.
There are no options(bigIRT.*) any more. Eighteen of them changed fits
without appearing in the call, so a fit could not be reproduced from
fit$call. The fourteen that steer a fit are irtControl() settings:
metric, metricDenseMax, metricMaxNnz, corrMax, corrMetric,
agqNodes3, agqInflate, laplace2Floor, ebayesRefine, runawaySlope,
runawayProbeItems, saturationFloor, certifyDenseMax and
certifyEscalate. bigIRT.checkpoint_compress is
improveModel(checkpointCompress = ). The R-versus-C++ switches and the
score-test comparison switch, which only tests used, are internal
arguments. fitIRT(), improveModel(), modificationTests(),
scoreTest() and scoreContributions() stop if a session still sets any
bigIRT.* option, and say where it went.
The warning about pervasive partial clipping of the Laplace curvature is
gone: it existed because the gradient was approximate there, and it no
longer is (see Fixes). laplaceStatus$clipped_partial still counts it.
irtControl(certify) is TRUE, FALSE, "fast" or "exact". The exact
route's time budget, formerly a number given as certify itself, is
irtControl(certifyBudget = ); a number for certify stops and says so.
gradTol and tolScale are documented as what they are: reporting
thresholds that label a fit converged when the certificate gives no
verdict. The optimiser has never read them.
cores defaults to six or the number of CPUs the process may use,
whichever is fewer, and is never more than that: the affinity mask on
Linux (as nproc counts, so a CPU reservation is respected), every CPU
elsewhere. fitIRT() and scoreContributions() put the session's
RcppParallel thread count back when they return; it was left at the fit's.
The scale column is optional: without one the data are
one-dimensional.
scoreTest(form = "auto") treats a numeric covariate with at most four
distinct values as a factor, as model building always has. For a binary
covariate the test is unchanged; its effect contrasts were not.
fitArgsFromSpec(data, spec, base) gives the fitIRT() arguments for a
model-building specification and heldoutGain(a, b) the held-out gain of
one fit over another, the two internals the analysis scripts reached for
with ::: (43 calls). fitArgsFromSpec() documents that base must come
from the same run, since it carries commonEffects, and takes the
improveModel() result itself as well.
personCovarianceMatrices(), personCovarianceMatrices_R() and
rowLoadingsToSparse() are no longer exported. Nothing in the package used
them, and they use Fisher weights, not the observed curvature the fits use.
broom's tidy(), glance() and augment() methods are registered
lazily, in NAMESPACE, instead of by an .onLoad hook that loaded broom
with bigIRT whenever it was installed. Depends on R >= 3.6.0.
Five exports are renamed to say what they do; the old names are gone:
heldoutLogLik() is heldoutPersonLogLik(): it needs whole persons
held out, and returns NULL for a holdout of single responses;heldoutMetrics() is heldoutResponseMetrics(): it scores held-out
responses, from any holdout;IRTic() is informationCriteria();checkMirtBridge() is checkAsMirt(): it checks what as.mirt()
produced;specArgs(), new in this round, is fitArgsFromSpec().
The held-out functions, heldoutGain() and looIRT() are documented as
one family, each saying its unit in its title.
heldoutPersonLogLik() documents the per-person values it has always
returned.Large fits scale better with threads. Three kinds of serial work grew with the data and ran ahead of every parallel kernel call:
as.integer() and as.numeric() copied eleven response-length vectors
per evaluation (13 per cent of a 2-million-response 3PL at 16 threads);solve() per person in R, the
certificate built every response's effective parameters to take a
per-parameter maximum, the output step computed a response-length value
it multiplied by zero, and data preparation ran duplicated() six times
on the same columns and converted integer ids to character to recode
them;solve(): 8.7 s of a 30 s three-dimensional fit at 16 threads. Fits
with an estimated correlation move by rounding (2e-10 on the parameters,
1e-11 on the log-likelihood), with the same evaluations and verdict;
every other reference fit is identical to the last bit.
On cd-dev1 (dedicated cores; medians of two interleaved runs), against the
package before this round (817f278), with the same evaluations and
log-likelihoods:| fit | 1 thread | 8 threads | 16 threads | |---|---|---|---| | 1-D 2PL, 4M responses | 40.3 to 35.1 s | 12.9 to 7.9 s | 11.1 to 5.9 s (1.89x) | | 1-D 3PL, 2M, quadrature | 160 to 153 s | 33.0 to 24.9 s | 23.7 to 15.1 s (1.56x) | | 3-D 2PL, 1.8M, correlation, quadrature | 233 to 224 s | 46.8 to 36.3 s | 33.7 to 23.3 s (1.45x) |
From one thread to sixteen they now run 6.0, 10.1 and 9.6 times as fast (3.6, 6.8 and 6.9 before). One evaluation's kernel runs 12 times as fast at sixteen threads on the 2PL, where a plain compute loop on the same machine gets 13.9; what is left serial is once per fit (the row context and the output tables, about 0.7 s each at four million responses).
One covariate basis (R/basis.R) replaces three builders, and one rule
decides what is categorical where five copies did. The score tests'
builder made indicators with model.matrix(), which drops a row with a
missing value, so the basis came back shorter than the covariate.
bigIRT_icc() is the item curve at six sites, and bigIRT_score_agq() the
quadrature pass the score and model-building code wrote out eight times. In C++, the covariate term
and its gradient, the jittered Cholesky loop and the PSD rebuild are each
written once (src/bigirt_common.h), with the arithmetic the copies had:
every reference fit is identical to the last bit.
src/Makevars drops CXX_STD = CXX17 (every source compiles at C++14),
EIGEN_NO_DEBUG (R's -DNDEBUG already does it) and
RCPP_PARALLEL_USE_TBB (RcppParallel's default). Two block offsets and
some quadrature counters were long, 32 bits on Windows; they are
std::ptrdiff_t.
Timed on a machine with dedicated cores (three interleaved repeats, medians; the worst repeat-to-repeat spread was 8 per cent), the package as it was before the November review (08131e6) against now. One-dimensional 2PL and 3PL fits of 0.3 to 4 million responses run 1.22 to 1.33 times as fast on one thread, 1.06 to 1.24 on eight and 1.01 to 1.17 on sixteen; a three-dimensional 2PL refined by quadrature 1.02 to 1.03. They are the same fits: the same evaluations and log-likelihoods within 4e-10 (1e-4 on the large 3PL, between thread counts, whose summation order differs). The last round of changes (options, block table, fitIRT split, the clip gradient) is within 4 per cent of the one before, inside that spread.
The Laplace person and item kernels no longer allocate on the heap per
response; scratch is sized once per chunk (from the threading branch).
The results are identical to the last bit single-threaded, and one
evaluation at one thread went from 1.03 to 0.76 s on a 50,000-person bank.
The per-response *_beta_row index arrays are gone from the row context
and from four native entry points. The kernels never read them, and they
cost 24 bytes a response: about 0.5 GB on Mindsteps maths, 1.8 GB at 80
million responses.
fitIRT()'s data preparation is its own function
(bigIRT_fit_prepare(), R/prepare.R); fitIRT() is 916 lines, from
1,250.
Tests that need NOT_CRAN skip with skip_on_cran() instead of sitting
inside a file-level if (NOT_CRAN), which in CRAN mode defined nothing and
reported nothing (42 "empty tests"). Needs testthat >= 3.0.
The convergence status is decided once, where it is built, and only then reported. It used to be revised some 400 lines later, after the certificate's warning had already gone out under the earlier reason.
One constructor builds the evaluation context everywhere
(bigIRT_laplace_context()). The four hand-built copies disagreed about
the correlation parameterisation's default. Inside the objective the 30
arguments shared by the four native kernels are one list per evaluation.
direct_block_fg calls the person step and the item block as plain C++
functions. As guarded R entry points, an error in either became an R
longjmp out of the caller, skipping its destructors.
src/bigirt_common.h holds what every kernel shares: one inverse logit,
softplus, probability clamp, loading link and R-array view, where there
were five inverse logits, four softplus functions and four identical view
structs.
bigIRT_laplace_optimize_direct() (600 lines, with closures sharing state
through <<-) is an evaluator object, a polish function and the
refinement controller (bigIRT_laplace_refine()). No <<- is left in
them.
The four largest R files are split by topic, as pure moves:
laplaceLayout.R, laplaceFit.R and laplaceEB.R out of
laplaceBackend.R; links.R, normalise.R, compare.R and
fitMethods.R out of fitIRT.R; scoreTest.R out of scores.R;
improveModel.R out of modify.R.
One table names each item-parameter block's fields (R/itemBlocks.R).
The names grew one block at a time -- the prior SD is BSDx for B but
invspASD for A -- and nine places spelled the table out for themselves;
the item prior is one loop over it.
fitIRT() is 461 lines, from 860: the empirical-Bayes rounds, the
convergence status, the diagnostics table, normalisation and the output
tables are functions of their own (R/fitAssemble.R).
Every change in this list gives results identical to the last bit on
seven reference fits (2PL, 3PL, 4PL, and 2-D fits with covariates, fixed
abilities, signed loadings and an estimated correlation).
The later ones (options, block table, fitIRT() split) were also checked
on whole fit objects, every element compared, over 17 fits that add joint
MAP, both empirical-Bayes methods, normalisation, item covariates, a
holdout, fixed items and the mize polish.
From a review of the whole package (November 2026). Each has a test that fails on the code before the fix.
The Laplace gradient was approximate wherever a person's data curvature
was clipped on some dimensions and not others: it zeroed that person's
whole log-determinant contribution, which is exact only when every
eigenvalue clips. 3PL respondents who guess right on items far beyond
them produce the partial case readily. On such a design the item gradient
missed central differences by 3e-4 and the correlation gradient by 55 per
cent, and a 2-D 3PL fit (refine = "none") stopped 3.2 nats short,
uncertified. The gradient now takes the clip's own derivative: the
Daleckii-Krein derivative of the eigenvalue clip for the log-determinant,
and the actual Hessian for the mode adjoint, in the item kernel and in the
ability-mean, ability-covariate and correlation gradients. The same fit is
certified (gap 2e-5). Unclipped persons are unaffected, and fits refined
by quadrature (the default for 3PL and 4PL) never used this gradient at
their end point.
The Laplace value counted a fixed ability's log(2 pi)/2 and its pinned
1/jitter precision, so logLik moved by about -6 nats per fixed
person-dimension and disagreed with the quadrature objective. A fixed
person now contributes exactly their likelihood and prior density.
Persons whose data curvature was clipped (log|H| flat) had their
log-determinant slope zeroed in the item gradient but not in the
ability-covariate, ability-mean and correlation adjoints: up to 85 per cent
off on a design with many such persons. The person step now reports them
(posterior$clipped).
simIRT(): personPars held one arbitrary dimension at K > 1 (each
person's first row's scale); the legacy item-predictor merge dropped every
response to scales 2..K.
scoreTest(parameters = "Ability") with a row-level covariate failed or
misaligned person means when whole persons were held out; the DTF summary
ignored purifyAlpha and the anchors purification kept.
normalise = TRUE at K > 1 left pars$B without item names (downstream
code labelled items 1..J).
AbilityStd divided by the SD of shrunk point estimates (0.114 too large
on a sparse check whose truth was .6); now the latent SD. AStd no longer
counts structural zeros of the loading matrix.
IRTcurve(plot = TRUE) returned NULL; looIRT(seed =) and six internal
seeded steps changed or replaced the caller's RNG state; glanceIRT
computed logLik three times.
wleIRT() solved Warm's equation in the difficulty form,
logistic(a (theta - b)), while fitIRT() reports B as an intercept
(a theta - b), and it dropped the upper asymptote. Every item with a != 1
was scored under the wrong model: on a 1D 2PL the estimate missed the
weighted-likelihood maximum by up to 0.34, by 1.1 with asymptotes. It now
uses each response's parameters as the fit reports them (item covariates
included), groups responses in one pass rather than a quadratic scan, and
warns when cross-loadings are ignored.
testInformation(scale =) was matched against a column itemPars never
has, so it was ignored, and a multidimensional fit's result summed every
item's information about its own dimension on one axis. scale (name or
index) is now honoured, and required when the fit has more than one.
Empirical Bayes updated its hyperparameters from every row, so held-out
responses added their design to the update (their scores did not enter,
which is why flipping them never showed it). With half of each person's
responses held out, the loadings of an EB fit differed from the fit to the
training rows alone by 0.11 on average. A trainingRows fit now equals the
fit to the training subset.
itemDat without a C column failed ("column C not found"); only the
columns given are fixed now.
A fit stopping on a non-finite objective or gradient warned "stopped with person-mode failures" whether or not any mode had failed.
The optimiser's coordinates for item-covariate coefficients were meant to be divided by each covariate's SD, but the lookup was by name on columns stored by index, so every SD was 1. Conditioning only; values and gradients were right.
dropPerfectScores() added its working columns to the caller's
data.table by reference and left them there, so a second call on the same
table could drop different persons.
Four native entry points had no BEGIN_RCPP/END_RCPP, so an input error
there terminated the R session instead of raising an error (the
correlation gradient, the person covariance kernel, the direct Laplace
evaluation and the legacy item objective).
The finite-difference gradient tests now cover the default correlation
parameterisation ("stan_corsqrt"); all of them used "normalized_chol".
They pass, and a deliberately mismatched evaluator shows they can fail.
src/Makevars: objects depend on the package headers, so an in-place
R CMD INSTALL . can no longer link objects built against an older header.
A fit's stored call (fit$call) no longer carries its data, a previous
fit passed as startFrom, or long vectors such as trainingRows: called
through do.call(), as improveModel() calls fitIRT(), match.call()
holds the values themselves, and a warm-started Mindsteps fit stored its
20-million-row data and the whole previous fit -- 33 GB where the fit is
9 -- which filled the disk. Large values are now placeholders naming their
class. options(bigIRT.checkpoint_compress = TRUE) writes
improveModel()'s checkpoint gzipped (about sevenfold smaller).
improveModel(): DIF added in a later round along a covariate that already
had DIF in the model collided with it. Each batch was a separate term
whose column took the same name, so the later batch's items overwrote the
earlier column -- the earlier items lost their DIF -- and the column was
listed twice. On Mindsteps a round-2 batch of 2 age items dropped the age
DIF of 3,883 items and ran for ten hours before it was stopped. DIF along
one covariate and parameter is now one term over the union of its items,
and the build asserts that no predictor is listed twice. The same
collision probably explains an earlier round-2 DIF batch that "failed
numerically". None of the simulation runs hit it (no run tried round-2
DIF along a covariate kept in round 1).
improveModel(checkpoint = ) keeps the fit of the current model only:
every other step keeps its verdict (held-out values, gain, z) but drops
its fit once a later batch is accepted or another alternative wins. A
resumed run needs nothing more. It kept every passing batch's fit, and
on Mindsteps, where a fit is 13 GB and a round can accept a batch per DIF
covariate, that was more than the disk held.
A 0/1 response covariate that is rarely 1 (a repeated attempt at an item,
say) got a DIF effect size of zero on every item: read as continuous, its
effect contrast was its 90th against its 10th percentile, both 0. No DIF
along it could pass improveModel()'s effect floor however strong (on
Mindsteps 434 items significant, none kept). modificationTests() now
treats a response covariate with four or fewer values as a factor, as it
already did a person covariate, and scoreTest() contrasts a covariate's
range when its 10th and 90th percentiles coincide.
The Laplace and quadrature L-BFGS search now also stops ("noise_floor") when the objective has fallen by less than a tenth of the tolerance over ten iterations together. It stopped only when the predicted gain fell below the tolerance, which it need never do when the gradient is not quite the derivative of what is evaluated -- one person whose mode cannot be resolved is enough -- while line searches keep accepting steps on the objective's noise. A 3-D 3PL fit in the model-building simulations ran to the 500-iteration cap this way on Linux (3,700 evaluations, the objective flat to 3e-5 nats), turning a two-minute replication into hours; on Windows the same fit's noise happened to stop it at iteration 31.
marginalApprox = "map" ("none"
is accepted as its former name) and is written in C++
(src/map_backend.cpp). It replaces a Stan model that optimised every
person and item coordinate at once by L-BFGS. The persons are now profiled
out: each evaluation solves every person's mode by Newton and differentiates
the joint posterior there, which needs no adjoint because the abilities are
at their maximum. The result runs through the Laplace path's optimiser,
empirical-Bayes rounds, convergence certificate and output, so the two
estimators take the same arguments and return the same object. Multiple
dimensions, confirmatory and cross-loadings, signedLoadings, person and
item covariates (within-person ones included), estimated means, fixed items
and persons, and whole-person holdouts all work under either.itemInformation(method = "exact")
use, to about 2e-9 (test-map.R). Two deliberately broken kernels were each
caught by the test aimed at them.estimateAbilityCorr = TRUE is an error under "map". The joint mode sets
its latent scale from the priors rather than the data (its loadings come out
inflated, by a factor of two to four on simulated 2PL data, exactly as the
Stan implementation's did), and no correlation estimated alongside it held
up. As a parameter of the joint posterior it runs to the boundary. Estimated
by EM from the person posteriors it was biased: .39 to .50 for a true .6,
where the Laplace estimator gave .61.logLik(), and so IRTic(), AIC(), glanceIRT() and summary(), report
for a "map" fit the same model's marginal likelihood at the MAP estimates,
by the approximation laplaceRefine selects. Fits by either estimator are
then compared on one scale.personDat), marginalApprox = "laplace" now
fits the joint mode, which is the same objective there up to a constant,
instead of falling back to the Stan path.personDat): off by 1.06 relative against central differences with
40 of 150 persons fixed, and by 1.15 in two dimensions, so such fits were
optimised along something that was not the derivative of what they
evaluated. Two faults. The item kernel took each person's log-determinant
over every dimension, while the person kernel, whose log-determinant the
objective's value uses, pins fixed abilities out. And the
positive-semidefinite guard on the data's curvature ran over every
dimension, so a fixed ability's curvature could clip the free block and
drop a log-determinant term that was not flat (7.8e-2 once the first was
fixed). Both now work on the free abilities only. Second-order Laplace
inherited both; quadrature and joint MAP were not affected. Fits without
fixed abilities are unchanged. Test: test-laplace-direct-gradients.R,
which fails on the old code, and on the first fix alone.normalise = TRUE now moves the whole fit with the abilities. In one
dimension the person posterior was left on the old metric, so
reliability() changed under what is only a reparameterisation (.859 to
.868 in one check) and the person SD columns were withheld. In any
dimension the covariate coefficients were not rescaled, so the reported
abilities stopped being base plus covariate effect, and the row-level
loadings and difficulties were not moved, so looIRT() gave different
answers for a normalised multidimensional fit. And logLik() read the
ability prior's normalising constant from the rewritten abilityPrior, so
it moved too; it now uses the prior the fit was estimated with. Test:
test-normalise-latent-scale.R, six of whose fifteen checks fail on the
old code. normaliseIRT() also returns the center and scale it
applied.itemSpecificBetas = TRUE) as if it were shared
across items, at two evaluations each. On a 500-item fit with three
covariates on A and B that is nearly 6,000 evaluations: the certificate
had run for more than ten minutes, on a fit that took a few seconds, when
it was stopped. Each coefficient belongs to one item, and both kernels now
return it in that item's block of the Hessian from the same pass, carried
row by row so that a covariate varying within an item is exact too. The
same fit is now certified in 6.3 s in all, 0.65 s of it the certificate.
The exact route's block form keeps them with their items as well. Tests:
test-map.R and test-certify.R check the blocks against central
differences (agreement 6e-10; built from the item's summed block instead,
they are off by 0.25 to 1.2) and the certificate's cost on such a fit.test_fixedItemPars.R ended 0.033 nats short and
uncertified, and a 300-person, 30-item fit with item SDs of 0.001 ended
8.9 nats short after two iterations. The metric now keeps the prior's
border, applied through the shared block's Schur complement: both fits
are certified, the first in one iteration, as the dense metric did it. On
tight priors the bordered metric also needs a third to a fifth of the
iterations where the old one did get there. Fits whose item means are
fixed have no border and are unchanged. Test: test-laplace-optimizer.R,
two of whose checks fail on the old code.estMeans leaving it out)
measured the items' spread about their own mean, then applied that SD about
the fixed centre. With the loading centre placed away from the items (raw
2.5, a = 2.6, against items generated at a = 1) it fitted an SD of about
.45 and dragged the loadings to a mean of 1.94 to 2.15; it now measures the
spread about the centre in force, fits 1.86 to 2.03, and the loadings
average 1.03 to 1.13 (three seeds). With estimated centres, the default,
nothing changes. Test: test-ebayes-centre.R, which fails on the old code.priors = TRUE; it says the priors in force did not prevent it (joint MAP
with empirical Bayes can do this, through an estimated centre or a floored
SD).personDat), the optimiser's metric
(src/fisher_blocks.cpp) formed its Schur term over every person's
columns, including the all-zero ones of the fixed persons. It now skips
them: 87.5 s to 7.2 s on a 500-item fit with every ability fixed, with
identical results.iter, tol, init and carefulfit (the Stan path's
optimiser controls; the laplace* ones govern both estimators),
noptimsteps (unused since the alternating Laplace backend was withdrawn),
ebayesFromFixed (never ran: it tested length(personDat) == 1, which a
data frame with more than one column never is) and stochastic. Old calls
keep working: plot and noptimsteps, which never changed a result, are
accepted in silence, and the others are accepted with a warning that they
are ignored. ... takes nothing else. It used to forward anything to the
Stan optimiser, so a misspelled argument could pass unnoticed; an unknown
argument is now an error. verbose is a named argument.marginalApprox moved next to pl in the argument list, and the arguments
are grouped by purpose. The laplace* optimiser controls keep their names.stochastic optimiser is gone with the Stan path that
shared it.print() and summary() name the estimator and objective; a MAP fit's
printed objective is labelled as the joint log posterior.rstan, rstantools, StanHeaders and BH dependencies,
the rstantools configure scripts, optimIRT(), the ctsem-derived SGD
optimiser and the sampled-ability helpers only it used, and the sigma-point
kernels in src/person_covariance.cpp. Installing no longer compiles a Stan
model, and native routines are registered in src/init.c.fitIRT() now defaults to marginalApprox = "laplace" (marginal maximum
likelihood with the Laplace backend; previously "none", penalised joint
estimation on the Stan path) and ebayes = FALSE (previously TRUE). On
simulated 3PL and 4PL data, empirical-Bayes priors narrowed toward a first
pass gave item intervals covering .004 to .84 of the time against a nominal
.95. Pass marginalApprox = "none", ebayes = TRUE for the old behaviour.laplaceRefine = "auto" now finishes a two-parameter fit in two or three
dimensions on adaptive quadrature (previously second-order Laplace, which
remains the choice beyond three dimensions). With few responses per person
and dimension the second-order finish sat off the quadrature optimum in
every fit (sparse 3D simulations, about eight fitted responses per person:
primary-loading RMSE .283 against .235), and with 1,350 persons 2 to 4 per
cent of its fits were certified at a spurious optimum with one item at a
discrimination of 4 to 10, where quadrature put it at 1.6. Quadrature costs
four to five times as much there. Pass laplaceRefine = "laplace2" for the
old behaviour. Test: test-refine-auto.R, which fails on the old rule.laplaceRefine = "agq", and "auto" wherever
it refines by quadrature) could walk an item out along its own ray and
accept a worse fit. Its rounds on held quadrature nodes treat abilities as
a discrete distribution, which a step between two nodes fits better the
steeper it is; each stretch gained on the held nodes and lost on the
objective. On a three-dimensional 2PL with priors = FALSE an item went
from a discrimination of 58 to 6197, and the refinement ended 46 nats
below where it started -- where quadrature was 62 nats off the exact
marginal, whose maximum had that item at 3.3. Each stretch on held nodes is
now judged with the nodes re-centred and undone if it lost more than
re-centring alone moves the value (1e-4 nats per person; regularised fits
moved by 6e-6 at most, the exploited stretches lost 2e-3 and more), the
fit going on with moving nodes. Items that have run off are pulled back along their
rays and the refinement restarted from there, keeping whichever end is
best on its own objective; pulling only where that gains was not enough,
because a run-off item's neighbours adapt to it. A refinement that ends
lower on its own objective than it started now reverts to the Laplace
optimum, as one that moves it implausibly far already did. On that case the
fit now ends 20 nats higher, with no item run off and quadrature within 0.6
nats of the exact marginal. Regularised fits barely change: of 18
simulated 2PL to 4PL fits in one to three dimensions, 14 are bit-identical
and four end up to 0.005 nats higher, each at one more evaluation. Tests:
test-agq-runoff.R, against the exact marginal on frozen data, which fails
on the old code.test-agq-runoff.R, which fails on the old code for all four metrics.dropPerfectScores (on by default) decided whom to drop from all rows,
held-out ones included, so held-out responses changed which persons and
items were fitted. It now decides from the trainingRows only.itemDat) failed on the Laplace path with "missing value where TRUE/FALSE
needed": the item-prior centres were still estimated with no free items to
inform them. A family's centre is now estimated only when it has a free
item parameter.personPars, _SD columns), from the per-person posterior precision every
fit keeps; before, they needed laplaceKeepCovariance = TRUE.normaliseIRT(), normaliseMIRT(), compareIRTmodels()'s "normalized"
form and fitIRT(normalise = TRUE) standardised a solution by the spread of
its ability point estimates, which are shrunk toward the prior mean, so
discriminations came out flattened by roughly the square root of the
reliability. They now use the latent scale: extractMIRTpars() returns
latent_mean and latent_sd (for a bigIRT fit from the posterior, by the
law of total variance; for mirt from its estimated latent distribution),
and the normalisation functions take AbilitySD/AbilityMean.
normaliseIRT(robust = TRUE) also set the interquartile range, not a
standard deviation, to the target scale.getOption("bigIRT.certify_escalate", 60) seconds, and the exact verdict is
reported (the fast one is kept as $fast).extractMIRTpars() set every simulated upper asymptote to 1, so 4PL curve
recovery was scored against the wrong curves.tidyIRT() returned NA standard errors on a default fit although the
certificate keeps the covariance.itemDat, fixed loadings). With
ebayesMethod = "moment" the prior SD of any parameter type with a fixed
member floored at ebayesMinSD whatever the data said (on a 2PL with six of thirty
difficulties anchored, 0.05 against a correct 1.03). With the default
"laplace" the hyperparameter objective dropped the responses to items with
a fixed difficulty, added spurious prior-only terms, and integrated over
fixed loadings and asymptotes as if they were free (prior SDs about 15 per
cent high in the same case). Fits with every item parameter free are
bit-identical to before.itemInformation() stopped with "Item index out of range." on a Laplace
fit with some difficulties anchored through itemDat and the loadings (or
asymptotes) all free, whenever it did not read the certificate's exact
covariance: the closed-form information was gathered by the
free-difficulty index, which is 0 on an anchored item, rather than by item.
Most callers caught the error and carried on without saying so. On such a
fit the Newton polish (laplaceOptimizer = "mize") took no steps, the
person-covariate standard errors dropped their marginal correction and came
back conditional, tidyIRT(method = "profile") gave NA standard errors,
and without a kept certificate covariance summary() and
checkConvergence() lost their standard errors and Newton decrement. Fits
with every difficulty free are bit-identical to before.itemDat removed the
difficulty block from every method, so all the free difficulties lost their
standard errors (on a 2PL with four of thirty anchored, none of the 26 free
ones had one, by "exact" or "profile"). Anchoring difficulties on some
items and loadings on others made itemInformation() stop with "No per-item
free parameter blocks were found in this fit." Each item's block now covers
the parameters it has free, and a fixed one is NA. The same pairing of item
j with free parameter j ran through the certificate's exact covariance, the
Newton polish, the person-covariate standard errors' marginal correction
(which left out the free difficulties, about 1 per cent of the variance with
an uncentred covariate), checkConvergence()'s Newton decrement and the
effective degrees of freedom in logLik() and IRTic(). All now follow
each item's free parameters. Fits with every item parameter free are
bit-identical to before.priors = FALSE) whose item ran off reported its
loading as Inf, an intercept in the thousands and NaN in
pars$pcorrect, with laplaceStatus saying only that it had not
converged (16 of 80 three-dimensional 2PL fits with laplaceRefine = "agq"). The raw discrimination was finite (6197 in one case); R formed
the loading as log1p(exp(raw)), which overflows past 709, while the
native kernels had always used the stable form. The linear predictor then
took Inf * 0 for persons whose ability was exactly the prior mean. The R
side now uses the native softplus, so loadings and probabilities are finite.
A loading steeper than getOption("bigIRT.runaway_slope", 20) logits per
ability SD, or an intercept that far beyond every ability within four SDs,
is now reported. The certificate's verdict is "diverged", and
laplaceStatus$diverged names each such parameter with what pulling its
item back along its own ray gains. converged is FALSE, and the table is
filled even with the certificate off. The case above was the
approximation's: on the exact marginal that item's maximum-likelihood
discrimination was 3.3.signedLoadings: loadings on non-primary dimensions with an unconstrained
sign (identity link, zero-centred prior).laplaceRefine documents that the two-parameter Laplace marginals rely on
the item priors; without them, use "agq".