| Title: | Continuous Time Structural Equation Modelling |
|---|---|
| Description: | Hierarchical continuous (and discrete) time state space modelling, for linear and nonlinear systems measured by continuous, binary, ordinal, count or censored variables. The subject specific dynamic system is a stochastic differential equation (SDE) or difference equation, measured through a (typically) multivariate factor model. Individual differences on any parameter, at one or more nested levels, are integrated out by an augmented Kalman filter or a Laplace approximation, or sampled. Estimation is by maximum likelihood or maximum a posteriori optimisation, with uncertainty from the Hessian, importance sampling or the bootstrap, or by sampling the posterior, using either 'Stan' or an optional 'Julia' engine that ctJuliaInstall() sets up. See <https://github.com/cdriveraus/ctsem/raw/master/vignettes/hierarchicalmanual.pdf> for details, and <https://osf.io/preprints/psyarxiv/4q9ex_v3> for a detailed tutorial. For the conceptual overview of the hierarchical Bayesian linear SDE approach, see <https://www.researchgate.net/publication/324093594_Hierarchical_Bayesian_Continuous_Time_Dynamic_Modeling>. Exogenous inputs may also be included; for an overview of such possibilities see <https://www.researchgate.net/publication/328221807_Understanding_the_Time_Course_of_Interventions_with_Continuous_Time_Dynamic_Models>. <https://cdriver.netlify.app/> contains some tutorial blog posts. |
| Authors: | Charles Driver [aut, cre, cph], Manuel Voelkle [aut, cph], Han Oud [aut, cph], Trustees of Columbia University [cph] |
| Maintainer: | Charles Driver <[email protected]> |
| License: | GPL-3 |
| Version: | 4.0.0 |
| Built: | 2026-11-19 01:42:14 UTC |
| Source: | https://github.com/cdriveraus/ctsem |
ctsem is an R package for continuous time structural equation modelling of panel (N > 1) and time series (N = 1) data, using either a frequentist or Bayesian approach, or middle ground forms like maximum a posteriori.
The general workflow begins by specifying a model using the ctModel function,
then fitting it to data using ctFit (alias ctStanFit).
For original OpenMx / SEM functionality from the first ctsem versions, use the
ctsemOMX package.
For most purposes, the Stan-based forms in ctsem are more robust and flexible.
For examples, see ctFit.
For citation info, please run citation('ctsem') .
Maintainer: Charles Driver [email protected] [copyright holder]
Authors:
Charles Driver [email protected] [copyright holder]
Manuel Voelkle [copyright holder]
Han Oud [copyright holder]
Other contributors:
Trustees of Columbia University [copyright holder]
https://www.jstatsoft.org/article/view/v077i05
Driver, C. C., & Voelkle, M. C. (2018). Hierarchical Bayesian continuous time dynamic modeling. Psychological Methods. Advance online publication.http://dx.doi.org/10.1037/met0000168
Stan Development Team (2018). RStan: the R interface to Stan. R package version 2.17.3. http://mc-stan.org
#' @keywords internal
Useful links:
Report bugs at https://github.com/cdriveraus/ctsem/issues
A dataset containing panel data assessments of individuals Anomia and Authoritarianism.
data frame with 2722 rows, 14 columns. Column Y1 represents anomia, Y2 Authoritarianism, dTx the time interval for measurement occasion x.
See doi:10.1037/a0027543 for details.
anova() on two or more optimised fits of nested models to the same
data gives each fit's number of parameters, AIC, BIC, log likelihood and -2
log likelihood, and tests each fit against the one above it: the chi-square
statistic (twice the gain in log likelihood), its degrees of freedom (the
extra parameters) and its p-value. The fits may come from either backend.
## S3 method for class 'ctJuliaFit' anova(object, ...) ## S3 method for class 'ctStanFit' anova(object, ...)## S3 method for class 'ctJuliaFit' anova(object, ...) ## S3 method for class 'ctStanFit' anova(object, ...)
object |
An optimised fit from |
... |
Further optimised fits, of models nested with |
The fits are ordered by their number of parameters, and each is tested
against the one with the next fewer, so the models must be nested in that
order; anova() cannot check that. A negative statistic, which warns,
means a model fits worse than one nested in it: the models are not nested,
or an optimisation stopped short.
The p-value is the chi-square distribution's on the difference in
parameters. Where the smaller model fixes a variance at zero, such as a
random effect's, the null lies on the boundary of the parameter space and
that p-value is conservative (Stram & Lee, 1994); ctScoreTest
tests a new random effect against the corrected reference without
refitting.
The log likelihoods are those logLik() reports. Where a fit's
random-effects check or a refined fit integrated the random effects, that
is the integrated value, and the heading says so. A fit with priors
maximised the posterior, so its log likelihood is read at the posterior
mode rather than at the likelihood's maximum, and the heading says that too.
Sampled fits have no single log likelihood; compare them with
ctLOO.
A data frame of class anova, one row per fit in order of
parameters, with columns npar, AIC, BIC,
logLik, -2LL, Chisq, Df and
Pr(>Chisq); the last three are NA on the first row. Rows are
named by the arguments.
Stram, D. O., & Lee, J. W. (1994). Variance components testing in the longitudinal mixed effects model. Biometrics, 50(4), 1171-1177.
ctChisqTest for the p-value alone.
df <- data.frame(id=1, time=1:length(sunspot.year), Y1=sunspot.year) m1 <- ctModel(type='dt', LAMBDA=diag(1),MANIFESTVAR=0) m2 <- ctModel(type='dt', LAMBDA=diag(1),MANIFESTVAR=0,DRIFT = .9) if (isTRUE(ctJuliaStatus()$available)) { f1 <- ctFit(df,m1,cores=1) f2 <- ctFit(df,m2,cores=1) test <- anova(f1, f2) test test$Chisq[2] }df <- data.frame(id=1, time=1:length(sunspot.year), Y1=sunspot.year) m1 <- ctModel(type='dt', LAMBDA=diag(1),MANIFESTVAR=0) m2 <- ctModel(type='dt', LAMBDA=diag(1),MANIFESTVAR=0,DRIFT = .9) if (isTRUE(ctJuliaStatus()$available)) { f1 <- ctFit(df,m1,cores=1) f2 <- ctFit(df,m2,cores=1) test <- anova(f1, f2) test test$Chisq[2] }
Computes autocorrelations and cross correlations by time lag, pooled over subjects, for data observed at irregular times.
ctACF( dat, varnames = "auto", ccfnames = "all", idcol = "id", timecol = "time", plot = TRUE, timestep = "auto", time.max = "auto", nboot = 100, scale = FALSE, center = FALSE )ctACF( dat, varnames = "auto", ccfnames = "all", idcol = "id", timecol = "time", plot = TRUE, timestep = "auto", time.max = "auto", nboot = 100, scale = FALSE, center = FALSE )
dat |
The input data in data frame or data table format, in long format. |
varnames |
Character vector of variable names in the data to compute the ACF for. 'auto' uses all columns that are not time / id. |
ccfnames |
Character vector of variable names in the data to compute cross correlation for. 'all' uses all variables in varnames, NA uses none. |
idcol |
The name of the column containing subject IDs (default is 'id'). |
timecol |
The name of the column containing time values (default is 'time'). |
plot |
A logical value indicating whether to create a plot (default is TRUE). |
timestep |
The width of the time lag bins. 'auto' (default) uses half the median time interval between consecutive rows of a subject. |
time.max |
The largest time lag to compute. 'auto' (default) uses two thirds of the mean, over rows, of the time span of the row's subject. |
nboot |
The number of bootstrap samples, each resampling subjects with replacement (default is 100). |
scale |
if TRUE, divide each variable by its within-subject standard deviation. |
center |
if TRUE, subtract each variable's within-subject mean. |
Within each subject, every pair of observations is put in the time lag bin nearest
to the time between them, at multiples of timestep up to time.max. The value at a
lag is the correlation of the pairs in its bin, pooled over subjects: of a variable at time
t + lag with the same variable at time t for an autocorrelation, and for the cross correlation
'x_y' of x at time t + lag with y at time t, so that at a positive lag y is the earlier.
Lags with fewer than three pairs are left out. Autocorrelations are returned from the first
non-zero lag, as the lag 0 autocorrelation is always 1. Cross correlations include lag 0, the
contemporaneous correlation. Bootstrap samples resample whole subjects, and the plot shows a
spline through the estimates with a band from their bootstrap spread (see plotctACF).
If 'plot' is TRUE, the function returns a ggplot object of the ACF plot. If 'plot' is
FALSE, it returns a data table with a row for each sample, variable and time lag:
Sample (0 for the data, then the bootstrap samples), ACF,
TimeInterval, Variable, SignificanceLevel (the 95% bound of a
correlation of zero computed from that many independent pairs), and Pairs, the
number of pairs the correlation is computed from.
data.table::setDTthreads(1) #ignore this line # Example usage: head(ctstantestdat) ctACF(ctstantestdat,varnames=c('Y1'),idcol='id',timecol='time',nboot=5)data.table::setDTthreads(1) #ignore this line # Example usage: head(ctstantestdat) ctACF(ctstantestdat,varnames=c('Y1'),idcol='id',timecol='time',nboot=5)
This function takes a fit object from ctsem and computes the continuous time autocorrelation function (ACF) on the standardized residuals.
ctACFresiduals(fit, ...)ctACFresiduals(fit, ...)
fit |
A fitted model object generated by the ctsem package. |
... |
Additional arguments to be passed to the |
This function first extracts the standardized residuals from the fit object using
the ctKalmanArray function. Then, it calculates the continuous time ACF for these residuals
and returns the results as a data table.
A data table containing the continuous time ACF estimates for standardized residuals.
data.table::setDTthreads(1) #ignore this line # Example usage: ctACFresiduals(ctstantestfit, varnames='Y1',nboot=5)data.table::setDTthreads(1) #ignore this line # Example usage: ctACFresiduals(ctstantestfit, varnames='Y1',nboot=5)
The p-value of a likelihood-ratio test between two nested model fits, from
either backend. anova(fit1, fit2) gives
the whole test: each fit's log likelihood, the chi-square statistic, its
degrees of freedom and the p-value.
ctChisqTest(fit1, fit2)ctChisqTest(fit1, fit2)
fit1 |
One of the fits to be compared. |
fit2 |
Second fit to be compared. The fit with more parameters is tested against the other, whichever order they come in. |
The p-value: that of the fit with more parameters in
anova(fit1, fit2). NA where the two have the same number of
parameters.
df <- data.frame(id=1, time=1:length(sunspot.year), Y1=sunspot.year) m1 <- ctModel(type='dt', LAMBDA=diag(1),MANIFESTVAR=0) m2 <- ctModel(type='dt', LAMBDA=diag(1),MANIFESTVAR=0,DRIFT = .9) if (isTRUE(ctJuliaStatus()$available)) { f1 <- ctFit(df,m1,cores=1) f2 <- ctFit(df,m2,cores=1) ctChisqTest(f1,f2) }df <- data.frame(id=1, time=1:length(sunspot.year), Y1=sunspot.year) m1 <- ctModel(type='dt', LAMBDA=diag(1),MANIFESTVAR=0) m2 <- ctModel(type='dt', LAMBDA=diag(1),MANIFESTVAR=0,DRIFT = .9) if (isTRUE(ctJuliaStatus()$available)) { f1 <- ctFit(df,m1,cores=1) f2 <- ctFit(df,m2,cores=1) ctChisqTest(f1,f2) }
ctCollapse Easily collapse an array margin using a specified function.
ctCollapse(inarray, collapsemargin, collapsefunc, plyr = TRUE, ...)ctCollapse(inarray, collapsemargin, collapsefunc, plyr = TRUE, ...)
inarray |
Input array of more than one dimension. |
collapsemargin |
Integers denoting which margins to collapse. |
collapsefunc |
function to use over the collapsing margin. |
plyr |
Whether to use plyr. |
... |
additional parameters to pass to collapsefunc. |
An array (or vector) with the collapsemargin dimensions
removed, holding collapsefunc applied over them.
testarray <- array(rnorm(900,2,1),dim=c(100,3,3)) ctCollapse(testarray,1,mean)testarray <- array(rnorm(900,2,1),dim=c(100,3,3)) ctCollapse(testarray,1,mean)
A ctsem matrix cell may be written as an expression referencing a latent
process or a time dependent predictor – for instance
DRIFT[1,1] = '-log1p(exp(param)) * eta2'. Such a cell has no single
value. It has a value at an evaluation context: a latent state, a set
of time dependent predictor values, a time and an interval.
ctsem's summaries and plots report a number for every cell, so for a model like this they must pick a point. There are two, and which one you get depends on what you asked for:
Population matrices (summary, ctSummaryMatrices,
ctDiscretePars(subjects='popmean'), ctTIpredEffects) are
evaluated at the population T0MEANS state, with time dependent
predictors at zero.
Subject matrices (ctSubjectPars,
ctDiscretePars(subjects=)) are evaluated at each subject's last
observed row.
Only ctKalman and ctPredict avoid the choice
entirely, because they run the filter and re-evaluate every cell at every
step.
A reference to an individually varying parameter is not affected by any of this. ctsem represents such a parameter as a carrier latent state with no drift and no diffusion, so a cell reading that state simply is the parameter, and the estimate at the subject's last row is the one that has seen all of that subject's data – the best available, not an arbitrary point. Only references to real dynamic processes, and to time dependent predictors, make a reported value conditional.
Functions affected by this attach the cells in question as
attr(x, 'contextDependent'), and say so once in a message.
ctModelIsNonlinear answers the question directly. With
backend='julia' the evaluation point can be chosen; see the
state argument of ctBackendParMatrices.
ctModelIsNonlinear, ctBackendParMatrices
ctsem's covariance matrices – T0VAR, DIFFUSION,
MANIFESTVAR, and the population spreads RAWPOPVAR and
RAWPOPVAR_<idname> – are specified as a lower triangle: standard
deviations on the diagonal, and below it coordinates that the model's
covmattransform turns into correlations. Except for two variables
under 'z', where a coordinate is Fisher's z, what one coordinate
yields depends on the others, so the cells for a given covariance are not
something to work out by hand. ctCov() computes them.
ctCov(cov, covmattransform = "z")ctCov(cov, covmattransform = "z")
cov |
A symmetric positive definite covariance matrix. |
covmattransform |
The construction to write the cells for:
|
Pass the result wherever a fixed matrix goes: ctModel(DIFFUSION =
ctCov(S)), m$matrices$T0VAR <- ctCov(S), or m$RAWPOPVAR <-
ctCov(S), in every case for the whole matrix. The model keeps the
covariance as well as the cells, so the matrix keeps its meaning if
covmattransform is changed afterwards, and the R generator of
ctGenerate – which reads covariance cells as a Cholesky
factor – generates from the same covariance as the julia one. A cell edited
by hand afterwards replaces what ctCov() wrote.
For RAWPOPVAR, the covariance is that of the raw, untransformed
parameters across subjects (or groups, for RAWPOPVAR_<idname>),
which is the scale a fit reports it on.
The cells, a lower triangular numeric matrix of class ctCov,
with the covariance as attribute covariance. 'rawcorr' cannot
reach every correlation matrix, and a covariance it cannot represent is
refused.
S <- matrix(c(1, .6, .3, .6, 2, .5, .3, .5, 1.5), 3) ctCov(S) m <- ctModel(LAMBDA = diag(3), DIFFUSION = ctCov(S), Tpoints = 5)S <- matrix(c(1, .6, .3, .6, 2, .5, .3, .5, 1.5), 3) ctCov(S) m <- ctModel(LAMBDA = diag(3), DIFFUSION = ctCov(S), Tpoints = 5)
Performs a coverage check analysis by generating data from a model, fitting it multiple times
with different fit arguments, and plotting the results. Works with fitArgs/
fittingModel combinations that fit with either backend='julia' (the
ctFit default) or backend='stan'.
ctCoverageCheck( initialData, fittingModel, niter, fitArgs, cores = getOption("mc.cores", 2L), fitCores = 1, generateCores = fitCores, plotEvery = max(c(10, cores)) )ctCoverageCheck( initialData, fittingModel, niter, fitArgs, cores = getOption("mc.cores", 2L), fitCores = 1, generateCores = fitCores, plotEvery = max(c(10, cores)) )
initialData |
An initial dataset to fit to determine 'true' parameters for further generation. |
fittingModel |
A ctModel object used for fitting the data |
niter |
Number of iterations to run |
fitArgs |
Named list of fit argument sets to test (e.g., list(boot = list(optimcontrol = list(uncertainty = 'bootstrap')), hess = list(optimcontrol = list(uncertainty = 'hessian')))) |
cores |
The most CPU cores the check uses at once. Iterations run
|
fitCores |
Number of cores for each inner |
generateCores |
Number of cores for |
plotEvery |
Print plots every n iterations; by default every 10, or
every |
A list containing the results data.table and final plots
## Not run: # Does the Hessian interval cover as often as it says? 100 datasets # generated from the fit to ctstantestdat, each fitted twice. model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1)) check <- ctCoverageCheck(ctstantestdat, model, niter = 100, fitArgs = list( hessian = list(optimcontrol = list(uncertainty = 'hessian')), bootstrap = list(optimcontrol = list(uncertainty = 'bootstrap'))), cores = 4) ## End(Not run)## Not run: # Does the Hessian interval cover as often as it says? 100 datasets # generated from the fit to ctstantestdat, each fitted twice. model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1)) check <- ctCoverageCheck(ctstantestdat, model, niter = 100, fitArgs = list( hessian = list(optimcontrol = list(uncertainty = 'hessian')), bootstrap = list(optimcontrol = list(uncertainty = 'bootstrap'))), cores = 4) ## End(Not run)
Converts intervals in ctsem long format data to absolute time
ctDeintervalise(datalong, id = "id", dT = "dT", startoffset = 0)ctDeintervalise(datalong, id = "id", dT = "dT", startoffset = 0)
datalong |
data to use, in ctsem long format (attained via function ctWideToLong) |
id |
character string denoting column of data containing numeric identifier for each subject. |
dT |
character string denoting column of data containing time interval preceding observations in that row. |
startoffset |
Number of units of time to offset by when converting. |
datalong, with the interval column dT replaced by
absolute times and renamed time.
Calculate model implied regressions for a sequence of time intervals (if ct) or steps (if dt) based on
a ctStanFit object, for specified subjects. Wrap with print() when used inside for loops!
ctStanDiscretePars is maintained as a backward-compatible alias.
ctDiscretePars( fit, subjects = "popmean", times = seq(from = 0, to = 10, by = 0.1), nsamples = 200, impulseType = "unit", standardise = FALSE, cov = FALSE, plot = FALSE, cores = getOption("mc.cores", 2L), state = NULL, method = "linearise", realid = TRUE, ..., ctstanfitobj, observational )ctDiscretePars( fit, subjects = "popmean", times = seq(from = 0, to = 10, by = 0.1), nsamples = 200, impulseType = "unit", standardise = FALSE, cov = FALSE, plot = FALSE, cores = getOption("mc.cores", 2L), state = NULL, method = "linearise", realid = TRUE, ..., ctstanfitobj, observational )
fit |
model fit from |
subjects |
Either 'popmean', to use the population mean parameter, or the
subjects to use, as ids in the data (see |
times |
Numeric vector of positive values, discrete time parameters will be calculated for each. If the fit object is a discrete time model, these should be positive integers. |
nsamples |
Number of samples from the stanfit to use for plotting. Higher values will increase smoothness / accuracy, at cost of plotting speed. Values greater than the total number of samples will be set to total samples. |
impulseType |
What a "one unit change in process c" is taken to bring
with it. Every variant is
The companion matrices are asymmetric, and must be: a one unit move in a
wide-spread process implies more movement in a narrow one than the reverse.
|
standardise |
Logical, |
cov |
Logical. If TRUE, covariances are returned instead of regression coefficients. |
plot |
Logical. If TRUE, ggplots output using |
cores |
Number of cpu cores to use for computing subject matrices. If subject matrices were saved during fiting, not used. |
state |
Evaluation point for any matrix cell that depends on the latent
state or a time dependent predictor – see |
method |
How the interval regressions are obtained. |
realid |
If TRUE (the default), subjects are read as the ids in the data, and integers that are not ids there are read as ctsem's numbering of the subjects, 1 to N, with a message. If FALSE, they are read as that numbering. |
... |
additional plotting arguments to control |
ctstanfitobj |
Deprecated. Use |
observational |
Deprecated. Use |
Two of these combinations are quantities you may already know under other
names. impulseType='unit' with standardise=FALSE is the
discrete time autoregression and cross-lagged matrix, expm(DRIFT*t).
impulseType='observed' with standardise=TRUE is the model
implied cross-correlation function,
Cor(x_r(t+u), x_c(t)) – the model's counterpart to the empirical
autocorrelations ctACF computes from the data, and equal to
the latent correlation matrix at t=0.
If plot=TRUE, the function will return a ggplot2 object (and hence needs to be printed if intended to display within a loop). This can be modified by the various ggplot2 functions, or displayed using print(x).
With plot = FALSE, an array of the discrete time
coefficients, by sample, subject, time interval, row and column, of class
ctDiscretePars: summary() gives each curve's peak, the lag of
the peak, the lag at which it has fallen to half the peak (a half-life), and
the area under it, with their uncertainty (summary.ctDiscretePars).
On an optimised fit, with subjects = 'popmean', the same
coefficients at the fit's estimate are attr(x, 'estimate'), by
subject, time interval, row and column: for 'linearise' computed from
the matrices ctSummaryMatrices(fit, state = )$estimate reports, for
'simulate' simulated at the estimate. The draws' mean or median need
not equal them. summary() reads their quantities into an
estimate column, and ctDiscreteParsPlot draws them as
its line. A sampled fit has no estimate, and gives none; nor do
subjects other than 'popmean', whose matrices come from each
subject's filter pass over the draws.
With plot = TRUE, a ggplot object.
data.table::setDTthreads(1) #ignore this line ctDiscretePars(ctstantestfit,times=seq(.5,4,.1), plot=TRUE,indices='CR') #modify plot require(ggplot2) g=ctDiscretePars(ctstantestfit,times=seq(.5,4,.1), plot=TRUE,indices='CR') g= g+ labs(title='Cross effects') print(g)data.table::setDTthreads(1) #ignore this line ctDiscretePars(ctstantestfit,times=seq(.5,4,.1), plot=TRUE,indices='CR') #modify plot require(ggplot2) g=ctDiscretePars(ctstantestfit,times=seq(.5,4,.1), plot=TRUE,indices='CR') g= g+ labs(title='Cross effects') print(g)
Plots the output from ctDiscretePars, for model implied regression strengths at specified times for continuous time models fit with ctStanFit.
ctStanDiscreteParsPlot is maintained as a backward-compatible alias.
ctDiscreteParsPlot( x, indices = "all", quantiles = c(0.025, 0.5, 0.975), latentNames = "auto", ylab = "Coefficient", xlab = "Time interval", ylim = NA, facets = NA, splitSubjects = TRUE, colour = "Effect", title = "auto", polygonalpha = 0.1, ggcode = FALSE, line = c("estimate", "median") )ctDiscreteParsPlot( x, indices = "all", quantiles = c(0.025, 0.5, 0.975), latentNames = "auto", ylab = "Coefficient", xlab = "Time interval", ylim = NA, facets = NA, splitSubjects = TRUE, colour = "Effect", title = "auto", polygonalpha = 0.1, ggcode = FALSE, line = c("estimate", "median") )
x |
list object returned from |
indices |
Either a string specifying type of plot to create, or an n by 2 matrix specifying which indices of the output matrix to plot. 'AR' specifies all diagonals, for discrete time autoregression parameters. 'CR' specifies all off-diagonals,for discrete time cross regression parameters. 'all' plots all AR and CR effects at once. |
quantiles |
numeric vector of length 3, with values between 0 and 1, specifying which quantiles to plot.
The default plots 95% credible intervals as the band, and the posterior median at 50% as the line
where |
latentNames |
Vector of character strings denoting names for the latent variables. 'auto' just uses eta1 eta2 etc. |
ylab |
y label. |
xlab |
x label. |
ylim |
Two values, the lower and upper limits of the y axis. The plot zooms to them; intervals reaching past them are drawn to the edge. |
facets |
May be 'Subject' or 'Effect'. |
splitSubjects |
if TRUE, subjects are plotted separately, if FALSE they are combined. |
colour |
Character string denoting how colour varies. 'Effect' or 'Subject'. |
title |
Character string. 'auto' generates automatically, NULL can be used to disable title. |
polygonalpha |
Numeric between 0 and 1 to multiply the alpha of the fill. |
ggcode |
if TRUE, returns a list containing the data.table to plot (for each curve and time interval,
the outer |
line |
What the line is. |
For a model whose dynamics depend on the state, the curves are the model
linearised at one point, or simulated from it, and the subtitle names the
point. The default, the population initial state with time dependent
predictors at zero, can be far from where the data are; choose the point, and
whether to linearise or simulate, with ctDiscretePars(state = , method = ),
whose help says when the linearisation is a fair interval regression.
A ggplot2 object. This can be modified by the various ggplot2 functions, or displayed using print(x).
data.table::setDTthreads(1) #ignore this line x <- ctDiscretePars(ctstantestfit) ctDiscreteParsPlot(x, indices='CR') #to modify plot: g <- ctDiscreteParsPlot(x, indices='CR') + ggplot2::labs(title='My ggplot modification') print(g)data.table::setDTthreads(1) #ignore this line x <- ctDiscretePars(ctstantestfit) ctDiscreteParsPlot(x, indices='CR') #to modify plot: g <- ctDiscreteParsPlot(x, indices='CR') + ggplot2::labs(title='My ggplot modification') print(g)
Rounds timing information so equal intervals, according to specified timestep, are achieved. Each subject's times are moved to the nearest point of a grid that starts at the subject's first time; values that land on the same point are averaged, with a warning. NA's are inserted in other columns as necessary. Any columns specified by TDpredNames have zeroes rather than NA's inserted (because some estimation routines do not tolerate NA's in covariates), while columns specified by TIpredNames are left as NA's – since a time independent predictor is constant within a subject, inserting zeroes for it would be incorrect.
ctDiscretiseData( dlong, timestep, timecol = "time", idcol = "id", TDpredNames = NULL, TIpredNames = NULL )ctDiscretiseData( dlong, timestep, timecol = "time", idcol = "id", TDpredNames = NULL, TIpredNames = NULL )
dlong |
Long format data |
timestep |
Positive real value to discretise |
timecol |
Name of column containing absolute (not intervals) time information. |
idcol |
Name of column containing subject id variable. |
TDpredNames |
Vector of column names of any time dependent predictors |
TIpredNames |
Vector of column names of any time independent predictors |
long format ctsem data.
long <- ctDiscretiseData(dlong=ctstantestdat, timestep = .1, TDpredNames=c('TD1'),TIpredNames=c('TI1','TI2','TI3'))long <- ctDiscretiseData(dlong=ctstantestdat, timestep = .1, TDpredNames=c('TD1'),TIpredNames=c('TI1','TI2','TI3'))
Get documentation pdf for ctsem
ctDocs()ctDocs()
Nothing. Opens a pdf when run interactively.
ctDocs()ctDocs()
Fits one ctsem model per subject using the model prior, estimates the empirical marginal distribution of the raw parameters, then fits each subject again using the resulting empirical Bayes prior.
ctEmpiricalBayesFit( datalong, model, subjects = "all", priors = TRUE, optimize = TRUE, cores = getOption("mc.cores", 2L), subjectFitArgs = list(), Npasses = 2, ebUse = c("rawest", "rawposterior"), ebRobust = TRUE, ebOutlierMAD = 6, ebOutlierQuantiles = c(0.025, 0.975), ebWinsorize = TRUE, minsd = 1e-06, verbose = 0, progress = TRUE, ... )ctEmpiricalBayesFit( datalong, model, subjects = "all", priors = TRUE, optimize = TRUE, cores = getOption("mc.cores", 2L), subjectFitArgs = list(), Npasses = 2, ebUse = c("rawest", "rawposterior"), ebRobust = TRUE, ebOutlierMAD = 6, ebOutlierQuantiles = c(0.025, 0.975), ebWinsorize = TRUE, minsd = 1e-06, verbose = 0, progress = TRUE, ... )
datalong |
Long format data containing multiple subjects. |
model |
Model object from |
subjects |
Vector of subject identifiers to fit, or |
priors |
Logical. Passed to |
optimize |
Logical. Passed to |
cores |
Number of subjects to fit in parallel. Each individual
subject-level |
subjectFitArgs |
Named list of additional arguments passed to each
|
Npasses |
Total number of subject-wise fitting passes. The default
|
ebUse |
|
ebRobust |
Logical. If TRUE, the empirical Bayes prior is built from robust raw summaries after outlier handling. |
ebOutlierMAD |
Positive numeric. Raw values farther than this
many MAD-scaled deviations from the median are treated as outliers. Use
|
ebOutlierQuantiles |
Length two numeric vector of lower and upper
quantiles used to bound first-pass raw values, or |
ebWinsorize |
Logical. If TRUE, outlying first-pass raw values are clamped to the outlier bounds before computing the EB prior. If FALSE, they are set to missing for EB prior construction. |
minsd |
Lower bound used for empirical raw SDs before model adjustment. |
verbose |
Integer from 0 to 2. Passed to |
progress |
Logical. If TRUE, report the current EB fitting stage and overwrite a single console line with the subject fitting percentage. |
... |
Additional arguments passed to each |
Works with either backend='stan' or backend='julia', passed via
subjectFitArgs or ...; every per-subject ctFit
call uses the same backend. ebUse='rawposterior' needs each subject
fit to carry posterior draws (the default for both backends' optimised fits,
unless optimcontrol$estonly=TRUE), or a sampled fit
(optimize=FALSE, or for backend='julia' a Laplace fit passed
through ctFitUncertainty with uncertainty = 'sample').
Object of class ctEmpiricalBayesFit, containing the subject
fit lists and metadata. $initialfits contains the first-pass model
prior fits, $fits contains the final empirical Bayes prior fits, and
$passfits contains every pass. Use summary() to compute final
transformed-parameter means, SDs, covariances, correlations, and outlier
diagnostics.
if (isTRUE(ctJuliaStatus()$available)) { model <- ctModel(type='ct', manifestNames='Y1', LAMBDA=matrix(1)) eb <- ctEmpiricalBayesFit(ctstantestdat, model, cores=2, subjectFitArgs=list(optimcontrol=list(finishsamples=20))) summary(eb) }if (isTRUE(ctJuliaStatus()$available)) { model <- ctModel(type='ct', manifestNames='Y1', LAMBDA=matrix(1)) eb <- ctEmpiricalBayesFit(ctstantestdat, model, cores=2, subjectFitArgs=list(optimcontrol=list(finishsamples=20))) summary(eb) }
Simulated example dataset for the ctsem package
100 by 17 matrix containing containing ctsem wide format data. 6 measurement occasions of leisure time and happiness and 5 measurement intervals for each of 100 individuals.
Simulated example dataset for the ctsem package
100 by 18 matrix containing containing ctsem wide format data. 6 measurement occasions of leisure time and happiness, 1 measurement of number of friends, and 5 measurement intervals for each of 100 individuals.
Simulated example dataset for the ctsem package
100 by 18 matrix containing containing ctsem wide format data. 8 measurement occasions of leisure time and happiness, 7 measurement occasions of a money intervention dummy, and 7 measurement intervals for each of 50 individuals.
Simulated example dataset for the ctsem package
100 by 18 matrix containing ctsem wide format data. 8 measurement occasions of leisure time and happiness, 7 measurement occasions of a money intervention dummy, and 7 measurement intervals for each of 50 individuals.
Simulated example dataset for the ctsem package
1 by 399 matrix containing containing ctsem wide format data. 100 observations of variables Y1 and Y2 and 199 measurement intervals, for 1 subject.
Simulated example dataset for the ctsem package
20 by 79 matrix containing 20 observations of variables Y1, Y2, Y3, and 19 measurement intervals dTx, for each of 20 individuals.
Returns the posterior (or approximate posterior) draws of a fit's parameters, as arrays named for the model matrices.
ctExtract( object, subjectMatrices = FALSE, cores = getOption("mc.cores", 2L), nsamples = "all", subjects = "all", state = NULL, realid = TRUE, ... )ctExtract( object, subjectMatrices = FALSE, cores = getOption("mc.cores", 2L), nsamples = "all", subjects = "all", state = NULL, realid = TRUE, ... )
object |
A fit from |
subjectMatrices |
Calculate subject specific system matrices? On a
|
cores |
Only used if subjectMatrices = TRUE . For faster computation use more cores. |
nsamples |
either 'all' or an integer denoting number of random samples to extract. |
subjects |
either 'all', or the subjects to extract, as ids in the data
(see |
state |
The latent state at which state dependent cells of the
|
realid |
If TRUE (the default), subjects are read as the ids in the data, and integers that are not ids there are read as ctsem's numbering of the subjects, 1 to N, with a message. If FALSE, they are read as that numbering. |
... |
not used. |
A list of posterior sample arrays. The point the pop_*
matrices were evaluated at is recorded in the 'evaluatedAt' attribute;
nothing is messaged, because the summaries call this repeatedly and say it
once themselves. ctModelIsNonlinear answers whether the point
matters for this model at all.
Where parameters vary over subjects, the population of them, per draw:
popsd (draws by parameters), rawpopcov and rawpopcorr
(the raw scale's covariance and correlations, draws by parameters by
parameters) and popcov (the covariance on the natural scale). On a
backend='julia' fit they are named by parameter, and a model with
several grouping levels gives each level's as rawpopcov.<level> and so
on; its popcov combines popsd with the raw correlations, exact
where a parameter's transform is affine.
e = ctExtract(ctstantestfit) attr(e, 'evaluatedAt')e = ctExtract(ctstantestfit) attr(e, 'evaluatedAt')
Fits a ctsem model specified via ctModel with type either 'ct' or 'dt'.
ctStanFit is maintained as a backward-compatible alias.
ctFit( datalong, model, stanmodeltext = NA, iter = 1000, intoverstates = TRUE, binomial = FALSE, fit = TRUE, intoverpop = "auto", poprank = "auto", sameInitialTimes = FALSE, stationary = FALSE, plot = FALSE, derrind = NA, optimize = TRUE, optimcontrol = list(), backend = c("julia", "stan"), nlcontrol = list(), nopriors = NA, priors = FALSE, chains = 2, cores = getOption("mc.cores", 2L), inits = NULL, compileArgs = list(), forcerecompile = FALSE, saveCompile = TRUE, savescores = FALSE, savesubjectmatrices = FALSE, saveComplexPars = FALSE, gendata = FALSE, sampleControl = list(), control = list(), verbose = 0, ..., popunique = NULL, popblocks = NULL, weights = NULL, ctstanmodel )ctFit( datalong, model, stanmodeltext = NA, iter = 1000, intoverstates = TRUE, binomial = FALSE, fit = TRUE, intoverpop = "auto", poprank = "auto", sameInitialTimes = FALSE, stationary = FALSE, plot = FALSE, derrind = NA, optimize = TRUE, optimcontrol = list(), backend = c("julia", "stan"), nlcontrol = list(), nopriors = NA, priors = FALSE, chains = 2, cores = getOption("mc.cores", 2L), inits = NULL, compileArgs = list(), forcerecompile = FALSE, saveCompile = TRUE, savescores = FALSE, savesubjectmatrices = FALSE, saveComplexPars = FALSE, gendata = FALSE, sampleControl = list(), control = list(), verbose = 0, ..., popunique = NULL, popblocks = NULL, weights = NULL, ctstanmodel )
datalong |
long format data containing columns for subject id, time, manifest variables, any time dependent (i.e. varying within subject) predictors, and any time independent (not varying within subject) predictors. |
model |
model object as generated by |
stanmodeltext |
already specified Stan model character string, generally leave NA unless modifying Stan model directly.
(Possible after modification of output from fitting with argument fit=FALSE.) |
iter |
Deprecated – use |
intoverstates |
Logical. TRUE (the default) integrates over the latent
states with a Kalman filter. FALSE makes the states part of what is
estimated – the target becomes the joint density of the data and the
states, and the fit reports the trajectory in |
binomial |
Deprecated. Logical indicating the use of binary rather than Gaussian data, as with IRT analyses.
This now sets the |
fit |
If FALSE, prepare the model and data without fitting: stan returns its fit object unfitted, julia the specification its engine reads. |
intoverpop |
How declared individual differences (random effects) are handled: integrated out of the likelihood, or sampled with everything else.
|
poprank |
Rank of the population covariance of the individually
varying parameters. Requires |
sameInitialTimes |
if TRUE, include an empty observation for every subject that has no observation at the earliest observation time of the dataset. This ensures that the T0MEANS occurs for every subject at the same time, rather than just at the earliest observation for that subject. Important when modelling trends over time, age, etc. |
stationary |
Logical. If TRUE, each subject's latent processes start
from the distribution DRIFT, CINT and DIFFUSION settle into – the mean
To start only some initial conditions there, name those cells
|
plot |
if TRUE, for sampling, a Shiny program is launched upon fitting to interactively plot samples.
May struggle with many (e.g., > 5000) parameters. For optimizing, the optimisation trace is plotted –
with |
derrind |
deprecated, latents involved in dynamic error calculations are determined automatically now. |
optimize |
if TRUE, estimate by optimisation – maximum likelihood, or
maximum a posteriori with |
optimcontrol |
List of optimiser settings, read by both backends with
the same meaning: stopping rules ( |
backend |
Either 'julia' (the default) or 'stan'. The two take the same
model definitions and give the same summaries. The julia backend
optimises and samples ( |
nlcontrol |
List of non-linear control parameters.
|
nopriors |
deprecated, use priors argument. logical. If TRUE, any priors are disabled – sometimes desirable for optimization. |
priors |
|
chains |
Deprecated – use |
cores |
number of cpu cores to use: a positive whole number, or
|
inits |
|
compileArgs |
List of arguments to pass to |
forcerecompile |
logical. For |
saveCompile |
if TRUE and compilation is needed / requested, writes the stan model to the parent frame as ctsem.compiled (unless that object already exists and is not from ctsem), to avoid unnecessary recompilation. |
savescores |
Logical. If TRUE, output from the Kalman filter is saved in output. For datasets with many variables
or time points, will increase file size substantially. |
savesubjectmatrices |
Logical. If TRUE, subject specific matrices are saved –
only relevant when either time dependent predictors or individual differences are
used. Can increase memory usage dramatically in large models, and can be computed after fitting using ctExtract
or ctSubjectPars. |
saveComplexPars |
Logical. If TRUE, also save rowwise output of any complex parameters specified, i.e. combinations of parameters, functions and states. |
gendata |
Logical – If TRUE, uses provided data for only covariates and a time and missingness structure, and
generates random data according to the specified model / priors.
Generated data is in the $Ygen subobject after running |
sampleControl |
Used when For |
control |
Deprecated – use |
verbose |
Integer from 0 to 2. 1 reports progress while the model
fits; 2 additionally keeps every progress line rather than overwriting one
in place, and prints more for debugging. Whether overwriting is possible is
detected from where the output is going; set
|
... |
additional arguments to pass to |
popunique |
Own variances on a reduced-rank level: the parameters
selected get a variance of their own beside the |
popblocks |
Shared dimensions in blocks, on a reduced-rank level: a
list of blocks, each a list with |
weights |
Sampling weights, one per subject: the name of a column of
|
ctstanmodel |
Deprecated. Use |
A julia fit with random effects reports, at its estimate, how much of each
random effect each subject's own data determine, and how many subjects' worth
of information each population standard deviation rests on
(fit$identifiability$effects). An effect resting on fewer than two is
named in a message, in summary() and in ctReport, with a
lower poprank suggested where its level can take one and
indvarying = FALSE after it.
A fitted object of class ctStanFit (backend='stan') or
ctJuliaFit (backend='julia'), both also classed ctFit.
Besides backend-specific components, every fit carries $args, a list
with two sublists that mean the same thing on both backends:
inputExactly what was passed to ctFit(), or the
formal default when an argument was not supplied – intoverpop may
still read 'auto', and cores 'maxneeded' if that is
what was passed.
resolvedWhat the fit actually ran with, after 'auto'
routing, deprecated-argument merges (e.g. nopriors into
priors) and backend-specific defaults were settled – cores
is a concrete integer, intoverpop is one of 'augmented',
'laplace' or 'none', and priors is TRUE,
FALSE or 'randomCorr'. A call that gives the same input
on both backends gives the same resolved on both.
Before this, $args itself held the raw call on a stan fit and only
the resolved settings on a julia fit, so the same field name meant opposite
things across backends; code reading fit$args$intoverpop or
fit$args$cores directly should now read fit$args$resolved (or
$input, if the literal call is what is wanted, as
ctFitUpdate does).
ctOptimControl for every optimiser setting;
ctFitUncertainty for uncertainty after a fit;
ctIdentify reports which parameters the data can inform,
before a fit is spent finding out; ctTracePlot draws the
optimisation trace a julia fit records.
#generate a modern ctsem model relying heavily on defaults model<-ctModel(type='ct', latentNames=c('eta1','eta2'), manifestNames=c('Y1','Y2'), MANIFESTVAR=diag(.1,2), TDpredNames='TD1', TIpredNames=c('TI1','TI2','TI3'), LAMBDA=diag(2)) # The default backend is julia; ctJuliaInstall() sets it up. if (isTRUE(ctJuliaStatus()$available)) { fit<-ctFit(ctstantestdat, model,priors=TRUE) summary(fit) plot(fit,wait=FALSE) } #### extended examples library(ctsem) set.seed(3) # Data generation (run this, but no need to understand!) ----------------- Tpoints <- 20 nmanifest <- 4 nlatent <- 2 nsubjects<-20 #random effects age <- rnorm(nsubjects) #standardised cint1<-rnorm(nsubjects,2,.3)+age*.5 cint2 <- cint1*.5+rnorm(nsubjects,1,.2)+age*.5 tdpredeffect <- rnorm(nsubjects,5,.3)+age*.5 for(i in 1:nsubjects){ #generating model gm<-ctModel(Tpoints=Tpoints,n.manifest = nmanifest,n.latent = nlatent,n.TDpred = 1, type='omx', LAMBDA = matrix(c(1,0,0,0, 0,1,.8,1.3),nrow=nmanifest,ncol=nlatent), DRIFT=matrix(c(-.3, .2, 0, -.5),nlatent,nlatent), TDPREDMEANS=matrix(c(rep(0,Tpoints-10),1,rep(0,9)),ncol=1), TDPREDEFFECT=matrix(c(tdpredeffect[i],0),nrow=nlatent), DIFFUSION = matrix(c(1, 0, 0, .5),2,2), CINT = matrix(c(cint1[i],cint2[i]),ncol=1), T0VAR=diag(2,nlatent,nlatent), MANIFESTVAR = diag(.5, nmanifest)) #generate data newdat <- ctGenerate(ctmodelobj = gm,n = 1,burnin = 2, dtmat=rbind(c(rep(.5,8),3,rep(.5,Tpoints-9)))) newdat[,'id'] <- i #set id for each subject newdat <- cbind(newdat,age[i]) #include time independent predictor if(i ==1) { dat <- newdat[1:(Tpoints-10),] #pre intervention data dat2 <- newdat #including post intervention data } if(i > 1) { dat <- rbind(dat, newdat[1:(Tpoints-10),]) dat2 <- rbind(dat2,newdat) } } colnames(dat)[ncol(dat)] <- 'age' colnames(dat2)[ncol(dat)] <- 'age' #plot generated data for sanity plot(age) matplot(dat[,gm$manifestNames],type='l',pch=1) plotvar <- 'Y1' plot(dat[dat[,'id']==1,'time'],dat[dat[,'id']==1,plotvar],type='l', ylim=range(dat[,plotvar],na.rm=TRUE)) for(i in 2:nsubjects){ points(dat[dat[,'id']==i,'time'],dat[dat[,'id']==i,plotvar],type='l',col=i) } dat2[,gm$manifestNames][sample(1:length(dat2[,gm$manifestNames]),size = 100)] <- NA #data structure head(dat2) # Model fitting ----------------------------------------------------------- ##simple univariate default model m <- ctModel(type = 'ct', manifestNames = c('Y1'), LAMBDA = diag(1)) ctModelLatex(m) #Specify univariate linear growth curve m1 <- ctModel(type = 'ct', manifestNames = c('Y1'), latentNames=c('eta1'), DRIFT=matrix(0,nrow=1,ncol=1), DIFFUSION=matrix(0,nrow=1,ncol=1), T0VAR=matrix(0,nrow=1,ncol=1), CINT=matrix(c('cint1'),ncol=1), T0MEANS=matrix(c('t0m1'),ncol=1), LAMBDA = diag(1), MANIFESTMEANS=matrix(0,ncol=1), MANIFESTVAR=matrix(c('merror'),nrow=1,ncol=1)) ctModelLatex(m1) #fit if (isTRUE(ctJuliaStatus()$available)) { f1 <- ctFit(datalong = dat2, model = m1, optimize=TRUE, priors=FALSE) summary(f1) #plots of individual subject models v data ctPredict(f1,plot=TRUE,subjects=1,kalmanvec=c('y','yprior'),timestep=.01) ctPredict(f1,plot=TRUE,subjects=1:3,kalmanvec=c('y','ysmooth'),timestep=.01,errorvec=NA) #compare randomly generated data from the posterior to the observed data, #including the lagged covariance structure and the mean trajectory over time ctPostPredict(f1, wait=FALSE) } ### Further example models #Include intervention m2 <- ctModel(type = 'ct', manifestNames = c('Y1'), latentNames=c('eta1'), n.TDpred=1,TDpredNames = 'TD1', #this line includes the intervention TDPREDEFFECT=matrix(c('tdpredeffect'),nrow=1,ncol=1), #intervention effect DRIFT=matrix(-1e-5,nrow=1,ncol=1), DIFFUSION=matrix(0,nrow=1,ncol=1), CINT=matrix(c('cint1'),ncol=1), T0MEANS=matrix(c('t0m1'),ncol=1), T0VAR=matrix(0,nrow=1,ncol=1), LAMBDA = diag(1), MANIFESTMEANS=matrix(0,ncol=1), MANIFESTVAR=matrix(c('merror'),nrow=1,ncol=1)) #Individual differences in intervention, Bayesian estimation, covariates m2i <- ctModel(type = 'ct', manifestNames = c('Y1'), latentNames=c('eta1'), TIpredNames = 'age', TDpredNames = 'TD1', #this line includes the intervention TDPREDEFFECT=matrix(c('tdpredeffect||TRUE'),nrow=1,ncol=1), #intervention effect DRIFT=matrix(-1e-5,nrow=1,ncol=1), DIFFUSION=matrix(0,nrow=1,ncol=1), CINT=matrix(c('cint1'),ncol=1), T0MEANS=matrix(c('t0m1'),ncol=1), T0VAR=matrix(0,nrow=1,ncol=1), LAMBDA = diag(1), MANIFESTMEANS=matrix(0,ncol=1), MANIFESTVAR=matrix(c('merror'),nrow=1,ncol=1)) #Including covariate effects m2ic <- ctModel(type = 'ct', manifestNames = c('Y1'), latentNames=c('eta1'), n.TIpred = 1, TIpredNames = 'age', n.TDpred=1,TDpredNames = 'TD1', #this line includes the intervention TDPREDEFFECT=matrix(c('tdpredeffect'),nrow=1,ncol=1), #intervention effect DRIFT=matrix(-1e-5,nrow=1,ncol=1), DIFFUSION=matrix(0,nrow=1,ncol=1), CINT=matrix(c('cint1'),ncol=1), T0MEANS=matrix(c('t0m1'),ncol=1), T0VAR=matrix(0,nrow=1,ncol=1), LAMBDA = diag(1), MANIFESTMEANS=matrix(0,ncol=1), MANIFESTVAR=matrix(c('merror'),nrow=1,ncol=1)) m2ic$pars$indvarying[m2ic$pars$matrix %in% 'TDPREDEFFECT'] <- TRUE #Include deterministic dynamics m3 <- ctModel(type = 'ct', manifestNames = c('Y1'), latentNames=c('eta1'), n.TDpred=1,TDpredNames = 'TD1', #this line includes the intervention TDPREDEFFECT=matrix(c('tdpredeffect'),nrow=1,ncol=1), #intervention effect DRIFT=matrix('drift11',nrow=1,ncol=1), DIFFUSION=matrix(0,nrow=1,ncol=1), CINT=matrix(c('cint1'),ncol=1), T0MEANS=matrix(c('t0m1'),ncol=1), T0VAR=matrix('t0var11',nrow=1,ncol=1), LAMBDA = diag(1), MANIFESTMEANS=matrix(0,ncol=1), MANIFESTVAR=matrix(c('merror1'),nrow=1,ncol=1)) #Add system noise to allow for fluctuations that persist in time m3n <- ctModel(type = 'ct', manifestNames = c('Y1'), latentNames=c('eta1'), n.TDpred=1,TDpredNames = 'TD1', #this line includes the intervention TDPREDEFFECT=matrix(c('tdpredeffect'),nrow=1,ncol=1), #intervention effect DRIFT=matrix('drift11',nrow=1,ncol=1), DIFFUSION=matrix('diffusion',nrow=1,ncol=1), CINT=matrix(c('cint1'),ncol=1), T0MEANS=matrix(c('t0m1'),ncol=1), T0VAR=matrix('t0var11',nrow=1,ncol=1), LAMBDA = diag(1), MANIFESTMEANS=matrix(0,ncol=1), MANIFESTVAR=matrix(c(0),nrow=1,ncol=1)) #include 2nd latent process m4 <- ctModel(n.manifest = 2,n.latent = 2, type = 'ct', manifestNames = c('Y1','Y2'), latentNames=c('L1','L2'), n.TDpred=1,TDpredNames = 'TD1', TDPREDEFFECT=matrix(c('tdpredeffect1','tdpredeffect2'),nrow=2,ncol=1), DRIFT=matrix(c('drift11','drift21','drift12','drift22'),nrow=2,ncol=2), DIFFUSION=matrix(c('diffusion11','diffusion21',0,'diffusion22'),nrow=2,ncol=2), CINT=matrix(c('cint1','cint2'),nrow=2,ncol=1), T0MEANS=matrix(c('t0m1','t0m2'),nrow=2,ncol=1), T0VAR=matrix(c('t0var11','t0var21',0,'t0var22'),nrow=2,ncol=2), LAMBDA = matrix(c(1,0,0,1),nrow=2,ncol=2), MANIFESTMEANS=matrix(c(0,0),nrow=2,ncol=1), MANIFESTVAR=matrix(c('merror1',0,0,'merror2'),nrow=2,ncol=2)) #dynamic factor model -- fixing CINT to 0 and freeing indicator level intercepts m3df <- ctModel(type = 'ct', manifestNames = c('Y2','Y3'), latentNames=c('eta1'), n.TDpred=1,TDpredNames = 'TD1', #this line includes the intervention TDPREDEFFECT=matrix(c('tdpredeffect'),nrow=1,ncol=1), #intervention effect DRIFT=matrix('drift11',nrow=1,ncol=1), DIFFUSION=matrix('diffusion',nrow=1,ncol=1), CINT=matrix(c(0),ncol=1), T0MEANS=matrix(c('t0m1'),ncol=1), T0VAR=matrix('t0var11',nrow=1,ncol=1), LAMBDA = matrix(c(1,'Y3loading'),nrow=2,ncol=1), MANIFESTMEANS=matrix(c('Y2_int','Y3_int'),nrow=2,ncol=1), MANIFESTVAR=matrix(c('Y2residual',0,0,'Y3residual'),nrow=2,ncol=2))#generate a modern ctsem model relying heavily on defaults model<-ctModel(type='ct', latentNames=c('eta1','eta2'), manifestNames=c('Y1','Y2'), MANIFESTVAR=diag(.1,2), TDpredNames='TD1', TIpredNames=c('TI1','TI2','TI3'), LAMBDA=diag(2)) # The default backend is julia; ctJuliaInstall() sets it up. if (isTRUE(ctJuliaStatus()$available)) { fit<-ctFit(ctstantestdat, model,priors=TRUE) summary(fit) plot(fit,wait=FALSE) } #### extended examples library(ctsem) set.seed(3) # Data generation (run this, but no need to understand!) ----------------- Tpoints <- 20 nmanifest <- 4 nlatent <- 2 nsubjects<-20 #random effects age <- rnorm(nsubjects) #standardised cint1<-rnorm(nsubjects,2,.3)+age*.5 cint2 <- cint1*.5+rnorm(nsubjects,1,.2)+age*.5 tdpredeffect <- rnorm(nsubjects,5,.3)+age*.5 for(i in 1:nsubjects){ #generating model gm<-ctModel(Tpoints=Tpoints,n.manifest = nmanifest,n.latent = nlatent,n.TDpred = 1, type='omx', LAMBDA = matrix(c(1,0,0,0, 0,1,.8,1.3),nrow=nmanifest,ncol=nlatent), DRIFT=matrix(c(-.3, .2, 0, -.5),nlatent,nlatent), TDPREDMEANS=matrix(c(rep(0,Tpoints-10),1,rep(0,9)),ncol=1), TDPREDEFFECT=matrix(c(tdpredeffect[i],0),nrow=nlatent), DIFFUSION = matrix(c(1, 0, 0, .5),2,2), CINT = matrix(c(cint1[i],cint2[i]),ncol=1), T0VAR=diag(2,nlatent,nlatent), MANIFESTVAR = diag(.5, nmanifest)) #generate data newdat <- ctGenerate(ctmodelobj = gm,n = 1,burnin = 2, dtmat=rbind(c(rep(.5,8),3,rep(.5,Tpoints-9)))) newdat[,'id'] <- i #set id for each subject newdat <- cbind(newdat,age[i]) #include time independent predictor if(i ==1) { dat <- newdat[1:(Tpoints-10),] #pre intervention data dat2 <- newdat #including post intervention data } if(i > 1) { dat <- rbind(dat, newdat[1:(Tpoints-10),]) dat2 <- rbind(dat2,newdat) } } colnames(dat)[ncol(dat)] <- 'age' colnames(dat2)[ncol(dat)] <- 'age' #plot generated data for sanity plot(age) matplot(dat[,gm$manifestNames],type='l',pch=1) plotvar <- 'Y1' plot(dat[dat[,'id']==1,'time'],dat[dat[,'id']==1,plotvar],type='l', ylim=range(dat[,plotvar],na.rm=TRUE)) for(i in 2:nsubjects){ points(dat[dat[,'id']==i,'time'],dat[dat[,'id']==i,plotvar],type='l',col=i) } dat2[,gm$manifestNames][sample(1:length(dat2[,gm$manifestNames]),size = 100)] <- NA #data structure head(dat2) # Model fitting ----------------------------------------------------------- ##simple univariate default model m <- ctModel(type = 'ct', manifestNames = c('Y1'), LAMBDA = diag(1)) ctModelLatex(m) #Specify univariate linear growth curve m1 <- ctModel(type = 'ct', manifestNames = c('Y1'), latentNames=c('eta1'), DRIFT=matrix(0,nrow=1,ncol=1), DIFFUSION=matrix(0,nrow=1,ncol=1), T0VAR=matrix(0,nrow=1,ncol=1), CINT=matrix(c('cint1'),ncol=1), T0MEANS=matrix(c('t0m1'),ncol=1), LAMBDA = diag(1), MANIFESTMEANS=matrix(0,ncol=1), MANIFESTVAR=matrix(c('merror'),nrow=1,ncol=1)) ctModelLatex(m1) #fit if (isTRUE(ctJuliaStatus()$available)) { f1 <- ctFit(datalong = dat2, model = m1, optimize=TRUE, priors=FALSE) summary(f1) #plots of individual subject models v data ctPredict(f1,plot=TRUE,subjects=1,kalmanvec=c('y','yprior'),timestep=.01) ctPredict(f1,plot=TRUE,subjects=1:3,kalmanvec=c('y','ysmooth'),timestep=.01,errorvec=NA) #compare randomly generated data from the posterior to the observed data, #including the lagged covariance structure and the mean trajectory over time ctPostPredict(f1, wait=FALSE) } ### Further example models #Include intervention m2 <- ctModel(type = 'ct', manifestNames = c('Y1'), latentNames=c('eta1'), n.TDpred=1,TDpredNames = 'TD1', #this line includes the intervention TDPREDEFFECT=matrix(c('tdpredeffect'),nrow=1,ncol=1), #intervention effect DRIFT=matrix(-1e-5,nrow=1,ncol=1), DIFFUSION=matrix(0,nrow=1,ncol=1), CINT=matrix(c('cint1'),ncol=1), T0MEANS=matrix(c('t0m1'),ncol=1), T0VAR=matrix(0,nrow=1,ncol=1), LAMBDA = diag(1), MANIFESTMEANS=matrix(0,ncol=1), MANIFESTVAR=matrix(c('merror'),nrow=1,ncol=1)) #Individual differences in intervention, Bayesian estimation, covariates m2i <- ctModel(type = 'ct', manifestNames = c('Y1'), latentNames=c('eta1'), TIpredNames = 'age', TDpredNames = 'TD1', #this line includes the intervention TDPREDEFFECT=matrix(c('tdpredeffect||TRUE'),nrow=1,ncol=1), #intervention effect DRIFT=matrix(-1e-5,nrow=1,ncol=1), DIFFUSION=matrix(0,nrow=1,ncol=1), CINT=matrix(c('cint1'),ncol=1), T0MEANS=matrix(c('t0m1'),ncol=1), T0VAR=matrix(0,nrow=1,ncol=1), LAMBDA = diag(1), MANIFESTMEANS=matrix(0,ncol=1), MANIFESTVAR=matrix(c('merror'),nrow=1,ncol=1)) #Including covariate effects m2ic <- ctModel(type = 'ct', manifestNames = c('Y1'), latentNames=c('eta1'), n.TIpred = 1, TIpredNames = 'age', n.TDpred=1,TDpredNames = 'TD1', #this line includes the intervention TDPREDEFFECT=matrix(c('tdpredeffect'),nrow=1,ncol=1), #intervention effect DRIFT=matrix(-1e-5,nrow=1,ncol=1), DIFFUSION=matrix(0,nrow=1,ncol=1), CINT=matrix(c('cint1'),ncol=1), T0MEANS=matrix(c('t0m1'),ncol=1), T0VAR=matrix(0,nrow=1,ncol=1), LAMBDA = diag(1), MANIFESTMEANS=matrix(0,ncol=1), MANIFESTVAR=matrix(c('merror'),nrow=1,ncol=1)) m2ic$pars$indvarying[m2ic$pars$matrix %in% 'TDPREDEFFECT'] <- TRUE #Include deterministic dynamics m3 <- ctModel(type = 'ct', manifestNames = c('Y1'), latentNames=c('eta1'), n.TDpred=1,TDpredNames = 'TD1', #this line includes the intervention TDPREDEFFECT=matrix(c('tdpredeffect'),nrow=1,ncol=1), #intervention effect DRIFT=matrix('drift11',nrow=1,ncol=1), DIFFUSION=matrix(0,nrow=1,ncol=1), CINT=matrix(c('cint1'),ncol=1), T0MEANS=matrix(c('t0m1'),ncol=1), T0VAR=matrix('t0var11',nrow=1,ncol=1), LAMBDA = diag(1), MANIFESTMEANS=matrix(0,ncol=1), MANIFESTVAR=matrix(c('merror1'),nrow=1,ncol=1)) #Add system noise to allow for fluctuations that persist in time m3n <- ctModel(type = 'ct', manifestNames = c('Y1'), latentNames=c('eta1'), n.TDpred=1,TDpredNames = 'TD1', #this line includes the intervention TDPREDEFFECT=matrix(c('tdpredeffect'),nrow=1,ncol=1), #intervention effect DRIFT=matrix('drift11',nrow=1,ncol=1), DIFFUSION=matrix('diffusion',nrow=1,ncol=1), CINT=matrix(c('cint1'),ncol=1), T0MEANS=matrix(c('t0m1'),ncol=1), T0VAR=matrix('t0var11',nrow=1,ncol=1), LAMBDA = diag(1), MANIFESTMEANS=matrix(0,ncol=1), MANIFESTVAR=matrix(c(0),nrow=1,ncol=1)) #include 2nd latent process m4 <- ctModel(n.manifest = 2,n.latent = 2, type = 'ct', manifestNames = c('Y1','Y2'), latentNames=c('L1','L2'), n.TDpred=1,TDpredNames = 'TD1', TDPREDEFFECT=matrix(c('tdpredeffect1','tdpredeffect2'),nrow=2,ncol=1), DRIFT=matrix(c('drift11','drift21','drift12','drift22'),nrow=2,ncol=2), DIFFUSION=matrix(c('diffusion11','diffusion21',0,'diffusion22'),nrow=2,ncol=2), CINT=matrix(c('cint1','cint2'),nrow=2,ncol=1), T0MEANS=matrix(c('t0m1','t0m2'),nrow=2,ncol=1), T0VAR=matrix(c('t0var11','t0var21',0,'t0var22'),nrow=2,ncol=2), LAMBDA = matrix(c(1,0,0,1),nrow=2,ncol=2), MANIFESTMEANS=matrix(c(0,0),nrow=2,ncol=1), MANIFESTVAR=matrix(c('merror1',0,0,'merror2'),nrow=2,ncol=2)) #dynamic factor model -- fixing CINT to 0 and freeing indicator level intercepts m3df <- ctModel(type = 'ct', manifestNames = c('Y2','Y3'), latentNames=c('eta1'), n.TDpred=1,TDpredNames = 'TD1', #this line includes the intervention TDPREDEFFECT=matrix(c('tdpredeffect'),nrow=1,ncol=1), #intervention effect DRIFT=matrix('drift11',nrow=1,ncol=1), DIFFUSION=matrix('diffusion',nrow=1,ncol=1), CINT=matrix(c(0),ncol=1), T0MEANS=matrix(c('t0m1'),ncol=1), T0VAR=matrix('t0var11',nrow=1,ncol=1), LAMBDA = matrix(c(1,'Y3loading'),nrow=2,ncol=1), MANIFESTMEANS=matrix(c('Y2_int','Y3_int'),nrow=2,ncol=1), MANIFESTVAR=matrix(c('Y2residual',0,0,'Y3residual'),nrow=2,ncol=2))
Appends normal draws from the fit's already-computed covariance to its
raw posterior. Deprecated in favour of ctOptimUncertainty with
uncertainty = 'stored', which does the same on both backends.
ctFitAddSamples(fit, nsamples, cores = 2)ctFitAddSamples(fit, nsamples, cores = 2)
fit |
fit object |
nsamples |
number of samples desired |
cores |
number of cores to use |
These are pseudo-posterior draws from the fitted covariance, not
posterior draws from a sampler. ctFitUncertainty with
uncertainty = 'sample' is the latter – Hamiltonian Monte Carlo
from an optimized ctJuliaFit – and is a different object, not
another route to this one.
Deprecated in favour of ctOptimUncertainty(fit, uncertainty =
'stored', finishsamples = n), which draws from the same covariance for
the same zero model evaluations, works on a ctJuliaFit as well as a
ctStanFit, and records the new sample count in
$uncertainty$settings. It also replaces the draws rather than
appending to them: appending normal draws to a posterior that came from
uncertainty='is' or 'bootstrap' leaves part of each in one
matrix with nothing recording the mixture, which is what this function
does.
ctAddSamples is the same function under its pre-3.11 name; both are
deprecated together.
fit object with extra samples
ctOptimUncertainty, ctFitUncertainty
## Not run: newfit <- ctOptimUncertainty(ctstantestfit, uncertainty = 'stored', finishsamples = 30, cores = 1) ## End(Not run)## Not run: newfit <- ctOptimUncertainty(ctstantestfit, uncertainty = 'stored', finishsamples = 30, cores = 1) ## End(Not run)
Tests a fitted model in families, each aimed at one way a continuous time model can fall short, and gives each family a p-value. Most families use the one-step-ahead prediction errors, which under the model are independent and standard normal whatever the spacing of the observations; each variable's error is taken given the others observed at the same time, so a result names a variable.
ctFitAdequacy( fit, covariates = NULL, families = c("residuals", "latent", "volatility", "pairs", "periodic", "distribution", "fit", "dependence", "effects", "tvcovariates", "subjects", "nonlinear", "covariates"), gaps = NULL, periods = NULL, tvcovariates = NULL, close = 0.05, alpha = 0.05, projection = c("auto", "derivative", "plugin", "opg", "none"), parameters = NULL, nsamples = 100, lags = 0:3, nboot = 1999L, cores = getOption("mc.cores", 2L) )ctFitAdequacy( fit, covariates = NULL, families = c("residuals", "latent", "volatility", "pairs", "periodic", "distribution", "fit", "dependence", "effects", "tvcovariates", "subjects", "nonlinear", "covariates"), gaps = NULL, periods = NULL, tvcovariates = NULL, close = 0.05, alpha = 0.05, projection = c("auto", "derivative", "plugin", "opg", "none"), parameters = NULL, nsamples = 100, lags = 0:3, nboot = 1999L, cores = getOption("mc.cores", 2L) )
fit |
A fit from |
covariates |
Optional subject-level covariates, as for
|
families |
Which families to run; see Details. |
gaps |
Break points, in the data's time units, for the residual correlations' gap bins. By default the eighth, quarter and half quantiles of the time between every pair of a unit's rows. |
periods |
Periods, in the data's time units, for the |
tvcovariates |
Optional time varying covariates for the
|
close |
The root mean square residual correlation regarded as a close fit. |
alpha |
Level for |
projection |
How the error-based statistics account for estimation:
|
parameters |
For the |
nsamples |
Number of generated datasets, for the families that need
them. Reused from |
lags |
Observation lags for the |
nboot |
Bootstrap draws for each family's calibration. |
cores |
Cores for the engine and for generating. |
Correlations of the errors within units, for every pair
of rows, by time gap (gaps): the residual autocorrelation and
cross-correlation in continuous time. Slow structure the model lacks
shows as correlation over the gaps it spans.
The same for each latent process: the update each row makes to the predicted state, standardised, correlated with the same subject's updates at other times. One per process rather than one per item, so a misfit in the dynamics is not diluted by item-specific structure.
The same for the squared errors, per variable: variance that clusters in time.
Each pair of variables' errors in the same row against the correlation the model implies: a wrong measurement error covariance or loading.
A cycle the model lacks, such as a daily rhythm: each
variable's errors, and each latent's updates, correlated with the same
unit's at other times weighted by the cosine of the time between them
over a period – the periodogram of the errors at that period. It
depends only on the time between rows, so it finds a cycle whatever its
phase in each unit and needs no clock. At the named periods, or
by default scanning the periods a typical record resolves, from half its
length to twice the typical interval between rows. The estimate is the
errors' autocorrelation at the period: a cycle of amplitude a, in error
standard deviations, gives a^2 / 2.
Skewness and kurtosis of each variable's errors, and each row's squared whitened distance against its chi-square mean and variance, together with Anderson-Darling tests against the normal and chi-square.
How unevenly units are fitted: the spread of each unit's summed distance against the chi-square's, and the worst unit.
The distance correlation of each variable's error, and of its square, with what came before – the predicted latent states, the previous errors, the time interval – by permutation: a conditional mean or variance the model gets wrong, in any form.
On a single-level Laplace fit, whether the random effects' normal distribution is adequate: each subject's posterior probability, smoothed, that its standardised effect lies below a cut, at cuts along each axis, against the normal's. Under the model the posterior averages to the prior, so the differences have mean zero; a skewed or bimodal distribution, or a subgroup, moves them. These integrate the gradient function (Verbeke & Molenberghs, 2013; Drikvandi, Verbeke & Molenberghs, 2017), which is too heavy tailed to test directly.
With tvcovariates, whether time varying
variables the model does not use carry information about what it
predicts: each variable's error, and each latent's update, against each
covariate at the same row and at the unit's previous row, the covariate
standardised. The estimate is the regression of the standardised error
on the covariate. The previous row's value is a test of prediction
(Granger's sense); the same row's is an association, which a covariate
the outcome itself moves would show as well. Its null is larger than
the other families': the errors have mean zero given what the model
conditions on, the outcomes' own past, and need not be uncorrelated
with anything else. A correct model of the outcomes fails it when a
covariate is another measure of the same states, since that measure's
previous value sharpens the estimate of the state and so predicts the
next error. So it says whether a model using these variables would
predict better, not that the model of the outcomes is wrong, and it is
not counted in adequate.
Not run unless asked for. White's (1987) dynamic information
matrix test: each data row's score, the derivative of its one-step
log likelihood, for the dynamic and measurement parameters
(parameters), correlated with the scores of the same subject's
other rows, by time gap. Under the model a subject's row scores are
uncorrelated whatever the departure, so this is a check for dynamic
misspecification of any kind; being that broad, over many statistics,
it is the least powerful of the families, and the targeted tests of
ctScoreTest (stability, state dependence) find what they
cover far more often. It costs a dual pass of the filter per eight
parameters, and on the Laplace route one per truncated record, which is
hours on large data.
Statistics of the data
against datasets generated from the fit: how much subjects differ in
mean, standard deviation and lag-one correlations; whether change
depends on level nonlinearly; with covariates, whether subjects'
statistics depend on them; and the data's own lagged correlations
(lags). These use the data twice and are conservative; the
lagged correlations especially, as the fit reproduces them, so they are
not run unless asked for.
The error-based statistics are moment tests: each unit's contribution is
projected on its scores, which removes what estimating the parameters did
to the statistic (projection), and the variance is the errors' own:
they are independent over rows under the model, so a sum of products over
pairs of rows has the sum of the products' squares as its variance, which
holds with heavy tails or a changing variance; estimating the parameters
takes a known part of it away. That is the variance of the residual,
latent, volatility, within-row, periodic and covariate families. A periodic
statistic is in each unit a quadratic form of rank two in its errors, the
periodogram, skewed to the right however many units there are; its z is
read through a chi-square of the skewness that form has under the model,
worked out exactly from the unit's times. The distribution and fit
families, whose statistics are sums of high powers, use the spread of the
units' contributions and a studentised bootstrap over units. Within a family the
largest standardised statistic is calibrated with the statistics'
correlation over units, and the sum of their squares, where the variances
are the model's, by its terms between units, whose mean is zero under the
model (Chen & Qin, 2010), and otherwise by the bootstrap; the two are
combined, with any exact p-values the family has, by the Cauchy
combination; families are combined by Holm. adequate is TRUE
when no family's adjusted p-value is below alpha. The
tvcovariates family tests a larger information set than the model's
and is held out of both: its p-value is its own, and informative
reports it.
On a Laplace fit (intoverpop = 'laplace') whose random effects reach
the dynamics or the variances, the errors come from the model's own
predictive distribution: each row predicted from the subject's earlier rows
with the random effects integrated against their posterior given those rows
by importance sampling (getOption('ctsem.sequential.particles', 512)
quasi-random draws a subject), the own variances integrated in the filter
exactly. The draws are fixed by getOption('ctsem.sequential.seed'),
so a call repeats; another seed, or more draws, shows how much of a result
is the sampling's. The approximation the fit itself uses there – the posterior's
mode on each truncated record, its uncertainty added by the delta method –
is exact only for effects that enter linearly; elsewhere it put a 1.5 per
cent bias into the predictive variance, and correct models were rejected
through it. Where that predictive cannot be had – random effects at more
than one level – the row distances are left out and the latent family's
p-values are approximate, and the call says so.
A binary, ordinal, count or censored item's error is not normal, so its place is taken by its randomised probability integral transform through the normal quantile: the item's predictive distribution function at what was observed, given the subject's earlier rows and the items the filter had already applied in the row (non-Gaussian items one at a time, in item order), with a fixed uniform between its values just below and at the observation where that has probability mass. Under the model these are standard normal and independent, so every family reads them as it reads Gaussian errors; on the Laplace route they come from the exact predictive. The latent family's updates on such rows rest on the filter's Gaussian projection of each item, so its p-values are approximate there, and the call says so. Within a row, a non-Gaussian item's transform and a Gaussian item's error have no model-implied correlation the pairs family could test against, and are not paired.
With many observations every misfit is significant, so the sizes of the
residual, latent and within-row correlations the model leaves are reported
too, as
root mean squares, each squared correlation less its sampling variance (as
for the SRMR, Maydeu-Olivares 2017): the mean square is unbiased, its root,
truncated at zero, bias-corrected. Each has a 95 per cent interval and
p_close, the p-value of the test that it is at least close.
close_fit is TRUE when every family that fails is one of those
three and that test rejects at alpha – the size is shown to be
below close, its one-sided upper bound under it. A size whose
interval reaches close is not a close fit: not showing that a misfit
is large does not show that it is small. A close fit is about these
averages over many correlations; one large discrepancy among them, or a
biased estimate of a quantity of interest, can sit under a small average.
Detection is not localization: a failure in one variable's equation reaches
the others' errors through the fitted covariance. ctScoreTest
is the more sensitive check for the departures it covers, and
ctModelImprove uses both.
An object of class ctFitAdequacy: families (one row
per family: the number of statistics, the worst, its z, the family's
p-value and the Holm-adjusted one), statistics (every statistic:
what it is about – variable, with (the other variable of
a pair, or what the variable is correlated with), gap (the gap
bin of a pair of rows, or a lag), period and covariate,
each NA where it does not apply – and its observed value, z and
two-sided p), size (the root mean square residual correlations,
their intervals and close-fit tests), errors (the rows the
error-based families are computed from, in long form: Subject,
Time, Element – 'error', each variable's
standardised one-step-ahead error given the row's other observed
values; 'update', each latent's standardised update;
'etaprior', its predicted mean – Row, the variable, and
value; NULL when no such family ran), adequate,
close_fit, informative (with tvcovariates, whether
they predict the errors at alpha; NA without), projection
(the method used), and fit
(with any generated data, for reuse). plot() draws the families,
or one family's statistics.
Newey, W. K. (1985). Maximum likelihood specification testing and conditional moment tests. Econometrica, 53, 1047-1070.
White, H. (1987). Specification testing in dynamic models. In T. Bewley (Ed.), Advances in Econometrics: Fifth World Congress. Cambridge University Press.
Verbeke, G., & Molenberghs, G. (2013). The gradient function as an exploratory goodness-of-fit assessment of the random-effects distribution in mixed models. Biostatistics, 14, 477-490.
Drikvandi, R., Verbeke, G., & Molenberghs, G. (2017). Diagnosing misspecification of the random-effects distribution in mixed models. Biometrics, 73, 63-71.
Chang, J., Yao, Q., & Zhou, W. (2017). Testing for high-dimensional white noise using maximum cross-correlations. Biometrika, 104, 111-127.
Chen, S. X., & Qin, Y.-L. (2010). A two-sample test for high-dimensional data with applications to gene-set testing. The Annals of Statistics, 38, 808-835.
Liu, Y., & Xie, J. (2020). Cauchy combination test. Journal of the American Statistical Association, 115, 393-402.
Maydeu-Olivares, A. (2017). Assessing the size of model misfit in structural equation models. Psychometrika, 82, 533-558.
ctPostPredPlots and ctFitCheckCov to
see the data against the fit.
Compares empirical lagged covariances or correlations with the same quantity
computed from data generated by the fitted model. For each requested lag,
covariances or correlations are computed across subjects for each matching
observation-index pair. By default (x = "lag") they are then averaged
over observation-index pairs, which hides misfit that differs between waves,
such as a variance that changes at one wave; x = "obs" or a data
column instead shows each wave, or each value or bin of the column, as a
point of its own.
ctFitCovCheck is maintained as a backward-compatible alias.
ctFitCheckCov( fit, cor = TRUE, plot = TRUE, splitby = NULL, data = NULL, lags = NULL, variables = NULL, splitdata = data, split = NULL, breaks = 2, nsamples = NULL, minpairn = 10, cores = getOption("mc.cores", 2L), keep = c("summary", "samples"), x = "lag", xbins = 10, lagrows = c("all", "observed") )ctFitCheckCov( fit, cor = TRUE, plot = TRUE, splitby = NULL, data = NULL, lags = NULL, variables = NULL, splitdata = data, split = NULL, breaks = 2, nsamples = NULL, minpairn = 10, cores = getOption("mc.cores", 2L), keep = c("summary", "samples"), x = "lag", xbins = 10, lagrows = c("all", "observed") )
fit |
A fitted object from |
cor |
Logical; if 'TRUE' correlations are analysed instead of covariances. |
plot |
Logical; if 'TRUE' (default) a 'ggplot2' object (or list of plots) is returned. If 'FALSE', the raw 'data.table' with diagnostics is returned. |
splitby |
Optional character string giving the variable name on which to split subjects or observations. Numeric variables are split at the subject mean and then at the grand mean by default; non-numeric variables are treated as groups and may vary within subject. |
data |
Optional long-format data to use for the empirical lagged covariances or correlations. If omitted, the fitted data are used. |
lags |
Non-negative integer vector giving observation lags to compare. If 'NULL', '0:10' for 'x = "lag"' and '0:1' (variances or covariances, and lag-1 relations) otherwise. |
variables |
Optional character vector of manifest variables to include. Defaults to all manifest variables in the model. |
splitdata |
Optional data containing 'splitby', useful when the split variable was not in the fitted model. If omitted, 'data' is used when supplied, otherwise the fitted data are used. |
split |
Split rule for 'splitby': '"median"', '"mean"', '"quantile"', '"factor"', or '"none"'. If 'NULL', numeric split variables use '"mean"' and non-numeric split variables use '"factor"'. |
breaks |
Number of quantile groups when 'split = "quantile"'. |
nsamples |
Optional number of generated data sets to use. Existing generated data are reused when present; otherwise 'ctGenerateFromFit' is called. |
minpairn |
Minimum number of complete paired observations required for an empirical or generated lagged covariance/correlation: for each observation-index pair averaged under 'x = "lag"', and for each point otherwise. A point with fewer is left out of the plot and of the share outside the model's interval; raise it to drop bins that hold only a few off-schedule observations. |
cores |
Cores for generating the data when |
keep |
If '"samples"', draw-level model diagnostics are attached as the '"samples"' attribute of the returned data. |
x |
What the x axis shows. '"lag"' (default): one point per lag, averaged over observation indices. '"obs"': one point per observation index within subject (1 for each subject's first row), a panel per lag. Or the name of a column of 'data' (or of the fitted data), such as the time column or an age: one point per value of that column, or per quantile bin of it (see 'xbins'), a panel per lag. Each pair of observations is placed by its later observation, as a row-level split is. |
xbins |
Number of quantile bins for a numeric 'x' column with more distinct values than this; a column with no more is used as it is. The x axis shows each bin's mean. |
lagrows |
Which rows a lag counts. '"all"' (default): lag k pairs a row with the subject's k-th earlier row. '"observed"': with the k-th earlier row at which the column variable was observed, so a variable measured at only some waves is paired with its own previous measurement rather than with rows that hold only other variables. The gap in time between the two then varies more, and the model's data, which share the observed data's missingness, are paired the same way. |
Either a plot/list of plots (default, given by 'ctFitCovCheckPlot') or a 'data.table'.
ctPostPredPlots, the posterior predictive dashboard,
whose LaggedCovariance panel is this function without the splitting.
# Lagged correlations for all manifest variables, lags 0 to 3. ctFitCheckCov(ctstantestfit, cor = TRUE, lags = 0:3, plot = TRUE) # Lagged covariances for one manifest variable. ctFitCheckCov(ctstantestfit, cor = FALSE, variables = "Y1", lags = 0:5, plot = FALSE) # Variances, covariances and lag-1 covariances at each wave rather than # averaged over waves, and the same by bins of time. ctFitCheckCov(ctstantestfit, cor = FALSE, x = "obs") ctFitCheckCov(ctstantestfit, cor = FALSE, x = "time", xbins = 5) # Lag each variable by its own previous observation, for variables # measured at only some waves. ctFitCheckCov(ctstantestfit, x = "obs", lagrows = "observed") # Split the diagnostic by a time independent predictor in the fitted data. gg <- ctFitCheckCov(ctstantestfit, cor = TRUE, splitby = "TI1", lags = 0:3) print(gg[[1]]) # Use an external subject-level variable for the split. splitdat <- unique(data.frame(ctstantestdat)[, "id", drop = FALSE]) splitdat$group <- rep(c("a", "b"), length.out = nrow(splitdat)) ctFitCheckCov(ctstantestfit, splitby = "group", splitdata = splitdat, lags = 0:2) # Split observations into early and late periods within each subject. dat <- data.table::as.data.table(data.frame(ctstantestdat)) dat[, period := ifelse(time <= stats::median(time, na.rm = TRUE), "early", "late"), by = id] ctFitCheckCov(ctstantestfit, data = dat, splitby = "period", split = "factor", lags = 0:2)# Lagged correlations for all manifest variables, lags 0 to 3. ctFitCheckCov(ctstantestfit, cor = TRUE, lags = 0:3, plot = TRUE) # Lagged covariances for one manifest variable. ctFitCheckCov(ctstantestfit, cor = FALSE, variables = "Y1", lags = 0:5, plot = FALSE) # Variances, covariances and lag-1 covariances at each wave rather than # averaged over waves, and the same by bins of time. ctFitCheckCov(ctstantestfit, cor = FALSE, x = "obs") ctFitCheckCov(ctstantestfit, cor = FALSE, x = "time", xbins = 5) # Lag each variable by its own previous observation, for variables # measured at only some waves. ctFitCheckCov(ctstantestfit, x = "obs", lagrows = "observed") # Split the diagnostic by a time independent predictor in the fitted data. gg <- ctFitCheckCov(ctstantestfit, cor = TRUE, splitby = "TI1", lags = 0:3) print(gg[[1]]) # Use an external subject-level variable for the split. splitdat <- unique(data.frame(ctstantestdat)[, "id", drop = FALSE]) splitdat$group <- rep(c("a", "b"), length.out = nrow(splitdat)) ctFitCheckCov(ctstantestfit, splitby = "group", splitdata = splitdat, lags = 0:2) # Split observations into early and late periods within each subject. dat <- data.table::as.data.table(data.frame(ctstantestdat)) dat[, period := ifelse(time <= stats::median(time, na.rm = TRUE), "early", "late"), by = id] ctFitCheckCov(ctstantestfit, data = dat, splitby = "period", split = "factor", lags = 0:2)
Plot the results of ctFitCheckCov.
ctFitCovCheckPlot(x, maxlag = 10, vars = NA, splitvar = NA, cor = FALSE, ...)ctFitCovCheckPlot(x, maxlag = 10, vars = NA, splitvar = NA, cor = FALSE, ...)
x |
Data output from 'ctFitCheckCov(..., plot = FALSE)'. If a ggplot object or list of ggplot objects is supplied, it is returned unchanged. Data with an 'x' column (from 'ctFitCheckCov(x = )' other than '"lag"') are drawn against it, a panel per lag and variable. |
maxlag |
Maximum lag to plot. |
vars |
Optional character vector of variable names to plot. If 'NA', all variables are plotted. |
splitvar |
Optional character string specifying a variable to split the plot by. |
cor |
Logical; if 'TRUE', plot correlations instead of covariances. (label change) |
... |
not used. |
ggplot object.
## Not run: ctFitCovCheckPlot(ctFitCheckCov(ctstantestfit,cor=TRUE,plot=FALSE),maxlag=3) ## End(Not run)## Not run: ctFitCovCheckPlot(ctFitCheckCov(ctstantestfit,cor=TRUE,plot=FALSE),maxlag=3) ## End(Not run)
Fixes one raw parameter at a sequence of values, re-optimising all the
others at each, and reports what the likelihood does. This answers the
identification question directly rather than through the curvature at the
estimate, which on a flat direction measures the transform rather than the
data – see the note in ctFitUncertainty on flat directions.
ctFitProfile( fit, parameters = NULL, points = 8L, step = NULL, growth = 1.6, level = 0.95, maxiter = 500L, verbose = 0, cores = getOption("mc.cores", 2L) )ctFitProfile( fit, parameters = NULL, points = 8L, step = NULL, growth = 1.6, level = 0.95, maxiter = 500L, verbose = 0, cores = getOption("mc.cores", 2L) )
fit |
a |
parameters |
which parameters to profile: raw parameter names, indices,
|
points |
maximum points per side. |
step |
first displacement, in raw units. |
growth |
factor the step is multiplied by after a point that moved the likelihood by less than the target, so a flat direction is walked out quickly rather than in a hundred equal steps. |
level |
confidence level setting the likelihood-ratio bar. |
maxiter |
iteration cap for each constrained optimisation. |
verbose |
0 for silence, 1 to report each point as it is computed. |
cores |
The most CPU cores the profile uses at once. The walks – each
parameter's two sides – run one after another in this session, on that
many engine threads. Only when the fit itself took more than five minutes
to optimise do they run up to |
Each point is a constrained optimisation, so a profile can discover that the
fit was not at a maximum. When that happens the run stops and the better
point is returned in $better; refit from there and profile again.
An object of class ctFitProfile, which
print and plot both understand. $profile is one row
per point (parameter, raw value, transformed value where the coordinate
has one, log likelihood, drop from the estimate); $summary is one
row per parameter (verdict, limits, how far each side was walked); and
$better is NULL unless a constrained fit beat the estimate,
in which case it holds that point and the run stopped there.
# Needs a working julia backend, so the example is inert where Julia is # absent -- including on CRAN, whose check machines have none. if (isTRUE(ctJuliaStatus()$available)) { model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1)) model$pars$indvarying <- FALSE fit <- ctFit(ctstantestdat, model, backend = 'julia', cores = 1) # Two named parameters; parameters = "flagged" profiles only those the # fit's own diagnostics doubt, and NULL profiles every one. result <- ctFitProfile(fit, parameters = c('drift_eta1', 'mm_Y1')) print(result) plot(result) # Julia keeps running between calls, holding what it has compiled; # this ends it. ctJuliaKill("session") }# Needs a working julia backend, so the example is inert where Julia is # absent -- including on CRAN, whose check machines have none. if (isTRUE(ctJuliaStatus()$available)) { model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1)) model$pars$indvarying <- FALSE fit <- ctFit(ctstantestdat, model, backend = 'julia', cores = 1) # Two named parameters; parameters = "flagged" profiles only those the # fit's own diagnostics doubt, and NULL profiles every one. result <- ctFitProfile(fit, parameters = c('drift_eta1', 'mm_Y1')) print(result) plot(result) # Julia keeps running between calls, holding what it has compiled; # this ends it. ctJuliaKill("session") }
Recomputes the approximate raw-parameter uncertainty for an optimized
ctFit object and refreshes the approximate raw-parameter
samples, or samples the posterior by Markov chain Monte Carlo. This is the
entry point for both backends; ctFit itself calls it to finish an
optimized fit. ctOptimUncertainty is the previous name, kept
for code written against ctsem 3.11.1, with that release's arguments.
ctFitUncertainty( fit, method = c("hessian", "checked", "is", "sample", "sandwich", "opg", "bootstrap", "fullbootstrap", "stored"), intoverpop = NULL, filter = c("kalman", "particle"), checks = TRUE, finishsamples = NULL, cores = getOption("mc.cores", 2L), control = list(), verbose = 0, ... ) ctOptimUncertainty( fit, uncertainty = c("hessian", "checked", "is", "bootstrap", "fullbootstrap", "sandwich", "opg", "stored"), draws = c("auto", "normal", "empirical", "imis"), finishsamples = NULL, cores = getOption("mc.cores", 2L), control = list(), verbose = 0, ... )ctFitUncertainty( fit, method = c("hessian", "checked", "is", "sample", "sandwich", "opg", "bootstrap", "fullbootstrap", "stored"), intoverpop = NULL, filter = c("kalman", "particle"), checks = TRUE, finishsamples = NULL, cores = getOption("mc.cores", 2L), control = list(), verbose = 0, ... ) ctOptimUncertainty( fit, uncertainty = c("hessian", "checked", "is", "bootstrap", "fullbootstrap", "sandwich", "opg", "stored"), draws = c("auto", "normal", "empirical", "imis"), finishsamples = NULL, cores = getOption("mc.cores", 2L), control = list(), verbose = 0, ... )
fit |
Optimized |
method |
How the parameters' posterior is described:
|
intoverpop |
How the random effects are integrated, in
|
filter |
How the latent states are integrated: |
checks |
Which approximations to check where the method itself
leaves them in place: |
finishsamples |
Number of approximate raw-parameter samples. If
|
cores |
The most CPU cores the call uses at once, counting every
process it starts. |
control |
List of method-specific options. |
verbose |
Integer controlling progress detail. |
... |
Not used. Anything passed here is an error rather than ignored:
a method's settings are entries of |
uncertainty, draws
|
|
Every method except 'sample' writes pseudo-posterior draws to
$rawposterior: a sample from a covariance fitted to the
log-posterior surface around the optimum, not a sample from the posterior
itself, and the point estimate stays at the optimum – unless the method
is taken against a more exact likelihood than the fit's
(intoverpop = 'quadrature', filter = 'particle'), when it
moves to the corrected point.
method = 'sample' is the other thing, genuine posterior draws
by Markov chain Monte Carlo, and it moves the point estimate too:
$estimate$raw becomes the posterior mean, the way
ctFit(optimize = FALSE) reports a sampled fit. Everything downstream
reads either kind of draw from the same slot, so the difference is recorded
rather than visible in the shape of the result: a curvature-based fit
carries $uncertainty$settings, a sampled one carries $sample.
Every method except 'sample' refuses a fit that already carries a
posterior rather than replacing its draws with an approximation;
'sample' may be run again on a sampled fit, to draw more, or
differently, from where it stands.
To change only the number of draws, use method = 'stored', which
redraws from the covariance the fit already carries and evaluates no model.
The fit, of the class it came in as. For every method except
'sample': the resolved method, draw strategy, sample count, cores,
and non-internal controls are recorded in
fit$stanfit$uncertainty$settings for a ctStanFit and in
fit$uncertainty$settings for a ctJuliaFit; see Backend
differences for the other slots each writes. For
method = 'sample': a ctJuliaFit with
estimate$rawposterior holding the draws and $sample holding
the chain diagnostics (split R-hat and effective sample size per
parameter, divergences, tree depths, step sizes, E-BFMI (NUTS only),
sampler naming the kernel, converged
and diagnosis, and target naming which posterior was
sampled), exactly as ctFit(optimize = FALSE) returns.
ctOptimUncertainty(): The name this function shipped under in ctsem
3.11.1, kept so that a call written against that release keeps meaning
exactly what it did. The same as ctFitUncertainty in every
respect but its argument names (uncertainty, draws) and
that it has neither intoverpop, filter, checks nor
'sample'; new code should prefer ctFitUncertainty.
'checked'The Hessian's normal, checked, and
corrected by importance sampling in the directions where it is wrong. The
log posterior's gradient at draws from the normal says in which directions
the posterior departs from it, in batches of 8 until the directions named
would be the same with every departure two standard errors larger or
smaller, and the directions named are fewer than half the draws (16 draws
at least, and at most twice the parameters, between 64 and 256). Where a direction would leave some parameter's 95% interval
covering under 90% or over 99%, or move one tail outside 0.5 to 5% by
a skew or by a width that moves with another parameter, every departing
direction is sampled as 'is' samples (below), the normal kept in
all the rest. This is
meant to correct a variance from few groups and the effects it scales, but
it weighs the sampled directions with the others held near the centre, so
where their spread moves with the sampled ones – a population sd and the
effects it scales – intervals can remain too narrow, and nothing below
detects it: on models with random effects, sample ('sample') for
intervals to report. Where
that sampling cannot correct the normal – a curved ridge such as a loading
vector whose length the data determine and whose direction they barely do,
where the Hessian's intervals can be many times too wide – the Hessian's
normal is kept, with a warning naming the parameters it misstates. The
normal is centred where the same gradients, and those at the draws
reflected (at least 32 pairs), put the posterior's mean, which is not the
estimate wherever the posterior is skewed or a covariance is estimated;
the estimate stays the fit's, and summary() reports it beside the
draws. Where nothing is sampled, the same pairs' widths replace the
Hessian's along the directions they resolve, where they move an sd by 5%
or more. The sampled directions are weighed with the volume the others'
spread adds to them, to first order, and the sampled draws are checked
against the posterior's gradient at 100 of them (Stein's identity): where
a mean is off along the sampled directions, sampling goes on from a
component the gradient places (twice at most), and a warning names the
parameters whose mean the gradient puts half a normal sd or more away, or
whose spread it puts under half or over twice the posterior's. A fit with nothing named
gets the Hessian's covariance, so corrected. The check is recorded
in $uncertainty$details$normalcheck, and the sampling, when there was
any, in $uncertainty$details$importance_sampling.
'hessian'The default, as ctFit's: a normal
approximation from the Hessian at the estimate – exact on julia, which differentiates its own
gradient, and by finite differences of the gradient on stan. On julia
the interval check (checks) is made beside it, and the normal is
reported as it is.
'bootstrap', 'sandwich', 'opg'
Built from per-subject score contributions: one-step score bootstrap draws with Hessian bread, Hessian bread with score covariance meat, and the outer product of the scores. A single-subject model uses case-level contributions, with a warning; fewer than ten subjects, or no more score rows than parameters, also warn.
'fullbootstrap'Resamples subjects and re-optimizes each resample from the estimate with mize L-BFGS. Stan only.
'is'Adaptive importance sampling against the fit's own
objective. Each raw parameter is first walked out on both sides of the
mode, from 3 to 96 standard errors, with the other parameters at their
conditional mode, and a proposal component is placed wherever that walk
finds a tail the curvature does not show – a skewed one, or one bending
along a ridge, as a variance with few subjects often has. Rounds of
isitersize draws from these multivariate t components then add
components at the highest-weighted draws, and up to three more where the
weights show the proposal falls short, each moved by a Newton step and
shaped by the log posterior's curvature there, until the effective sample
size reaches isESS and the Pareto k of the weights is below 0.7 (k
needs the loo package). The effective size, k, the evaluations used and the
parameters with a tail are recorded in
fit$uncertainty$details$importance_sampling; k above 0.7 warns, and
so does a parameter whose walk has not fallen off by 96 standard errors,
which is what an improper posterior looks like. A rank deficient Hessian
covariance is sampled in the directions it has curvature in, holding the
rest at the estimate.
On a fit whose random effects are integrated (intoverpop =
'laplace' or 'augmented') the objective is the route's approximate
marginal posterior, so by default 'is' corrects the shape of the
normal approximation but keeps the route's own approximation: where the
Laplace approximation is biased – random effects on variances, in
nonlinear dynamics, or with non-Gaussian indicators – 'is'
reproduces the bias. Expect it to cost several times the fit.
intoverpop = 'quadrature' (julia, a Laplace fit) draws against a
more exact density, with 'is' or 'checked': the quadrature
hybrid, which has the units the screen flags,
and those the fit's check named, integrated by quadrature rules
(control$nodes per effect, 5 by default) or draws, the rest by
their Laplace terms, with the error of that density reported in
fit$laplace$correction$error and in summary(). A fit not
refined when it was fitted – the default, which only checks and warns –
is climbed to that density's optimum first; one refined when it was fitted
stays where it is, as does any fit under correctEstimate = FALSE.
With 'quadrature', 'hessian' is taken too: the hybrid's
curvature where the climb ends, which is what
ctFit(intoverpop = 'quadrature') gives at fit time, with the
interval check made on the hybrid there.
Where the climb stops short of its optimum, the intervals are the Laplace
objective's at its optimum, moved with the estimate, as at fit time.
filter = 'particle', for a fit on the filter's route and with
'is', replaces the
filter's Gaussian likelihood by a particle filter's (control$particles,
1000; substeps, 10; transition, 'exponential' or
'euler'; seed, 1, with draw i using seed + i -
1), the likelihood ctParticleLik measures the filter
against; its Monte Carlo noise widens the weights without biasing them, each
draw costs one particle-filter evaluation, the proposal starts from the
fit's covariance widened by imisScaleInit (1.5) and runs at most
imisMaxIter (10) rounds of isitersize draws, and the estimate
moves to the weighted posterior mean unless correctEstimate = FALSE.
What was done is recorded in fit$laplace$correction or, printing,
fit$particle_correction. Where too few effective draws survive the
approximation is too far from the target to repair by reweighting, and
'sample' with intoverpop = 'none' is the answer instead.
'stored'Computes nothing: redraws finishsamples normal
draws from the covariance already on the fit, leaving the covariance, the
Hessian and the recorded method untouched, and marks
$uncertainty$settings$redrawn = TRUE. It warns if the existing draws
came from 'is' or a bootstrap, which normal draws from that
covariance do not reproduce.
'sample'Markov chain Monte Carlo; see the next section.
backend = 'julia' only.
Every method that produces draws aims for the same effective sample size
unless told otherwise, 200: minESS for 'sample' (of the worst
parameter, since the 2.5% and 97.5% quantiles need it for each) and
isESS for 'is', whatever its likelihood. Each warns when it
ends short of its target.
method = 'sample')Draws from the posterior by Markov chain Monte Carlo, through the same
runner ctFit(backend = 'julia', optimize = FALSE) uses, and with the
same defaults. Its settings are entries of control, and
draws and finishsamples are not used: chains (4),
warmup (200; not used by the map sampler, which learns its map over
warm-up windows of its own first), draws
(800, after warmup), seed,
saveEffects (FALSE; whether to keep every draw of every random
effect rather than only their summary – either way they are each
parameter's deviation from its population value on the raw scale, per
subject or group, in fit$sample$effects, effect_mean and
effect_sd) and processes (FALSE).
What is sampled is what the fit's intoverpop says, read as
ctFit(optimize = FALSE) reads it: the filter's marginal for
'augmented', the Laplace marginal for a fit that named
'laplace' (an approximate posterior whose dimension does not grow
with the number of subjects), and the joint posterior over population
parameters and every subject's random effects for 'none' or
for a Laplace route that intoverpop = 'auto' chose, where the
Laplace fit only places the chains. The argument intoverpop overrides it
in the same words: 'none' for the joint posterior or
'laplace' for the Laplace marginal on a Laplace fit;
'augmented' needs a fit made that way, and an augmented fit takes
nothing else. Missing TI predictor values that the fit imputed are sampled
too, unless the fit's optimcontrol$tipredMissing or
control$tipredMissing is 'impute'.
The kernel. control$sampler chooses it for the joint
posterior. By default it is 'map'. 'saem' draws the random effects by
SAEM's Metropolis-within-Gibbs sweeps and the parameters given them by NUTS,
with moves that re-express the effects for each level's means and scales;
'nuts' runs NUTS on the parameters and every random effect at once;
'map' runs NUTS through a learnt map, with the random effects the
Kalman filter can integrate exactly (on intercepts, initial means, manifest
means and time dependent predictor effects, in a linear Gaussian model)
integrated out and the rest partially non-centred, unit by unit, from
Gaussian summaries of each subject's likelihood learnt in warm-up; it
starts from the fit's estimate (placement = 'fit') and reports every
random effect, the integrated ones drawn from their exact conditional.
All draw the same posterior. In comparisons on variance, nested,
categorical and nonlinear models the map reached the default effective
sample size where SAEM's kernel often ran out of draws, and was usually the
faster; SAEM's kernel was the more reliable and the faster than
'nuts', which mixes poorly on the joint vector. A marginal target is sampled by tempered
sequential Monte Carlo where the interval check of 'checked' named
directions in which the posterior departs from the fit's Hessian, and by
NUTS where it named none, or where the call sets a control only NUTS reads
(chains, warmup, stepsize, adapt_metric,
adapt_effects, settleTol); sampler = 'nuts' or
'smc' chooses, and 'smc' beside chains,
warmup or stepsize is an error, since it reads none of
them. A fit made without that check gets one first, at a few hundred
gradients. NUTS starts from a metric set to the widths the check measured
along those directions, or, where 'checked' moved the normal to the
posterior's moments, to that normal's covariance. Tempered SMC carries
draws from that normal to the posterior by stages of NUTS moves,
particles (200) chains a stage, keeping every state they visit,
then runs as many chains on as the effective sample size target needs.
It is for posteriors NUTS does not
converge on from the normal – a mass far from the mode, a loading shell –
and costs about what NUTS does in gradients. Its
particles move on a coreset of the units (coreset, TRUE): the
full density, evaluated once a particle a stage, corrects the moves
exactly, so a fit with many units pays a fraction of a gradient a move and
coreset = FALSE is the oracle. Its chains are short and start
from particles already drawn from the posterior, so it reports no
R-hat; the effective sample size is its stop. NUTS
too runs its trajectories on a coreset, fitted before warm-up and again
during it, each draw's end accepted or refused by the full density, which
keeps the draws exact whatever the coreset. adapt_metric, adapt_effects and
settleTol belong to NUTS's warmup and are refused with SAEM's kernel and with
tempered SMC, and the latter two with the map. control$placement says
where the chains start for SAEM's kernel and NUTS: 'saem', the
default on the joint posterior,
runs SAEM from the fit's estimate and starts each chain from SAEM's
estimate and its draws of the random effects; 'fit' starts them
around the fit's own estimate, and is the only placement for a marginal
target.
When it stops. A run stops once every parameter's effective sample
size reaches minESS (200) and the chains' R-hat is below
rhatTarget (1.01) – meanESS adds a target for the average –
within a budget of four times draws per chain, or maxDraws
when given; minESS = 0 takes exactly draws. A run that
reaches its budget short of the target warns. settleTol ends warmup
early once the metric stops moving (off by default). Lower targets give a
quicker, approximate posterior, and say how approximate: a reported
quantile's Monte Carlo error is about 1.96 * 2.1 / sqrt(ESS)
posterior sds at the 5% and 95% points and 1.96 * 1.25 / sqrt(ESS)
at the median, and chains whose means disagree by d posterior sds
give an R-hat of about sqrt(1 + d^2). So minESS = 50,
rhatTarget = 1.07 aims at about half a posterior sd at the centre and more
in the tails, and the defaults at about a fifth at the centre and two
fifths in the tails. Short runs are usually limited by whether the chains
agree, so R-hat is the binding check.
Processes. By default the chains run at once in this session,
sharing its cores. With processes = TRUE they run in up to
cores worker processes instead, each starting its own Julia session
– a fixed cost that a short run does not recover – and a worker runs its
share of the chains in turn, so four chains on two cores run two at a time.
ctJuliaKill('workers') releases the worker processes, which
persist for the next call.
The methods are the same on both backends and the covariance they produce is comparable, but some things differ and are worth knowing before comparing output.
Where the result is stored. A ctStanFit comes back with
fit$stanfit$cov, fit$stanfit$rawposterior and
fit$stanfit$uncertainty; a ctJuliaFit with
fit$estimate$cov, fit$estimate$se,
fit$estimate$rawposterior and fit$uncertainty. Both record the
resolved settings in $uncertainty$settings. On both backends the
draws, the covariance and the standard errors are labelled by raw parameter,
as is coef(fit), so fit$estimate$se['drift'] or
apply(fit$estimate$rawposterior, 2, quantile) reads without counting
columns. The two backends spell the population-SD and correlation blocks
differently (popsd_x/rawcor_x__y on julia,
x_SD/y_x_corr on stan); the order is the same.
The Hessian. The stan path finite-differences its gradient with a
single global step (control$hessianStep). The julia engine
differentiates its own reverse-mode gradient in forward mode, so its
Hessian is exact; control$analyticHessian = FALSE falls back to the
shared finite difference for comparison. The two agree to about 1e-4 on a
well-conditioned model.
Directions with no curvature. A direction whose curvature is
negligible against the sharpest one is left out of the inversion rather
than floored at control$ridge, so the parameters that carry it come
back with no spread at all instead of a standard error of 1/ridge.
Floored, that number leaks into the parameters the data does
determine, by an amount rounding decides. A direction with little
curvature along which the likelihood itself is measured flat – within the
likelihood-ratio bar of 1.92 over four raw units on one side, following the
ridge where it curves – is treated the same way. $uncertainty$cov
records which directions were dropped, and fit$identifiability names
the parameters.
Whether the intervals are as wide as the curvature allows. A julia
fit carries fit$uncertainty$intervalcheck: for each raw parameter,
the reported standard error against 1/sqrt(information[i,i]), the
width that parameter's own curvature supports. Their ratio is 1 when the
parameter is separable from the rest and grows without bound as it stops
being; anything past about 100 means the reported width comes from the
entanglement rather than from the data, and will not repeat between runs.
$nullmass is each coordinate's share of the directions left out of
the inversion, and $unidentified names those with enough of it that
their true asymptotic variance is infinite; summary() reports their
sd, interval and z as NA rather than as precisely estimated.
Backend-specific arguments. method = 'fullbootstrap' and
its control$bootstrapFitCores / control$bootstrapTol, and
control$parsteps, are stan-only and are refused by name on a
ctJuliaFit; method = 'sample' and
control$analyticHessian are julia-only. cores means R worker
processes on stan and engine threads on julia. The IMIS proposal defaults
also differ: imisScaleInit = 1.1 and imisTailScale = 1.1 on
stan; on julia the same for 'checked', whose proposal starts at the
widths its check measured, and 1.5 and 1.2 for 'is',
whose proposal starts at the Hessian curvature, the wider pair being what a
small-sample posterior wider than that curvature needs to be seen at all.
Set them explicitly to compare the two. Transformed-parameter
summaries are refreshed on stan and not on julia; the raw-scale covariance
and draws are complete on both.
ctFitAddSamples is the deprecated stan-only
predecessor of method = 'stored'. ctFit for
optimize = FALSE, which reaches method = 'sample'
through the same pipeline that placed the fit being sampled.
ctLaplaceCheck and ctParticleLik measure the
approximations a julia fit's likelihood makes, which
intoverpop = 'quadrature' and filter = 'particle' correct.
# Redraw from the covariance already on the fit, evaluating no model: fit <- ctFitUncertainty(ctstantestfit, method = 'stored', finishsamples = 200) fit$stanfit$uncertainty$settings$redrawn # The Hessian's normal, checked, and importance sampled where it is wrong fit <- ctFitUncertainty(ctstantestfit, method = 'checked') summary(fit)$popmeans # Needs a working julia backend, so the example is inert where Julia is # absent -- including on CRAN, whose check machines have none. if (isTRUE(ctJuliaStatus()$available)) { model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1)) model$pars$indvarying <- FALSE jfit <- ctFit(ctstantestdat, model, backend = 'julia', cores = 1, priors = TRUE) # Posterior draws by MCMC, with lower targets for a quick look jfit <- ctFitUncertainty(jfit, method = 'sample', cores = 1, control = list(chains = 2, minESS = 50, rhatTarget = 1.05)) jfit$sample # Julia keeps running between calls, holding what it has compiled; # this ends it. ctJuliaKill("session") }# Redraw from the covariance already on the fit, evaluating no model: fit <- ctFitUncertainty(ctstantestfit, method = 'stored', finishsamples = 200) fit$stanfit$uncertainty$settings$redrawn # The Hessian's normal, checked, and importance sampled where it is wrong fit <- ctFitUncertainty(ctstantestfit, method = 'checked') summary(fit)$popmeans # Needs a working julia backend, so the example is inert where Julia is # absent -- including on CRAN, whose check machines have none. if (isTRUE(ctJuliaStatus()$available)) { model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1)) model$pars$indvarying <- FALSE jfit <- ctFit(ctstantestdat, model, backend = 'julia', cores = 1, priors = TRUE) # Posterior draws by MCMC, with lower targets for a quick look jfit <- ctFitUncertainty(jfit, method = 'sample', cores = 1, control = list(chains = 2, minESS = 50, rhatTarget = 1.05)) jfit$sample # Julia keeps running between calls, holding what it has compiled; # this ends it. ctJuliaKill("session") }
Either to include different data, or because you have upgraded ctsem and the internal data structure has changed. Works on a fit from either backend.
ctFitUpdate(oldfit, data = NA, recompile = FALSE, refit = FALSE, ...)ctFitUpdate(oldfit, data = NA, recompile = FALSE, refit = FALSE, ...)
oldfit |
fit object to be updated, from |
data |
replacement long format data object. If not supplied, the data the fit was made with. |
recompile |
whether to force a recompile of the Stan model – safer but slower and usually unnecessary. Stan fits only. |
refit |
if TRUE, refits the model using the old estimates as a starting point. Only applicable for optimized fits, not sampling. If FALSE, the old fit is returned with its data and model specification replaced; its estimates, and everything computed from them when it was fitted, are left as they were. |
... |
extra arguments to pass to ctFit |
The fit is rebuilt from its own call: the arguments it was made with are
passed to ctFit again, with the model it was given and, unless
data is supplied, the data it was fitted to. Arguments in ...
replace the fit's own.
With refit=FALSE and data, the data are instead prepared under
the model the fit ran, so the fit keeps its structure along with its
estimates whatever the data – a single subject included, which
ctFit would set up without random effects. To predict for
other data, ctPredict(fit, newdata = data) does the same
without an update.
updated fit object, of the same class as oldfit.
newfit <- ctFitUpdate(ctstantestfit,refit=FALSE)newfit <- ctFitUpdate(ctstantestfit,refit=FALSE)
This function generates data according to the specified ctsem model object.
ctGenerate( ctmodelobj, n = 100, burnin = 0, dtmean = 1, logdtsd = 0, dtmat = NA, Tpoints = NULL, wide = FALSE, backend = c("auto", "r", "julia", "stan"), intoverstates = "auto", fromPriors = FALSE, nsamples = 200, datastruct = NA, cores = getOption("mc.cores", 2L), values = NULL, n.subjects )ctGenerate( ctmodelobj, n = 100, burnin = 0, dtmean = 1, logdtsd = 0, dtmat = NA, Tpoints = NULL, wide = FALSE, backend = c("auto", "r", "julia", "stan"), intoverstates = "auto", fromPriors = FALSE, nsamples = 200, datastruct = NA, cores = getOption("mc.cores", 2L), values = NULL, n.subjects )
ctmodelobj |
ctsem model object from |
n |
Number of subjects to output. For a model with grouping
levels above the subject ( |
burnin |
Number of initial time points to discard (to simulate stationary data) |
dtmean |
Positive numeric. Median time interval (delta T) to use.
Intervals are drawn as |
logdtsd |
Numeric. Standard deviation of the log time interval. Zero gives an equal-interval design. |
dtmat |
Either NA, or numeric matrix of n rows and Tpoints-1 columns, containing positive numeric values for all time intervals between measurements. If not NA, dtmean and logdtsd are ignored. |
Tpoints |
Optional number of time points to generate. If supplied, this overrides
any |
wide |
Logical. Output in wide format? |
backend |
Which generator to use: |
intoverstates |
For
|
fromPriors |
Draw the parameters from the model's priors rather than
using the values the model specifies, and return |
nsamples |
With |
datastruct |
A long format data frame giving the design to generate
for, in place of the balanced one |
cores |
With |
values |
For |
n.subjects |
Deprecated; use |
Covariance related matrices are treated as Cholesky factors. TRAITTDPREDCOV and TIPREDCOV matrices are not accounted for, at present. The first 1:n.TDpred rows and columns of TDPREDVAR are used for generating tdpreds at each time point.
From the R generator, a long format matrix with one row per subject
and time point, or wide format with wide = TRUE. From the julia
generator, the same long format, carrying the attributes described under
Generating values. With fromPriors = TRUE, a list: Y, an
nsamples by data rows by manifest variables array of generated
data, and llrow, the matching array of row log likelihoods.
With backend='julia' every value the model leaves free is fixed by
the scale of its prior, and all of them are named in a message:
a free parameter is at its prior's centre, raw zero, unless
values names it;
an individually varying parameter's population sd is its
sdscale, at every level it varies at, on the raw,
untransformed scale, unless RAWPOPVAR (RAWPOPVAR_<level>)
states a number, and random effects are uncorrelated unless it states a
coordinate – ctCov writes one from a covariance;
a time independent predictor is drawn standard normal per subject,
and an effect fixed in the model ('mm||||TI1=0.5') shifts the raw
parameter by that much per unit of the predictor; an effect left free is
zero.
fromPriors=TRUE draws all of these from the priors instead.
The julia generator also returns what it generated from, as attributes:
attr(, 'subjectpars'), each subject's parameters on their natural
scale with its id and groups; attr(, 'grouppars'), for a model with
grouping levels, each group's parameters at each level – its own effects
and those above it, as a subject of average deviation within it would have;
and attr(, 'states'), the latent
states at every row by id and time (not with intoverstates=TRUE,
which draws observations from the filter rather than states).
#a 2 process model, each process measured by a noisy indicator, with stable #individual differences in process levels generatingModel<-ctModel(Tpoints=8,n.latent=2,n.TDpred=0,n.TIpred=0,n.manifest=2, MANIFESTVAR=diag(.1,2), LAMBDA=diag(1,2), DRIFT=matrix(c(-.2,-.05,-.1,-.1),nrow=2), DIFFUSION=matrix(c(1,.2,0,4),2), CINT=matrix(c('cint1, indvarying=TRUE, sdscale=0.05', 'cint2, indvarying=TRUE, sdscale=0.08'),nrow=2), MANIFESTMEANS=matrix(0,ncol=1,nrow=2), T0MEANS=matrix(0,ncol=1,nrow=2), T0VAR=diag(1,2)) #the R generator: every subject at the values the model states, with free #cells filled by fixed defaults and no individual differences data <- ctGenerate(generatingModel,n=15,burnin=10) #the julia engine: each subject's CINT drawn from the population, with the #means stated on the natural scale and the sds given by sdscale on the raw #scale (CINT is 10 * raw, so 0.5 and 0.8) if (isTRUE(ctJuliaStatus()$available)) { data <- ctGenerate(generatingModel,n=15,burnin=10, backend='julia',values=c(cint1=1,cint2=0)) } #the prior predictive: what the model says data could look like before it #has seen any. No fit, and no data. priorpred <- ctGenerate(generatingModel, fromPriors = TRUE, Tpoints = 6, n = 10, nsamples = 20, cores = 2) str(priorpred$Y) # backend = 'auto' generates with julia where it is available. Julia keeps # running between calls, holding what it has compiled; this ends it. if (isTRUE(ctJuliaStatus()$available)) ctJuliaKill("session")#a 2 process model, each process measured by a noisy indicator, with stable #individual differences in process levels generatingModel<-ctModel(Tpoints=8,n.latent=2,n.TDpred=0,n.TIpred=0,n.manifest=2, MANIFESTVAR=diag(.1,2), LAMBDA=diag(1,2), DRIFT=matrix(c(-.2,-.05,-.1,-.1),nrow=2), DIFFUSION=matrix(c(1,.2,0,4),2), CINT=matrix(c('cint1, indvarying=TRUE, sdscale=0.05', 'cint2, indvarying=TRUE, sdscale=0.08'),nrow=2), MANIFESTMEANS=matrix(0,ncol=1,nrow=2), T0MEANS=matrix(0,ncol=1,nrow=2), T0VAR=diag(1,2)) #the R generator: every subject at the values the model states, with free #cells filled by fixed defaults and no individual differences data <- ctGenerate(generatingModel,n=15,burnin=10) #the julia engine: each subject's CINT drawn from the population, with the #means stated on the natural scale and the sds given by sdscale on the raw #scale (CINT is 10 * raw, so 0.5 and 0.8) if (isTRUE(ctJuliaStatus()$available)) { data <- ctGenerate(generatingModel,n=15,burnin=10, backend='julia',values=c(cint1=1,cint2=0)) } #the prior predictive: what the model says data could look like before it #has seen any. No fit, and no data. priorpred <- ctGenerate(generatingModel, fromPriors = TRUE, Tpoints = 6, n = 10, nsamples = 20, cores = 2) str(priorpred$Y) # backend = 'auto' generates with julia where it is available. Julia keeps # running between calls, holding what it has compiled; this ends it. if (isTRUE(ctJuliaStatus()$available)) ctJuliaKill("session")
Adds a $generated object to a fit, holding random data generated
from the fit's posterior (or its estimate), on either backend.
ctStanGenerateFromFit is maintained as a backward-compatible alias.
ctGenerateFromFit( fit, nsamples = 200, fullposterior = FALSE, verboseErrors = FALSE, cores = getOption("mc.cores", 2L), intoverstates = "fit", effects = c("population", "fitted") )ctGenerateFromFit( fit, nsamples = 200, fullposterior = FALSE, verboseErrors = FALSE, cores = getOption("mc.cores", 2L), intoverstates = "fit", effects = c("population", "fitted") )
fit |
A fit from |
nsamples |
Positive integer specifying number of datasets to generate. |
fullposterior |
Logical indicating whether to sample from the full posterior (original nsamples) or the posterior mean. |
verboseErrors |
if TRUE, print verbose output when errors in generation encountered. |
cores |
Number of cpu cores to use: R worker processes for a stan fit, and for a julia fit the engine threads its draws are generated on. |
intoverstates |
For
Either way the states are regenerated rather than reused. |
effects |
For |
The fit, with a $generated element: Y, the generated
data as an array of sample by data row by manifest variable, on the
original time and missingness structure, and llrow, each generated
row's log likelihood.
gen <- ctGenerateFromFit(ctstantestfit, nsamples=3,fullposterior=TRUE,cores=1) plot(gen$generated$Y[3,,2],type='l') #Third random data sample, 2nd manifest var, all time points.gen <- ctGenerateFromFit(ctstantestfit, nsamples=3,fullposterior=TRUE,cores=1) plot(gen$generated$Y[3,,2],type='l') #Third random data sample, 2nd manifest var, all time points.
Deprecated. Use ctGenerate(fromPriors = TRUE), which is the
same computation reached through the function that generates data from a
ctsem model generally. ctStanGenerate is a backward-compatible alias
and is deprecated with it.
ctGenerateFromPriors( cts, datastruct = NA, is = FALSE, fullposterior = TRUE, nsamples = 200, parsonly = FALSE, cores = getOption("mc.cores", 2L), backend = c("auto", "julia", "stan"), n.subjects = 20, Tpoints = NULL, dtmean = 1 )ctGenerateFromPriors( cts, datastruct = NA, is = FALSE, fullposterior = TRUE, nsamples = 200, parsonly = FALSE, cores = getOption("mc.cores", 2L), backend = c("auto", "julia", "stan"), n.subjects = 20, Tpoints = NULL, dtmean = 1 )
cts |
A model – |
datastruct |
Long format data structure as used by ctsem, giving the
subjects, times and missingness to generate for. Ignored when |
is |
Deprecated and ignored, with a warning if set. Importance sampling reweights draws taken from a gaussian approximation onto a target that cannot be sampled directly. Fitted to an empty dataset there is no such target: the likelihood contributes nothing and what remains is the prior, which ctsem holds as independent univariate densities in the raw space these draws are taken in. There is no approximation to correct. |
fullposterior |
Generate from the full prior, or from its mode (the raw origin)? |
nsamples |
How many samples to generate? |
parsonly |
If TRUE, only return samples of raw parameters, don't generate data. |
cores |
Number of cpu cores to use. |
backend |
Which engine turns a parameter draw into data: |
n.subjects, Tpoints, dtmean
|
The design to build when |
List containing Y, an array of nsamples by data rows by manifest
variables, and llrow, an array of nsamples by data rows log likelihoods.
With parsonly=TRUE, the prepared model carrying the prior draws
under $stanfit$rawposterior and $stanfit$transformedpars.
#generate and plot samples from prior predictive priorpred <- ctGenerateFromPriors(cts = ctstantestfit,cores=2,nsamples = 50) # backend = 'auto' generates with julia where it is available. Julia keeps # running between calls, holding what it has compiled; this ends it. if (isTRUE(ctJuliaStatus()$available)) ctJuliaKill("session")#generate and plot samples from prior predictive priorpred <- ctGenerateFromPriors(cts = ctstantestfit,cores=2,nsamples = 50) # backend = 'auto' generates with julia where it is available. Julia keeps # running between calls, holding what it has compiled; this ends it. if (isTRUE(ctJuliaStatus()$available)) ctJuliaKill("session")
Reports the directions of the parameter space this data carries no information about, and the parameters involved in them, without fitting the model.
ctIdentify( datalong, model, inits = NULL, nstart = 3L, spread = 0.5, priors = FALSE, cores = getOption("mc.cores", 2L), verbose = 0L, rtol = 1e-13, seed = 1L, ... )ctIdentify( datalong, model, inits = NULL, nstart = 3L, spread = 0.5, priors = FALSE, cores = getOption("mc.cores", 2L), verbose = 0L, rtol = 1e-13, seed = 1L, ... )
datalong |
Long format data, as for |
model |
Model from |
inits |
Optional evaluation point. Supplying one uses that single point and disables the comparison across points described in Details. |
nstart |
Number of points to evaluate. The first is the origin of the unconstrained space; the rest are dispersed draws around it. |
spread |
Standard deviation of the additional evaluation points on the
raw scale. Large enough that they are genuinely different points, which
|
priors |
Whether to include the prior. |
cores |
As for |
verbose |
As for |
rtol |
A direction counts as uninformed when the information along it falls below this fraction of the best-informed direction's. The default separates zero from merely small: on the scaled information a structurally unidentified direction sits at machine precision (measured: 7e-17), while a weakly identified one on a comparable model sat at 3e-12. Raise it to catch weak directions too, at the cost of reporting some a fit would have settled. |
seed |
Seed for the dispersed evaluation points, so that two identical
calls give identical output. The session's random stream is restored
afterwards, so this neither depends on it nor disturbs it. |
... |
Further arguments to |
This catches structural non-identification – parameters that only appear together, a latent process nothing measures, a scale that is never fixed – which is a property of the model and holds wherever it is evaluated. It is not a verdict on the model: a direction that is flat at one point may be perfectly informed at the estimate, which is why several points are used and only parameters lying in the flat subspace at every one of them are reported as uninformed.
The model is prepared by ctFit, with any of its arguments
given in ..., so what is checked is the model a fit with the same
arguments would estimate.
Not every flat direction means the same thing. A random effect on a variance
cell (DIFFUSION, MANIFESTVAR) under intoverpop = 'augmented' with a
full-rank population covariance (poprank = NA; the route's default
rank leaves out the directions its filter cannot see) is
partially identified: the population covariances it generates with the other
individually varying parameters are determined, while their decomposition
into a scale and correlations is a ridge. The printed summary distinguishes
the two cases, because the advice differs – fixing a value discards
information in the first case and is the fix in the second.
Requires the Julia backend, which is where the per-subject scores come from. The statistic has rank at most the number of subjects, so a model with more free parameters than subjects reports flat directions for that reason alone; the printed summary says when that applies.
How well the data determine each random effect's population standard
deviation, and how much of each subject's effect its own data determine
– the check ctFit makes at its estimate, stored as
fit$identifiability$effects – is taken here only at a supplied
inits, such as a fit's estimate$raw. It depends on the
population standard deviations, which only a fit estimates: at the
default evaluation points it describes the starting values rather than
the data, and there it read a well identified random drift as less
informed than one the data barely inform.
An object of class ctIdentify with $parameters (names
in the flat subspace at every evaluation point), $partial and
$structural splitting those into the partially and completely
unidentified, $sometimes (in a flat direction beyond that subspace
at some points), $rotating (whether the basis of the flat subspace
turns between points), $nweak, $condition,
$starts holding the per-point detail, and $effects, the
random-effect information at a supplied inits (NULL
otherwise). Printing it summarises what was found.
ctFit runs the equivalent check on a finished fit,
evaluated at the estimate and stored as fit$identifiability.
# ctIdentify() always uses the julia backend internally, whatever the # eventual fit will use, so the example is inert where Julia is absent -- # including on CRAN, whose check machines have none. if (isTRUE(ctJuliaStatus()$available)) { gen <- suppressMessages(ctModel(type = 'ct', n.latent = 1, n.manifest = 1, manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1), DRIFT = matrix(-0.4), DIFFUSION = matrix(0.6), MANIFESTVAR = matrix(0.3), T0VAR = matrix(1), T0MEANS = matrix(0), CINT = matrix(0), MANIFESTMEANS = matrix(0), Tpoints = 8)) datalong <- ctGenerate(gen, n = 40, Tpoints = 8, backend = 'r') model <- suppressMessages(ctModel(type = 'ct', n.latent = 1, n.manifest = 1, manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1), T0MEANS = matrix(0), CINT = matrix(0), MANIFESTMEANS = matrix(0))) result <- ctIdentify(datalong, model, cores = 1) print(result) # Julia keeps running between calls, holding what it has compiled; # this ends it. ctJuliaKill("session") }# ctIdentify() always uses the julia backend internally, whatever the # eventual fit will use, so the example is inert where Julia is absent -- # including on CRAN, whose check machines have none. if (isTRUE(ctJuliaStatus()$available)) { gen <- suppressMessages(ctModel(type = 'ct', n.latent = 1, n.manifest = 1, manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1), DRIFT = matrix(-0.4), DIFFUSION = matrix(0.6), MANIFESTVAR = matrix(0.3), T0VAR = matrix(1), T0MEANS = matrix(0), CINT = matrix(0), MANIFESTMEANS = matrix(0), Tpoints = 8)) datalong <- ctGenerate(gen, n = 40, Tpoints = 8, backend = 'r') model <- suppressMessages(ctModel(type = 'ct', n.latent = 1, n.manifest = 1, manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1), T0MEANS = matrix(0), CINT = matrix(0), MANIFESTMEANS = matrix(0))) result <- ctIdentify(datalong, model, cores = 1) print(result) # Julia keeps running between calls, holding what it has compiled; # this ends it. ctJuliaKill("session") }
Converts absolute times to intervals for wide format ctsem panel data
ctIntervalise( datawide, Tpoints, n.manifest, n.TDpred = 0, n.TIpred = 0, imputedefs = F, manifestNames = "auto", TDpredNames = "auto", TIpredNames = "auto", digits = 5, mininterval = 0.001, individualRelativeTime = TRUE, startoffset = 0 )ctIntervalise( datawide, Tpoints, n.manifest, n.TDpred = 0, n.TIpred = 0, imputedefs = F, manifestNames = "auto", TDpredNames = "auto", TIpredNames = "auto", digits = 5, mininterval = 0.001, individualRelativeTime = TRUE, startoffset = 0 )
datawide |
Wide format data, containing absolute time measurements,
to convert to interval time scale.
See |
Tpoints |
Maximum number of discrete time points (waves of data, or measurement occasions) for an individual in the input data structure. |
n.manifest |
number of manifest variables per time point in the data. |
n.TDpred |
number of time dependent predictors in the data structure. |
n.TIpred |
number of time independent predictors in the data structure. |
imputedefs |
if TRUE, impute time intervals based on the measurement occasion (i.e. column) they are in, if FALSE (default), set related observations to NA. FALSE is recommended unless you are certain that the imputed value (mean of the relevant time column) is appropriate. Noise and bias in estimates will result if wrongly set to TRUE. |
manifestNames |
vector of character strings giving variable names of manifest indicator variables (without _Tx suffix for measurement occasion). |
TDpredNames |
vector of character strings giving variable names of time dependent predictor variables (without _Tx suffix for measurement occasion). |
TIpredNames |
vector of character strings giving variable names of time independent predictor variables. |
digits |
How many digits to round to for interval calculations. |
mininterval |
set to lower than any possible observed measurement interval, but above 0 - this is used for filling NA values where necessary and has no impact on estimates when set in the correct range. (If all observed intervals are greater than 1, mininterval=1 may be a good choice) |
individualRelativeTime |
if TRUE (default), the first measurement for each individual is assumed to be taken at time 0, and all other times are adjusted accordingly. If FALSE, new columns for an initial wave are created, consisting only of observations which occurred at the earliest observation time of the entire sample. |
startoffset |
if 0 (default) uses earliest observation as start time. If greater than 0, all first observations are NA, with distance of startoffset to first recorded observation. |
Time column must be numeric! Columns are read in ctsem's wide
order: the manifest variables and then the time dependent predictors, each
by occasion as ctWideNames lists them, then the absolute
times T0, T1, ..., then the time independent predictors, as
ctLongToWide returns them. Columns with exactly those names
in another order are put in that order by name, with a message; columns
named otherwise are read by position. An occasion with no time has its
observations set to NA unless imputedefs = TRUE.
The wide format data as a matrix, with the absolute time columns
replaced by the intervals preceding each time point, dT1,
dT2, ..., and columns named as ctWideNames names
them.
wideexample <- ctLongToWide(datalong = ctstantestdat, id = "id", time = "time", manifestNames = c("Y1", "Y2"), TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3")) #Then convert the absolute times to intervals, using the Tpoints reported from the prior step. wide <- ctIntervalise(datawide = wideexample, Tpoints = 10, n.manifest = 2, n.TDpred = 1, n.TIpred = 3, manifestNames = c("Y1", "Y2"), TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3") ) print(wide)wideexample <- ctLongToWide(datalong = ctstantestdat, id = "id", time = "time", manifestNames = c("Y1", "Y2"), TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3")) #Then convert the absolute times to intervals, using the Tpoints reported from the prior step. wide <- ctIntervalise(datawide = wideexample, Tpoints = 10, n.manifest = 2, n.TDpred = 1, n.TIpred = 3, manifestNames = c("Y1", "Y2"), TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3") ) print(wide)
Evaluates the objective the julia backend optimises – the log likelihood,
with the random effects integrated as the model's intoverpop route
integrates them – at a vector of raw (unconstrained) parameters, and its
gradient. For checking a fit, profiling by hand, or handing ctsem's
likelihood to an optimiser or sampler of your own.
ctJuliaEvaluate( object, pars = NULL, gradient = TRUE, contributions = FALSE, gradient_method = c("adjoint", "forward"), cores = getOption("mc.cores", 2L) )ctJuliaEvaluate( object, pars = NULL, gradient = TRUE, contributions = FALSE, gradient_method = c("adjoint", "forward"), cores = getOption("mc.cores", 2L) )
object |
A fit from |
pars |
Raw (unconstrained) parameters, in the order of
|
gradient |
Also return the gradient with respect to |
contributions |
Return subject and row contribution details when available. |
gradient_method |
Either "adjoint" (reverse mode, the default) or
"forward" (ForwardDiff). Both compute the same gradient. "adjoint" costs
the same regardless of how many free parameters a model has, and is
faster than "forward" at every model size measured, by a margin that
grows with the parameter count. See |
cores |
The most engine threads this evaluation uses, restored
afterwards. |
A list containing the log likelihood and, when requested, the gradient and the contributions.
# Needs a working julia backend, so the example is inert where Julia is # absent -- including on CRAN, whose check machines have none. if (isTRUE(ctJuliaStatus()$available)) { model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1)) model$pars$indvarying <- FALSE prepared <- ctFit(ctstantestdat, model, backend = 'julia', fit = FALSE) start <- ctJuliaEvaluate(prepared, pars = rep(0, 6), cores = 1) str(start) # Julia keeps running between calls, holding what it has compiled; # this ends it. ctJuliaKill("session") }# Needs a working julia backend, so the example is inert where Julia is # absent -- including on CRAN, whose check machines have none. if (isTRUE(ctJuliaStatus()$available)) { model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1)) model$pars$indvarying <- FALSE prepared <- ctFit(ctstantestdat, model, backend = 'julia', fit = FALSE) start <- ctJuliaEvaluate(prepared, pars = rep(0, 6), cores = 1) str(start) # Julia keeps running between calls, holding what it has compiled; # this ends it. ctJuliaKill("session") }
Installs everything ctFit(backend='julia') needs, in one call and
without restarting R: the JuliaConnectoR bridge package, a Julia
installation, and the Julia dependencies of the engine that ships inside
ctsem. Any step that is already satisfied is skipped, so on a machine that
already has Julia this only prepares the engine.
ctJuliaInstall( threads = NULL, version = NULL, agree = NULL, force = FALSE, quiet = FALSE )ctJuliaInstall( threads = NULL, version = NULL, agree = NULL, force = FALSE, quiet = FALSE )
threads |
Number of Julia threads for the engine's subject loop. Julia
fixes this when its process starts, so it applies to the session this call
creates; see |
version |
Julia version to install. Defaults to the version ctsem's vendored engine was resolved against. The download is checksum-verified only for that default. |
agree |
|
force |
Reinstall Julia even if one is already available, and restart any running Julia session. |
quiet |
Suppress download and installation progress output. |
Each step that installs something asks first, naming the download and its
destination. A non-interactive session has nobody to ask, so nothing is
installed there unless agree = TRUE or the environment variable
CTSEM_JULIA_AGREE=yes gave permission in advance.
Julia is downloaded only if none can be found. ctsem looks in
JULIA_BINDIR, then at a Julia it installed itself, then on the
PATH, then among juliaup's installations. A Julia that ctsem installs
goes under tools::R_user_dir("ctsem", "data") and later sessions find
it automatically – there is no environment variable to set, and no PATH or
shell profile is modified. To use a different Julia afterwards, set
JULIA_BINDIR.
A Julia-engine status list, invisibly; see ctJuliaStatus.
## Not run: ctJuliaInstall() # ask, then install whatever is missing ctJuliaInstall(threads = 4) # and start the Julia session with four threads ## End(Not run)## Not run: ctJuliaInstall() # ask, then install whatever is missing ctJuliaInstall(threads = 4) # and start the Julia session with four threads ## End(Not run)
Stops the Julia processes ctJuliaProcesses lists. By default
only those left by R sessions that have ended, which nothing can be using.
ctJuliaKill( which = c("orphaned", "session", "workers", "mine", "all"), pid = NULL, ask = interactive() )ctJuliaKill( which = c("orphaned", "session", "workers", "mine", "all"), pid = NULL, ask = interactive() )
which |
Which to stop: |
pid |
Process ids to stop instead of choosing by |
ask |
Whether to ask before stopping another R session's engine, as
|
The rows of ctJuliaProcesses() that were stopped,
invisibly.
ctJuliaKill() # engines left by R sessions that have ended ctJuliaKill("session") # this session's engine, e.g. stuck compilingctJuliaKill() # engines left by R sessions that have ended ctJuliaKill("session") # this session's engine, e.g. stuck compiling
Lists the Julia processes on this machine that ctsem started or that
belong to them: this R session's engine, the engines of its sampling
workers, those of other R sessions, and any left running by an R session
that has ended. A Julia process can outlive what it was started for –
Escape leaves one finishing an uninterruptible step in the background, and a
crashed R session leaves its engine running until the engine notices – so
this is where to look when a machine is busier than it should be.
ctJuliaKill stops them.
ctJuliaProcesses(interval = 0.5)ctJuliaProcesses(interval = 0.5)
interval |
Seconds over which to measure CPU use, which is what the
|
ctsem records each Julia session it starts, which is how its processes are told from any other Julia. A JuliaConnectoR session it has no record of – one started by other code, or by a ctsem too old to record it – is listed too, as is a Julia process started by one of these (precompiling the engine, typically). CPU, memory, thread and parent details need the ps package; without it, only which processes exist and whose they are.
A data frame, one row per process, with columns pid;
role ("this session", "worker of this session",
"another R session", "orphaned" for one whose R session has
ended, "JuliaConnectoR, unrecorded", or "child of <pid>");
busy and cpu_percent over interval;
cpu_seconds; memory_mb; julia_threads, the Julia
thread count ctsem started it with; os_threads; r_pid, the
R process it serves; started; and note, which says when
this session's engine is still finishing an interrupted call.
ctJuliaProcesses()ctJuliaProcesses()
Prepares the copy of ContinuousTimeSEM.jl that ships inside this ctsem
installation. The engine source itself is vendored in inst/julia/ –
no repository credentials and no network access are involved in getting
that – but its Julia package dependencies still have to be instantiated,
which reaches the network on a machine where they are not already cached.
That step is consent-gated exactly as in ctJuliaInstall: asked
for interactively, skipped with a message when the environment is already
instantiated, and declined by default in a non-interactive session unless
agree = TRUE or CTSEM_JULIA_AGREE=yes said otherwise in
advance.
ctJuliaSetup( project = NULL, julia_bin = NULL, threads = NULL, force = FALSE, agree = NULL )ctJuliaSetup( project = NULL, julia_bin = NULL, threads = NULL, force = FALSE, agree = NULL )
project |
Optional local ContinuousTimeSEM.jl checkout to use instead of
the copy that ships with ctsem. Rarely needed: the engine is part of ctsem
and is edited in place under |
julia_bin |
Optional Julia binary directory. |
threads |
Number of Julia threads, fixed for the life of the session.
The engine splits its subject loop across them, and |
force |
Reconfigure an existing Julia session. |
agree |
|
The first call downloads and precompiles those dependencies (roughly 120 MB) and compiles the engine, which takes several minutes; later calls in new sessions reuse both.
The engine is compiled in advance, by fitting a small model on each route,
and every model of a route runs that same compiled code: a model's sizes,
parameters and state-dependent cells are data to the engine rather than part
of its types. So the first fit of a new model in a session compiles little,
and takes seconds more than later fits rather than minutes. A model with a
categorical indicator, or a Laplace unit with more than four random effects,
compiles a little more the first time. ctFit(forcerecompile = TRUE)
compiles a heavily state-dependent model's expressions for faster gradients,
at a minute or more of compile.
Pressing Escape (or Ctrl-C) during a julia fit returns to the R prompt at
once, and keeps the Julia session and everything it has compiled. Julia
stops the interrupted work at its next iteration; work that has no
iterations, such as compiling for a new model shape, finishes in the
background. If it is still running when the next julia call starts, that
call says so and waits, and Escape during that wait stops Julia outright.
ctJuliaProcesses lists the Julia processes ctsem has running,
including any left by R sessions that have ended, and
ctJuliaKill stops them.
If JuliaConnectoR or Julia itself is missing, this offers to install it
rather than failing – the same thing ctJuliaInstall does, which
is the function to reach for when setting the backend up deliberately, or
from a script.
A Julia-engine status list, invisibly.
## Not run: ctJuliaSetup() # prepare the engine, asking first ctJuliaSetup(threads = 8, force = TRUE) # restart the session wider ## End(Not run)## Not run: ctJuliaSetup() # prepare the engine, asking first ctJuliaSetup(threads = 8, force = TRUE) # restart the session wider ## End(Not run)
Reports what backend='julia' would find, and installs nothing: it is
safe to call on a machine with no Julia and no JuliaConnectoR, where it
says so rather than erroring. ctJuliaInstall supplies whatever
it reports as missing.
ctJuliaStatus(project = NULL, julia_bin = NULL)ctJuliaStatus(project = NULL, julia_bin = NULL)
project |
Optional local ContinuousTimeSEM.jl checkout to use instead of
the copy that ships with ctsem. Rarely needed: the engine is part of ctsem
and is edited in place under |
julia_bin |
Optional Julia binary directory. |
A list describing the selected Julia engine: whether it is
available – a julia fit could run now without asking anything,
which needs a Julia that starts and the engine's Julia packages installed,
or consent to install them given in advance with
CTSEM_JULIA_AGREE=yes (checked by a short julia process,
once per session, never a session) – whether the connectoR bridge
package is installed at a version the backend runs on,
the julia_bin directory in use and the julia version there,
the engine version (a hash of the engine source shipped with this
ctsem, which is also what its cached project directory is keyed on), and
the number of threads in a running session (NA when none is
running: this never starts one, so it leaves the session's width to the
first call that needs it).
status <- ctJuliaStatus() status$availablestatus <- ctJuliaStatus() status$available
Get Kalman filter estimates from a ctStanFit object.
ctStanKalman is maintained as a backward-compatible alias for this
function. Its pre-3.11 partner ctKalman is an alias for
ctPredict, not for this function, and returns a long data
frame rather than a list of arrays.
ctKalmanArray( fit, nsamples = NA, pointest = TRUE, collapsefunc = NA, cores = getOption("mc.cores", 2L), subjects = "all", timestep = "asdata", maxtime = "asdata", standardisederrors = FALSE, subjectpars = TRUE, indvarstates = FALSE, removeObs = F, realid = TRUE, newdata = NULL, ... )ctKalmanArray( fit, nsamples = NA, pointest = TRUE, collapsefunc = NA, cores = getOption("mc.cores", 2L), subjects = "all", timestep = "asdata", maxtime = "asdata", standardisederrors = FALSE, subjectpars = TRUE, indvarstates = FALSE, removeObs = F, realid = TRUE, newdata = NULL, ... )
fit |
fit object from |
nsamples |
either NA (to extract all) or a positive integer from 1 to maximum samples in the fit. |
pointest |
If TRUE, uses the posterior mode as the single sample. |
collapsefunc |
function to apply over samples, such as |
cores |
Integer number of cpu cores to use. Only needed if savescores was set to FALSE when fitting.
|
subjects |
'all' (the default), or the subjects to compute for, as ids in the
data (see |
timestep |
Either a positive numeric value, 'asdata' to use the times in the dataset, or 'auto' to select a timestep automatically (resulting in some interpolation but not excessive computation). |
maxtime |
only relevant if timestep is not 'asdata'. Positive numeric denoting max time for computations. |
standardisederrors |
If TRUE, computes standardised errors for prior, upd, smooth conditions. |
subjectpars |
if TRUE, state estimates are not returned, instead, predictions of each subjects parameters are returned, for parameters that had random effects specified. |
indvarstates |
if TRUE, do not remove indvarying states from output |
removeObs |
Logical or integer. If TRUE, observations (but not covariates) are set to NA, so only expectations based on parameters and covariates are returned. If a positive integer N, every N observations are retained while others are set NA for computing model expectations – useful for observing prediction performance forward further in time than one observation. |
realid |
If TRUE (the default), subjects are read as the ids in the data, and integers that are not ids there are read as ctsem's numbering of the subjects, 1 to N, with a message. If FALSE, they are read as that numbering. |
newdata |
Optional long format data, laid out as the data the model
was fitted to, to compute for instead of the fit's own: new subjects, the
fit's subjects with other covariate values, or a record cut short. The
fit's estimates (or, with |
... |
additional arguments to collpsefunc. |
list containing Kalman filter elements, each element in array of iterations, data row, variables. llrow is the log likelihood for each row of data.
For a binary or ordinal manifest variable the returned manifest quantities are
the *linear predictor* , on the logit scale,
not a probability and not a category. Turn one into a probability with
plogis(nu) for binary, or into cumulative category probabilities with
plogis(tau_k - nu) for ordinal, taking the thresholds from
summary(fit)$popmeans. The manifest covariance for such a variable is
likewise the covariance of the linear predictor.
k=ctKalmanArray(ctstantestfit,subjectpars=TRUE,collapsefunc=mean)k=ctKalmanArray(ctstantestfit,subjectpars=TRUE,collapsefunc=mean)
Recomputes a backend='julia', intoverpop='laplace' fit's log
marginal likelihood by adaptive Gauss-Hermite quadrature over the same
random-effect integral the Laplace approximation approximates, and reports
how far the estimate would move if that integral were taken exactly.
ctLaplaceCheck( fit, nodes = 5L, correction = TRUE, step = 0.001, refine = FALSE, maxiter = 50L, cores = getOption("mc.cores", 2L), verbose = 0L )ctLaplaceCheck( fit, nodes = 5L, correction = TRUE, step = 0.001, refine = FALSE, maxiter = 50L, cores = getOption("mc.cores", 2L), verbose = 0L )
fit |
A |
nodes |
Quadrature nodes per random effect. Cost is
|
correction |
Compute the first-order correction to the estimate. Costs
|
step |
Finite-difference step for the gap gradient, on the raw scale. |
refine |
Continue the fit on the quadrature objective rather than
correcting linearly: the refined fit |
maxiter |
Iteration cap for the climb of |
cores |
Engine threads for the quadrature. |
verbose |
Integer; 1 or more prints progress. |
The quadrature is adaptive in the usual sense: it is centred at each subject's inner mode and scaled by the inverse of the inner curvature there, both of which the fit has already computed. The curvature's eigenvalues are clipped from below at one, the prior's own curvature, before scaling: where the likelihood has gone convex in the random effects an unclipped rule puts its nodes far outside the prior and misses the mass it is meant to measure (1.8 nats too high on average on near-singular units in simulation, 0.06 clipped). Where every eigenvalue is at least one nothing changes.
The gap is quadrature minus the fit's own Laplace term, the quantity the fit
maximised. With nodes = 1 that gap is zero wherever the curvature is
at least the prior's, which is what makes the comparison meaningful – the
two are integrating the same thing by different rules. On a unit with an
eigenvalue below one it is not zero even at one node: the one-point clipped
rule is the eigenwise-floored term, below the fit's.
The correction is a single Newton step, delta = solve(-hessian,
gradient of (quadrature - laplace)), evaluated at the estimate. Read it
against the standard errors: delta / se well below one says the
approximation is not what limits the answer, and near or above one says the
point estimate is approximation-limited and the interval will not cover
whatever width it has.
A fit can be refined, with its units' random effects integrated by
quadrature, when it is fitted
(ctFit(intoverpop = 'quadrature')) or after
(ctFitUncertainty(fit, intoverpop = 'quadrature')); by default it
only runs the random-effects check, which warns where integrating would
move an estimate. On a corrected fit this check is made at the corrected estimate,
which is not a Laplace optimum: the correction reported is then the
further Newton step on the quadrature objective from there, with the
Laplace gradient included, and at on the result is
'corrected'. A small further step says the correction got there; a
large one says it did not. On any other fit it is the first step.
Nested groupings are handled. A group's integral does not factor over its members, but it does factor conditionally – given the group effect the members are independent – so the rule recurses over the same block tree the fit already builds, at a cost linear in the number of groups at each level rather than exponential in the number of members.
A list with gap (quadrature minus Laplace log marginal at the
estimate), gap_per_subject, nodes, and, when
correction is TRUE, a parameters data frame with one
row per raw parameter giving estimate, delta,
corrected, se and delta_se. With refine it also
carries refined. at says whether the estimate checked was the
Laplace optimum ('laplace') or already quadrature-corrected
('corrected').
dropped_directions counts the directions of the information matrix
the correction does not step along, whose correction is reported as zero
rather than as the arbitrarily large step an unguarded solve would
produce: exactly the directions fit$identifiability names, those
whose curvature is negligible or not positive and those the likelihood was
measured flat along. On a model where the Laplace approximation is
already exact the gap gradient is floating-point noise, and dividing that
by a near-singular curvature is how a meaningless correction gets a
plausible-looking number.
ctFitUncertainty with method = 'sample'
and intoverpop = 'none' removes the approximation
instead of measuring it, by sampling the joint posterior; much slower,
and the right answer when delta_se says a linear correction is not
credible.
## Not run: data <- ctstantestdat model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1)) model$pars$indvarying <- model$pars$matrix %in% 'MANIFESTMEANS' fit <- ctFit(data, model, backend = 'julia', intoverpop = 'laplace') check <- ctLaplaceCheck(fit) check # gap, and the largest corrections in standard errors check$parameters # the full per-parameter table ## End(Not run)## Not run: data <- ctstantestdat model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1)) model$pars$indvarying <- model$pars$matrix %in% 'MANIFESTMEANS' fit <- ctFit(data, model, backend = 'julia', intoverpop = 'laplace') check <- ctLaplaceCheck(fit) check # gap, and the largest corrections in standard errors check$parameters # the full per-parameter table ## End(Not run)
ctLongToWide Restructures time series / panel data from long format to wide format for ctsem analysis
ctLongToWide( datalong, id, time, manifestNames, TDpredNames = NULL, TIpredNames = NULL )ctLongToWide( datalong, id, time, manifestNames, TDpredNames = NULL, TIpredNames = NULL )
datalong |
dataset in long format, including subject/id column, observation time (or change in observation time, with 0 for first observation) column, indicator (manifest / observed) variables, any time dependent predictors, and any time independent predictors. |
id |
character string giving column name of the subject/id column |
time |
character string giving column name of the time columnn |
manifestNames |
vector of character strings giving column names of manifest indicator variables |
TDpredNames |
vector of character strings giving column names of time dependent predictor variables |
TIpredNames |
vector of character strings giving column names of time independent predictor variables |
Time column must be numeric
A wide format matrix, one row per subject: each manifest variable
and time dependent predictor at each time point (Y1_T0,
Y1_T1, ...), the observation times T0, T1, ...,
then the time independent predictors.
wideexample <- ctLongToWide(datalong = ctstantestdat, id = "id", time = "time", manifestNames = c("Y1", "Y2"), TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3")) #Then convert the absolute times to intervals, using the Tpoints reported from the prior step. wide <- ctIntervalise(datawide = wideexample, Tpoints = 10, n.manifest = 2, n.TDpred = 1, n.TIpred = 3, manifestNames = c("Y1", "Y2"), TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3") ) print(wide)wideexample <- ctLongToWide(datalong = ctstantestdat, id = "id", time = "time", manifestNames = c("Y1", "Y2"), TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3")) #Then convert the absolute times to intervals, using the Tpoints reported from the prior step. wide <- ctIntervalise(datawide = wideexample, Tpoints = 10, n.manifest = 2, n.TDpred = 1, n.TIpred = 3, manifestNames = c("Y1", "Y2"), TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3") ) print(wide)
Estimates out-of-sample predictive performance, by refitting the model with each fold of the data held out in turn or, for a julia Laplace fit, by Pareto-smoothed importance sampling without refitting.
ctLOO( fit, folds = 10, cores = getOption("mc.cores", 2L), parallelFolds = FALSE, tol = NULL, subjectwise = ifelse(length(unique(.ctFitRowSubject(fit))) >= folds, TRUE, FALSE), keepfirstobs = FALSE, leaveOutN = NA, refit = TRUE, casewiseApproximation = FALSE, method = c("kfold", "psis"), nsamples = 1000 )ctLOO( fit, folds = 10, cores = getOption("mc.cores", 2L), parallelFolds = FALSE, tol = NULL, subjectwise = ifelse(length(unique(.ctFitRowSubject(fit))) >= folds, TRUE, FALSE), keepfirstobs = FALSE, leaveOutN = NA, refit = TRUE, casewiseApproximation = FALSE, method = c("kfold", "psis"), nsamples = 1000 )
fit |
A fit from |
folds |
Number of cross validation splits to use – 10 folds implies that the model is re-fit 10 times, each time to a data set with 1/10 of the observations randomly removed. |
cores |
Number of processor cores to use. |
parallelFolds |
compute folds in parallel or use cores to finish single folds faster. parallelFolds will use folds times as much memory. |
tol |
objective tolerance for each fold's refit. On a stan fit it
defaults to 1e-5. On a julia fit a fold by default stops once a step is
predicted to gain less than 1e-4 in log likelihood, which keeps each
held-out subject's error to about 0.014 times the square root of its
influence (see |
subjectwise |
drop random subjects instead of data rows? With
|
keepfirstobs |
do not drop first observation (more stable estimates) |
leaveOutN |
if a positive integer is given, the folds argument is ignored and instead the folds are calculated by leaving out every Nth row from the data when fitting. Leaving 2 out would result in 3 folds (starting at rows 1,2,3), each containing one third of the data. |
refit |
if FALSE, do not optimise parameters for the new data set, just compute the likelihoods etc from the original parameters |
casewiseApproximation |
|
method |
|
nsamples |
Number of importance sampling draws for |
Works for backend='julia' fits as well as stan ones. There,
a held-out row is withheld by setting its manifest observations to missing
and re-preparing, which is a row the filter propagates through without an
update and without a likelihood contribution – the same thing
standata$dokalmanrows does for the stan backend. For julia fits
parallelFolds = TRUE runs up to cores folds at once, each in a
background R session with a one-thread Julia, started and compiled for the
model before the first fold (it needs the future package; the
sessions persist for the next call, and ctJuliaKill('workers')
releases them). Without it the folds run one at a time on cores
threads. Folds are separate fits, so they gain more from processes than
from threads, except where each fold takes only seconds and starting the
sessions costs more than it saves. $processes on the result says
which ran.
Be aware that each fold is an independent re-optimisation, and neither backend's optimizer restarts from a flat direction. On a weakly identified model, or one where withholding a fold leaves a parameter poorly determined, two folds – or two backends – can converge to noticeably different raw parameters for a similar likelihood.
K fold on an intoverpop = 'laplace' fit. The random effects
are integrated out, so what is scored depends on the fold type. With row
folds, a held-out row is scored with each subject's random effects at their
mode given the fold's training rows, at the fold's parameters – the
held-out row never informs the mode it is scored at. With subject folds a
held-out subject's rows are not scored at all: the out-of-sample quantity is
the subject's Laplace marginal likelihood at the fold's parameters,
returned as outsampleLogLikSubject, and outsampleLogLikRow is
NA because the marginal does not decompose into rows. In a multilevel
model a subject's marginal is conditional on the rest of its outermost group,
as the difference of that group's marginal with and without the held-out
subjects; several subjects of one group held out together share that joint
value evenly. insampleLogLik is correspondingly the Laplace marginal at
the estimate for subject folds, and the sum of the rows at the full-data
modes for row folds.
casewiseApproximation. Each fold's refit is replaced by one
Newton step from the full estimate, est - cov %*% s, where
s is the score of the held-out data at the estimate: each unit the
fold empties (a subject, or under the Laplace layer an outermost group)
contributes its likelihood score, and a unit it leaves partly observed – a
row fold, or a subject inside a group – the difference between its score on
the full and on the training data, at one gradient of the training objective
per fold. The prior stays, as it does in a refit. This is the one-step
(infinitesimal jackknife) approximation to the refit; its error grows faster
than the step. A fold's influence t(s) %*% cov %*% s measures the step in
posterior standard deviations squared, and a number given as
casewiseApproximation refits exactly every fold whose influence is at
or above it, so the approximation is used only where the step is short.
foldstepped says which folds were stepped. On a julia fit a stepped
fold scores its held-out rows only, together with every other stepped fold
in one pass over the data, so its column of LogLikRowFolds is
NA elsewhere: the training rows at each step would cost a pass a
fold, which for leave-one-subject-out is the whole cost.
method = 'psis'. Pareto-smoothed importance sampling
(Vehtari, Gelman and Gabry, 2017), without refitting. It estimates the
Bayesian leave-one-out predictive density – a unit's likelihood averaged
over the posterior without it – where K fold scores each fold at its refit
estimate, so the two differ by more than Monte Carlo error, most for a unit
that moves the estimate. Two levels:
subjectwise = TRUE: leave one unit out, where a unit is the
outermost level of random effect – a subject in a one-level model, a
group in a multilevel one, since subjects sharing a group effect are not
independent given the population parameters. A unit's term is its
likelihood with its random effects integrated out: by the filter under
intoverpop = 'augmented', by the Laplace approximation otherwise.
On a sampled fit (optimize = FALSE, or
ctFitUncertainty(fit, 'sample')) the population parameters are
its posterior draws. On an optimized fit they are drawn from its normal
approximation (fit$estimate$cov, so the fit must carry one),
widened by a factor of 1.5 and importance corrected to the posterior. A
unit whose Pareto k is above 0.7 is estimated again with nsamples
more draws, centred where leaving it out moves the estimate – the step
casewiseApproximation takes – pooled with draws around the
estimate, and the estimate with the lower k is kept
(pointwise$refined). That needs the fit's objective to be the
posterior it describes, so it is not done for a fit sampled jointly over
parameters and random effects (intoverpop = 'none'), whose unit
terms are Laplace approximations to the sampled marginal. When the normal
approximation itself misses the posterior (proposal_k, printed,
above 0.7) every unit's estimate is unreliable, and a warning says to
sample the fit and use its draws instead. No warning is not proof the
approximation is good: k is estimated too low from few draws, so a value
approaching 0.7, or a posterior you expect to be skewed (an outlying
subject, a variance near zero, binary indicators), is worth rerunning
with more nsamples or on a sampled fit.
subjectwise = FALSE, on an optimized
intoverpop = 'laplace' fit only: leave one row out, conditional on the
estimated population parameters and integrating over each unit's random
effects, drawn from the Gaussian the Laplace approximation fits at their
mode, widened by a factor of 1.5. This is pointwise leave-one-out of the
filter's one-step-ahead terms – each row's density given the rows before
it – not exact time series
leave-one-out, and it does not propagate uncertainty in the population
parameters.
Draws whose inner mode solve fails are dropped and counted. A Pareto k above 0.7 flags a unit or row whose estimate is unreliable; K fold is then the better tool for it.
Both methods cross-validate the Laplace objective. A Laplace fit's
estimate can be refitted with its units integrated (intoverpop =
'quadrature', in ctFit or ctFitUncertainty).
The folds are
refits of the Laplace objective, scored by it, and are not corrected – each
would need its own Hessian and quadrature gradient – so the in-sample side,
the folds' starting point and the PSIS proposal all use the Laplace optimum,
fit$laplace$correction$laplace_estimate, rather than the corrected
estimate. scoring says so when the fit was corrected.
A fold's estimate is never certified. Whatever the route, a refit
fold is optimised with optimcontrol$certify = FALSE: certifying and
correcting a fold the way the full fit is would cost each fold its own
Hessian, which is the expense K-fold exists to avoid. scoring says so
for any refit (refit = TRUE, the default, and not
casewiseApproximation), on both routes.
For method = 'kfold', a list with foldrows,
foldpars, insampleLogLikRow, LogLikRowFolds,
outsampleLogLikRow, insampleLogLik, outsampleLogLik
and entropy and standard deviation summaries, and processes,
whether the folds ran in worker processes. When a julia fit's folds are
refit, foldoptim has a row per fold: method ('refit'
or 'step'), and for a refit the optimiser's iterations,
gradients, converged, gain (what it was predicted
still to gain, in log likelihood, when it stopped) and seconds; a
warning counts the refits that stopped without converging. With
casewiseApproximation, foldinfluence and
foldstepped give each fold's influence and whether it was stepped
rather than refit. scoring is added
whenever the folds were refit, naming what was scored and that they are
uncertified (see Details); an intoverpop = 'laplace' fit also adds
insampleLogLikSubject and outsampleLogLikSubject. For
method = 'psis', an object of class
ctLOOpsis: elpd_loo, se_elpd_loo, p_loo,
looic, pointwise (a data frame of elpd_loo,
lpd, p_loo and pareto_k per unit or row, and at
the unit level refined),
n_high_k, ndraws, ndropped, proposal_k (the
Pareto k of the correction from proposal to posterior: one value at the
unit level, NA for a sampled fit, one per unit at the row level),
level and note.
Vehtari, A., Gelman, A. and Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27, 1413–1432.
ctLOO(ctstantestfit)ctLOO(ctstantestfit)
This function is used to specify a continuous time structural equation model,
which can then be fit to data with function ctFit.
ctModel( LAMBDA, type = "ct", n.manifest = "auto", n.latent = "auto", Tpoints = NULL, manifestNames = "auto", manifesttype = rep(0, nrow(LAMBDA)), ncategories = NULL, censormin = NULL, censormax = NULL, latentNames = "auto", id = "id", time = "time", silent = FALSE, T0VAR = "auto", T0MEANS = "auto", MANIFESTMEANS = "auto", MANIFESTVAR = "diag", ASYMPTOTES = NA, RESPONSE = NA, DRIFT = "auto", CINT = 0, DIFFUSION = "auto", n.TDpred = "auto", TDpredNames = "auto", TDPREDEFFECT = "auto", TDPREDMEANS = "auto", TDPREDVAR = "auto", n.TIpred = "auto", TIpredNames = "auto", tipredDefault = TRUE, PARS = NULL, driftScaled = character() )ctModel( LAMBDA, type = "ct", n.manifest = "auto", n.latent = "auto", Tpoints = NULL, manifestNames = "auto", manifesttype = rep(0, nrow(LAMBDA)), ncategories = NULL, censormin = NULL, censormax = NULL, latentNames = "auto", id = "id", time = "time", silent = FALSE, T0VAR = "auto", T0MEANS = "auto", MANIFESTMEANS = "auto", MANIFESTVAR = "diag", ASYMPTOTES = NA, RESPONSE = NA, DRIFT = "auto", CINT = 0, DIFFUSION = "auto", n.TDpred = "auto", TDpredNames = "auto", TDPREDEFFECT = "auto", TDPREDMEANS = "auto", TDPREDVAR = "auto", n.TIpred = "auto", TIpredNames = "auto", tipredDefault = TRUE, PARS = NULL, driftScaled = character() )
LAMBDA |
n.manifest*n.latent loading matrix relating latent to manifest variables, with latent processes 1:n.latent along the columns, and manifest variables 1:n.manifest in the rows. |
type |
character string. Use 'ct' (continuous time) or 'dt'
(discrete time) for Stan-based fitting with |
n.manifest |
Number of manifest indicators per individual at each measurement occasion / time point. Manifest variables are included as the first element of the wide data matrix, with all the 1:n.manifest manifest variables at time 1 followed by those of time 2, and so on. |
n.latent |
Number of latent processes. |
Tpoints |
Number of time points, or measurement occasions, in the data. This will generally be the maximum
number of time points for a single individual, but may be one extra if sample relative time intervals are used,
see |
manifestNames |
n.manifest length vector of manifest variable names as they appear in the data structure, without any _Tx time point suffix that may be present in wide data. Defaults to Y1, Y2, etc. |
manifesttype |
n.manifest length vector of manifest variable types. 0 (the
default) is a continuous, Gaussian variable; 1 is binary, coded 0/1; 2 is
ordinal, coded as consecutive integers from 1; 3 is a count, coded as
non-negative integers and modelled as Poisson with a log link, so the latent
process gives the log rate. A count keeps a free |
ncategories |
n.manifest length integer vector, giving the number of
categories of each ordinal manifest variable and ignored for the others.
Required when any |
censormin, censormax
|
n.manifest length numeric vectors giving the lower
and upper censoring limits of each |
latentNames |
n.latent length vector of latent variable names (used for naming parameters, defaults to eta1, eta2, etc). |
id |
character string denoting column name containing subject identification variables.
id data may be of any form, though will be coerced internally to an integer sequence rising from 1.
May instead be a character vector naming a nested hierarchy, innermost first:
|
time |
character string denoting column name containing timing data. Timing data must be numeric. |
silent |
Suppress all output to console. |
T0VAR |
lower triangular n.latent*n.latent cholesky matrix of latent process initial variance / covariance. "auto" freely estimates all parameters. |
T0MEANS |
n.latent*1 matrix of latent process means at first time point, T0. "auto" freely estimates all parameters. |
MANIFESTMEANS |
n.manifest*1 matrix of manifest intercept parameters. "auto" frees all parameters. |
MANIFESTVAR |
lower triangular n.manifest*n.manifest cholesky matrix of variance / covariance between manifests at each measurement occasion (i.e. measurement error / residual). "auto" freely estimates variance parameters, and fixes covariances between manifests to 0. "free" frees all values, including covariances. |
ASYMPTOTES |
Matrix with one row per binary indicator, in manifest
order, and two columns: the lower and upper asymptotes of its response
probability, which is then lower + (upper - lower) times the logistic of
the linear predictor. |
RESPONSE |
Matrix with one row per manifest variable and two columns,
the response scale of a censored indicator (manifesttype 4); rows for other
indicators must stay at their defaults. Column 1, |
DRIFT |
n.latent*n.latent DRIFT matrix of continuous auto and cross effects, relating the processes over time. "auto" freely estimates all parameters. |
CINT |
n.latent * 1 matrix of latent process intercepts, allowing for non 0 asymptotic levels of the latent processes. Generally only necessary for additional trends and more complex dynamics. |
DIFFUSION |
lower triangular n.latent*n.latent cholesky matrix of diffusion process variance and covariance (latent error / dynamic innovation). "auto" freely estimates all parameters. |
n.TDpred |
Number of time dependent predictor variables in the dataset. |
TDpredNames |
n.TDpred length vector of time dependent predictor variable names, as they appear in the data structure, without any _Tx time point suffix that may appear in wide data. Default names are TD1, TD2, etc. |
TDPREDEFFECT |
n.latent*n.TDpred matrix of effects from time dependent predictors to latent processes. Effects from 1:n.TDpred columns TDpredictors go to 1:n.latent rows of latent processes. "auto" freely estimates all parameters. |
TDPREDMEANS |
Legacy argument for time-dependent predictor means, mainly to support data generation workflows. Expected dimensions are (n.TDpred * Tpoints) rows and 1 column. "auto" creates a free-label matrix. |
TDPREDVAR |
Legacy argument for time-dependent predictor covariance structure, mainly to support data generation workflows. Expected dimensions are (n.TDpred * Tpoints) square. "auto" creates a free-label matrix. |
n.TIpred |
Number of time independent predictors. Each TIpredictor is inserted at the right of the data matrix, after the time intervals. |
TIpredNames |
n.TIpred length vector of time independent predictor variable names,
as they appear in the data structure. Default names are TI1, TI2, etc.
Or a formula whose terms are such names and smooth terms,
|
tipredDefault |
Logical. TRUE sets any parameters with unspecified time independent
predictor effects to have effects estimated, FALSE fixes the effect to zero unless individually specified.
A cell's effects count as unspecified only when its |
PARS |
for types 'ct' and 'dt' only. May be of any structure, only needed to contain extra parameters for certain non-linear models. |
driftScaled |
for type 'ct' only: which of "DIFFUSION" and "CINT" to
parameterise by what they accrue over one time unit given the process's
own drift diagonal, rather than by their instantaneous rates. A diffusion
sd |
A matrix cell is one of four things, and which one is decided by what its text looks like:
a number, which fixes the cell;
a parameter name, which frees it – the same name in two cells is one parameter, which is how an equality constraint is written;
a name with fields, adding the transform, the levels of
id at which the parameter varies, the prior scaling of that
variation, and the time-independent predictors acting on the parameter;
an expression, computed from the latent states, the time
dependent predictors and the PARS cells.
Fields may be given in either of two interchangeable ways: in a fixed order
separated by | ('mm||TRUE|0.5'), or by name
('mm, indvarying=TRUE, sdscale=0.5'). With nested ids,
indvarying may name levels instead, as in 'mm||study' or
'mm, indvarying=c(subject, study)'. A cell containing | is
read the ordered way; otherwise a cell with a field of the form
name=value is read the named way, and an unrecognised field name is
an error rather than part of a parameter name.
ctPar writes the ordered form from the same named
arguments, for when a specification is built programmatically.
Every name in an expression has to be declared: a latent state, a time
dependent predictor, or a parameter given a PARS cell. Writing a
fresh name inside an expression does not declare it, so
DRIFT = matrix('-exp(dr)*eta1') needs
PARS = c('dr') alongside it. To transform one parameter and nothing
else the second field is shorter –
DRIFT = matrix('dr|-exp(param)'). A cell may also read a time
independent predictor by name, the subject's value of it – a study's known
response rounding as MANIFESTVAR = 'rsd + mvar', say, with
PARS = c('mvar|1e-10 + 5 * log1p_exp(2 * param)'); this is julia
only. A subject's missing value of such a predictor is imputed or sampled
as any other (optimcontrol$tipredMissing), and the cell reads that
value.
Population summaries evaluate the cell at a predictor value of zero, where
they report predictor effects. A
cell cannot reference a manifest variable or the time interval at all.
Predictor effects on a parameter are set with the tipreds
field.
Not every simple name in a cell is a parameter name. A latent state or
time dependent predictor name is a reference to it, and any number
of matrices may reference the same one. A name declared in a PARS
cell is likewise a reference wherever else it is written: the
PARS cell holds the parameter and carries its transform, and the
cells that name it are rewritten into references to that cell.
Repeating a name that is not one of those is the equality
constraint, and those cells are one parameter – one value, one random
effect – so one transform has to apply to all of them. Cells that would
transform it differently are refused, and that includes cells whose
defaults differ: the DRIFT diagonal is bounded negative
while its off-diagonals are not, so DRIFT = matrix('a', 2, 2) is
not a way to tie the whole matrix to one parameter. To share a parameter
across cells that would otherwise transform it differently, declare it in
one PARS cell (PARS = c('a')) and then apply whatever
transform each cell needs in an expression there –
DRIFT[1,1] = '-exp(a)' and DIFFUSION[1,1] = 'exp(a)'. One
parameter then serves cells that need different transforms, and each
transform is stated where it applies. Giving the PARS cell the
transform instead (PARS = c('a|-exp(param)')) and writing just the
name in each cell works when one transform suits them all. Two PARS
cells sharing one label are co-holders rather than references, so they must
agree on the transform as any other repeated name must.
A predictor effect belongs to the parameter rather than to the cell, so
cells that are one parameter must state the same tipreds. Naming an
effect ('mypar||||age=shared') constrains every parameter given that
name under that predictor to one coefficient, which costs one fewer free
parameter and makes the two parameters' reported effects identical. Because
one coefficient displaces each parameter's raw value equally, it means the
same thing on the natural scale only where the transforms agree, so sharing
a name between parameters transformed differently is refused. A name is
scoped per predictor: the same label under two predictors is two
coefficients. A number instead of a name ('mypar||||age=0.5') fixes
the effect: no coefficient is estimated, and each subject's raw value of the
parameter is displaced by 0.5 times their age, before the transform.
Fixed effects are fitted by backend = 'julia' and used by
ctGenerate; the stan backend refuses them.
A cell that is exactly a latent state or time dependent predictor name is
taken as a reference to it rather than as a parameter of that name, and
warns, because the two readings are different models. 'state[1]'
and 'tdpreds[rowi, 1]' say the same thing without the ambiguity. In
PARS it is an error rather than a warning: a bare name there reads
as the declaration of a parameter, so write the reference as part of an
expression ('log1p(exp(eta1))') if a state-dependent PARS
cell is what is wanted.
For type = 'ct' or 'dt', a model of class
ctStanModel: $pars holds one row per matrix cell, and
$matrices the same cells as matrices, editable either way. For
type = 'omx', a list of matrices of class ctsemInit, for
ctGenerate or ctModelConvertOMX.
ctPar for the fields a matrix cell can carry,
ctFit for fitting the result.
### Frequentist example: ### impulse and level change time dependent predictor ### example from Driver, Oud, Voelkle (2015) data('ctExample2') tdpredmodel <- ctModel(type='omx', n.manifest = 2, n.latent = 3, n.TDpred = 1, Tpoints = 8, manifestNames = c('LeisureTime', 'Happiness'), TDpredNames = 'MoneyInt', latentNames = c('LeisureTime', 'Happiness', 'MoneyIntLatent'), LAMBDA = matrix(c(1,0, 0,1, 0,0), ncol = 3)) tdpredmodel$DIFFUSION[, 3] <- 0 tdpredmodel$DIFFUSION[3, ] <- 0 tdpredmodel$T0VAR[3, ] <- 0 tdpredmodel$T0VAR[, 3] <- 0 tdpredmodel$CINT[3] <- 0 tdpredmodel$T0MEANS[3] <- 0 tdpredmodel$DRIFT[3, ] <- 0 ###Bayesian example: model<-ctModel(type='ct', n.latent=2, latentNames=c('eta1','eta2'), n.manifest=2, manifestNames=c('Y1','Y2'), n.TDpred=1, TDpredNames='TD1', n.TIpred=3, TIpredNames=c('TI1','TI2','TI3'), LAMBDA=diag(2))### Frequentist example: ### impulse and level change time dependent predictor ### example from Driver, Oud, Voelkle (2015) data('ctExample2') tdpredmodel <- ctModel(type='omx', n.manifest = 2, n.latent = 3, n.TDpred = 1, Tpoints = 8, manifestNames = c('LeisureTime', 'Happiness'), TDpredNames = 'MoneyInt', latentNames = c('LeisureTime', 'Happiness', 'MoneyIntLatent'), LAMBDA = matrix(c(1,0, 0,1, 0,0), ncol = 3)) tdpredmodel$DIFFUSION[, 3] <- 0 tdpredmodel$DIFFUSION[3, ] <- 0 tdpredmodel$T0VAR[3, ] <- 0 tdpredmodel$T0VAR[, 3] <- 0 tdpredmodel$CINT[3] <- 0 tdpredmodel$T0MEANS[3] <- 0 tdpredmodel$DRIFT[3, ] <- 0 ###Bayesian example: model<-ctModel(type='ct', n.latent=2, latentNames=c('eta1','eta2'), n.manifest=2, manifestNames=c('Y1','Y2'), n.TDpred=1, TDpredNames='TD1', n.TIpred=3, TIpredNames=c('TI1','TI2','TI3'), LAMBDA=diag(2))
Convert an old OpenMx-style ctsem model to the modern ctsem model format.
ctStanModel is maintained as a backward-compatible alias.
ctModelConvertOMX(ctmodelobj, type = "ct", tipredDefault = TRUE)ctModelConvertOMX(ctmodelobj, type = "ct", tipredDefault = TRUE)
ctmodelobj |
ctsem model object created by |
type |
Either |
tipredDefault |
Logical. TRUE sets any parameters with unspecified time independent predictor effects to have effects estimated, FALSE fixes the effect to zero unless individually specified. |
ctStanModel is maintained as a backward-compatible alias.
Call this when you have a matrix-list model built with
ctModel(type='omx') and want to fit it: ctFit() takes a
built model, and an omx object is not one until it comes through
here. That conversion is the reason the function is exported.
New models need it only implicitly. ctModel(type='ct') and
ctModel(type='dt') call it for you and return a model that is
already built, so there is nothing left to convert.
List object of class ctStanModel in the modern ctsem model format,
with a pars data frame used by ctFit. Random effects are
specified for any intercept type parameters (T0MEANS, MANIFESTMEANS, and or
CINT), and time independent predictor effects are specified for all
parameters. Adjust these after initial specification by directly editing the
pars subobject, so model$pars, or by editing the pars-backed
matrix view, so model$matrices.
model <- ctModel(type='omx', Tpoints=50, n.latent=2, n.manifest=1, manifestNames='sunspots', latentNames=c('ss_level', 'ss_velocity'), LAMBDA=matrix(c( 1, 'ma1' ), nrow=1, ncol=2), DRIFT=matrix(c(0, 1, 'a21', 'a22'), nrow=2, ncol=2, byrow=TRUE), MANIFESTMEANS=matrix(c('m1'), nrow=1, ncol=1), # MANIFESTVAR=matrix(0, nrow=1, ncol=1), CINT=matrix(c(0, 0), nrow=2, ncol=1), DIFFUSION=matrix(c( 0, 0, 0, "diffusion"), ncol=2, nrow=2, byrow=TRUE)) modernmodel=ctModelConvertOMX(model)model <- ctModel(type='omx', Tpoints=50, n.latent=2, n.manifest=1, manifestNames='sunspots', latentNames=c('ss_level', 'ss_velocity'), LAMBDA=matrix(c( 1, 'ma1' ), nrow=1, ncol=2), DRIFT=matrix(c(0, 1, 'a21', 'a22'), nrow=2, ncol=2, byrow=TRUE), MANIFESTMEANS=matrix(c('m1'), nrow=1, ncol=1), # MANIFESTVAR=matrix(0, nrow=1, ncol=1), CINT=matrix(c(0, 0), nrow=2, ncol=1), DIFFUSION=matrix(c( 0, 0, 0, "diffusion"), ncol=2, nrow=2, byrow=TRUE)) modernmodel=ctModelConvertOMX(model)
Each latent in indices gains a derivative latent (named d
and the latent's name), which drives the level with a fixed unit
coefficient. That makes each a damped oscillator: a second order process,
whose autocorrelation can dip below zero and rise again, which no first
order process can.
ctModelHigherOrder( ctm, indices, diffusion = TRUE, crosseffects = FALSE, cint = FALSE, explosive = FALSE )ctModelHigherOrder( ctm, indices, diffusion = TRUE, crosseffects = FALSE, cint = FALSE, explosive = FALSE )
ctm |
ctModel, |
indices |
Vector of integers, which latents to raise the order of. |
diffusion |
Move the process noise from the level to its derivative? |
crosseffects |
Move the effects of the other processes on this one to
its derivative (divided by |
cint |
Move the continuous intercept to the derivative (divided by
|
explosive |
For a |
A model from ctModel(type = 'ct') is rewritten cell by cell on its
parameter table, with a new inertia time for each latent
(inertia_ and the latent's name, in PARS):
where is the level's own drift and its diffusion, both
keeping their names and every setting – transform, individual variation,
predictor effects. The derivative's row of DRIFT reads
and , and its diffusion . As goes to
zero the derivative follows the level at once and the process is the
first order one it was, so the new model contains the old one with every
old parameter at its value; from on, the process
oscillates. The derivative starts where the process alone holds it at
equilibrium: mean zero, uncorrelated with every state, and variance
, so the inertia time is the one new parameter. With the
noise left on the level (diffusion = FALSE), or none, its initial
mean, variance and covariances are free.
A model in the older matrix-list form from ctModel(type = 'omx') is
written instead as the derivative pulled back by the level through an
estimated coefficient (negative unless explosive), with the level's
own autoregression as the derivative's damping. That is the same family of
processes, but its first order member is reached only as the damping, the
coefficient and the diffusion all grow without bound.
extended ctModel
om <- ctModel(LAMBDA=diag(1,2),DRIFT=0, MANIFESTMEANS=0,type='omx',Tpoints=4) om <- ctModelHigherOrder(om,1:2) print(om$DRIFT) print(om$pars)om <- ctModel(LAMBDA=diag(1,2),DRIFT=0, MANIFESTMEANS=0,type='omx',Tpoints=4) om <- ctModelHigherOrder(om,1:2) print(om$DRIFT) print(om$pars)
Starting from a fitted model, repeatedly: screens it with
ctScoreTest; refits it with the most strongly indicated
relaxations, one at a time, warm started from the current estimates; keeps
the best of them if the refit's likelihood confirms it; and stops when no
relaxation is indicated and the model is adequate by
ctFitAdequacy. When nothing is indicated but the model is not
adequate, a slow process driving a latent
(ctModelSlowComponent), a cycle such as a daily rhythm
(ctModelPeriodicComponent), a process of its own for an item
(ctModelUniqueProcess), raising the order of a latent process
(ctModelHigherOrder) and any models given in
candidates are tried the same way, each ranked by the adequacy
check's evidence for it: a slow process for a latent whose updates stay
correlated over time gaps (the latent family), a cycle at the period
the periodic family points to, a process of its own for an item
whose errors stay correlated with its own (the residuals family).
ctModelImprove( fit, covariates = NULL, relax = c("heterogeneity", "covariates", "statedependence", "volatility", "fixed", "level", "slow", "unique", "periodic", "order"), changes = NULL, candidates = NULL, alpha = 0.05, maxsteps = 10L, ntry = 3L, criterion = c("LR", "AIC", "BIC"), adequacy = TRUE, prune = TRUE, close = NULL, holdout = 0, form = "auto", periods = NULL, nsamples = 100L, projection = "auto", cores = getOption("mc.cores", 2L), checkpoint = NULL, verbose = TRUE )ctModelImprove( fit, covariates = NULL, relax = c("heterogeneity", "covariates", "statedependence", "volatility", "fixed", "level", "slow", "unique", "periodic", "order"), changes = NULL, candidates = NULL, alpha = 0.05, maxsteps = 10L, ntry = 3L, criterion = c("LR", "AIC", "BIC"), adequacy = TRUE, prune = TRUE, close = NULL, holdout = 0, form = "auto", periods = NULL, nsamples = 100L, projection = "auto", cores = getOption("mc.cores", 2L), checkpoint = NULL, verbose = TRUE )
fit |
A fit from |
covariates |
Optional subject-level covariates the model may add, as
for |
relax |
Which relaxations may be made. |
changes |
Optionally rows of |
candidates |
Optional list of alternative models (from
|
alpha |
Level for the screen (Benjamini-Hochberg within each family), for the likelihood ratio decision, and for adequacy. |
maxsteps |
Most refit rounds. |
ntry |
Proposals from the score tests refitted in a round, at most: one at a time, the strongest first, keeping the first that improves the fit. Proposals from the adequacy check are tried the same way until one improves or none is left, since the check does not change until the model does. Pruning and a held-out confirmation refit this many side by side. |
criterion |
|
adequacy |
Use |
prune |
After the search, try dropping each kept change in turn and drop it if the likelihood does not need it given the others: a change kept early can be made redundant by one kept later. |
close |
Optional root mean square residual correlation regarded as a close enough fit to stop at; see Details. |
holdout |
Share of subjects, between 0 and 0.5, held out of the search to confirm its changes on. |
form |
How covariates are screened, as for |
periods |
Periods for the adequacy check's |
nsamples |
Generated datasets for the adequacy checks. |
projection |
How the adequacy checks remove the estimation, as for
|
cores |
Cores for fitting and testing. With two or more, once a
refit has taken longer than five minutes, the refits of pruning and of a
held-out confirmation run side by side in worker processes, |
checkpoint |
Optional file the search saves its state to after every
round and after pruning. Called again with the same file, the same fit and
the same settings, it resumes after the last round saved; a file from a
different search is refused. A long search on a cluster otherwise keeps
nothing until it ends. With |
verbose |
Report progress: each round's steps, each refit's time, and while a refit runs, its iteration, objective and gradient every ten minutes. |
The relaxations, in the language of the model:
indvarying = TRUE for a parameter and the
T0MEANS parameters, whose covariances with it the test took in;
for a fixed MANIFESTMEANS cell, a parameter there varying about
the fixed value as its stated mean.
a covariate added as a time independent predictor, with effects on the parameters its tests single out.
a term d * eta_k * eta_m in a latent's
drift, written into CINT with d a PARS parameter:
a drift coefficient that varies with a state.
a latent's innovation sd scaled by exp(v *
eta_m): innovations whose size depends on a state.
a cell fixed at zero freed: a drift cross effect, a diffusion covariance, a cross loading.
a parameter of the dynamics or error variances that depends
on a subject's level (a random intercept) let vary over subjects, its
random effect correlated with the intercepts (ctScoreTest's
level terms).
a slow process driving a latent through its drift, its effect started at one and its persistence at a tenth of the latent's; where the model has a slow process for another latent, that process driving this one too is tried first.
a cycle of the period the adequacy check points to, driving every latent with innovations of its own, each subject's phase set from the clock at its first row and free to deviate; the phase predictors are added to the data.
a process of its own, persistent, loading on one item;
and, where the pairs family finds two items' errors correlated
within an occasion, a process the two share
(ctModelUniqueProcess(shared = TRUE)): an error is meant to be
random, and co-movement within an occasion that the latents' innovation
covariances do not explain is read as a process faster than the
occasions. Like the other structural changes it is tried once the
score tests indicate nothing, ranked against them by the check's
evidence, so that it does not absorb misfit that belongs to a cycle or
to one item's own process.
not by default: a measurement error covariance the score test flags is freed, for designs with a reason for correlated error, such as a method the items share, and no shared process is proposed.
a latent process made second order.
On a reduced-rank level (poprank) a heterogeneity result for a
parameter that already varies through the shared dimensions gives it a
variance of its own (popunique); without reduced rank, a
significant dimension test makes the parameters its direction
loads on vary together. Relaxations the score tests cannot tell apart (the same
class) are not refitted in the same round.
Each random effect the search adds brings its correlations with the others,
and with many effects for few subjects the data no longer determine them.
When a kept change leaves the fit's curvature weak or negative along the
population covariance (fit$identifiability), the covariance is
refitted at one rank lower (poprank), starting from its leading
dimensions, and the lower rank kept unless the likelihood ratio test needs
the dropped dimension; it steps down until the fit is identified. Later
random effects then load on the shared dimensions, and later heterogeneity
results give own variances. Single-level models only.
When the search stops, each change kept is tried out again, by refitting
without it (prune), and dropped if the others make it redundant.
A relaxation is kept when its refit improves on the current model by
criterion: a likelihood ratio test at alpha divided by the
number of refits tried at that step ('LR'), or a lower AIC or BIC
(subjects as the sample size). The likelihood ratio test of a new random
effect against the chi-square is conservative, as its variance starts on a
boundary.
An object of class ctModelImprove: fit (the final
fit), history (every refit tried: the round, the relaxation, its
screening p-value, the change in log likelihood and parameters, the
criterion's verdict, and whether it was kept; a pruning row, family
'prune', says whether the change was dropped, and a 'rank' row
whether the lower rank was kept), screens (the score
tests of each round), adequacy (the final checks),
confirmation (with holdout: each kept change's likelihood
ratio test on the held-out subjects), and stopped (why the search
ended: 'adequate', no adequacy family detected misfit;
'close fit', those that did were shown within close;
'not adequate, no candidate improved' or 'not adequate,
nothing left to try', the checks still detect misfit; 'nothing
indicated', with adequacy = FALSE, the screen was clean;
'maxsteps'). A refit that failed, or ended below the model it
extends, is in history with a note saying so. With holdout,
the held-out subjects take no part in the search; each kept change is
tested once on them, at the end, at alpha. fit is the model with the confirmed
changes refitted to every subject, and selected the search's own
choice fitted to the subjects it searched on.
The screen and the refit read the same data, so a
refit confirms most of what the screen proposes, and the search's error
rate is close to alpha per decision. Measured on simulated data (one
latent, 80 subjects, 20 irregular rows, 40 searches each): with nothing
missing, alpha = 0.05 kept something in 5 per cent of searches and
alpha = 0.01 in none; an omitted covariate shifting a random
intercept by half its sd was added in 75 and 55 per cent; an unrelated
covariate never. criterion = 'BIC' kept something in 8 per cent,
so at that size it is not the stricter choice. A drift that varied over
subjects was never added as such at that size; a random diffusion or
measurement error variance stood in for it in 10 per cent, since what the
data showed was subjects differing in variance. A search makes a decision
per family it screens, so its error rate grows with the families: on two
processes with three items each (60 subjects, about 48 rows each), where
the screen also tests state dependence and volatility in each process,
alpha = 0.05 kept something in 4 of 20 searches with nothing
missing (a volatility term twice, state dependence once, a random effect
once), and alpha = 0.01 is the stricter choice. With an omitted
cross effect, a process of an item's own and a daily cycle in the same
data, the search kept the cross effect in 19 of 20 searches, the cycle in
16 and the item's process in all 20.
The result is a model chosen by the data it is fitted to, and its
standard errors and tests do not account for that choice (MacCallum,
Roznowski & Necowitz, 1992). holdout sets aside a share of the
subjects: the search runs on the rest, and each change it keeps is then
tested on the held-out subjects alone, by refitting the final model without
it there; confirmation says which held, and the final model, refitted
to every subject, keeps only those. Every step, and every refit tried and
rejected, is in history.
With many observations a model can be rejected by the checks for
departures that are small on these summaries. With close, the search
also stops when every check that fails is a residual or within-row
correlation family
whose size is shown below close (ctFitAdequacy).
Each refit is judged by criterion; with 'LR', the likelihood
ratio against a chi-square on the parameters the change adds, at
alpha divided by the refits the round tries. That reference is
exact for none of the changes the search makes: a variance added at zero
sits on its boundary, where the chi-square is conservative, and a process
or cycle added brings parameters that mean nothing when its coupling is
zero, where it can err either way. Screening, rounds and pruning add
further choices the level does not count. The search is a way of
developing a model, and its p-values describe the steps it took rather than
testing the result. A refit that ends below the model it extends has not
reached an optimum, and is recorded as such.
On the Laplace route the filter integrates a fit's intercept-type random
effects exactly where it can, and Laplace approximates only the others (the
two-step term); a change that makes the dynamics read the state takes that
away, and its refit is on the one-step term
(optimcontrol$laplace_onestep in ctFit). Beside random
dynamics the two approximate their integrals differently, so such a refit
is compared with the current model refitted one-step from its own
estimate: a fast comparison of two approximations. To compare integrals
instead, start the search from a fit with intoverpop =
'quadrature': every refit is then refined the same way.
vignette('score-tests').
A ctsem matrix cell may be specified as an expression referencing a latent
process or a time dependent predictor – for instance
DRIFT[1,1] = '-log1p(exp(param)) * eta2'. Such a cell has no single
value, only a value at a given state and set of predictor values, and the
package's summaries and plots must therefore name the point they evaluated
it at.
ctModelIsNonlinear(x)ctModelIsNonlinear(x)
x |
A |
References to an individually varying parameter's carrier state (the
intoverpop representation of a random effect) do not count: those are
parameters, not dynamics, and are reported exactly.
Logical.
ctModelIsNonlinear(ctstantestfit)ctModelIsNonlinear(ctstantestfit)
In the usual model a subject's level sits in random manifest intercepts
(MANIFESTMEANS), so the latent state is the subject's deviation from
it. That is the same model, for linear dynamics, as one whose latent
process carries the level itself through a random latent intercept
(CINT) – but not once the dynamics depend on the state: a term that
reads the state reads the deviation in the first, the level in the second.
Persistence that differs near the bottom of a scale, or innovations that
grow with how high a subject usually is, are dependence on the level, which
state dependence written into the first model cannot express and its tests
(ctScoreTest) cannot see; ctScoreTest(tests = 'level')
asks about it between subjects.
ctModelLatentLevel(ctm, latents = NULL)ctModelLatentLevel(ctm, latents = NULL)
ctm |
A model from |
latents |
Which latents, by name or position; by default every one. |
For each latent in latents with a marker – an indicator whose loading
on it is fixed at 1 and on every other latent at 0 – whose
MANIFESTMEANS intercept varies over subjects, the intercept's
variation moves into the latent: CINT becomes a free parameter,
cint_ and the latent's name, with the intercept's random effect and
predictor effects; the marker's intercept is fixed at 0; and the latent's
initial mean is its stationary one (T0MEANS 'stationary'), so
a subject starts at their own level. The other indicators' intercepts are
left as they are, fixed or varying. A latent without such a marker, or
whose CINT is already free, is left as it is, with a message.
The level then rides on the dynamics: CINT is the level times the
rate of return to it, so a level's random effect is no longer one an
indicator's filter can integrate on its own, and a fit on the Laplace route
carries it as a coordinate.
The model, with the latents whose level moved in attribute
moved.
ctScoreTest for the tests of level dependence.
m <- ctModel(type = 'ct', manifestNames = c('Y1', 'Y2'), latentNames = c('eta1', 'eta2'), LAMBDA = diag(2), CINT = matrix(0, 2), MANIFESTMEANS = matrix(c('mm1', 'mm2'), 2)) lm <- ctModelLatentLevel(m) attr(lm, 'moved') p <- lm$pars p[p$matrix %in% c('CINT', 'MANIFESTMEANS', 'T0MEANS'), c('matrix', 'row', 'param', 'value', 'indvarying')]m <- ctModel(type = 'ct', manifestNames = c('Y1', 'Y2'), latentNames = c('eta1', 'eta2'), LAMBDA = diag(2), CINT = matrix(0, 2), MANIFESTMEANS = matrix(c('mm1', 'mm2'), 2)) lm <- ctModelLatentLevel(m) attr(lm, 'moved') p <- lm$pars p[p$matrix %in% c('CINT', 'MANIFESTMEANS', 'T0MEANS'), c('matrix', 'row', 'param', 'value', 'indvarying')]
Generate and optionally compile latex equation of subject level ctsem model.
ctModelLatex( x, matrixnames = TRUE, digits = 3, linearise = inherits(x, "ctStanFit"), textsize = "normalsize", folder = tempdir(), filename = paste0("ctsemTex", as.numeric(Sys.time())), tex = TRUE, equationonly = FALSE, compile = TRUE, open = TRUE, includeNote = TRUE, minimal = FALSE, splitDynamics = TRUE, splitMeasurement = TRUE, savepng = FALSE )ctModelLatex( x, matrixnames = TRUE, digits = 3, linearise = inherits(x, "ctStanFit"), textsize = "normalsize", folder = tempdir(), filename = paste0("ctsemTex", as.numeric(Sys.time())), tex = TRUE, equationonly = FALSE, compile = TRUE, open = TRUE, includeNote = TRUE, minimal = FALSE, splitDynamics = TRUE, splitMeasurement = TRUE, savepng = FALSE )
x |
A model from |
matrixnames |
Logical. If TRUE, includes ctsem matrix names such as DRIFT and DIFFUSION under the matrices. |
digits |
Precision of decimals for numeric values. |
linearise |
Logical. Show the linearised normal approximation for subject parameters and
covariate effects, or the raw parameters? Only relevant where a subject
parameter distribution is shown, and not available for a |
textsize |
Standard latex text sizes – tiny scriptsize footnotesize small normalsize large Large LARGE huge Huge. Useful if output overflows page. |
folder |
Character string specifying folder to save to, defaults to temporary directory, use "./" for working directory. |
filename |
filename, without suffix, to output .tex and .pdf files too. |
tex |
Save .tex file? Otherwise latex is simply returned within R as a string. |
equationonly |
Logical. If TRUE, output is only the latex relevant to the equation, not a compileable document. |
compile |
Compile to .pdf? (Depends on |
open |
Open after compiling? (Depends on |
includeNote |
Include text describing matrix transformations and subject notation? triangular matrices (which results in a covariance or Cholesky matrix) is shown – the latter is a more direct representation of the model, while the former is often simpler to convey. |
minimal |
if TRUE, outputs reduced form version displaying matrix dimensions and equation structure only. |
splitDynamics |
Logical. If TRUE, split latent process dynamics across deterministic and random change lines. If FALSE, show the full dynamics equation on one line. |
splitMeasurement |
Logical. If TRUE, split measurement equations across observation and observation noise lines. If FALSE, show the full measurement equation on one line. |
savepng |
Logical. If TRUE, renders the equation as a png file instead of a pdf, viewing the png in RStudio viewer when available. |
character string of latex code. Side effects include saving a .tex, .pdf, and displaying the pdf.
ctmodel <- ctModel(type='ct', n.latent=2, n.manifest=1, manifestNames='sunspots', latentNames=c('ss_level', 'ss_velocity'), LAMBDA=matrix(c( 1, 'ma1' ), nrow=1, ncol=2), DRIFT=matrix(c(0, 1, 'a21', 'a22'), nrow=2, ncol=2, byrow=TRUE), MANIFESTMEANS=matrix(c('m1'), nrow=1, ncol=1), CINT=matrix(c(0, 0), nrow=2, ncol=1), DIFFUSION=matrix(c( 0, 0, 0, "diffusion"), ncol=2, nrow=2, byrow=TRUE)) l=ctModelLatex(ctmodel,compile=FALSE, open=FALSE) cat(l)ctmodel <- ctModel(type='ct', n.latent=2, n.manifest=1, manifestNames='sunspots', latentNames=c('ss_level', 'ss_velocity'), LAMBDA=matrix(c( 1, 'ma1' ), nrow=1, ncol=2), DRIFT=matrix(c(0, 1, 'a21', 'a22'), nrow=2, ncol=2, byrow=TRUE), MANIFESTMEANS=matrix(c('m1'), nrow=1, ncol=1), CINT=matrix(c(0, 0), nrow=2, ncol=1), DIFFUSION=matrix(c( 0, 0, 0, "diffusion"), ncol=2, nrow=2, byrow=TRUE)) l=ctModelLatex(ctmodel,compile=FALSE, open=FALSE) cat(l)
Access or replace the matrix representation of a ctStanModel while
keeping the pars data frame as the canonical model specification.
ctModelMatrices(x) ctModelMatrices(x) <- valuectModelMatrices(x) ctModelMatrices(x) <- value
x |
A |
value |
A named list of matrices, typically copied from
|
ctModelMatrices(x) reconstructs matrices from x$pars. The
replacement form updates matching matrix, row, and col
entries in x$pars. Numeric matrix entries become fixed values;
non-numeric entries become parameter labels. Metadata such as transforms and
individual variation settings is preserved when possible and reset to
fit-ready defaults when fixed/free status changes.
The same view is available as x$matrices. Direct replacements such as
x$matrices$DRIFT[1, 2] <- "cross" update x$pars; detached copies
must be assigned back with x$matrices <- mats. A placeholder
matrices element is stored in the object so the view is visible in
names(x) and printed objects, but the matrix values are always derived
from x$pars.
The view also holds the population spread of the individually varying
parameters: RAWPOPVAR for the subject level and
RAWPOPVAR_<idname> for each grouping level, one row and column per
parameter varying at that level, also reachable as x$RAWPOPVAR.
The diagonal holds standard deviations and the lower triangle correlation
coordinates, both for the raw, untransformed parameters, the scale a fit
reports them on. A number fixes a cell, on every route, and a label leaves
it free. ctCov writes a whole matrix from a covariance, which
is also how to state T0VAR, DIFFUSION or MANIFESTVAR
as the covariance it is meant to be.
ctModelMatrices() returns a named list of matrices. The
replacement form returns the updated ctStanModel.
Adds a pair of latents, <name>_cos and <name>_sin, that turn
at a fixed period with no noise of their own – DRIFT[cos, sin]
= -2 pi / period and DRIFT[sin, cos] = 2 pi / period – so they
trace the cosine and sine of the time of day (or week, or whatever the
period is). Each latent in indices is driven by both, through
estimated effects DRIFT[k, <name>_cos] and DRIFT[k,
<name>_sin], so it is pushed through the cycle with an amplitude and phase
the data decide, and responds with its own dynamics: a fast process follows
the cycle closely, a slow one smooths and delays it.
ctModelPeriodicComponent( ctm, period, indices = seq_len(ctm$n.latent), name = "cycle", individual = TRUE )ctModelPeriodicComponent( ctm, period, indices = seq_len(ctm$n.latent), name = "cycle", individual = TRUE )
ctm |
A model from |
period |
The cycle's period, in the data's time units: 24 for a daily rhythm with time in hours. |
indices |
Which latents the cycle drives, by position. |
name |
Prefix for the added latents, predictors and parameters. |
individual |
Let each subject's cycle deviate from the clock. |
The cycle is set to the clock at each subject's first observation, from
two time independent predictors, <name>_cos0 and <name>_sin0,
the cosine and sine of that time over the period, which
ctPeriodicPhases adds to the data. So the data's time must
count from an origin shared by every subject – hours since midnight of a
common day, say – for the cycle to mean the same clock time for all.
With individual = TRUE each subject's cycle may also be shifted from
the clock and differ in strength, by an initial deviation of the pair with
an estimated standard deviation, which the filter learns from each
subject's data: rhythms that differ between people, or a time origin that
is not shared, are then still modelled.
The effects start at zero, where they have a gradient: unlike a slow process, the cycle is known, so a refit from the model without it needs no particular start.
Needs the julia backend, which reads time independent predictors in the initial means.
The extended model.
ctPeriodicPhases for the predictors it reads;
ctFitAdequacy's periodic family, which finds cycles a
model lacks; ctModelImprove, which adds one where that
family points.
m <- ctModel(type = 'ct', manifestNames = c('Y1', 'Y2'), latentNames = c('eta1', 'eta2'), LAMBDA = diag(2)) m2 <- ctModelPeriodicComponent(m, period = 24) m2$latentNames m2$TIpredNames p <- m2$pars p[p$matrix %in% 'DRIFT' & (p$row > 2 | p$col > 2) & (!is.na(p$param) | p$value != 0), c('matrix', 'row', 'col', 'param', 'value')]m <- ctModel(type = 'ct', manifestNames = c('Y1', 'Y2'), latentNames = c('eta1', 'eta2'), LAMBDA = diag(2)) m2 <- ctModelPeriodicComponent(m, period = 24) m2$latentNames m2$TIpredNames p <- m2$pars p[p$matrix %in% 'DRIFT' & (p$row > 2 | p$col > 2) & (!is.na(p$param) | p$value != 0), c('matrix', 'row', 'col', 'param', 'value')]
The changes ctModelImprove would consider for a fit: the
relaxations its score tests indicate and, given an adequacy check, the
structural changes the check's families point to – a process of an item's
own or shared by two, a slow process, a cycle, a second order. Rows can be
handed to ctModelImprove(changes = ), which refits with them and
judges each by the search's criterion: a search made a step at a time, by
hand.
ctModelProposals( fit, score = NULL, adequacy = NULL, covariates = NULL, relax = c("heterogeneity", "covariates", "statedependence", "volatility", "fixed", "level", "slow", "unique", "periodic", "order"), alpha = 0.05, cores = getOption("mc.cores", 2L) )ctModelProposals( fit, score = NULL, adequacy = NULL, covariates = NULL, relax = c("heterogeneity", "covariates", "statedependence", "volatility", "fixed", "level", "slow", "unique", "periodic", "order"), alpha = 0.05, cores = getOption("mc.cores", 2L) )
fit |
A fit from |
score |
A |
adequacy |
Optionally a |
covariates, relax, alpha, cores
|
As for |
Unlike a search's rounds, every indicated change is listed, including
relaxations the tests cannot tell apart (one class), of which a
round refits only the strongest; and the structural changes are listed
beside the relaxations, where a round turns to them only once the screen
is clean.
A data frame of class ctModelProposals, one row per change,
strongest first within the relaxations and within the structural
changes: change (as a search's history names it),
family, p (the evidence ranked by), class, and
proposal, what ctModelImprove() needs to make the change.
vignette('score-tests').
Each latent in indices gains a slow latent, named slow_ and
the latent's name, that drives it through the drift: an estimated effect
DRIFT[k, slow_k] moves the level the latent returns to with the slow
process, so a process can wander over weeks while it fluctuates over hours.
Its equilibrium is DRIFT[k, slow_k] / |DRIFT[k, k]| times the slow
process, so a person with faster dynamics follows the slow process less
closely.
ctModelSlowComponent(ctm, indices = seq_len(ctm$n.latent))ctModelSlowComponent(ctm, indices = seq_len(ctm$n.latent))
ctm |
A model from |
indices |
Which latents get a slow process, by position. |
A slow process is slower than the latent it drives by construction: its
drift is the latent's times inv_logit(r), with r a
PARS parameter named drift_ and the slow process's name, so
its rate is a fraction of the latent's. Left free, the two rates can trade
places: with only the latent measured, a latent of drift a driven by
a slow process of drift b has the spectrum of one with the two
exchanged and the effect adjusted, so the "slow" process can come out the
fast one, and refits from other starting values did. The latent's drift
parameter is declared in PARS for this, keeping its name,
transform, individual variation and predictor effects, so where the
latent's rate varies over people the slow process's follows it. Where the
latent's drift is fixed the fraction is of that value; where it is an
expression the slow process's persistence is left free, as before.
A slow process has innovations of unit variance (the effect carries its
scale), correlated with those of the other slow processes, including ones
added earlier, and not with the fast ones.
It has no loadings and no intercept, and no individual variation or
predictor effects. Its initial mean and variance are its stationary ones
(T0MEANS and the T0VAR diagonal 'stationary', see
ctFit): zero and 1 / (2 |a|) for persistence
a. Its initial covariance with the latent it drives is free – that
latent follows it, so the two start related – and with the other states
zero.
The effect has to start away from zero. At zero neither the slow process nor
its effect has a gradient, so an optimiser started there stays. When
refitting from a model without the slow process, start the effect at one
(the slow process's sign is not identified, so either sign will do) and
the fraction small; ctModelImprove starts it at a tenth.
The extended model.
ctModelHigherOrder for the other structural extension
the search tries.
m <- ctModel(type = 'ct', manifestNames = c('Y1', 'Y2'), latentNames = c('eta1', 'eta2'), LAMBDA = diag(2)) m2 <- ctModelSlowComponent(m, 1:2) m2$latentNames p <- m2$pars p[p$matrix %in% c('DRIFT', 'DIFFUSION') & (p$row > 2 | p$col > 2) & (!is.na(p$param) | p$value != 0), c('matrix', 'row', 'col', 'param', 'value')]m <- ctModel(type = 'ct', manifestNames = c('Y1', 'Y2'), latentNames = c('eta1', 'eta2'), LAMBDA = diag(2)) m2 <- ctModelSlowComponent(m, 1:2) m2$latentNames p <- m2$pars p[p$matrix %in% c('DRIFT', 'DIFFUSION') & (p$row > 2 | p$col > 2) & (!is.na(p$param) | p$value != 0), c('matrix', 'row', 'col', 'param', 'value')]
Each manifest variable in items gains a latent, named u_ and
the variable's name, that loads on it alone with a fixed unit loading and
has its own persistence and innovations. The variable then carries, beside
what it shares through the model's other latents and its measurement error,
a part that is specific to it and carried over time: an item whose wording
picks up something besides the construct, and picks it up persistently.
ctModelUniqueProcess(ctm, items, shared = FALSE, name = NULL)ctModelUniqueProcess(ctm, items, shared = FALSE, name = NULL)
ctm |
A model from |
items |
Which manifest variables, by name or position. |
shared |
One process for all of |
name |
The new latents' names, in place of those: one for a shared process, one per item otherwise. For a second process on the same items, at another time scale. |
Without it, that part can only go to the measurement error, which is
independent over time, so the item's prediction errors stay correlated
with its own at nearby times – what the residuals family of
ctFitAdequacy shows for that item alone – or to the shared
latents, whose dynamics it then distorts. The specific process and the
measurement error are told apart by persistence: the one carries over
time, the other does not.
The process has no intercept (the item's own carries its level), ctsem's
default drift transform, so it is mean reverting, and innovations
independent of every other latent's. It starts from its stationary
distribution: mean zero and the variance its persistence and innovations
imply (the T0VAR diagonal 'stationary', see
ctFit).
With shared = TRUE the items share one process instead, loading on
the first at one and on the others freely: what moves them together beyond
the model's other latents. Measurement errors that are correlated within an
occasion are usually better read this way – an error is meant to be
random, and co-movement it does not carry over to the next occasion is a
process faster than the occasions – unless the design gives a reason for
correlated error, such as a method shared by the items.
The extended model.
ctModelImprove, which adds one for the items the
adequacy check points to.
m <- ctModel(type = 'ct', manifestNames = c('Y1', 'Y2', 'Y3'), latentNames = 'eta', LAMBDA = matrix(c(1, 'l2', 'l3'), 3)) m2 <- ctModelUniqueProcess(m, 'Y2') m2$latentNames p <- m2$pars p[p$col %in% 2 & p$matrix %in% c('LAMBDA', 'DRIFT', 'DIFFUSION'), c('matrix', 'row', 'col', 'param', 'value')] ctModelUniqueProcess(m, c('Y2', 'Y3'), shared = TRUE)$latentNamesm <- ctModel(type = 'ct', manifestNames = c('Y1', 'Y2', 'Y3'), latentNames = 'eta', LAMBDA = matrix(c(1, 'l2', 'l3'), 3)) m2 <- ctModelUniqueProcess(m, 'Y2') m2$latentNames p <- m2$pars p[p$col %in% 2 & p$matrix %in% c('LAMBDA', 'DRIFT', 'DIFFUSION'), c('matrix', 'row', 'col', 'param', 'value')] ctModelUniqueProcess(m, c('Y2', 'Y3'), shared = TRUE)$latentNames
The two graphs network psychometrics draws from a multivariate time series – the directed temporal network and the undirected contemporaneous network – computed from a ctsem model specification or from a fit on either backend.
Both are functions of the time interval, which is what a continuous time
model has to say here that a discrete time VAR does not. Call it at several
dt and the networks change shape, not merely scale.
ctNetwork( x, dt = 1, state = NULL, standardise = TRUE, impulseType = "unit", threshold = 0, networks = c("temporal", "contemporaneous"), plot = FALSE, quiet = FALSE, ..., observational )ctNetwork( x, dt = 1, state = NULL, standardise = TRUE, impulseType = "unit", threshold = 0, networks = c("temporal", "contemporaneous"), plot = FALSE, quiet = FALSE, ..., observational )
x |
A |
dt |
Time interval for the temporal and contemporaneous networks. A
whole number of steps for a discrete time model. Length one; to see the
interval dependence, call |
state |
Where a state dependent cell is evaluated, when |
standardise |
If TRUE (the default), temporal edges are in standard
deviation units of the processes themselves, using
|
impulseType |
What a one unit change in a process brings with it; see
|
threshold |
Edges with |
networks |
Which networks appear in |
plot |
If TRUE, returns the |
quiet |
Suppress the messages about how the numbers were arrived at. |
... |
Passed to |
observational |
Deprecated. Use |
Three networks over the latent processes, plus the measurement model:
temporalexpm(DRIFT * dt), the discrete time
autoregression and cross-lagged matrix at interval dt. Directed,
with the diagonal as self-loops (the autoregressions). Cell
[i,j] is the effect of process j now on process i
one interval later, so an edge runs from the column to the row. This is
ctDiscretePars at a single interval, and shares its
impulseType and standardise arguments.
contemporaneousPartial correlations of the innovation
covariance accumulated over dt,
Q(dt) = int_0^dt expm(DRIFT s) DIFFUSIONcov t(expm(DRIFT s)) ds.
This is the continuous time counterpart of the graphicalVAR
contemporaneous network. Partial correlations are scale free, so as
dt shrinks this tends to the partial correlations of
DIFFUSIONcov itself, and as dt grows to those of the
asymptotic covariance.
asymptoticPartial correlations of
asymDIFFUSIONcov, the long run within-subject covariance – the
network a cross-sectional study of a stationary system would see.
NULL if the system is not stationary.
measurementLAMBDA, as a bipartite manifest by latent
graph. Directed from latent to manifest.
From a fit, the matrices are the posterior medians
ctSummaryMatrices reports. From an unfitted specification, a
fixed cell is its own value and a free cell takes the value its transform
gives at a raw parameter of zero, which is where estimation starts; how many
DRIFT, DIFFUSION and LAMBDA cells were treated that way is reported, those
being the only matrices an edge can come from.
A list of class ctNetwork:
temporal, contemporaneous, asymptotic,
measurement
Weight matrices with the model's own names, ready to
hand to qgraph::qgraph(). asymptotic may be NULL.
edgesTidy data frame: network, from,
to, weight, directed, self. Thresholded, and
restricted to networks.
nodesData frame of name and type
('latent' or 'manifest').
DRIFT, DIFFUSIONcov, asymDIFFUSIONcov,
innovation
The matrices the networks were derived from, so every
number above can be checked. On a julia fit DRIFT is the drift's
Jacobian at the evaluation point, DRIFT itself where no cell reads the
state (see the section on state dependent models).
With attributes dt, standardise, impulseType,
threshold, networks, continuoustime, source,
stateLabel (the evaluation point, for a fit) and
stateDependent (the cells with no single value, or NULL).
A DRIFT, DIFFUSION or LAMBDA cell may be written as an expression
referencing a latent process, and then there is no such thing as the
network. What a fit reports is the system linearised at one point of the
state space, so the graph drawn from it is that linearisation's graph:
somewhere else the edge weights differ, and for a strongly nonlinear model so
does which edges are there at all. The temporal and contemporaneous networks
then come from the drift's Jacobian at that point (JAx) and the
measurement network from the measurement's (Jy), which differ from
DRIFT and LAMBDA there where a cell is nonlinear in the state: a cell
a + b*Y in the column of process Y has slope a + 2*b*Y.
A stan fit reports no Jacobians, so its edges are the cells' values at
T0MEANS, and it says so.
When any such cell is present, ctNetwork says so once, names the
evaluation point, and records the cells in
attr(x, 'stateDependent'); ctNetworkPlot puts the point
in the figure's subtitle so a saved plot carries its own caveat. Use
state to move the point, and
ctPhasePortrait or ctStateDependencePlot to see
the variation itself rather than one slice of it. Nothing is said for a
linear model, whose matrices are the same everywhere.
From an unfitted specification such a cell is reported and left out instead, since filling it in would draw a model that was not written.
ctNetworkPlot to draw it,
ctDiscretePars for the temporal network across a continuum of
intervals, ctPhasePortrait and
ctStateDependencePlot for what a state dependent system does
away from the point these edges were evaluated at.
# Two processes, b driven by a, drawn at one interval. m <- ctModel(type='ct', n.latent=2, n.manifest=2, manifestNames=c('y1','y2'), latentNames=c('a','b'), LAMBDA=diag(2), DRIFT=matrix(c(-1, 0, .5, -2), 2, 2, byrow=TRUE), DIFFUSION=matrix(c(1, 0, .3, 1), 2, 2, byrow=TRUE)) net <- ctNetwork(m, dt=1, standardise=FALSE) round(net$temporal, 4) # exp(-1) and exp(-2) on the diagonal round(net$contemporaneous, 4) net$edges# Two processes, b driven by a, drawn at one interval. m <- ctModel(type='ct', n.latent=2, n.manifest=2, manifestNames=c('y1','y2'), latentNames=c('a','b'), LAMBDA=diag(2), DRIFT=matrix(c(-1, 0, .5, -2), 2, 2, byrow=TRUE), DIFFUSION=matrix(c(1, 0, .3, 1), 2, 2, byrow=TRUE)) net <- ctNetwork(m, dt=1, standardise=FALSE) round(net$temporal, 4) # exp(-1) and exp(-2) on the diagonal round(net$contemporaneous, 4) net$edges
Plots what ctNetwork computes, in the style of the network
psychometrics literature: signed edges, width by magnitude, arrowheads and
self-loops on the directed temporal network, and no arrowheads on the
undirected contemporaneous one.
Give dt more than one value and the same networks are drawn side by
side at each interval, which is the continuous time point: the graph a study
would have recovered depends on how often it measured.
ctNetworkPlot( x, dt = 1, state = NULL, networks = c("temporal", "contemporaneous"), threshold = 0, engine = c("ggplot", "qgraph"), maxwidth = 3, arrowsize = 2.6, nodesize = 10, labelsize = 3.2, poscolour = "#2166AC", negcolour = "#B2182B", title = "auto", ... )ctNetworkPlot( x, dt = 1, state = NULL, networks = c("temporal", "contemporaneous"), threshold = 0, engine = c("ggplot", "qgraph"), maxwidth = 3, arrowsize = 2.6, nodesize = 10, labelsize = 3.2, poscolour = "#2166AC", negcolour = "#B2182B", title = "auto", ... )
x |
A |
dt |
One or more time intervals. Ignored, with a message, when |
state |
Where a state dependent cell is evaluated; see
|
networks |
Which networks to draw; see |
threshold |
Edges with |
engine |
|
maxwidth |
Line width of the largest edge drawn. |
arrowsize |
Arrowhead length in mm. |
nodesize, labelsize
|
Node and label size. |
poscolour, negcolour
|
Colours for positive and negative edges. The
defaults are a colour-blind safe blue and red; the literature's green and
red are |
title |
Panel title, or NULL for none. |
... |
Passed to |
Drawn with ggplot2, which ctsem already depends on, so no extra package is
needed. engine='qgraph' hands the weight matrix to
qgraph::qgraph() instead, if that package is installed –
ctNetwork's matrices are in the shape qgraph expects, so
qgraph::qgraph(ctNetwork(fit)$temporal, directed=TRUE) needs nothing
from this function.
A ggplot2 object, which can be modified further or printed. With
engine='qgraph', the qgraph object, invisibly, having drawn.
m <- ctModel(type='ct', n.latent=3, n.manifest=3, manifestNames=c('y1','y2','y3'), latentNames=c('a','b','c'), LAMBDA=diag(3), DRIFT=matrix(c(-.4,0,0, .3,-.5,0, 0,.4,-.6), 3, 3, byrow=TRUE), DIFFUSION=matrix(c(1,0,0, .3,1,0, 0,-.4,1), 3, 3, byrow=TRUE)) # The indirect path a -> b -> c is invisible at a short interval and present # at a long one -- the same model, two networks. ctNetworkPlot(m, dt=c(.2, 2), networks='temporal')m <- ctModel(type='ct', n.latent=3, n.manifest=3, manifestNames=c('y1','y2','y3'), latentNames=c('a','b','c'), LAMBDA=diag(3), DRIFT=matrix(c(-.4,0,0, .3,-.5,0, 0,.4,-.6), 3, 3, byrow=TRUE), DIFFUSION=matrix(c(1,0,0, .3,1,0, 0,-.4,1), 3, 3, byrow=TRUE)) # The indirect path a -> b -> c is invisible at a short interval and present # at a long one -- the same model, two networks. ctNetworkPlot(m, dt=c(.2, 2), networks='temporal')
The names ctFit reads from optimcontrol. A name means
the same thing on both backends. One a backend does not have is refused by
name, unless its value asks for nothing that backend does not already do
(gradient = 'adjoint' on stan, nsubsets = 1 on julia), and a
misspelled name is an error. Leaving a name unset keeps that backend's own
default.
tolBoth. Stop when the objective changes by less than this.
Default 1e-8 on stan, off on julia.
g_tolBoth. Stop when the l2 norm of the gradient falls below
this. Default 1e-8 on julia, off on stan.
x_tolBoth. Stop when the parameter step falls below this. Off by default.
maxiterBoth. Iteration cap. Default 1000 on julia, mize's on stan.
lbfgs_scoresJulia. Default TRUE. Runs beside L-BFGS a second model of the curvature, the centred outer product of the per-unit scores plus the prior's precision and a learned correction, and takes each step on whichever model predicted the last step's gain better; while a fit is batched, only on the whole data.
lbfgs_localJulia. Default TRUE. On the Laplace route, where some parameters are read by few units (a study's, a group's, an item few answered), starts L-BFGS's inverse Hessian from their exact Hessian columns, differenced over those units alone, and the score model of the rest; built once the optimisation has spent what it costs.
lbfgs_memoryBoth. How many curvature pairs L-BFGS keeps. Default on julia one per free parameter, at least 20 and at most 1000; mize's on stan.
stallretriesBoth. Default 2. How often a fit that stopped
short of a maximum is tried again: stan restarts from fresh random values
when the gradient exceeds stalltol per data point; julia resumes
from the better point its probe found by pulling flat coordinates back,
or from where it stopped when the check finds the fit short, keeping a
resumed stage only if it improves the objective, and stopping one that
stops gaining. 0 turns it off.
initsdBoth. Default 0.01. The sd of the random start, on the raw scale.
datastartJulia. Default TRUE. Starting values for the
diagonals of DRIFT, DIFFUSION, T0VAR and MANIFESTVAR are read off the
data's scale – through the link, so a count is read through log1p
and a binary or ordinal indicator contributes none – rather than from one
fixed point whatever the units. They are kept only where the objective can
be evaluated there; otherwise the fit starts from the default and says so.
Supplied inits are never overridden.
carefulfitBoth. A rough first pass with ctsem's
normal(0,1) raw priors on every coordinate, for starting values; the
fit's own objective is then maximised from there. Skipped with
priors = TRUE, which already has those priors, and when inits
are supplied. Default TRUE on both backends. On julia it runs to its own
convergence, at most 200 iterations, and a number sets that cap; it keeps
the fit from starting where a boundary (a correlation at one, a variance at
zero) has already captured it. fit$optim$carefulfit records whether
it ran.
stochasticBoth. Start with ctsem's stochastic-gradient
optimiser and finish with L-BFGS; 'auto' does so above 50
parameters. Default TRUE on stan, FALSE on julia. On large, poorly
conditioned models it can be much faster than L-BFGS alone, and it can
carry a fit further toward a degenerate limit, such as a variance running
to zero. fit$optim$sgd_iterations records the phase.
gradientJulia. 'adjoint' (reverse mode, the default)
or 'forward'. The same gradient; the adjoint's cost does not grow
with the number of parameters.
batchJulia. Default TRUE, or FALSE when inits are
supplied. Start on a random subset of the subjects and grow it as the fit
needs more data; needs at least 80 subjects or top-level groups. A
supplied start is not batched from, since the subsets' gradients walk off a
point already chosen; TRUE asks for it anyway.
coresetJulia. Default 'auto'. Run stretches of
L-BFGS on a weighted few of the units (subjects, or top-level groups where
random effects are integrated by Laplace), chosen so that
their changes in gradient and their spread reproduce all the units', each
stretch checked by a full gradient; the full objective finishes, so the
estimate and its standard errors are those of every unit. 'auto'
runs them for data of at least 200 subjects or top-level groups and 10,000
data cells (rows times manifest variables), where they save time; on
smaller data they save none and can change which of several maxima a fit
reaches. TRUE runs them whenever a coreset is expected to save
gradients, FALSE never. fit$optim$coreset_sizes records the
stages.
explosive_forwardJulia. Default FALSE. For a subject whose
predictions grow more than 100-fold between observations (explosive
dynamics over a long gap), take its gradient by forward mode instead of
the reverse pass, as a cross-check: the reverse pass is accurate there
too. The first such subject in a session compiles the filter again,
minutes on a large model. The fit counts such subjects either way
(fit$optim$explosive_passes) and warns when any is explosive at
the estimate.
lbfgs_diagonalJulia. Default TRUE. L-BFGS's initial inverse
Hessian takes a scale per parameter, learned from the curvature pairs
(Gilbert and Lemarechal); FALSE uses one scale for all, which can be much
slower on large models. Like stochastic, it can carry a fit further
toward a degenerate limit.
lbfgs_gllJulia. Default 0. A step is accepted against the
worst of the last W objective values (Grippo, Lampariello and
Lucidi), so it may cross a curved valley.
lbfgs_nonmonotoneJulia. Between 0 (the default) and 1. A step is accepted against a running average of recent objective values (Zhang and Hager).
initial_alphaJulia. Default 0.1. The length of the first trial step, in raw units.
preconditionJulia. Default FALSE. TRUE starts L-BFGS's per-parameter scaling from the model's transforms rather than learning it from the curvature alone. It made fits whose intercepts start far from the data settle in worse optima.
saemJulia, intoverpop = 'laplace'. Default FALSE.
Start with SAEM: the random effects are sampled from their conditional
distribution rather than integrated by the Laplace approximation, heading
for the exact marginal posterior mode. The usual optimiser then polishes
and certifies its point on the Laplace objective, so the estimate usually
ends where a fit without SAEM does; what it can change is which maximum is
reached when the Laplace objective has more than one. Its point carries
Monte Carlo error, so where the posterior itself has several maxima –
weakly identified random-effect scales and correlations – that error, not
the data, can decide which one is reached. So the fit also finishes from
its own start without SAEM, one more optimisation, and keeps whichever
maximum has the higher log posterior on the quadrature-corrected objective
(quadrature for each unit narrow enough, as the refined fit
screens), where a maximum the Laplace approximation over-credits loses;
fit$optim$saem_guard records both. Not a better
approximate estimate on its own – for that, sample with
optimize = FALSE. A number caps its iterations (TRUE is 10000).
fit$optim$saem_iterations, saem_settled, saem_trend,
saem_chains, saem_acceptance and saem_trace record the
phase.
saem_proposalJulia. SAEM's proposal, 'rw' or
'laplace'.
restartsJulia. Default 0. Random restarts for a fit that has
still not converged after its corrections – typically a likelihood with
more than one basin – each from the fit's own start plus normal noise of sd
restartsd (default 1) on the raw scale, keeping the best if it is
better. They run in worker processes when cores > 1 and the fit took
more than 30 seconds. Not when inits or maxiter were supplied.
fit$optim$restarts records each start.
newtonDefault TRUE. Newton steps once close, on an exact
Hessian updated after each step and formed again when its steps stop paying
for it, ending on the Hessian the certification and the standard errors
use; it tries the direction of negative curvature at a saddle and checks
what the directions with no curvature are worth. Under
intoverpop = 'laplace' it runs only when the fit is certified. May
name the curvature its steps use: 'chord' (the default except as
below), 'exact' (a fresh Hessian whenever a step contracts slowly),
'subset', or 'krylov': the outer product of the units'
scores corrected by Hessian-vector products taken at each step, with the
exact Hessian formed where its steps converge. 'krylov' is the
default under intoverpop = 'laplace' where the exact Hessian would
cost at least 200 gradients. fit$optim$newton_steps records the
steps, and newton_krylov_products the gradients the Krylov steps
took.
certifyDefault TRUE. The curvature check that certifies a
fit has converged. estonly skips it too.
gaptolDefault 5e-5. How much objective the curvature
may still predict at the estimate before the fit is continued. The gap is
half the squared distance from the maximum in standard errors, so the
default is 0.01 standard errors: close
enough that the estimate's Hessian is the maximum's. A smaller value, such
as 1e-6, holds the estimate closer still.
hessianreuseDefault 0.3. How far, in standard errors, the estimate may move from where a Hessian was formed for that Hessian still to give the standard errors. A Wald standard error assumes the curvature is constant over about a standard error around the estimate, and is in error by however much it is not; a Hessian formed this far away adds about this fraction to that error, where forming a fresh one costs a gradient per parameter (two on the Laplace route). 0 forms one at the estimate.
innergaptolWhat a prediction of the remaining gain that
can be optimistic – L-BFGS's, or a Newton step's on a Hessian taken at
another point – has to fall below to stop; by default a hundredth of
gaptol. A gain read off the exact Hessian at the estimate is held to
gaptol itself.
overshoot'magnitude' (the default),
'saturation' or 'off': which directions are pulled back after
a fit, to test whether it stopped at a maximum.
stallwindow, stallratio, stallcooldown
The stall check (defaults 80, 0.001, 30): over how many iterations a fit's gain is read (half a nat or less starts a look for a transform gone flat), how much of its responsiveness a transform must have lost to count as the reason the fit stopped, and how long the check waits after finding a fit slow rather than stuck.
escapesaturatedDefault FALSE. Refit a fit stalled on a saturated transform from a zeroed boundary, keeping whichever wins.
escapepinDefault FALSE. Hold the escaped coordinates still while the rest re-optimises around them, then release them.
intoverpop = 'laplace')laplace_inner_maxiter, laplace_inner_tol
Defaults 200
and 1e-10. The inner solve that finds each unit's random-effect mode.
Raise the first when the fit reports modes that did not converge; the
tolerance bounds the inner gradient, floored at
1e-10 * (1 + abs(value)).
laplace_onestepFALSE (the default) or TRUE.
Where the filter can integrate a person's intercept-type effects (initial
means, intercepts, a predictor's effect) exactly, given their other random
effects, it does, and Laplace approximates only the integral over the
others: the two-step term. TRUE takes one Laplace approximation over
all of them together instead, as before version 4.0's change of default.
Beside random dynamics (a random drift or diffusion) the two-step term is
the more accurate; the one-step term is what a model whose filter cannot
integrate them (one whose dynamics read the state) takes in any case, and
ctModelImprove() and ctScoreTest() use it to compare such a
model with one that can.
laplace_soft_directions'quadrature' (the default) or
'logdet_floor': how a unit is scored whose likelihood is convex in
some direction of its random effects, where plain Laplace would credit that
direction more width than the integrand has. 'quadrature' integrates
the softest direction by a three-point quadrature where the unit's
curvature has an eigenvalue below 0.7, blending into plain Laplace between
0.2 and 0.7; 'logdet_floor' floors the log determinant at zero.
Recorded in fit$laplace$soft_directions, and used by every function
that rebuilds the objective; fit$laplace$conditioning counts such
units.
It applies too when sampling with intoverpop = FALSE and
sampleControl$placement = 'fit'.
uncertaintyBoth. Default 'hessian'
('sandwich' with subject weights on julia). The method computed
when the optimiser ends, as in ctFitUncertainty. The
default is for building a model, with the checks below beside it; for
intervals to report, sample with ctFitUncertainty(fit, 'sample').
'sample' is refused here: use optimize = FALSE, or
ctFitUncertainty(fit, 'sample') on the fit.
checksJulia. Default TRUE: the two checks
a fit makes last, of the approximations its numbers rest on, as
ctFitUncertainty's checks makes them; FALSE
for neither, or one of 'randomeffects' and 'intervals'.
The random-effects check, on intoverpop = 'laplace': every unit,
whatever its number of effects, scored at draws of its random effects,
and a warning where integrating them exactly would move an estimate by
0.1 standard errors or more, giving the move's size below one standard
error (past that the check's linear step is no guide to it) and how far
Laplace's log likelihood is from the integrated one, and naming
ctFitUncertainty(fit, intoverpop = 'quadrature') and joint
sampling. Wherever it runs, fit$estimate$loglik is Laplace's plus
the check's integral of every unit's error (loglik_se its standard
error, loglik_laplace the value the optimiser maximised,
loglik_method 'checked'). Every model whose effects enter
linearly passes at once, and a fit whose optimiser stopped where a unit's
Laplace term jumps warns instead, naming the units;
fit$laplace$correction records it. The interval check, with
uncertainty = 'hessian': 'checked''s check of the
curvature's normal against the posterior's gradients, on the density the
intervals come from, said in a message and in summary() where the
posterior departs from that normal. Under
intoverpop = 'quadrature' that density is the refined fit's, where
the check can cost more than the refinement (see ctFitUncertainty's
checks); checks = 'randomeffects' leaves it out.
Neither runs with estonly.
uncertaintyDraws, finishsamples,
uncertaintyControl
Both. ctOptimUncertainty's draws
(default 'auto'), and ctFitUncertainty's
finishsamples (1000) and control.
accuracy, sampleControl
Julia. An accuracy in
posterior standard deviations, off by default: once uncertainty's
method has run, its own account of its error is read against it, and
where the method falls short, or has none, the fit is sampled to that
accuracy (ctFitUncertainty's control$accuracy);
sampleControl holds the sampler's other settings for that run.
estonlyBoth. Default FALSE. Point estimates only: no Hessian,
so no standard errors and no certification; under
intoverpop = 'laplace' no Newton finish and no quadrature
correction either.
callbackA function called while the fit runs, with
(iteration, total, objective, gradient_norm, parameters), where
parameters is the raw vector the objective and gradient describe; one
declaring only the first four is called with four. It is called on a time
cadence and once more at the end, for live progress in a front end;
checkpoint below writes a resumable checkpoint itself. An error
inside it disables it with a warning. fit$optim$trace holds
every iteration after the fit, and ctTracePlot draws it.
checkpointA file path. While the fit optimises, its point is
written there every checkpoint_every seconds (default 60) and at the
end of each run, with each subject's random-effect modes on the Laplace
route and the smoothing parameters of any smooth terms; each exact Hessian
the fit forms is kept beside it (<checkpoint>.hessian). Where the
file already exists, the fit starts from it: the same call again resumes an
interrupted fit, its modes, smoothing parameters and Hessian included,
which starting from inits alone does not keep. A file written for a
different model stops the fit rather than being overwritten.
progressForce the progress line on or off.
tipredMissingHow missing TI predictor values are handled.
Each gets a normal distribution from a regression on the other variables;
'impute' holds the value at its mean, 'sample' makes it a
parameter with that distribution as its prior, and 'auto', the
default, imputes when optimizing and samples when sampling. A sampled fit
is placed with the values imputed. ctFitUncertainty(fit, 'sample')
reads the fit's setting, and its control$tipredMissing overrides
it.
tipredMissingIncludeOutcomeDefault TRUE. Whether the outcome informs the imputation of missing TI predictors.
saveEffectsA sampling setting; read here for older scripts.
Use sampleControl$saveEffects.
nsubsets, subsamplesize, lproughnesstarget,
stochasticTolAdjust, parsteps
Tuning of stan's stochastic-gradient optimiser and its stepwise phase.
stalltolDefault 0.01. The gradient per data point above which
stan calls a fit stalled and restarts it (see stallretries).
ctFit, ctFitUncertainty,
ctTracePlot.
Builds the |-separated cell specification ctsem's model matrices use,
from named arguments rather than positional fields.
ctPar( name, transform = NA, indvarying = NA, sdscale = NA, tipreds = NULL, mean = NA )ctPar( name, transform = NA, indvarying = NA, sdscale = NA, tipreds = NULL, mean = NA )
name |
Parameter name. Two cells given the same name are the same parameter, which is how equality constraints are written. |
transform |
Transform expression in terms of |
indvarying |
Where the parameter varies: |
sdscale |
Multiplier on the population standard deviation's prior, at every level where the parameter varies. Only meaningful when it varies. |
tipreds |
Time-independent predictors acting on this parameter: a
character vector of names for effects to estimate, e.g. |
mean |
A population mean stated rather than estimated, for a parameter
that varies ( |
The output is the same string the | syntax produces, so this
is a way of writing one rather than an alternative to it. Fields left at
their default are emitted empty, which is what the parser reads as
"unspecified".
The same named fields can also be written directly into a model matrix
cell, in which case nothing needs to be called. A cell containing
| is read as a |-separated specification; otherwise a cell
with a field of the form name=value is read as a named
specification, and anything else as a plain parameter name. All three of
these specify the same parameter:
MANIFESTMEANS = matrix('mm||TRUE|0.5')
MANIFESTMEANS = matrix(ctPar('mm', indvarying = TRUE, sdscale = 0.5))
MANIFESTMEANS = matrix('mm, indvarying=TRUE, sdscale=0.5')
In the named form, fields are separated by commas (commas inside brackets
belong to the value, so transform=pnorm(param, 0, 1) is one field),
the parameter name comes first, and several predictors or levels are given
as tipreds=c(age, sex) or indvarying=c(person, couple); in
the | form they are separated by commas, as in
'cross||person,couple'. A field name that is not one of the above is
an error naming the nearest valid field, so a misspelling such as
topreds=age is refused rather than becoming part of a parameter
name. Comparisons are unaffected: a state-dependent cell may contain
==, >=, <= or !=, and a named argument
inside a call belongs to the value it appears in.
A single character string, suitable for a model matrix cell.
ctModel for the matrices these cells go in.
ctPar('mm', indvarying = TRUE) ctPar('drift11', transform = '-exp(param)') ctPar('mm', indvarying = TRUE, tipreds = c('age', 'sex')) # varying between couples only, in a model with id = c('person', 'couple') ctPar('cross', indvarying = 'couple') # a person-level intercept whose population mean is zero ctPar('lev', indvarying = TRUE, mean = 0) ## Not run: # These two models are identical. model <- ctModel(type = 'ct', n.latent = 1, n.manifest = 1, LAMBDA = matrix(1), MANIFESTMEANS = matrix(ctPar('mm', indvarying = TRUE, sdscale = 0.5))) model <- ctModel(type = 'ct', n.latent = 1, n.manifest = 1, LAMBDA = matrix(1), MANIFESTMEANS = matrix('mm, indvarying=TRUE, sdscale=0.5')) ## End(Not run)ctPar('mm', indvarying = TRUE) ctPar('drift11', transform = '-exp(param)') ctPar('mm', indvarying = TRUE, tipreds = c('age', 'sex')) # varying between couples only, in a model with id = c('person', 'couple') ctPar('cross', indvarying = 'couple') # a person-level intercept whose population mean is zero ctPar('lev', indvarying = TRUE, mean = 0) ## Not run: # These two models are identical. model <- ctModel(type = 'ct', n.latent = 1, n.manifest = 1, LAMBDA = matrix(1), MANIFESTMEANS = matrix(ctPar('mm', indvarying = TRUE, sdscale = 0.5))) model <- ctModel(type = 'ct', n.latent = 1, n.manifest = 1, LAMBDA = matrix(1), MANIFESTMEANS = matrix('mm, indvarying=TRUE, sdscale=0.5')) ## End(Not run)
The filter ctsem fits with is an assumed-density approximation: it carries a Gaussian for the latent states, linearises the drift over each prediction step, and projects non-Gaussian measurement updates back onto a Gaussian. For a linear model with Gaussian indicators it is exact; for anything else the size of its error is invisible from inside it. This function estimates the same marginal log likelihood at the fit's estimates with a bootstrap particle filter that makes none of those approximations, and reports the difference, in total and per row.
ctParticleLik( fit, particles = 2000, substeps = 20, transition = c("exponential", "euler"), seed = 1, replicates = 1, cores = getOption("mc.cores", 2L) )ctParticleLik( fit, particles = 2000, substeps = 20, transition = c("exponential", "euler"), seed = 1, replicates = 1, cores = getOption("mc.cores", 2L) )
fit |
A fit from |
particles |
Particles per subject. |
substeps |
Transition steps per observation interval. This is the
reference mesh, independent of anything the fit used, and floored by the
fit's own |
transition |
|
seed |
Fixes every draw; the result is reproducible given it. |
replicates |
Independent runs (with seeds |
cores |
Engine threads. The particle filter splits the subjects across them, each subject with its own random stream, so the result does not depend on the count. |
Each particle steps through the process with drift, diffusion and every
state-dependent parameter evaluated at its own state, is weighted by the
conditional density of the row's observations given that state (Gaussian,
binary, ordinal, count and censored indicators are all handled), and is
resampled when the weights degenerate. The estimate is unbiased with Monte
Carlo error falling as one over the square root of particles.
It is a diagnostic at fixed parameters, not an estimator. Two things decide
whether to trust the comparison: se should be well below the
difference you care about (raise particles, or set replicates
for an empirical standard error), and doubling substeps should not
move the answer.
A list: loglik, the particle estimate (mean over replicates);
se; fit_loglik, the filter's log likelihood from the fit;
difference, particle minus filter; rows, a data frame with
the subject, time, particle and filter log likelihood of every row and
their difference; replicates, the individual estimates;
ess_min, the smallest effective sample size any row saw; and the
settings.
ctFitUncertainty with method = 'is' and
filter = 'particle' applies the correction this
measures, to the fit's posterior draws.
# Needs a working julia backend, so the example is inert where Julia is # absent -- including on CRAN, whose check machines have none. if (isTRUE(ctJuliaStatus()$available)) { model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1)) model$pars$indvarying <- FALSE fit <- ctFit(ctstantestdat, model, backend = 'julia', cores = 1) # A linear Gaussian model, where the filter is exact: the difference # should be within a few standard errors of zero. check <- ctParticleLik(fit, particles = 1000, substeps = 5) check[c('loglik', 'se', 'fit_loglik', 'difference')] # Julia keeps running between calls, holding what it has compiled; # this ends it. ctJuliaKill("session") }# Needs a working julia backend, so the example is inert where Julia is # absent -- including on CRAN, whose check machines have none. if (isTRUE(ctJuliaStatus()$available)) { model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1)) model$pars$indvarying <- FALSE fit <- ctFit(ctstantestdat, model, backend = 'julia', cores = 1) # A linear Gaussian model, where the filter is exact: the difference # should be within a few standard errors of zero. check <- ctParticleLik(fit, particles = 1000, substeps = 5) check[c('loglik', 'se', 'fit_loglik', 'difference')] # Julia keeps running between calls, holding what it has compiled; # this ends it. ctJuliaKill("session") }
Adds to long format data, for each subject, the cosine and sine of the
time of its first row over period: the columns <name>_cos0
and <name>_sin0 that ctModelPeriodicComponent sets its
cycle from.
ctPeriodicPhases(datalong, period, name = "cycle", id = "id", time = "time")ctPeriodicPhases(datalong, period, name = "cycle", id = "id", time = "time")
datalong |
Long format data, one row per observation. |
period |
The cycle's period, in the data's time units. |
name |
The cycle's name, as given to
|
id, time
|
The subject and time columns. |
datalong with the two columns added.
d <- data.frame(id = rep(1:2, each = 3), time = c(8, 12, 20, 33, 37, 45)) ctPeriodicPhases(d, period = 24)d <- data.frame(id = rep(1:2, each = 3), time = c(8, 12, 20, 33, 37, 45)) ctPeriodicPhases(d, period = 24)
Draw the model's vector field over a plane of the latent state space: at each point, the direction and speed the deterministic part of the system moves. For a model whose matrices depend on the latent state – the reason the julia backend exists – this shows behaviour that no single reported DRIFT matrix can, because there is no single DRIFT matrix.
ctPhasePortrait( fit, latents = 1:2, extent = "data", gridsize = 15, state = "asymptotic", tipreds = NULL, nullclines = TRUE, fixedpoint = TRUE, trajectories = 0, plot = TRUE )ctPhasePortrait( fit, latents = 1:2, extent = "data", gridsize = 15, state = "asymptotic", tipreds = NULL, nullclines = TRUE, fixedpoint = TRUE, trajectories = 0, plot = TRUE )
fit |
A ctJuliaFit, or a ctStanFit of a linear model. |
latents |
Two latent processes, by name or index, forming the plane. |
extent |
|
gridsize |
Arrows per side. The field costs one engine call per point, so cost is quadratic in this. |
state |
Where the *other* latent processes are held, as in
|
tipreds |
Time independent predictor values, or NULL for the population. |
nullclines |
Draw the curves where each process stops changing? |
fixedpoint |
Mark the asymptotic state when it is inside the region? |
trajectories |
Number of subjects' smoothed trajectories to overlay, or 0 for none. |
plot |
If FALSE, return the field, nullclines and trajectories as data rather than a plot. |
Three things are drawn over the field, each of which can be turned off:
Nullclines, the curves where one process stops changing. Where the nullclines of two processes cross, the whole system stops: that is a fixed point, and a nonlinear system may have several.
The fixed point found by state='asymptotic', marked
when it lies inside the plotted region. Only one is found, by Newton from
T0MEANS; the nullcline crossings show whether there are others.
Smoothed trajectories for a few subjects, so the field can be read against where the data actually went.
With more than two latent processes the remaining ones are held at
state, so the picture is a slice rather than the whole story –
change state and the slice changes.
A linear model has an affine field, straight nullclines and exactly one fixed point. It is drawn correctly, from a stan fit as readily as a julia one, but it shows nothing the DRIFT matrix did not already say. A nonlinear stan fit is refused rather than drawn as if it were linear.
A ggplot, or when plot=FALSE a list with field, the
latents on the plane, stateLabel naming where the off-plane
processes were held, the held process names, and nullclines,
fixedpoint and trajectories when asked for. The plot names
stateLabel in its subtitle whenever anything is held off-plane.
ctContextDependence, ctDiscretePars
# A linear stan fit needs no engine, so the bundled example fit is enough. ctPhasePortrait(ctstantestfit, gridsize = 7, plot = FALSE)# A linear stan fit needs no engine, so the bundled example fit is enough. ctPhasePortrait(ctstantestfit, gridsize = 7, plot = FALSE)
1st margin of $Y sets line values, 2nd sets variables, 3rd quantiles.
ctPlotArray( input, grid = FALSE, add = FALSE, colvec = "auto", lwdvec = "auto", ltyvec = "auto", typevec = "auto", plotcontrol = list(ylab = "Array values", xaxs = "i"), legend = TRUE, legendcontrol = list(), polygon = TRUE, polygonalpha = 0.1, polygoncontrol = list(steps = 25) )ctPlotArray( input, grid = FALSE, add = FALSE, colvec = "auto", lwdvec = "auto", ltyvec = "auto", typevec = "auto", plotcontrol = list(ylab = "Array values", xaxs = "i"), legend = TRUE, legendcontrol = list(), polygon = TRUE, polygonalpha = 0.1, polygoncontrol = list(steps = 25) )
input |
list containing 3 dimensional array to use for Y values, |
grid |
Logical. Plot with a grid? |
add |
Logical. If TRUE, plotting is overlayed on current plot, without creating new plot. |
colvec |
color vector of same length as 2nd margin. |
lwdvec |
lwd vector of same length as 2nd margin. |
ltyvec |
lty vector of same length as 2nd margin. |
typevec |
type vector of same length as 2nd margin. |
plotcontrol |
list of arguments to pass to plot. |
legend |
Logical. Draw a legend? |
legendcontrol |
list of arguments to pass to |
polygon |
Logical. Draw the uncertainty polygon? |
polygonalpha |
Numeric, multiplier for alpha (transparency) of the uncertainty polygon. |
polygoncontrol |
list of arguments to pass to |
Nothing. Generates plots.
#' input<-ctTIpredEffects(ctstantestfit, plot=FALSE, whichpars='CINT', nsamples=10,nsubjects=10) ctPlotArray(input=input)#' input<-ctTIpredEffects(ctstantestfit, plot=FALSE, whichpars='CINT', nsamples=10,nsubjects=10) ctPlotArray(input=input)
Plots, for each free population parameter of a fit, its prior and its
posterior on the natural scale: the population mean, and for an individually
varying parameter the population standard deviation, at each level it varies
at. Works for fits from either backend. The prior is ctsem's
normal(0,1) raw prior carried through each parameter's transform –
the prior priors = TRUE fits with – so on a fit made without it the
plot shows how far the data moved each parameter from where that prior
would put it. The posterior is the fit's draws: genuine posterior draws for
a sampled fit, the approximate ones ctFitUncertainty writes
for an optimized one. ctStanPlotPost is maintained as a
backward-compatible alias.
ctPlotPosterior( obj, rows = "all", npp = 4, priorwidth = TRUE, smoothness = 1, priorsamples = 10000, plot = TRUE, wait = FALSE, ... )ctPlotPosterior( obj, rows = "all", npp = 4, priorwidth = TRUE, smoothness = 1, priorsamples = 10000, plot = TRUE, wait = FALSE, ... )
obj |
A fit from |
rows |
Character string 'all', or which parameters to plot, by position in the fit's free population parameters, in parameter order. |
npp |
Integer number of parameters to show per page. |
priorwidth |
if TRUE, plots will be scaled to show bulk of both the prior and posterior distributions. If FALSE, scale is based only on the posterior. |
smoothness |
Positive numeric – multiplier to modify smoothness of density plots, higher is smoother but can cause plots to exceed natural boundaries, such as standard deviations below zero. |
priorsamples |
number of samples from prior to use. More is slower. |
plot |
Logical, if FALSE, ggplot objects are returned in a list instead of plotting. |
wait |
If true, user is prompted to continue before plotting next graph. If false, graphs are plotted one after another without waiting. |
... |
For a stan fit, passed to the preparation of the prior draws:
|
With plot = TRUE, NULL invisibly, having drawn the
plots; with plot = FALSE, a list of ggplot objects.
ctPlotPosterior(ctstantestfit, rows=3:4)ctPlotPosterior(ctstantestfit, rows=3:4)
Plots uncertainty bands with shading
ctPoly(x, y, ylow, yhigh, steps = 20, ...)ctPoly(x, y, ylow, yhigh, steps = 20, ...)
x |
x values |
y |
y values |
ylow |
lower limits of y |
yhigh |
upper limits of y |
steps |
number of polygons to overlay - higher integers lead to smoother changes in transparency between y and yhigh / ylow. |
... |
arguments to pass to polygon() |
Nothing. Adds a polygon to existing plot.
plot(0:100,sqrt(0:100),type='l') ctPoly(x=0:100, y=sqrt(0:100), yhigh=sqrt(0:100) - runif(101), ylow=sqrt(0:100) + runif(101), col=adjustcolor('red',alpha.f=.1))plot(0:100,sqrt(0:100),type='l') ctPoly(x=0:100, y=sqrt(0:100), yhigh=sqrt(0:100) - runif(101), ylow=sqrt(0:100) + runif(101), col=adjustcolor('red',alpha.f=.1))
This function allows for easy comparison of data generated from a fitted ctsem model with the original data used to fit the model. It provides options to include residuals in the comparison.
ctPostPredData(fit, residuals = FALSE, nsamples = NA)ctPostPredData(fit, residuals = FALSE, nsamples = NA)
fit |
A fitted ctsem model, from either backend. |
residuals |
If set to TRUE, includes standardised prior residuals in the comparison,
as extra rows whose variable names are suffixed |
nsamples |
Integer or NA. Number of generated datasets to use. If the fit carries
more than this, draws are subsampled; if it carries none, this many are generated. NA
uses whatever the fit already has, or the |
A long data table with one row per generated draw x data row x variable, carrying
the generated value, the corresponding obsValue, and the id,
Time and TimeInterval of that data row. LogLik appears as an extra
'variable', holding the row-wise log likelihood.
ctPostPredPlots, which turns this into diagnostic plots.
data_comparison <- ctPostPredData(ctstantestfit)data_comparison <- ctPostPredData(ctstantestfit)
ctPostPredict and ctStanPostPredict are aliases for
ctPostPredPlots, which the two functions were merged into. They differ from
it only in printing the panels by default rather than returning them.
ctPostPredict(fit, ...)ctPostPredict(fit, ...)
fit |
A fitted ctsem model. |
... |
Passed to |
A named list of ggplot objects, invisibly.
ctPostPredict(ctstantestfit, wait = FALSE)ctPostPredict(ctstantestfit, wait = FALSE)
Compares data generated from the fitted model against the data it was fitted to, as a
set of diagnostic plots. Works for both the stan and julia backends.
ctPostPredict and ctStanPostPredict are maintained as aliases.
ctPostPredPlots( fit, panels = "all", variables = NULL, nsamples = NA, datarows = "all", diffsize = 1, interval = 0.95, resolution = 100, jitter = 0, residuals = FALSE, timebins = 10, lags = 0:5, lagcor = TRUE, notes = TRUE, plot = FALSE, wait = FALSE )ctPostPredPlots( fit, panels = "all", variables = NULL, nsamples = NA, datarows = "all", diffsize = 1, interval = 0.95, resolution = 100, jitter = 0, residuals = FALSE, timebins = 10, lags = 0:5, lagcor = TRUE, notes = TRUE, plot = FALSE, wait = FALSE )
fit |
A fitted ctsem model. |
panels |
Which panels to produce. |
variables |
Character vector of variable names to restrict to, or NULL for all.
|
nsamples |
Integer or NA. Number of generated datasets to compare against. NA uses whatever the fit already carries. |
datarows |
Integer vector of data rows to use, or |
diffsize |
Integer vector > 0. Lags, in observations, for the rate-of-change panels. One pair of panels is produced per lag. |
interval |
Numeric in (0,1). Width of the central predictive interval used by the
|
resolution |
Positive integer. Grid resolution of the shaded model density in the bivariate panels. |
jitter |
Non-negative numeric. Jitter added to the values in the rate-of-change panels, as a proportion of each variable's standard deviation, applied to observed and generated alike. Useful when data are rounded or ordinal. 0, the default, adds none. |
residuals |
Logical. Include standardised prior residuals as extra variables. Slow – it needs one Kalman pass per generated dataset. |
timebins |
Positive integer. Number of equal-count bins of time for the
|
lags |
Non-negative integer vector. Observation lags for the |
lagcor |
Logical. If TRUE (the default) the |
notes |
Logical. If TRUE (the default), each panel carries a caption saying what it shows and what a departure from its reference line means. Set FALSE for bare plots. |
plot |
Logical. If TRUE, prints the panels and returns them invisibly. If FALSE (the default) returns the list without printing. |
wait |
Logical. If TRUE and |
Two families of panel are produced.
structure asks whether the model reproduces the shape of the data:
Density (marginal distributions), MeanTrajectory (the mean over subjects
against time), ValueByTime and ValueByOccasion (the model-implied density
of each variable against time and against occasion number, with the observed data as
points on top), ChangeByValue and ChangeByTime (rate of change against
current value and against time), and LaggedCovariance (the lagged covariance or
correlation structure). ChangeByValue is the phase plane, and is the panel that
shows nonlinear dynamics: a linear model implies a straight band with a constant slope, so
curvature in the observed points relative to the model shading is evidence the drift is
not linear.
MeanTrajectory and ValueByTime both plot against time and answer different
questions. ValueByTime shows where the model puts an individual observation, so its
envelope is as wide as the data; MeanTrajectory shows where it puts the mean
of a bin of observations, whose sampling distribution is narrower by roughly the square
root of the bin count. A bias in the average level is therefore visible in the second while
sitting comfortably inside the first.
LaggedCovariance is ctFitCheckCov rendered as a dashboard panel: one
plot per variable, faceted by the variable it is paired with, empirical against the central
95% of the generated datasets at each lag. It is the only panel that looks at dependence
across time rather than at each time, which is what makes it the one that responds
to a wrong drift or diffusion when every marginal already matches. Call
ctFitCheckCov directly to split it by a group or a predictor.
calibration asks whether the predictive distribution is the right width and in
the right place: PredictedVsObserved, IntervalCoverage, PIT, and the
PIT resolved by CalibrationByInterval and CalibrationBySubject. A departure
that depends on the time interval implicates the dynamics rather than the measurement
model, since the measurement model cannot know the interval.
The row-wise log likelihood appears throughout as an extra variable, LogLik –
its distribution should match between model and data like any other quantity. It is
omitted from the rate-of-change panels, where the derivative of a log likelihood with
respect to time is not a meaningful quantity. Note that the observed side of the LogLik
comparison is evaluated at the point estimate, so it carries no parameter uncertainty
while the generated side does.
SubjectLogLik asks the same question one level up: each subject's total log
likelihood, summed over that subject's rows, compared as a cumulative distribution
across subjects against the same curve from each generated dataset. Aggregating within
a subject first is what makes a subject the model fits badly across many occasions show
up, rather than being spread thin across the row-wise panels. The comparison is against
replicate datasets of the same size rather than a smooth density, so it is the right
reference however few subjects there are.
The approximation is only as good as the number of generated datasets, so a fit carrying
few draws gives a coarse picture – the PIT in particular is discrete on
nsamples + 1 values.
A named list of ggplot objects, invisibly if plot=TRUE.
ctPostPredData for the underlying table,
ctFitCheckCov for the lagged covariance panel on its own, with splits by
group or predictor, and ctPhasePortrait for the model-implied phase plane
on its own.
plots <- ctPostPredPlots(ctstantestfit) print(plots$PIT)plots <- ctPostPredPlots(ctstantestfit) print(plots$PIT)
Outputs predicted, updated, and smoothed estimates of manifest indicators and latent states,
with covariances, for specific subjects from data fit with ctFit,
based on either the mode (if optimized) or mean (if sampled) of parameter distribution.
ctKalman is an alias for this function and returns the same long data
frame. The pre-3.11 name ctStanKalman is not its Stan-prefixed twin:
that is an alias for ctKalmanArray, which returns a list of
arrays instead.
ctPredict( fit, timerange = "asdata", timestep = "auto", subjects = NULL, removeObs = FALSE, plot = FALSE, standardisederrors = FALSE, realid = TRUE, randomEffects = NULL, newdata = NULL, ... )ctPredict( fit, timerange = "asdata", timestep = "auto", subjects = NULL, removeObs = FALSE, plot = FALSE, standardisederrors = FALSE, realid = TRUE, randomEffects = NULL, newdata = NULL, ... )
fit |
fit object as generated by |
timerange |
Either 'asdata' to just use the observed data range, or a numeric vector of length 2 denoting start and end of time range, allowing for estimates outside the range of observed data. Ranges smaller than the observed data are ignored. |
timestep |
Either 'asdata' to just use the observed data (which also requires 'asdata' for timerange) or a positive numeric value indicating the time step to use for interpolating values. Lower values give a more accurate / smooth representation, but take a little more time to calculate. |
subjects |
subject ids to predict for, or NULL (the default) for the first
subject in the data, or every subject in |
removeObs |
Logical or integer. If TRUE, observations (but not covariates) are withheld from the filter, so only expectations based on parameters and covariates are returned (the observations themselves are still reported, for comparison). If a positive integer N, every N observations are retained while others are withheld – useful for observing prediction performance forward further in time than one observation. |
plot |
Logical. If TRUE, plots output instead of returning it.
See |
standardisederrors |
if TRUE, also include standardised error output (based on covariance per time point). |
realid |
If TRUE (the default), subjects are read as the ids in the data, and integers that are not ids there are read as ctsem's numbering of the subjects, 1 to N, with a message. If FALSE, they are read as that numbering. |
randomEffects |
For an With one level of random effects, left in, each subject's predicted and
updated values take its random effects at their posterior given its rows
up to each time, and its smoothed values at their mode given its whole
record. Otherwise every pass takes the modes from each subject's whole
record. Those whole-record modes stay the fit's however this call changes
the rows being filtered (selecting subjects, a time grid,
|
newdata |
Optional long format data to predict for instead of the
data the model was fitted to, with the same columns: an id, a time, the
manifest variables ( |
... |
additional arguments to pass to |
Returns a list containing matrix objects etaprior, etaupd, etasmooth, y, yprior,
yupd, ysmooth, errprior, errupd, errsmooth, time, loglik, with values for each time point in each row.
eta refers to latent states and y to manifest indicators - y itself is thus just
the input data. With standardisederrors=TRUE, standardised versions
errstdprior, errstdupd, errstdsmooth are included too.
Covariance matrices etapriorcov, etaupdcov, etasmoothcov, ypriorcov, yupdcov, ysmoothcov,
are returned in a row * column * time array.
Some outputs are unavailable for ctStan fits at present.
If plot=TRUE, nothing is returned but a plot is generated.
#Basic ctPredict(ctstantestfit, timerange=c(0,60), plot=TRUE) #Multiple subjects, y and yprior, showing plot arguments plot1<-ctPredict(ctstantestfit, timerange=c(0,60), timestep=.1, plot=TRUE, subjects=2:3, kalmanvec=c('y','yprior'), errorvec=c(NA,'ypriorcov')) #'auto' would also have achieved this #modify plot as per normal with ggplot print(plot1+ggplot2::coord_cartesian(xlim=c(0,10))) #or generate custom plot from scratch:#' k=ctPredict(ctstantestfit, timerange=c(0,60), timestep=.1, subjects=2:3) library(ggplot2) ggplot(k[k$Element %in% 'yprior',], aes(x=Time, y=value,colour=Subject,linetype=Row)) + geom_line() + theme_bw() #New subjects: two hypothetical people differing only in TI1, each #observed once at time 0, predicted to time 4. newsubjects <- data.frame(id = c('lowTI1', 'highTI1'), time = 0, Y1 = 8, Y2 = 12, TD1 = 0, TI1 = c(-1, 1)) newsubjects <- rbind(newsubjects, transform(newsubjects, time = 4, Y1 = NA, Y2 = NA)) k <- ctPredict(ctstantestfit, newdata = newsubjects, timerange = c(0, 4), timestep = .1) k[k$Element %in% 'yprior' & k$Time == 4, ] ctPredict(ctstantestfit, newdata = newsubjects, timerange = c(0, 4), timestep = .1, plot = TRUE, kalmanvec = 'yprior', errorvec = 'ypriorcov')#Basic ctPredict(ctstantestfit, timerange=c(0,60), plot=TRUE) #Multiple subjects, y and yprior, showing plot arguments plot1<-ctPredict(ctstantestfit, timerange=c(0,60), timestep=.1, plot=TRUE, subjects=2:3, kalmanvec=c('y','yprior'), errorvec=c(NA,'ypriorcov')) #'auto' would also have achieved this #modify plot as per normal with ggplot print(plot1+ggplot2::coord_cartesian(xlim=c(0,10))) #or generate custom plot from scratch:#' k=ctPredict(ctstantestfit, timerange=c(0,60), timestep=.1, subjects=2:3) library(ggplot2) ggplot(k[k$Element %in% 'yprior',], aes(x=Time, y=value,colour=Subject,linetype=Row)) + geom_line() + theme_bw() #New subjects: two hypothetical people differing only in TI1, each #observed once at time 0, predicted to time 4. newsubjects <- data.frame(id = c('lowTI1', 'highTI1'), time = 0, Y1 = 8, Y2 = 12, TD1 = 0, TI1 = c(-1, 1)) newsubjects <- rbind(newsubjects, transform(newsubjects, time = 4, Y1 = NA, Y2 = NA)) k <- ctPredict(ctstantestfit, newdata = newsubjects, timerange = c(0, 4), timestep = .1) k[k$Element %in% 'yprior' & k$Time == 4, ] ctPredict(ctstantestfit, newdata = newsubjects, timerange = c(0, 4), timestep = .1, plot = TRUE, kalmanvec = 'yprior', errorvec = 'ypriorcov')
Outputs the estimated effect of time independent predictors (covariate moderators) on the expected observations.
ctPredictTIP( sf, tipreds = "all", subject = NULL, timestep = "auto", doDynamics = TRUE, plot = TRUE, quantiles = c(0.16, 0.5, 0.84), discreteTimeQuantiles = c(0.025, 0.5, 0.975), dynamicsControl = list(), showUncertainty = TRUE, TIPvalues = NA, realid = TRUE )ctPredictTIP( sf, tipreds = "all", subject = NULL, timestep = "auto", doDynamics = TRUE, plot = TRUE, quantiles = c(0.16, 0.5, 0.84), discreteTimeQuantiles = c(0.025, 0.5, 0.975), dynamicsControl = list(), showUncertainty = TRUE, TIPvalues = NA, realid = TRUE )
sf |
A fitted object from |
tipreds |
A character vector specifying which time independent predictors to use. Default is 'all', which uses all time independent predictors in the model. |
subject |
The subject for which predictions are made, as an id in the data
(see |
timestep |
A numeric value specifying the time step for predictions. Default is 'auto', which tries to automatically determine an appropriate time step. |
doDynamics |
A logical value indicating whether to plot the effects of time independent predictors on the dynamics of the system. Default is TRUE. Can be problematic for systems with many dimensions. |
plot |
A logical value indicating whether to ggplot the results instead of returning a data.frame of predictions. Default is TRUE. |
quantiles |
A numeric vector specifying the quantiles of the time independent predictors to plot. Default is c(.16,.5,.84): the median, and one standard deviation either side for a normally distributed predictor. |
discreteTimeQuantiles |
a numeric vector of length 3 specifying the quantiles of the discrete time points to plot, when showUncertainty is TRUE. |
dynamicsControl |
A named list of additional arguments to pass to
|
showUncertainty |
A logical value indicating whether to plot the uncertainty of the predictions. Default is TRUE. |
TIPvalues |
An nvalue * nTIpred numeric matrix specifying the fixed values for each time independent predictor effect to plot. Default is NA, which instead relies on the quantiles specified in the quantiles argument. |
realid |
If TRUE (the default), subject is read as an id in the data, and an integer that is not an id there as ctsem's numbering of the subjects, 1 to N, with a message. If FALSE, it is read as that numbering. |
This function estimates the effects of covariate moderators on the expected process and observations for a specified subject in a dynamic system. The covariate moderators are defined at the specified quantiles, and their effects on the trajectory are plotted or returned as a data frame.
If plot is TRUE, a list of ggplot objects showing the estimated effects of covariate moderators. Otherwise, a data frame with the predictions.
# Example usage: ctPredictTIP(ctstantestfit, tipreds='all', doDynamics=FALSE, plot=TRUE)# Example usage: ctPredictTIP(ctstantestfit, tipreds='all', doDynamics=FALSE, plot=TRUE)
Gets internal stan parameter names of a ctStanFit object sampled via stan based on specified substrings.
ctStanParnames is maintained as a backward-compatible alias.
ctRawParnames(x, substrings = c("pop_", "popsd"))ctRawParnames(x, substrings = c("pop_", "popsd"))
x |
ctStanFit object, sampled ( |
substrings |
vector of character strings, parameter names of the stan model containing any of these strings will be returned. Useful strings may be 'pop_' for population means, 'popsd' for population standard deviations, or specific combinations such as 'pop_DRIFT' for the population means of temporal dynamics parameters |
vector of character strings.
sunspots<-sunspot.year sunspots<-sunspots[50: (length(sunspots) - (1988-1924))] id <- 1 time <- 1749:1924 datalong <- cbind(id, time, sunspots) #setup model ssmodel <- ctModel(type='ct', n.latent=2, n.manifest=1, manifestNames='sunspots', latentNames=c('ss_level', 'ss_velocity'), LAMBDA=matrix(c( 1, 'ma1| log(1+(exp(param)))' ), nrow=1, ncol=2), DRIFT=matrix(c(0, 'a21 | -log(1+exp(param))', 1, 'a22'), nrow=2, ncol=2), MANIFESTMEANS=matrix(c('m1|param * 10 + 44'), nrow=1, ncol=1), MANIFESTVAR=diag(0,1), #As per original spec CINT=matrix(c(0, 0), nrow=2, ncol=1), DIFFUSION=matrix(c(0, 0, 0, "diffusion"), ncol=2, nrow=2)) #fit ssfit <- ctFit(datalong, ssmodel, optimize=FALSE, priors=TRUE, sampleControl=list(iter=2, chains=1), backend='stan') ctRawParnames(ssfit,substrings=c('pop_','popsd'))sunspots<-sunspot.year sunspots<-sunspots[50: (length(sunspots) - (1988-1924))] id <- 1 time <- 1749:1924 datalong <- cbind(id, time, sunspots) #setup model ssmodel <- ctModel(type='ct', n.latent=2, n.manifest=1, manifestNames='sunspots', latentNames=c('ss_level', 'ss_velocity'), LAMBDA=matrix(c( 1, 'ma1| log(1+(exp(param)))' ), nrow=1, ncol=2), DRIFT=matrix(c(0, 'a21 | -log(1+exp(param))', 1, 'a22'), nrow=2, ncol=2), MANIFESTMEANS=matrix(c('m1|param * 10 + 44'), nrow=1, ncol=1), MANIFESTVAR=diag(0,1), #As per original spec CINT=matrix(c(0, 0), nrow=2, ncol=1), DIFFUSION=matrix(c(0, 0, 0, "diffusion"), ncol=2, nrow=2)) #fit ssfit <- ctFit(datalong, ssmodel, optimize=FALSE, priors=TRUE, sampleControl=list(iter=2, chains=1), backend='stan') ctRawParnames(ssfit,substrings=c('pop_','popsd'))
Runs the package's reporting and fit-checking functions over a fitted model and writes the results into one folder, with a markdown index that says what each file is and how to read it. Works for either backend; components a backend or a model does not support are recorded in the index as not produced, rather than stopping the run.
ctReport( fit, folder = "ctReport", components = "default", nsamples = 200, cores = getOption("mc.cores", 2L), quantiles = c(0.025, 0.5, 0.975), times = "auto", timeinterval = 1, nsubjects = 6, lags = 0:3, folds = 10, profilepoints = 9, verbose = TRUE )ctReport( fit, folder = "ctReport", components = "default", nsamples = 200, cores = getOption("mc.cores", 2L), quantiles = c(0.025, 0.5, 0.975), times = "auto", timeinterval = 1, nsubjects = 6, lags = 0:3, folds = 10, profilepoints = 9, verbose = TRUE )
fit |
A fit from |
folder |
Directory to write into; created if it does not exist. Files
from a previous |
components |
Either one of |
nsamples |
Draws used by the components that sample from the fit's
uncertainty, and by data generation for |
cores |
Cores for the components that can use more than one
( |
quantiles |
Quantiles for the parameter matrices and the discrete parameter plots. |
times |
Interval grid for |
timeinterval |
Time interval at which the networks, the discrete-time parameter matrices and the covariate effects are reported. |
nsubjects |
Number of subjects plotted by |
lags |
Lags for the lagged covariance check. |
folds |
Folds for |
profilepoints |
Grid points per parameter in the likelihood profile. |
verbose |
Report each component as it runs. |
The expensive parts are adequacy, covcheck and
postpred, which generate data from the fit (once, shared),
scoretests, which evaluates the model extended with new terms, and
crossval, which refits. components = 'quick' excludes them
all.
profile walks the log probability along each raw parameter, holding
the others at the estimate, over a grid measured in the standard error that
parameter's own curvature implies. It flags parameters whose slice peaks away
from the estimate (the optimizer did not arrive), whose surface is not the
shape a standard error assumes, and whose reported interval is much wider
than their own curvature (they sit on a ridge with other parameters). It is a
conditional slice, not a profile likelihood, which is why it is cheap enough
to run by default; identification names the ridge directions.
Invisibly, a list with the folder and, per component, whether it was written, skipped or failed. Called for the files it writes.
ctIdentify before
fitting; fit$identifiability and ctFitProfile after.
print(fit) says
whether it converged, summary(fit) adds R-hat and effective sample
sizes for a sampled fit, and ctTracePlot draws the
optimiser's trace or each chain's draws.
ctPostPredPlots, and ctFitCheckCov for the
lagged covariances, split by a group or predictor if wanted.
ctFitAdequacy tests them by family, allowing for the
estimation; ctResiduals and ctACFresiduals
show them.
ctScoreTest,
without refitting; ctModelProposals lists the changes and
ctModelImprove refits them.
anova for nested models, ctLOO for
out-of-sample prediction.
ctLaplaceCheck for the Laplace approximation of random
effects, ctParticleLik for the filter's Gaussian
approximation of the states.
ctCoverageCheck, by simulation.
f <- ctReport(ctstantestfit, folder = file.path(tempdir(), 'ctReport'), components = c('summary', 'identification', 'profile'), profilepoints = 5) list.files(f$folder)f <- ctReport(ctstantestfit, folder = file.path(tempdir(), 'ctReport'), components = c('summary', 'identification', 'profile'), profilepoints = 5) list.files(f$folder)
This function takes a fit object from the ctsem package and extracts the standardized residuals.
ctResiduals(fit)ctResiduals(fit)
fit |
A fitted model object generated by the ctsem package. |
This function uses the ctKalmanArray function to calculate the standardized residuals
and then extracts and formats them as a data table. The standardized residuals represent the differences
between the observed and predicted values, divided by the standard errors of the observations.
A data table containing the standardized residuals for each subject and time point, with subjects labelled by their ids in the data.
data.table::setDTthreads(1) #ignore this line # Example usage: residuals <- ctResiduals(ctstantestfit)data.table::setDTthreads(1) #ignore this line # Example usage: residuals <- ctResiduals(ctstantestfit)
Tests, from one fitted model and without refitting it, whether the model would fit better with a relaxation it does not contain: parameters that differ between subjects (an unmodelled random effect), parameters that depend on a time independent covariate the model does not include, or dynamics that depend on the latent state (a missing nonlinearity). Each test is a score (Lagrange multiplier) test: the slope of the log likelihood in the direction of the relaxation, at the fitted model, with the fitted parameters projected out. Each also gives a one-step estimate of the relaxation's size, for screening: refit with the relaxations that matter before interpreting their sizes.
ctScoreTest( fit, covariates = NULL, tests = c("heterogeneity", "covariates", "statedependence", "volatility", "fixed", "level", "instability"), form = c("auto", "linear", "spline", "factor", "threshold"), df = 3, parameters = NULL, order = 2L, variance = c("robust", "model"), nboot = 1999L, exact = 5L, intoverpop = c("laplace", "quadrature"), cores = getOption("mc.cores", 2L), verbose = TRUE )ctScoreTest( fit, covariates = NULL, tests = c("heterogeneity", "covariates", "statedependence", "volatility", "fixed", "level", "instability"), form = c("auto", "linear", "spline", "factor", "threshold"), df = 3, parameters = NULL, order = 2L, variance = c("robust", "model"), nboot = 1999L, exact = 5L, intoverpop = c("laplace", "quadrature"), cores = getOption("mc.cores", 2L), verbose = TRUE )
fit |
A fit from |
covariates |
Optional data frame of subject-level covariates to test:
a column named as the fit's subject id (or |
tests |
Which families to run: any of |
form |
How a covariate enters: |
df |
Degrees of freedom of the spline basis. |
parameters |
Optional parameter names to restrict the heterogeneity and covariate tests to. |
order |
Highest degree of the state-dependence terms: 2, each drift coefficient varying linearly with each state; 3 adds variation with products and squares of states, which is what a symmetric dependence – regulation that strengthens at both extremes, say – shows up as, and which degree 2 cannot see. |
variance |
|
nboot |
Bootstrap resamples for the robust p-values. |
exact |
For |
intoverpop |
How the extended families ( |
cores |
Engine cores for the evaluations. |
verbose |
Report progress. |
An object of class ctScoreTest: a list with
heterogeneity (a data frame, one row per parameter or fixed
MANIFESTMEANS cell),
dimension (the largest eigenvalue, its p-value and the direction,
when two or more parameters are tested),
covariates (one element per covariate, each a list of its
joint test and a data frame parameters),
statedependence (a list of terms, a data frame, and
joint, the tests per equation and overall), volatility
(the same, for the innovation sds), fixed (a list
of cells and their joint test), instability (a list
of parameters, one row per ordering and parameter, and
joint, per ordering), and info (the
route, the number of subjects and parameters, the variance used, and
errors, why any family was not tested). Tables carry class
where relaxations are alike, and the extended families' projected,
whether a term's estimation was removed exactly (see exact).
For each parameter that is free, does not already
vary over subjects, and is not an initial condition (T0MEANS and T0VAR,
whose variation T0VAR already carries), the score test of letting it vary
as ctFit would fit it: its variance over subjects together with
its covariances with what already varies – each subject's initial state
(the free T0MEANS parameters) and the parameters with random
effects, named in with. Every subject has an initial state, so
even a model without random effects has those covariances to test. They
matter: an intercept in CINT that varies while the initial means
are shared makes subjects drift apart from a common start, which data
whose subjects differ stably from the first occasion reject, whereas the
same intercept correlated with where each subject starts is a random
intercept. The covariances are tested as ordinary score tests and the
variance, given them, one sided (Stram & Lee, 1994): LM is
referred to an equal mixture of chi-square on df - 1 and
df. Where the new effect is independent of what varies, those
covariances only cost power, so p is the stronger of that test
and of the variance alone (z, one sided, its covariances held at
zero), by their smaller p-value, calibrated by resampling subjects. A
negative z beside a large LM says the parameter varies only
together with what already varies, as an intercept does with where each
subject starts. Nothing is assumed about how the initial states relate
to the dynamics: neither that subjects start at their own equilibrium
nor that they share a start. The statistic is the information matrix
test in those coordinates, which is the score test for neglected
heterogeneity (Chesher, 1984). It responds to any failure of the
information equality there, of which subject differences are the usual
cause but not the only one: a subject-varying drift also moves the
diffusion's test, because both change how much a subject varies, and a
missing nonlinearity moves several. sd is the one-step estimate of
the subjects' standard deviation on the raw scale, with the covariances
estimated beside it, and sd_natural approximately on the
parameter's own; NA where the covariances carry the test and the
variance given them reads zero, which happens for an intercept, and
which only a refit can size. To act on a result, set indvarying = TRUE for the
parameter and the T0MEANS parameters, refit, and test again;
ctModelImprove does that.
Each MANIFESTMEANS cell the model fixes is tested too, as a
random intercept on that indicator about its fixed value (rows named
MANIFESTMEANS[...]), from the model with those cells written as
parameters. With one indicator per latent such an intercept is a
direction the CINT rows' covariances also reach; with more
indicators it is the indicator's own; for a latent with no free
intercept at all it is the only test of its level. To act on it, give
the cell a parameter with the fixed value as its stated mean
(ctPar(mean=)) and indvarying = TRUE.
On a reduced-rank level (poprank), a parameter that varies
only through the shared dimensions, without a variance of its own, is
tested too: for variation outside those dimensions. There the shared
dimensions carry the covariances, and each test is of the variance
alone. A parameter whose test the fitted model already determines –
under one per cent of its spread left once estimation is taken out, as
for an intercept that loads on the shared dimensions – is not tested
(NA) and is left out of dimension.
dimension tests every direction at once: whether subjects vary
along some combination of the tested parameters, a new dimension of
person variation, its covariances with what already varies held at
zero. It is not computed on a reduced-rank level, where in
simulation it found a dimension the rank lacked 10 per cent of the time
and the per-parameter tests of variation outside the shared dimensions 58
per cent; those are the test there. Adding
variance along a direction u changes a subject's log likelihood,
at zero, by (u's_i)^2/2 + u'H_iu/2, so the variances are
the diagonal of one score matrix; its largest eigenvalue, scaled by each
diagonal's spread, is the statistic, against a bootstrap over subjects,
and its eigenvector is the direction, read as relative loadings (raw
scale). It can detect variation spread over parameters that no single
test singles out, and is less sensitive than the best single test when
one parameter carries it.
For each covariate, the effects it would have on every
tested parameter if added as a time independent predictor (on the raw
scale, as a predictor effect acts), including on the population standard
deviations of the parameters that vary: a covariate moderating
heterogeneity. The joint test of all of a covariate's effects is the
screen. Per parameter, estimate is the one-step effect per unit of
the covariate with the fitted parameters held at their estimates, jointly
over the parameters, so a covariate that shifts one parameter is not read
off another correlated with it; estimate_natural puts it
approximately on the parameter's own scale, and p tests it. Holding
the fit's parameters is what keeps the size near a refit's: a covariate
the model lacks is absorbed by the parameters it has – a random effect's
standard deviation grows to take in a covariate that shifts its mean –
and letting them move in a one-step extrapolates the likelihood across
that whole change. A covariate enters linearly (centred), as treatment
indicators for a factor, or as a natural spline with df degrees of
freedom, for which only p is given; subjects missing it contribute
at the mean. To act on a result, add the covariate to the data and to
TIpredNames and refit.
form = 'threshold' tests a step at an unknown cut instead: the
joint test at each cut from the 15th to the 85th percentile, and the
largest of them (Andrews, 1993) – the split a score-guided tree makes –
against the same scan with the covariate permuted over subjects (Hothorn
& Lausen, 2003), which takes the search into account exactly and refits
nothing. The effects reported are those of being above the best cut,
cut in the joint row.
On a reduced-rank level the spread of a varying parameter has no
popsd_ coordinate, so a covariate's effect on it is tested along
the direction that scales that parameter's row of the loadings and its own
sd together, reported as scale(<parameter>): the effect on its log
standard deviation per unit of the covariate, the correlations held. In
simulation (rank 1) these rows rejected 6.5 to 8 per cent of correct
models at .05 with 100 subjects and 5 to 6 with 300, where the
covariate's joint test held its level: with few subjects the projection
on the fitted parameters leaves a variance a little too small. Read them
as attribution once the joint test has screened.
For each latent equation and each pair of
latent states k <= m, the score test of a term d * eta_k *
eta_m in that equation's drift: a drift coefficient that varies linearly
with a state, the second-order expansion of a drift that depends on the
state (Luukkonen, Saikkonen & Terasvirta, 1988). A term eta_k *
eta_m in the equation of eta_i is DRIFT[i,k] varying with
eta_m – and, when k != m, equally DRIFT[i,m] varying
with eta_k; at this order the data cannot tell those apart, and
reads says both. estimate is the one-step change in that
drift coefficient per unit of the state. It is NA when the fitted model's
curvature cannot size the term, which happens when the departure is
strong; the test itself is unaffected. order = 3 adds the terms
of degree three. The terms are evaluated by the model's own filter at the
state predicted from earlier observations, which keeps the test valid; a
state estimate that has seen the observation being scored – the updated
or smoothed state – would not. To act on a result, write the dependence
into the drift cell (DRIFT[1,1] = 'dr11 + d * eta2' with
PARS = c('dr11|-log1p_exp(-param)', 'd')) and refit.
tests = 'volatility' asks whether the size of
each latent's innovations depends on a state: the score test of
DIFFUSION[i, i], the innovation sd, scaled by exp(v *
eta_m), at v = 0, for every latent with innovations and every
state. Variance that rises with a process's own level, as when negative
affect is more variable when high, is the diagonal term. Evaluated by the
filter at the predicted state, as for state dependence. To act on a
result, write the scaling into the cell (DIFFUSION[2,2] = 'sd2 *
exp(v * eta2)' with PARS = c('sd2|log1p_exp(param)', 'v')) and
refit; ctModelImprove does that. The terms make the model
nonlinear, so the test, and a refit, run the extended Kalman filter.
tests = 'level' asks whether a parameter of the
dynamics or of the error variances depends on a subject's level, where
that level is a random intercept: for each free DRIFT,
DIFFUSION and MANIFESTVAR parameter that does not vary over
subjects, and each MANIFESTMEANS or CINT intercept that
does, the score test of a term g * intercept added to the
parameter, at g = 0. When subjects' levels sit in the intercepts,
the latent states are deviations from them, so the state dependence and
volatility tests, which read the states, see dependence on the deviation
only; a parameter that depends on where a subject sits – persistence or
innovation size that differs between people who are usually high and
people who are usually low – shows there only as heterogeneity, in no
particular direction. These terms ask about that direction. The score
is taken over each subject's posterior for the intercept, from the
extended model's own gradient, which the product of a subject's score
and estimated level is not. estimate is the one-step change in the
parameter, on its own scale, per unit of the intercept. To act on a
result, let the parameter vary over subjects, so its random effect is
estimated with its correlation with the intercept;
ctModelImprove does that. A model with no random intercept
has nothing to test.
For each cell the model fixes at zero that could
sensibly be freed – a DRIFT cross effect, a DIFFUSION or
MANIFESTVAR covariance between two that have a variance, a
LAMBDA cross loading, a
TDPREDEFFECT – the score test of freeing it: the modification
index (Sorbom, 1989), with estimate the expected parameter change.
CINT and MANIFESTMEANS are not offered, since with the
other free either is identified only through the initial transient.
State dependence, volatility and fixed cells are tested on one extended
model, so asking for several costs one preparation.
tests = 'instability', not run unless asked
for, tests whether each parameter is stable across the data: ordered by
time, and by each latent's state as predicted from earlier observations.
It uses each data row's score – the derivative of its one-step log
likelihood – whose partial sums, under the model, behave as a Brownian
bridge in any order fixed in advance (Nyblom, 1989). A parameter that
changes over the study, or differs at high and low levels of a state,
bends them. Ordered by a state, that is a test of state dependence of
unknown form, beside the polynomial terms of statedependence. The
estimation effect is removed with the information accumulated along the
ordering, and the p-values are from a wild bootstrap over rows. It costs a
dual pass of the filter per eight parameters, and on the Laplace route one
per truncated record of every subject.
Two relaxations whose per-subject projected
contributions are nearly collinear (correlation of 0.9 or more) would
each explain the same signal, and the data cannot tell them apart at this
order (MacCallum et al., 1993). They share a letter in class;
NA means alike to none. Refit with the most plausible, then test
again.
The fitted model's curvature projects the
fitted parameters out of each test and sets the one-step sizes; the
variance of each test is, by default (variance = 'robust'), the
spread over subjects of their projected contributions, and its p-value a
bootstrap over subjects of the studentised statistic (nboot
resamples, from a fixed stream, so a call is reproducible and leaves the
session's random numbers alone). That keeps each test valid when the model
is wrong in some other way, which is when a screen is used, and corrects
for the skewness of a variance score. variance = 'model' uses the
curvature for the variance and chi-square and normal references instead;
it is what is used below 20 subjects, where a spread over subjects is too
thin to estimate.
The tests are screens. A badly misspecified
model lights up many at once, and their one-step sizes are local
approximations – especially for drift parameters, which act through the
matrix exponential. Relax what stands out most, refit, and test again.
Within each family p_adj is the Benjamini-Hochberg adjustment over
the tests shown. The tests use the fit's own objective, so they apply to
the route it was fitted on (intoverpop); they are made at an
optimum, and refuse a sampled fit. Heterogeneity and covariate tests need
subjects as the independent units, so a model with a level above the
subject can only be tested for state dependence. A family that cannot be
tested on a model says why, as a warning and in info$errors, and
the other families' results stand. With priors = TRUE the new terms
of the extended model would carry the normal(0, 1) prior every parameter
has; they are tested on the likelihood, without it.
On the Laplace route the filter integrates a fit's intercept-type random
effects exactly where it can, and Laplace approximates only the others
(the two-step term; optimcontrol$laplace_onestep in
ctFit). The state dependence and volatility families extend
the model with terms that read the state, which that filter cannot take;
where the fit has random dynamics beside such effects, those families are
then computed with the fit and its extension both on the one-step term, a
fast screen whose approximation error stays in the scores, and
info$onestep names them. intoverpop = 'quadrature'
integrates every subject's random effects for these families instead,
leaving no Laplace error in them (info$quadrature).
Boos, D. D. (1992). On generalized score tests. The American Statistician, 46, 327-333.
Chesher, A. (1984). Testing for neglected heterogeneity. Econometrica, 52, 865-872.
Stram, D. O., & Lee, J. W. (1994). Variance components testing in the longitudinal mixed effects model. Biometrics, 50, 1171-1177.
Self, S. G., & Liang, K.-Y. (1987). Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. Journal of the American Statistical Association, 82, 605-610.
Hothorn, T., & Lausen, B. (2003). On the exact distribution of maximally selected rank statistics. Computational Statistics & Data Analysis, 43, 121-137.
Andrews, D. W. K. (1993). Tests for parameter instability and structural change with unknown change point. Econometrica, 61, 821-856.
Nyblom, J. (1989). Testing for the constancy of parameters over time. Journal of the American Statistical Association, 84, 223-230.
Hall, D. B., & Praestgaard, J. T. (2001). Order-restricted score tests for homogeneity in generalised linear and nonlinear mixed models. Biometrika, 88, 739-751.
MacCallum, R. C., Wegener, D. T., Uchino, B. N., & Fabrigar, L. R. (1993). The problem of equivalent models in applications of covariance structure analysis. Psychological Bulletin, 114, 185-199.
Luukkonen, R., Saikkonen, P., & Terasvirta, T. (1988). Testing linearity against smooth transition autoregressive models. Biometrika, 75, 491-499.
Sorbom, D. (1989). Model modification. Psychometrika, 54, 371-384.
vignette('score-tests') for three worked examples.
## Not run: fit <- ctFit(dat, model, backend = 'julia') st <- ctScoreTest(fit, covariates = subjectdata) st st$heterogeneity ## End(Not run)## Not run: fit <- ctFit(dat, model, backend = 'julia') st <- ctScoreTest(fit, covariates = subjectdata) st st$heterogeneity ## End(Not run)
Settings that change how ctsem behaves across a session, rather than
per-call. Set them with options(), typically once at the top of a
script or in a front end's startup.
Nothing; this page documents options rather than defining a function.
ctsem.progress.overwriteWhether progress reporting rewrites
a single line with a carriage return (TRUE) or prints occasional
separate lines (FALSE). Unset, ctsem detects it: an active Shiny
session, a sink(), or a knitr chunk all capture output, where a
carriage return is a character rather than a cursor movement. Set it
explicitly if that detection is wrong for your front end.
ctsem.cluster.outfileWhere parallel workers send their
output. Unset, they are silent. Setting it to "" forwards worker
stdout and stderr to the console, which is useful only when debugging a
worker – it also restores the startup chatter that made a two-core fit
announce itself six times.
Progress and reporting are also controlled per fit, and those win over these:
verbose in ctFit (0 quiet, 1 reports progress, 2 keeps
every line rather than overwriting), optimcontrol$callback for a
function called while the fit runs, and optimcontrol$progress to
force progress on or off for one fit.
The Julia backend is mostly located through the environment rather than
options: JULIA_BINDIR names the binary directory,
JULIA_NUM_THREADS the thread count of the session that starts next,
and ctJuliaSetup(project=) points the session at a different engine
environment. See ctJuliaSetup.
ctsem.julia.restartWhether a fit asking for more
cores than the running Julia session has threads may restart the
session to get them. FALSE by default: Julia fixes its thread
count at process start, so the only way to raise it is to end the
process, which discards the engine's compiled model shapes and costs
seconds. Left FALSE, such a fit runs at the threads it has and
says so.
Dummy fit for testing functions from ctsem package.
ctStanFit object
Plot every context-dependent matrix cell against one latent process, across the range that process occupies, with the posterior spread at each point.
This is the plot that says what a state-dependent model actually claims.
summary() reports each cell as a single number at a single state;
here the same cell is drawn as the function of the state it really is, and
the summary's number is one point on that curve.
ctStateDependencePlot( fit, along = 1, extent = "data", gridsize = 25, nsamples = 20, quantiles = c(0.025, 0.5, 0.975), state = "asymptotic", tipreds = NULL, tolerance = 1e-06, plot = TRUE )ctStateDependencePlot( fit, along = 1, extent = "data", gridsize = 25, nsamples = 20, quantiles = c(0.025, 0.5, 0.975), state = "asymptotic", tipreds = NULL, tolerance = 1e-06, plot = TRUE )
fit |
A ctJuliaFit. |
along |
The latent process to vary, by name or index. |
extent |
|
gridsize |
Points along the range. One engine call each. |
nsamples |
Posterior draws for the interval, or 1 for the point estimate alone. |
quantiles |
Three quantiles: lower ribbon, line, upper ribbon. |
state |
Where the other latent processes are held: |
tipreds |
Time independent predictor values, or NULL for the population. |
tolerance |
A cell is treated as varying when its range across the grid exceeds this fraction of its own magnitude. |
plot |
If FALSE, return the values as a data frame rather than a plot. |
Cells are chosen by measuring rather than by parsing: every model matrix is materialised across the grid, and a cell is kept when its value actually moves. That catches dependence arriving indirectly – through a PARS entry, or through a covariance built from a state-dependent cell – without having to trace it symbolically.
A model with no state-dependent cells has nothing to draw, and says so rather than returning an empty plot.
A ggplot, or a data frame when plot=FALSE.
ctContextDependence, ctPhasePortrait
# Needs a working julia backend, so the example is inert where Julia is # absent -- including on CRAN, whose check machines have none. if (isTRUE(ctJuliaStatus()$available)) { set.seed(1) generating <- suppressMessages(ctModel(type = 'ct', n.latent = 2, n.manifest = 2, manifestNames = c('Y1', 'Y2'), latentNames = c('eta1', 'eta2'), LAMBDA = diag(2), DRIFT = matrix(c(-.4, .1, 0, -.3), 2, 2), CINT = matrix(c(.2, .1), 2, 1), MANIFESTMEANS = matrix(0, 2, 1), MANIFESTVAR = diag(.2, 2), DIFFUSION = matrix(c(.5, 0, 0, .4), 2, 2))) datalong <- as.data.frame(suppressMessages(ctGenerate(generating, n = 25, burnin = 5, dtmean = 1, logdtsd = .1, wide = FALSE, Tpoints = 12))) # eta1's own decay depends on where eta2 is -- a state-dependent DRIFT cell. model <- suppressMessages(ctModel(type = 'ct', n.latent = 2, n.manifest = 2, manifestNames = c('Y1', 'Y2'), latentNames = c('eta1', 'eta2'), LAMBDA = diag(2), PARS = c('dr11|-log1p_exp(param)'), DRIFT = matrix(c('dr11 * (1 + 0.2 * eta2)', 'd21', 0, 'd22'), 2, 2), CINT = matrix(c('c1', 'c2'), 2, 1), MANIFESTMEANS = matrix(0, 2, 1), MANIFESTVAR = diag(.2, 2), DIFFUSION = matrix(c('df1', 0, 0, 'df2'), 2, 2))) model$pars$indvarying <- FALSE fit <- ctFit(datalong, model, backend = 'julia', cores = 1, verbose = 0) ctStateDependencePlot(fit, along = 'eta2', gridsize = 11, nsamples = 5) # Julia keeps running between calls, holding what it has compiled; # this ends it. ctJuliaKill("session") }# Needs a working julia backend, so the example is inert where Julia is # absent -- including on CRAN, whose check machines have none. if (isTRUE(ctJuliaStatus()$available)) { set.seed(1) generating <- suppressMessages(ctModel(type = 'ct', n.latent = 2, n.manifest = 2, manifestNames = c('Y1', 'Y2'), latentNames = c('eta1', 'eta2'), LAMBDA = diag(2), DRIFT = matrix(c(-.4, .1, 0, -.3), 2, 2), CINT = matrix(c(.2, .1), 2, 1), MANIFESTMEANS = matrix(0, 2, 1), MANIFESTVAR = diag(.2, 2), DIFFUSION = matrix(c(.5, 0, 0, .4), 2, 2))) datalong <- as.data.frame(suppressMessages(ctGenerate(generating, n = 25, burnin = 5, dtmean = 1, logdtsd = .1, wide = FALSE, Tpoints = 12))) # eta1's own decay depends on where eta2 is -- a state-dependent DRIFT cell. model <- suppressMessages(ctModel(type = 'ct', n.latent = 2, n.manifest = 2, manifestNames = c('Y1', 'Y2'), latentNames = c('eta1', 'eta2'), LAMBDA = diag(2), PARS = c('dr11|-log1p_exp(param)'), DRIFT = matrix(c('dr11 * (1 + 0.2 * eta2)', 'd21', 0, 'd22'), 2, 2), CINT = matrix(c('c1', 'c2'), 2, 1), MANIFESTMEANS = matrix(0, 2, 1), MANIFESTVAR = diag(.2, 2), DIFFUSION = matrix(c('df1', 0, 0, 'df2'), 2, 2))) model$pars$indvarying <- FALSE fit <- ctFit(datalong, model, backend = 'julia', cores = 1, verbose = 0) ctStateDependencePlot(fit, along = 'eta2', gridsize = 11, nsamples = 5) # Julia keeps running between calls, holding what it has compiled; # this ends it. ctJuliaKill("session") }
Extract an array of subject specific parameters from a fit from
ctFit, on either backend.
ctStanSubjectPars is maintained as a backward-compatible alias.
ctSubjectPars( fit, pointest = TRUE, cores = getOption("mc.cores", 2L), nsamples = "all", level = NULL, uncertainty = NULL )ctSubjectPars( fit, pointest = TRUE, cores = getOption("mc.cores", 2L), nsamples = "all", level = NULL, uncertainty = NULL )
fit |
A fit from |
pointest |
if TRUE, returns only the set of individual difference parameters based on the max a posteriori estimate (or the median if sampling approaches were used). |
cores |
Number of cores to use. |
nsamples |
Number of samples to calculate parameters for. Not used if pointest=TRUE. |
level |
For a model with nested ids ( |
uncertainty |
For |
This function returns the estimates of individual parameters, taking into account any covariates and random effects, at every level of a nested model.
A julia fit also stores, for every level with random effects, each group's
conditional posterior of its effects on the raw parameter scale –
fit$randomeffects, one entry per level with the groups' mean
deviation from the population value and their covariance packed column by
column (cov). The draws with uncertainty='total' are built from
the same posterior.
an nsamples by nsubjects by nparams array, with the subjects (or
groups, for a grouping level) labelled by their ids in the data.
indpars <- ctSubjectPars(ctstantestfit) dimnames(indpars) plot(indpars[1,,'cint1'],indpars[1,,'cint2'])indpars <- ctSubjectPars(ctstantestfit) dimnames(indpars) plot(indpars[1,,'cint1'],indpars[1,,'cint2'])
Summarise model-implied parameter matrices from a ctsem fit object.
ctStanContinuousPars is maintained as a backward-compatible alias.
ctSummaryMatrices( fit, calcfunc = quantile, calcfuncargs = list(probs = 0.5), timeinterval = 1, state = NULL, ... )ctSummaryMatrices( fit, calcfunc = quantile, calcfuncargs = list(probs = 0.5), timeinterval = 1, state = NULL, ... )
fit |
fit object from |
calcfunc |
Function to apply over samples, must return a single value.
By default the median over all samples is returned using the |
calcfuncargs |
A list of additional parameters to pass to calcfunc. For instance, with the default of calcfunc = quantile, the probs argument is needed to ensure only a single value is returned. |
timeinterval |
time interval for discrete time parameter matrix computation. |
state |
The latent state at which state dependent matrix cells are
evaluated: |
... |
arguments passed to the method for the fit's backend. |
A named list of matrices, each summarised over the posterior (or
approximate posterior) draws by calcfunc: the model's own
(DRIFT, DIFFUSION, LAMBDA, ...), the covariances
built from them (DIFFUSIONcov, MANIFESTcov, and T0cov,
the initial latent covariance over subjects: T0VAR plus the
population covariance of any individually varying T0MEANS), the
asymptotic asymCINT and asymDIFFUSIONcov, and
dtDRIFT at timeinterval. Beside each covariance its
correlations, DIFFUSIONcor, MANIFESTcor, T0cor and
asymDIFFUSIONcor, taken draw by draw before calcfunc
summarises them; a variable whose variance is zero has NA there. On backend='julia', also the
Jacobians of the drift and of the measurement at the evaluation point,
JAx (latent by latent) and Jy (manifest by latent): the local
effects of the model linearised at state. They equal DRIFT and
LAMBDA for a linear model; where a cell is nonlinear in the state they
do not, since a DRIFT cell a + q*Y in the column of process
Y holds a + q*Y there and contributes a + 2*q*Y to
JAx. dtDRIFT is expm(JAx * timeinterval) where there is
a JAx, the transition ctDiscretePars reports, and
expm(DRIFT * timeinterval) on stan. For a model whose dynamics depend
on the state it holds JAx fixed over the interval: a fair interval
regression near a fixed point or over a short interval, and away from a fixed
point one ctDiscretePars(method = 'simulate') should check. Where parameters vary over subjects,
also their population covariance on the natural scale, popcov, and
their raw-scale correlations, rawpopcorr, named by parameter (with
several grouping levels, popcov.<level> and rawpopcorr.<level>).
popcov combines the sds summary() reports as popsd with
those correlations, so it is exact where a parameter's transform is affine,
as for intercepts and initial means.
On an optimised fit, estimate holds the same matrices at the fit's
estimate, unrounded: ctSummaryMatrices(fit)$estimate$DRIFT is the
estimated drift matrix, which a summary of the draws need not equal, and
ctSummaryMatrices(fit, state = s)$estimate$JAx the model linearised
at s at the estimate. A sampled fit has no estimate, and a
backend='stan' fit's has no popcov, which stan simulates.
#posterior median over all subjects (also reflects mean of unconstrained pars) ctSummaryMatrices(ctstantestfit)#posterior median over all subjects (also reflects mean of unconstrained pars) ctSummaryMatrices(ctstantestfit)
Computes and plots combined effects and quantiles for effects of time independent predictors
on subject level parameters of a ctStanFit object.
ctStanTIpredeffects is maintained as a backward-compatible alias.
ctTIpredEffects( fit, returndifference = FALSE, probs = c(0.025, 0.5, 0.975), includeMeanUncertainty = FALSE, whichTIpreds = 1, parmatrices = TRUE, whichpars = "all", nsamples = 100, timeinterval = 1, nsubjects = 20, filter = NA, plot = FALSE )ctTIpredEffects( fit, returndifference = FALSE, probs = c(0.025, 0.5, 0.975), includeMeanUncertainty = FALSE, whichTIpreds = 1, parmatrices = TRUE, whichpars = "all", nsamples = 100, timeinterval = 1, nsubjects = 20, filter = NA, plot = FALSE )
fit |
fit object from |
returndifference |
logical. If FALSE, absolute parameter values are returned.
If TRUE, only the effect of the covariate (i.e. without the average value of the parameter)
are returned. The former can be easier to interpret, but the latter are more likely to fit multiple plots together.
Not used if |
probs |
numeric vector of quantile probabilities from 0 to 1. Specify 3 values if plotting, the 2nd will be drawn as a line with uncertainty polygon based on 1st and 3rd. |
includeMeanUncertainty |
if TRUE, output includes sampling variation in the mean parameters. If FALSE, mean parameters are fixed at their median, only uncertainty in time independent predictor effects is included. |
whichTIpreds |
Which of the model's time independent predictors to
calculate effects for: their names, as given in |
parmatrices |
Logical. If TRUE (default), system matrices rather than specific parameters are referenced – e.g. 'DRIFT' instead of a parameter name like drift12. |
whichpars |
if parmatrices==TRUE, character vector specifying which matrices, and potentially which
indices of the matrices, to plot. c('dtDRIFT[2,1]', 'DRIFT') would output for row 2 and column 1 of
the discrete time drift matrix, as well as all indices of the continuous time drift matrix.
If parmatrices==FALSE, integer vector specifying which of the subject
level parameters to compute effects on. The integers corresponding to certain parameters can be found in the
|
nsamples |
Positive integer specifying the maximum number of saved iterations to use. Character string 'all' can also be used. |
timeinterval |
positive numeric indicating time interval to use for discrete time parameter matrices,
if |
nsubjects |
Positive integer specifying the number of subjects to compute values for. When only one TIpred is used, this specifies the number of points along the curve. Character string 'all' can also be used. Time taken for plotting is a function of nsubjects*niterations. |
filter |
either NA, or a length 2 vector, where the first element contains the time independent predictor index to filter by, and the second contains the comparison operator in string form (e.g. "< 3", to only calculate effects for subjects where the tipreds of the denoted index are less than 3). |
plot |
Logical. If TRUE, nothing is returned but instead |
Either a three dimensional array of predictor effects, or nothing with a plot generated.
ctTIpredEffects(ctstantestfit, whichpars=c('CINT','dtDIFFUSION[2,2]'), plot=TRUE)ctTIpredEffects(ctstantestfit, whichpars=c('CINT','dtDIFFUSION[2,2]'), plot=TRUE)
For an optimized fit, draws the log posterior and the gradient norm against
iteration, from the record the engine kept while fitting. For a sampled fit,
draws each chain's draws of every population mean and standard deviation,
on the scale summary() reports them on, so chains that have not
settled on one distribution can be seen.
ctTracePlot(fit, which = NULL, plot = TRUE)ctTracePlot(fit, which = NULL, plot = TRUE)
fit |
A fit from |
which |
Which panels to draw: trace columns ( |
plot |
If |
The Julia backend records the optimisation trace during the fit and
returns it as fit$optim$trace. For output while a fit is still
running, see optimcontrol$callback in ctFit. The
chains of a sampled stan fit are drawn by plot(fit, types =
'trace').
A ggplot, or with plot = FALSE a data frame.
# Needs a working julia backend, so the example is inert where Julia is # absent -- including on CRAN, whose check machines have none. if (isTRUE(ctJuliaStatus()$available)) { model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1)) model$pars$indvarying <- FALSE fit <- ctFit(ctstantestdat, model, backend = 'julia', cores = 1) ctTracePlot(fit) # Julia keeps running between calls, holding what it has compiled; # this ends it. ctJuliaKill("session") }# Needs a working julia backend, so the example is inert where Julia is # absent -- including on CRAN, whose check machines have none. if (isTRUE(ctJuliaStatus()$available)) { model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1)) model$pars$indvarying <- FALSE fit <- ctFit(ctstantestdat, model, backend = 'julia', cores = 1) ctTracePlot(fit) # Julia keeps running between calls, holding what it has compiled; # this ends it. ctJuliaKill("session") }
Splits the variance each indicator of a fitted model implies into a between person part, a within person part from the latent process, and a within person part from measurement error – with the process part further split into what the system noise contributes and what the mean path moving contributes.
ctVarianceDecomposition( fit, method = c("auto", "moment", "simulation"), persons = c("auto", "model", "estimated"), scale = c("latent", "response"), times = NULL, npersons = 200L, npaths = 20L, latents = TRUE, quadpoints = 21L, subjects = "all", realid = TRUE, sources = NULL )ctVarianceDecomposition( fit, method = c("auto", "moment", "simulation"), persons = c("auto", "model", "estimated"), scale = c("latent", "response"), times = NULL, npersons = 200L, npaths = 20L, latents = TRUE, quadpoints = 21L, subjects = "all", realid = TRUE, sources = NULL )
fit |
fit object as generated by |
method |
How the model implied moments are obtained. |
persons |
Which population the between person variance refers to.
|
scale |
For a binary or ordinal indicator, whether to decompose the
linear predictor behind the logit ( |
times |
If given, decompose at each of these times rather than
averaging over the window: a numeric vector, or |
npersons |
Number of persons to draw when |
npaths |
Trajectories drawn per person when |
latents |
If TRUE, also decompose the latent processes, which have no measurement component. |
quadpoints |
Gauss-Hermite nodes used when |
subjects |
|
realid |
If TRUE (the default), subjects are read as the ids in the data, and integers that are not ids there are read as ctsem's numbering of the subjects, 1 to N, with a message. If FALSE, they are read as that numbering. |
sources |
Optional named list of latent names, one entry per group of
processes whose innovations are independent of the other groups' – fast
affect, a day-scale tier, a slow tier, say. The stochastic within person
term is then split by where it came from: |
The decomposition is a nested law of total variance over the
population of (person, occasion) pairs at the design the fit was built on.
With the model's expected observation given the
latent state,
whose four terms are the four columns reported. They are non-negative and sum to the total exactly.
within.deterministic is where non-stationarity appears: it is the
variance of a person's own mean path over their observation times, so it
is zero for a stationary process with no time dependent predictors and
positive when there is a trend, a predictor driven shift, or a process
still relaxing from its starting distribution.
The moments come from the forward recursion over each subject's actual observation times, with no data entering, so unequal spacing, missingness and time dependent predictors are handled exactly rather than by assuming stationarity. The result is exact for a model with linear dynamics.
A model whose DRIFT or DIFFUSION cells depend on the state
has no single transition to run that recursion with, so
method='simulation' draws its trajectories from the engine and
takes the same decomposition over the draws. It is refused rather than
linearised at a point nothing chose. A state dependent measurement
cell – LAMBDA, MANIFESTMEANS, MANIFESTVAR – is
refused by both routes, since turning a drawn state into an expected
observation then needs the measurement model re-materialised at every row.
Random effect levels. A model with a grouping level above the
subject (id = c('subject','study') in ctModel) is
fitted with intoverpop='laplace' – or sampled, which prepares the
same description and is handled the same way here – and gets one
between.<level> column per level, each the variance of that
level's own departure from the level outside it. They are orthogonal by
construction and sum to between.
Each level's fitted modes are shrunk toward the level outside them by an
amount that follows the information per unit rather than the number of
units: on a 6-study, 30-subject example the subject level's modes retained
52 per cent of the population standard deviation (ten occasions each)
while the study level's retained 97 per cent (five subjects each).
persons='estimated' therefore adds to each mode a draw from that
unit's posterior, drawn jointly over its levels: the spread of such
plausible values is the population's, where the spread of the modes is
not. persons='model' draws from each level's own fitted covariance
instead, and is the default.
When the split changes over the window. The columns above are
averages over each person's occasions, and for a process that has not
settled they average a split that is genuinely different at different
times. within.deterministic is the flag that this is happening –
it is the mean path moving – and times= is how to look. On a
fixture whose T0MEANS is fixed, so that every person starts in the
same place, the between person share of an indicator's variance runs from
zero at the first occasion to 20 per cent by the twentieth, against a
window average of 17 per cent that describes no occasion in particular.
Designs. Nothing here assumes a balanced one. Each person is evaluated over their own observation times, so everyone measured at the same occasions, everyone at their own, and subjects with different numbers of occasions all work, as does an observation being missing – a row still has a model implied distribution whether or not its indicator was seen, so every design row counts and the answer does not move with the missingness pattern. Persons are weighted equally rather than by row count: the estimand is over persons, so a frequently measured subject should not be a larger share of the population than a rarely measured one.
method='simulation' checks, by generating twice, that the engine
draws a model's individually varying parameters rather than leaving every
simulated person at the population values, and refuses rather than
returning a between person variance of zero if it does not.
A data frame of class ctVarianceDecomposition, one row per
variable, with columns variable, type, between,
within.deterministic, within.stochastic,
within.measurement, within, total and the
corresponding proportions. The population the between column refers to,
the scale, and the number of persons are recorded as attributes and shown
by print.
ctSummaryMatrices for the matrices this is built
from, and ctStateDependencePlot for a model whose dynamics
vary over the state space.
ctVarianceDecomposition(ctstantestfit)ctVarianceDecomposition(ctstantestfit)
ctWideNames sets default column names for wide ctsem datasets. Primarily intended for internal ctsem usage.
ctWideNames( n.manifest, Tpoints, n.TDpred = 0, n.TIpred = 0, manifestNames = "auto", TDpredNames = "auto", TIpredNames = "auto" )ctWideNames( n.manifest, Tpoints, n.TDpred = 0, n.TIpred = 0, manifestNames = "auto", TDpredNames = "auto", TIpredNames = "auto" )
n.manifest |
number of manifest variables per time point in the data. |
Tpoints |
Maximum number of discrete time points (waves of data, or measurement occasions) for an individual in the input data structure. |
n.TDpred |
number of time dependent predictors in the data structure. |
n.TIpred |
number of time independent predictors in the data structure. |
manifestNames |
vector of character strings giving column names of manifest indicator variables |
TDpredNames |
vector of character strings giving column names of time dependent predictor variables |
TIpredNames |
vector of character strings giving column names of time independent predictor variables |
A character vector of column names for a wide format dataset: the
manifest variables and time dependent predictors at each time point,
the intervals dT1, dT2, ..., then the time independent
predictors.
ctWideToLong Convert ctsem wide to long format
ctWideToLong( datawide, Tpoints, n.manifest, n.TDpred = 0, n.TIpred = 0, manifestNames = "auto", TDpredNames = "auto", TIpredNames = "auto" )ctWideToLong( datawide, Tpoints, n.manifest, n.TDpred = 0, n.TIpred = 0, manifestNames = "auto", TDpredNames = "auto", TIpredNames = "auto" )
datawide |
ctsem wide format data |
Tpoints |
number of measurement occasions in data |
n.manifest |
number of manifest variables |
n.TDpred |
number of time dependent predictors |
n.TIpred |
number of time independent predictors |
manifestNames |
Character vector of manifest variable names. |
TDpredNames |
Character vector of time dependent predictor names. |
TIpredNames |
Character vector of time independent predictor names. |
Names must account for *all* the columns in the data - i.e. do not leave certain variables out just because you do not need them.
A long format matrix, one row per subject and time point, with the
id, the time interval dT (absolute times are restored by
ctDeintervalise), the manifest variables and the
predictors.
#create wide data wideexample <- ctLongToWide(datalong = ctstantestdat, id = "id", time = "time", manifestNames = c("Y1", "Y2"), TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3")) wide <- ctIntervalise(datawide = wideexample, Tpoints = 10, n.manifest = 2, n.TDpred = 1, n.TIpred = 3, manifestNames = c("Y1", "Y2"), TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3") ) #Then convert to long format longexample <- ctWideToLong(datawide = wideexample, Tpoints=10, n.manifest=2, manifestNames = c("Y1", "Y2"), n.TDpred=1, TDpredNames = "TD1", n.TIpred=3, TIpredNames = c("TI1", "TI2","TI3")) #Then convert the time intervals to absolute time long <- ctDeintervalise(datalong = longexample, id='id', dT='dT') head(long,22)#create wide data wideexample <- ctLongToWide(datalong = ctstantestdat, id = "id", time = "time", manifestNames = c("Y1", "Y2"), TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3")) wide <- ctIntervalise(datawide = wideexample, Tpoints = 10, n.manifest = 2, n.TDpred = 1, n.TIpred = 3, manifestNames = c("Y1", "Y2"), TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3") ) #Then convert to long format longexample <- ctWideToLong(datawide = wideexample, Tpoints=10, n.manifest=2, manifestNames = c("Y1", "Y2"), n.TDpred=1, TDpredNames = "TD1", n.TIpred=3, TIpredNames = c("TI1", "TI2","TI3")) #Then convert the time intervals to absolute time long <- ctDeintervalise(datalong = longexample, id='id', dT='dT') head(long,22)
Simulated example dataset for the ctsem package
2 by 15 matrix containing containing ctsem wide format data. 3 measurement occasions of manifest variables Y1 and Y2, 2 measurement occasions of time dependent predictor TD1, 2 measurement intervals dTx, and 2 time independent predictors TI1 and TI2, for 2 individuals.
Maps the stan function so the same code works in R.
inv_logit(x)inv_logit(x)
x |
value to calculate the inverse logit for. |
1 / (1 + exp(-x)), elementwise.
inv_logit(-3)inv_logit(-3)
Maps the stan function so the same code works in R.
log1p_exp(x)log1p_exp(x)
x |
value to use. |
log(1 + exp(x)), elementwise, finite where exp(x)
overflows.
log1p_exp(-3)log1p_exp(-3)
Simulated example dataset for the ctsem package
7 by 8 matrix containing ctsem long format data, for two subjects, with three manifest variables Y1, Y2, Y3, one time dependent predictor TD1, two time independent predictors TI1 and TI2, and absolute timing information Time.
Simulated example dataset for the ctsem package.
200 by 21 matrix containing containing ctsem wide format data. 11 measurement occasions and 10 measurement intervals for each of 200 individuals
See https://bpspsychub.onlinelibrary.wiley.com/doi/10.1111/j.2044-8317.2012.02043.x
Without family, each family's Holm-adjusted p-value on a log scale,
against alpha. With a family, its
statistics: for the families over the gaps between a subject's rows
(residuals, latent, volatility, scores),
each pair's estimate by gap with a 95 per cent interval; for
periodic, each variable's z by period; for any other, the
statistics' z, the 40 largest. The intervals and z are each statistic's
own: the family's p-value, which accounts for how many there are, is the
test.
## S3 method for class 'ctFitAdequacy' plot(x, family = NULL, ...)## S3 method for class 'ctFitAdequacy' plot(x, family = NULL, ...)
x |
A |
family |
One family, or NULL for all of them. |
... |
Unused. |
A ggplot.
One panel per parameter: the drop in log likelihood against the value the parameter was held at, with the likelihood-ratio bar drawn across. A curve that crosses the bar on both sides has a confidence interval; one that runs along the bottom is a direction the data does not determine.
## S3 method for class 'ctFitProfile' plot(x, parameters = NULL, ...)## S3 method for class 'ctFitProfile' plot(x, parameters = NULL, ...)
x |
a |
parameters |
which to draw; defaults to all that were profiled. |
... |
passed to |
x, invisibly.
The backend='julia' counterpart of plot.ctStanFit.
'regression' plots model implied regression coefficients against the
time interval via ctDiscretePars, 'kalman' plots
expectations via ctPredict, 'priorcheck' plots each
population parameter's prior against its posterior via
ctPlotPosterior, and 'trace' plots how the fit reached
its estimate via ctTracePlot: the optimiser's trace, or each
chain's draws for a sampled fit. Stan's own density and interval plots of a
sampled fit ('density', 'intervals') have no julia
counterpart; 'priorcheck' shows the posterior densities.
## S3 method for class 'ctJuliaFit' plot(x, types = "all", wait = TRUE, ...)## S3 method for class 'ctJuliaFit' plot(x, types = "all", wait = TRUE, ...)
x |
Fit object from |
types |
Vector of plot types: 'all', 'regression', 'kalman', 'priorcheck', 'trace'. 'all' leaves out 'regression' for a discrete time model, 'priorcheck' for a fit with no posterior draws, and 'trace' for a fit with nothing to trace. |
wait |
Logical. Pause between plots? |
... |
Passed through to |
Nothing. Generates plots.
# Needs a working julia backend, so the example is inert where Julia is # absent -- including on CRAN, whose check machines have none. if (isTRUE(ctJuliaStatus()$available)) { data <- ctstantestdat model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1)) model$pars$indvarying <- FALSE fit <- ctFit(data, model, backend = 'julia', cores = 1) plot(fit, wait = FALSE) # Julia keeps running between calls, holding what it has compiled; # this ends it. ctJuliaKill("session") }# Needs a working julia backend, so the example is inert where Julia is # absent -- including on CRAN, whose check machines have none. if (isTRUE(ctJuliaStatus()$available)) { data <- ctstantestdat model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1)) model$pars$indvarying <- FALSE fit <- ctFit(data, model, backend = 'julia', cores = 1) plot(fit, wait = FALSE) # Julia keeps running between calls, holding what it has compiled; # this ends it. ctJuliaKill("session") }
Plots prediction output from ctPredict.
## S3 method for class 'ctKalmanDF' plot( x, subjects = unique(x$Subject), kalmanvec = c("y", "yprior"), errorvec = "auto", errormultiply = 1.96, plot = TRUE, elementNames = NA, polygonsteps = 10, polygonalpha = 0.2, facets = "Variable", ... )## S3 method for class 'ctKalmanDF' plot( x, subjects = unique(x$Subject), kalmanvec = c("y", "yprior"), errorvec = "auto", errormultiply = 1.96, plot = TRUE, elementNames = NA, polygonsteps = 10, polygonalpha = 0.2, facets = "Variable", ... )
x |
Output from |
subjects |
which subjects to plot, as they appear in the Subject column of
|
kalmanvec |
string vector of the elements of the output to plot: 'y', the
data, or 'y', 'eta', 'err' or 'errstd' followed by 'prior', 'upd' or 'smooth'. 'y' is
the observed variables and 'eta' the latent states; 'prior' is conditional on all
previous data, 'upd' on all data up to the current time point, and 'smooth' on all
data. 'err' is the data minus the estimate, and 'errstd' its standardised form, present
when the output was made with |
errorvec |
vector of names indicating which kalmanvec elements to plot uncertainty bands for. 'auto' plots all possible. |
errormultiply |
Numeric denoting the multiplication factor of the std deviation of errorvec objects. Defaults to 1.96, for 95% intervals. |
plot |
if FALSE, plots are not generated and the ggplot object is simply returned invisibly. |
elementNames |
if NA, all relevant object elements are included – e.g. if yprior is in the kalmanvec argument, all manifest variables are plotted, and likewise for latent states if etasmooth was specified. Alternatively, a character vector specifying the manifest and latent names to plot explicitly can be specified. |
polygonsteps |
Number of steps to use for uncertainty band shading. |
polygonalpha |
Numeric for the opacity of the uncertainty region. |
facets |
when multiple subjects are included in multivariate plots, the default is to facet plots by variable type. This can be set to NA for no facets, or "Subject" for facetting by subject. |
... |
not used. |
A ggplot2 object. Side effect – Generates plots.
### Get output from ctPredict x<-ctPredict(ctstantestfit,subjects=2,timestep=.01) ### Plot with plot.ctKalmanDF plot(x, subjects=2) ###Single step procedure: ctPredict(ctstantestfit,subjects=2, kalmanvec=c('y','yprior'), elementNames=c('Y1','Y2'), plot=TRUE,timestep=.01)### Get output from ctPredict x<-ctPredict(ctstantestfit,subjects=2,timestep=.01) ### Plot with plot.ctKalmanDF plot(x, subjects=2) ###Single step procedure: ctPredict(ctstantestfit,subjects=2, kalmanvec=c('y','yprior'), elementNames=c('Y1','Y2'), plot=TRUE,timestep=.01)
The search's path: one row per refit, in the order tried, from the log likelihood its round started at to the one the refit reached, both as gained over the starting fit; kept changes filled, rejected ones open. Pruning's decisions are in the subtitle.
## S3 method for class 'ctModelImprove' plot(x, ...)## S3 method for class 'ctModelImprove' plot(x, ...)
x |
A |
... |
Unused. |
A ggplot.
Each family's strongest relaxations: their p-values adjusted within the family (Benjamini-Hochberg), on a log scale, against .05.
## S3 method for class 'ctScoreTest' plot(x, family = NULL, top = 10, ...)## S3 method for class 'ctScoreTest' plot(x, family = NULL, top = 10, ...)
x |
A |
family |
Families to show; by default every family tested. |
top |
Relaxations shown per family, the strongest. |
... |
Unused. |
A ggplot.
Plots for ctStanFit objects
## S3 method for class 'ctStanFit' plot(x, types = "all", wait = TRUE, ...)## S3 method for class 'ctStanFit' plot(x, types = "all", wait = TRUE, ...)
x |
Fit object from |
types |
Vector of character strings defining which plots to create. 'all' plots all possible types, including: 'regression', 'kalman', 'priorcheck', 'trace', 'density','intervals'. |
wait |
Logical. Pause between plots? |
... |
Arguments to pass through to the specific plot functions. Bewar of clashes may occur if types='all'. For details see the specific functions generating each type of plot. |
This function is just a wrapper calling the necessary functions for plotting - it
may be simpler in many cases to access those directly. They are:
ctDiscretePars,ctPredict,
ctPlotPosterior,stan_trace,
stan_dens,stan_plot
rstan offers many plotting possibilities not available here, to use that functionality
one must simply call the relevant rstan plotting function. Use x$stanfit as the stan fit object
(where x is the name of your ctStanFit object). Because a ctStanFit object has many
parameters, the additional argument pars=ctRawParnames(x,'pop_') is recommended.
This denotes population means, but see ctRawParnames for
other options.
Nothing. Generates plots.
plot(ctstantestfit,types=c('regression','kalman','priorcheck'), wait=FALSE)plot(ctstantestfit,types=c('regression','kalman','priorcheck'), wait=FALSE)
Plots priors for free model parameters in a ctStanModel.
## S3 method for class 'ctStanModel' plot( x, rows = "all", wait = FALSE, nsamples = 1e+06, rawpopsd = "marginalise", inddifdevs = c(-1, 1), inddifsd = 0.1, plot = TRUE, ... )## S3 method for class 'ctStanModel' plot( x, rows = "all", wait = FALSE, nsamples = 1e+06, rawpopsd = "marginalise", inddifdevs = c(-1, 1), inddifsd = 0.1, plot = TRUE, ... )
x |
ctStanModel object as generated by |
rows |
vector of integers denoting which rows of ctstanmodel$pars to plot priors for. Character string 'all' plots all rows with parameters to be estimated. |
wait |
If true, user is prompted to continue before plotting next graph. |
nsamples |
Numeric. Higher values increase fidelity (smoothness / accuracy) of density plots, at cost of speed. |
rawpopsd |
Either 'marginalise' to sample from the specified (in the ctstanmodel) prior distribution for the raw population standard deviation, or a numeric value to use for the raw population standard deviation for all subject level prior plots - the plots in dotted blue or red. |
inddifdevs |
numeric vector of length 2, setting the means for the individual differences distributions. |
inddifsd |
numeric, setting the standard deviation of the population means used to generate individual difference distributions. |
plot |
If FALSE, ouputs list of GGplot objects that can be further modified. |
... |
not used. |
Plotted in black is the prior for the population mean. In red and blue are the subject level priors that result given that the population mean is estimated as 1 std deviation above the mean of the prior, or 1 std deviation below. The distributions around these two points are then obtained by marginalising over the prior for the raw population std deviation - so the red and blue distributions do not represent any specific subject level prior, but rather characterise the general amount and shape of possible subject level priors at the specific points of the population mean prior.
model <- ctModel(type='ct', manifestNames='sunspots', latentNames=c('ss_level', 'ss_velocity'), LAMBDA=matrix(c( 1, 'ma1' ), nrow=1, ncol=2), DRIFT=matrix(c(0, 1, 'a21', 'a22'), nrow=2, ncol=2, byrow=TRUE), MANIFESTMEANS=matrix(c('m1'), nrow=1, ncol=1), # MANIFESTVAR=matrix(0, nrow=1, ncol=1), CINT=matrix(c(0, 0), nrow=2, ncol=1), DIFFUSION=matrix(c( 0, 0, 0, "diffusion"), ncol=2, nrow=2, byrow=TRUE)) plot(model,rows=8)model <- ctModel(type='ct', manifestNames='sunspots', latentNames=c('ss_level', 'ss_velocity'), LAMBDA=matrix(c( 1, 'ma1' ), nrow=1, ncol=2), DRIFT=matrix(c(0, 1, 'a21', 'a22'), nrow=2, ncol=2, byrow=TRUE), MANIFESTMEANS=matrix(c('m1'), nrow=1, ncol=1), # MANIFESTVAR=matrix(0, nrow=1, ncol=1), CINT=matrix(c(0, 0), nrow=2, ncol=1), DIFFUSION=matrix(c( 0, 0, 0, "diffusion"), ncol=2, nrow=2, byrow=TRUE)) plot(model,rows=8)
Plot an approximate continuous-time ACF object from ctACF
plotctACF( ctacfobj, df = "auto", quantiles = c(0.025, 0.5, 0.975), separateLearnRates = FALSE, reducedXlim = 1, estimateSpline = TRUE, method = c("weighted", "quantile") )plotctACF( ctacfobj, df = "auto", quantiles = c(0.025, 0.5, 0.975), separateLearnRates = FALSE, reducedXlim = 1, estimateSpline = TRUE, method = c("weighted", "quantile") )
ctacfobj |
object |
df |
basis dimension for the spline. 'auto' uses up to 40 for |
quantiles |
quantiles to plot. |
separateLearnRates |
if TRUE, estimate the learning rate for the quantile splines for each combination of variables. Slower but theoretically more accurate.
Only used with |
reducedXlim |
if non-zero, n timesteps are removed from the upper and lower end of the x range where the spline estimates are less likely to be reasonable. |
estimateSpline |
if TRUE, a spline is estimated according to |
method |
|
With a spline, the full data estimate at each time interval is shown as a point, the spline as a line, and the band spans the outer quantiles.
a ggplot object
data.table::setDTthreads(1) #ignore this line # Example usage: head(ctstantestdat) ac=ctACF(ctstantestdat,varnames=c('Y1'),idcol='id',timecol='time',timestep=.5,nboot=5,plot=FALSE) plotctACF(ac, reducedXlim=0)data.table::setDTthreads(1) #ignore this line # Example usage: head(ctstantestdat) ac=ctACF(ctstantestdat,varnames=c('Y1'),idcol='id',timecol='time',timestep=.5,nboot=5,plot=FALSE) plotctACF(ac, reducedXlim=0)
Print a profile likelihood
## S3 method for class 'ctFitProfile' print(x, digits = 3, ...)## S3 method for class 'ctFitProfile' print(x, digits = 3, ...)
x |
a |
digits |
significant digits for the reported values. |
... |
ignored. |
x, invisibly.
Summarises what the model is, what it will estimate, and what its individually varying parameters imply, rather than printing the underlying list.
## S3 method for class 'ctStanModel' print(x, matrices = NULL, ...)## S3 method for class 'ctStanModel' print(x, matrices = NULL, ...)
x |
A model from |
matrices |
Whether to print the model matrices. Defaults to TRUE for models small enough to read. |
... |
Ignored. |
x$pars remains the canonical specification and can be
printed directly; ctModelMatrices gives the matrix view.
x, invisibly.
Print ctStanFit summaries
## S3 method for class 'summary.ctStanFit' print( x, width = getOption("width"), sections = names(x), row.names = TRUE, ... )## S3 method for class 'summary.ctStanFit' print( x, width = getOption("width"), sections = names(x), row.names = TRUE, ... )
x |
Object returned by |
width |
Console width to use for wrapping note text and formatting tables. |
sections |
Optional character vector of summary sections to print. |
row.names |
Logical. Print row names for data frame sections? |
... |
Additional arguments passed to |
Invisibly returns x.
Converts a lower triangular matrix with standard deviations on the diagonal and partial correlations on lower triangle, to a covariance (or cholesky decomposed covariance)
sdpcor2cov(mat, coronly = FALSE, cholesky = FALSE)sdpcor2cov(mat, coronly = FALSE, cholesky = FALSE)
mat |
input square matrix with std dev on diagonal and lower tri of partial correlations. |
coronly |
if TRUE, ignores everything except the lower triangle and outputs correlation. |
cholesky |
Logical. To return the cholesky decomposition instead of full covariance, set to TRUE. |
A covariance matrix; its lower Cholesky factor with
cholesky = TRUE; a correlation matrix with coronly = TRUE.
testmat <- diag(exp(rnorm(5,-3,2)),5) #generate arbitrary std deviations testmat[row(testmat) > col(testmat)] <- runif((5^2-5)/2, -1, 1) print(testmat) covmat <- sdpcor2cov(testmat) #convert to covariance cov2cor(covmat) #convert covariance to correlationtestmat <- diag(exp(rnorm(5,-3,2)),5) #generate arbitrary std deviations testmat[row(testmat) > col(testmat)] <- runif((5^2-5)/2, -1, 1) print(testmat) covmat <- sdpcor2cov(testmat) #convert to covariance cov2cor(covmat) #convert covariance to correlation
Quickly initialise stanfit object from model and data
stan_reinitsf(model, data, fast = FALSE)stan_reinitsf(model, data, fast = FALSE)
model |
stanmodel |
data |
standata |
fast |
Use cut down form for speed |
stanfit object
sf <- stan_reinitsf(ctstantestfit$stanmodel,ctstantestfit$standata)sf <- stan_reinitsf(ctstantestfit$stanmodel,ctstantestfit$standata)
Convert samples from a stanfit object to the unconstrained scale
stan_unconstrainsamples(fit, standata = NA)stan_unconstrainsamples(fit, standata = NA)
fit |
stanfit object. |
standata |
only necessary if R session has been restarted since fitting model – used to reinitialize stanfit object. |
Matrix containing columns of unconstrained parameters for each post-warmup iteration.
#get data sunspots<-sunspot.year sunspots<-sunspots[50: (length(sunspots) - (1988-1924))] id <- 1 time <- 1749:1924 datalong <- cbind(id, time, sunspots) #setup model ssmodel <- ctModel(type='ct', n.latent=2, n.manifest=1, manifestNames='sunspots', latentNames=c('ss_level', 'ss_velocity'), LAMBDA=matrix(c( 1, 'ma1| log(1+(exp(param)))' ), nrow=1, ncol=2), DRIFT=matrix(c(0, 'a21 | -log(1+exp(param))', 1, 'a22'), nrow=2, ncol=2), MANIFESTMEANS=matrix(c('m1|param * 10 + 44'), nrow=1, ncol=1), MANIFESTVAR=diag(0,1), #As per original spec CINT=matrix(c(0, 0), nrow=2, ncol=1), DIFFUSION=matrix(c(0, 0, 0, "diffusion"), ncol=2, nrow=2)) #fit ssfit <- ctFit(datalong, ssmodel, optimize=FALSE, priors=TRUE, sampleControl=list(iter=200, chains=2, max_treedepth=4), backend='stan') umat <- stan_unconstrainsamples(ssfit$stanfit$stanfit)#get data sunspots<-sunspot.year sunspots<-sunspots[50: (length(sunspots) - (1988-1924))] id <- 1 time <- 1749:1924 datalong <- cbind(id, time, sunspots) #setup model ssmodel <- ctModel(type='ct', n.latent=2, n.manifest=1, manifestNames='sunspots', latentNames=c('ss_level', 'ss_velocity'), LAMBDA=matrix(c( 1, 'ma1| log(1+(exp(param)))' ), nrow=1, ncol=2), DRIFT=matrix(c(0, 'a21 | -log(1+exp(param))', 1, 'a22'), nrow=2, ncol=2), MANIFESTMEANS=matrix(c('m1|param * 10 + 44'), nrow=1, ncol=1), MANIFESTVAR=diag(0,1), #As per original spec CINT=matrix(c(0, 0), nrow=2, ncol=1), DIFFUSION=matrix(c(0, 0, 0, "diffusion"), ncol=2, nrow=2)) #fit ssfit <- ctFit(datalong, ssmodel, optimize=FALSE, priors=TRUE, sampleControl=list(iter=200, chains=2, max_treedepth=4), backend='stan') umat <- stan_unconstrainsamples(ssfit$stanfit$stanfit)
Adjust standata from ctsem to only use specific subjects
standatact_specificsubjects(standata, subjects, timestep = NA)standatact_specificsubjects(standata, subjects, timestep = NA)
standata |
standata |
subjects |
vector of subjects |
timestep |
ignored at present |
list of updated structure
d <- standatact_specificsubjects(ctstantestfit$standata, 1:2)d <- standatact_specificsubjects(ctstantestfit$standata, 1:2)
Optimize / importance sample a stan or ctStan model.
stanoptimis( standata, sm, init = "random", initsd = 0.01, estonly = FALSE, tol = 1e-08, stochastic = TRUE, priors = TRUE, carefulfit = TRUE, stallretries = 2, stalltol = 0.01, uncertainty = c("hessian", "checked", "is", "bootstrap", "fullbootstrap", "sandwich", "opg"), uncertaintyDraws = "auto", uncertaintyControl = list(), subsamplesize = 1, parsteps = c(), plot = FALSE, finishsamples = 1000, lproughnesstarget = 0.2, verbose = 0, cores = 2, matsetup = NA, nsubsets = 1, stochasticTolAdjust = 1000, g_tol = NULL, x_tol = NULL, maxiter = NULL, lbfgs_memory = NULL )stanoptimis( standata, sm, init = "random", initsd = 0.01, estonly = FALSE, tol = 1e-08, stochastic = TRUE, priors = TRUE, carefulfit = TRUE, stallretries = 2, stalltol = 0.01, uncertainty = c("hessian", "checked", "is", "bootstrap", "fullbootstrap", "sandwich", "opg"), uncertaintyDraws = "auto", uncertaintyControl = list(), subsamplesize = 1, parsteps = c(), plot = FALSE, finishsamples = 1000, lproughnesstarget = 0.2, verbose = 0, cores = 2, matsetup = NA, nsubsets = 1, stochasticTolAdjust = 1000, g_tol = NULL, x_tol = NULL, maxiter = NULL, lbfgs_memory = NULL )
standata |
list object conforming to rstan data standards. |
sm |
compiled stan model object. |
init |
vector of unconstrained parameter values, or character string 'random' to initialise with random values very close to zero. |
initsd |
positive numeric specifying sd of normal distribution governing random sample of init parameters, if init='random' . |
estonly |
if TRUE,just return point estimates under $rawest subobject. |
tol |
objective tolerance – the optimizer stops when the objective changes by less than this between evaluations. |
stochastic |
Logical. Use stochastic gradient descent as main optimizer. Always finishes (double checks) with mize (bfgs) optimizer. |
priors |
logical. If TRUE, a priors integer is set to 1 (TRUE) in the standata object – only has an effect if the stan model uses this value. |
carefulfit |
Logical. If TRUE, priors are always used for a rough first pass to obtain starting values when priors=FALSE |
stallretries |
Integer. Number of times to restart the optimizer from fresh random
values when it finishes somewhere that is not a maximum – see |
stalltol |
Gradient per data point above which the optimizer is taken to have stopped short rather than converged. Rough likelihoods – binary indicators are the case this was built for – can leave both optimizers unable to find an improving step, which they report as convergence, returning the starting values as the estimate. Converged fits sit near 1e-5 per data point, stalled ones at 1 or more. |
uncertainty |
Character string selecting the optimized-fit uncertainty
approximation. Options are |
uncertaintyDraws |
Character string controlling approximate
raw-parameter draws
from the approximate uncertainty. |
uncertaintyControl |
List of method-specific options passed to
|
subsamplesize |
value between 0 and 1 representing proportion of subjects to include in first pass fit. |
parsteps |
ordered list of vectors of integers denoting which parameters should begin fixed at zero, and freed sequentially (by list order). Useful for complex models, e.g. keep all cross couplings fixed to zero as a first step, free them in second step. |
plot |
Logical. If TRUE, plot iteration details. Probably slower. |
finishsamples |
Number of samples to draw (either from hessian based covariance or posterior distribution) for final results computation. |
lproughnesstarget |
target log posterior roughness for stochastic optimizer (suggest between .05 and .4). |
verbose |
Integer from 0 to 2. Higher values print more information during model fit – for debugging. |
cores |
Number of cpu cores to use, should be at least 2. |
matsetup |
subobject of ctStanFit output. If provided, parameter names instead of numbers are output for any problem indications. |
nsubsets |
number of subsets for stochastic optimizer. Subsets are further split across cores, but each subjects data remains whole – processed by one core in one subset. |
stochasticTolAdjust |
Multiplier for stochastic optimizer tolerance. |
g_tol |
gradient tolerance: stop when the l2 norm of the gradient falls
below this. |
x_tol |
parameter-step tolerance: stop when the update to the parameter
vector is smaller than this. |
maxiter |
maximum optimizer iterations. |
lbfgs_memory |
number of curvature pairs L-BFGS keeps. |
list containing fit elements
Runs stan, and plots sampling information while sampling.
stanWplot(object, iter = 2000, chains = 4, ...)stanWplot(object, iter = 2000, chains = 4, ...)
object |
stan model object |
iter |
Number of iterations |
chains |
Number of chains |
... |
All the other regular arguments to stan() |
On windows, requires Rtools installed and able to be found by pkgbuild::rtools_path()
The stanfit object rstan::sampling returns.
library(rstan) #### example 1 scode <- " parameters { real y[2]; } model { y[1] ~ normal(0, .5); y[2] ~ double_exponential(0, 2); } " #Uncomment the following lines -- launches rscript not compatible with cran check. #sm <- stan_model(model_code = scode) #fit1 <- stanWplot(object = sm,iter = 100000,chains=2,cores=1)library(rstan) #### example 1 scode <- " parameters { real y[2]; } model { y[1] ~ normal(0, .5); y[2] ~ double_exponential(0, 2); } " #Uncomment the following lines -- launches rscript not compatible with cran check. #sm <- stan_model(model_code = scode) #fit1 <- stanWplot(object = sm,iter = 100000,chains=2,cores=1)
For each curve ctDiscretePars returned – one per subject and
pair of processes – the quantities read off it, per draw, and their
distribution over draws.
## S3 method for class 'ctDiscretePars' summary(object, probs = c(0.025, 0.5, 0.975), ...)## S3 method for class 'ctDiscretePars' summary(object, probs = c(0.025, 0.5, 0.975), ...)
object |
Output of |
probs |
Quantiles of each quantity over draws. |
... |
Not used. |
A data frame, one row per subject, pair of processes (row the
one affected, col the one acting) and quantity, with the mean, sd
and quantiles over draws, and where the object carries the fit's estimate
(an optimised fit's population effects, attr(object, 'estimate')),
the quantity read off the estimate's curve, estimate. The
quantities:
peakThe value of largest magnitude, with its sign.
peaktimeThe lag at which it occurs.
halftimeThe first lag after the peak at which the curve's
magnitude has fallen to half the peak's, interpolated between lags.
For an autoregression, which peaks at lag zero, its half-life: with
impulseType = 'observed', standardise = TRUE, that of the
autocorrelation. NA for a draw whose curve does not fall that
far within the lags computed; beyond is the share of such
draws, and the summary is over the others.
areaThe area under the curve over the lags computed: for a unit impulse over long enough lags, the cumulative effect.
The lags are those ctDiscretePars was given, so their spacing
bounds the precision of peaktime and halftime, and their
range what halftime and area can see.
dp <- ctDiscretePars(ctstantestfit, times = seq(0, 5, .1), nsamples = 20) summary(dp)dp <- ctDiscretePars(ctstantestfit, times = seq(0, 5, .1), nsamples = 20) summary(dp)
Summarise empirical Bayes subject-wise ctsem fits
## S3 method for class 'ctEmpiricalBayesFit' summary( object, use = c("rawest", "rawposterior"), probs = c(0.025, 0.5, 0.975), sdscale = c("unit", "rawsd"), minsd = 1e-06, digits = 4, ... )## S3 method for class 'ctEmpiricalBayesFit' summary( object, use = c("rawest", "rawposterior"), probs = c(0.025, 0.5, 0.975), sdscale = c("unit", "rawsd"), minsd = 1e-06, digits = 4, ... )
object |
Object returned by |
use |
|
probs |
Quantiles to report for transformed parameters. |
sdscale |
How to set |
minsd |
Lower bound used for empirical raw SDs before model adjustment. |
digits |
Number of digits for printed summary tables. |
... |
Unused. |
Compact list containing fit settings, transformed-parameter
summaries, outlier diagnostics, transformed-parameter covariance /
correlation matrices, and subjectpars: each subject's final-pass
point estimates on the parameters' own scale, one row per subject with its
id in subject, unrounded, for comparisons across subjects. The subject
fit lists, raw estimate matrices, pass maps, and adjusted model remain on
the original ctEmpiricalBayesFit object.
Summarise a fit from ctFit, on either backend: population
parameters with their uncertainty, random-effect spreads and correlations,
time independent predictor effects, the parameter matrices, and the log
likelihood and AIC.
## S3 method for class 'ctJuliaFit' summary( object, timeinterval = 1, digits = 3, parmatrices = TRUE, priorcheck = TRUE, residualcov = TRUE, ... ) ## S3 method for class 'ctStanFit' summary( object, timeinterval = 1, digits = 3, parmatrices = TRUE, priorcheck = TRUE, residualcov = TRUE, ... )## S3 method for class 'ctJuliaFit' summary( object, timeinterval = 1, digits = 3, parmatrices = TRUE, priorcheck = TRUE, residualcov = TRUE, ... ) ## S3 method for class 'ctStanFit' summary( object, timeinterval = 1, digits = 3, parmatrices = TRUE, priorcheck = TRUE, residualcov = TRUE, ... )
object |
fit object from |
timeinterval |
positive numeric indicating time interval to use for discrete time parameter calculations reported in summary. |
digits |
integer denoting number of digits to report. |
parmatrices |
if TRUE, also return additional parameter matrices – can be slow to compute for large models with many samples. |
priorcheck |
Whether or not to use |
residualcov |
Whether or not to show standardised residual covariance. Takes a little longer to compute. |
... |
Additional arguments to pass to |
List containing summary items, with a print method for readable console and knitr output.
On an optimised backend = 'julia' fit with uncertainty draws, the
parameter tables and parmatrices carry a mode column beside
the draws' mean, sd and quantiles: the fit's estimate on the same scale,
the point a likelihood ratio or an information criterion refers to.
popmeans reports each free parameter. A T0VAR,
DIFFUSION or MANIFESTVAR parameter is a standard deviation
on the diagonal; off it, a coordinate the model's covmattransform
turns into correlations, not a correlation: under the default 'z'
it is an entry of the correlation matrix's logarithm, which for two
variables is the correlation's Fisher z (tanh() of it is the
correlation) and for more is no function of any one correlation. The
correlations are in parmatrices, as T0cor,
DIFFUSIONcor, MANIFESTcor and asymDIFFUSIONcor beside
the covariances, each summarised over the draws' own correlations.
summary(ctstantestfit)summary(ctstantestfit)
Tests if 2 values are close to each other
test_isclose(..., tol = 1e-08)test_isclose(..., tol = 1e-08)
... |
values to compare. At least two, and none of them empty: comparing a zero-length or NULL value is an error rather than a pass, because there is nothing to compare. |
tol |
tolerance |
Logical or testthat output.
test_isclose(1,1.0000001, tol=1e-4)test_isclose(1,1.0000001, tol=1e-4)