Package 'ctsem'

Title: Continuous Time Structural Equation Modelling
Description: Hierarchical continuous (and discrete) time state space modelling, for linear and nonlinear systems measured by continuous, binary, ordinal, count or censored variables. The subject specific dynamic system is a stochastic differential equation (SDE) or difference equation, measured through a (typically) multivariate factor model. Individual differences on any parameter, at one or more nested levels, are integrated out by an augmented Kalman filter or a Laplace approximation, or sampled. Estimation is by maximum likelihood or maximum a posteriori optimisation, with uncertainty from the Hessian, importance sampling or the bootstrap, or by sampling the posterior, using either 'Stan' or an optional 'Julia' engine that ctJuliaInstall() sets up. See <https://github.com/cdriveraus/ctsem/raw/master/vignettes/hierarchicalmanual.pdf> for details, and <https://osf.io/preprints/psyarxiv/4q9ex_v3> for a detailed tutorial. For the conceptual overview of the hierarchical Bayesian linear SDE approach, see <https://www.researchgate.net/publication/324093594_Hierarchical_Bayesian_Continuous_Time_Dynamic_Modeling>. Exogenous inputs may also be included; for an overview of such possibilities see <https://www.researchgate.net/publication/328221807_Understanding_the_Time_Course_of_Interventions_with_Continuous_Time_Dynamic_Models>. <https://cdriver.netlify.app/> contains some tutorial blog posts.
Authors: Charles Driver [aut, cre, cph], Manuel Voelkle [aut, cph], Han Oud [aut, cph], Trustees of Columbia University [cph]
Maintainer: Charles Driver <[email protected]>
License: GPL-3
Version: 4.0.0
Built: 2026-11-19 01:42:14 UTC
Source: https://github.com/cdriveraus/ctsem

Help Index


ctsem

Description

ctsem is an R package for continuous time structural equation modelling of panel (N > 1) and time series (N = 1) data, using either a frequentist or Bayesian approach, or middle ground forms like maximum a posteriori.

The general workflow begins by specifying a model using the ctModel function, then fitting it to data using ctFit (alias ctStanFit). For original OpenMx / SEM functionality from the first ctsem versions, use the ctsemOMX package. For most purposes, the Stan-based forms in ctsem are more robust and flexible. For examples, see ctFit. For citation info, please run citation('ctsem') .

Author(s)

Maintainer: Charles Driver [email protected] [copyright holder]

Authors:

  • Charles Driver [email protected] [copyright holder]

  • Manuel Voelkle [copyright holder]

  • Han Oud [copyright holder]

Other contributors:

  • Trustees of Columbia University [copyright holder]

References

https://www.jstatsoft.org/article/view/v077i05

Driver, C. C., & Voelkle, M. C. (2018). Hierarchical Bayesian continuous time dynamic modeling. Psychological Methods. Advance online publication.http://dx.doi.org/10.1037/met0000168

Stan Development Team (2018). RStan: the R interface to Stan. R package version 2.17.3. http://mc-stan.org

#' @keywords internal

See Also

Useful links:


AnomAuth

Description

A dataset containing panel data assessments of individuals Anomia and Authoritarianism.

Format

data frame with 2722 rows, 14 columns. Column Y1 represents anomia, Y2 Authoritarianism, dTx the time interval for measurement occasion x.

Source

See doi:10.1037/a0027543 for details.


Likelihood-ratio tests between nested ctsem fits

Description

anova() on two or more optimised fits of nested models to the same data gives each fit's number of parameters, AIC, BIC, log likelihood and -2 log likelihood, and tests each fit against the one above it: the chi-square statistic (twice the gain in log likelihood), its degrees of freedom (the extra parameters) and its p-value. The fits may come from either backend.

Usage

## S3 method for class 'ctJuliaFit'
anova(object, ...)

## S3 method for class 'ctStanFit'
anova(object, ...)

Arguments

object

An optimised fit from ctFit.

...

Further optimised fits, of models nested with object and fitted to the same data.

Details

The fits are ordered by their number of parameters, and each is tested against the one with the next fewer, so the models must be nested in that order; anova() cannot check that. A negative statistic, which warns, means a model fits worse than one nested in it: the models are not nested, or an optimisation stopped short.

The p-value is the chi-square distribution's on the difference in parameters. Where the smaller model fixes a variance at zero, such as a random effect's, the null lies on the boundary of the parameter space and that p-value is conservative (Stram & Lee, 1994); ctScoreTest tests a new random effect against the corrected reference without refitting.

The log likelihoods are those logLik() reports. Where a fit's random-effects check or a refined fit integrated the random effects, that is the integrated value, and the heading says so. A fit with priors maximised the posterior, so its log likelihood is read at the posterior mode rather than at the likelihood's maximum, and the heading says that too. Sampled fits have no single log likelihood; compare them with ctLOO.

Value

A data frame of class anova, one row per fit in order of parameters, with columns npar, AIC, BIC, logLik, -2LL, Chisq, Df and Pr(>Chisq); the last three are NA on the first row. Rows are named by the arguments.

References

Stram, D. O., & Lee, J. W. (1994). Variance components testing in the longitudinal mixed effects model. Biometrics, 50(4), 1171-1177.

See Also

ctChisqTest for the p-value alone.

Examples

df <- data.frame(id=1, time=1:length(sunspot.year), Y1=sunspot.year)

    m1 <- ctModel(type='dt', LAMBDA=diag(1),MANIFESTVAR=0)
    m2 <- ctModel(type='dt', LAMBDA=diag(1),MANIFESTVAR=0,DRIFT = .9)

    if (isTRUE(ctJuliaStatus()$available)) {
      f1 <- ctFit(df,m1,cores=1)
      f2 <- ctFit(df,m2,cores=1)

      test <- anova(f1, f2)
      test
      test$Chisq[2]
    }

Continuous Time Autocorrelation Function (ctACF)

Description

Computes autocorrelations and cross correlations by time lag, pooled over subjects, for data observed at irregular times.

Usage

ctACF(
  dat,
  varnames = "auto",
  ccfnames = "all",
  idcol = "id",
  timecol = "time",
  plot = TRUE,
  timestep = "auto",
  time.max = "auto",
  nboot = 100,
  scale = FALSE,
  center = FALSE
)

Arguments

dat

The input data in data frame or data table format, in long format.

varnames

Character vector of variable names in the data to compute the ACF for. 'auto' uses all columns that are not time / id.

ccfnames

Character vector of variable names in the data to compute cross correlation for. 'all' uses all variables in varnames, NA uses none.

idcol

The name of the column containing subject IDs (default is 'id').

timecol

The name of the column containing time values (default is 'time').

plot

A logical value indicating whether to create a plot (default is TRUE).

timestep

The width of the time lag bins. 'auto' (default) uses half the median time interval between consecutive rows of a subject.

time.max

The largest time lag to compute. 'auto' (default) uses two thirds of the mean, over rows, of the time span of the row's subject.

nboot

The number of bootstrap samples, each resampling subjects with replacement (default is 100).

scale

if TRUE, divide each variable by its within-subject standard deviation.

center

if TRUE, subtract each variable's within-subject mean.

Details

Within each subject, every pair of observations is put in the time lag bin nearest to the time between them, at multiples of timestep up to time.max. The value at a lag is the correlation of the pairs in its bin, pooled over subjects: of a variable at time t + lag with the same variable at time t for an autocorrelation, and for the cross correlation 'x_y' of x at time t + lag with y at time t, so that at a positive lag y is the earlier. Lags with fewer than three pairs are left out. Autocorrelations are returned from the first non-zero lag, as the lag 0 autocorrelation is always 1. Cross correlations include lag 0, the contemporaneous correlation. Bootstrap samples resample whole subjects, and the plot shows a spline through the estimates with a band from their bootstrap spread (see plotctACF).

Value

If 'plot' is TRUE, the function returns a ggplot object of the ACF plot. If 'plot' is FALSE, it returns a data table with a row for each sample, variable and time lag: Sample (0 for the data, then the bootstrap samples), ACF, TimeInterval, Variable, SignificanceLevel (the 95% bound of a correlation of zero computed from that many independent pairs), and Pairs, the number of pairs the correlation is computed from.

Examples

data.table::setDTthreads(1) #ignore this line
# Example usage:
head(ctstantestdat)
ctACF(ctstantestdat,varnames=c('Y1'),idcol='id',timecol='time',nboot=5)

Calculate Continuous Time Autocorrelation Function (ACF) for Standardized Residuals of ctsem fit.

Description

This function takes a fit object from ctsem and computes the continuous time autocorrelation function (ACF) on the standardized residuals.

Usage

ctACFresiduals(fit, ...)

Arguments

fit

A fitted model object generated by the ctsem package.

...

Additional arguments to be passed to the ctACF function.

Details

This function first extracts the standardized residuals from the fit object using the ctKalmanArray function. Then, it calculates the continuous time ACF for these residuals and returns the results as a data table.

Value

A data table containing the continuous time ACF estimates for standardized residuals.

See Also

ctKalmanArray

Examples

data.table::setDTthreads(1) #ignore this line
# Example usage:
ctACFresiduals(ctstantestfit, varnames='Y1',nboot=5)

Chi Square test wrapper for ctsem fit objects.

Description

The p-value of a likelihood-ratio test between two nested model fits, from either backend. anova(fit1, fit2) gives the whole test: each fit's log likelihood, the chi-square statistic, its degrees of freedom and the p-value.

Usage

ctChisqTest(fit1, fit2)

Arguments

fit1

One of the fits to be compared.

fit2

Second fit to be compared. The fit with more parameters is tested against the other, whichever order they come in.

Value

The p-value: that of the fit with more parameters in anova(fit1, fit2). NA where the two have the same number of parameters.

Examples

df <- data.frame(id=1, time=1:length(sunspot.year), Y1=sunspot.year)

    m1 <- ctModel(type='dt', LAMBDA=diag(1),MANIFESTVAR=0)
    m2 <- ctModel(type='dt', LAMBDA=diag(1),MANIFESTVAR=0,DRIFT = .9)

    if (isTRUE(ctJuliaStatus()$available)) {
      f1 <- ctFit(df,m1,cores=1)
      f2 <- ctFit(df,m2,cores=1)

      ctChisqTest(f1,f2)
    }

ctCollapse Easily collapse an array margin using a specified function.

Description

ctCollapse Easily collapse an array margin using a specified function.

Usage

ctCollapse(inarray, collapsemargin, collapsefunc, plyr = TRUE, ...)

Arguments

inarray

Input array of more than one dimension.

collapsemargin

Integers denoting which margins to collapse.

collapsefunc

function to use over the collapsing margin.

plyr

Whether to use plyr.

...

additional parameters to pass to collapsefunc.

Value

An array (or vector) with the collapsemargin dimensions removed, holding collapsefunc applied over them.

Examples

testarray <- array(rnorm(900,2,1),dim=c(100,3,3))
ctCollapse(testarray,1,mean)

Context dependence of reported model matrices

Description

A ctsem matrix cell may be written as an expression referencing a latent process or a time dependent predictor – for instance DRIFT[1,1] = '-log1p(exp(param)) * eta2'. Such a cell has no single value. It has a value at an evaluation context: a latent state, a set of time dependent predictor values, a time and an interval.

Details

ctsem's summaries and plots report a number for every cell, so for a model like this they must pick a point. There are two, and which one you get depends on what you asked for:

  • Population matrices (summary, ctSummaryMatrices, ctDiscretePars(subjects='popmean'), ctTIpredEffects) are evaluated at the population T0MEANS state, with time dependent predictors at zero.

  • Subject matrices (ctSubjectPars, ctDiscretePars(subjects=)) are evaluated at each subject's last observed row.

Only ctKalman and ctPredict avoid the choice entirely, because they run the filter and re-evaluate every cell at every step.

A reference to an individually varying parameter is not affected by any of this. ctsem represents such a parameter as a carrier latent state with no drift and no diffusion, so a cell reading that state simply is the parameter, and the estimate at the subject's last row is the one that has seen all of that subject's data – the best available, not an arbitrary point. Only references to real dynamic processes, and to time dependent predictors, make a reported value conditional.

Functions affected by this attach the cells in question as attr(x, 'contextDependent'), and say so once in a message. ctModelIsNonlinear answers the question directly. With backend='julia' the evaluation point can be chosen; see the state argument of ctBackendParMatrices.

See Also

ctModelIsNonlinear, ctBackendParMatrices


Write a covariance matrix as the cells a ctsem model reads

Description

ctsem's covariance matrices – T0VAR, DIFFUSION, MANIFESTVAR, and the population spreads RAWPOPVAR and RAWPOPVAR_<idname> – are specified as a lower triangle: standard deviations on the diagonal, and below it coordinates that the model's covmattransform turns into correlations. Except for two variables under 'z', where a coordinate is Fisher's z, what one coordinate yields depends on the others, so the cells for a given covariance are not something to work out by hand. ctCov() computes them.

Usage

ctCov(cov, covmattransform = "z")

Arguments

cov

A symmetric positive definite covariance matrix.

covmattransform

The construction to write the cells for: 'z' (the default, and what ctModel uses), 'rawcorr' or 'cholesky'. A ctCov() matrix assigned into a model is rewritten for that model's own construction, so this matters only when the cells are read directly.

Details

Pass the result wherever a fixed matrix goes: ctModel(DIFFUSION = ctCov(S)), m$matrices$T0VAR <- ctCov(S), or m$RAWPOPVAR <- ctCov(S), in every case for the whole matrix. The model keeps the covariance as well as the cells, so the matrix keeps its meaning if covmattransform is changed afterwards, and the R generator of ctGenerate – which reads covariance cells as a Cholesky factor – generates from the same covariance as the julia one. A cell edited by hand afterwards replaces what ctCov() wrote.

For RAWPOPVAR, the covariance is that of the raw, untransformed parameters across subjects (or groups, for RAWPOPVAR_<idname>), which is the scale a fit reports it on.

Value

The cells, a lower triangular numeric matrix of class ctCov, with the covariance as attribute covariance. 'rawcorr' cannot reach every correlation matrix, and a covariance it cannot represent is refused.

Examples

S <- matrix(c(1, .6, .3,
              .6, 2, .5,
              .3, .5, 1.5), 3)
ctCov(S)

m <- ctModel(LAMBDA = diag(3), DIFFUSION = ctCov(S), Tpoints = 5)

Check interval coverage by simulation

Description

Performs a coverage check analysis by generating data from a model, fitting it multiple times with different fit arguments, and plotting the results. Works with fitArgs/ fittingModel combinations that fit with either backend='julia' (the ctFit default) or backend='stan'.

Usage

ctCoverageCheck(
  initialData,
  fittingModel,
  niter,
  fitArgs,
  cores = getOption("mc.cores", 2L),
  fitCores = 1,
  generateCores = fitCores,
  plotEvery = max(c(10, cores))
)

Arguments

initialData

An initial dataset to fit to determine 'true' parameters for further generation.

fittingModel

A ctModel object used for fitting the data

niter

Number of iterations to run

fitArgs

Named list of fit argument sets to test (e.g., list(boot = list(optimcontrol = list(uncertainty = 'bootstrap')), hess = list(optimcontrol = list(uncertainty = 'hessian'))))

cores

The most CPU cores the check uses at once. Iterations run cores %/% fitCores at a time, each fit on fitCores.

fitCores

Number of cores for each inner ctFit call, at most cores. This is enforced after merging each fitArgs entry so that worker fits cannot take more than their share.

generateCores

Number of cores for ctGenerateFromFit when generating replicated datasets, at most cores. Defaults to fitCores; set it explicitly to use more cores for generation.

plotEvery

Print plots every n iterations; by default every 10, or every cores iterations when that is more.

Value

A list containing the results data.table and final plots

Examples

## Not run: 
# Does the Hessian interval cover as often as it says? 100 datasets
# generated from the fit to ctstantestdat, each fitted twice.
model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1',
  LAMBDA = matrix(1))
check <- ctCoverageCheck(ctstantestdat, model, niter = 100,
  fitArgs = list(
    hessian = list(optimcontrol = list(uncertainty = 'hessian')),
    bootstrap = list(optimcontrol = list(uncertainty = 'bootstrap'))),
  cores = 4)

## End(Not run)

ctDeintervalise

Description

Converts intervals in ctsem long format data to absolute time

Usage

ctDeintervalise(datalong, id = "id", dT = "dT", startoffset = 0)

Arguments

datalong

data to use, in ctsem long format (attained via function ctWideToLong)

id

character string denoting column of data containing numeric identifier for each subject.

dT

character string denoting column of data containing time interval preceding observations in that row.

startoffset

Number of units of time to offset by when converting.

Value

datalong, with the interval column dT replaced by absolute times and renamed time.


ctDiscretePars

Description

Calculate model implied regressions for a sequence of time intervals (if ct) or steps (if dt) based on a ctStanFit object, for specified subjects. Wrap with print() when used inside for loops! ctStanDiscretePars is maintained as a backward-compatible alias.

Usage

ctDiscretePars(
  fit,
  subjects = "popmean",
  times = seq(from = 0, to = 10, by = 0.1),
  nsamples = 200,
  impulseType = "unit",
  standardise = FALSE,
  cov = FALSE,
  plot = FALSE,
  cores = getOption("mc.cores", 2L),
  state = NULL,
  method = "linearise",
  realid = TRUE,
  ...,
  ctstanfitobj,
  observational
)

Arguments

fit

model fit from ctFit

subjects

Either 'popmean', to use the population mean parameter, or the subjects to use, as ids in the data (see realid).

times

Numeric vector of positive values, discrete time parameters will be calculated for each. If the fit object is a discrete time model, these should be positive integers.

nsamples

Number of samples from the stanfit to use for plotting. Higher values will increase smoothness / accuracy, at cost of plotting speed. Values greater than the total number of samples will be set to total samples.

impulseType

What a "one unit change in process c" is taken to bring with it. Every variant is dtDRIFT(t) %*% C for a different companion matrix C, whose column c says what else moves.

'unit' (the default)

C = I. An impulse of one unit (one standard deviation when standardise=TRUE) is applied to process c alone and nothing else moves, because the change was imposed. This is the partial regression E[x(t+u)|x(t)], and the only variant that is a property of the dynamics alone.

'observed'

C = Sigma diag(Sigma)^-1 with Sigma = asymDIFFUSIONcov. Process c is observed one unit above expectation and the others are not held fixed, so they move by Sigma_rc/Sigma_cc. This is the simple regression – what regressing the observed data on x_c alone recovers.

'shock'

C = Q diag(Q)^-1 with Q = DIFFUSIONcov. One hypothetical system noise innovation arrives in c, with companion innovations Q_rc/Q_cc. An impulse response rather than a regression.

'orthogonal'

From the Cholesky factor of DIFFUSION, so shocks are uncorrelated. Depends on the order of the latent processes, which is a modelling assumption rather than a property of the fit.

The companion matrices are asymmetric, and must be: a one unit move in a wide-spread process implies more movement in a narrow one than the reverse. 'observed' and 'shock' use different covariance matrices because they ask different questions, and coincide only under isotropic decay with no cross effects.

standardise

Logical, FALSE by default. If TRUE, report the effect of a one standard deviation change in process c as a change in r's own standard deviations, taking both scales from the stationary within subject covariance asymDIFFUSIONcov. That makes effects between processes on different scales comparable, and is what ctNetwork reports by default. FALSE leaves the effects in the processes' own units, so impulseType='unit', standardise=FALSE is the bare expm(DRIFT*t). A system with no stationary variance has nothing to standardise by, and those draws are returned as NaN with a message. The plot title states which scale was used either way.

cov

Logical. If TRUE, covariances are returned instead of regression coefficients.

plot

Logical. If TRUE, ggplots output using ctDiscreteParsPlot instead of returning output.

cores

Number of cpu cores to use for computing subject matrices. If subject matrices were saved during fiting, not used.

state

Evaluation point for any matrix cell that depends on the latent state or a time dependent predictor – see ctContextDependence. NULL (the default) or 'T0MEANS' uses the population initial state, with time dependent predictors at zero, which for a model whose dynamics depend on the state can be far from where the data are; 'mean' the mean smoothed latent state, 'asymptotic' the system's own fixed point; or supply a numeric state vector. The initial state and the fixed point are each draw's own, so the intervals include where they are; 'mean' is the estimate's, used for every draw, so its intervals leave out where that state is. A process that does not change (a latent predictor) is held at its initial value in finding the fixed point. Requires backend='julia' and subjects='popmean'. For a linear model the argument makes no difference, since no cell depends on the state.

method

How the interval regressions are obtained. 'linearise' (the default) exponentiates the drift's Jacobian at the evaluation point, expm(J*t) with J the model's JAx: the DRIFT matrix itself for a linear model, where it is exact, and for a model whose dynamics depend on the state, the response of the model linearised at state. Where a cell is nonlinear in the state the two differ: a DRIFT cell a + b*Y in the column of process Y is a + b*Y there, and contributes a + 2*b*Y to the Jacobian. A stan fit carries no Jacobian, and exponentiates DRIFT as evaluated at T0MEANS. The linearisation holds J fixed over the interval, so it is a fair interval regression near a fixed point, or over intervals short beside the dynamics; away from a fixed point the state moves during the interval, the nonlinear system's response can differ from it, and 'simulate' is the check. 'simulate' integrates the nonlinear system from state, and from state plus a shock of one stationary standard deviation in each process in turn (with the companion movements impulseType gives), and differences the trajectories, divided back to one unit (one standard deviation when standardise=TRUE): the model implied regression for a shock of that size. It starts at 1 and decays to 0 for a stable process, and reduces exactly to 'linearise' when no cell depends on the state; for a strongly nonlinear system the curve depends on the shock's size, and near a fixed point the difference from 'linearise' is that size's effect. 'simulate' requires backend='julia' and is slower, so it uses nsamples posterior draws with a small default.

realid

If TRUE (the default), subjects are read as the ids in the data, and integers that are not ids there are read as ctsem's numbering of the subjects, 1 to N, with a message. If FALSE, they are read as that numbering.

...

additional plotting arguments to control ctDiscreteParsPlot

ctstanfitobj

Deprecated. Use fit.

observational

Deprecated. Use impulseType: FALSE is 'unit' and TRUE is 'observed'.

Details

Two of these combinations are quantities you may already know under other names. impulseType='unit' with standardise=FALSE is the discrete time autoregression and cross-lagged matrix, expm(DRIFT*t). impulseType='observed' with standardise=TRUE is the model implied cross-correlation function, Cor(x_r(t+u), x_c(t)) – the model's counterpart to the empirical autocorrelations ctACF computes from the data, and equal to the latent correlation matrix at t=0.

If plot=TRUE, the function will return a ggplot2 object (and hence needs to be printed if intended to display within a loop). This can be modified by the various ggplot2 functions, or displayed using print(x).

Value

With plot = FALSE, an array of the discrete time coefficients, by sample, subject, time interval, row and column, of class ctDiscretePars: summary() gives each curve's peak, the lag of the peak, the lag at which it has fallen to half the peak (a half-life), and the area under it, with their uncertainty (summary.ctDiscretePars). On an optimised fit, with subjects = 'popmean', the same coefficients at the fit's estimate are attr(x, 'estimate'), by subject, time interval, row and column: for 'linearise' computed from the matrices ctSummaryMatrices(fit, state = )$estimate reports, for 'simulate' simulated at the estimate. The draws' mean or median need not equal them. summary() reads their quantities into an estimate column, and ctDiscreteParsPlot draws them as its line. A sampled fit has no estimate, and gives none; nor do subjects other than 'popmean', whose matrices come from each subject's filter pass over the draws. With plot = TRUE, a ggplot object.

Examples

data.table::setDTthreads(1) #ignore this line
ctDiscretePars(ctstantestfit,times=seq(.5,4,.1),
 plot=TRUE,indices='CR')

#modify plot
require(ggplot2)
g=ctDiscretePars(ctstantestfit,times=seq(.5,4,.1),
 plot=TRUE,indices='CR')
g= g+ labs(title='Cross effects')
print(g)

ctDiscreteParsPlot

Description

Plots the output from ctDiscretePars, for model implied regression strengths at specified times for continuous time models fit with ctStanFit. ctStanDiscreteParsPlot is maintained as a backward-compatible alias.

Usage

ctDiscreteParsPlot(
  x,
  indices = "all",
  quantiles = c(0.025, 0.5, 0.975),
  latentNames = "auto",
  ylab = "Coefficient",
  xlab = "Time interval",
  ylim = NA,
  facets = NA,
  splitSubjects = TRUE,
  colour = "Effect",
  title = "auto",
  polygonalpha = 0.1,
  ggcode = FALSE,
  line = c("estimate", "median")
)

Arguments

x

list object returned from ctDiscretePars.

indices

Either a string specifying type of plot to create, or an n by 2 matrix specifying which indices of the output matrix to plot. 'AR' specifies all diagonals, for discrete time autoregression parameters. 'CR' specifies all off-diagonals,for discrete time cross regression parameters. 'all' plots all AR and CR effects at once.

quantiles

numeric vector of length 3, with values between 0 and 1, specifying which quantiles to plot. The default plots 95% credible intervals as the band, and the posterior median at 50% as the line where line does not draw the estimate.

latentNames

Vector of character strings denoting names for the latent variables. 'auto' just uses eta1 eta2 etc.

ylab

y label.

xlab

x label.

ylim

Two values, the lower and upper limits of the y axis. The plot zooms to them; intervals reaching past them are drawn to the edge.

facets

May be 'Subject' or 'Effect'.

splitSubjects

if TRUE, subjects are plotted separately, if FALSE they are combined.

colour

Character string denoting how colour varies. 'Effect' or 'Subject'.

title

Character string. 'auto' generates automatically, NULL can be used to disable title.

polygonalpha

Numeric between 0 and 1 to multiply the alpha of the fill.

ggcode

if TRUE, returns a list containing the data.table to plot (for each curve and time interval, the outer quantiles of the draws as ymin and ymax, and the line as value), and a character string that can be evaluated (with the necessary arguments such as ylab etc filled in). For modifying plots.

line

What the line is. 'estimate' (the default) draws the effects at the fit's estimate where x carries them (attr(x, 'estimate'): an optimised fit's population effects), and the middle of quantiles otherwise; 'median' always draws the middle of quantiles. The band is the draws' either way.

Details

For a model whose dynamics depend on the state, the curves are the model linearised at one point, or simulated from it, and the subtitle names the point. The default, the population initial state with time dependent predictors at zero, can be far from where the data are; choose the point, and whether to linearise or simulate, with ctDiscretePars(state = , method = ), whose help says when the linearisation is a fair interval regression.

Value

A ggplot2 object. This can be modified by the various ggplot2 functions, or displayed using print(x).

Examples

data.table::setDTthreads(1) #ignore this line
x <- ctDiscretePars(ctstantestfit)
ctDiscreteParsPlot(x, indices='CR')

#to modify plot:
g <- ctDiscreteParsPlot(x, indices='CR') +
 ggplot2::labs(title='My ggplot modification')
print(g)

Discretise long format continuous time (ctsem) data to specific timestep.

Description

Rounds timing information so equal intervals, according to specified timestep, are achieved. Each subject's times are moved to the nearest point of a grid that starts at the subject's first time; values that land on the same point are averaged, with a warning. NA's are inserted in other columns as necessary. Any columns specified by TDpredNames have zeroes rather than NA's inserted (because some estimation routines do not tolerate NA's in covariates), while columns specified by TIpredNames are left as NA's – since a time independent predictor is constant within a subject, inserting zeroes for it would be incorrect.

Usage

ctDiscretiseData(
  dlong,
  timestep,
  timecol = "time",
  idcol = "id",
  TDpredNames = NULL,
  TIpredNames = NULL
)

Arguments

dlong

Long format data

timestep

Positive real value to discretise

timecol

Name of column containing absolute (not intervals) time information.

idcol

Name of column containing subject id variable.

TDpredNames

Vector of column names of any time dependent predictors

TIpredNames

Vector of column names of any time independent predictors

Value

long format ctsem data.

Examples

long <- ctDiscretiseData(dlong=ctstantestdat, timestep = .1,
TDpredNames=c('TD1'),TIpredNames=c('TI1','TI2','TI3'))

Get documentation pdf for ctsem

Description

Get documentation pdf for ctsem

Usage

ctDocs()

Value

Nothing. Opens a pdf when run interactively.

Examples

ctDocs()

Empirical Bayes subject-wise ctsem fits

Description

Fits one ctsem model per subject using the model prior, estimates the empirical marginal distribution of the raw parameters, then fits each subject again using the resulting empirical Bayes prior.

Usage

ctEmpiricalBayesFit(
  datalong,
  model,
  subjects = "all",
  priors = TRUE,
  optimize = TRUE,
  cores = getOption("mc.cores", 2L),
  subjectFitArgs = list(),
  Npasses = 2,
  ebUse = c("rawest", "rawposterior"),
  ebRobust = TRUE,
  ebOutlierMAD = 6,
  ebOutlierQuantiles = c(0.025, 0.975),
  ebWinsorize = TRUE,
  minsd = 1e-06,
  verbose = 0,
  progress = TRUE,
  ...
)

Arguments

datalong

Long format data containing multiple subjects.

model

Model object from ctModel. Time independent predictors are not supported. A random-effect-free copy of this model is used for the per-subject fits.

subjects

Vector of subject identifiers to fit, or 'all'.

priors

Logical. Passed to ctFit; defaults to TRUE.

optimize

Logical. Passed to ctFit; defaults to TRUE.

cores

Number of subjects to fit in parallel. Each individual subject-level ctFit call uses one core.

subjectFitArgs

Named list of additional arguments passed to each ctFit call. For optimized fits, optimcontrol$stochastic defaults to FALSE for all EB passes unless supplied here or in .... First-pass fits force optimcontrol$estonly=TRUE.

Npasses

Total number of subject-wise fitting passes. The default 2 fits once with the model prior, builds a marginal empirical Bayes prior, then fits once with that prior. Values above 2 repeatedly map the previous pass estimates back to the original raw scale, rebuild the marginal EB prior, and refit.

ebUse

'rawest' to build the empirical Bayes prior from first pass point estimates, or 'rawposterior' to pool raw posterior samples.

ebRobust

Logical. If TRUE, the empirical Bayes prior is built from robust raw summaries after outlier handling.

ebOutlierMAD

Positive numeric. Raw values farther than this many MAD-scaled deviations from the median are treated as outliers. Use Inf or NULL to disable this rule.

ebOutlierQuantiles

Length two numeric vector of lower and upper quantiles used to bound first-pass raw values, or NULL to disable.

ebWinsorize

Logical. If TRUE, outlying first-pass raw values are clamped to the outlier bounds before computing the EB prior. If FALSE, they are set to missing for EB prior construction.

minsd

Lower bound used for empirical raw SDs before model adjustment.

verbose

Integer from 0 to 2. Passed to ctFit.

progress

Logical. If TRUE, report the current EB fitting stage and overwrite a single console line with the subject fitting percentage.

...

Additional arguments passed to each ctFit call.

Details

Works with either backend='stan' or backend='julia', passed via subjectFitArgs or ...; every per-subject ctFit call uses the same backend. ebUse='rawposterior' needs each subject fit to carry posterior draws (the default for both backends' optimised fits, unless optimcontrol$estonly=TRUE), or a sampled fit (optimize=FALSE, or for backend='julia' a Laplace fit passed through ctFitUncertainty with uncertainty = 'sample').

Value

Object of class ctEmpiricalBayesFit, containing the subject fit lists and metadata. $initialfits contains the first-pass model prior fits, $fits contains the final empirical Bayes prior fits, and $passfits contains every pass. Use summary() to compute final transformed-parameter means, SDs, covariances, correlations, and outlier diagnostics.

Examples

if (isTRUE(ctJuliaStatus()$available)) {
  model <- ctModel(type='ct', manifestNames='Y1', LAMBDA=matrix(1))
  eb <- ctEmpiricalBayesFit(ctstantestdat, model, cores=2,
    subjectFitArgs=list(optimcontrol=list(finishsamples=20)))
  summary(eb)
}

ctExample1

Description

Simulated example dataset for the ctsem package

Format

100 by 17 matrix containing containing ctsem wide format data. 6 measurement occasions of leisure time and happiness and 5 measurement intervals for each of 100 individuals.


ctExample1TIpred

Description

Simulated example dataset for the ctsem package

Format

100 by 18 matrix containing containing ctsem wide format data. 6 measurement occasions of leisure time and happiness, 1 measurement of number of friends, and 5 measurement intervals for each of 100 individuals.


ctExample2

Description

Simulated example dataset for the ctsem package

Format

100 by 18 matrix containing containing ctsem wide format data. 8 measurement occasions of leisure time and happiness, 7 measurement occasions of a money intervention dummy, and 7 measurement intervals for each of 50 individuals.


ctExample2level

Description

Simulated example dataset for the ctsem package

Format

100 by 18 matrix containing ctsem wide format data. 8 measurement occasions of leisure time and happiness, 7 measurement occasions of a money intervention dummy, and 7 measurement intervals for each of 50 individuals.


ctExample3

Description

Simulated example dataset for the ctsem package

Format

1 by 399 matrix containing containing ctsem wide format data. 100 observations of variables Y1 and Y2 and 199 measurement intervals, for 1 subject.


ctExample4

Description

Simulated example dataset for the ctsem package

Format

20 by 79 matrix containing 20 observations of variables Y1, Y2, Y3, and 19 measurement intervals dTx, for each of 20 individuals.


Extract samples from a ctsem fit

Description

Returns the posterior (or approximate posterior) draws of a fit's parameters, as arrays named for the model matrices.

Usage

ctExtract(
  object,
  subjectMatrices = FALSE,
  cores = getOption("mc.cores", 2L),
  nsamples = "all",
  subjects = "all",
  state = NULL,
  realid = TRUE,
  ...
)

Arguments

object

A fit from ctFit, on either backend: sampled, or optimized with its approximate posterior draws.

subjectMatrices

Calculate subject specific system matrices? On a backend='julia' fit whose random effects are coordinates (intoverpop='laplace', or a sampled fit), each draw's subject matrices are at that draw's plausible values – the population draw with a draw of the subject's own effects from their posterior given its data – so their spread over draws is the subject's whole uncertainty, as ctSubjectPars reports by default.

cores

Only used if subjectMatrices = TRUE . For faster computation use more cores.

nsamples

either 'all' or an integer denoting number of random samples to extract.

subjects

either 'all', or the subjects to extract, as ids in the data (see realid).

state

The latent state at which state dependent cells of the pop_* matrices are evaluated: 'T0MEANS' (the default), 'mean', 'asymptotic', or a numeric state vector. Only backend='julia' can materialise the matrices anywhere other than the default, so anything else is an error on a stan fit rather than being silently ignored. Irrelevant to a linear model, whose matrices are the same everywhere. Note that it moves the pop_* arrays only: the subject matrices subjectMatrices=TRUE adds come from the filter, and are always at each subject's own last observed row.

realid

If TRUE (the default), subjects are read as the ids in the data, and integers that are not ids there are read as ctsem's numbering of the subjects, 1 to N, with a message. If FALSE, they are read as that numbering.

...

not used.

Value

A list of posterior sample arrays. The point the pop_* matrices were evaluated at is recorded in the 'evaluatedAt' attribute; nothing is messaged, because the summaries call this repeatedly and say it once themselves. ctModelIsNonlinear answers whether the point matters for this model at all.

Where parameters vary over subjects, the population of them, per draw: popsd (draws by parameters), rawpopcov and rawpopcorr (the raw scale's covariance and correlations, draws by parameters by parameters) and popcov (the covariance on the natural scale). On a backend='julia' fit they are named by parameter, and a model with several grouping levels gives each level's as rawpopcov.<level> and so on; its popcov combines popsd with the raw correlations, exact where a parameter's transform is affine.

Examples

e = ctExtract(ctstantestfit)
attr(e, 'evaluatedAt')

Fit a ctsem model

Description

Fits a ctsem model specified via ctModel with type either 'ct' or 'dt'. ctStanFit is maintained as a backward-compatible alias.

Usage

ctFit(
  datalong,
  model,
  stanmodeltext = NA,
  iter = 1000,
  intoverstates = TRUE,
  binomial = FALSE,
  fit = TRUE,
  intoverpop = "auto",
  poprank = "auto",
  sameInitialTimes = FALSE,
  stationary = FALSE,
  plot = FALSE,
  derrind = NA,
  optimize = TRUE,
  optimcontrol = list(),
  backend = c("julia", "stan"),
  nlcontrol = list(),
  nopriors = NA,
  priors = FALSE,
  chains = 2,
  cores = getOption("mc.cores", 2L),
  inits = NULL,
  compileArgs = list(),
  forcerecompile = FALSE,
  saveCompile = TRUE,
  savescores = FALSE,
  savesubjectmatrices = FALSE,
  saveComplexPars = FALSE,
  gendata = FALSE,
  sampleControl = list(),
  control = list(),
  verbose = 0,
  ...,
  popunique = NULL,
  popblocks = NULL,
  weights = NULL,
  ctstanmodel
)

Arguments

datalong

long format data containing columns for subject id, time, manifest variables, any time dependent (i.e. varying within subject) predictors, and any time independent (not varying within subject) predictors.

model

model object as generated by ctModel with type='ct' or 'dt', for continuous or discrete time models respectively.

stanmodeltext

already specified Stan model character string, generally leave NA unless modifying Stan model directly. (Possible after modification of output from fitting with argument fit=FALSE.) backend='stan' only.

iter

Deprecated – use sampleControl$iter. Still honoured, with a warning.

intoverstates

Logical. TRUE (the default) integrates over the latent states with a Kalman filter. FALSE makes the states part of what is estimated – the target becomes the joint density of the data and the states, and the fit reports the trajectory in fit$estimate$states – and is for sampling: with backend='julia' that route makes no Gaussian assumption about the state anywhere, where the filter's update for a binary, ordinal or count indicator is an assumed-density projection. optimize=TRUE is refused, because the joint mode is degenerate rather than an estimate; optimcontrol$estonly=TRUE returns it anyway, with no standard errors.

binomial

Deprecated. Logical indicating the use of binary rather than Gaussian data, as with IRT analyses. This now sets the manifesttype of every indicator to 1, for binary.

fit

If FALSE, prepare the model and data without fitting: stan returns its fit object unfitted, julia the specification its engine reads.

intoverpop

How declared individual differences (random effects) are handled: integrated out of the likelihood, or sampled with everything else.

'auto', the default, integrates them out when optimizing, as TRUE does, and samples them ('none') when optimize=FALSE – except on julia where every subject's likelihood is Gaussian in its effects (Gaussian indicators, effects only on means through the affine transform), where they are integrated exactly and the population parameters sampled ('laplace').

TRUE integrates them out: by 'laplace' on backend='julia', and by 'augmented' on stan, which has only that route, or with intoverstates=FALSE, which composes with it only. fit$args$resolved$intoverpopreason records why. Before 4.0.0 TRUE always meant 'augmented'.

'laplace' (backend='julia') integrates an effect that shifts a mean in a linear Gaussian model – a manifest mean, an initial mean, a continuous intercept – exactly, inside each subject's filter, and every other effect by a Laplace approximation at its mode, which summary() reports. 'augmented' makes each random effect a state of the filter: exact for effects that shift a mean in a linear Gaussian model, and a linearisation elsewhere. That shrinks random effects on the dynamics; where the dynamics depend on the state, an effect that reaches them – a random initial mean too, the more so beside a random manifest mean that keeps the data from pinning it down – is linearised over its population spread, which can overstate the likelihood and move the maximum; and variation on a DIFFUSION or MANIFESTVAR parameter is only partly identified. ctFit warns of both.

'quadrature' (backend='julia', optimizing) fits by 'laplace' and then refines the fit with the units the random-effects check names (optimcontrol$checks) integrated numerically – each unit by adaptive Gauss-Hermite or lattice rules, whichever reaches the accuracy at the lower cost, at any number of effects – reporting that objective's estimate, standard errors and log likelihood (Laplace's kept in loglik_laplace). Seconds to minutes; longer where the units nest; ctFitUncertainty(fit, intoverpop = 'quadrature') does the same after a fit.

FALSE, or 'none', samples them, so it needs optimize=FALSE: maximising over every subject's effects would drive the population variance to zero. With optimize=FALSE on julia, 'none' (and so 'auto', but for the exception above) samples the joint posterior of parameters and random effects; 'laplace' and 'augmented' sample the population parameters with the effects integrated out, by the Laplace approximation or the filter, and TRUE by whichever of the two it resolves to.

poprank

Rank of the population covariance of the individually varying parameters. 'auto', the default, uses the number of varying parameters that reach the observation mean: under intoverpop='augmented' that is all the route can identify, so it costs no likelihood, it does nothing on a model whose varying parameters all reach the mean, and it says so when it reduces. NA leaves the covariance unrestricted. A smaller whole number is an approximation – fewer dimensions than the data support, which lowers the likelihood and distorts the parameters retained – useful for parsimony or speed.

Requires backend='julia', and the default applies to 'augmented' only: under 'laplace' and 'none' the dropped coordinates are identified, so a rank has to be asked for. There it may be given per grouping level by name, as poprank=c(study=2), with 'auto' resolved per level, and it may not fall below the number of individually varying T0MEANS with a free T0VAR, whose latents then take their initial variance from this covariance. Under 'augmented' a varying T0MEANS cannot be reduced at all, and an explicit rank is refused. A restricted covariance is L %*% t(L) for a loading matrix L, reported as poploading_<parameter>_dim<j>.<level>; a loading's sign is arbitrary, so read the standard deviations and correlations summary() reports. A stated RAWPOPVAR cell turns a defaulted reduction off, and is refused alongside a rank asked for. May also be set on the model, as model$poprank <- 2. A rank of 0 is allowed at a level whose every varying parameter has an own variance (popunique).

sameInitialTimes

if TRUE, include an empty observation for every subject that has no observation at the earliest observation time of the dataset. This ensures that the T0MEANS occurs for every subject at the same time, rather than just at the earliest observation for that subject. Important when modelling trends over time, age, etc.

stationary

Logical. If TRUE, each subject's latent processes start from the distribution DRIFT, CINT and DIFFUSION settle into – the mean -DRIFT^-1 CINT and the asymptotic covariance – rather than from T0MEANS and T0VAR, which are then not estimated. Suits processes observed long after they began, and saves estimating the initial state. Requires backend='julia', a continuous time model, and a DRIFT, CINT and DIFFUSION that are fixed within each subject: no cell may depend on the latent states or on time dependent predictors. Individual differences in those matrices are allowed with intoverpop='laplace', which intoverpop='auto' then chooses; each subject starts from its own stationary distribution. A latent process whose DRIFT is fixed so that it never settles – a row fixed at zero, or a fixed oscillation such as ctModelPeriodicComponent's clock – is refused.

To start only some initial conditions there, name those cells 'stationary' in the model (backend='julia'); every other initial condition stays as written. A T0MEANS cell takes its entry of the stationary mean, asymCINT. A T0VAR diagonal cell gives its latent the stationary variance, asymDIFFUSION's diagonal, keeping that latent's T0VAR correlations with the others. A T0VAR off-diagonal cell gives the pair the stationary covariance, and needs both their variances, and every covariance among the latents it joins, named too. The stationary moments are those of the marked latents and every latent that drives them, so a latent driven by others works, and DRIFT, CINT and DIFFUSION need to be fixed within each subject only for those.

plot

if TRUE, for sampling, a Shiny program is launched upon fitting to interactively plot samples. May struggle with many (e.g., > 5000) parameters. For optimizing, the optimisation trace is plotted – with backend='julia' once the fit returns, since the fit is one call into the engine; for live output use optimcontrol$callback.

derrind

deprecated, latents involved in dynamic error calculations are determined automatically now.

optimize

if TRUE, estimate by optimisation – maximum likelihood, or maximum a posteriori with priors=TRUE – with uncertainty from the curvature at the estimate. If FALSE, sample the posterior: with Stan's HMC sampler for backend='stan', and for backend='julia' with NUTS through a learnt map on the joint posterior of parameters and random effects, or NUTS or tempered SMC when there are none or intoverpop integrates them out – slower, but exact rather than a normal approximation at a mode. Other uncertainty methods for an optimised fit, importance sampling among them, are in ctFitUncertainty. A sampled fit's point estimate is the per-parameter median of the draws on stan (fit$stanfit$rawest) and their mean on julia (fit$estimate$raw).

optimcontrol

List of optimiser settings, read by both backends with the same meaning: stopping rules (tol, g_tol, maxiter), the start (initsd, carefulfit), the uncertainty computed at the end (uncertainty, estonly), and optional phases such as stochastic and, under intoverpop='laplace', saem and checks. ctOptimControl lists every name with its backend and default. A misspelled name, or one the chosen backend cannot honour, is an error.

backend

Either 'julia' (the default) or 'stan'. The two take the same model definitions and give the same summaries. The julia backend optimises and samples (optimize=FALSE), and adds intoverpop='laplace', grouping levels above the subject, and ordinal, count and censored indicators, and it integrates a binary one rather than linearising it. It needs a working Julia, which ctJuliaInstall sets up once; see also ctJuliaSetup. 'stan' was the default up to ctsem 3.11 and needs no external toolchain; it is to be deprecated.

nlcontrol

List of non-linear control parameters. maxtimestep must be a positive numeric, specifying the largest time span covered by the numerical integration. The large default ensures that for each observation time interval, only a single step of exponential integration is used. When maxtimestep is smaller than the observation time interval, the integration is nested within an Euler like loop. Smaller values may offer greater accuracy, but are slower and not always necessary. Given the exponential integration, linear model elements are fit exactly with only a single step. nsubsteps = 'auto' (julia backend only) instead measures, at the starting values and again at the optimum, how nonlinear each observation interval is and refines only the intervals that need it (under intoverpop = 'laplace', with each subject at its random-effect modes); substeptol (default 0.01) is the largest acceptable linearisation error as a fraction of the predicted state standard deviation, and maxsubsteps (default 64) caps an interval. maxtimestep remains a ceiling on the step. The choice is reported in fit$substeps. With optimize = FALSE that optimum is the one that places the sampler, and the chains keep the mesh chosen there. transition = 'euler' (julia backend, intoverstates = FALSE only) replaces the exponential step between substeps of the state-explicit path with plain Euler-Maruyama, for a reference that shares no approximation with the filter; it needs a fine maxtimestep. See also ctParticleLik.

nopriors

deprecated, use priors argument. logical. If TRUE, any priors are disabled – sometimes desirable for optimization.

priors

FALSE (the default), TRUE or 'randomCorr'. FALSE uses no prior: maximum likelihood. TRUE adds ctsem's normal(0,1) raw-scale prior to every parameter, making an optimised fit maximum a posteriori. 'randomCorr' applies it to the random-effect correlations only, at every level: a bounded correlation on an unbounded coordinate can walk along a direction the data do not determine, and the prior holds it. Sampling (optimize=FALSE) without a prior on every parameter warns, because a parameter the data leave flat has an improper posterior; use priors=TRUE there. 'randomCorr' needs backend='julia'.

chains

Deprecated – use sampleControl$chains. Still honoured, with a warning.

cores

number of cpu cores to use: a positive whole number, or 'maxneeded' for all but one (capped at the number of chains on stan). Defaults to getOption("mc.cores", 2). It is the most CPU cores the call uses at once, counting every process it starts: on julia the threads a fit uses, BLAS included, and a sampling run's worker processes, which share it between them. A julia fit at cores > 1 is not reproducible to the last decimal, because its sums are divided among workers in an order that varies from run to run; use cores = 1 for a before-and-after comparison. The Julia session starts as wide as the call that starts it, and a later call asking for more runs at what it has and says so: ctJuliaSetup(threads = n, force = TRUE) restarts it wider, and options(ctsem.julia.restart = TRUE) has a fit do so itself.

inits

NULL (the default) for ctsem's own starting values, or a vector of starting values on the raw (unconstrained) scale, one per free parameter, as a fit holds its estimate: fit$estimate$raw on julia, fit$stanfit$rawest on stan. On julia, a fit itself: its estimates are matched to this model's parameters by name, so a model extended by a predictor or a smooth starts where the fit stopped, with the new coefficients at zero. With backend='stan' and optimize=FALSE, the string 'optimize' optimises first and starts the chains there.

compileArgs

List of arguments to pass to stan_model for compilation of the Stan model.

forcerecompile

logical. For backend='stan', recompiles the stan model regardless of apparent need (for development). For backend='julia', compiles the model's state-dependent expressions – matrix cells that read a latent state, and their derivatives – and any parameter transforms of your own into code for this model, rather than evaluating them as data. The model's first fit in a session then compiles for a minute or more, and its gradients run faster: by up to a third for a heavily nonlinear model (many state-dependent cells, or long formulas in them), and by little otherwise. Worth it only when such a model's fit takes many minutes.

saveCompile

if TRUE and compilation is needed / requested, writes the stan model to the parent frame as ctsem.compiled (unless that object already exists and is not from ctsem), to avoid unnecessary recompilation.

savescores

Logical. If TRUE, output from the Kalman filter is saved in output. For datasets with many variables or time points, will increase file size substantially. backend='stan' only; on julia, ctKalman computes it from the fit.

savesubjectmatrices

Logical. If TRUE, subject specific matrices are saved – only relevant when either time dependent predictors or individual differences are used. Can increase memory usage dramatically in large models, and can be computed after fitting using ctExtract or ctSubjectPars. backend='stan' only.

saveComplexPars

Logical. If TRUE, also save rowwise output of any complex parameters specified, i.e. combinations of parameters, functions and states.

gendata

Logical – If TRUE, uses provided data for only covariates and a time and missingness structure, and generates random data according to the specified model / priors. Generated data is in the $Ygen subobject after running extract on the fit object. For datasets with many manifest variables or time points, file size may be large. To generate data based on the posterior of a fitted model, see ctGenerateFromFit. backend='stan' only; julia generates with ctGenerate.

sampleControl

Used when optimize=FALSE: a list of sampler settings. chains (4 on julia, 2 on stan); iter (1000, warmup and sampling together); warmup (200, or half of iter if that is less; not used by the map sampler, which learns its map over warm-up windows of its own first); draws, the post-warmup count, which wins over iter when given; seed; saveEffects, whether every random-effect draw is kept; and processes (FALSE), whether the chains run in worker processes – at most cores at once, so four chains on two cores run two at a time – rather than together in this session.

For backend='julia' also sampler ('map', NUTS through a learnt map, on the joint posterior; 'saem', SAEM's kernel, or 'nuts' by choice), placement (the map starts from an optimised fit; for 'saem' and 'nuts' on the joint posterior 'saem': SAEM's state places the chains, so no Laplace optimum is needed; 'fit' places them around an optimised fit), the stopping target minESS (200, of the worst parameter) and rhatTarget (1.01), with meanESS, maxDraws and settleTol, and NUTS's maxdepth, target_accept, maxdelta, init_scale, adapt_metric and adapt_effects – all described under uncertainty = 'sample' in ctFitUncertainty, with how lower targets give a quicker approximate posterior. A run stops once the target is met, and may run to four times the post-warmup count to reach it. A name the sampler does not read is an error. For backend='stan' the settings are rstan's, passed to stan's control.

control

Deprecated – use sampleControl, which is the same list under a name that says what it controls. Still honoured, with a warning; where both name the same setting, sampleControl wins.

verbose

Integer from 0 to 2. 1 reports progress while the model fits; 2 additionally keeps every progress line rather than overwriting one in place, and prints more for debugging. Whether overwriting is possible is detected from where the output is going; set options(ctsem.progress.overwrite = FALSE) if that detection is wrong for your front end – a Shiny app capturing stdout, for instance – or TRUE to force it on.

...

additional arguments to pass to stan function (backend='stan' only; refused on julia).

popunique

Own variances on a reduced-rank level: the parameters selected get a variance of their own beside the poprank shared dimensions, so that level's covariance is L %*% t(L) plus a diagonal – the factor-analytic form. TRUE selects every parameter varying at a level, FALSE none, and names select matrices or parameters, e.g. popunique = c(subject = 'MANIFESTMEANS'). One unnamed value applies to every level; named entries, or a named list, apply per level. Each selected level needs a poprank below its number of varying parameters; poprank = 0 with every parameter selected gives independent random effects. Reported as popunique_<parameter>.<level>, on the same raw scale as a full-rank sd. Requires backend = 'julia' and intoverpop = 'laplace' or sampling ('none'); must be passed by name.

popblocks

Shared dimensions in blocks, on a reduced-rank level: a list of blocks, each a list with rank, how many shared dimensions, and effects, which parameters load on them – TRUE for every parameter varying at the level, or names of matrices or parameters, as for popunique. A parameter loads only on the dimensions of the blocks that name it, so parameters with no block in common are uncorrelated, and one named in no block needs an own variance (popunique). A level's rank is the sum of its blocks' ranks, so poprank need not be given for it; 'auto' there defers to the blocks, and a number must equal the sum. One unnamed list of blocks applies to every level; a list named by level gives each level its own. For example, person-level item intercepts that do not covary with a latent level but may covary with each other and with the dynamics: popblocks = list(level = list(rank = 1, effects = c('lev', 'DRIFT')), items = list(rank = 2, effects = c('MANIFESTMEANS', 'DRIFT'))). Within a block, its first rank parameters, in the order the model lists them, form its basis. Loadings are reported as poploading_<parameter>_dim<j>, the dimensions numbered across the blocks in order. Requires backend = 'julia' and intoverpop = 'laplace' or sampling ('none'); must be passed by name.

weights

Sampling weights, one per subject: the name of a column of datalong that is constant within subject, or a numeric vector named by subject id (or unnamed, in the order subjects first appear in the data). Each subject's contribution to the likelihood – its marginal likelihood, random effects integrated out – is multiplied by its weight (the weighted pseudo-likelihood), and the prior is not; a weight of zero leaves the subject out. Weights are normalised to a mean of one over the weighted subjects, and standard errors come from the sandwich estimator unless optimcontrol$uncertainty says otherwise. fit$estimate$loglik stays the data's log likelihood at the estimate; fit$estimate$loglik_weighted is the weighted one. Requires backend = 'julia' and optimisation; with nested ids, weights must be constant within each group of the outermost level. Must be passed by name.

ctstanmodel

Deprecated. Use model.

Details

A julia fit with random effects reports, at its estimate, how much of each random effect each subject's own data determine, and how many subjects' worth of information each population standard deviation rests on (fit$identifiability$effects). An effect resting on fewer than two is named in a message, in summary() and in ctReport, with a lower poprank suggested where its level can take one and indvarying = FALSE after it.

Value

A fitted object of class ctStanFit (backend='stan') or ctJuliaFit (backend='julia'), both also classed ctFit. Besides backend-specific components, every fit carries $args, a list with two sublists that mean the same thing on both backends:

input

Exactly what was passed to ctFit(), or the formal default when an argument was not supplied – intoverpop may still read 'auto', and cores 'maxneeded' if that is what was passed.

resolved

What the fit actually ran with, after 'auto' routing, deprecated-argument merges (e.g. nopriors into priors) and backend-specific defaults were settled – cores is a concrete integer, intoverpop is one of 'augmented', 'laplace' or 'none', and priors is TRUE, FALSE or 'randomCorr'. A call that gives the same input on both backends gives the same resolved on both.

Before this, $args itself held the raw call on a stan fit and only the resolved settings on a julia fit, so the same field name meant opposite things across backends; code reading fit$args$intoverpop or fit$args$cores directly should now read fit$args$resolved (or $input, if the literal call is what is wanted, as ctFitUpdate does).

See Also

ctOptimControl for every optimiser setting; ctFitUncertainty for uncertainty after a fit; ctIdentify reports which parameters the data can inform, before a fit is spent finding out; ctTracePlot draws the optimisation trace a julia fit records.

Examples

#generate a modern ctsem model relying heavily on defaults
model<-ctModel(type='ct',
  latentNames=c('eta1','eta2'),
  manifestNames=c('Y1','Y2'),
  MANIFESTVAR=diag(.1,2),
  TDpredNames='TD1',
  TIpredNames=c('TI1','TI2','TI3'),
  LAMBDA=diag(2))

# The default backend is julia; ctJuliaInstall() sets it up.
if (isTRUE(ctJuliaStatus()$available)) {
  fit<-ctFit(ctstantestdat, model,priors=TRUE)

  summary(fit)

  plot(fit,wait=FALSE)
}

#### extended examples

library(ctsem)
set.seed(3)

#  Data generation (run this, but no need to understand!) -----------------

Tpoints <- 20
nmanifest <- 4
nlatent <- 2
nsubjects<-20

#random effects
age <- rnorm(nsubjects) #standardised
cint1<-rnorm(nsubjects,2,.3)+age*.5
cint2 <- cint1*.5+rnorm(nsubjects,1,.2)+age*.5
tdpredeffect <- rnorm(nsubjects,5,.3)+age*.5

for(i in 1:nsubjects){
  #generating model
  gm<-ctModel(Tpoints=Tpoints,n.manifest = nmanifest,n.latent = nlatent,n.TDpred = 1,
  type='omx',
    LAMBDA = matrix(c(1,0,0,0, 0,1,.8,1.3),nrow=nmanifest,ncol=nlatent),
    DRIFT=matrix(c(-.3, .2, 0, -.5),nlatent,nlatent),
    TDPREDMEANS=matrix(c(rep(0,Tpoints-10),1,rep(0,9)),ncol=1),
    TDPREDEFFECT=matrix(c(tdpredeffect[i],0),nrow=nlatent),
    DIFFUSION = matrix(c(1, 0, 0, .5),2,2),
    CINT = matrix(c(cint1[i],cint2[i]),ncol=1),
    T0VAR=diag(2,nlatent,nlatent),
    MANIFESTVAR = diag(.5, nmanifest))

  #generate data
  newdat <- ctGenerate(ctmodelobj = gm,n = 1,burnin = 2,
    dtmat=rbind(c(rep(.5,8),3,rep(.5,Tpoints-9))))
  newdat[,'id'] <- i #set id for each subject
  newdat <- cbind(newdat,age[i]) #include time independent predictor
  if(i ==1) {
    dat <- newdat[1:(Tpoints-10),] #pre intervention data
    dat2 <- newdat #including post intervention data
  }
  if(i > 1) {
    dat <- rbind(dat, newdat[1:(Tpoints-10),])
    dat2 <- rbind(dat2,newdat)
  }
}
colnames(dat)[ncol(dat)] <- 'age'
colnames(dat2)[ncol(dat)] <- 'age'


#plot generated data for sanity
plot(age)
matplot(dat[,gm$manifestNames],type='l',pch=1)
plotvar <- 'Y1'
plot(dat[dat[,'id']==1,'time'],dat[dat[,'id']==1,plotvar],type='l',
  ylim=range(dat[,plotvar],na.rm=TRUE))
for(i in 2:nsubjects){
  points(dat[dat[,'id']==i,'time'],dat[dat[,'id']==i,plotvar],type='l',col=i)
}


dat2[,gm$manifestNames][sample(1:length(dat2[,gm$manifestNames]),size = 100)] <- NA


#data structure
head(dat2)


# Model fitting -----------------------------------------------------------

##simple univariate default model

m <- ctModel(type = 'ct', manifestNames = c('Y1'), LAMBDA = diag(1))
ctModelLatex(m)

#Specify univariate linear growth curve

m1 <- ctModel(type = 'ct',
  manifestNames = c('Y1'), latentNames=c('eta1'),
  DRIFT=matrix(0,nrow=1,ncol=1),
  DIFFUSION=matrix(0,nrow=1,ncol=1),
  T0VAR=matrix(0,nrow=1,ncol=1),
  CINT=matrix(c('cint1'),ncol=1),
  T0MEANS=matrix(c('t0m1'),ncol=1),
  LAMBDA = diag(1),
  MANIFESTMEANS=matrix(0,ncol=1),
  MANIFESTVAR=matrix(c('merror'),nrow=1,ncol=1))

ctModelLatex(m1)

#fit
if (isTRUE(ctJuliaStatus()$available)) {
  f1 <- ctFit(datalong = dat2, model = m1, optimize=TRUE, priors=FALSE)

  summary(f1)

  #plots of individual subject models v data
  ctPredict(f1,plot=TRUE,subjects=1,kalmanvec=c('y','yprior'),timestep=.01)
  ctPredict(f1,plot=TRUE,subjects=1:3,kalmanvec=c('y','ysmooth'),timestep=.01,errorvec=NA)

  #compare randomly generated data from the posterior to the observed data,
  #including the lagged covariance structure and the mean trajectory over time
  ctPostPredict(f1, wait=FALSE)
}

 ### Further example models

#Include intervention
m2 <- ctModel(type = 'ct',
  manifestNames = c('Y1'), latentNames=c('eta1'),
  n.TDpred=1,TDpredNames = 'TD1', #this line includes the intervention
  TDPREDEFFECT=matrix(c('tdpredeffect'),nrow=1,ncol=1), #intervention effect
  DRIFT=matrix(-1e-5,nrow=1,ncol=1),
  DIFFUSION=matrix(0,nrow=1,ncol=1),
  CINT=matrix(c('cint1'),ncol=1),
  T0MEANS=matrix(c('t0m1'),ncol=1),
  T0VAR=matrix(0,nrow=1,ncol=1),
  LAMBDA = diag(1),
  MANIFESTMEANS=matrix(0,ncol=1),
  MANIFESTVAR=matrix(c('merror'),nrow=1,ncol=1))



#Individual differences in intervention, Bayesian estimation, covariates
m2i <- ctModel(type = 'ct',
  manifestNames = c('Y1'), latentNames=c('eta1'),
  TIpredNames = 'age',
  TDpredNames = 'TD1', #this line includes the intervention
  TDPREDEFFECT=matrix(c('tdpredeffect||TRUE'),nrow=1,ncol=1), #intervention effect
  DRIFT=matrix(-1e-5,nrow=1,ncol=1),
  DIFFUSION=matrix(0,nrow=1,ncol=1),
  CINT=matrix(c('cint1'),ncol=1),
  T0MEANS=matrix(c('t0m1'),ncol=1),
  T0VAR=matrix(0,nrow=1,ncol=1),
  LAMBDA = diag(1),
  MANIFESTMEANS=matrix(0,ncol=1),
  MANIFESTVAR=matrix(c('merror'),nrow=1,ncol=1))


#Including covariate effects
m2ic <- ctModel(type = 'ct',
  manifestNames = c('Y1'), latentNames=c('eta1'),
  n.TIpred = 1, TIpredNames = 'age',
  n.TDpred=1,TDpredNames = 'TD1', #this line includes the intervention
  TDPREDEFFECT=matrix(c('tdpredeffect'),nrow=1,ncol=1), #intervention effect
  DRIFT=matrix(-1e-5,nrow=1,ncol=1),
  DIFFUSION=matrix(0,nrow=1,ncol=1),
  CINT=matrix(c('cint1'),ncol=1),
  T0MEANS=matrix(c('t0m1'),ncol=1),
  T0VAR=matrix(0,nrow=1,ncol=1),
  LAMBDA = diag(1),
  MANIFESTMEANS=matrix(0,ncol=1),
  MANIFESTVAR=matrix(c('merror'),nrow=1,ncol=1))

m2ic$pars$indvarying[m2ic$pars$matrix %in% 'TDPREDEFFECT'] <- TRUE


#Include deterministic dynamics
m3 <- ctModel(type = 'ct',
  manifestNames = c('Y1'), latentNames=c('eta1'),
  n.TDpred=1,TDpredNames = 'TD1', #this line includes the intervention
  TDPREDEFFECT=matrix(c('tdpredeffect'),nrow=1,ncol=1), #intervention effect
  DRIFT=matrix('drift11',nrow=1,ncol=1),
  DIFFUSION=matrix(0,nrow=1,ncol=1),
  CINT=matrix(c('cint1'),ncol=1),
  T0MEANS=matrix(c('t0m1'),ncol=1),
  T0VAR=matrix('t0var11',nrow=1,ncol=1),
  LAMBDA = diag(1),
  MANIFESTMEANS=matrix(0,ncol=1),
  MANIFESTVAR=matrix(c('merror1'),nrow=1,ncol=1))





#Add system noise to allow for fluctuations that persist in time
m3n <- ctModel(type = 'ct',
  manifestNames = c('Y1'), latentNames=c('eta1'),
  n.TDpred=1,TDpredNames = 'TD1', #this line includes the intervention
  TDPREDEFFECT=matrix(c('tdpredeffect'),nrow=1,ncol=1), #intervention effect
  DRIFT=matrix('drift11',nrow=1,ncol=1),
  DIFFUSION=matrix('diffusion',nrow=1,ncol=1),
  CINT=matrix(c('cint1'),ncol=1),
  T0MEANS=matrix(c('t0m1'),ncol=1),
  T0VAR=matrix('t0var11',nrow=1,ncol=1),
  LAMBDA = diag(1),
  MANIFESTMEANS=matrix(0,ncol=1),
  MANIFESTVAR=matrix(c(0),nrow=1,ncol=1))



#include 2nd latent process

m4 <- ctModel(n.manifest = 2,n.latent = 2, type = 'ct',
  manifestNames = c('Y1','Y2'), latentNames=c('L1','L2'),
  n.TDpred=1,TDpredNames = 'TD1',
  TDPREDEFFECT=matrix(c('tdpredeffect1','tdpredeffect2'),nrow=2,ncol=1),
  DRIFT=matrix(c('drift11','drift21','drift12','drift22'),nrow=2,ncol=2),
  DIFFUSION=matrix(c('diffusion11','diffusion21',0,'diffusion22'),nrow=2,ncol=2),
  CINT=matrix(c('cint1','cint2'),nrow=2,ncol=1),
  T0MEANS=matrix(c('t0m1','t0m2'),nrow=2,ncol=1),
  T0VAR=matrix(c('t0var11','t0var21',0,'t0var22'),nrow=2,ncol=2),
  LAMBDA = matrix(c(1,0,0,1),nrow=2,ncol=2),
  MANIFESTMEANS=matrix(c(0,0),nrow=2,ncol=1),
  MANIFESTVAR=matrix(c('merror1',0,0,'merror2'),nrow=2,ncol=2))

#dynamic factor model -- fixing CINT to 0 and freeing indicator level intercepts

m3df <- ctModel(type = 'ct',
  manifestNames = c('Y2','Y3'), latentNames=c('eta1'),
  n.TDpred=1,TDpredNames = 'TD1', #this line includes the intervention
  TDPREDEFFECT=matrix(c('tdpredeffect'),nrow=1,ncol=1), #intervention effect
  DRIFT=matrix('drift11',nrow=1,ncol=1),
  DIFFUSION=matrix('diffusion',nrow=1,ncol=1),
  CINT=matrix(c(0),ncol=1),
  T0MEANS=matrix(c('t0m1'),ncol=1),
  T0VAR=matrix('t0var11',nrow=1,ncol=1),
  LAMBDA = matrix(c(1,'Y3loading'),nrow=2,ncol=1),
  MANIFESTMEANS=matrix(c('Y2_int','Y3_int'),nrow=2,ncol=1),
  MANIFESTVAR=matrix(c('Y2residual',0,0,'Y3residual'),nrow=2,ncol=2))

Append pseudo-posterior samples to an optimized ctStanFit object (deprecated)

Description

Appends normal draws from the fit's already-computed covariance to its raw posterior. Deprecated in favour of ctOptimUncertainty with uncertainty = 'stored', which does the same on both backends.

Usage

ctFitAddSamples(fit, nsamples, cores = 2)

Arguments

fit

fit object

nsamples

number of samples desired

cores

number of cores to use

Details

These are pseudo-posterior draws from the fitted covariance, not posterior draws from a sampler. ctFitUncertainty with uncertainty = 'sample' is the latter – Hamiltonian Monte Carlo from an optimized ctJuliaFit – and is a different object, not another route to this one.

Deprecated in favour of ctOptimUncertainty(fit, uncertainty = 'stored', finishsamples = n), which draws from the same covariance for the same zero model evaluations, works on a ctJuliaFit as well as a ctStanFit, and records the new sample count in $uncertainty$settings. It also replaces the draws rather than appending to them: appending normal draws to a posterior that came from uncertainty='is' or 'bootstrap' leaves part of each in one matrix with nothing recording the mixture, which is what this function does.

ctAddSamples is the same function under its pre-3.11 name; both are deprecated together.

Value

fit object with extra samples

See Also

ctOptimUncertainty, ctFitUncertainty

Examples

## Not run: 
newfit <- ctOptimUncertainty(ctstantestfit, uncertainty = 'stored',
  finishsamples = 30, cores = 1)

## End(Not run)

Adequacy of a fitted model, as tests

Description

Tests a fitted model in families, each aimed at one way a continuous time model can fall short, and gives each family a p-value. Most families use the one-step-ahead prediction errors, which under the model are independent and standard normal whatever the spacing of the observations; each variable's error is taken given the others observed at the same time, so a result names a variable.

Usage

ctFitAdequacy(
  fit,
  covariates = NULL,
  families = c("residuals", "latent", "volatility", "pairs", "periodic", "distribution",
    "fit", "dependence", "effects", "tvcovariates", "subjects", "nonlinear",
    "covariates"),
  gaps = NULL,
  periods = NULL,
  tvcovariates = NULL,
  close = 0.05,
  alpha = 0.05,
  projection = c("auto", "derivative", "plugin", "opg", "none"),
  parameters = NULL,
  nsamples = 100,
  lags = 0:3,
  nboot = 1999L,
  cores = getOption("mc.cores", 2L)
)

Arguments

fit

A fit from ctFit, either backend. The projection and the scores it needs require backend = 'julia'.

covariates

Optional subject-level covariates, as for ctScoreTest, for the covariates family.

families

Which families to run; see Details.

gaps

Break points, in the data's time units, for the residual correlations' gap bins. By default the eighth, quarter and half quantiles of the time between every pair of a unit's rows.

periods

Periods, in the data's time units, for the periodic family: 24 for a daily cycle with time in hours. By default a scan.

tvcovariates

Optional time varying covariates for the tvcovariates family: a data frame with the model's subject id and time columns and one or more numeric columns, matched to the fit's rows by id and time.

close

The root mean square residual correlation regarded as a close fit.

alpha

Level for adequate and close_fit.

projection

How the error-based statistics account for estimation: 'derivative' differences the statistics along every parameter (two passes of the filter per parameter) and projects with the fit's Hessian; 'plugin', on a Laplace fit, takes those differences from plug-in passes, which filter each subject at effects solved once from the whole record rather than row by row – several times cheaper, and approximate (on a pilot-shaped model it recovered the distribution family's misfit, z 5.3 against 5.6 exact and 1.8 uncorrected). Where random effects reach the dynamics or the variances, either projects only the unit-level families (distribution, fit, effects, scores) and leaves the error-based ones unprojected: there their projected variance was unreliable, while unprojected they held their level on data generated from the model; 'opg' uses the outer product of the scores, valid when units outnumber parameters many times; 'none' ignores estimation, which is conservative, and for the distribution family can hide a strong misfit. 'auto' takes the first that is affordable: 'derivative' when the fit's Hessian is stored and the passes take under getOption('ctsem.adequacy.budget', 120) seconds, then 'plugin' on the same terms, 'opg' with at least ten units a parameter, else 'none'.

parameters

For the scores family, the raw parameters whose row scores are used: by default up to 16 of the dynamic and measurement parameters, those in DRIFT, DIFFUSION and CINT first.

nsamples

Number of generated datasets, for the families that need them. Reused from fit when it already carries at least as many.

lags

Observation lags for the lagged family.

nboot

Bootstrap draws for each family's calibration.

cores

Cores for the engine and for generating.

Details

residuals

Correlations of the errors within units, for every pair of rows, by time gap (gaps): the residual autocorrelation and cross-correlation in continuous time. Slow structure the model lacks shows as correlation over the gaps it spans.

latent

The same for each latent process: the update each row makes to the predicted state, standardised, correlated with the same subject's updates at other times. One per process rather than one per item, so a misfit in the dynamics is not diluted by item-specific structure.

volatility

The same for the squared errors, per variable: variance that clusters in time.

pairs

Each pair of variables' errors in the same row against the correlation the model implies: a wrong measurement error covariance or loading.

periodic

A cycle the model lacks, such as a daily rhythm: each variable's errors, and each latent's updates, correlated with the same unit's at other times weighted by the cosine of the time between them over a period – the periodogram of the errors at that period. It depends only on the time between rows, so it finds a cycle whatever its phase in each unit and needs no clock. At the named periods, or by default scanning the periods a typical record resolves, from half its length to twice the typical interval between rows. The estimate is the errors' autocorrelation at the period: a cycle of amplitude a, in error standard deviations, gives a^2 / 2.

distribution

Skewness and kurtosis of each variable's errors, and each row's squared whitened distance against its chi-square mean and variance, together with Anderson-Darling tests against the normal and chi-square.

fit

How unevenly units are fitted: the spread of each unit's summed distance against the chi-square's, and the worst unit.

dependence

The distance correlation of each variable's error, and of its square, with what came before – the predicted latent states, the previous errors, the time interval – by permutation: a conditional mean or variance the model gets wrong, in any form.

effects

On a single-level Laplace fit, whether the random effects' normal distribution is adequate: each subject's posterior probability, smoothed, that its standardised effect lies below a cut, at cuts along each axis, against the normal's. Under the model the posterior averages to the prior, so the differences have mean zero; a skewed or bimodal distribution, or a subgroup, moves them. These integrate the gradient function (Verbeke & Molenberghs, 2013; Drikvandi, Verbeke & Molenberghs, 2017), which is too heavy tailed to test directly.

tvcovariates

With tvcovariates, whether time varying variables the model does not use carry information about what it predicts: each variable's error, and each latent's update, against each covariate at the same row and at the unit's previous row, the covariate standardised. The estimate is the regression of the standardised error on the covariate. The previous row's value is a test of prediction (Granger's sense); the same row's is an association, which a covariate the outcome itself moves would show as well. Its null is larger than the other families': the errors have mean zero given what the model conditions on, the outcomes' own past, and need not be uncorrelated with anything else. A correct model of the outcomes fails it when a covariate is another measure of the same states, since that measure's previous value sharpens the estimate of the state and so predicts the next error. So it says whether a model using these variables would predict better, not that the model of the outcomes is wrong, and it is not counted in adequate.

scores

Not run unless asked for. White's (1987) dynamic information matrix test: each data row's score, the derivative of its one-step log likelihood, for the dynamic and measurement parameters (parameters), correlated with the scores of the same subject's other rows, by time gap. Under the model a subject's row scores are uncorrelated whatever the departure, so this is a check for dynamic misspecification of any kind; being that broad, over many statistics, it is the least powerful of the families, and the targeted tests of ctScoreTest (stability, state dependence) find what they cover far more often. It costs a dual pass of the filter per eight parameters, and on the Laplace route one per truncated record, which is hours on large data.

subjects, nonlinear, covariates, lagged

Statistics of the data against datasets generated from the fit: how much subjects differ in mean, standard deviation and lag-one correlations; whether change depends on level nonlinearly; with covariates, whether subjects' statistics depend on them; and the data's own lagged correlations (lags). These use the data twice and are conservative; the lagged correlations especially, as the fit reproduces them, so they are not run unless asked for.

The error-based statistics are moment tests: each unit's contribution is projected on its scores, which removes what estimating the parameters did to the statistic (projection), and the variance is the errors' own: they are independent over rows under the model, so a sum of products over pairs of rows has the sum of the products' squares as its variance, which holds with heavy tails or a changing variance; estimating the parameters takes a known part of it away. That is the variance of the residual, latent, volatility, within-row, periodic and covariate families. A periodic statistic is in each unit a quadratic form of rank two in its errors, the periodogram, skewed to the right however many units there are; its z is read through a chi-square of the skewness that form has under the model, worked out exactly from the unit's times. The distribution and fit families, whose statistics are sums of high powers, use the spread of the units' contributions and a studentised bootstrap over units. Within a family the largest standardised statistic is calibrated with the statistics' correlation over units, and the sum of their squares, where the variances are the model's, by its terms between units, whose mean is zero under the model (Chen & Qin, 2010), and otherwise by the bootstrap; the two are combined, with any exact p-values the family has, by the Cauchy combination; families are combined by Holm. adequate is TRUE when no family's adjusted p-value is below alpha. The tvcovariates family tests a larger information set than the model's and is held out of both: its p-value is its own, and informative reports it.

On a Laplace fit (intoverpop = 'laplace') whose random effects reach the dynamics or the variances, the errors come from the model's own predictive distribution: each row predicted from the subject's earlier rows with the random effects integrated against their posterior given those rows by importance sampling (getOption('ctsem.sequential.particles', 512) quasi-random draws a subject), the own variances integrated in the filter exactly. The draws are fixed by getOption('ctsem.sequential.seed'), so a call repeats; another seed, or more draws, shows how much of a result is the sampling's. The approximation the fit itself uses there – the posterior's mode on each truncated record, its uncertainty added by the delta method – is exact only for effects that enter linearly; elsewhere it put a 1.5 per cent bias into the predictive variance, and correct models were rejected through it. Where that predictive cannot be had – random effects at more than one level – the row distances are left out and the latent family's p-values are approximate, and the call says so.

A binary, ordinal, count or censored item's error is not normal, so its place is taken by its randomised probability integral transform through the normal quantile: the item's predictive distribution function at what was observed, given the subject's earlier rows and the items the filter had already applied in the row (non-Gaussian items one at a time, in item order), with a fixed uniform between its values just below and at the observation where that has probability mass. Under the model these are standard normal and independent, so every family reads them as it reads Gaussian errors; on the Laplace route they come from the exact predictive. The latent family's updates on such rows rest on the filter's Gaussian projection of each item, so its p-values are approximate there, and the call says so. Within a row, a non-Gaussian item's transform and a Gaussian item's error have no model-implied correlation the pairs family could test against, and are not paired.

With many observations every misfit is significant, so the sizes of the residual, latent and within-row correlations the model leaves are reported too, as root mean squares, each squared correlation less its sampling variance (as for the SRMR, Maydeu-Olivares 2017): the mean square is unbiased, its root, truncated at zero, bias-corrected. Each has a 95 per cent interval and p_close, the p-value of the test that it is at least close. close_fit is TRUE when every family that fails is one of those three and that test rejects at alpha – the size is shown to be below close, its one-sided upper bound under it. A size whose interval reaches close is not a close fit: not showing that a misfit is large does not show that it is small. A close fit is about these averages over many correlations; one large discrepancy among them, or a biased estimate of a quantity of interest, can sit under a small average.

Detection is not localization: a failure in one variable's equation reaches the others' errors through the fitted covariance. ctScoreTest is the more sensitive check for the departures it covers, and ctModelImprove uses both.

Value

An object of class ctFitAdequacy: families (one row per family: the number of statistics, the worst, its z, the family's p-value and the Holm-adjusted one), statistics (every statistic: what it is about – variable, with (the other variable of a pair, or what the variable is correlated with), gap (the gap bin of a pair of rows, or a lag), period and covariate, each NA where it does not apply – and its observed value, z and two-sided p), size (the root mean square residual correlations, their intervals and close-fit tests), errors (the rows the error-based families are computed from, in long form: Subject, Time, Element – 'error', each variable's standardised one-step-ahead error given the row's other observed values; 'update', each latent's standardised update; 'etaprior', its predicted mean – Row, the variable, and value; NULL when no such family ran), adequate, close_fit, informative (with tvcovariates, whether they predict the errors at alpha; NA without), projection (the method used), and fit (with any generated data, for reuse). plot() draws the families, or one family's statistics.

References

Newey, W. K. (1985). Maximum likelihood specification testing and conditional moment tests. Econometrica, 53, 1047-1070.

White, H. (1987). Specification testing in dynamic models. In T. Bewley (Ed.), Advances in Econometrics: Fifth World Congress. Cambridge University Press.

Verbeke, G., & Molenberghs, G. (2013). The gradient function as an exploratory goodness-of-fit assessment of the random-effects distribution in mixed models. Biostatistics, 14, 477-490.

Drikvandi, R., Verbeke, G., & Molenberghs, G. (2017). Diagnosing misspecification of the random-effects distribution in mixed models. Biometrics, 73, 63-71.

Chang, J., Yao, Q., & Zhou, W. (2017). Testing for high-dimensional white noise using maximum cross-correlations. Biometrika, 104, 111-127.

Chen, S. X., & Qin, Y.-L. (2010). A two-sample test for high-dimensional data with applications to gene-set testing. The Annals of Statistics, 38, 808-835.

Liu, Y., & Xie, J. (2020). Cauchy combination test. Journal of the American Statistical Association, 115, 393-402.

Maydeu-Olivares, A. (2017). Assessing the size of model misfit in structural equation models. Psychometrika, 82, 533-558.

See Also

ctPostPredPlots and ctFitCheckCov to see the data against the fit.


Visual lagged covariance or correlation diagnostics for ctsem fits.

Description

Compares empirical lagged covariances or correlations with the same quantity computed from data generated by the fitted model. For each requested lag, covariances or correlations are computed across subjects for each matching observation-index pair. By default (x = "lag") they are then averaged over observation-index pairs, which hides misfit that differs between waves, such as a variance that changes at one wave; x = "obs" or a data column instead shows each wave, or each value or bin of the column, as a point of its own. ctFitCovCheck is maintained as a backward-compatible alias.

Usage

ctFitCheckCov(
  fit,
  cor = TRUE,
  plot = TRUE,
  splitby = NULL,
  data = NULL,
  lags = NULL,
  variables = NULL,
  splitdata = data,
  split = NULL,
  breaks = 2,
  nsamples = NULL,
  minpairn = 10,
  cores = getOption("mc.cores", 2L),
  keep = c("summary", "samples"),
  x = "lag",
  xbins = 10,
  lagrows = c("all", "observed")
)

Arguments

fit

A fitted object from ctFit, from either backend.

cor

Logical; if 'TRUE' correlations are analysed instead of covariances.

plot

Logical; if 'TRUE' (default) a 'ggplot2' object (or list of plots) is returned. If 'FALSE', the raw 'data.table' with diagnostics is returned.

splitby

Optional character string giving the variable name on which to split subjects or observations. Numeric variables are split at the subject mean and then at the grand mean by default; non-numeric variables are treated as groups and may vary within subject.

data

Optional long-format data to use for the empirical lagged covariances or correlations. If omitted, the fitted data are used.

lags

Non-negative integer vector giving observation lags to compare. If 'NULL', '0:10' for 'x = "lag"' and '0:1' (variances or covariances, and lag-1 relations) otherwise.

variables

Optional character vector of manifest variables to include. Defaults to all manifest variables in the model.

splitdata

Optional data containing 'splitby', useful when the split variable was not in the fitted model. If omitted, 'data' is used when supplied, otherwise the fitted data are used.

split

Split rule for 'splitby': '"median"', '"mean"', '"quantile"', '"factor"', or '"none"'. If 'NULL', numeric split variables use '"mean"' and non-numeric split variables use '"factor"'.

breaks

Number of quantile groups when 'split = "quantile"'.

nsamples

Optional number of generated data sets to use. Existing generated data are reused when present; otherwise 'ctGenerateFromFit' is called.

minpairn

Minimum number of complete paired observations required for an empirical or generated lagged covariance/correlation: for each observation-index pair averaged under 'x = "lag"', and for each point otherwise. A point with fewer is left out of the plot and of the share outside the model's interval; raise it to drop bins that hold only a few off-schedule observations.

cores

Cores for generating the data when fit carries none: engine threads for a julia fit, and for a stan fit worker processes, which start only for a job large enough to repay them. The summary of the generated datasets is computed in this session.

keep

If '"samples"', draw-level model diagnostics are attached as the '"samples"' attribute of the returned data.

x

What the x axis shows. '"lag"' (default): one point per lag, averaged over observation indices. '"obs"': one point per observation index within subject (1 for each subject's first row), a panel per lag. Or the name of a column of 'data' (or of the fitted data), such as the time column or an age: one point per value of that column, or per quantile bin of it (see 'xbins'), a panel per lag. Each pair of observations is placed by its later observation, as a row-level split is.

xbins

Number of quantile bins for a numeric 'x' column with more distinct values than this; a column with no more is used as it is. The x axis shows each bin's mean.

lagrows

Which rows a lag counts. '"all"' (default): lag k pairs a row with the subject's k-th earlier row. '"observed"': with the k-th earlier row at which the column variable was observed, so a variable measured at only some waves is paired with its own previous measurement rather than with rows that hold only other variables. The gap in time between the two then varies more, and the model's data, which share the observed data's missingness, are paired the same way.

Value

Either a plot/list of plots (default, given by 'ctFitCovCheckPlot') or a 'data.table'.

See Also

ctPostPredPlots, the posterior predictive dashboard, whose LaggedCovariance panel is this function without the splitting.

Examples

# Lagged correlations for all manifest variables, lags 0 to 3.
ctFitCheckCov(ctstantestfit, cor = TRUE, lags = 0:3, plot = TRUE)

# Lagged covariances for one manifest variable.
ctFitCheckCov(ctstantestfit, cor = FALSE, variables = "Y1", lags = 0:5,
  plot = FALSE)

# Variances, covariances and lag-1 covariances at each wave rather than
# averaged over waves, and the same by bins of time.
ctFitCheckCov(ctstantestfit, cor = FALSE, x = "obs")
ctFitCheckCov(ctstantestfit, cor = FALSE, x = "time", xbins = 5)

# Lag each variable by its own previous observation, for variables
# measured at only some waves.
ctFitCheckCov(ctstantestfit, x = "obs", lagrows = "observed")

# Split the diagnostic by a time independent predictor in the fitted data.
gg <- ctFitCheckCov(ctstantestfit, cor = TRUE, splitby = "TI1", lags = 0:3)
print(gg[[1]])

# Use an external subject-level variable for the split.
splitdat <- unique(data.frame(ctstantestdat)[, "id", drop = FALSE])
splitdat$group <- rep(c("a", "b"), length.out = nrow(splitdat))
ctFitCheckCov(ctstantestfit, splitby = "group", splitdata = splitdat,
  lags = 0:2)

# Split observations into early and late periods within each subject.
dat <- data.table::as.data.table(data.frame(ctstantestdat))
dat[, period := ifelse(time <= stats::median(time, na.rm = TRUE),
  "early", "late"), by = id]
ctFitCheckCov(ctstantestfit, data = dat, splitby = "period",
  split = "factor", lags = 0:2)

ctFitCovCheckPlot

Description

Plot the results of ctFitCheckCov.

Usage

ctFitCovCheckPlot(x, maxlag = 10, vars = NA, splitvar = NA, cor = FALSE, ...)

Arguments

x

Data output from 'ctFitCheckCov(..., plot = FALSE)'. If a ggplot object or list of ggplot objects is supplied, it is returned unchanged. Data with an 'x' column (from 'ctFitCheckCov(x = )' other than '"lag"') are drawn against it, a panel per lag and variable.

maxlag

Maximum lag to plot.

vars

Optional character vector of variable names to plot. If 'NA', all variables are plotted.

splitvar

Optional character string specifying a variable to split the plot by.

cor

Logical; if 'TRUE', plot correlations instead of covariances. (label change)

...

not used.

Value

ggplot object.

Examples

## Not run: 
ctFitCovCheckPlot(ctFitCheckCov(ctstantestfit,cor=TRUE,plot=FALSE),maxlag=3)

## End(Not run)

Profile likelihood for a ctsem julia fit

Description

Fixes one raw parameter at a sequence of values, re-optimising all the others at each, and reports what the likelihood does. This answers the identification question directly rather than through the curvature at the estimate, which on a flat direction measures the transform rather than the data – see the note in ctFitUncertainty on flat directions.

Usage

ctFitProfile(
  fit,
  parameters = NULL,
  points = 8L,
  step = NULL,
  growth = 1.6,
  level = 0.95,
  maxiter = 500L,
  verbose = 0,
  cores = getOption("mc.cores", 2L)
)

Arguments

fit

a ctJuliaFit.

parameters

which parameters to profile: raw parameter names, indices, NULL for all of them, or "flagged" for only those the fit's own diagnostics already doubt – the saturated coordinates, the unidentified directions, and the intervals wider than the curvature supports. "flagged" is the cheap internal use; naming parameters is the usual one. Profiling every parameter of a large model is a great many optimisations.

points

maximum points per side.

step

first displacement, in raw units. NULL takes half of each parameter's own standard error where the fit has one, so an identified parameter gets several points inside its interval instead of crossing the bar on the first step; it falls back to 0.5 raw, which is a small move in ctsem's standardised raw coordinates, when there is no curvature to ask.

growth

factor the step is multiplied by after a point that moved the likelihood by less than the target, so a flat direction is walked out quickly rather than in a hundred equal steps.

level

confidence level setting the likelihood-ratio bar.

maxiter

iteration cap for each constrained optimisation.

verbose

0 for silence, 1 to report each point as it is computed.

cores

The most CPU cores the profile uses at once. The walks – each parameter's two sides – run one after another in this session, on that many engine threads. Only when the fit itself took more than five minutes to optimise do they run up to cores at a time instead, each in a background R session with a one-thread Julia (it needs the future package; the sessions persist for the next call, and ctJuliaKill('workers') releases them): a session costs up to a minute to start. The walks are independent, so the profile is the same either way, except that with several sessions every walk finishes before $better is reported.

Details

Each point is a constrained optimisation, so a profile can discover that the fit was not at a maximum. When that happens the run stops and the better point is returned in $better; refit from there and profile again.

Value

An object of class ctFitProfile, which print and plot both understand. $profile is one row per point (parameter, raw value, transformed value where the coordinate has one, log likelihood, drop from the estimate); $summary is one row per parameter (verdict, limits, how far each side was walked); and $better is NULL unless a constrained fit beat the estimate, in which case it holds that point and the run stopped there.

Examples

# Needs a working julia backend, so the example is inert where Julia is
# absent -- including on CRAN, whose check machines have none.
if (isTRUE(ctJuliaStatus()$available)) {
  model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1',
    LAMBDA = matrix(1))
  model$pars$indvarying <- FALSE
  fit <- ctFit(ctstantestdat, model, backend = 'julia', cores = 1)

  # Two named parameters; parameters = "flagged" profiles only those the
  # fit's own diagnostics doubt, and NULL profiles every one.
  result <- ctFitProfile(fit, parameters = c('drift_eta1', 'mm_Y1'))
  print(result)
  plot(result)
  # Julia keeps running between calls, holding what it has compiled;
  # this ends it.
  ctJuliaKill("session")
}

Compute or sample a fit's uncertainty

Description

Recomputes the approximate raw-parameter uncertainty for an optimized ctFit object and refreshes the approximate raw-parameter samples, or samples the posterior by Markov chain Monte Carlo. This is the entry point for both backends; ctFit itself calls it to finish an optimized fit. ctOptimUncertainty is the previous name, kept for code written against ctsem 3.11.1, with that release's arguments.

Usage

ctFitUncertainty(
  fit,
  method = c("hessian", "checked", "is", "sample", "sandwich", "opg", "bootstrap",
    "fullbootstrap", "stored"),
  intoverpop = NULL,
  filter = c("kalman", "particle"),
  checks = TRUE,
  finishsamples = NULL,
  cores = getOption("mc.cores", 2L),
  control = list(),
  verbose = 0,
  ...
)

ctOptimUncertainty(
  fit,
  uncertainty = c("hessian", "checked", "is", "bootstrap", "fullbootstrap", "sandwich",
    "opg", "stored"),
  draws = c("auto", "normal", "empirical", "imis"),
  finishsamples = NULL,
  cores = getOption("mc.cores", 2L),
  control = list(),
  verbose = 0,
  ...
)

Arguments

fit

Optimized ctStanFit or ctJuliaFit object, or for method = 'sample' a julia fit that is already sampled.

method

How the parameters' posterior is described: 'hessian' (the default), 'checked', 'is', 'sample', 'sandwich', 'opg', 'bootstrap', 'fullbootstrap' or 'stored'. See the Methods section, and Sampling for 'sample'. The kind of draws follows from it: normal draws from a curvature, importance-weighted ones for 'checked' and 'is', the bootstrap's own, MCMC for 'sample'.

intoverpop

How the random effects are integrated, in ctFit's words; NULL (the default) keeps the fit's own ('laplace' or 'augmented', as fit$args$resolved records). 'quadrature', for a Laplace fit, refines the fit from its estimate, with the units the random-effects check names integrated numerically – by adaptive Gauss-Hermite or lattice rules, whichever reaches the accuracy at the lower cost – and describes the posterior there, as ctFit(intoverpop = 'quadrature') does when fitting; with 'hessian', 'checked' or 'is'. With method = 'sample', 'none' samples the random effects jointly with the parameters, which takes no approximation of their integral, and 'laplace' the Laplace marginal (see Sampling).

filter

How the latent states are integrated: 'kalman' (the default), the fit's Kalman filter (extended where the model is nonlinear), or 'particle', a particle filter, the likelihood ctParticleLik measures the Kalman filter against; with method = 'is', on a fit whose random effects are not integrated by Laplace (see Methods).

checks

Which approximations to check where the method itself leaves them in place: TRUE (the default) for both, FALSE for neither, or one of 'randomeffects' and 'intervals'. The random-effects check (a julia fit on intoverpop = 'laplace') scores every unit's Laplace integral at draws of its random effects, and warns where integrating them exactly would move an estimate, naming intoverpop = 'quadrature'; a fit that has one on record is not checked again. The interval check (julia, with method = 'hessian') makes 'checked''s check of the curvature's normal against the posterior's gradients, on the density the intervals come from, records it in $uncertainty$details$normalcheck, and says in a message, and in summary(), when the posterior departs from that normal, naming method = 'sample'. Both checks take seconds on the fit's own objective. With intoverpop = 'quadrature' the interval check is made on the refined fit's objective, where each of its draws integrates the flagged units again: seconds on one level, but half an hour on a nested model whose refinement took minutes, longer than sampling; checks = 'randomeffects' leaves it out.

finishsamples

Number of approximate raw-parameter samples. If NULL, the existing number of rows in the fit's raw posterior (fit$stanfit$rawposterior for stan, fit$estimate$rawposterior for julia) is reused when available; otherwise 1000 samples are used. Not used by 'sample', whose draw count is control$draws.

cores

The most CPU cores the call uses at once, counting every process it starts. 'sample' shares them between at most that many worker processes, which run the chains. On a ctJuliaFit there is no R cluster; the value becomes the engine's subject-loop chunk ceiling for the duration of the call, and the engine caps it at its own thread count. A ctStanFit has no threads, so more than one core means R worker processes, which take seconds to start and are used only for work slow enough to repay that: the refits of 'fullbootstrap', and 'is' on data of a few thousand rows or more, where each log-probability evaluation is split across subjects. Everything else runs in this session. Neither route changes what is computed, though both change the order things are summed in, so results are reproducible at cores = 1 and agree to rounding above it.

control

List of method-specific options. accuracy (julia only), an accuracy in posterior standard deviations, applies to every method but 'sample': once the method has run, its own account of its error – for 'checked' and 'is' the Stein check's per-parameter mean error and spread – is read against it, and where the method falls short, or has no account ('hessian', the score methods), the fit is sampled to that accuracy ('sample' with minESS and rhatTarget set from it: 0.5 is a tail effective size of about 125 and R-hat 1.07, 0.3 about 350 and 1.025). The method reached is the one reported, and $uncertainty$escalation records what was read and done. For method = 'sample' the entries are the sampler settings described in the Sampling section (chains, warmup, draws, seed, saveEffects, processes, tipredMissing, sampler, particles, coreset, placement, minESS, meanESS, rhatTarget, maxDraws, settleTol, init_scale, and NUTS's maxdepth, target_accept, adapt_metric and adapt_effects); none of the entries below apply to it. For every other method, useful entries include ridge, hessianStep, bootstrapFitCores, bootstrapTol, imisMaxIter, imisScaleInit, imisTailScale, imisDf, isESS, and isitersize. Omitted entries use ridge = 1e-8, hessianStep = 1e-3, bootstrapFitCores = 1, bootstrapTol = 1e-5, imisMaxIter = 50 (100 for 'checked'), imisScaleInit = 1.1, imisTailScale = 1.1 (1.5 and 1.2 for 'is' on julia), imisDf = 5 (the proposal components' t degrees of freedom; Inf for normal), isESS = 200, and isitersize = 1000 (100 for 'checked'). imisBudget caps the log density evaluations importance sampling spends (no cap by default). imisComplement = TRUE ('checked') weighs the sampled directions with the others integrated over, rather than held at the centre with the volume their spread adds to first order: exact where the others' spread moves with them, as a large covariance's scale does, at up to seven times the evaluations. On a julia fit, importance sampling evaluates a weighted subset of the units chosen and checked as it runs, and corrects its weights where that subset strays; coreset = FALSE evaluates every unit instead. Where the check of 'checked' names a direction, the normal is first moved to the posterior's moments by doubly adaptive importance sampling (van den Boom, Cremaschi and Thiery 2026; julia fits, in the engine) and checked again there before anything is sampled; dais = FALSE skips that, daisDraws sets its draws an iteration (the dimension, between 64 and 256) and daisMaxIter its iterations (150). parsteps keeps stepwise-fixed raw parameters fixed (stan only). analyticHessian = FALSE (julia only) asks for the finite-difference Hessian instead of the engine's exact one. flat = 'natural' ('checked' and 'is', julia only) makes the parameters without a prior flat on their natural scale rather than their raw one in the density sampled: the standard deviation itself rather than its log, say, so that a likelihood that levels off as an sd goes to zero still has a proper posterior. The estimate is unchanged. With intoverpop = 'quadrature' or filter = 'particle', nodes, particles, substeps, transition, seed and correctEstimate are as described under Methods. A Hessian covariance that needs numerical repair – positive-definite projection, ridge flooring, or MASS::ginv – warns, with diagnostics in $uncertainty$details$covariance.

verbose

Integer controlling progress detail.

...

Not used. Anything passed here is an error rather than ignored: a method's settings are entries of control.

uncertainty, draws

ctOptimUncertainty() only, the names of ctsem 3.11.1: uncertainty is method; draws ('auto', 'normal', 'empirical' or 'imis') overrides the kind of draws the method makes.

Details

Every method except 'sample' writes pseudo-posterior draws to $rawposterior: a sample from a covariance fitted to the log-posterior surface around the optimum, not a sample from the posterior itself, and the point estimate stays at the optimum – unless the method is taken against a more exact likelihood than the fit's (intoverpop = 'quadrature', filter = 'particle'), when it moves to the corrected point. method = 'sample' is the other thing, genuine posterior draws by Markov chain Monte Carlo, and it moves the point estimate too: $estimate$raw becomes the posterior mean, the way ctFit(optimize = FALSE) reports a sampled fit. Everything downstream reads either kind of draw from the same slot, so the difference is recorded rather than visible in the shape of the result: a curvature-based fit carries $uncertainty$settings, a sampled one carries $sample. Every method except 'sample' refuses a fit that already carries a posterior rather than replacing its draws with an approximation; 'sample' may be run again on a sampled fit, to draw more, or differently, from where it stands.

To change only the number of draws, use method = 'stored', which redraws from the covariance the fit already carries and evaluates no model.

Value

The fit, of the class it came in as. For every method except 'sample': the resolved method, draw strategy, sample count, cores, and non-internal controls are recorded in fit$stanfit$uncertainty$settings for a ctStanFit and in fit$uncertainty$settings for a ctJuliaFit; see Backend differences for the other slots each writes. For method = 'sample': a ctJuliaFit with estimate$rawposterior holding the draws and $sample holding the chain diagnostics (split R-hat and effective sample size per parameter, divergences, tree depths, step sizes, E-BFMI (NUTS only), sampler naming the kernel, converged and diagnosis, and target naming which posterior was sampled), exactly as ctFit(optimize = FALSE) returns.

Functions

  • ctOptimUncertainty(): The name this function shipped under in ctsem 3.11.1, kept so that a call written against that release keeps meaning exactly what it did. The same as ctFitUncertainty in every respect but its argument names (uncertainty, draws) and that it has neither intoverpop, filter, checks nor 'sample'; new code should prefer ctFitUncertainty.

Methods

'checked'

The Hessian's normal, checked, and corrected by importance sampling in the directions where it is wrong. The log posterior's gradient at draws from the normal says in which directions the posterior departs from it, in batches of 8 until the directions named would be the same with every departure two standard errors larger or smaller, and the directions named are fewer than half the draws (16 draws at least, and at most twice the parameters, between 64 and 256). Where a direction would leave some parameter's 95% interval covering under 90% or over 99%, or move one tail outside 0.5 to 5% by a skew or by a width that moves with another parameter, every departing direction is sampled as 'is' samples (below), the normal kept in all the rest. This is meant to correct a variance from few groups and the effects it scales, but it weighs the sampled directions with the others held near the centre, so where their spread moves with the sampled ones – a population sd and the effects it scales – intervals can remain too narrow, and nothing below detects it: on models with random effects, sample ('sample') for intervals to report. Where that sampling cannot correct the normal – a curved ridge such as a loading vector whose length the data determine and whose direction they barely do, where the Hessian's intervals can be many times too wide – the Hessian's normal is kept, with a warning naming the parameters it misstates. The normal is centred where the same gradients, and those at the draws reflected (at least 32 pairs), put the posterior's mean, which is not the estimate wherever the posterior is skewed or a covariance is estimated; the estimate stays the fit's, and summary() reports it beside the draws. Where nothing is sampled, the same pairs' widths replace the Hessian's along the directions they resolve, where they move an sd by 5% or more. The sampled directions are weighed with the volume the others' spread adds to them, to first order, and the sampled draws are checked against the posterior's gradient at 100 of them (Stein's identity): where a mean is off along the sampled directions, sampling goes on from a component the gradient places (twice at most), and a warning names the parameters whose mean the gradient puts half a normal sd or more away, or whose spread it puts under half or over twice the posterior's. A fit with nothing named gets the Hessian's covariance, so corrected. The check is recorded in $uncertainty$details$normalcheck, and the sampling, when there was any, in $uncertainty$details$importance_sampling.

'hessian'

The default, as ctFit's: a normal approximation from the Hessian at the estimate – exact on julia, which differentiates its own gradient, and by finite differences of the gradient on stan. On julia the interval check (checks) is made beside it, and the normal is reported as it is.

'bootstrap', 'sandwich', 'opg'

Built from per-subject score contributions: one-step score bootstrap draws with Hessian bread, Hessian bread with score covariance meat, and the outer product of the scores. A single-subject model uses case-level contributions, with a warning; fewer than ten subjects, or no more score rows than parameters, also warn.

'fullbootstrap'

Resamples subjects and re-optimizes each resample from the estimate with mize L-BFGS. Stan only.

'is'

Adaptive importance sampling against the fit's own objective. Each raw parameter is first walked out on both sides of the mode, from 3 to 96 standard errors, with the other parameters at their conditional mode, and a proposal component is placed wherever that walk finds a tail the curvature does not show – a skewed one, or one bending along a ridge, as a variance with few subjects often has. Rounds of isitersize draws from these multivariate t components then add components at the highest-weighted draws, and up to three more where the weights show the proposal falls short, each moved by a Newton step and shaped by the log posterior's curvature there, until the effective sample size reaches isESS and the Pareto k of the weights is below 0.7 (k needs the loo package). The effective size, k, the evaluations used and the parameters with a tail are recorded in fit$uncertainty$details$importance_sampling; k above 0.7 warns, and so does a parameter whose walk has not fallen off by 96 standard errors, which is what an improper posterior looks like. A rank deficient Hessian covariance is sampled in the directions it has curvature in, holding the rest at the estimate.

On a fit whose random effects are integrated (intoverpop = 'laplace' or 'augmented') the objective is the route's approximate marginal posterior, so by default 'is' corrects the shape of the normal approximation but keeps the route's own approximation: where the Laplace approximation is biased – random effects on variances, in nonlinear dynamics, or with non-Gaussian indicators – 'is' reproduces the bias. Expect it to cost several times the fit.

intoverpop = 'quadrature' (julia, a Laplace fit) draws against a more exact density, with 'is' or 'checked': the quadrature hybrid, which has the units the screen flags, and those the fit's check named, integrated by quadrature rules (control$nodes per effect, 5 by default) or draws, the rest by their Laplace terms, with the error of that density reported in fit$laplace$correction$error and in summary(). A fit not refined when it was fitted – the default, which only checks and warns – is climbed to that density's optimum first; one refined when it was fitted stays where it is, as does any fit under correctEstimate = FALSE. With 'quadrature', 'hessian' is taken too: the hybrid's curvature where the climb ends, which is what ctFit(intoverpop = 'quadrature') gives at fit time, with the interval check made on the hybrid there. Where the climb stops short of its optimum, the intervals are the Laplace objective's at its optimum, moved with the estimate, as at fit time. filter = 'particle', for a fit on the filter's route and with 'is', replaces the filter's Gaussian likelihood by a particle filter's (control$particles, 1000; substeps, 10; transition, 'exponential' or 'euler'; seed, 1, with draw i using seed + i - 1), the likelihood ctParticleLik measures the filter against; its Monte Carlo noise widens the weights without biasing them, each draw costs one particle-filter evaluation, the proposal starts from the fit's covariance widened by imisScaleInit (1.5) and runs at most imisMaxIter (10) rounds of isitersize draws, and the estimate moves to the weighted posterior mean unless correctEstimate = FALSE. What was done is recorded in fit$laplace$correction or, printing, fit$particle_correction. Where too few effective draws survive the approximation is too far from the target to repair by reweighting, and 'sample' with intoverpop = 'none' is the answer instead.

'stored'

Computes nothing: redraws finishsamples normal draws from the covariance already on the fit, leaving the covariance, the Hessian and the recorded method untouched, and marks $uncertainty$settings$redrawn = TRUE. It warns if the existing draws came from 'is' or a bootstrap, which normal draws from that covariance do not reproduce.

'sample'

Markov chain Monte Carlo; see the next section. backend = 'julia' only.

Every method that produces draws aims for the same effective sample size unless told otherwise, 200: minESS for 'sample' (of the worst parameter, since the 2.5% and 97.5% quantiles need it for each) and isESS for 'is', whatever its likelihood. Each warns when it ends short of its target.

Sampling (method = 'sample')

Draws from the posterior by Markov chain Monte Carlo, through the same runner ctFit(backend = 'julia', optimize = FALSE) uses, and with the same defaults. Its settings are entries of control, and draws and finishsamples are not used: chains (4), warmup (200; not used by the map sampler, which learns its map over warm-up windows of its own first), draws (800, after warmup), seed, saveEffects (FALSE; whether to keep every draw of every random effect rather than only their summary – either way they are each parameter's deviation from its population value on the raw scale, per subject or group, in fit$sample$effects, effect_mean and effect_sd) and processes (FALSE).

What is sampled is what the fit's intoverpop says, read as ctFit(optimize = FALSE) reads it: the filter's marginal for 'augmented', the Laplace marginal for a fit that named 'laplace' (an approximate posterior whose dimension does not grow with the number of subjects), and the joint posterior over population parameters and every subject's random effects for 'none' or for a Laplace route that intoverpop = 'auto' chose, where the Laplace fit only places the chains. The argument intoverpop overrides it in the same words: 'none' for the joint posterior or 'laplace' for the Laplace marginal on a Laplace fit; 'augmented' needs a fit made that way, and an augmented fit takes nothing else. Missing TI predictor values that the fit imputed are sampled too, unless the fit's optimcontrol$tipredMissing or control$tipredMissing is 'impute'.

The kernel. control$sampler chooses it for the joint posterior. By default it is 'map'. 'saem' draws the random effects by SAEM's Metropolis-within-Gibbs sweeps and the parameters given them by NUTS, with moves that re-express the effects for each level's means and scales; 'nuts' runs NUTS on the parameters and every random effect at once; 'map' runs NUTS through a learnt map, with the random effects the Kalman filter can integrate exactly (on intercepts, initial means, manifest means and time dependent predictor effects, in a linear Gaussian model) integrated out and the rest partially non-centred, unit by unit, from Gaussian summaries of each subject's likelihood learnt in warm-up; it starts from the fit's estimate (placement = 'fit') and reports every random effect, the integrated ones drawn from their exact conditional. All draw the same posterior. In comparisons on variance, nested, categorical and nonlinear models the map reached the default effective sample size where SAEM's kernel often ran out of draws, and was usually the faster; SAEM's kernel was the more reliable and the faster than 'nuts', which mixes poorly on the joint vector. A marginal target is sampled by tempered sequential Monte Carlo where the interval check of 'checked' named directions in which the posterior departs from the fit's Hessian, and by NUTS where it named none, or where the call sets a control only NUTS reads (chains, warmup, stepsize, adapt_metric, adapt_effects, settleTol); sampler = 'nuts' or 'smc' chooses, and 'smc' beside chains, warmup or stepsize is an error, since it reads none of them. A fit made without that check gets one first, at a few hundred gradients. NUTS starts from a metric set to the widths the check measured along those directions, or, where 'checked' moved the normal to the posterior's moments, to that normal's covariance. Tempered SMC carries draws from that normal to the posterior by stages of NUTS moves, particles (200) chains a stage, keeping every state they visit, then runs as many chains on as the effective sample size target needs. It is for posteriors NUTS does not converge on from the normal – a mass far from the mode, a loading shell – and costs about what NUTS does in gradients. Its particles move on a coreset of the units (coreset, TRUE): the full density, evaluated once a particle a stage, corrects the moves exactly, so a fit with many units pays a fraction of a gradient a move and coreset = FALSE is the oracle. Its chains are short and start from particles already drawn from the posterior, so it reports no R-hat; the effective sample size is its stop. NUTS too runs its trajectories on a coreset, fitted before warm-up and again during it, each draw's end accepted or refused by the full density, which keeps the draws exact whatever the coreset. adapt_metric, adapt_effects and settleTol belong to NUTS's warmup and are refused with SAEM's kernel and with tempered SMC, and the latter two with the map. control$placement says where the chains start for SAEM's kernel and NUTS: 'saem', the default on the joint posterior, runs SAEM from the fit's estimate and starts each chain from SAEM's estimate and its draws of the random effects; 'fit' starts them around the fit's own estimate, and is the only placement for a marginal target.

When it stops. A run stops once every parameter's effective sample size reaches minESS (200) and the chains' R-hat is below rhatTarget (1.01) – meanESS adds a target for the average – within a budget of four times draws per chain, or maxDraws when given; minESS = 0 takes exactly draws. A run that reaches its budget short of the target warns. settleTol ends warmup early once the metric stops moving (off by default). Lower targets give a quicker, approximate posterior, and say how approximate: a reported quantile's Monte Carlo error is about 1.96 * 2.1 / sqrt(ESS) posterior sds at the 5% and 95% points and 1.96 * 1.25 / sqrt(ESS) at the median, and chains whose means disagree by d posterior sds give an R-hat of about sqrt(1 + d^2). So minESS = 50, rhatTarget = 1.07 aims at about half a posterior sd at the centre and more in the tails, and the defaults at about a fifth at the centre and two fifths in the tails. Short runs are usually limited by whether the chains agree, so R-hat is the binding check.

Processes. By default the chains run at once in this session, sharing its cores. With processes = TRUE they run in up to cores worker processes instead, each starting its own Julia session – a fixed cost that a short run does not recover – and a worker runs its share of the chains in turn, so four chains on two cores run two at a time. ctJuliaKill('workers') releases the worker processes, which persist for the next call.

Backend differences

The methods are the same on both backends and the covariance they produce is comparable, but some things differ and are worth knowing before comparing output.

Where the result is stored. A ctStanFit comes back with fit$stanfit$cov, fit$stanfit$rawposterior and fit$stanfit$uncertainty; a ctJuliaFit with fit$estimate$cov, fit$estimate$se, fit$estimate$rawposterior and fit$uncertainty. Both record the resolved settings in $uncertainty$settings. On both backends the draws, the covariance and the standard errors are labelled by raw parameter, as is coef(fit), so fit$estimate$se['drift'] or apply(fit$estimate$rawposterior, 2, quantile) reads without counting columns. The two backends spell the population-SD and correlation blocks differently (popsd_x/rawcor_x__y on julia, x_SD/y_x_corr on stan); the order is the same.

The Hessian. The stan path finite-differences its gradient with a single global step (control$hessianStep). The julia engine differentiates its own reverse-mode gradient in forward mode, so its Hessian is exact; control$analyticHessian = FALSE falls back to the shared finite difference for comparison. The two agree to about 1e-4 on a well-conditioned model.

Directions with no curvature. A direction whose curvature is negligible against the sharpest one is left out of the inversion rather than floored at control$ridge, so the parameters that carry it come back with no spread at all instead of a standard error of 1/ridge. Floored, that number leaks into the parameters the data does determine, by an amount rounding decides. A direction with little curvature along which the likelihood itself is measured flat – within the likelihood-ratio bar of 1.92 over four raw units on one side, following the ridge where it curves – is treated the same way. $uncertainty$cov records which directions were dropped, and fit$identifiability names the parameters.

Whether the intervals are as wide as the curvature allows. A julia fit carries fit$uncertainty$intervalcheck: for each raw parameter, the reported standard error against 1/sqrt(information[i,i]), the width that parameter's own curvature supports. Their ratio is 1 when the parameter is separable from the rest and grows without bound as it stops being; anything past about 100 means the reported width comes from the entanglement rather than from the data, and will not repeat between runs. $nullmass is each coordinate's share of the directions left out of the inversion, and $unidentified names those with enough of it that their true asymptotic variance is infinite; summary() reports their sd, interval and z as NA rather than as precisely estimated.

Backend-specific arguments. method = 'fullbootstrap' and its control$bootstrapFitCores / control$bootstrapTol, and control$parsteps, are stan-only and are refused by name on a ctJuliaFit; method = 'sample' and control$analyticHessian are julia-only. cores means R worker processes on stan and engine threads on julia. The IMIS proposal defaults also differ: imisScaleInit = 1.1 and imisTailScale = 1.1 on stan; on julia the same for 'checked', whose proposal starts at the widths its check measured, and 1.5 and 1.2 for 'is', whose proposal starts at the Hessian curvature, the wider pair being what a small-sample posterior wider than that curvature needs to be seen at all. Set them explicitly to compare the two. Transformed-parameter summaries are refreshed on stan and not on julia; the raw-scale covariance and draws are complete on both.

See Also

ctFitAddSamples is the deprecated stan-only predecessor of method = 'stored'. ctFit for optimize = FALSE, which reaches method = 'sample' through the same pipeline that placed the fit being sampled. ctLaplaceCheck and ctParticleLik measure the approximations a julia fit's likelihood makes, which intoverpop = 'quadrature' and filter = 'particle' correct.

Examples

# Redraw from the covariance already on the fit, evaluating no model:
fit <- ctFitUncertainty(ctstantestfit, method = 'stored',
  finishsamples = 200)
fit$stanfit$uncertainty$settings$redrawn


# The Hessian's normal, checked, and importance sampled where it is wrong
fit <- ctFitUncertainty(ctstantestfit, method = 'checked')
summary(fit)$popmeans

# Needs a working julia backend, so the example is inert where Julia is
# absent -- including on CRAN, whose check machines have none.
if (isTRUE(ctJuliaStatus()$available)) {
  model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1',
    LAMBDA = matrix(1))
  model$pars$indvarying <- FALSE
  jfit <- ctFit(ctstantestdat, model, backend = 'julia', cores = 1,
    priors = TRUE)
  # Posterior draws by MCMC, with lower targets for a quick look
  jfit <- ctFitUncertainty(jfit, method = 'sample', cores = 1,
    control = list(chains = 2, minESS = 50, rhatTarget = 1.05))
  jfit$sample
  # Julia keeps running between calls, holding what it has compiled;
  # this ends it.
  ctJuliaKill("session")
}

Update a fit to new data, or to the current version of ctsem

Description

Either to include different data, or because you have upgraded ctsem and the internal data structure has changed. Works on a fit from either backend.

Usage

ctFitUpdate(oldfit, data = NA, recompile = FALSE, refit = FALSE, ...)

Arguments

oldfit

fit object to be updated, from ctFit with either backend.

data

replacement long format data object. If not supplied, the data the fit was made with.

recompile

whether to force a recompile of the Stan model – safer but slower and usually unnecessary. Stan fits only.

refit

if TRUE, refits the model using the old estimates as a starting point. Only applicable for optimized fits, not sampling. If FALSE, the old fit is returned with its data and model specification replaced; its estimates, and everything computed from them when it was fitted, are left as they were.

...

extra arguments to pass to ctFit

Details

The fit is rebuilt from its own call: the arguments it was made with are passed to ctFit again, with the model it was given and, unless data is supplied, the data it was fitted to. Arguments in ... replace the fit's own.

With refit=FALSE and data, the data are instead prepared under the model the fit ran, so the fit keeps its structure along with its estimates whatever the data – a single subject included, which ctFit would set up without random effects. To predict for other data, ctPredict(fit, newdata = data) does the same without an update.

Value

updated fit object, of the same class as oldfit.

Examples

newfit <- ctFitUpdate(ctstantestfit,refit=FALSE)

ctGenerate

Description

This function generates data according to the specified ctsem model object.

Usage

ctGenerate(
  ctmodelobj,
  n = 100,
  burnin = 0,
  dtmean = 1,
  logdtsd = 0,
  dtmat = NA,
  Tpoints = NULL,
  wide = FALSE,
  backend = c("auto", "r", "julia", "stan"),
  intoverstates = "auto",
  fromPriors = FALSE,
  nsamples = 200,
  datastruct = NA,
  cores = getOption("mc.cores", 2L),
  values = NULL,
  n.subjects
)

Arguments

ctmodelobj

ctsem model object from ctModel.

n

Number of subjects to output. For a model with grouping levels above the subject (ctModel(id = c('subject', 'study'))), one count per level in the same order, innermost first: c(100, 10) is 100 subjects split evenly over 10 studies. A level not given has one group.

burnin

Number of initial time points to discard (to simulate stationary data)

dtmean

Positive numeric. Median time interval (delta T) to use. Intervals are drawn as exp(rnorm(n, log(dtmean), logdtsd)), so dtmean is the median rather than the mean whenever logdtsd is non-zero: the mean interval is dtmean * exp(logdtsd^2/2), which at logdtsd = 0.6 is about 20 percent longer than dtmean.

logdtsd

Numeric. Standard deviation of the log time interval. Zero gives an equal-interval design.

dtmat

Either NA, or numeric matrix of n rows and Tpoints-1 columns, containing positive numeric values for all time intervals between measurements. If not NA, dtmean and logdtsd are ignored.

Tpoints

Optional number of time points to generate. If supplied, this overrides any Tpoints stored in ctmodelobj. If not supplied, ctGenerate uses ctmodelobj$Tpoints when available.

wide

Logical. Output in wide format?

backend

Which generator to use: 'r', 'julia', or 'auto' (the default). 'julia' generates from the model itself: each subject's parameters are drawn from the population at every level, its latent trajectory is simulated from the process, and each observation is drawn from its measurement model, so nonlinear dynamics, random effects on any parameter and binary, ordinal or count indicators are all generated exactly. It needs ctmodelobj to be a ctModel(type='ct'/'dt') object rather than the matrix-list form. 'r' integrates a linear Gaussian system with a matrix exponential at the values the model states, with individual differences only through TRAITVAR and MANIFESTTRAITVAR, and fills free parameters with fixed defaults; it refuses a nonlinear model and has no measurement link. 'auto' uses 'r', as ctGenerate always has, unless the model or the call needs 'julia': a nonlinear model, a non-Gaussian indicator, more than one count in n, or values. On a model that declares random effects or TI predictor effects, 'r' says what it leaves out.

intoverstates

For backend='julia': 'auto' (the default, which is FALSE), TRUE or FALSE. A diagnostic handle rather than a modelling choice.

FALSE samples the latent trajectory from the process and then each observation from its conditional distribution given the state at its row: a draw from the model. TRUE instead draws each row from the filter's one-step-ahead predictive, for the same subjects, which is the density a fit integrating over the states maximises. The two agree in distribution for linear dynamics with Gaussian indicators and differ otherwise, which is what makes the comparison useful.

backend='r' already generates this way and ignores the argument.

fromPriors

Draw the parameters from the model's priors rather than using the values the model specifies, and return nsamples datasets instead of one. This is the prior predictive: what the model says the data could look like before it has seen any. It needs no fitted model and no data – with no datastruct, a balanced design of n subjects at Tpoints occasions dtmean apart is used.

nsamples

With fromPriors=TRUE, how many datasets to draw.

datastruct

A long format data frame giving the design to generate for, in place of the balanced one n, Tpoints and dtmean build: the subject (and any group) ids and times, TD and TI predictor values, and which observations exist – a manifest cell that is NA stays missing, any other value or a manifest column left out is generated. A predictor the model names and datastruct lacks is zero (TD) or drawn standard normal per subject (TI). The same data frame is returned with the generated values filled in. Needs backend='julia', or fromPriors=TRUE, where a fit passed as ctmodelobj brings its own.

cores

With fromPriors=TRUE, cpu cores to use.

values

For backend='julia': the population values to generate from, named and scaled as summary() reports them, so a model written for fitting can generate data at chosen values. A named numeric vector gives free parameters' values on their natural scale, e.g. c(mm = 2, drift = -0.3). A list gives summary()'s sections: popmeans (the same), popsd (random-effect sds on the natural scale, named by parameter), rawpopcorr (correlations of the random effects, named 'a__b') and tipreds (linearised TI predictor effects, named 'tip_<predictor>_<parameter>'), as in list(popmeans = c(mm = 2), popsd = c(mm = 0.5), tipreds = c(tip_TI1_mm = 1)); with grouping levels, popsd.<level> and rawpopcorr.<level> name a level's. What is not given takes its prior's centre: raw zero for a mean, sdscale for a raw sd, zero for a correlation or a free effect. A value fixed in the model cannot also be given here.

n.subjects

Deprecated; use n.

Details

Covariance related matrices are treated as Cholesky factors. TRAITTDPREDCOV and TIPREDCOV matrices are not accounted for, at present. The first 1:n.TDpred rows and columns of TDPREDVAR are used for generating tdpreds at each time point.

Value

From the R generator, a long format matrix with one row per subject and time point, or wide format with wide = TRUE. From the julia generator, the same long format, carrying the attributes described under Generating values. With fromPriors = TRUE, a list: Y, an nsamples by data rows by manifest variables array of generated data, and llrow, the matching array of row log likelihoods.

Generating values

With backend='julia' every value the model leaves free is fixed by the scale of its prior, and all of them are named in a message:

  • a free parameter is at its prior's centre, raw zero, unless values names it;

  • an individually varying parameter's population sd is its sdscale, at every level it varies at, on the raw, untransformed scale, unless RAWPOPVAR (RAWPOPVAR_<level>) states a number, and random effects are uncorrelated unless it states a coordinate – ctCov writes one from a covariance;

  • a time independent predictor is drawn standard normal per subject, and an effect fixed in the model ('mm||||TI1=0.5') shifts the raw parameter by that much per unit of the predictor; an effect left free is zero.

fromPriors=TRUE draws all of these from the priors instead.

The julia generator also returns what it generated from, as attributes: attr(, 'subjectpars'), each subject's parameters on their natural scale with its id and groups; attr(, 'grouppars'), for a model with grouping levels, each group's parameters at each level – its own effects and those above it, as a subject of average deviation within it would have; and attr(, 'states'), the latent states at every row by id and time (not with intoverstates=TRUE, which draws observations from the filter rather than states).

Examples

#a 2 process model, each process measured by a noisy indicator, with stable
#individual differences in process levels
generatingModel<-ctModel(Tpoints=8,n.latent=2,n.TDpred=0,n.TIpred=0,n.manifest=2,
 MANIFESTVAR=diag(.1,2),
 LAMBDA=diag(1,2),
 DRIFT=matrix(c(-.2,-.05,-.1,-.1),nrow=2),
 DIFFUSION=matrix(c(1,.2,0,4),2),
 CINT=matrix(c('cint1, indvarying=TRUE, sdscale=0.05',
   'cint2, indvarying=TRUE, sdscale=0.08'),nrow=2),
 MANIFESTMEANS=matrix(0,ncol=1,nrow=2),
 T0MEANS=matrix(0,ncol=1,nrow=2),
 T0VAR=diag(1,2))

#the R generator: every subject at the values the model states, with free
#cells filled by fixed defaults and no individual differences
data <- ctGenerate(generatingModel,n=15,burnin=10)


#the julia engine: each subject's CINT drawn from the population, with the
#means stated on the natural scale and the sds given by sdscale on the raw
#scale (CINT is 10 * raw, so 0.5 and 0.8)
if (isTRUE(ctJuliaStatus()$available)) {
  data <- ctGenerate(generatingModel,n=15,burnin=10,
    backend='julia',values=c(cint1=1,cint2=0))
}


#the prior predictive: what the model says data could look like before it
#has seen any. No fit, and no data.

priorpred <- ctGenerate(generatingModel, fromPriors = TRUE, Tpoints = 6,
  n = 10, nsamples = 20, cores = 2)
str(priorpred$Y)
# backend = 'auto' generates with julia where it is available. Julia keeps
# running between calls, holding what it has compiled; this ends it.
if (isTRUE(ctJuliaStatus()$available)) ctJuliaKill("session")

Generate data from a fitted model

Description

Adds a $generated object to a fit, holding random data generated from the fit's posterior (or its estimate), on either backend. ctStanGenerateFromFit is maintained as a backward-compatible alias.

Usage

ctGenerateFromFit(
  fit,
  nsamples = 200,
  fullposterior = FALSE,
  verboseErrors = FALSE,
  cores = getOption("mc.cores", 2L),
  intoverstates = "fit",
  effects = c("population", "fitted")
)

Arguments

fit

A fit from ctFit, on either backend.

nsamples

Positive integer specifying number of datasets to generate.

fullposterior

Logical indicating whether to sample from the full posterior (original nsamples) or the posterior mean.

verboseErrors

if TRUE, print verbose output when errors in generation encountered.

cores

Number of cpu cores to use: R worker processes for a stan fit, and for a julia fit the engine threads its draws are generated on.

intoverstates

For backend='julia' fits: which route generates the latent states. 'fit' (the default) mirrors the fit, so a state-explicit fit gets a state-explicit predictive and every other fit generates through the filter's one-step-ahead predictive: the density the fit worked with, and the identity that a generated dataset's llrow is the likelihood reported while generating it.

FALSE simulates the model exactly whatever the fit did: the trajectory from the process at each subject's own parameters, then each observation from its measurement model given the state at its row, as ctGenerate does. The two differ for a non-Gaussian indicator – where the filter's measurement update is an assumed-density projection – or for state-dependent dynamics. Comparing data generated each way, with the same effects, separates the fitting approximation from model misspecification, which comparing either with the observed data tests. TRUE forces the filter route.

Either way the states are regenerated rather than reused.

effects

For backend='julia' fits with random effects: 'population' (the default) draws new subjects – and groups, at every level – from the fitted population distribution at each parameter draw, so the generated data are the model's prediction for a new sample from the same design. 'fitted' keeps the fitted subjects: each at its sampled effects when the fit sampled them and saved the draws, otherwise at its conditional mode. A fit that integrates the random effects as states (intoverpop='augmented') carries no per-subject effects, so only 'population' applies there. Stan fits generate as the fit was made and refuse the argument.

Value

The fit, with a $generated element: Y, the generated data as an array of sample by data row by manifest variable, on the original time and missingness structure, and llrow, each generated row's log likelihood.

Examples

gen <- ctGenerateFromFit(ctstantestfit, nsamples=3,fullposterior=TRUE,cores=1)
plot(gen$generated$Y[3,,2],type='l') #Third random data sample, 2nd manifest var, all time points.

Generate data from a model's priors (deprecated)

Description

Deprecated. Use ctGenerate(fromPriors = TRUE), which is the same computation reached through the function that generates data from a ctsem model generally. ctStanGenerate is a backward-compatible alias and is deprecated with it.

Usage

ctGenerateFromPriors(
  cts,
  datastruct = NA,
  is = FALSE,
  fullposterior = TRUE,
  nsamples = 200,
  parsonly = FALSE,
  cores = getOption("mc.cores", 2L),
  backend = c("auto", "julia", "stan"),
  n.subjects = 20,
  Tpoints = NULL,
  dtmean = 1
)

Arguments

cts

A model – ctModel or ctModelConvertOMX – or a fit from either backend, which is simply a model that arrives with a design attached. Nothing here needs a fit: generating from the prior needs the model and a set of subjects and times to generate over, and a fit is one way to supply both.

datastruct

Long format data structure as used by ctsem, giving the subjects, times and missingness to generate for. Ignored when cts is a fit, which carries its own. Left at NA for a model, a balanced design of n.subjects subjects observed at Tpoints occasions dtmean apart is built instead.

is

Deprecated and ignored, with a warning if set. Importance sampling reweights draws taken from a gaussian approximation onto a target that cannot be sampled directly. Fitted to an empty dataset there is no such target: the likelihood contributes nothing and what remains is the prior, which ctsem holds as independent univariate densities in the raw space these draws are taken in. There is no approximation to correct.

fullposterior

Generate from the full prior, or from its mode (the raw origin)?

nsamples

How many samples to generate?

parsonly

If TRUE, only return samples of raw parameters, don't generate data.

cores

Number of cpu cores to use.

backend

Which engine turns a parameter draw into data: 'auto' (the default – 'julia' when a julia session is available, 'stan' otherwise), 'julia' or 'stan'. The draws themselves do not depend on this: they come from the model's prior, which is the same object either way. parsonly=TRUE returns those draws and generates nothing, so it does not use this argument.

n.subjects, Tpoints, dtmean

The design to build when cts is a model and no datastruct is given, and unused otherwise. Tpoints falls back to the model's own when it carries one. n.subjects is deliberately smaller than ctGenerate's: this returns nsamples datasets rather than one.

Value

List containing Y, an array of nsamples by data rows by manifest variables, and llrow, an array of nsamples by data rows log likelihoods. With parsonly=TRUE, the prepared model carrying the prior draws under $stanfit$rawposterior and $stanfit$transformedpars.

Examples

#generate and plot samples from prior predictive
priorpred <- ctGenerateFromPriors(cts = ctstantestfit,cores=2,nsamples = 50)
# backend = 'auto' generates with julia where it is available. Julia keeps
# running between calls, holding what it has compiled; this ends it.
if (isTRUE(ctJuliaStatus()$available)) ctJuliaKill("session")

Check which parameters a dataset can inform, before fitting

Description

Reports the directions of the parameter space this data carries no information about, and the parameters involved in them, without fitting the model.

Usage

ctIdentify(
  datalong,
  model,
  inits = NULL,
  nstart = 3L,
  spread = 0.5,
  priors = FALSE,
  cores = getOption("mc.cores", 2L),
  verbose = 0L,
  rtol = 1e-13,
  seed = 1L,
  ...
)

Arguments

datalong

Long format data, as for ctFit.

model

Model from ctModel.

inits

Optional evaluation point. Supplying one uses that single point and disables the comparison across points described in Details.

nstart

Number of points to evaluate. The first is the origin of the unconstrained space; the rest are dispersed draws around it.

spread

Standard deviation of the additional evaluation points on the raw scale. Large enough that they are genuinely different points, which ctFit's own rnorm(npar, 0, 0.01) initialisation is not.

priors

Whether to include the prior. FALSE by default, because the question is what the data can inform: a prior informs every direction, so a model unidentified by its data looks fine with priors on and the check stops being able to say anything.

cores

As for ctFit, and also the subject-chunk ceiling for every engine call this function makes, restored on exit.

verbose

As for ctFit.

rtol

A direction counts as uninformed when the information along it falls below this fraction of the best-informed direction's. The default separates zero from merely small: on the scaled information a structurally unidentified direction sits at machine precision (measured: 7e-17), while a weakly identified one on a comparable model sat at 3e-12. Raise it to catch weak directions too, at the cost of reporting some a fit would have settled.

seed

Seed for the dispersed evaluation points, so that two identical calls give identical output. The session's random stream is restored afterwards, so this neither depends on it nor disturbs it. NULL draws from the session stream instead, which makes the result depend on where that had got to.

...

Further arguments to ctFit that shape the model, such as intoverpop or poprank, with ctFit's defaults.

Details

This catches structural non-identification – parameters that only appear together, a latent process nothing measures, a scale that is never fixed – which is a property of the model and holds wherever it is evaluated. It is not a verdict on the model: a direction that is flat at one point may be perfectly informed at the estimate, which is why several points are used and only parameters lying in the flat subspace at every one of them are reported as uninformed.

The model is prepared by ctFit, with any of its arguments given in ..., so what is checked is the model a fit with the same arguments would estimate.

Not every flat direction means the same thing. A random effect on a variance cell (DIFFUSION, MANIFESTVAR) under intoverpop = 'augmented' with a full-rank population covariance (poprank = NA; the route's default rank leaves out the directions its filter cannot see) is partially identified: the population covariances it generates with the other individually varying parameters are determined, while their decomposition into a scale and correlations is a ridge. The printed summary distinguishes the two cases, because the advice differs – fixing a value discards information in the first case and is the fix in the second.

Requires the Julia backend, which is where the per-subject scores come from. The statistic has rank at most the number of subjects, so a model with more free parameters than subjects reports flat directions for that reason alone; the printed summary says when that applies.

How well the data determine each random effect's population standard deviation, and how much of each subject's effect its own data determine – the check ctFit makes at its estimate, stored as fit$identifiability$effects – is taken here only at a supplied inits, such as a fit's estimate$raw. It depends on the population standard deviations, which only a fit estimates: at the default evaluation points it describes the starting values rather than the data, and there it read a well identified random drift as less informed than one the data barely inform.

Value

An object of class ctIdentify with $parameters (names in the flat subspace at every evaluation point), $partial and $structural splitting those into the partially and completely unidentified, $sometimes (in a flat direction beyond that subspace at some points), $rotating (whether the basis of the flat subspace turns between points), $nweak, $condition, $starts holding the per-point detail, and $effects, the random-effect information at a supplied inits (NULL otherwise). Printing it summarises what was found.

See Also

ctFit runs the equivalent check on a finished fit, evaluated at the estimate and stored as fit$identifiability.

Examples

# ctIdentify() always uses the julia backend internally, whatever the
# eventual fit will use, so the example is inert where Julia is absent --
# including on CRAN, whose check machines have none.
if (isTRUE(ctJuliaStatus()$available)) {
  gen <- suppressMessages(ctModel(type = 'ct', n.latent = 1, n.manifest = 1,
    manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1),
    DRIFT = matrix(-0.4), DIFFUSION = matrix(0.6), MANIFESTVAR = matrix(0.3),
    T0VAR = matrix(1), T0MEANS = matrix(0), CINT = matrix(0),
    MANIFESTMEANS = matrix(0), Tpoints = 8))
  datalong <- ctGenerate(gen, n = 40, Tpoints = 8, backend = 'r')

  model <- suppressMessages(ctModel(type = 'ct', n.latent = 1, n.manifest = 1,
    manifestNames = 'Y1', latentNames = 'eta1', LAMBDA = matrix(1),
    T0MEANS = matrix(0), CINT = matrix(0), MANIFESTMEANS = matrix(0)))

  result <- ctIdentify(datalong, model, cores = 1)
  print(result)
  # Julia keeps running between calls, holding what it has compiled;
  # this ends it.
  ctJuliaKill("session")
}

Converts absolute times to intervals for wide format ctsem panel data

Description

Converts absolute times to intervals for wide format ctsem panel data

Usage

ctIntervalise(
  datawide,
  Tpoints,
  n.manifest,
  n.TDpred = 0,
  n.TIpred = 0,
  imputedefs = F,
  manifestNames = "auto",
  TDpredNames = "auto",
  TIpredNames = "auto",
  digits = 5,
  mininterval = 0.001,
  individualRelativeTime = TRUE,
  startoffset = 0
)

Arguments

datawide

Wide format data, containing absolute time measurements, to convert to interval time scale. See ctLongToWide to easily convert long format data.

Tpoints

Maximum number of discrete time points (waves of data, or measurement occasions) for an individual in the input data structure.

n.manifest

number of manifest variables per time point in the data.

n.TDpred

number of time dependent predictors in the data structure.

n.TIpred

number of time independent predictors in the data structure.

imputedefs

if TRUE, impute time intervals based on the measurement occasion (i.e. column) they are in, if FALSE (default), set related observations to NA. FALSE is recommended unless you are certain that the imputed value (mean of the relevant time column) is appropriate. Noise and bias in estimates will result if wrongly set to TRUE.

manifestNames

vector of character strings giving variable names of manifest indicator variables (without _Tx suffix for measurement occasion).

TDpredNames

vector of character strings giving variable names of time dependent predictor variables (without _Tx suffix for measurement occasion).

TIpredNames

vector of character strings giving variable names of time independent predictor variables.

digits

How many digits to round to for interval calculations.

mininterval

set to lower than any possible observed measurement interval, but above 0 - this is used for filling NA values where necessary and has no impact on estimates when set in the correct range. (If all observed intervals are greater than 1, mininterval=1 may be a good choice)

individualRelativeTime

if TRUE (default), the first measurement for each individual is assumed to be taken at time 0, and all other times are adjusted accordingly. If FALSE, new columns for an initial wave are created, consisting only of observations which occurred at the earliest observation time of the entire sample.

startoffset

if 0 (default) uses earliest observation as start time. If greater than 0, all first observations are NA, with distance of startoffset to first recorded observation.

Details

Time column must be numeric! Columns are read in ctsem's wide order: the manifest variables and then the time dependent predictors, each by occasion as ctWideNames lists them, then the absolute times T0, T1, ..., then the time independent predictors, as ctLongToWide returns them. Columns with exactly those names in another order are put in that order by name, with a message; columns named otherwise are read by position. An occasion with no time has its observations set to NA unless imputedefs = TRUE.

Value

The wide format data as a matrix, with the absolute time columns replaced by the intervals preceding each time point, dT1, dT2, ..., and columns named as ctWideNames names them.

Examples

wideexample <- ctLongToWide(datalong = ctstantestdat, id = "id", 
time = "time", manifestNames = c("Y1", "Y2"), 
TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3"))

#Then convert the absolute times to intervals, using the Tpoints reported from the prior step.
wide <- ctIntervalise(datawide = wideexample, Tpoints = 10, n.manifest = 2, 
n.TDpred = 1, n.TIpred = 3, manifestNames = c("Y1", "Y2"), 
TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3") )
 
print(wide)

Evaluate a julia model's likelihood and gradient at given parameters

Description

Evaluates the objective the julia backend optimises – the log likelihood, with the random effects integrated as the model's intoverpop route integrates them – at a vector of raw (unconstrained) parameters, and its gradient. For checking a fit, profiling by hand, or handing ctsem's likelihood to an optimiser or sampler of your own.

Usage

ctJuliaEvaluate(
  object,
  pars = NULL,
  gradient = TRUE,
  contributions = FALSE,
  gradient_method = c("adjoint", "forward"),
  cores = getOption("mc.cores", 2L)
)

Arguments

object

A fit from ctFit(..., backend = 'julia'), or a model prepared without fitting by ctFit(..., backend = 'julia', fit = FALSE) (class ctJuliaModel).

pars

Raw (unconstrained) parameters, in the order of coef(fit); defaults to a fit's estimate, and must be given for a prepared model.

gradient

Also return the gradient with respect to pars.

contributions

Return subject and row contribution details when available.

gradient_method

Either "adjoint" (reverse mode, the default) or "forward" (ForwardDiff). Both compute the same gradient. "adjoint" costs the same regardless of how many free parameters a model has, and is faster than "forward" at every model size measured, by a margin that grows with the parameter count. See optimcontrol$gradient in ctFit.

cores

The most engine threads this evaluation uses, restored afterwards. NA leaves the ceiling already in force, for a caller that has set one around a run of evaluations.

Value

A list containing the log likelihood and, when requested, the gradient and the contributions.

Examples

# Needs a working julia backend, so the example is inert where Julia is
# absent -- including on CRAN, whose check machines have none.
if (isTRUE(ctJuliaStatus()$available)) {
  model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1',
    LAMBDA = matrix(1))
  model$pars$indvarying <- FALSE
  prepared <- ctFit(ctstantestdat, model, backend = 'julia', fit = FALSE)
  start <- ctJuliaEvaluate(prepared, pars = rep(0, 6), cores = 1)
  str(start)
  # Julia keeps running between calls, holding what it has compiled;
  # this ends it.
  ctJuliaKill("session")
}

Set up the Julia backend

Description

Installs everything ctFit(backend='julia') needs, in one call and without restarting R: the JuliaConnectoR bridge package, a Julia installation, and the Julia dependencies of the engine that ships inside ctsem. Any step that is already satisfied is skipped, so on a machine that already has Julia this only prepares the engine.

Usage

ctJuliaInstall(
  threads = NULL,
  version = NULL,
  agree = NULL,
  force = FALSE,
  quiet = FALSE
)

Arguments

threads

Number of Julia threads for the engine's subject loop. Julia fixes this when its process starts, so it applies to the session this call creates; see ctJuliaSetup.

version

Julia version to install. Defaults to the version ctsem's vendored engine was resolved against. The download is checksum-verified only for that default.

agree

TRUE to consent to the installs without being asked, FALSE to refuse them. NULL, the default, asks in an interactive session and refuses otherwise.

force

Reinstall Julia even if one is already available, and restart any running Julia session.

quiet

Suppress download and installation progress output.

Details

Each step that installs something asks first, naming the download and its destination. A non-interactive session has nobody to ask, so nothing is installed there unless agree = TRUE or the environment variable CTSEM_JULIA_AGREE=yes gave permission in advance.

Julia is downloaded only if none can be found. ctsem looks in JULIA_BINDIR, then at a Julia it installed itself, then on the PATH, then among juliaup's installations. A Julia that ctsem installs goes under tools::R_user_dir("ctsem", "data") and later sessions find it automatically – there is no environment variable to set, and no PATH or shell profile is modified. To use a different Julia afterwards, set JULIA_BINDIR.

Value

A Julia-engine status list, invisibly; see ctJuliaStatus.

See Also

ctJuliaSetup, ctJuliaStatus

Examples

## Not run: 
ctJuliaInstall()            # ask, then install whatever is missing
ctJuliaInstall(threads = 4) # and start the Julia session with four threads

## End(Not run)

Stop Julia processes started for ctsem

Description

Stops the Julia processes ctJuliaProcesses lists. By default only those left by R sessions that have ended, which nothing can be using.

Usage

ctJuliaKill(
  which = c("orphaned", "session", "workers", "mine", "all"),
  pid = NULL,
  ask = interactive()
)

Arguments

which

Which to stop: "orphaned" (the default) for processes whose R session has ended; "session" for this R session's engine, which the next julia call replaces with a new one (losing what it had compiled); "workers" for this session's sampling workers – the pool of background R sessions that sampling with processes = TRUE warms and keeps for the next call, released here with their Julia processes; "mine" for all three; "all" for every process listed, including the engines of other running R sessions, which then fail.

pid

Process ids to stop instead of choosing by which. Only processes ctJuliaProcesses() lists can be stopped this way.

ask

Whether to ask before stopping another R session's engine, as which = "all" or pid can.

Value

The rows of ctJuliaProcesses() that were stopped, invisibly.

See Also

ctJuliaProcesses

Examples

ctJuliaKill()             # engines left by R sessions that have ended
ctJuliaKill("session")    # this session's engine, e.g. stuck compiling

Julia processes started for ctsem

Description

Lists the Julia processes on this machine that ctsem started or that belong to them: this R session's engine, the engines of its sampling workers, those of other R sessions, and any left running by an R session that has ended. A Julia process can outlive what it was started for – Escape leaves one finishing an uninterruptible step in the background, and a crashed R session leaves its engine running until the engine notices – so this is where to look when a machine is busier than it should be. ctJuliaKill stops them.

Usage

ctJuliaProcesses(interval = 0.5)

Arguments

interval

Seconds over which to measure CPU use, which is what the busy column reports; 0 skips the measurement.

Details

ctsem records each Julia session it starts, which is how its processes are told from any other Julia. A JuliaConnectoR session it has no record of – one started by other code, or by a ctsem too old to record it – is listed too, as is a Julia process started by one of these (precompiling the engine, typically). CPU, memory, thread and parent details need the ps package; without it, only which processes exist and whose they are.

Value

A data frame, one row per process, with columns pid; role ("this session", "worker of this session", "another R session", "orphaned" for one whose R session has ended, "JuliaConnectoR, unrecorded", or "child of <pid>"); busy and cpu_percent over interval; cpu_seconds; memory_mb; julia_threads, the Julia thread count ctsem started it with; os_threads; r_pid, the R process it serves; started; and note, which says when this session's engine is still finishing an interrupted call.

See Also

ctJuliaKill, ctJuliaStatus

Examples

ctJuliaProcesses()

Configure the Julia engine used by ctsem

Description

Prepares the copy of ContinuousTimeSEM.jl that ships inside this ctsem installation. The engine source itself is vendored in inst/julia/ – no repository credentials and no network access are involved in getting that – but its Julia package dependencies still have to be instantiated, which reaches the network on a machine where they are not already cached. That step is consent-gated exactly as in ctJuliaInstall: asked for interactively, skipped with a message when the environment is already instantiated, and declined by default in a non-interactive session unless agree = TRUE or CTSEM_JULIA_AGREE=yes said otherwise in advance.

Usage

ctJuliaSetup(
  project = NULL,
  julia_bin = NULL,
  threads = NULL,
  force = FALSE,
  agree = NULL
)

Arguments

project

Optional local ContinuousTimeSEM.jl checkout to use instead of the copy that ships with ctsem. Rarely needed: the engine is part of ctsem and is edited in place under inst/julia/.

julia_bin

Optional Julia binary directory.

threads

Number of Julia threads, fixed for the life of the session. The engine splits its subject loop across them, and ctFit's cores cannot exceed this count: Julia fixes it at process start, so setting it here takes effect only if no Julia session is running yet – pass force = TRUE to restart one. NULL, the default, starts a session at getOption("mc.cores", 2) threads unless JULIA_NUM_THREADS is set; a session started by a call with a cores argument – a fit, ctFitUncertainty, ctLOO and the rest – is instead as wide as that call's cores, if more. A number given here is also the ceiling for any call that sets none. A fit that asks for more cores than the session has threads says so; see ctFit.

force

Reconfigure an existing Julia session.

agree

TRUE to consent to instantiating the engine's Julia package dependencies without being asked, FALSE to refuse. NULL, the default, asks in an interactive session and refuses otherwise; see ctJuliaInstall.

Details

The first call downloads and precompiles those dependencies (roughly 120 MB) and compiles the engine, which takes several minutes; later calls in new sessions reuse both.

The engine is compiled in advance, by fitting a small model on each route, and every model of a route runs that same compiled code: a model's sizes, parameters and state-dependent cells are data to the engine rather than part of its types. So the first fit of a new model in a session compiles little, and takes seconds more than later fits rather than minutes. A model with a categorical indicator, or a Laplace unit with more than four random effects, compiles a little more the first time. ctFit(forcerecompile = TRUE) compiles a heavily state-dependent model's expressions for faster gradients, at a minute or more of compile.

Pressing Escape (or Ctrl-C) during a julia fit returns to the R prompt at once, and keeps the Julia session and everything it has compiled. Julia stops the interrupted work at its next iteration; work that has no iterations, such as compiling for a new model shape, finishes in the background. If it is still running when the next julia call starts, that call says so and waits, and Escape during that wait stops Julia outright. ctJuliaProcesses lists the Julia processes ctsem has running, including any left by R sessions that have ended, and ctJuliaKill stops them.

If JuliaConnectoR or Julia itself is missing, this offers to install it rather than failing – the same thing ctJuliaInstall does, which is the function to reach for when setting the backend up deliberately, or from a script.

Value

A Julia-engine status list, invisibly.

See Also

ctJuliaInstall, ctJuliaStatus

Examples

## Not run: 
ctJuliaSetup()                         # prepare the engine, asking first
ctJuliaSetup(threads = 8, force = TRUE) # restart the session wider

## End(Not run)

Report Julia backend availability

Description

Reports what backend='julia' would find, and installs nothing: it is safe to call on a machine with no Julia and no JuliaConnectoR, where it says so rather than erroring. ctJuliaInstall supplies whatever it reports as missing.

Usage

ctJuliaStatus(project = NULL, julia_bin = NULL)

Arguments

project

Optional local ContinuousTimeSEM.jl checkout to use instead of the copy that ships with ctsem. Rarely needed: the engine is part of ctsem and is edited in place under inst/julia/.

julia_bin

Optional Julia binary directory.

Value

A list describing the selected Julia engine: whether it is available – a julia fit could run now without asking anything, which needs a Julia that starts and the engine's Julia packages installed, or consent to install them given in advance with CTSEM_JULIA_AGREE=yes (checked by a short julia process, once per session, never a session) – whether the connectoR bridge package is installed at a version the backend runs on, the julia_bin directory in use and the julia version there, the engine version (a hash of the engine source shipped with this ctsem, which is also what its cached project directory is keyed on), and the number of threads in a running session (NA when none is running: this never starts one, so it leaves the session's width to the first call that needs it).

Examples

status <- ctJuliaStatus()
status$available

Get Kalman filter estimates from a ctStanFit object

Description

Get Kalman filter estimates from a ctStanFit object. ctStanKalman is maintained as a backward-compatible alias for this function. Its pre-3.11 partner ctKalman is an alias for ctPredict, not for this function, and returns a long data frame rather than a list of arrays.

Usage

ctKalmanArray(
  fit,
  nsamples = NA,
  pointest = TRUE,
  collapsefunc = NA,
  cores = getOption("mc.cores", 2L),
  subjects = "all",
  timestep = "asdata",
  maxtime = "asdata",
  standardisederrors = FALSE,
  subjectpars = TRUE,
  indvarstates = FALSE,
  removeObs = F,
  realid = TRUE,
  newdata = NULL,
  ...
)

Arguments

fit

fit object from ctFit, from any backend.

nsamples

either NA (to extract all) or a positive integer from 1 to maximum samples in the fit.

pointest

If TRUE, uses the posterior mode as the single sample.

collapsefunc

function to apply over samples, such as mean

cores

Integer number of cpu cores to use. Only needed if savescores was set to FALSE when fitting. backend='stan' only – the julia path runs this loop serially in the engine and refuses cores by name rather than accepting it and ignoring it.

subjects

'all' (the default), or the subjects to compute for, as ids in the data (see realid). The output numbers them in the order given, sorted.

timestep

Either a positive numeric value, 'asdata' to use the times in the dataset, or 'auto' to select a timestep automatically (resulting in some interpolation but not excessive computation).

maxtime

only relevant if timestep is not 'asdata'. Positive numeric denoting max time for computations.

standardisederrors

If TRUE, computes standardised errors for prior, upd, smooth conditions.

subjectpars

if TRUE, state estimates are not returned, instead, predictions of each subjects parameters are returned, for parameters that had random effects specified.

indvarstates

if TRUE, do not remove indvarying states from output

removeObs

Logical or integer. If TRUE, observations (but not covariates) are set to NA, so only expectations based on parameters and covariates are returned. If a positive integer N, every N observations are retained while others are set NA for computing model expectations – useful for observing prediction performance forward further in time than one observation.

realid

If TRUE (the default), subjects are read as the ids in the data, and integers that are not ids there are read as ctsem's numbering of the subjects, 1 to N, with a message. If FALSE, they are read as that numbering.

newdata

Optional long format data, laid out as the data the model was fitted to, to compute for instead of the fit's own: new subjects, the fit's subjects with other covariate values, or a record cut short. The fit's estimates (or, with pointest=FALSE, its draws) are used as they stand, under the model the fit ran; nothing is refitted. See ctPredict.

...

additional arguments to collpsefunc.

Value

list containing Kalman filter elements, each element in array of iterations, data row, variables. llrow is the log likelihood for each row of data.

For a binary or ordinal manifest variable the returned manifest quantities are the *linear predictor* ν=Λη+τ\nu = \Lambda \eta + \tau, on the logit scale, not a probability and not a category. Turn one into a probability with plogis(nu) for binary, or into cumulative category probabilities with plogis(tau_k - nu) for ordinal, taking the thresholds from summary(fit)$popmeans. The manifest covariance for such a variable is likewise the covariance of the linear predictor.

Examples

k=ctKalmanArray(ctstantestfit,subjectpars=TRUE,collapsefunc=mean)

Measure, and optionally correct, the Laplace approximation error in a fit

Description

Recomputes a backend='julia', intoverpop='laplace' fit's log marginal likelihood by adaptive Gauss-Hermite quadrature over the same random-effect integral the Laplace approximation approximates, and reports how far the estimate would move if that integral were taken exactly.

Usage

ctLaplaceCheck(
  fit,
  nodes = 5L,
  correction = TRUE,
  step = 0.001,
  refine = FALSE,
  maxiter = 50L,
  cores = getOption("mc.cores", 2L),
  verbose = 0L
)

Arguments

fit

A ctJuliaFit fitted with intoverpop='laplace'.

nodes

Quadrature nodes per random effect. Cost is nodes^k process log likelihoods per block, where k is that level's number of random effects; 5 is enough to locate the maximum in the cases tested, 9 to settle the value.

correction

Compute the first-order correction to the estimate. Costs 2 * npar quadrature evaluations, and needs the fit's Hessian, so it is skipped when uncertainty was not computed.

step

Finite-difference step for the gap gradient, on the raw scale.

refine

Continue the fit on the quadrature objective rather than correcting linearly: the refined fit intoverpop = 'quadrature' runs at fit time (see ctFit), from this fit's estimate, with the quadrature rules placed at every point. Its gradient is exact, so it costs a quadrature gradient per step rather than the 2 * npar quadrature evaluations per gradient the finite-difference refinement this replaced cost, and it reports the point it reaches in refined.

maxiter

Iteration cap for the climb of refine.

cores

Engine threads for the quadrature.

verbose

Integer; 1 or more prints progress.

Details

The quadrature is adaptive in the usual sense: it is centred at each subject's inner mode and scaled by the inverse of the inner curvature there, both of which the fit has already computed. The curvature's eigenvalues are clipped from below at one, the prior's own curvature, before scaling: where the likelihood has gone convex in the random effects an unclipped rule puts its nodes far outside the prior and misses the mass it is meant to measure (1.8 nats too high on average on near-singular units in simulation, 0.06 clipped). Where every eigenvalue is at least one nothing changes.

The gap is quadrature minus the fit's own Laplace term, the quantity the fit maximised. With nodes = 1 that gap is zero wherever the curvature is at least the prior's, which is what makes the comparison meaningful – the two are integrating the same thing by different rules. On a unit with an eigenvalue below one it is not zero even at one node: the one-point clipped rule is the eigenwise-floored term, below the fit's.

The correction is a single Newton step, delta = solve(-hessian, gradient of (quadrature - laplace)), evaluated at the estimate. Read it against the standard errors: delta / se well below one says the approximation is not what limits the answer, and near or above one says the point estimate is approximation-limited and the interval will not cover whatever width it has.

A fit can be refined, with its units' random effects integrated by quadrature, when it is fitted (ctFit(intoverpop = 'quadrature')) or after (ctFitUncertainty(fit, intoverpop = 'quadrature')); by default it only runs the random-effects check, which warns where integrating would move an estimate. On a corrected fit this check is made at the corrected estimate, which is not a Laplace optimum: the correction reported is then the further Newton step on the quadrature objective from there, with the Laplace gradient included, and at on the result is 'corrected'. A small further step says the correction got there; a large one says it did not. On any other fit it is the first step.

Nested groupings are handled. A group's integral does not factor over its members, but it does factor conditionally – given the group effect the members are independent – so the rule recurses over the same block tree the fit already builds, at a cost linear in the number of groups at each level rather than exponential in the number of members.

Value

A list with gap (quadrature minus Laplace log marginal at the estimate), gap_per_subject, nodes, and, when correction is TRUE, a parameters data frame with one row per raw parameter giving estimate, delta, corrected, se and delta_se. With refine it also carries refined. at says whether the estimate checked was the Laplace optimum ('laplace') or already quadrature-corrected ('corrected').

dropped_directions counts the directions of the information matrix the correction does not step along, whose correction is reported as zero rather than as the arbitrarily large step an unguarded solve would produce: exactly the directions fit$identifiability names, those whose curvature is negligible or not positive and those the likelihood was measured flat along. On a model where the Laplace approximation is already exact the gap gradient is floating-point noise, and dividing that by a near-singular curvature is how a meaningless correction gets a plausible-looking number.

See Also

ctFitUncertainty with method = 'sample' and intoverpop = 'none' removes the approximation instead of measuring it, by sampling the joint posterior; much slower, and the right answer when delta_se says a linear correction is not credible.

Examples

## Not run: 
data <- ctstantestdat
model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1',
  LAMBDA = matrix(1))
model$pars$indvarying <- model$pars$matrix %in% 'MANIFESTMEANS'
fit <- ctFit(data, model, backend = 'julia', intoverpop = 'laplace')

check <- ctLaplaceCheck(fit)
check                     # gap, and the largest corrections in standard errors
check$parameters          # the full per-parameter table

## End(Not run)

ctLongToWide Restructures time series / panel data from long format to wide format for ctsem analysis

Description

ctLongToWide Restructures time series / panel data from long format to wide format for ctsem analysis

Usage

ctLongToWide(
  datalong,
  id,
  time,
  manifestNames,
  TDpredNames = NULL,
  TIpredNames = NULL
)

Arguments

datalong

dataset in long format, including subject/id column, observation time (or change in observation time, with 0 for first observation) column, indicator (manifest / observed) variables, any time dependent predictors, and any time independent predictors.

id

character string giving column name of the subject/id column

time

character string giving column name of the time columnn

manifestNames

vector of character strings giving column names of manifest indicator variables

TDpredNames

vector of character strings giving column names of time dependent predictor variables

TIpredNames

vector of character strings giving column names of time independent predictor variables

Details

Time column must be numeric

Value

A wide format matrix, one row per subject: each manifest variable and time dependent predictor at each time point (Y1_T0, Y1_T1, ...), the observation times T0, T1, ..., then the time independent predictors.

See Also

ctIntervalise

Examples

wideexample <- ctLongToWide(datalong = ctstantestdat, id = "id", 
time = "time", manifestNames = c("Y1", "Y2"), 
TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3"))

#Then convert the absolute times to intervals, using the Tpoints reported from the prior step.
wide <- ctIntervalise(datawide = wideexample, Tpoints = 10, n.manifest = 2, 
n.TDpred = 1, n.TIpred = 3, manifestNames = c("Y1", "Y2"), 
TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3") )
 
print(wide)

Cross validation for ctsem fits: K fold by refitting, or PSIS for Laplace fits

Description

Estimates out-of-sample predictive performance, by refitting the model with each fold of the data held out in turn or, for a julia Laplace fit, by Pareto-smoothed importance sampling without refitting.

Usage

ctLOO(
  fit,
  folds = 10,
  cores = getOption("mc.cores", 2L),
  parallelFolds = FALSE,
  tol = NULL,
  subjectwise = ifelse(length(unique(.ctFitRowSubject(fit))) >= folds, TRUE, FALSE),
  keepfirstobs = FALSE,
  leaveOutN = NA,
  refit = TRUE,
  casewiseApproximation = FALSE,
  method = c("kfold", "psis"),
  nsamples = 1000
)

Arguments

fit

A fit from ctFit, on either backend.

folds

Number of cross validation splits to use – 10 folds implies that the model is re-fit 10 times, each time to a data set with 1/10 of the observations randomly removed.

cores

Number of processor cores to use.

parallelFolds

compute folds in parallel or use cores to finish single folds faster. parallelFolds will use folds times as much memory.

tol

objective tolerance for each fold's refit. On a stan fit it defaults to 1e-5. On a julia fit a fold by default stops once a step is predicted to gain less than 1e-4 in log likelihood, which keeps each held-out subject's error to about 0.014 times the square root of its influence (see casewiseApproximation); an objective tolerance stops it while the gradient can still be large, and at 1e-5 moved one 40-subject model's out-of-sample log likelihood by 1 in total.

subjectwise

drop random subjects instead of data rows? With method = 'psis' this chooses the level instead – leave one subject (unit) out when TRUE, the default there, and leave one row out when FALSE.

keepfirstobs

do not drop first observation (more stable estimates)

leaveOutN

if a positive integer is given, the folds argument is ignored and instead the folds are calculated by leaving out every Nth row from the data when fitting. Leaving 2 out would result in 3 folds (starting at rows 1,2,3), each containing one third of the data.

refit

if FALSE, do not optimise parameters for the new data set, just compute the likelihoods etc from the original parameters

casewiseApproximation

TRUE replaces each fold's refit by one Newton step from the full estimate: the held-out data's score times the fit's covariance. Much faster than refitting, and close to it when no fold moves the estimate far. On a julia fit each fold's influence, the step's squared length in the covariance's metric, is returned in foldinfluence, and a number instead of TRUE steps only the folds whose influence is below it and refits the rest (see Details). Needs an optimized fit with a covariance; on a stan fit, subject folds and TRUE or FALSE only.

method

'kfold' (the default) refits the model once per fold. 'psis' is refit-free Pareto-smoothed importance sampling leave-one-out for backend = 'julia' fits (see Details); it needs the loo package. The K-fold arguments folds, parallelFolds, tol, keepfirstobs, leaveOutN, refit and casewiseApproximation mean nothing to it and are refused if given.

nsamples

Number of importance sampling draws for method = 'psis'. A sampled fit's posterior draws are used instead: all of them, or thinned to nsamples when it is given. Fewer draws estimate the Pareto k of a heavy-tailed correction too low: on one fit with an outlying subject the proposal's k read 0.54 at 1000 draws and 0.83 at 4000.

Details

Works for backend='julia' fits as well as stan ones. There, a held-out row is withheld by setting its manifest observations to missing and re-preparing, which is a row the filter propagates through without an update and without a likelihood contribution – the same thing standata$dokalmanrows does for the stan backend. For julia fits parallelFolds = TRUE runs up to cores folds at once, each in a background R session with a one-thread Julia, started and compiled for the model before the first fold (it needs the future package; the sessions persist for the next call, and ctJuliaKill('workers') releases them). Without it the folds run one at a time on cores threads. Folds are separate fits, so they gain more from processes than from threads, except where each fold takes only seconds and starting the sessions costs more than it saves. $processes on the result says which ran.

Be aware that each fold is an independent re-optimisation, and neither backend's optimizer restarts from a flat direction. On a weakly identified model, or one where withholding a fold leaves a parameter poorly determined, two folds – or two backends – can converge to noticeably different raw parameters for a similar likelihood.

K fold on an intoverpop = 'laplace' fit. The random effects are integrated out, so what is scored depends on the fold type. With row folds, a held-out row is scored with each subject's random effects at their mode given the fold's training rows, at the fold's parameters – the held-out row never informs the mode it is scored at. With subject folds a held-out subject's rows are not scored at all: the out-of-sample quantity is the subject's Laplace marginal likelihood at the fold's parameters, returned as outsampleLogLikSubject, and outsampleLogLikRow is NA because the marginal does not decompose into rows. In a multilevel model a subject's marginal is conditional on the rest of its outermost group, as the difference of that group's marginal with and without the held-out subjects; several subjects of one group held out together share that joint value evenly. insampleLogLik is correspondingly the Laplace marginal at the estimate for subject folds, and the sum of the rows at the full-data modes for row folds.

casewiseApproximation. Each fold's refit is replaced by one Newton step from the full estimate, est - cov %*% s, where s is the score of the held-out data at the estimate: each unit the fold empties (a subject, or under the Laplace layer an outermost group) contributes its likelihood score, and a unit it leaves partly observed – a row fold, or a subject inside a group – the difference between its score on the full and on the training data, at one gradient of the training objective per fold. The prior stays, as it does in a refit. This is the one-step (infinitesimal jackknife) approximation to the refit; its error grows faster than the step. A fold's influence t(s) %*% cov %*% s measures the step in posterior standard deviations squared, and a number given as casewiseApproximation refits exactly every fold whose influence is at or above it, so the approximation is used only where the step is short. foldstepped says which folds were stepped. On a julia fit a stepped fold scores its held-out rows only, together with every other stepped fold in one pass over the data, so its column of LogLikRowFolds is NA elsewhere: the training rows at each step would cost a pass a fold, which for leave-one-subject-out is the whole cost.

method = 'psis'. Pareto-smoothed importance sampling (Vehtari, Gelman and Gabry, 2017), without refitting. It estimates the Bayesian leave-one-out predictive density – a unit's likelihood averaged over the posterior without it – where K fold scores each fold at its refit estimate, so the two differ by more than Monte Carlo error, most for a unit that moves the estimate. Two levels:

  • subjectwise = TRUE: leave one unit out, where a unit is the outermost level of random effect – a subject in a one-level model, a group in a multilevel one, since subjects sharing a group effect are not independent given the population parameters. A unit's term is its likelihood with its random effects integrated out: by the filter under intoverpop = 'augmented', by the Laplace approximation otherwise. On a sampled fit (optimize = FALSE, or ctFitUncertainty(fit, 'sample')) the population parameters are its posterior draws. On an optimized fit they are drawn from its normal approximation (fit$estimate$cov, so the fit must carry one), widened by a factor of 1.5 and importance corrected to the posterior. A unit whose Pareto k is above 0.7 is estimated again with nsamples more draws, centred where leaving it out moves the estimate – the step casewiseApproximation takes – pooled with draws around the estimate, and the estimate with the lower k is kept (pointwise$refined). That needs the fit's objective to be the posterior it describes, so it is not done for a fit sampled jointly over parameters and random effects (intoverpop = 'none'), whose unit terms are Laplace approximations to the sampled marginal. When the normal approximation itself misses the posterior (proposal_k, printed, above 0.7) every unit's estimate is unreliable, and a warning says to sample the fit and use its draws instead. No warning is not proof the approximation is good: k is estimated too low from few draws, so a value approaching 0.7, or a posterior you expect to be skewed (an outlying subject, a variance near zero, binary indicators), is worth rerunning with more nsamples or on a sampled fit.

  • subjectwise = FALSE, on an optimized intoverpop = 'laplace' fit only: leave one row out, conditional on the estimated population parameters and integrating over each unit's random effects, drawn from the Gaussian the Laplace approximation fits at their mode, widened by a factor of 1.5. This is pointwise leave-one-out of the filter's one-step-ahead terms – each row's density given the rows before it – not exact time series leave-one-out, and it does not propagate uncertainty in the population parameters.

Draws whose inner mode solve fails are dropped and counted. A Pareto k above 0.7 flags a unit or row whose estimate is unreliable; K fold is then the better tool for it.

Both methods cross-validate the Laplace objective. A Laplace fit's estimate can be refitted with its units integrated (intoverpop = 'quadrature', in ctFit or ctFitUncertainty). The folds are refits of the Laplace objective, scored by it, and are not corrected – each would need its own Hessian and quadrature gradient – so the in-sample side, the folds' starting point and the PSIS proposal all use the Laplace optimum, fit$laplace$correction$laplace_estimate, rather than the corrected estimate. scoring says so when the fit was corrected.

A fold's estimate is never certified. Whatever the route, a refit fold is optimised with optimcontrol$certify = FALSE: certifying and correcting a fold the way the full fit is would cost each fold its own Hessian, which is the expense K-fold exists to avoid. scoring says so for any refit (refit = TRUE, the default, and not casewiseApproximation), on both routes.

Value

For method = 'kfold', a list with foldrows, foldpars, insampleLogLikRow, LogLikRowFolds, outsampleLogLikRow, insampleLogLik, outsampleLogLik and entropy and standard deviation summaries, and processes, whether the folds ran in worker processes. When a julia fit's folds are refit, foldoptim has a row per fold: method ('refit' or 'step'), and for a refit the optimiser's iterations, gradients, converged, gain (what it was predicted still to gain, in log likelihood, when it stopped) and seconds; a warning counts the refits that stopped without converging. With casewiseApproximation, foldinfluence and foldstepped give each fold's influence and whether it was stepped rather than refit. scoring is added whenever the folds were refit, naming what was scored and that they are uncertified (see Details); an intoverpop = 'laplace' fit also adds insampleLogLikSubject and outsampleLogLikSubject. For method = 'psis', an object of class ctLOOpsis: elpd_loo, se_elpd_loo, p_loo, looic, pointwise (a data frame of elpd_loo, lpd, p_loo and pareto_k per unit or row, and at the unit level refined), n_high_k, ndraws, ndropped, proposal_k (the Pareto k of the correction from proposal to posterior: one value at the unit level, NA for a sampled fit, one per unit at the row level), level and note.

References

Vehtari, A., Gelman, A. and Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27, 1413–1432.

Examples

ctLOO(ctstantestfit)

Define a ctsem model

Description

This function is used to specify a continuous time structural equation model, which can then be fit to data with function ctFit.

Usage

ctModel(
  LAMBDA,
  type = "ct",
  n.manifest = "auto",
  n.latent = "auto",
  Tpoints = NULL,
  manifestNames = "auto",
  manifesttype = rep(0, nrow(LAMBDA)),
  ncategories = NULL,
  censormin = NULL,
  censormax = NULL,
  latentNames = "auto",
  id = "id",
  time = "time",
  silent = FALSE,
  T0VAR = "auto",
  T0MEANS = "auto",
  MANIFESTMEANS = "auto",
  MANIFESTVAR = "diag",
  ASYMPTOTES = NA,
  RESPONSE = NA,
  DRIFT = "auto",
  CINT = 0,
  DIFFUSION = "auto",
  n.TDpred = "auto",
  TDpredNames = "auto",
  TDPREDEFFECT = "auto",
  TDPREDMEANS = "auto",
  TDPREDVAR = "auto",
  n.TIpred = "auto",
  TIpredNames = "auto",
  tipredDefault = TRUE,
  PARS = NULL,
  driftScaled = character()
)

Arguments

LAMBDA

n.manifest*n.latent loading matrix relating latent to manifest variables, with latent processes 1:n.latent along the columns, and manifest variables 1:n.manifest in the rows.

type

character string. Use 'ct' (continuous time) or 'dt' (discrete time) for Stan-based fitting with ctFit. These return a modern ctsem model object whose pars data frame can also be edited through the pars-backed matrices view. For legacy workflows, 'omx' returns a matrix-list model object that can be edited directly and then converted via ctModelConvertOMX.

n.manifest

Number of manifest indicators per individual at each measurement occasion / time point. Manifest variables are included as the first element of the wide data matrix, with all the 1:n.manifest manifest variables at time 1 followed by those of time 2, and so on.

n.latent

Number of latent processes.

Tpoints

Number of time points, or measurement occasions, in the data. This will generally be the maximum number of time points for a single individual, but may be one extra if sample relative time intervals are used, see ctIntervalise.

manifestNames

n.manifest length vector of manifest variable names as they appear in the data structure, without any _Tx time point suffix that may be present in wide data. Defaults to Y1, Y2, etc.

manifesttype

n.manifest length vector of manifest variable types. 0 (the default) is a continuous, Gaussian variable; 1 is binary, coded 0/1; 2 is ordinal, coded as consecutive integers from 1; 3 is a count, coded as non-negative integers and modelled as Poisson with a log link, so the latent process gives the log rate. A count keeps a free MANIFESTVAR diagonal, unlike a binary or ordinal variable: for a count that entry is a log-scale dispersion, making the observation Poisson-lognormal with variance μ+μ2(eσ2−1)\mu + \mu^2(e^{\sigma^2}-1) against the Poisson's μ\mu. Fix it to zero for the equidispersed Poisson. 4 is censored, a Gaussian variable observed only within censormin and censormax, with values at a limit contributing the probability of being there rather than a density. Non-Gaussian variables are supported by the julia backend only, and binary additionally by stan.

ncategories

n.manifest length integer vector, giving the number of categories of each ordinal manifest variable and ignored for the others. Required when any manifesttype is 2, because the model has to know how many thresholds to estimate before it sees any data – ctGenerate needs it too.

censormin, censormax

n.manifest length numeric vectors giving the lower and upper censoring limits of each manifesttype = 4 variable, and ignored for the others. Defaults are -Inf and Inf; a variable censored on one side only leaves the other infinite. Unlike thresholds these are known constants rather than parameters – a scale's floor and ceiling are properties of the instrument, and the data cannot inform them.

latentNames

n.latent length vector of latent variable names (used for naming parameters, defaults to eta1, eta2, etc).

id

character string denoting column name containing subject identification variables. id data may be of any form, though will be coerced internally to an integer sequence rising from 1. May instead be a character vector naming a nested hierarchy, innermost first: id = c('subject','study') treats each subject as belonging to one study. A cell says where its parameter varies with the indvarying field (see ctPar): TRUE at every level, FALSE at none, or the names of the levels, so 'cint||study' gives an intercept that varies between studies with subjects exchangeable within a study, and 'cint||subject,study' (or TRUE) one that varies at both. model$pars$indvarying holds the same values, one per cell, and can be edited directly, with TRUE again meaning every level; a cell's sdscale applies at every level it varies at. Intercepts (T0MEANS, MANIFESTMEANS, CINT) left unspecified vary at every level. Levels above the subject require backend='julia' with intoverpop='laplace', and the hierarchy must be strictly nested.

time

character string denoting column name containing timing data. Timing data must be numeric.

silent

Suppress all output to console.

T0VAR

lower triangular n.latent*n.latent cholesky matrix of latent process initial variance / covariance. "auto" freely estimates all parameters.

T0MEANS

n.latent*1 matrix of latent process means at first time point, T0. "auto" freely estimates all parameters.

MANIFESTMEANS

n.manifest*1 matrix of manifest intercept parameters. "auto" frees all parameters.

MANIFESTVAR

lower triangular n.manifest*n.manifest cholesky matrix of variance / covariance between manifests at each measurement occasion (i.e. measurement error / residual). "auto" freely estimates variance parameters, and fixes covariances between manifests to 0. "free" frees all values, including covariances.

ASYMPTOTES

Matrix with one row per binary indicator, in manifest order, and two columns: the lower and upper asymptotes of its response probability, which is then lower + (upper - lower) times the logistic of the linear predictor. NA means no asymptote on that side, so the default – all NA – is the two parameter logistic, and the model has no ASYMPTOTES matrix at all. Name a lower cell to estimate a guessing probability (the three parameter model), an upper cell for a ceiling below one, or both (four parameter); a number between 0 and 1 fixes it, and one name shared across rows estimates one asymptote for those items. A length 2 vector applies to every binary indicator, so c('guess', NA) is one guessing parameter for the whole test. Julia backend only.

RESPONSE

Matrix with one row per manifest variable and two columns, the response scale of a censored indicator (manifesttype 4); rows for other indicators must stay at their defaults. Column 1, halfwidth: the half-width of the interval each recorded value stands for – a value y is the event that the latent response lies between the transformed y - halfwidth and y + halfwidth, and a value at censormin or censormax that it lies below or above half a step in from it. 0, the default, is a value recorded exactly. For a rating scale it is half the step between categories, so the bottom category of a 7-point item and a slider's 0 are each what they are; a step that differs by study is a time independent predictor in the cell. Column 2, power: the transform from the recorded scale to the latent response, G(r)=L+((1+r−L)p−1)/pG(r) = L + ((1 + r - L)^p - 1)/p with LL the censormin (0 where there is none). 1, the default, is linear; below 1 the latent response moves less per recorded unit high on the scale, so the recorded value moves faster there; 0 is the log, whose inverse is the exponential link. One transform serves every form an item is asked in, so a name shared across items or studies estimates one curvature from all of them. NA keeps a default, and a length 2 vector fills every censored row, so c(0.5, 'power') is integer data with one estimated transform. Julia backend only.

DRIFT

n.latent*n.latent DRIFT matrix of continuous auto and cross effects, relating the processes over time. "auto" freely estimates all parameters.

CINT

n.latent * 1 matrix of latent process intercepts, allowing for non 0 asymptotic levels of the latent processes. Generally only necessary for additional trends and more complex dynamics.

DIFFUSION

lower triangular n.latent*n.latent cholesky matrix of diffusion process variance and covariance (latent error / dynamic innovation). "auto" freely estimates all parameters.

n.TDpred

Number of time dependent predictor variables in the dataset.

TDpredNames

n.TDpred length vector of time dependent predictor variable names, as they appear in the data structure, without any _Tx time point suffix that may appear in wide data. Default names are TD1, TD2, etc.

TDPREDEFFECT

n.latent*n.TDpred matrix of effects from time dependent predictors to latent processes. Effects from 1:n.TDpred columns TDpredictors go to 1:n.latent rows of latent processes. "auto" freely estimates all parameters.

TDPREDMEANS

Legacy argument for time-dependent predictor means, mainly to support data generation workflows. Expected dimensions are (n.TDpred * Tpoints) rows and 1 column. "auto" creates a free-label matrix.

TDPREDVAR

Legacy argument for time-dependent predictor covariance structure, mainly to support data generation workflows. Expected dimensions are (n.TDpred * Tpoints) square. "auto" creates a free-label matrix.

n.TIpred

Number of time independent predictors. Each TIpredictor is inserted at the right of the data matrix, after the time intervals.

TIpredNames

n.TIpred length vector of time independent predictor variable names, as they appear in the data structure. Default names are TI1, TI2, etc. Or a formula whose terms are such names and smooth terms, ~ s(age, k = 8) + female: a smooth is a penalised spline of a person-level covariate, with its smoothness estimated from the data (backend = 'julia'). It enters the model as k - 1 predictor columns, age_lin (the linear trend) and age_s1, ..., which are built from the data when the model is fitted. A cell's tipreds field names the covariate ('cint||||age') to give it the smooth; without one, under tipredDefault = TRUE, a smooth reaches every free parameter that varies between persons.

tipredDefault

Logical. TRUE sets any parameters with unspecified time independent predictor effects to have effects estimated, FALSE fixes the effect to zero unless individually specified. A cell's effects count as unspecified only when its tipreds field is left out: an empty one ('mypar||||') states that the parameter has no predictor effects, which is the way to exclude a parameter under tipredDefault = TRUE. The setting is kept on the returned model as $tipredDefault.

PARS

for types 'ct' and 'dt' only. May be of any structure, only needed to contain extra parameters for certain non-linear models.

driftScaled

for type 'ct' only: which of "DIFFUSION" and "CINT" to parameterise by what they accrue over one time unit given the process's own drift diagonal, rather than by their instantaneous rates. A diffusion sd s becomes s * sqrt(2 a / expm1(2 a)), s^2 the variance the process gains over one time unit from its own noise (its stationary variance where the drift is fast, the diffusion variance where it is slow); an intercept u becomes u * a / expm1(a). The fit and its likelihood are the default parameterisation's; what changes is that these parameters no longer have to move with the drift diagonal where fast mean reversion and more noise look alike, so that uncertainty and sampling are better behaved there. Only the diffusion's diagonal is rescaled, so its correlations keep their continuous-time meaning. Each such parameter moves, name, transform, random effects and predictor effects with it, to a new cell of PARS, and its own cell becomes an expression of that cell and the drift diagonal. Not available with covmattransform = 'cholesky'.

Details

A matrix cell is one of four things, and which one is decided by what its text looks like:

  • a number, which fixes the cell;

  • a parameter name, which frees it – the same name in two cells is one parameter, which is how an equality constraint is written;

  • a name with fields, adding the transform, the levels of id at which the parameter varies, the prior scaling of that variation, and the time-independent predictors acting on the parameter;

  • an expression, computed from the latent states, the time dependent predictors and the PARS cells.

Fields may be given in either of two interchangeable ways: in a fixed order separated by | ('mm||TRUE|0.5'), or by name ('mm, indvarying=TRUE, sdscale=0.5'). With nested ids, indvarying may name levels instead, as in 'mm||study' or 'mm, indvarying=c(subject, study)'. A cell containing | is read the ordered way; otherwise a cell with a field of the form name=value is read the named way, and an unrecognised field name is an error rather than part of a parameter name. ctPar writes the ordered form from the same named arguments, for when a specification is built programmatically.

Every name in an expression has to be declared: a latent state, a time dependent predictor, or a parameter given a PARS cell. Writing a fresh name inside an expression does not declare it, so DRIFT = matrix('-exp(dr)*eta1') needs PARS = c('dr') alongside it. To transform one parameter and nothing else the second field is shorter – DRIFT = matrix('dr|-exp(param)'). A cell may also read a time independent predictor by name, the subject's value of it – a study's known response rounding as MANIFESTVAR = 'rsd + mvar', say, with PARS = c('mvar|1e-10 + 5 * log1p_exp(2 * param)'); this is julia only. A subject's missing value of such a predictor is imputed or sampled as any other (optimcontrol$tipredMissing), and the cell reads that value. Population summaries evaluate the cell at a predictor value of zero, where they report predictor effects. A cell cannot reference a manifest variable or the time interval at all. Predictor effects on a parameter are set with the tipreds field.

Not every simple name in a cell is a parameter name. A latent state or time dependent predictor name is a reference to it, and any number of matrices may reference the same one. A name declared in a PARS cell is likewise a reference wherever else it is written: the PARS cell holds the parameter and carries its transform, and the cells that name it are rewritten into references to that cell.

Repeating a name that is not one of those is the equality constraint, and those cells are one parameter – one value, one random effect – so one transform has to apply to all of them. Cells that would transform it differently are refused, and that includes cells whose defaults differ: the DRIFT diagonal is bounded negative while its off-diagonals are not, so DRIFT = matrix('a', 2, 2) is not a way to tie the whole matrix to one parameter. To share a parameter across cells that would otherwise transform it differently, declare it in one PARS cell (PARS = c('a')) and then apply whatever transform each cell needs in an expression there – DRIFT[1,1] = '-exp(a)' and DIFFUSION[1,1] = 'exp(a)'. One parameter then serves cells that need different transforms, and each transform is stated where it applies. Giving the PARS cell the transform instead (PARS = c('a|-exp(param)')) and writing just the name in each cell works when one transform suits them all. Two PARS cells sharing one label are co-holders rather than references, so they must agree on the transform as any other repeated name must.

A predictor effect belongs to the parameter rather than to the cell, so cells that are one parameter must state the same tipreds. Naming an effect ('mypar||||age=shared') constrains every parameter given that name under that predictor to one coefficient, which costs one fewer free parameter and makes the two parameters' reported effects identical. Because one coefficient displaces each parameter's raw value equally, it means the same thing on the natural scale only where the transforms agree, so sharing a name between parameters transformed differently is refused. A name is scoped per predictor: the same label under two predictors is two coefficients. A number instead of a name ('mypar||||age=0.5') fixes the effect: no coefficient is estimated, and each subject's raw value of the parameter is displaced by 0.5 times their age, before the transform. Fixed effects are fitted by backend = 'julia' and used by ctGenerate; the stan backend refuses them.

A cell that is exactly a latent state or time dependent predictor name is taken as a reference to it rather than as a parameter of that name, and warns, because the two readings are different models. 'state[1]' and 'tdpreds[rowi, 1]' say the same thing without the ambiguity. In PARS it is an error rather than a warning: a bare name there reads as the declaration of a parameter, so write the reference as part of an expression ('log1p(exp(eta1))') if a state-dependent PARS cell is what is wanted.

Value

For type = 'ct' or 'dt', a model of class ctStanModel: $pars holds one row per matrix cell, and $matrices the same cells as matrices, editable either way. For type = 'omx', a list of matrices of class ctsemInit, for ctGenerate or ctModelConvertOMX.

See Also

ctPar for the fields a matrix cell can carry, ctFit for fitting the result.

Examples

### Frequentist example:
 ### impulse and level change time dependent predictor 
 ### example from Driver, Oud, Voelkle (2015)
 data('ctExample2')
 tdpredmodel <- ctModel(type='omx', n.manifest = 2, n.latent = 3, n.TDpred = 1, 
 Tpoints = 8, manifestNames = c('LeisureTime', 'Happiness'), 
 TDpredNames = 'MoneyInt', 
 latentNames = c('LeisureTime', 'Happiness', 'MoneyIntLatent'),
 LAMBDA = matrix(c(1,0, 0,1, 0,0), ncol = 3))

 tdpredmodel$DIFFUSION[, 3] <- 0
 tdpredmodel$DIFFUSION[3, ] <- 0
 tdpredmodel$T0VAR[3, ] <- 0
 tdpredmodel$T0VAR[, 3] <- 0
 tdpredmodel$CINT[3] <- 0
 tdpredmodel$T0MEANS[3] <- 0
 tdpredmodel$DRIFT[3, ] <- 0
 
 
###Bayesian example:
model<-ctModel(type='ct',
n.latent=2, latentNames=c('eta1','eta2'),
n.manifest=2, manifestNames=c('Y1','Y2'),
n.TDpred=1, TDpredNames='TD1', 
n.TIpred=3, TIpredNames=c('TI1','TI2','TI3'),
LAMBDA=diag(2))

Convert an old OpenMx-style ctsem model to the modern ctsem model format.

Description

Convert an old OpenMx-style ctsem model to the modern ctsem model format. ctStanModel is maintained as a backward-compatible alias.

Usage

ctModelConvertOMX(ctmodelobj, type = "ct", tipredDefault = TRUE)

Arguments

ctmodelobj

ctsem model object created by ctModel with type='omx'.

type

Either 'ct' for continuous time, or 'dt' for discrete time.

tipredDefault

Logical. TRUE sets any parameters with unspecified time independent predictor effects to have effects estimated, FALSE fixes the effect to zero unless individually specified.

Details

ctStanModel is maintained as a backward-compatible alias.

Call this when you have a matrix-list model built with ctModel(type='omx') and want to fit it: ctFit() takes a built model, and an omx object is not one until it comes through here. That conversion is the reason the function is exported.

New models need it only implicitly. ctModel(type='ct') and ctModel(type='dt') call it for you and return a model that is already built, so there is nothing left to convert.

Value

List object of class ctStanModel in the modern ctsem model format, with a pars data frame used by ctFit. Random effects are specified for any intercept type parameters (T0MEANS, MANIFESTMEANS, and or CINT), and time independent predictor effects are specified for all parameters. Adjust these after initial specification by directly editing the pars subobject, so model$pars, or by editing the pars-backed matrix view, so model$matrices.

Examples

model <- ctModel(type='omx', Tpoints=50,
n.latent=2, n.manifest=1, 
manifestNames='sunspots', 
latentNames=c('ss_level', 'ss_velocity'),
LAMBDA=matrix(c( 1, 'ma1' ), nrow=1, ncol=2),
DRIFT=matrix(c(0, 1,   'a21', 'a22'), nrow=2, ncol=2, byrow=TRUE),
MANIFESTMEANS=matrix(c('m1'), nrow=1, ncol=1),
# MANIFESTVAR=matrix(0, nrow=1, ncol=1),
CINT=matrix(c(0, 0), nrow=2, ncol=1),
DIFFUSION=matrix(c(
  0, 0,
  0, "diffusion"), ncol=2, nrow=2, byrow=TRUE))

modernmodel=ctModelConvertOMX(model)

Raise the order of a ctsem model's latent processes.

Description

Each latent in indices gains a derivative latent (named d and the latent's name), which drives the level with a fixed unit coefficient. That makes each a damped oscillator: a second order process, whose autocorrelation can dip below zero and rise again, which no first order process can.

Usage

ctModelHigherOrder(
  ctm,
  indices,
  diffusion = TRUE,
  crosseffects = FALSE,
  cint = FALSE,
  explosive = FALSE
)

Arguments

ctm

ctModel, type = 'ct' or type = 'omx'.

indices

Vector of integers, which latents to raise the order of.

diffusion

Move the process noise from the level to its derivative?

crosseffects

Move the effects of the other processes on this one to its derivative (divided by τ\tau, for a type = 'ct' model)? For a type = 'omx' model, its effects on them move as well.

cint

Move the continuous intercept to the derivative (divided by τ\tau, for a type = 'ct' model)?

explosive

For a type = 'omx' model, allow explosive (non equilibrium returning) processes. A type = 'ct' model's level keeps its own drift's transform, which decides that.

Details

A model from ctModel(type = 'ct') is rewritten cell by cell on its parameter table, with a new inertia time τ>0\tau > 0 for each latent (inertia_ and the latent's name, in PARS):

τx¨+x˙=ax+q dW,\tau \ddot{x} + \dot{x} = a x + q \, dW,

where aa is the level's own drift and qq its diffusion, both keeping their names and every setting – transform, individual variation, predictor effects. The derivative's row of DRIFT reads a/τa/\tau and −1/τ-1/\tau, and its diffusion q/τq/\tau. As τ\tau goes to zero the derivative follows the level at once and the process is the first order one it was, so the new model contains the old one with every old parameter at its value; from τ=1/(4∣a∣)\tau = 1/(4|a|) on, the process oscillates. The derivative starts where the process alone holds it at equilibrium: mean zero, uncorrelated with every state, and variance q2/(2τ)q^2/(2\tau), so the inertia time is the one new parameter. With the noise left on the level (diffusion = FALSE), or none, its initial mean, variance and covariances are free.

A model in the older matrix-list form from ctModel(type = 'omx') is written instead as the derivative pulled back by the level through an estimated coefficient (negative unless explosive), with the level's own autoregression as the derivative's damping. That is the same family of processes, but its first order member is reached only as the damping, the coefficient and the diffusion all grow without bound.

Value

extended ctModel

Examples

om <- ctModel(LAMBDA=diag(1,2),DRIFT=0, 
  MANIFESTMEANS=0,type='omx',Tpoints=4)
  
om <- ctModelHigherOrder(om,1:2)
print(om$DRIFT)

print(om$pars)

Improve a model step by step, from score tests and predictive checks

Description

Starting from a fitted model, repeatedly: screens it with ctScoreTest; refits it with the most strongly indicated relaxations, one at a time, warm started from the current estimates; keeps the best of them if the refit's likelihood confirms it; and stops when no relaxation is indicated and the model is adequate by ctFitAdequacy. When nothing is indicated but the model is not adequate, a slow process driving a latent (ctModelSlowComponent), a cycle such as a daily rhythm (ctModelPeriodicComponent), a process of its own for an item (ctModelUniqueProcess), raising the order of a latent process (ctModelHigherOrder) and any models given in candidates are tried the same way, each ranked by the adequacy check's evidence for it: a slow process for a latent whose updates stay correlated over time gaps (the latent family), a cycle at the period the periodic family points to, a process of its own for an item whose errors stay correlated with its own (the residuals family).

Usage

ctModelImprove(
  fit,
  covariates = NULL,
  relax = c("heterogeneity", "covariates", "statedependence", "volatility", "fixed",
    "level", "slow", "unique", "periodic", "order"),
  changes = NULL,
  candidates = NULL,
  alpha = 0.05,
  maxsteps = 10L,
  ntry = 3L,
  criterion = c("LR", "AIC", "BIC"),
  adequacy = TRUE,
  prune = TRUE,
  close = NULL,
  holdout = 0,
  form = "auto",
  periods = NULL,
  nsamples = 100L,
  projection = "auto",
  cores = getOption("mc.cores", 2L),
  checkpoint = NULL,
  verbose = TRUE
)

Arguments

fit

A fit from ctFit with backend = 'julia'.

covariates

Optional subject-level covariates the model may add, as for ctScoreTest.

relax

Which relaxations may be made.

changes

Optionally rows of ctModelProposals: the first round refits with these, in the order given, instead of screening, and keeps the first that improves the fit by criterion. With maxsteps = 1, one step of a search made by hand; later rounds screen as usual.

candidates

Optional list of alternative models (from ctModel) to try when the screen is clean but the checks fail.

alpha

Level for the screen (Benjamini-Hochberg within each family), for the likelihood ratio decision, and for adequacy.

maxsteps

Most refit rounds.

ntry

Proposals from the score tests refitted in a round, at most: one at a time, the strongest first, keeping the first that improves the fit. Proposals from the adequacy check are tried the same way until one improves or none is left, since the check does not change until the model does. Pruning and a held-out confirmation refit this many side by side.

criterion

'LR', 'AIC' or 'BIC'.

adequacy

Use ctFitAdequacy to decide when to stop.

prune

After the search, try dropping each kept change in turn and drop it if the likelihood does not need it given the others: a change kept early can be made redundant by one kept later.

close

Optional root mean square residual correlation regarded as a close enough fit to stop at; see Details.

holdout

Share of subjects, between 0 and 0.5, held out of the search to confirm its changes on.

form

How covariates are screened, as for ctScoreTest; 'threshold' lets the search add a covariate as a step.

periods

Periods for the adequacy check's periodic family, as for ctFitAdequacy; by default it scans.

nsamples

Generated datasets for the adequacy checks.

projection

How the adequacy checks remove the estimation, as for ctFitAdequacy.

cores

Cores for fitting and testing. With two or more, once a refit has taken longer than five minutes, the refits of pruning and of a held-out confirmation run side by side in worker processes, ntry at a time, as sampling chains do, each with an equal share of cores: they are independent, one fit uses its cores only partly, and a worker takes about half a minute to start. options(ctsem.improve.processes = FALSE) keeps them in this session.

checkpoint

Optional file the search saves its state to after every round and after pruning. Called again with the same file, the same fit and the same settings, it resumes after the last round saved; a file from a different search is refused. A long search on a cluster otherwise keeps nothing until it ends. With verbose, the refit running is also saved every ten minutes beside it (<checkpoint>.refit.rds): the round, the change, the iteration and the raw parameters it had reached, and, where a change's starts are refitted side by side, the start.

verbose

Report progress: each round's steps, each refit's time, and while a refit runs, its iteration, objective and gradient every ten minutes.

Details

The relaxations, in the language of the model:

heterogeneity

indvarying = TRUE for a parameter and the T0MEANS parameters, whose covariances with it the test took in; for a fixed MANIFESTMEANS cell, a parameter there varying about the fixed value as its stated mean.

covariates

a covariate added as a time independent predictor, with effects on the parameters its tests single out.

statedependence

a term d * eta_k * eta_m in a latent's drift, written into CINT with d a PARS parameter: a drift coefficient that varies with a state.

volatility

a latent's innovation sd scaled by exp(v * eta_m): innovations whose size depends on a state.

fixed

a cell fixed at zero freed: a drift cross effect, a diffusion covariance, a cross loading.

level

a parameter of the dynamics or error variances that depends on a subject's level (a random intercept) let vary over subjects, its random effect correlated with the intercepts (ctScoreTest's level terms).

slow

a slow process driving a latent through its drift, its effect started at one and its persistence at a tenth of the latent's; where the model has a slow process for another latent, that process driving this one too is tried first.

periodic

a cycle of the period the adequacy check points to, driving every latent with innovations of its own, each subject's phase set from the clock at its first row and free to deviate; the phase predictors are added to the data.

unique

a process of its own, persistent, loading on one item; and, where the pairs family finds two items' errors correlated within an occasion, a process the two share (ctModelUniqueProcess(shared = TRUE)): an error is meant to be random, and co-movement within an occasion that the latents' innovation covariances do not explain is read as a process faster than the occasions. Like the other structural changes it is tried once the score tests indicate nothing, ranked against them by the check's evidence, so that it does not absorb misfit that belongs to a cycle or to one item's own process.

errorcov

not by default: a measurement error covariance the score test flags is freed, for designs with a reason for correlated error, such as a method the items share, and no shared process is proposed.

order

a latent process made second order.

On a reduced-rank level (poprank) a heterogeneity result for a parameter that already varies through the shared dimensions gives it a variance of its own (popunique); without reduced rank, a significant dimension test makes the parameters its direction loads on vary together. Relaxations the score tests cannot tell apart (the same class) are not refitted in the same round.

Each random effect the search adds brings its correlations with the others, and with many effects for few subjects the data no longer determine them. When a kept change leaves the fit's curvature weak or negative along the population covariance (fit$identifiability), the covariance is refitted at one rank lower (poprank), starting from its leading dimensions, and the lower rank kept unless the likelihood ratio test needs the dropped dimension; it steps down until the fit is identified. Later random effects then load on the shared dimensions, and later heterogeneity results give own variances. Single-level models only.

When the search stops, each change kept is tried out again, by refitting without it (prune), and dropped if the others make it redundant.

A relaxation is kept when its refit improves on the current model by criterion: a likelihood ratio test at alpha divided by the number of refits tried at that step ('LR'), or a lower AIC or BIC (subjects as the sample size). The likelihood ratio test of a new random effect against the chi-square is conservative, as its variance starts on a boundary.

Value

An object of class ctModelImprove: fit (the final fit), history (every refit tried: the round, the relaxation, its screening p-value, the change in log likelihood and parameters, the criterion's verdict, and whether it was kept; a pruning row, family 'prune', says whether the change was dropped, and a 'rank' row whether the lower rank was kept), screens (the score tests of each round), adequacy (the final checks), confirmation (with holdout: each kept change's likelihood ratio test on the held-out subjects), and stopped (why the search ended: 'adequate', no adequacy family detected misfit; 'close fit', those that did were shown within close; 'not adequate, no candidate improved' or 'not adequate, nothing left to try', the checks still detect misfit; 'nothing indicated', with adequacy = FALSE, the screen was clean; 'maxsteps'). A refit that failed, or ended below the model it extends, is in history with a note saying so. With holdout, the held-out subjects take no part in the search; each kept change is tested once on them, at the end, at alpha. fit is the model with the confirmed changes refitted to every subject, and selected the search's own choice fitted to the subjects it searched on.

Error rates

The screen and the refit read the same data, so a refit confirms most of what the screen proposes, and the search's error rate is close to alpha per decision. Measured on simulated data (one latent, 80 subjects, 20 irregular rows, 40 searches each): with nothing missing, alpha = 0.05 kept something in 5 per cent of searches and alpha = 0.01 in none; an omitted covariate shifting a random intercept by half its sd was added in 75 and 55 per cent; an unrelated covariate never. criterion = 'BIC' kept something in 8 per cent, so at that size it is not the stricter choice. A drift that varied over subjects was never added as such at that size; a random diffusion or measurement error variance stood in for it in 10 per cent, since what the data showed was subjects differing in variance. A search makes a decision per family it screens, so its error rate grows with the families: on two processes with three items each (60 subjects, about 48 rows each), where the screen also tests state dependence and volatility in each process, alpha = 0.05 kept something in 4 of 20 searches with nothing missing (a volatility term twice, state dependence once, a random effect once), and alpha = 0.01 is the stricter choice. With an omitted cross effect, a process of an item's own and a daily cycle in the same data, the search kept the cross effect in 19 of 20 searches, the cycle in 16 and the item's process in all 20.

The result is a model chosen by the data it is fitted to, and its standard errors and tests do not account for that choice (MacCallum, Roznowski & Necowitz, 1992). holdout sets aside a share of the subjects: the search runs on the rest, and each change it keeps is then tested on the held-out subjects alone, by refitting the final model without it there; confirmation says which held, and the final model, refitted to every subject, keeps only those. Every step, and every refit tried and rejected, is in history.

With many observations a model can be rejected by the checks for departures that are small on these summaries. With close, the search also stops when every check that fails is a residual or within-row correlation family whose size is shown below close (ctFitAdequacy).

Each refit is judged by criterion; with 'LR', the likelihood ratio against a chi-square on the parameters the change adds, at alpha divided by the refits the round tries. That reference is exact for none of the changes the search makes: a variance added at zero sits on its boundary, where the chi-square is conservative, and a process or cycle added brings parameters that mean nothing when its coupling is zero, where it can err either way. Screening, rounds and pruning add further choices the level does not count. The search is a way of developing a model, and its p-values describe the steps it took rather than testing the result. A refit that ends below the model it extends has not reached an optimum, and is recorded as such.

On the Laplace route the filter integrates a fit's intercept-type random effects exactly where it can, and Laplace approximates only the others (the two-step term); a change that makes the dynamics read the state takes that away, and its refit is on the one-step term (optimcontrol$laplace_onestep in ctFit). Beside random dynamics the two approximate their integrals differently, so such a refit is compared with the current model refitted one-step from its own estimate: a fast comparison of two approximations. To compare integrals instead, start the search from a fit with intoverpop = 'quadrature': every refit is then refined the same way.

See Also

vignette('score-tests').


Does a model or fit have context-dependent (nonlinear) matrix cells?

Description

A ctsem matrix cell may be specified as an expression referencing a latent process or a time dependent predictor – for instance DRIFT[1,1] = '-log1p(exp(param)) * eta2'. Such a cell has no single value, only a value at a given state and set of predictor values, and the package's summaries and plots must therefore name the point they evaluated it at.

Usage

ctModelIsNonlinear(x)

Arguments

x

A ctStanModel, ctStanFit or ctJuliaFit.

Details

References to an individually varying parameter's carrier state (the intoverpop representation of a random effect) do not count: those are parameters, not dynamics, and are reported exactly.

Value

Logical.

Examples

ctModelIsNonlinear(ctstantestfit)

Put subjects' levels in the latent processes rather than in the indicators

Description

In the usual model a subject's level sits in random manifest intercepts (MANIFESTMEANS), so the latent state is the subject's deviation from it. That is the same model, for linear dynamics, as one whose latent process carries the level itself through a random latent intercept (CINT) – but not once the dynamics depend on the state: a term that reads the state reads the deviation in the first, the level in the second. Persistence that differs near the bottom of a scale, or innovations that grow with how high a subject usually is, are dependence on the level, which state dependence written into the first model cannot express and its tests (ctScoreTest) cannot see; ctScoreTest(tests = 'level') asks about it between subjects.

Usage

ctModelLatentLevel(ctm, latents = NULL)

Arguments

ctm

A model from ctModel(type = 'ct').

latents

Which latents, by name or position; by default every one.

Details

For each latent in latents with a marker – an indicator whose loading on it is fixed at 1 and on every other latent at 0 – whose MANIFESTMEANS intercept varies over subjects, the intercept's variation moves into the latent: CINT becomes a free parameter, cint_ and the latent's name, with the intercept's random effect and predictor effects; the marker's intercept is fixed at 0; and the latent's initial mean is its stationary one (T0MEANS 'stationary'), so a subject starts at their own level. The other indicators' intercepts are left as they are, fixed or varying. A latent without such a marker, or whose CINT is already free, is left as it is, with a message.

The level then rides on the dynamics: CINT is the level times the rate of return to it, so a level's random effect is no longer one an indicator's filter can integrate on its own, and a fit on the Laplace route carries it as a coordinate.

Value

The model, with the latents whose level moved in attribute moved.

See Also

ctScoreTest for the tests of level dependence.

Examples

m <- ctModel(type = 'ct', manifestNames = c('Y1', 'Y2'),
  latentNames = c('eta1', 'eta2'), LAMBDA = diag(2), CINT = matrix(0, 2),
  MANIFESTMEANS = matrix(c('mm1', 'mm2'), 2))
lm <- ctModelLatentLevel(m)
attr(lm, 'moved')
p <- lm$pars
p[p$matrix %in% c('CINT', 'MANIFESTMEANS', 'T0MEANS'), c('matrix', 'row', 'param',
  'value', 'indvarying')]

Generate and optionally compile latex equation of subject level ctsem model.

Description

Generate and optionally compile latex equation of subject level ctsem model.

Usage

ctModelLatex(
  x,
  matrixnames = TRUE,
  digits = 3,
  linearise = inherits(x, "ctStanFit"),
  textsize = "normalsize",
  folder = tempdir(),
  filename = paste0("ctsemTex", as.numeric(Sys.time())),
  tex = TRUE,
  equationonly = FALSE,
  compile = TRUE,
  open = TRUE,
  includeNote = TRUE,
  minimal = FALSE,
  splitDynamics = TRUE,
  splitMeasurement = TRUE,
  savepng = FALSE
)

Arguments

x

A model from ctModel, or a fit from ctFit on either backend (ctStanFit or ctJuliaFit). Given a fit, the estimates are substituted for the parameter labels.

matrixnames

Logical. If TRUE, includes ctsem matrix names such as DRIFT and DIFFUSION under the matrices.

digits

Precision of decimals for numeric values.

linearise

Logical. Show the linearised normal approximation for subject parameters and covariate effects, or the raw parameters? Only relevant where a subject parameter distribution is shown, and not available for a ctJuliaFit, which holds that distribution on the raw scale.

textsize

Standard latex text sizes – tiny scriptsize footnotesize small normalsize large Large LARGE huge Huge. Useful if output overflows page.

folder

Character string specifying folder to save to, defaults to temporary directory, use "./" for working directory.

filename

filename, without suffix, to output .tex and .pdf files too.

tex

Save .tex file? Otherwise latex is simply returned within R as a string.

equationonly

Logical. If TRUE, output is only the latex relevant to the equation, not a compileable document.

compile

Compile to .pdf? (Depends on tex = TRUE)

open

Open after compiling? (Depends on compile = TRUE)

includeNote

Include text describing matrix transformations and subject notation? triangular matrices (which results in a covariance or Cholesky matrix) is shown – the latter is a more direct representation of the model, while the former is often simpler to convey.

minimal

if TRUE, outputs reduced form version displaying matrix dimensions and equation structure only.

splitDynamics

Logical. If TRUE, split latent process dynamics across deterministic and random change lines. If FALSE, show the full dynamics equation on one line.

splitMeasurement

Logical. If TRUE, split measurement equations across observation and observation noise lines. If FALSE, show the full measurement equation on one line.

savepng

Logical. If TRUE, renders the equation as a png file instead of a pdf, viewing the png in RStudio viewer when available.

Value

character string of latex code. Side effects include saving a .tex, .pdf, and displaying the pdf.

Examples

ctmodel <- ctModel(type='ct', 
n.latent=2, n.manifest=1, 
manifestNames='sunspots', 
latentNames=c('ss_level', 'ss_velocity'),
LAMBDA=matrix(c( 1, 'ma1' ), nrow=1, ncol=2),
DRIFT=matrix(c(0, 1,   'a21', 'a22'), nrow=2, ncol=2, byrow=TRUE),
MANIFESTMEANS=matrix(c('m1'), nrow=1, ncol=1),
CINT=matrix(c(0, 0), nrow=2, ncol=1),
DIFFUSION=matrix(c(
  0, 0,
  0, "diffusion"), ncol=2, nrow=2, byrow=TRUE))
  
l=ctModelLatex(ctmodel,compile=FALSE, open=FALSE)
cat(l)

Matrix view for ctStanModel objects

Description

Access or replace the matrix representation of a ctStanModel while keeping the pars data frame as the canonical model specification.

Usage

ctModelMatrices(x)

ctModelMatrices(x) <- value

Arguments

x

A ctStanModel object.

value

A named list of matrices, typically copied from ctModelMatrices(x) or x$matrices.

Details

ctModelMatrices(x) reconstructs matrices from x$pars. The replacement form updates matching matrix, row, and col entries in x$pars. Numeric matrix entries become fixed values; non-numeric entries become parameter labels. Metadata such as transforms and individual variation settings is preserved when possible and reset to fit-ready defaults when fixed/free status changes.

The same view is available as x$matrices. Direct replacements such as x$matrices$DRIFT[1, 2] <- "cross" update x$pars; detached copies must be assigned back with x$matrices <- mats. A placeholder matrices element is stored in the object so the view is visible in names(x) and printed objects, but the matrix values are always derived from x$pars.

The view also holds the population spread of the individually varying parameters: RAWPOPVAR for the subject level and RAWPOPVAR_<idname> for each grouping level, one row and column per parameter varying at that level, also reachable as x$RAWPOPVAR. The diagonal holds standard deviations and the lower triangle correlation coordinates, both for the raw, untransformed parameters, the scale a fit reports them on. A number fixes a cell, on every route, and a label leaves it free. ctCov writes a whole matrix from a covariance, which is also how to state T0VAR, DIFFUSION or MANIFESTVAR as the covariance it is meant to be.

Value

ctModelMatrices() returns a named list of matrices. The replacement form returns the updated ctStanModel.


Add a cycle, such as a daily rhythm, that drives a model's latent processes

Description

Adds a pair of latents, <name>_cos and <name>_sin, that turn at a fixed period with no noise of their own – DRIFT[cos, sin] = -2 pi / period and DRIFT[sin, cos] = 2 pi / period – so they trace the cosine and sine of the time of day (or week, or whatever the period is). Each latent in indices is driven by both, through estimated effects DRIFT[k, <name>_cos] and DRIFT[k, <name>_sin], so it is pushed through the cycle with an amplitude and phase the data decide, and responds with its own dynamics: a fast process follows the cycle closely, a slow one smooths and delays it.

Usage

ctModelPeriodicComponent(
  ctm,
  period,
  indices = seq_len(ctm$n.latent),
  name = "cycle",
  individual = TRUE
)

Arguments

ctm

A model from ctModel(type = 'ct').

period

The cycle's period, in the data's time units: 24 for a daily rhythm with time in hours.

indices

Which latents the cycle drives, by position.

name

Prefix for the added latents, predictors and parameters.

individual

Let each subject's cycle deviate from the clock.

Details

The cycle is set to the clock at each subject's first observation, from two time independent predictors, <name>_cos0 and <name>_sin0, the cosine and sine of that time over the period, which ctPeriodicPhases adds to the data. So the data's time must count from an origin shared by every subject – hours since midnight of a common day, say – for the cycle to mean the same clock time for all. With individual = TRUE each subject's cycle may also be shifted from the clock and differ in strength, by an initial deviation of the pair with an estimated standard deviation, which the filter learns from each subject's data: rhythms that differ between people, or a time origin that is not shared, are then still modelled.

The effects start at zero, where they have a gradient: unlike a slow process, the cycle is known, so a refit from the model without it needs no particular start.

Needs the julia backend, which reads time independent predictors in the initial means.

Value

The extended model.

See Also

ctPeriodicPhases for the predictors it reads; ctFitAdequacy's periodic family, which finds cycles a model lacks; ctModelImprove, which adds one where that family points.

Examples

m <- ctModel(type = 'ct', manifestNames = c('Y1', 'Y2'),
  latentNames = c('eta1', 'eta2'), LAMBDA = diag(2))
m2 <- ctModelPeriodicComponent(m, period = 24)
m2$latentNames
m2$TIpredNames
p <- m2$pars
p[p$matrix %in% 'DRIFT' & (p$row > 2 | p$col > 2) &
  (!is.na(p$param) | p$value != 0), c('matrix', 'row', 'col', 'param', 'value')]

Changes the score tests and adequacy checks propose

Description

The changes ctModelImprove would consider for a fit: the relaxations its score tests indicate and, given an adequacy check, the structural changes the check's families point to – a process of an item's own or shared by two, a slow process, a cycle, a second order. Rows can be handed to ctModelImprove(changes = ), which refits with them and judges each by the search's criterion: a search made a step at a time, by hand.

Usage

ctModelProposals(
  fit,
  score = NULL,
  adequacy = NULL,
  covariates = NULL,
  relax = c("heterogeneity", "covariates", "statedependence", "volatility", "fixed",
    "level", "slow", "unique", "periodic", "order"),
  alpha = 0.05,
  cores = getOption("mc.cores", 2L)
)

Arguments

fit

A fit from ctFit with backend = 'julia'.

score

A ctScoreTest of fit. Run when neither it nor adequacy is given.

adequacy

Optionally a ctFitAdequacy of fit, for the structural changes.

covariates, relax, alpha, cores

As for ctModelImprove.

Details

Unlike a search's rounds, every indicated change is listed, including relaxations the tests cannot tell apart (one class), of which a round refits only the strongest; and the structural changes are listed beside the relaxations, where a round turns to them only once the screen is clean.

Value

A data frame of class ctModelProposals, one row per change, strongest first within the relaxations and within the structural changes: change (as a search's history names it), family, p (the evidence ranked by), class, and proposal, what ctModelImprove() needs to make the change.

See Also

vignette('score-tests').


Add slow processes that drive a model's latent processes

Description

Each latent in indices gains a slow latent, named slow_ and the latent's name, that drives it through the drift: an estimated effect DRIFT[k, slow_k] moves the level the latent returns to with the slow process, so a process can wander over weeks while it fluctuates over hours. Its equilibrium is DRIFT[k, slow_k] / |DRIFT[k, k]| times the slow process, so a person with faster dynamics follows the slow process less closely.

Usage

ctModelSlowComponent(ctm, indices = seq_len(ctm$n.latent))

Arguments

ctm

A model from ctModel(type = 'ct').

indices

Which latents get a slow process, by position.

Details

A slow process is slower than the latent it drives by construction: its drift is the latent's times inv_logit(r), with r a PARS parameter named drift_ and the slow process's name, so its rate is a fraction of the latent's. Left free, the two rates can trade places: with only the latent measured, a latent of drift a driven by a slow process of drift b has the spectrum of one with the two exchanged and the effect adjusted, so the "slow" process can come out the fast one, and refits from other starting values did. The latent's drift parameter is declared in PARS for this, keeping its name, transform, individual variation and predictor effects, so where the latent's rate varies over people the slow process's follows it. Where the latent's drift is fixed the fraction is of that value; where it is an expression the slow process's persistence is left free, as before.

A slow process has innovations of unit variance (the effect carries its scale), correlated with those of the other slow processes, including ones added earlier, and not with the fast ones. It has no loadings and no intercept, and no individual variation or predictor effects. Its initial mean and variance are its stationary ones (T0MEANS and the T0VAR diagonal 'stationary', see ctFit): zero and 1 / (2 |a|) for persistence a. Its initial covariance with the latent it drives is free – that latent follows it, so the two start related – and with the other states zero.

The effect has to start away from zero. At zero neither the slow process nor its effect has a gradient, so an optimiser started there stays. When refitting from a model without the slow process, start the effect at one (the slow process's sign is not identified, so either sign will do) and the fraction small; ctModelImprove starts it at a tenth.

Value

The extended model.

See Also

ctModelHigherOrder for the other structural extension the search tries.

Examples

m <- ctModel(type = 'ct', manifestNames = c('Y1', 'Y2'),
  latentNames = c('eta1', 'eta2'), LAMBDA = diag(2))
m2 <- ctModelSlowComponent(m, 1:2)
m2$latentNames
p <- m2$pars
p[p$matrix %in% c('DRIFT', 'DIFFUSION') & (p$row > 2 | p$col > 2) &
  (!is.na(p$param) | p$value != 0), c('matrix', 'row', 'col', 'param', 'value')]

Give indicators processes of their own

Description

Each manifest variable in items gains a latent, named u_ and the variable's name, that loads on it alone with a fixed unit loading and has its own persistence and innovations. The variable then carries, beside what it shares through the model's other latents and its measurement error, a part that is specific to it and carried over time: an item whose wording picks up something besides the construct, and picks it up persistently.

Usage

ctModelUniqueProcess(ctm, items, shared = FALSE, name = NULL)

Arguments

ctm

A model from ctModel(type = 'ct').

items

Which manifest variables, by name or position.

shared

One process for all of items, named u_ and their names joined by _, rather than one each.

name

The new latents' names, in place of those: one for a shared process, one per item otherwise. For a second process on the same items, at another time scale.

Details

Without it, that part can only go to the measurement error, which is independent over time, so the item's prediction errors stay correlated with its own at nearby times – what the residuals family of ctFitAdequacy shows for that item alone – or to the shared latents, whose dynamics it then distorts. The specific process and the measurement error are told apart by persistence: the one carries over time, the other does not.

The process has no intercept (the item's own carries its level), ctsem's default drift transform, so it is mean reverting, and innovations independent of every other latent's. It starts from its stationary distribution: mean zero and the variance its persistence and innovations imply (the T0VAR diagonal 'stationary', see ctFit).

With shared = TRUE the items share one process instead, loading on the first at one and on the others freely: what moves them together beyond the model's other latents. Measurement errors that are correlated within an occasion are usually better read this way – an error is meant to be random, and co-movement it does not carry over to the next occasion is a process faster than the occasions – unless the design gives a reason for correlated error, such as a method shared by the items.

Value

The extended model.

See Also

ctModelImprove, which adds one for the items the adequacy check points to.

Examples

m <- ctModel(type = 'ct', manifestNames = c('Y1', 'Y2', 'Y3'),
  latentNames = 'eta', LAMBDA = matrix(c(1, 'l2', 'l3'), 3))
m2 <- ctModelUniqueProcess(m, 'Y2')
m2$latentNames
p <- m2$pars
p[p$col %in% 2 & p$matrix %in% c('LAMBDA', 'DRIFT', 'DIFFUSION'),
  c('matrix', 'row', 'col', 'param', 'value')]
ctModelUniqueProcess(m, c('Y2', 'Y3'), shared = TRUE)$latentNames

Temporal and contemporaneous networks of a continuous time model

Description

The two graphs network psychometrics draws from a multivariate time series – the directed temporal network and the undirected contemporaneous network – computed from a ctsem model specification or from a fit on either backend.

Both are functions of the time interval, which is what a continuous time model has to say here that a discrete time VAR does not. Call it at several dt and the networks change shape, not merely scale.

Usage

ctNetwork(
  x,
  dt = 1,
  state = NULL,
  standardise = TRUE,
  impulseType = "unit",
  threshold = 0,
  networks = c("temporal", "contemporaneous"),
  plot = FALSE,
  quiet = FALSE,
  ...,
  observational
)

Arguments

x

A ctStanModel from ctModel, or a fit from ctFit on either backend.

dt

Time interval for the temporal and contemporaneous networks. A whole number of steps for a discrete time model. Length one; to see the interval dependence, call ctNetworkPlot with a vector.

state

Where a state dependent cell is evaluated, when x is a fit: 'T0MEANS' (the default, and what ctSummaryMatrices uses), 'mean' for the mean smoothed latent state, 'asymptotic' for the system's own fixed point, or a numeric state vector. Only backend='julia' can re-materialise the matrices elsewhere, so anything but the default is an error on a stan fit rather than being quietly ignored. Irrelevant to a linear model, and to an unfitted specification.

standardise

If TRUE (the default), temporal edges are in standard deviation units of the processes themselves, using asymDIFFUSIONcov, so edges between processes on different scales are comparable. Falls back to FALSE, with a message, when the system has no stationary variance to standardise by.

impulseType

What a one unit change in a process brings with it; see ctDiscretePars, whose argument this is. 'unit' (the default) is an impulse of one to that process alone – the partial regression, and a property of the dynamics alone.

threshold

Edges with abs(weight) at or below this are dropped from $edges and from any plot. The matrices are never thresholded.

networks

Which networks appear in $edges and in the plot. Any of 'temporal', 'contemporaneous', 'asymptotic', 'measurement'. All four matrices are computed regardless.

plot

If TRUE, returns the ctNetworkPlot of the result instead of the result.

quiet

Suppress the messages about how the numbers were arrived at.

...

Passed to ctSummaryMatrices when x is a fit, and to ctNetworkPlot when plot=TRUE.

observational

Deprecated. Use impulseType.

Details

Three networks over the latent processes, plus the measurement model:

temporal

expm(DRIFT * dt), the discrete time autoregression and cross-lagged matrix at interval dt. Directed, with the diagonal as self-loops (the autoregressions). Cell [i,j] is the effect of process j now on process i one interval later, so an edge runs from the column to the row. This is ctDiscretePars at a single interval, and shares its impulseType and standardise arguments.

contemporaneous

Partial correlations of the innovation covariance accumulated over dt, Q(dt) = int_0^dt expm(DRIFT s) DIFFUSIONcov t(expm(DRIFT s)) ds. This is the continuous time counterpart of the graphicalVAR contemporaneous network. Partial correlations are scale free, so as dt shrinks this tends to the partial correlations of DIFFUSIONcov itself, and as dt grows to those of the asymptotic covariance.

asymptotic

Partial correlations of asymDIFFUSIONcov, the long run within-subject covariance – the network a cross-sectional study of a stationary system would see. NULL if the system is not stationary.

measurement

LAMBDA, as a bipartite manifest by latent graph. Directed from latent to manifest.

From a fit, the matrices are the posterior medians ctSummaryMatrices reports. From an unfitted specification, a fixed cell is its own value and a free cell takes the value its transform gives at a raw parameter of zero, which is where estimation starts; how many DRIFT, DIFFUSION and LAMBDA cells were treated that way is reported, those being the only matrices an edge can come from.

Value

A list of class ctNetwork:

temporal, contemporaneous, asymptotic, measurement

Weight matrices with the model's own names, ready to hand to qgraph::qgraph(). asymptotic may be NULL.

edges

Tidy data frame: network, from, to, weight, directed, self. Thresholded, and restricted to networks.

nodes

Data frame of name and type ('latent' or 'manifest').

DRIFT, DIFFUSIONcov, asymDIFFUSIONcov, innovation

The matrices the networks were derived from, so every number above can be checked. On a julia fit DRIFT is the drift's Jacobian at the evaluation point, DRIFT itself where no cell reads the state (see the section on state dependent models).

With attributes dt, standardise, impulseType, threshold, networks, continuoustime, source, stateLabel (the evaluation point, for a fit) and stateDependent (the cells with no single value, or NULL).

State dependent and nonlinear models

A DRIFT, DIFFUSION or LAMBDA cell may be written as an expression referencing a latent process, and then there is no such thing as the network. What a fit reports is the system linearised at one point of the state space, so the graph drawn from it is that linearisation's graph: somewhere else the edge weights differ, and for a strongly nonlinear model so does which edges are there at all. The temporal and contemporaneous networks then come from the drift's Jacobian at that point (JAx) and the measurement network from the measurement's (Jy), which differ from DRIFT and LAMBDA there where a cell is nonlinear in the state: a cell a + b*Y in the column of process Y has slope a + 2*b*Y. A stan fit reports no Jacobians, so its edges are the cells' values at T0MEANS, and it says so.

When any such cell is present, ctNetwork says so once, names the evaluation point, and records the cells in attr(x, 'stateDependent'); ctNetworkPlot puts the point in the figure's subtitle so a saved plot carries its own caveat. Use state to move the point, and ctPhasePortrait or ctStateDependencePlot to see the variation itself rather than one slice of it. Nothing is said for a linear model, whose matrices are the same everywhere.

From an unfitted specification such a cell is reported and left out instead, since filling it in would draw a model that was not written.

See Also

ctNetworkPlot to draw it, ctDiscretePars for the temporal network across a continuum of intervals, ctPhasePortrait and ctStateDependencePlot for what a state dependent system does away from the point these edges were evaluated at.

Examples

# Two processes, b driven by a, drawn at one interval.
m <- ctModel(type='ct', n.latent=2, n.manifest=2,
  manifestNames=c('y1','y2'), latentNames=c('a','b'),
  LAMBDA=diag(2),
  DRIFT=matrix(c(-1, 0, .5, -2), 2, 2, byrow=TRUE),
  DIFFUSION=matrix(c(1, 0, .3, 1), 2, 2, byrow=TRUE))

net <- ctNetwork(m, dt=1, standardise=FALSE)
round(net$temporal, 4)          # exp(-1) and exp(-2) on the diagonal
round(net$contemporaneous, 4)
net$edges

Draw the temporal and contemporaneous networks of a continuous time model

Description

Plots what ctNetwork computes, in the style of the network psychometrics literature: signed edges, width by magnitude, arrowheads and self-loops on the directed temporal network, and no arrowheads on the undirected contemporaneous one.

Give dt more than one value and the same networks are drawn side by side at each interval, which is the continuous time point: the graph a study would have recovered depends on how often it measured.

Usage

ctNetworkPlot(
  x,
  dt = 1,
  state = NULL,
  networks = c("temporal", "contemporaneous"),
  threshold = 0,
  engine = c("ggplot", "qgraph"),
  maxwidth = 3,
  arrowsize = 2.6,
  nodesize = 10,
  labelsize = 3.2,
  poscolour = "#2166AC",
  negcolour = "#B2182B",
  title = "auto",
  ...
)

Arguments

x

A ctStanModel, a fit, or a ctNetwork from ctNetwork.

dt

One or more time intervals. Ignored, with a message, when x is already a ctNetwork.

state

Where a state dependent cell is evaluated; see ctNetwork. Ignored, with a message, when x is already a ctNetwork, which was built at a point of its own. When the system is state dependent the point appears in the plot's subtitle, so a saved figure carries its own caveat; a linear system gets no subtitle.

networks

Which networks to draw; see ctNetwork. Each becomes a panel, as does each dt.

threshold

Edges with abs(weight) at or below this are not drawn.

engine

'ggplot' (the default) or 'qgraph'.

maxwidth

Line width of the largest edge drawn.

arrowsize

Arrowhead length in mm.

nodesize, labelsize

Node and label size.

poscolour, negcolour

Colours for positive and negative edges. The defaults are a colour-blind safe blue and red; the literature's green and red are 'darkgreen' and 'red'.

title

Panel title, or NULL for none. 'auto' names the source and the interval.

...

Passed to ctNetwork, or to qgraph::qgraph() when engine='qgraph'.

Details

Drawn with ggplot2, which ctsem already depends on, so no extra package is needed. engine='qgraph' hands the weight matrix to qgraph::qgraph() instead, if that package is installed – ctNetwork's matrices are in the shape qgraph expects, so qgraph::qgraph(ctNetwork(fit)$temporal, directed=TRUE) needs nothing from this function.

Value

A ggplot2 object, which can be modified further or printed. With engine='qgraph', the qgraph object, invisibly, having drawn.

Examples

m <- ctModel(type='ct', n.latent=3, n.manifest=3,
  manifestNames=c('y1','y2','y3'), latentNames=c('a','b','c'),
  LAMBDA=diag(3),
  DRIFT=matrix(c(-.4,0,0, .3,-.5,0, 0,.4,-.6), 3, 3, byrow=TRUE),
  DIFFUSION=matrix(c(1,0,0, .3,1,0, 0,-.4,1), 3, 3, byrow=TRUE))

# The indirect path a -> b -> c is invisible at a short interval and present
# at a long one -- the same model, two networks.
ctNetworkPlot(m, dt=c(.2, 2), networks='temporal')

Optimiser settings for ctFit

Description

The names ctFit reads from optimcontrol. A name means the same thing on both backends. One a backend does not have is refused by name, unless its value asks for nothing that backend does not already do (gradient = 'adjoint' on stan, nsubsets = 1 on julia), and a misspelled name is an error. Leaving a name unset keeps that backend's own default.

Stopping

tol

Both. Stop when the objective changes by less than this. Default 1e-8 on stan, off on julia.

g_tol

Both. Stop when the l2 norm of the gradient falls below this. Default 1e-8 on julia, off on stan.

x_tol

Both. Stop when the parameter step falls below this. Off by default.

maxiter

Both. Iteration cap. Default 1000 on julia, mize's on stan.

lbfgs_scores

Julia. Default TRUE. Runs beside L-BFGS a second model of the curvature, the centred outer product of the per-unit scores plus the prior's precision and a learned correction, and takes each step on whichever model predicted the last step's gain better; while a fit is batched, only on the whole data.

lbfgs_local

Julia. Default TRUE. On the Laplace route, where some parameters are read by few units (a study's, a group's, an item few answered), starts L-BFGS's inverse Hessian from their exact Hessian columns, differenced over those units alone, and the score model of the rest; built once the optimisation has spent what it costs.

lbfgs_memory

Both. How many curvature pairs L-BFGS keeps. Default on julia one per free parameter, at least 20 and at most 1000; mize's on stan.

stallretries

Both. Default 2. How often a fit that stopped short of a maximum is tried again: stan restarts from fresh random values when the gradient exceeds stalltol per data point; julia resumes from the better point its probe found by pulling flat coordinates back, or from where it stopped when the check finds the fit short, keeping a resumed stage only if it improves the objective, and stopping one that stops gaining. 0 turns it off.

Starting values

initsd

Both. Default 0.01. The sd of the random start, on the raw scale.

datastart

Julia. Default TRUE. Starting values for the diagonals of DRIFT, DIFFUSION, T0VAR and MANIFESTVAR are read off the data's scale – through the link, so a count is read through log1p and a binary or ordinal indicator contributes none – rather than from one fixed point whatever the units. They are kept only where the objective can be evaluated there; otherwise the fit starts from the default and says so. Supplied inits are never overridden.

carefulfit

Both. A rough first pass with ctsem's normal(0,1) raw priors on every coordinate, for starting values; the fit's own objective is then maximised from there. Skipped with priors = TRUE, which already has those priors, and when inits are supplied. Default TRUE on both backends. On julia it runs to its own convergence, at most 200 iterations, and a number sets that cap; it keeps the fit from starting where a boundary (a correlation at one, a variance at zero) has already captured it. fit$optim$carefulfit records whether it ran.

The optimiser

stochastic

Both. Start with ctsem's stochastic-gradient optimiser and finish with L-BFGS; 'auto' does so above 50 parameters. Default TRUE on stan, FALSE on julia. On large, poorly conditioned models it can be much faster than L-BFGS alone, and it can carry a fit further toward a degenerate limit, such as a variance running to zero. fit$optim$sgd_iterations records the phase.

gradient

Julia. 'adjoint' (reverse mode, the default) or 'forward'. The same gradient; the adjoint's cost does not grow with the number of parameters.

batch

Julia. Default TRUE, or FALSE when inits are supplied. Start on a random subset of the subjects and grow it as the fit needs more data; needs at least 80 subjects or top-level groups. A supplied start is not batched from, since the subsets' gradients walk off a point already chosen; TRUE asks for it anyway.

coreset

Julia. Default 'auto'. Run stretches of L-BFGS on a weighted few of the units (subjects, or top-level groups where random effects are integrated by Laplace), chosen so that their changes in gradient and their spread reproduce all the units', each stretch checked by a full gradient; the full objective finishes, so the estimate and its standard errors are those of every unit. 'auto' runs them for data of at least 200 subjects or top-level groups and 10,000 data cells (rows times manifest variables), where they save time; on smaller data they save none and can change which of several maxima a fit reaches. TRUE runs them whenever a coreset is expected to save gradients, FALSE never. fit$optim$coreset_sizes records the stages.

explosive_forward

Julia. Default FALSE. For a subject whose predictions grow more than 100-fold between observations (explosive dynamics over a long gap), take its gradient by forward mode instead of the reverse pass, as a cross-check: the reverse pass is accurate there too. The first such subject in a session compiles the filter again, minutes on a large model. The fit counts such subjects either way (fit$optim$explosive_passes) and warns when any is explosive at the estimate.

lbfgs_diagonal

Julia. Default TRUE. L-BFGS's initial inverse Hessian takes a scale per parameter, learned from the curvature pairs (Gilbert and Lemarechal); FALSE uses one scale for all, which can be much slower on large models. Like stochastic, it can carry a fit further toward a degenerate limit.

lbfgs_gll

Julia. Default 0. A step is accepted against the worst of the last W objective values (Grippo, Lampariello and Lucidi), so it may cross a curved valley.

lbfgs_nonmonotone

Julia. Between 0 (the default) and 1. A step is accepted against a running average of recent objective values (Zhang and Hager).

initial_alpha

Julia. Default 0.1. The length of the first trial step, in raw units.

precondition

Julia. Default FALSE. TRUE starts L-BFGS's per-parameter scaling from the model's transforms rather than learning it from the curvature alone. It made fits whose intercepts start far from the data settle in worse optima.

saem

Julia, intoverpop = 'laplace'. Default FALSE. Start with SAEM: the random effects are sampled from their conditional distribution rather than integrated by the Laplace approximation, heading for the exact marginal posterior mode. The usual optimiser then polishes and certifies its point on the Laplace objective, so the estimate usually ends where a fit without SAEM does; what it can change is which maximum is reached when the Laplace objective has more than one. Its point carries Monte Carlo error, so where the posterior itself has several maxima – weakly identified random-effect scales and correlations – that error, not the data, can decide which one is reached. So the fit also finishes from its own start without SAEM, one more optimisation, and keeps whichever maximum has the higher log posterior on the quadrature-corrected objective (quadrature for each unit narrow enough, as the refined fit screens), where a maximum the Laplace approximation over-credits loses; fit$optim$saem_guard records both. Not a better approximate estimate on its own – for that, sample with optimize = FALSE. A number caps its iterations (TRUE is 10000). fit$optim$saem_iterations, saem_settled, saem_trend, saem_chains, saem_acceptance and saem_trace record the phase.

saem_proposal

Julia. SAEM's proposal, 'rw' or 'laplace'.

restarts

Julia. Default 0. Random restarts for a fit that has still not converged after its corrections – typically a likelihood with more than one basin – each from the fit's own start plus normal noise of sd restartsd (default 1) on the raw scale, keeping the best if it is better. They run in worker processes when cores > 1 and the fit took more than 30 seconds. Not when inits or maxiter were supplied. fit$optim$restarts records each start.

The finish and its certification (julia)

newton

Default TRUE. Newton steps once close, on an exact Hessian updated after each step and formed again when its steps stop paying for it, ending on the Hessian the certification and the standard errors use; it tries the direction of negative curvature at a saddle and checks what the directions with no curvature are worth. Under intoverpop = 'laplace' it runs only when the fit is certified. May name the curvature its steps use: 'chord' (the default except as below), 'exact' (a fresh Hessian whenever a step contracts slowly), 'subset', or 'krylov': the outer product of the units' scores corrected by Hessian-vector products taken at each step, with the exact Hessian formed where its steps converge. 'krylov' is the default under intoverpop = 'laplace' where the exact Hessian would cost at least 200 gradients. fit$optim$newton_steps records the steps, and newton_krylov_products the gradients the Krylov steps took.

certify

Default TRUE. The curvature check that certifies a fit has converged. estonly skips it too.

gaptol

Default 5e-5. How much objective the curvature may still predict at the estimate before the fit is continued. The gap is half the squared distance from the maximum in standard errors, so the default is 0.01 standard errors: close enough that the estimate's Hessian is the maximum's. A smaller value, such as 1e-6, holds the estimate closer still.

hessianreuse

Default 0.3. How far, in standard errors, the estimate may move from where a Hessian was formed for that Hessian still to give the standard errors. A Wald standard error assumes the curvature is constant over about a standard error around the estimate, and is in error by however much it is not; a Hessian formed this far away adds about this fraction to that error, where forming a fresh one costs a gradient per parameter (two on the Laplace route). 0 forms one at the estimate.

innergaptol

What a prediction of the remaining gain that can be optimistic – L-BFGS's, or a Newton step's on a Hessian taken at another point – has to fall below to stop; by default a hundredth of gaptol. A gain read off the exact Hessian at the estimate is held to gaptol itself.

overshoot

'magnitude' (the default), 'saturation' or 'off': which directions are pulled back after a fit, to test whether it stopped at a maximum.

stallwindow, stallratio, stallcooldown

The stall check (defaults 80, 0.001, 30): over how many iterations a fit's gain is read (half a nat or less starts a look for a transform gone flat), how much of its responsiveness a transform must have lost to count as the reason the fit stopped, and how long the check waits after finding a fit slow rather than stuck.

escapesaturated

Default FALSE. Refit a fit stalled on a saturated transform from a zeroed boundary, keeping whichever wins.

escapepin

Default FALSE. Hold the escaped coordinates still while the rest re-optimises around them, then release them.

The Laplace route (julia, intoverpop = 'laplace')

laplace_inner_maxiter, laplace_inner_tol

Defaults 200 and 1e-10. The inner solve that finds each unit's random-effect mode. Raise the first when the fit reports modes that did not converge; the tolerance bounds the inner gradient, floored at 1e-10 * (1 + abs(value)).

laplace_onestep

FALSE (the default) or TRUE. Where the filter can integrate a person's intercept-type effects (initial means, intercepts, a predictor's effect) exactly, given their other random effects, it does, and Laplace approximates only the integral over the others: the two-step term. TRUE takes one Laplace approximation over all of them together instead, as before version 4.0's change of default. Beside random dynamics (a random drift or diffusion) the two-step term is the more accurate; the one-step term is what a model whose filter cannot integrate them (one whose dynamics read the state) takes in any case, and ctModelImprove() and ctScoreTest() use it to compare such a model with one that can.

laplace_soft_directions

'quadrature' (the default) or 'logdet_floor': how a unit is scored whose likelihood is convex in some direction of its random effects, where plain Laplace would credit that direction more width than the integrand has. 'quadrature' integrates the softest direction by a three-point quadrature where the unit's curvature has an eigenvalue below 0.7, blending into plain Laplace between 0.2 and 0.7; 'logdet_floor' floors the log determinant at zero. Recorded in fit$laplace$soft_directions, and used by every function that rebuilds the objective; fit$laplace$conditioning counts such units. It applies too when sampling with intoverpop = FALSE and sampleControl$placement = 'fit'.

Uncertainty

uncertainty

Both. Default 'hessian' ('sandwich' with subject weights on julia). The method computed when the optimiser ends, as in ctFitUncertainty. The default is for building a model, with the checks below beside it; for intervals to report, sample with ctFitUncertainty(fit, 'sample'). 'sample' is refused here: use optimize = FALSE, or ctFitUncertainty(fit, 'sample') on the fit.

checks

Julia. Default TRUE: the two checks a fit makes last, of the approximations its numbers rest on, as ctFitUncertainty's checks makes them; FALSE for neither, or one of 'randomeffects' and 'intervals'. The random-effects check, on intoverpop = 'laplace': every unit, whatever its number of effects, scored at draws of its random effects, and a warning where integrating them exactly would move an estimate by 0.1 standard errors or more, giving the move's size below one standard error (past that the check's linear step is no guide to it) and how far Laplace's log likelihood is from the integrated one, and naming ctFitUncertainty(fit, intoverpop = 'quadrature') and joint sampling. Wherever it runs, fit$estimate$loglik is Laplace's plus the check's integral of every unit's error (loglik_se its standard error, loglik_laplace the value the optimiser maximised, loglik_method 'checked'). Every model whose effects enter linearly passes at once, and a fit whose optimiser stopped where a unit's Laplace term jumps warns instead, naming the units; fit$laplace$correction records it. The interval check, with uncertainty = 'hessian': 'checked''s check of the curvature's normal against the posterior's gradients, on the density the intervals come from, said in a message and in summary() where the posterior departs from that normal. Under intoverpop = 'quadrature' that density is the refined fit's, where the check can cost more than the refinement (see ctFitUncertainty's checks); checks = 'randomeffects' leaves it out. Neither runs with estonly.

uncertaintyDraws, finishsamples, uncertaintyControl

Both. ctOptimUncertainty's draws (default 'auto'), and ctFitUncertainty's finishsamples (1000) and control.

accuracy, sampleControl

Julia. An accuracy in posterior standard deviations, off by default: once uncertainty's method has run, its own account of its error is read against it, and where the method falls short, or has none, the fit is sampled to that accuracy (ctFitUncertainty's control$accuracy); sampleControl holds the sampler's other settings for that run.

estonly

Both. Default FALSE. Point estimates only: no Hessian, so no standard errors and no certification; under intoverpop = 'laplace' no Newton finish and no quadrature correction either.

Progress and output (julia)

callback

A function called while the fit runs, with (iteration, total, objective, gradient_norm, parameters), where parameters is the raw vector the objective and gradient describe; one declaring only the first four is called with four. It is called on a time cadence and once more at the end, for live progress in a front end; checkpoint below writes a resumable checkpoint itself. An error inside it disables it with a warning. fit$optim$trace holds every iteration after the fit, and ctTracePlot draws it.

checkpoint

A file path. While the fit optimises, its point is written there every checkpoint_every seconds (default 60) and at the end of each run, with each subject's random-effect modes on the Laplace route and the smoothing parameters of any smooth terms; each exact Hessian the fit forms is kept beside it (<checkpoint>.hessian). Where the file already exists, the fit starts from it: the same call again resumes an interrupted fit, its modes, smoothing parameters and Hessian included, which starting from inits alone does not keep. A file written for a different model stops the fit rather than being overwritten.

progress

Force the progress line on or off.

tipredMissing

How missing TI predictor values are handled. Each gets a normal distribution from a regression on the other variables; 'impute' holds the value at its mean, 'sample' makes it a parameter with that distribution as its prior, and 'auto', the default, imputes when optimizing and samples when sampling. A sampled fit is placed with the values imputed. ctFitUncertainty(fit, 'sample') reads the fit's setting, and its control$tipredMissing overrides it.

tipredMissingIncludeOutcome

Default TRUE. Whether the outcome informs the imputation of missing TI predictors.

saveEffects

A sampling setting; read here for older scripts. Use sampleControl$saveEffects.

Stan only

nsubsets, subsamplesize, lproughnesstarget, stochasticTolAdjust, parsteps

Tuning of stan's stochastic-gradient optimiser and its stepwise phase.

stalltol

Default 0.01. The gradient per data point above which stan calls a fit stalled and restarts it (see stallretries).

See Also

ctFit, ctFitUncertainty, ctTracePlot.


Write a parameter specification

Description

Builds the |-separated cell specification ctsem's model matrices use, from named arguments rather than positional fields.

Usage

ctPar(
  name,
  transform = NA,
  indvarying = NA,
  sdscale = NA,
  tipreds = NULL,
  mean = NA
)

Arguments

name

Parameter name. Two cells given the same name are the same parameter, which is how equality constraints are written.

transform

Transform expression in terms of param, e.g. 'exp(param)'. Omitted leaves the default for that matrix, which depends on the cell – a variance, a drift diagonal and a mean all differ.

indvarying

Where the parameter varies: TRUE at every level of the model's id hierarchy, FALSE at none, or the names of the levels it varies at. With id = c('person', 'couple') in ctModel, indvarying = 'couple' gives a parameter that differs between couples but not between the persons within one, and TRUE is the same as c('person', 'couple'). With a single id the two forms mean the same thing. The model's pars$indvarying column holds the same values and may be edited directly.

sdscale

Multiplier on the population standard deviation's prior, at every level where the parameter varies. Only meaningful when it varies.

tipreds

Time-independent predictors acting on this parameter: a character vector of names for effects to estimate, e.g. 'age' or c('age', 'sex'). Naming any predictor switches the rest off for this cell, so the list is the whole set of effects on it. A name must be one of the model's TIpredNames; anything else is an error when the model is built. character(0) (or tipreds= with nothing after it, in a cell) means no predictor acts on this cell, which is not the same as leaving the argument out – that keeps the model's default.

mean

A population mean stated rather than estimated, for a parameter that varies (indvarying): a number on the scale summary() reports population means on. Only the random effects about it are then fitted, as for a person-level intercept with mean zero. Written into the cell as 'name=value|...', and held by the model in fixedpopmeans, a named vector that may also be set directly. Fits with backend = 'julia'.

Details

The output is the same string the | syntax produces, so this is a way of writing one rather than an alternative to it. Fields left at their default are emitted empty, which is what the parser reads as "unspecified".

The same named fields can also be written directly into a model matrix cell, in which case nothing needs to be called. A cell containing | is read as a |-separated specification; otherwise a cell with a field of the form name=value is read as a named specification, and anything else as a plain parameter name. All three of these specify the same parameter:

  MANIFESTMEANS = matrix('mm||TRUE|0.5')
  MANIFESTMEANS = matrix(ctPar('mm', indvarying = TRUE, sdscale = 0.5))
  MANIFESTMEANS = matrix('mm, indvarying=TRUE, sdscale=0.5')
  

In the named form, fields are separated by commas (commas inside brackets belong to the value, so transform=pnorm(param, 0, 1) is one field), the parameter name comes first, and several predictors or levels are given as tipreds=c(age, sex) or indvarying=c(person, couple); in the | form they are separated by commas, as in 'cross||person,couple'. A field name that is not one of the above is an error naming the nearest valid field, so a misspelling such as topreds=age is refused rather than becoming part of a parameter name. Comparisons are unaffected: a state-dependent cell may contain ==, >=, <= or !=, and a named argument inside a call belongs to the value it appears in.

Value

A single character string, suitable for a model matrix cell.

See Also

ctModel for the matrices these cells go in.

Examples

ctPar('mm', indvarying = TRUE)
ctPar('drift11', transform = '-exp(param)')
ctPar('mm', indvarying = TRUE, tipreds = c('age', 'sex'))
# varying between couples only, in a model with id = c('person', 'couple')
ctPar('cross', indvarying = 'couple')
# a person-level intercept whose population mean is zero
ctPar('lev', indvarying = TRUE, mean = 0)

## Not run: 
# These two models are identical.
model <- ctModel(type = 'ct', n.latent = 1, n.manifest = 1,
  LAMBDA = matrix(1),
  MANIFESTMEANS = matrix(ctPar('mm', indvarying = TRUE, sdscale = 0.5)))

model <- ctModel(type = 'ct', n.latent = 1, n.manifest = 1,
  LAMBDA = matrix(1),
  MANIFESTMEANS = matrix('mm, indvarying=TRUE, sdscale=0.5'))

## End(Not run)

Check a julia-backend fit against a particle-filter likelihood

Description

The filter ctsem fits with is an assumed-density approximation: it carries a Gaussian for the latent states, linearises the drift over each prediction step, and projects non-Gaussian measurement updates back onto a Gaussian. For a linear model with Gaussian indicators it is exact; for anything else the size of its error is invisible from inside it. This function estimates the same marginal log likelihood at the fit's estimates with a bootstrap particle filter that makes none of those approximations, and reports the difference, in total and per row.

Usage

ctParticleLik(
  fit,
  particles = 2000,
  substeps = 20,
  transition = c("exponential", "euler"),
  seed = 1,
  replicates = 1,
  cores = getOption("mc.cores", 2L)
)

Arguments

fit

A fit from ctFit(..., backend = 'julia') with the default intoverpop; random effects carried as augmented states are integrated by the particles. Laplace-route fits are refused.

particles

Particles per subject.

substeps

Transition steps per observation interval. This is the reference mesh, independent of anything the fit used, and floored by the fit's own maxtimestep or automatic mesh.

transition

'exponential' applies the filter's own discretisation at each particle's state: exact for the linear parts of the model, second order otherwise, so the comparison isolates the filter's Gaussian assumption. 'euler' is plain Euler-Maruyama, which shares no approximation with the filter at all but needs several times more substeps.

seed

Fixes every draw; the result is reproducible given it.

replicates

Independent runs (with seeds seed, seed + 1, ...). One run reports an approximate standard error from the effective sample sizes; more than one reports the standard deviation across runs.

cores

Engine threads. The particle filter splits the subjects across them, each subject with its own random stream, so the result does not depend on the count.

Details

Each particle steps through the process with drift, diffusion and every state-dependent parameter evaluated at its own state, is weighted by the conditional density of the row's observations given that state (Gaussian, binary, ordinal, count and censored indicators are all handled), and is resampled when the weights degenerate. The estimate is unbiased with Monte Carlo error falling as one over the square root of particles.

It is a diagnostic at fixed parameters, not an estimator. Two things decide whether to trust the comparison: se should be well below the difference you care about (raise particles, or set replicates for an empirical standard error), and doubling substeps should not move the answer.

Value

A list: loglik, the particle estimate (mean over replicates); se; fit_loglik, the filter's log likelihood from the fit; difference, particle minus filter; rows, a data frame with the subject, time, particle and filter log likelihood of every row and their difference; replicates, the individual estimates; ess_min, the smallest effective sample size any row saw; and the settings.

See Also

ctFitUncertainty with method = 'is' and filter = 'particle' applies the correction this measures, to the fit's posterior draws.

Examples

# Needs a working julia backend, so the example is inert where Julia is
# absent -- including on CRAN, whose check machines have none.
if (isTRUE(ctJuliaStatus()$available)) {
  model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1',
    LAMBDA = matrix(1))
  model$pars$indvarying <- FALSE
  fit <- ctFit(ctstantestdat, model, backend = 'julia', cores = 1)
  # A linear Gaussian model, where the filter is exact: the difference
  # should be within a few standard errors of zero.
  check <- ctParticleLik(fit, particles = 1000, substeps = 5)
  check[c('loglik', 'se', 'fit_loglik', 'difference')]
  # Julia keeps running between calls, holding what it has compiled;
  # this ends it.
  ctJuliaKill("session")
}

The phase predictors a cycle added by ctModelPeriodicComponent reads

Description

Adds to long format data, for each subject, the cosine and sine of the time of its first row over period: the columns <name>_cos0 and <name>_sin0 that ctModelPeriodicComponent sets its cycle from.

Usage

ctPeriodicPhases(datalong, period, name = "cycle", id = "id", time = "time")

Arguments

datalong

Long format data, one row per observation.

period

The cycle's period, in the data's time units.

name

The cycle's name, as given to ctModelPeriodicComponent.

id, time

The subject and time columns.

Value

datalong with the two columns added.

Examples

d <- data.frame(id = rep(1:2, each = 3), time = c(8, 12, 20, 33, 37, 45))
ctPeriodicPhases(d, period = 24)

Phase portrait of a fitted continuous time model

Description

Draw the model's vector field over a plane of the latent state space: at each point, the direction and speed the deterministic part of the system moves. For a model whose matrices depend on the latent state – the reason the julia backend exists – this shows behaviour that no single reported DRIFT matrix can, because there is no single DRIFT matrix.

Usage

ctPhasePortrait(
  fit,
  latents = 1:2,
  extent = "data",
  gridsize = 15,
  state = "asymptotic",
  tipreds = NULL,
  nullclines = TRUE,
  fixedpoint = TRUE,
  trajectories = 0,
  plot = TRUE
)

Arguments

fit

A ctJuliaFit, or a ctStanFit of a linear model.

latents

Two latent processes, by name or index, forming the plane.

extent

'data' (the default) spans the smoothed latent states, 'sd' spans two stationary standard deviations either side of T0MEANS, or supply a list of two numeric ranges.

gridsize

Arrows per side. The field costs one engine call per point, so cost is quadratic in this.

state

Where the *other* latent processes are held, as in ctDiscretePars: 'asymptotic', 'mean', 'T0MEANS', or a numeric state.

tipreds

Time independent predictor values, or NULL for the population.

nullclines

Draw the curves where each process stops changing?

fixedpoint

Mark the asymptotic state when it is inside the region?

trajectories

Number of subjects' smoothed trajectories to overlay, or 0 for none.

plot

If FALSE, return the field, nullclines and trajectories as data rather than a plot.

Details

Three things are drawn over the field, each of which can be turned off:

  • Nullclines, the curves where one process stops changing. Where the nullclines of two processes cross, the whole system stops: that is a fixed point, and a nonlinear system may have several.

  • The fixed point found by state='asymptotic', marked when it lies inside the plotted region. Only one is found, by Newton from T0MEANS; the nullcline crossings show whether there are others.

  • Smoothed trajectories for a few subjects, so the field can be read against where the data actually went.

With more than two latent processes the remaining ones are held at state, so the picture is a slice rather than the whole story – change state and the slice changes.

A linear model has an affine field, straight nullclines and exactly one fixed point. It is drawn correctly, from a stan fit as readily as a julia one, but it shows nothing the DRIFT matrix did not already say. A nonlinear stan fit is refused rather than drawn as if it were linear.

Value

A ggplot, or when plot=FALSE a list with field, the latents on the plane, stateLabel naming where the off-plane processes were held, the held process names, and nullclines, fixedpoint and trajectories when asked for. The plot names stateLabel in its subtitle whenever anything is held off-plane.

See Also

ctContextDependence, ctDiscretePars

Examples

# A linear stan fit needs no engine, so the bundled example fit is enough.
ctPhasePortrait(ctstantestfit, gridsize = 7, plot = FALSE)

Plots three dimensional y values for quantile plots

Description

1st margin of $Y sets line values, 2nd sets variables, 3rd quantiles.

Usage

ctPlotArray(
  input,
  grid = FALSE,
  add = FALSE,
  colvec = "auto",
  lwdvec = "auto",
  ltyvec = "auto",
  typevec = "auto",
  plotcontrol = list(ylab = "Array values", xaxs = "i"),
  legend = TRUE,
  legendcontrol = list(),
  polygon = TRUE,
  polygonalpha = 0.1,
  polygoncontrol = list(steps = 25)
)

Arguments

input

list containing 3 dimensional array to use for Y values, $y and vector of corresponding x values $x.

grid

Logical. Plot with a grid?

add

Logical. If TRUE, plotting is overlayed on current plot, without creating new plot.

colvec

color vector of same length as 2nd margin.

lwdvec

lwd vector of same length as 2nd margin.

ltyvec

lty vector of same length as 2nd margin.

typevec

type vector of same length as 2nd margin.

plotcontrol

list of arguments to pass to plot.

legend

Logical. Draw a legend?

legendcontrol

list of arguments to pass to legend.

polygon

Logical. Draw the uncertainty polygon?

polygonalpha

Numeric, multiplier for alpha (transparency) of the uncertainty polygon.

polygoncontrol

list of arguments to pass to ctPoly

Value

Nothing. Generates plots.

Examples

#'
input<-ctTIpredEffects(ctstantestfit, plot=FALSE, whichpars='CINT',
 nsamples=10,nsubjects=10)
    
ctPlotArray(input=input)

Prior and posterior distributions of a fit's parameters

Description

Plots, for each free population parameter of a fit, its prior and its posterior on the natural scale: the population mean, and for an individually varying parameter the population standard deviation, at each level it varies at. Works for fits from either backend. The prior is ctsem's normal(0,1) raw prior carried through each parameter's transform – the prior priors = TRUE fits with – so on a fit made without it the plot shows how far the data moved each parameter from where that prior would put it. The posterior is the fit's draws: genuine posterior draws for a sampled fit, the approximate ones ctFitUncertainty writes for an optimized one. ctStanPlotPost is maintained as a backward-compatible alias.

Usage

ctPlotPosterior(
  obj,
  rows = "all",
  npp = 4,
  priorwidth = TRUE,
  smoothness = 1,
  priorsamples = 10000,
  plot = TRUE,
  wait = FALSE,
  ...
)

Arguments

obj

A fit from ctFit, on either backend. A model on its own has no posterior; plot(model) plots its priors.

rows

Character string 'all', or which parameters to plot, by position in the fit's free population parameters, in parameter order.

npp

Integer number of parameters to show per page.

priorwidth

if TRUE, plots will be scaled to show bulk of both the prior and posterior distributions. If FALSE, scale is based only on the posterior.

smoothness

Positive numeric – multiplier to modify smoothness of density plots, higher is smoother but can cause plots to exceed natural boundaries, such as standard deviations below zero.

priorsamples

number of samples from prior to use. More is slower.

plot

Logical, if FALSE, ggplot objects are returned in a list instead of plotting.

wait

If true, user is prompted to continue before plotting next graph. If false, graphs are plotted one after another without waiting.

...

For a stan fit, passed to the preparation of the prior draws: cores = x will speed things up, where x is the number of cpu cores to use. A julia fit takes nothing here.

Value

With plot = TRUE, NULL invisibly, having drawn the plots; with plot = FALSE, a list of ggplot objects.

Examples

ctPlotPosterior(ctstantestfit, rows=3:4)

Plots uncertainty bands with shading

Description

Plots uncertainty bands with shading

Usage

ctPoly(x, y, ylow, yhigh, steps = 20, ...)

Arguments

x

x values

y

y values

ylow

lower limits of y

yhigh

upper limits of y

steps

number of polygons to overlay - higher integers lead to smoother changes in transparency between y and yhigh / ylow.

...

arguments to pass to polygon()

Value

Nothing. Adds a polygon to existing plot.

Examples

plot(0:100,sqrt(0:100),type='l')
ctPoly(x=0:100, y=sqrt(0:100), 
yhigh=sqrt(0:100) - runif(101), 
ylow=sqrt(0:100) + runif(101),
col=adjustcolor('red',alpha.f=.1))

Create a data.table to compare data generated from a ctsem fit with the original data.

Description

This function allows for easy comparison of data generated from a fitted ctsem model with the original data used to fit the model. It provides options to include residuals in the comparison.

Usage

ctPostPredData(fit, residuals = FALSE, nsamples = NA)

Arguments

fit

A fitted ctsem model, from either backend.

residuals

If set to TRUE, includes standardised prior residuals in the comparison, as extra rows whose variable names are suffixed ' std. res.'. Needs one Kalman filter pass per generated dataset, so it is much slower than the default.

nsamples

Integer or NA. Number of generated datasets to use. If the fit carries more than this, draws are subsampled; if it carries none, this many are generated. NA uses whatever the fit already has, or the ctGenerateFromFit default.

Value

A long data table with one row per generated draw x data row x variable, carrying the generated value, the corresponding obsValue, and the id, Time and TimeInterval of that data row. LogLik appears as an extra 'variable', holding the row-wise log likelihood.

See Also

ctPostPredPlots, which turns this into diagnostic plots.

Examples

data_comparison <- ctPostPredData(ctstantestfit)

Posterior predictive checks for a ctsem fit

Description

ctPostPredict and ctStanPostPredict are aliases for ctPostPredPlots, which the two functions were merged into. They differ from it only in printing the panels by default rather than returning them.

Usage

ctPostPredict(fit, ...)

Arguments

fit

A fitted ctsem model.

...

Passed to ctPostPredPlots.

Value

A named list of ggplot objects, invisibly.

Examples

ctPostPredict(ctstantestfit, wait = FALSE)

Posterior predictive checks for a ctsem fit

Description

Compares data generated from the fitted model against the data it was fitted to, as a set of diagnostic plots. Works for both the stan and julia backends. ctPostPredict and ctStanPostPredict are maintained as aliases.

Usage

ctPostPredPlots(
  fit,
  panels = "all",
  variables = NULL,
  nsamples = NA,
  datarows = "all",
  diffsize = 1,
  interval = 0.95,
  resolution = 100,
  jitter = 0,
  residuals = FALSE,
  timebins = 10,
  lags = 0:5,
  lagcor = TRUE,
  notes = TRUE,
  plot = FALSE,
  wait = FALSE
)

Arguments

fit

A fitted ctsem model.

panels

Which panels to produce. 'all' (the default), 'structure', 'calibration', or a character vector of panel names – see Details.

variables

Character vector of variable names to restrict to, or NULL for all. 'LogLik' is a valid name here.

nsamples

Integer or NA. Number of generated datasets to compare against. NA uses whatever the fit already carries.

datarows

Integer vector of data rows to use, or 'all'.

diffsize

Integer vector > 0. Lags, in observations, for the rate-of-change panels. One pair of panels is produced per lag.

interval

Numeric in (0,1). Width of the central predictive interval used by the PredictedVsObserved and IntervalCoverage panels.

resolution

Positive integer. Grid resolution of the shaded model density in the bivariate panels.

jitter

Non-negative numeric. Jitter added to the values in the rate-of-change panels, as a proportion of each variable's standard deviation, applied to observed and generated alike. Useful when data are rounded or ordinal. 0, the default, adds none.

residuals

Logical. Include standardised prior residuals as extra variables. Slow – it needs one Kalman pass per generated dataset.

timebins

Positive integer. Number of equal-count bins of time for the MeanTrajectory panel. Subjects are rarely observed at identical times, so the mean at a time is a mean within a bin of times.

lags

Non-negative integer vector. Observation lags for the LaggedCovariance panel, passed to ctFitCheckCov.

lagcor

Logical. If TRUE (the default) the LaggedCovariance panel shows correlations, which are usually easier to read; FALSE shows covariances.

notes

Logical. If TRUE (the default), each panel carries a caption saying what it shows and what a departure from its reference line means. Set FALSE for bare plots.

plot

Logical. If TRUE, prints the panels and returns them invisibly. If FALSE (the default) returns the list without printing.

wait

Logical. If TRUE and plot=TRUE, waits for a keypress between panels.

Details

Two families of panel are produced.

structure asks whether the model reproduces the shape of the data: Density (marginal distributions), MeanTrajectory (the mean over subjects against time), ValueByTime and ValueByOccasion (the model-implied density of each variable against time and against occasion number, with the observed data as points on top), ChangeByValue and ChangeByTime (rate of change against current value and against time), and LaggedCovariance (the lagged covariance or correlation structure). ChangeByValue is the phase plane, and is the panel that shows nonlinear dynamics: a linear model implies a straight band with a constant slope, so curvature in the observed points relative to the model shading is evidence the drift is not linear.

MeanTrajectory and ValueByTime both plot against time and answer different questions. ValueByTime shows where the model puts an individual observation, so its envelope is as wide as the data; MeanTrajectory shows where it puts the mean of a bin of observations, whose sampling distribution is narrower by roughly the square root of the bin count. A bias in the average level is therefore visible in the second while sitting comfortably inside the first.

LaggedCovariance is ctFitCheckCov rendered as a dashboard panel: one plot per variable, faceted by the variable it is paired with, empirical against the central 95% of the generated datasets at each lag. It is the only panel that looks at dependence across time rather than at each time, which is what makes it the one that responds to a wrong drift or diffusion when every marginal already matches. Call ctFitCheckCov directly to split it by a group or a predictor.

calibration asks whether the predictive distribution is the right width and in the right place: PredictedVsObserved, IntervalCoverage, PIT, and the PIT resolved by CalibrationByInterval and CalibrationBySubject. A departure that depends on the time interval implicates the dynamics rather than the measurement model, since the measurement model cannot know the interval.

The row-wise log likelihood appears throughout as an extra variable, LogLik – its distribution should match between model and data like any other quantity. It is omitted from the rate-of-change panels, where the derivative of a log likelihood with respect to time is not a meaningful quantity. Note that the observed side of the LogLik comparison is evaluated at the point estimate, so it carries no parameter uncertainty while the generated side does.

SubjectLogLik asks the same question one level up: each subject's total log likelihood, summed over that subject's rows, compared as a cumulative distribution across subjects against the same curve from each generated dataset. Aggregating within a subject first is what makes a subject the model fits badly across many occasions show up, rather than being spread thin across the row-wise panels. The comparison is against replicate datasets of the same size rather than a smooth density, so it is the right reference however few subjects there are.

The approximation is only as good as the number of generated datasets, so a fit carrying few draws gives a coarse picture – the PIT in particular is discrete on nsamples + 1 values.

Value

A named list of ggplot objects, invisibly if plot=TRUE.

See Also

ctPostPredData for the underlying table, ctFitCheckCov for the lagged covariance panel on its own, with splits by group or predictor, and ctPhasePortrait for the model-implied phase plane on its own.

Examples

plots <- ctPostPredPlots(ctstantestfit)
print(plots$PIT)

ctPredict

Description

Outputs predicted, updated, and smoothed estimates of manifest indicators and latent states, with covariances, for specific subjects from data fit with ctFit, based on either the mode (if optimized) or mean (if sampled) of parameter distribution. ctKalman is an alias for this function and returns the same long data frame. The pre-3.11 name ctStanKalman is not its Stan-prefixed twin: that is an alias for ctKalmanArray, which returns a list of arrays instead.

Usage

ctPredict(
  fit,
  timerange = "asdata",
  timestep = "auto",
  subjects = NULL,
  removeObs = FALSE,
  plot = FALSE,
  standardisederrors = FALSE,
  realid = TRUE,
  randomEffects = NULL,
  newdata = NULL,
  ...
)

Arguments

fit

fit object as generated by ctFit.

timerange

Either 'asdata' to just use the observed data range, or a numeric vector of length 2 denoting start and end of time range, allowing for estimates outside the range of observed data. Ranges smaller than the observed data are ignored.

timestep

Either 'asdata' to just use the observed data (which also requires 'asdata' for timerange) or a positive numeric value indicating the time step to use for interpolating values. Lower values give a more accurate / smooth representation, but take a little more time to calculate.

subjects

subject ids to predict for, or NULL (the default) for the first subject in the data, or every subject in newdata when that is given. With realid=FALSE, integers denoting subjects from 1 to N.

removeObs

Logical or integer. If TRUE, observations (but not covariates) are withheld from the filter, so only expectations based on parameters and covariates are returned (the observations themselves are still reported, for comparison). If a positive integer N, every N observations are retained while others are withheld – useful for observing prediction performance forward further in time than one observation.

plot

Logical. If TRUE, plots output instead of returning it. See plot.ctKalmanDF (Stan based fit) for the possible arguments.

standardisederrors

if TRUE, also include standardised error output (based on covariance per time point).

realid

If TRUE (the default), subjects are read as the ids in the data, and integers that are not ids there are read as ctsem's numbering of the subjects, 1 to N, with a message. If FALSE, they are read as that numbering.

randomEffects

For an intoverpop='laplace' fit, which levels of random effect the trajectories are built from. Naming one of the model's id columns includes that level and every level outside it, so the innermost id gives each subject its own trajectory and an outer id gives the group mean, shared by every subject in the group. 'population' includes none, giving the trajectory implied by the population parameters alone. Defaults to the innermost id. Ignored for other fits.

With one level of random effects, left in, each subject's predicted and updated values take its random effects at their posterior given its rows up to each time, and its smoothed values at their mode given its whole record. Otherwise every pass takes the modes from each subject's whole record. Those whole-record modes stay the fit's however this call changes the rows being filtered (selecting subjects, a time grid, removeObs), so with every observation withheld the smoothed values still use that subject's own random effects.

newdata

Optional long format data to predict for instead of the data the model was fitted to, with the same columns: an id, a time, the manifest variables (NA where nothing is observed) and the predictors. New subjects, such as hypothetical people with chosen covariate values; a subject of the fit under other covariate values; or a record cut short at some time, to see what was predictable then. The data are put under the model the fit ran, with its estimates as they stand: nothing is refitted, and the model keeps its structure whatever the number of subjects. A subject's predicted (prior) and updated values condition on its rows in newdata up to each time, and its smoothed values on all of them. For a fit with random effects that includes the subject's random effects, at their posterior given those rows (for a Laplace fit with one level of random effects; otherwise as randomEffects describes), so a subject with no observations gets the population's prediction for its covariate values. A row at a time with nothing observed gives a prediction there; a time dependent predictor left NA is taken as zero, as ctFit takes it.

...

additional arguments to pass to plot.ctKalmanDF.

Value

Returns a list containing matrix objects etaprior, etaupd, etasmooth, y, yprior, yupd, ysmooth, errprior, errupd, errsmooth, time, loglik, with values for each time point in each row. eta refers to latent states and y to manifest indicators - y itself is thus just the input data. With standardisederrors=TRUE, standardised versions errstdprior, errstdupd, errstdsmooth are included too. Covariance matrices etapriorcov, etaupdcov, etasmoothcov, ypriorcov, yupdcov, ysmoothcov, are returned in a row * column * time array. Some outputs are unavailable for ctStan fits at present. If plot=TRUE, nothing is returned but a plot is generated.

Examples

#Basic
ctPredict(ctstantestfit, timerange=c(0,60), plot=TRUE)

#Multiple subjects, y and yprior, showing plot arguments
plot1<-ctPredict(ctstantestfit, timerange=c(0,60), timestep=.1, plot=TRUE,
  subjects=2:3, 
  kalmanvec=c('y','yprior'),
  errorvec=c(NA,'ypriorcov')) #'auto' would also have achieved this
  
 #modify plot as per normal with ggplot
 print(plot1+ggplot2::coord_cartesian(xlim=c(0,10)))
 
 #or generate custom plot from scratch:#'  
 k=ctPredict(ctstantestfit, timerange=c(0,60), timestep=.1, subjects=2:3)
 library(ggplot2)
 ggplot(k[k$Element %in% 'yprior',],
   aes(x=Time, y=value,colour=Subject,linetype=Row)) +
   geom_line() +
   theme_bw()

#New subjects: two hypothetical people differing only in TI1, each
#observed once at time 0, predicted to time 4.
newsubjects <- data.frame(id = c('lowTI1', 'highTI1'), time = 0,
  Y1 = 8, Y2 = 12, TD1 = 0, TI1 = c(-1, 1))
newsubjects <- rbind(newsubjects,
  transform(newsubjects, time = 4, Y1 = NA, Y2 = NA))
k <- ctPredict(ctstantestfit, newdata = newsubjects, timerange = c(0, 4),
  timestep = .1)
k[k$Element %in% 'yprior' & k$Time == 4, ]
ctPredict(ctstantestfit, newdata = newsubjects, timerange = c(0, 4),
  timestep = .1, plot = TRUE, kalmanvec = 'yprior', errorvec = 'ypriorcov')

ctPredictTIP

Description

Outputs the estimated effect of time independent predictors (covariate moderators) on the expected observations.

Usage

ctPredictTIP(
  sf,
  tipreds = "all",
  subject = NULL,
  timestep = "auto",
  doDynamics = TRUE,
  plot = TRUE,
  quantiles = c(0.16, 0.5, 0.84),
  discreteTimeQuantiles = c(0.025, 0.5, 0.975),
  dynamicsControl = list(),
  showUncertainty = TRUE,
  TIPvalues = NA,
  realid = TRUE
)

Arguments

sf

A fitted object from ctFit, from any backend.

tipreds

A character vector specifying which time independent predictors to use. Default is 'all', which uses all time independent predictors in the model.

subject

The subject for which predictions are made, as an id in the data (see realid), or NULL (the default) for the first subject. This is relevant only when time dependent predictors are also included in the model.

timestep

A numeric value specifying the time step for predictions. Default is 'auto', which tries to automatically determine an appropriate time step.

doDynamics

A logical value indicating whether to plot the effects of time independent predictors on the dynamics of the system. Default is TRUE. Can be problematic for systems with many dimensions.

plot

A logical value indicating whether to ggplot the results instead of returning a data.frame of predictions. Default is TRUE.

quantiles

A numeric vector specifying the quantiles of the time independent predictors to plot. Default is c(.16,.5,.84): the median, and one standard deviation either side for a normally distributed predictor.

discreteTimeQuantiles

a numeric vector of length 3 specifying the quantiles of the discrete time points to plot, when showUncertainty is TRUE.

dynamicsControl

A named list of additional arguments to pass to ctDiscretePars and ctDiscreteParsPlot when doDynamics=TRUE. Arguments matching ctDiscreteParsPlot are automatically routed to the plot call. Internally controlled arguments fit, ctstanfitobj, plot, subjects, realid, impulseType (and its deprecated spelling observational), x, quantiles, and splitSubjects are ignored if supplied.

showUncertainty

A logical value indicating whether to plot the uncertainty of the predictions. Default is TRUE.

TIPvalues

An nvalue * nTIpred numeric matrix specifying the fixed values for each time independent predictor effect to plot. Default is NA, which instead relies on the quantiles specified in the quantiles argument.

realid

If TRUE (the default), subject is read as an id in the data, and an integer that is not an id there as ctsem's numbering of the subjects, 1 to N, with a message. If FALSE, it is read as that numbering.

Details

This function estimates the effects of covariate moderators on the expected process and observations for a specified subject in a dynamic system. The covariate moderators are defined at the specified quantiles, and their effects on the trajectory are plotted or returned as a data frame.

Value

If plot is TRUE, a list of ggplot objects showing the estimated effects of covariate moderators. Otherwise, a data frame with the predictions.

Examples

# Example usage:
ctPredictTIP(ctstantestfit, tipreds='all', doDynamics=FALSE, plot=TRUE)

ctRawParnames

Description

Gets internal stan parameter names of a ctStanFit object sampled via stan based on specified substrings. ctStanParnames is maintained as a backward-compatible alias.

Usage

ctRawParnames(x, substrings = c("pop_", "popsd"))

Arguments

x

ctStanFit object, sampled (ctFit(backend = 'stan', optimize = FALSE)).

substrings

vector of character strings, parameter names of the stan model containing any of these strings will be returned. Useful strings may be 'pop_' for population means, 'popsd' for population standard deviations, or specific combinations such as 'pop_DRIFT' for the population means of temporal dynamics parameters

Value

vector of character strings.

Examples

sunspots<-sunspot.year
sunspots<-sunspots[50: (length(sunspots) - (1988-1924))]
id <- 1
time <- 1749:1924
datalong <- cbind(id, time, sunspots)

#setup model
ssmodel <- ctModel(type='ct', n.latent=2, n.manifest=1,
 manifestNames='sunspots',
 latentNames=c('ss_level', 'ss_velocity'),
 LAMBDA=matrix(c( 1, 'ma1| log(1+(exp(param)))' ), nrow=1, ncol=2),
 DRIFT=matrix(c(0, 'a21 | -log(1+exp(param))', 1, 'a22'), nrow=2, ncol=2),
 MANIFESTMEANS=matrix(c('m1|param * 10 + 44'), nrow=1, ncol=1),
 MANIFESTVAR=diag(0,1), #As per original spec
 CINT=matrix(c(0, 0), nrow=2, ncol=1),
 DIFFUSION=matrix(c(0, 0, 0, "diffusion"), ncol=2, nrow=2))

#fit
ssfit <- ctFit(datalong, ssmodel, optimize=FALSE, priors=TRUE,
  sampleControl=list(iter=2, chains=1), backend='stan')
ctRawParnames(ssfit,substrings=c('pop_','popsd'))

Write a folder of summary, diagnostic and interpretation output for a fit

Description

Runs the package's reporting and fit-checking functions over a fitted model and writes the results into one folder, with a markdown index that says what each file is and how to read it. Works for either backend; components a backend or a model does not support are recorded in the index as not produced, rather than stopping the run.

Usage

ctReport(
  fit,
  folder = "ctReport",
  components = "default",
  nsamples = 200,
  cores = getOption("mc.cores", 2L),
  quantiles = c(0.025, 0.5, 0.975),
  times = "auto",
  timeinterval = 1,
  nsubjects = 6,
  lags = 0:3,
  folds = 10,
  profilepoints = 9,
  verbose = TRUE
)

Arguments

fit

A fit from ctFit (class ctStanFit or ctJuliaFit).

folder

Directory to write into; created if it does not exist. Files from a previous ctReport run in the same folder are removed first, so the folder always matches its index.

components

Either one of 'quick' (nothing that needs generated data or further model evaluations), 'default', 'all' (adds cross validation and LaTeX), or a character vector of component names: summary, parmatrices, identification, profile, discretepars, network, predictions, residuals, adequacy (ctFitAdequacy), scoretests (ctScoreTest, julia fits), covcheck, postpred, tipredeffects, crossval, equations.

nsamples

Draws used by the components that sample from the fit's uncertainty, and by data generation for adequacy, covcheck and postpred.

cores

Cores for the components that can use more than one (discretepars, adequacy, scoretests, covcheck, postpred, crossval).

quantiles

Quantiles for the parameter matrices and the discrete parameter plots.

times

Interval grid for discretepars. 'auto' uses zero to the 90th percentile of the observed per-subject time span.

timeinterval

Time interval at which the networks, the discrete-time parameter matrices and the covariate effects are reported.

nsubjects

Number of subjects plotted by predictions, chosen as those with the most observations.

lags

Lags for the lagged covariance check.

folds

Folds for crossval.

profilepoints

Grid points per parameter in the likelihood profile.

verbose

Report each component as it runs.

Details

The expensive parts are adequacy, covcheck and postpred, which generate data from the fit (once, shared), scoretests, which evaluates the model extended with new terms, and crossval, which refits. components = 'quick' excludes them all.

profile walks the log probability along each raw parameter, holding the others at the estimate, over a grid measured in the standard error that parameter's own curvature implies. It flags parameters whose slice peaks away from the estimate (the optimizer did not arrive), whose surface is not the shape a standard error assumes, and whose reported interval is much wider than their own curvature (they sit on a ridge with other parameters). It is a conditional slice, not a profile likelihood, which is why it is cheap enough to run by default; identification names the ridge directions.

Value

Invisibly, a list with the folder and, per component, whether it was written, skipped or failed. Called for the files it writes.

Checking a fit – which function answers what

Can the data inform every parameter?

ctIdentify before fitting; fit$identifiability and ctFitProfile after.

Did the optimiser, or the sampler, get there?

print(fit) says whether it converged, summary(fit) adds R-hat and effective sample sizes for a sampled fit, and ctTracePlot draws the optimiser's trace or each chain's draws.

How far did the data move each parameter from its prior?

ctPlotPosterior.

Does data generated from the fit look like the data?

ctPostPredPlots, and ctFitCheckCov for the lagged covariances, split by a group or predictor if wanted.

Are the one-step-ahead errors what the model says they are?

ctFitAdequacy tests them by family, allowing for the estimation; ctResiduals and ctACFresiduals show them.

What would relaxing the model change?

ctScoreTest, without refitting; ctModelProposals lists the changes and ctModelImprove refits them.

Does a richer model fit better, or predict better?

anova for nested models, ctLOO for out-of-sample prediction.

How much do the julia engine's approximations cost?

ctLaplaceCheck for the Laplace approximation of random effects, ctParticleLik for the filter's Gaussian approximation of the states.

Do the intervals cover as often as they claim?

ctCoverageCheck, by simulation.

Examples

f <- ctReport(ctstantestfit, folder = file.path(tempdir(), 'ctReport'),
  components = c('summary', 'identification', 'profile'), profilepoints = 5)
list.files(f$folder)

Extract Standardized Residuals from a ctsem Fit

Description

This function takes a fit object from the ctsem package and extracts the standardized residuals.

Usage

ctResiduals(fit)

Arguments

fit

A fitted model object generated by the ctsem package.

Details

This function uses the ctKalmanArray function to calculate the standardized residuals and then extracts and formats them as a data table. The standardized residuals represent the differences between the observed and predicted values, divided by the standard errors of the observations.

Value

A data table containing the standardized residuals for each subject and time point, with subjects labelled by their ids in the data.

See Also

ctKalmanArray

Examples

data.table::setDTthreads(1) #ignore this line
# Example usage:
residuals <- ctResiduals(ctstantestfit)

Score tests for relaxing a fitted model

Description

Tests, from one fitted model and without refitting it, whether the model would fit better with a relaxation it does not contain: parameters that differ between subjects (an unmodelled random effect), parameters that depend on a time independent covariate the model does not include, or dynamics that depend on the latent state (a missing nonlinearity). Each test is a score (Lagrange multiplier) test: the slope of the log likelihood in the direction of the relaxation, at the fitted model, with the fitted parameters projected out. Each also gives a one-step estimate of the relaxation's size, for screening: refit with the relaxations that matter before interpreting their sizes.

Usage

ctScoreTest(
  fit,
  covariates = NULL,
  tests = c("heterogeneity", "covariates", "statedependence", "volatility", "fixed",
    "level", "instability"),
  form = c("auto", "linear", "spline", "factor", "threshold"),
  df = 3,
  parameters = NULL,
  order = 2L,
  variance = c("robust", "model"),
  nboot = 1999L,
  exact = 5L,
  intoverpop = c("laplace", "quadrature"),
  cores = getOption("mc.cores", 2L),
  verbose = TRUE
)

Arguments

fit

A fit from ctFit with backend = 'julia', optimised.

covariates

Optional data frame of subject-level covariates to test: a column named as the fit's subject id (or id) and one column per covariate. Long data with repeated rows per subject is accepted when each covariate is constant within subject. Without an id column, one row per subject in the fit's subject order.

tests

Which families to run: any of 'heterogeneity', 'covariates', 'statedependence', 'volatility', 'fixed', 'level', 'instability'. By default all but 'instability'; covariates are tested only when covariates is given, and levels only where the model has a random intercept.

form

How a covariate enters: 'auto' (indicators for a factor, character or logical covariate; linear otherwise), 'linear', 'spline', 'factor' or 'threshold' (a step at an unknown cut). One value, or one per covariate.

df

Degrees of freedom of the spline basis.

parameters

Optional parameter names to restrict the heterogeneity and covariate tests to.

order

Highest degree of the state-dependence terms: 2, each drift coefficient varying linearly with each state; 3 adds variation with products and squares of states, which is what a symmetric dependence – regulation that strengthens at both extremes, say – shows up as, and which degree 2 cannot see.

variance

'robust' (the default) or 'model'; see the section on variance.

nboot

Bootstrap resamples for the robust p-values.

exact

For statedependence, volatility and fixed with the robust variance: how many of each family's terms have the fit's estimation removed exactly, which takes two evaluations of the extended model per term. Every term is first tested on its scores without that correction, which is conservative, and the strongest exact by that test are then corrected (projected in the tables). Inf corrects every term.

intoverpop

How the extended families (statedependence, volatility, fixed, level) integrate the random effects on the Laplace route. 'laplace' (the default) takes the Laplace term, a fast screen. 'quadrature' integrates every subject's random effects by quadrature rules placed at their mode and curvature, as the refined fit does (ctFit(intoverpop = 'quadrature')), for the scores, their spread and the curvature: no Laplace error left in them, at the cost of placing a rule for every subject and the extended model's whole Hessian on it. The other families read the fit's own objective either way.

cores

Engine cores for the evaluations.

verbose

Report progress.

Value

An object of class ctScoreTest: a list with heterogeneity (a data frame, one row per parameter or fixed MANIFESTMEANS cell), dimension (the largest eigenvalue, its p-value and the direction, when two or more parameters are tested), covariates (one element per covariate, each a list of its joint test and a data frame parameters), statedependence (a list of terms, a data frame, and joint, the tests per equation and overall), volatility (the same, for the innovation sds), fixed (a list of cells and their joint test), instability (a list of parameters, one row per ordering and parameter, and joint, per ordering), and info (the route, the number of subjects and parameters, the variance used, and errors, why any family was not tested). Tables carry class where relaxations are alike, and the extended families' projected, whether a term's estimation was removed exactly (see exact).

Heterogeneity

For each parameter that is free, does not already vary over subjects, and is not an initial condition (T0MEANS and T0VAR, whose variation T0VAR already carries), the score test of letting it vary as ctFit would fit it: its variance over subjects together with its covariances with what already varies – each subject's initial state (the free T0MEANS parameters) and the parameters with random effects, named in with. Every subject has an initial state, so even a model without random effects has those covariances to test. They matter: an intercept in CINT that varies while the initial means are shared makes subjects drift apart from a common start, which data whose subjects differ stably from the first occasion reject, whereas the same intercept correlated with where each subject starts is a random intercept. The covariances are tested as ordinary score tests and the variance, given them, one sided (Stram & Lee, 1994): LM is referred to an equal mixture of chi-square on df - 1 and df. Where the new effect is independent of what varies, those covariances only cost power, so p is the stronger of that test and of the variance alone (z, one sided, its covariances held at zero), by their smaller p-value, calibrated by resampling subjects. A negative z beside a large LM says the parameter varies only together with what already varies, as an intercept does with where each subject starts. Nothing is assumed about how the initial states relate to the dynamics: neither that subjects start at their own equilibrium nor that they share a start. The statistic is the information matrix test in those coordinates, which is the score test for neglected heterogeneity (Chesher, 1984). It responds to any failure of the information equality there, of which subject differences are the usual cause but not the only one: a subject-varying drift also moves the diffusion's test, because both change how much a subject varies, and a missing nonlinearity moves several. sd is the one-step estimate of the subjects' standard deviation on the raw scale, with the covariances estimated beside it, and sd_natural approximately on the parameter's own; NA where the covariances carry the test and the variance given them reads zero, which happens for an intercept, and which only a refit can size. To act on a result, set indvarying = TRUE for the parameter and the T0MEANS parameters, refit, and test again; ctModelImprove does that.

Each MANIFESTMEANS cell the model fixes is tested too, as a random intercept on that indicator about its fixed value (rows named MANIFESTMEANS[...]), from the model with those cells written as parameters. With one indicator per latent such an intercept is a direction the CINT rows' covariances also reach; with more indicators it is the indicator's own; for a latent with no free intercept at all it is the only test of its level. To act on it, give the cell a parameter with the fixed value as its stated mean (ctPar(mean=)) and indvarying = TRUE.

On a reduced-rank level (poprank), a parameter that varies only through the shared dimensions, without a variance of its own, is tested too: for variation outside those dimensions. There the shared dimensions carry the covariances, and each test is of the variance alone. A parameter whose test the fitted model already determines – under one per cent of its spread left once estimation is taken out, as for an intercept that loads on the shared dimensions – is not tested (NA) and is left out of dimension.

dimension tests every direction at once: whether subjects vary along some combination of the tested parameters, a new dimension of person variation, its covariances with what already varies held at zero. It is not computed on a reduced-rank level, where in simulation it found a dimension the rank lacked 10 per cent of the time and the per-parameter tests of variation outside the shared dimensions 58 per cent; those are the test there. Adding variance along a direction u changes a subject's log likelihood, at zero, by (u's_i)^2/2 + u'H_iu/2, so the variances are the diagonal of one score matrix; its largest eigenvalue, scaled by each diagonal's spread, is the statistic, against a bootstrap over subjects, and its eigenvector is the direction, read as relative loadings (raw scale). It can detect variation spread over parameters that no single test singles out, and is less sensitive than the best single test when one parameter carries it.

Covariates

For each covariate, the effects it would have on every tested parameter if added as a time independent predictor (on the raw scale, as a predictor effect acts), including on the population standard deviations of the parameters that vary: a covariate moderating heterogeneity. The joint test of all of a covariate's effects is the screen. Per parameter, estimate is the one-step effect per unit of the covariate with the fitted parameters held at their estimates, jointly over the parameters, so a covariate that shifts one parameter is not read off another correlated with it; estimate_natural puts it approximately on the parameter's own scale, and p tests it. Holding the fit's parameters is what keeps the size near a refit's: a covariate the model lacks is absorbed by the parameters it has – a random effect's standard deviation grows to take in a covariate that shifts its mean – and letting them move in a one-step extrapolates the likelihood across that whole change. A covariate enters linearly (centred), as treatment indicators for a factor, or as a natural spline with df degrees of freedom, for which only p is given; subjects missing it contribute at the mean. To act on a result, add the covariate to the data and to TIpredNames and refit.

form = 'threshold' tests a step at an unknown cut instead: the joint test at each cut from the 15th to the 85th percentile, and the largest of them (Andrews, 1993) – the split a score-guided tree makes – against the same scan with the covariate permuted over subjects (Hothorn & Lausen, 2003), which takes the search into account exactly and refits nothing. The effects reported are those of being above the best cut, cut in the joint row.

On a reduced-rank level the spread of a varying parameter has no popsd_ coordinate, so a covariate's effect on it is tested along the direction that scales that parameter's row of the loadings and its own sd together, reported as scale(<parameter>): the effect on its log standard deviation per unit of the covariate, the correlations held. In simulation (rank 1) these rows rejected 6.5 to 8 per cent of correct models at .05 with 100 subjects and 5 to 6 with 300, where the covariate's joint test held its level: with few subjects the projection on the fitted parameters leaves a variance a little too small. Read them as attribution once the joint test has screened.

State dependence

For each latent equation and each pair of latent states k <= m, the score test of a term d * eta_k * eta_m in that equation's drift: a drift coefficient that varies linearly with a state, the second-order expansion of a drift that depends on the state (Luukkonen, Saikkonen & Terasvirta, 1988). A term eta_k * eta_m in the equation of eta_i is DRIFT[i,k] varying with eta_m – and, when k != m, equally DRIFT[i,m] varying with eta_k; at this order the data cannot tell those apart, and reads says both. estimate is the one-step change in that drift coefficient per unit of the state. It is NA when the fitted model's curvature cannot size the term, which happens when the departure is strong; the test itself is unaffected. order = 3 adds the terms of degree three. The terms are evaluated by the model's own filter at the state predicted from earlier observations, which keeps the test valid; a state estimate that has seen the observation being scored – the updated or smoothed state – would not. To act on a result, write the dependence into the drift cell (DRIFT[1,1] = 'dr11 + d * eta2' with PARS = c('dr11|-log1p_exp(-param)', 'd')) and refit.

Volatility

tests = 'volatility' asks whether the size of each latent's innovations depends on a state: the score test of DIFFUSION[i, i], the innovation sd, scaled by exp(v * eta_m), at v = 0, for every latent with innovations and every state. Variance that rises with a process's own level, as when negative affect is more variable when high, is the diagonal term. Evaluated by the filter at the predicted state, as for state dependence. To act on a result, write the scaling into the cell (DIFFUSION[2,2] = 'sd2 * exp(v * eta2)' with PARS = c('sd2|log1p_exp(param)', 'v')) and refit; ctModelImprove does that. The terms make the model nonlinear, so the test, and a refit, run the extended Kalman filter.

Level

tests = 'level' asks whether a parameter of the dynamics or of the error variances depends on a subject's level, where that level is a random intercept: for each free DRIFT, DIFFUSION and MANIFESTVAR parameter that does not vary over subjects, and each MANIFESTMEANS or CINT intercept that does, the score test of a term g * intercept added to the parameter, at g = 0. When subjects' levels sit in the intercepts, the latent states are deviations from them, so the state dependence and volatility tests, which read the states, see dependence on the deviation only; a parameter that depends on where a subject sits – persistence or innovation size that differs between people who are usually high and people who are usually low – shows there only as heterogeneity, in no particular direction. These terms ask about that direction. The score is taken over each subject's posterior for the intercept, from the extended model's own gradient, which the product of a subject's score and estimated level is not. estimate is the one-step change in the parameter, on its own scale, per unit of the intercept. To act on a result, let the parameter vary over subjects, so its random effect is estimated with its correlation with the intercept; ctModelImprove does that. A model with no random intercept has nothing to test.

Fixed cells

For each cell the model fixes at zero that could sensibly be freed – a DRIFT cross effect, a DIFFUSION or MANIFESTVAR covariance between two that have a variance, a LAMBDA cross loading, a TDPREDEFFECT – the score test of freeing it: the modification index (Sorbom, 1989), with estimate the expected parameter change. CINT and MANIFESTMEANS are not offered, since with the other free either is identified only through the initial transient. State dependence, volatility and fixed cells are tested on one extended model, so asking for several costs one preparation.

Stability

tests = 'instability', not run unless asked for, tests whether each parameter is stable across the data: ordered by time, and by each latent's state as predicted from earlier observations. It uses each data row's score – the derivative of its one-step log likelihood – whose partial sums, under the model, behave as a Brownian bridge in any order fixed in advance (Nyblom, 1989). A parameter that changes over the study, or differs at high and low levels of a state, bends them. Ordered by a state, that is a test of state dependence of unknown form, beside the polynomial terms of statedependence. The estimation effect is removed with the information accumulated along the ordering, and the p-values are from a wild bootstrap over rows. It costs a dual pass of the filter per eight parameters, and on the Laplace route one per truncated record of every subject.

Alike relaxations

Two relaxations whose per-subject projected contributions are nearly collinear (correlation of 0.9 or more) would each explain the same signal, and the data cannot tell them apart at this order (MacCallum et al., 1993). They share a letter in class; NA means alike to none. Refit with the most plausible, then test again.

Variance and p-values

The fitted model's curvature projects the fitted parameters out of each test and sets the one-step sizes; the variance of each test is, by default (variance = 'robust'), the spread over subjects of their projected contributions, and its p-value a bootstrap over subjects of the studentised statistic (nboot resamples, from a fixed stream, so a call is reproducible and leaves the session's random numbers alone). That keeps each test valid when the model is wrong in some other way, which is when a screen is used, and corrects for the skewness of a variance score. variance = 'model' uses the curvature for the variance and chi-square and normal references instead; it is what is used below 20 subjects, where a spread over subjects is too thin to estimate.

Reading the results

The tests are screens. A badly misspecified model lights up many at once, and their one-step sizes are local approximations – especially for drift parameters, which act through the matrix exponential. Relax what stands out most, refit, and test again. Within each family p_adj is the Benjamini-Hochberg adjustment over the tests shown. The tests use the fit's own objective, so they apply to the route it was fitted on (intoverpop); they are made at an optimum, and refuse a sampled fit. Heterogeneity and covariate tests need subjects as the independent units, so a model with a level above the subject can only be tested for state dependence. A family that cannot be tested on a model says why, as a warning and in info$errors, and the other families' results stand. With priors = TRUE the new terms of the extended model would carry the normal(0, 1) prior every parameter has; they are tested on the likelihood, without it.

On the Laplace route the filter integrates a fit's intercept-type random effects exactly where it can, and Laplace approximates only the others (the two-step term; optimcontrol$laplace_onestep in ctFit). The state dependence and volatility families extend the model with terms that read the state, which that filter cannot take; where the fit has random dynamics beside such effects, those families are then computed with the fit and its extension both on the one-step term, a fast screen whose approximation error stays in the scores, and info$onestep names them. intoverpop = 'quadrature' integrates every subject's random effects for these families instead, leaving no Laplace error in them (info$quadrature).

References

Boos, D. D. (1992). On generalized score tests. The American Statistician, 46, 327-333.

Chesher, A. (1984). Testing for neglected heterogeneity. Econometrica, 52, 865-872.

Stram, D. O., & Lee, J. W. (1994). Variance components testing in the longitudinal mixed effects model. Biometrics, 50, 1171-1177.

Self, S. G., & Liang, K.-Y. (1987). Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. Journal of the American Statistical Association, 82, 605-610.

Hothorn, T., & Lausen, B. (2003). On the exact distribution of maximally selected rank statistics. Computational Statistics & Data Analysis, 43, 121-137.

Andrews, D. W. K. (1993). Tests for parameter instability and structural change with unknown change point. Econometrica, 61, 821-856.

Nyblom, J. (1989). Testing for the constancy of parameters over time. Journal of the American Statistical Association, 84, 223-230.

Hall, D. B., & Praestgaard, J. T. (2001). Order-restricted score tests for homogeneity in generalised linear and nonlinear mixed models. Biometrika, 88, 739-751.

MacCallum, R. C., Wegener, D. T., Uchino, B. N., & Fabrigar, L. R. (1993). The problem of equivalent models in applications of covariance structure analysis. Psychological Bulletin, 114, 185-199.

Luukkonen, R., Saikkonen, P., & Terasvirta, T. (1988). Testing linearity against smooth transition autoregressive models. Biometrika, 75, 491-499.

Sorbom, D. (1989). Model modification. Psychometrika, 54, 371-384.

See Also

vignette('score-tests') for three worked examples.

Examples

## Not run: 
fit <- ctFit(dat, model, backend = 'julia')
st <- ctScoreTest(fit, covariates = subjectdata)
st
st$heterogeneity

## End(Not run)

Options ctsem reads

Description

Settings that change how ctsem behaves across a session, rather than per-call. Set them with options(), typically once at the top of a script or in a front end's startup.

Value

Nothing; this page documents options rather than defining a function.

Progress and output

ctsem.progress.overwrite

Whether progress reporting rewrites a single line with a carriage return (TRUE) or prints occasional separate lines (FALSE). Unset, ctsem detects it: an active Shiny session, a sink(), or a knitr chunk all capture output, where a carriage return is a character rather than a cursor movement. Set it explicitly if that detection is wrong for your front end.

ctsem.cluster.outfile

Where parallel workers send their output. Unset, they are silent. Setting it to "" forwards worker stdout and stderr to the console, which is useful only when debugging a worker – it also restores the startup chatter that made a two-core fit announce itself six times.

Per-call alternatives

Progress and reporting are also controlled per fit, and those win over these: verbose in ctFit (0 quiet, 1 reports progress, 2 keeps every line rather than overwriting), optimcontrol$callback for a function called while the fit runs, and optimcontrol$progress to force progress on or off for one fit.

Julia

The Julia backend is mostly located through the environment rather than options: JULIA_BINDIR names the binary directory, JULIA_NUM_THREADS the thread count of the session that starts next, and ctJuliaSetup(project=) points the session at a different engine environment. See ctJuliaSetup.

ctsem.julia.restart

Whether a fit asking for more cores than the running Julia session has threads may restart the session to get them. FALSE by default: Julia fixes its thread count at process start, so the only way to raise it is to end the process, which discards the engine's compiled model shapes and costs seconds. Left FALSE, such a fit runs at the threads it has and says so.


ctstantestdat

Description

Generated dataset for testing ctFit from ctsem package.

Format

matrix


ctstantestfit

Description

Dummy fit for testing functions from ctsem package.

Format

ctStanFit object


How model matrix cells vary with a latent state

Description

Plot every context-dependent matrix cell against one latent process, across the range that process occupies, with the posterior spread at each point.

This is the plot that says what a state-dependent model actually claims. summary() reports each cell as a single number at a single state; here the same cell is drawn as the function of the state it really is, and the summary's number is one point on that curve.

Usage

ctStateDependencePlot(
  fit,
  along = 1,
  extent = "data",
  gridsize = 25,
  nsamples = 20,
  quantiles = c(0.025, 0.5, 0.975),
  state = "asymptotic",
  tipreds = NULL,
  tolerance = 1e-06,
  plot = TRUE
)

Arguments

fit

A ctJuliaFit.

along

The latent process to vary, by name or index.

extent

'data' (the default) spans the smoothed latent states, 'sd' spans two stationary standard deviations either side of T0MEANS, or supply a numeric range of length two.

gridsize

Points along the range. One engine call each.

nsamples

Posterior draws for the interval, or 1 for the point estimate alone.

quantiles

Three quantiles: lower ribbon, line, upper ribbon.

state

Where the other latent processes are held: 'asymptotic', 'mean', 'T0MEANS', or a numeric state.

tipreds

Time independent predictor values, or NULL for the population.

tolerance

A cell is treated as varying when its range across the grid exceeds this fraction of its own magnitude.

plot

If FALSE, return the values as a data frame rather than a plot.

Details

Cells are chosen by measuring rather than by parsing: every model matrix is materialised across the grid, and a cell is kept when its value actually moves. That catches dependence arriving indirectly – through a PARS entry, or through a covariance built from a state-dependent cell – without having to trace it symbolically.

A model with no state-dependent cells has nothing to draw, and says so rather than returning an empty plot.

Value

A ggplot, or a data frame when plot=FALSE.

See Also

ctContextDependence, ctPhasePortrait

Examples

# Needs a working julia backend, so the example is inert where Julia is
# absent -- including on CRAN, whose check machines have none.
if (isTRUE(ctJuliaStatus()$available)) {
  set.seed(1)
  generating <- suppressMessages(ctModel(type = 'ct', n.latent = 2, n.manifest = 2,
    manifestNames = c('Y1', 'Y2'), latentNames = c('eta1', 'eta2'),
    LAMBDA = diag(2), DRIFT = matrix(c(-.4, .1, 0, -.3), 2, 2),
    CINT = matrix(c(.2, .1), 2, 1), MANIFESTMEANS = matrix(0, 2, 1),
    MANIFESTVAR = diag(.2, 2), DIFFUSION = matrix(c(.5, 0, 0, .4), 2, 2)))
  datalong <- as.data.frame(suppressMessages(ctGenerate(generating, n = 25,
    burnin = 5, dtmean = 1, logdtsd = .1, wide = FALSE, Tpoints = 12)))

  # eta1's own decay depends on where eta2 is -- a state-dependent DRIFT cell.
  model <- suppressMessages(ctModel(type = 'ct', n.latent = 2, n.manifest = 2,
    manifestNames = c('Y1', 'Y2'), latentNames = c('eta1', 'eta2'),
    LAMBDA = diag(2), PARS = c('dr11|-log1p_exp(param)'),
    DRIFT = matrix(c('dr11 * (1 + 0.2 * eta2)', 'd21', 0, 'd22'), 2, 2),
    CINT = matrix(c('c1', 'c2'), 2, 1), MANIFESTMEANS = matrix(0, 2, 1),
    MANIFESTVAR = diag(.2, 2), DIFFUSION = matrix(c('df1', 0, 0, 'df2'), 2, 2)))
  model$pars$indvarying <- FALSE

  fit <- ctFit(datalong, model, backend = 'julia', cores = 1, verbose = 0)
  ctStateDependencePlot(fit, along = 'eta2', gridsize = 11, nsamples = 5)
  # Julia keeps running between calls, holding what it has compiled;
  # this ends it.
  ctJuliaKill("session")
}

Extract an array of subject specific parameters from a fit

Description

Extract an array of subject specific parameters from a fit from ctFit, on either backend. ctStanSubjectPars is maintained as a backward-compatible alias.

Usage

ctSubjectPars(
  fit,
  pointest = TRUE,
  cores = getOption("mc.cores", 2L),
  nsamples = "all",
  level = NULL,
  uncertainty = NULL
)

Arguments

fit

A fit from ctFit.

pointest

if TRUE, returns only the set of individual difference parameters based on the max a posteriori estimate (or the median if sampling approaches were used).

cores

Number of cores to use. backend='stan' only – a julia fit already carries the subject matrices, so there is nothing here to split, and cores is refused by name rather than accepted and ignored.

nsamples

Number of samples to calculate parameters for. Not used if pointest=TRUE.

level

For a model with nested ids (id = c('person', 'couple') in ctModel), the level to report: the subject level (the default, the first id) or a grouping level above it, which gives each group's own parameter values – the population values shifted by that group's random effects and those of the levels above it, without the subject-level ones or TI predictor effects. Grouping levels need a backend='julia' fit; their draws take uncertainty as a subject's do.

uncertainty

For backend='julia' fits with pointest=FALSE: what the draws carry. 'total' (the default) pairs each draw of the population parameters with a draw of each subject's random effects from their posterior given that subject's data – plausible values, whose spread is both the population parameters' uncertainty and how well the subject's own data determine its parameters. 'conditional' keeps the population parameters at the estimate and draws only the subject's effects; 'population' varies only the population parameters, with each subject at its estimate given them, which understates a subject's uncertainty whenever its own data say little. A fit that sampled its random effects (optimize=FALSE) already has draws of both, so only 'total' applies there.

Details

This function returns the estimates of individual parameters, taking into account any covariates and random effects, at every level of a nested model.

A julia fit also stores, for every level with random effects, each group's conditional posterior of its effects on the raw parameter scale – fit$randomeffects, one entry per level with the groups' mean deviation from the population value and their covariance packed column by column (cov). The draws with uncertainty='total' are built from the same posterior.

Value

an nsamples by nsubjects by nparams array, with the subjects (or groups, for a grouping level) labelled by their ids in the data.

Examples

indpars <- ctSubjectPars(ctstantestfit)
dimnames(indpars)
plot(indpars[1,,'cint1'],indpars[1,,'cint2'])

ctSummaryMatrices

Description

Summarise model-implied parameter matrices from a ctsem fit object. ctStanContinuousPars is maintained as a backward-compatible alias.

Usage

ctSummaryMatrices(
  fit,
  calcfunc = quantile,
  calcfuncargs = list(probs = 0.5),
  timeinterval = 1,
  state = NULL,
  ...
)

Arguments

fit

fit object from ctFit

calcfunc

Function to apply over samples, must return a single value. By default the median over all samples is returned using the quantile function, but one might also be interested in the mean or sd, for instance.

calcfuncargs

A list of additional parameters to pass to calcfunc. For instance, with the default of calcfunc = quantile, the probs argument is needed to ensure only a single value is returned.

timeinterval

time interval for discrete time parameter matrix computation.

state

The latent state at which state dependent matrix cells are evaluated: 'T0MEANS' (the default), 'mean', 'asymptotic', or a numeric state vector. The default is the population initial state, with time dependent predictors at zero, which for a model whose dynamics depend on the state can be far from where the data are; 'mean' (the mean smoothed latent state) or 'asymptotic' (the system's fixed point) is then usually the better point. The initial state and the fixed point are each draw's own; 'mean' is the estimate's, used for every draw, so a summary of the draws there leaves out where that state is. Only backend='julia' can materialise the matrices anywhere other than the default, so anything else is an error on a stan fit rather than being silently ignored. Irrelevant to a linear model, whose matrices are the same everywhere.

...

arguments passed to the method for the fit's backend.

Value

A named list of matrices, each summarised over the posterior (or approximate posterior) draws by calcfunc: the model's own (DRIFT, DIFFUSION, LAMBDA, ...), the covariances built from them (DIFFUSIONcov, MANIFESTcov, and T0cov, the initial latent covariance over subjects: T0VAR plus the population covariance of any individually varying T0MEANS), the asymptotic asymCINT and asymDIFFUSIONcov, and dtDRIFT at timeinterval. Beside each covariance its correlations, DIFFUSIONcor, MANIFESTcor, T0cor and asymDIFFUSIONcor, taken draw by draw before calcfunc summarises them; a variable whose variance is zero has NA there. On backend='julia', also the Jacobians of the drift and of the measurement at the evaluation point, JAx (latent by latent) and Jy (manifest by latent): the local effects of the model linearised at state. They equal DRIFT and LAMBDA for a linear model; where a cell is nonlinear in the state they do not, since a DRIFT cell a + q*Y in the column of process Y holds a + q*Y there and contributes a + 2*q*Y to JAx. dtDRIFT is expm(JAx * timeinterval) where there is a JAx, the transition ctDiscretePars reports, and expm(DRIFT * timeinterval) on stan. For a model whose dynamics depend on the state it holds JAx fixed over the interval: a fair interval regression near a fixed point or over a short interval, and away from a fixed point one ctDiscretePars(method = 'simulate') should check. Where parameters vary over subjects, also their population covariance on the natural scale, popcov, and their raw-scale correlations, rawpopcorr, named by parameter (with several grouping levels, popcov.<level> and rawpopcorr.<level>). popcov combines the sds summary() reports as popsd with those correlations, so it is exact where a parameter's transform is affine, as for intercepts and initial means.

On an optimised fit, estimate holds the same matrices at the fit's estimate, unrounded: ctSummaryMatrices(fit)$estimate$DRIFT is the estimated drift matrix, which a summary of the draws need not equal, and ctSummaryMatrices(fit, state = s)$estimate$JAx the model linearised at s at the estimate. A sampled fit has no estimate, and a backend='stan' fit's has no popcov, which stan simulates.

Examples

#posterior median over all subjects (also reflects mean of unconstrained pars)
ctSummaryMatrices(ctstantestfit)

Get time independent predictor effect estimates

Description

Computes and plots combined effects and quantiles for effects of time independent predictors on subject level parameters of a ctStanFit object. ctStanTIpredeffects is maintained as a backward-compatible alias.

Usage

ctTIpredEffects(
  fit,
  returndifference = FALSE,
  probs = c(0.025, 0.5, 0.975),
  includeMeanUncertainty = FALSE,
  whichTIpreds = 1,
  parmatrices = TRUE,
  whichpars = "all",
  nsamples = 100,
  timeinterval = 1,
  nsubjects = 20,
  filter = NA,
  plot = FALSE
)

Arguments

fit

fit object from ctFit

returndifference

logical. If FALSE, absolute parameter values are returned. If TRUE, only the effect of the covariate (i.e. without the average value of the parameter) are returned. The former can be easier to interpret, but the latter are more likely to fit multiple plots together. Not used if parmatrices=TRUE.

probs

numeric vector of quantile probabilities from 0 to 1. Specify 3 values if plotting, the 2nd will be drawn as a line with uncertainty polygon based on 1st and 3rd.

includeMeanUncertainty

if TRUE, output includes sampling variation in the mean parameters. If FALSE, mean parameters are fixed at their median, only uncertainty in time independent predictor effects is included.

whichTIpreds

Which of the model's time independent predictors to calculate effects for: their names, as given in TIpredNames in ctModel (whichTIpreds = 'TI2'), or their positions in that list (whichTIpreds = 2). Unless quadratic / higher order versions of predictors have been included, selecting more than one probably doesn't make sense. If for instance a squared predictor has been included, then you can specify both the linear and squared version. The x axis of the plot (if generated) is based on the first predictor selected. A smooth term (s(age) in ctModel's TIpredNames) is selected by its covariate's name, whichTIpreds = 'age', and is drawn over that covariate, with each person's own value at their covariate value as a point (returned as $persons): the spread of the points about the curve is what the smooth leaves to the person differences.

parmatrices

Logical. If TRUE (default), system matrices rather than specific parameters are referenced – e.g. 'DRIFT' instead of a parameter name like drift12.

whichpars

if parmatrices==TRUE, character vector specifying which matrices, and potentially which indices of the matrices, to plot. c('dtDRIFT[2,1]', 'DRIFT') would output for row 2 and column 1 of the discrete time drift matrix, as well as all indices of the continuous time drift matrix. If parmatrices==FALSE, integer vector specifying which of the subject level parameters to compute effects on. The integers corresponding to certain parameters can be found in the param column of the fit$setup$matsetup object. In either case 'all' uses all available parameters.

nsamples

Positive integer specifying the maximum number of saved iterations to use. Character string 'all' can also be used.

timeinterval

positive numeric indicating time interval to use for discrete time parameter matrices, if parmatrices=TRUE.

nsubjects

Positive integer specifying the number of subjects to compute values for. When only one TIpred is used, this specifies the number of points along the curve. Character string 'all' can also be used. Time taken for plotting is a function of nsubjects*niterations.

filter

either NA, or a length 2 vector, where the first element contains the time independent predictor index to filter by, and the second contains the comparison operator in string form (e.g. "< 3", to only calculate effects for subjects where the tipreds of the denoted index are less than 3).

plot

Logical. If TRUE, nothing is returned but instead ctPlotArray is used to plot the output instead.

Value

Either a three dimensional array of predictor effects, or nothing with a plot generated.

Examples

ctTIpredEffects(ctstantestfit,
 whichpars=c('CINT','dtDIFFUSION[2,2]'), plot=TRUE)

Plot how a fit reached its estimate

Description

For an optimized fit, draws the log posterior and the gradient norm against iteration, from the record the engine kept while fitting. For a sampled fit, draws each chain's draws of every population mean and standard deviation, on the scale summary() reports them on, so chains that have not settled on one distribution can be seen.

Usage

ctTracePlot(fit, which = NULL, plot = TRUE)

Arguments

fit

A fit from ctFit with the Julia backend.

which

Which panels to draw: trace columns ('objective', 'gradient_norm', 'inner_converged') for an optimized fit, parameter names for a sampled one. Defaults to all of them.

plot

If FALSE, return the data rather than a plot: the trace for an optimized fit, one row per draw and parameter for a sampled one.

Details

The Julia backend records the optimisation trace during the fit and returns it as fit$optim$trace. For output while a fit is still running, see optimcontrol$callback in ctFit. The chains of a sampled stan fit are drawn by plot(fit, types = 'trace').

Value

A ggplot, or with plot = FALSE a data frame.

Examples

# Needs a working julia backend, so the example is inert where Julia is
# absent -- including on CRAN, whose check machines have none.
if (isTRUE(ctJuliaStatus()$available)) {
  model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1',
    LAMBDA = matrix(1))
  model$pars$indvarying <- FALSE
  fit <- ctFit(ctstantestdat, model, backend = 'julia', cores = 1)
  ctTracePlot(fit)
  # Julia keeps running between calls, holding what it has compiled;
  # this ends it.
  ctJuliaKill("session")
}

Decompose model implied variance into between person, process and measurement

Description

Splits the variance each indicator of a fitted model implies into a between person part, a within person part from the latent process, and a within person part from measurement error – with the process part further split into what the system noise contributes and what the mean path moving contributes.

Usage

ctVarianceDecomposition(
  fit,
  method = c("auto", "moment", "simulation"),
  persons = c("auto", "model", "estimated"),
  scale = c("latent", "response"),
  times = NULL,
  npersons = 200L,
  npaths = 20L,
  latents = TRUE,
  quadpoints = 21L,
  subjects = "all",
  realid = TRUE,
  sources = NULL
)

Arguments

fit

fit object as generated by ctFit, from either backend.

method

How the model implied moments are obtained. 'moment' runs the forward moment recursion, which is exact and needs the dynamics to be linear. 'simulation' draws trajectories from the julia engine instead, which is what a model with state dependent DRIFT or DIFFUSION cells needs. 'auto', the default, is 'moment' unless the model has such cells.

persons

Which population the between person variance refers to. 'model' draws persons from the fitted population distribution of the random effects, which is the model's own claim about the population and is not shrunk. 'estimated' uses the subjects in the data, which describes this sample: on a backend='julia' fit each at plausible values – its estimated effects plus a draw from their posterior given its data – which are not shrunk either; on a stan fit at their estimated (empirical Bayes) matrices, which attenuates the between person variance. 'auto', the default, is 'model' for a backend='julia' fit and 'estimated' for a stan one, saying which it used – only the julia engine can materialise the model matrices at a drawn set of random effects.

scale

For a binary or ordinal indicator, whether to decompose the linear predictor behind the logit ('latent', the default, with a measurement variance of π2/3\pi^2/3) or the observed response ('response', integrating the link over each row's own normal marginal). The two coincide for a Gaussian indicator.

times

If given, decompose at each of these times rather than averaging over the window: a numeric vector, or 'asdata' for the observed occasions. Every person is evaluated on the one grid, which is what makes a variance across persons at a given time mean anything. The result gains a time column and loses within.deterministic – that term is the variance of the mean path over time, so it exists only in the average. Not available with method='simulation' or with time dependent predictors, neither of which has values on a grid of its own.

npersons

Number of persons to draw when persons='model'. With persons='estimated' on a julia fit, each subject gets as many plausible values as make at least this many persons in all.

npaths

Trajectories drawn per person when method='simulation'. Each person needs several: with one path, the average over that person's rows wanders with the path rather than estimating their mean, and the wander lands in the between person term. The two Monte Carlo corrections this makes necessary are applied, so the result is unbiased at any npaths, but small values are noisy.

latents

If TRUE, also decompose the latent processes, which have no measurement component.

quadpoints

Gauss-Hermite nodes used when scale='response'.

subjects

'all', or the subjects whose designs the decomposition is computed over, as ids in the data (see realid).

realid

If TRUE (the default), subjects are read as the ids in the data, and integers that are not ids there are read as ctsem's numbering of the subjects, 1 to N, with a message. If FALSE, they are read as that numbering.

sources

Optional named list of latent names, one entry per group of processes whose innovations are independent of the other groups' – fast affect, a day-scale tier, a slow tier, say. The stochastic within person term is then split by where it came from: within.stochastic.<group> for each group's innovations and within.stochastic.initial for the initial state's variance still present, which add up to within.stochastic. Averaged over the window, level.<group> is the part of each that is the variance of the record's own mean: a process as slow as the record is long acts within it mostly as a level, and this says how much. Needs method='moment'.

Details

The decomposition is a nested law of total variance over the population of (person, occasion) pairs at the design the fit was built on. With g=E[y∣η]g = E[y | \eta] the model's expected observation given the latent state,

Var(y)=E[Var(y∣η)]+EiEtVar(g)+EiVart(E[g])+Vari(E[g])Var(y) = E[Var(y|\eta)] + E_i E_t Var(g) + E_i Var_t(E[g]) + Var_i(E[g])

whose four terms are the four columns reported. They are non-negative and sum to the total exactly.

within.deterministic is where non-stationarity appears: it is the variance of a person's own mean path over their observation times, so it is zero for a stationary process with no time dependent predictors and positive when there is a trend, a predictor driven shift, or a process still relaxing from its starting distribution.

The moments come from the forward recursion over each subject's actual observation times, with no data entering, so unequal spacing, missingness and time dependent predictors are handled exactly rather than by assuming stationarity. The result is exact for a model with linear dynamics.

A model whose DRIFT or DIFFUSION cells depend on the state has no single transition to run that recursion with, so method='simulation' draws its trajectories from the engine and takes the same decomposition over the draws. It is refused rather than linearised at a point nothing chose. A state dependent measurement cell – LAMBDA, MANIFESTMEANS, MANIFESTVAR – is refused by both routes, since turning a drawn state into an expected observation then needs the measurement model re-materialised at every row.

Random effect levels. A model with a grouping level above the subject (id = c('subject','study') in ctModel) is fitted with intoverpop='laplace' – or sampled, which prepares the same description and is handled the same way here – and gets one between.<level> column per level, each the variance of that level's own departure from the level outside it. They are orthogonal by construction and sum to between.

Each level's fitted modes are shrunk toward the level outside them by an amount that follows the information per unit rather than the number of units: on a 6-study, 30-subject example the subject level's modes retained 52 per cent of the population standard deviation (ten occasions each) while the study level's retained 97 per cent (five subjects each). persons='estimated' therefore adds to each mode a draw from that unit's posterior, drawn jointly over its levels: the spread of such plausible values is the population's, where the spread of the modes is not. persons='model' draws from each level's own fitted covariance instead, and is the default.

When the split changes over the window. The columns above are averages over each person's occasions, and for a process that has not settled they average a split that is genuinely different at different times. within.deterministic is the flag that this is happening – it is the mean path moving – and times= is how to look. On a fixture whose T0MEANS is fixed, so that every person starts in the same place, the between person share of an indicator's variance runs from zero at the first occasion to 20 per cent by the twentieth, against a window average of 17 per cent that describes no occasion in particular.

Designs. Nothing here assumes a balanced one. Each person is evaluated over their own observation times, so everyone measured at the same occasions, everyone at their own, and subjects with different numbers of occasions all work, as does an observation being missing – a row still has a model implied distribution whether or not its indicator was seen, so every design row counts and the answer does not move with the missingness pattern. Persons are weighted equally rather than by row count: the estimand is over persons, so a frequently measured subject should not be a larger share of the population than a rarely measured one.

method='simulation' checks, by generating twice, that the engine draws a model's individually varying parameters rather than leaving every simulated person at the population values, and refuses rather than returning a between person variance of zero if it does not.

Value

A data frame of class ctVarianceDecomposition, one row per variable, with columns variable, type, between, within.deterministic, within.stochastic, within.measurement, within, total and the corresponding proportions. The population the between column refers to, the scale, and the number of persons are recorded as attributes and shown by print.

See Also

ctSummaryMatrices for the matrices this is built from, and ctStateDependencePlot for a model whose dynamics vary over the state space.

Examples

ctVarianceDecomposition(ctstantestfit)

ctWideNames sets default column names for wide ctsem datasets. Primarily intended for internal ctsem usage.

Description

ctWideNames sets default column names for wide ctsem datasets. Primarily intended for internal ctsem usage.

Usage

ctWideNames(
  n.manifest,
  Tpoints,
  n.TDpred = 0,
  n.TIpred = 0,
  manifestNames = "auto",
  TDpredNames = "auto",
  TIpredNames = "auto"
)

Arguments

n.manifest

number of manifest variables per time point in the data.

Tpoints

Maximum number of discrete time points (waves of data, or measurement occasions) for an individual in the input data structure.

n.TDpred

number of time dependent predictors in the data structure.

n.TIpred

number of time independent predictors in the data structure.

manifestNames

vector of character strings giving column names of manifest indicator variables

TDpredNames

vector of character strings giving column names of time dependent predictor variables

TIpredNames

vector of character strings giving column names of time independent predictor variables

Value

A character vector of column names for a wide format dataset: the manifest variables and time dependent predictors at each time point, the intervals dT1, dT2, ..., then the time independent predictors.


ctWideToLong Convert ctsem wide to long format

Description

ctWideToLong Convert ctsem wide to long format

Usage

ctWideToLong(
  datawide,
  Tpoints,
  n.manifest,
  n.TDpred = 0,
  n.TIpred = 0,
  manifestNames = "auto",
  TDpredNames = "auto",
  TIpredNames = "auto"
)

Arguments

datawide

ctsem wide format data

Tpoints

number of measurement occasions in data

n.manifest

number of manifest variables

n.TDpred

number of time dependent predictors

n.TIpred

number of time independent predictors

manifestNames

Character vector of manifest variable names.

TDpredNames

Character vector of time dependent predictor names.

TIpredNames

Character vector of time independent predictor names.

Details

Names must account for *all* the columns in the data - i.e. do not leave certain variables out just because you do not need them.

Value

A long format matrix, one row per subject and time point, with the id, the time interval dT (absolute times are restored by ctDeintervalise), the manifest variables and the predictors.

Examples

#create wide data
wideexample <- ctLongToWide(datalong = ctstantestdat, id = "id", 
time = "time", manifestNames = c("Y1", "Y2"), 
TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3"))

wide <- ctIntervalise(datawide = wideexample, Tpoints = 10, n.manifest = 2, 
n.TDpred = 1, n.TIpred = 3, manifestNames = c("Y1", "Y2"), 
TDpredNames = "TD1", TIpredNames = c("TI1", "TI2","TI3") )

 #Then convert to long format
 longexample <- ctWideToLong(datawide = wideexample, Tpoints=10, 
 n.manifest=2, manifestNames = c("Y1", "Y2"),
 n.TDpred=1, TDpredNames = "TD1", 
 n.TIpred=3, TIpredNames = c("TI1", "TI2","TI3"))

 #Then convert the time intervals to absolute time
 long <- ctDeintervalise(datalong = longexample, id='id', dT='dT')
 head(long,22)

datastructure

Description

Simulated example dataset for the ctsem package

Format

2 by 15 matrix containing containing ctsem wide format data. 3 measurement occasions of manifest variables Y1 and Y2, 2 measurement occasions of time dependent predictor TD1, 2 measurement intervals dTx, and 2 time independent predictors TI1 and TI2, for 2 individuals.


Inverse logit

Description

Maps the stan function so the same code works in R.

Usage

inv_logit(x)

Arguments

x

value to calculate the inverse logit for.

Value

1 / (1 + exp(-x)), elementwise.

Examples

inv_logit(-3)

log1p_exp

Description

Maps the stan function so the same code works in R.

Usage

log1p_exp(x)

Arguments

x

value to use.

Value

log(1 + exp(x)), elementwise, finite where exp(x) overflows.

Examples

log1p_exp(-3)

longexample

Description

Simulated example dataset for the ctsem package

Format

7 by 8 matrix containing ctsem long format data, for two subjects, with three manifest variables Y1, Y2, Y3, one time dependent predictor TD1, two time independent predictors TI1 and TI2, and absolute timing information Time.


Oscillating

Description

Simulated example dataset for the ctsem package.

Format

200 by 21 matrix containing containing ctsem wide format data. 11 measurement occasions and 10 measurement intervals for each of 200 individuals

Source

See https://bpspsychub.onlinelibrary.wiley.com/doi/10.1111/j.2044-8317.2012.02043.x


Plot the adequacy checks

Description

Without family, each family's Holm-adjusted p-value on a log scale, against alpha. With a family, its statistics: for the families over the gaps between a subject's rows (residuals, latent, volatility, scores), each pair's estimate by gap with a 95 per cent interval; for periodic, each variable's z by period; for any other, the statistics' z, the 40 largest. The intervals and z are each statistic's own: the family's p-value, which accounts for how many there are, is the test.

Usage

## S3 method for class 'ctFitAdequacy'
plot(x, family = NULL, ...)

Arguments

x

A ctFitAdequacy.

family

One family, or NULL for all of them.

...

Unused.

Value

A ggplot.


Plot a profile likelihood

Description

One panel per parameter: the drop in log likelihood against the value the parameter was held at, with the likelihood-ratio bar drawn across. A curve that crosses the bar on both sides has a confidence interval; one that runs along the bottom is a direction the data does not determine.

Usage

## S3 method for class 'ctFitProfile'
plot(x, parameters = NULL, ...)

Arguments

x

a ctFitProfile.

parameters

which to draw; defaults to all that were profiled.

...

passed to plot.

Value

x, invisibly.


Plots for ctJuliaFit objects

Description

The backend='julia' counterpart of plot.ctStanFit. 'regression' plots model implied regression coefficients against the time interval via ctDiscretePars, 'kalman' plots expectations via ctPredict, 'priorcheck' plots each population parameter's prior against its posterior via ctPlotPosterior, and 'trace' plots how the fit reached its estimate via ctTracePlot: the optimiser's trace, or each chain's draws for a sampled fit. Stan's own density and interval plots of a sampled fit ('density', 'intervals') have no julia counterpart; 'priorcheck' shows the posterior densities.

Usage

## S3 method for class 'ctJuliaFit'
plot(x, types = "all", wait = TRUE, ...)

Arguments

x

Fit object from ctFit(..., backend='julia').

types

Vector of plot types: 'all', 'regression', 'kalman', 'priorcheck', 'trace'. 'all' leaves out 'regression' for a discrete time model, 'priorcheck' for a fit with no posterior draws, and 'trace' for a fit with nothing to trace.

wait

Logical. Pause between plots?

...

Passed through to ctDiscretePars and ctPredict. Beware clashes when types='all'.

Value

Nothing. Generates plots.

Examples

# Needs a working julia backend, so the example is inert where Julia is
# absent -- including on CRAN, whose check machines have none.
if (isTRUE(ctJuliaStatus()$available)) {
  data <- ctstantestdat
  model <- ctModel(type = 'ct', manifestNames = 'Y1', latentNames = 'eta1',
    LAMBDA = matrix(1))
  model$pars$indvarying <- FALSE
  fit <- ctFit(data, model, backend = 'julia', cores = 1)
  plot(fit, wait = FALSE)
  # Julia keeps running between calls, holding what it has compiled;
  # this ends it.
  ctJuliaKill("session")
}

Plots prediction output from ctPredict.

Description

Plots prediction output from ctPredict.

Usage

## S3 method for class 'ctKalmanDF'
plot(
  x,
  subjects = unique(x$Subject),
  kalmanvec = c("y", "yprior"),
  errorvec = "auto",
  errormultiply = 1.96,
  plot = TRUE,
  elementNames = NA,
  polygonsteps = 10,
  polygonalpha = 0.2,
  facets = "Variable",
  ...
)

Arguments

x

Output from ctPredict. In general it is easier to call ctPredict directly with the plot=TRUE argument, which calls this function.

subjects

which subjects to plot, as they appear in the Subject column of x: the ids in the data, unless ctPredict read them as ctsem's numbering.

kalmanvec

string vector of the elements of the output to plot: 'y', the data, or 'y', 'eta', 'err' or 'errstd' followed by 'prior', 'upd' or 'smooth'. 'y' is the observed variables and 'eta' the latent states; 'prior' is conditional on all previous data, 'upd' on all data up to the current time point, and 'smooth' on all data. 'err' is the data minus the estimate, and 'errstd' its standardised form, present when the output was made with standardisederrors=TRUE. The default, 'y' and 'yprior', plots the data against the one step ahead predictions.

errorvec

vector of names indicating which kalmanvec elements to plot uncertainty bands for. 'auto' plots all possible.

errormultiply

Numeric denoting the multiplication factor of the std deviation of errorvec objects. Defaults to 1.96, for 95% intervals.

plot

if FALSE, plots are not generated and the ggplot object is simply returned invisibly.

elementNames

if NA, all relevant object elements are included – e.g. if yprior is in the kalmanvec argument, all manifest variables are plotted, and likewise for latent states if etasmooth was specified. Alternatively, a character vector specifying the manifest and latent names to plot explicitly can be specified.

polygonsteps

Number of steps to use for uncertainty band shading.

polygonalpha

Numeric for the opacity of the uncertainty region.

facets

when multiple subjects are included in multivariate plots, the default is to facet plots by variable type. This can be set to NA for no facets, or "Subject" for facetting by subject.

...

not used.

Value

A ggplot2 object. Side effect – Generates plots.

Examples

### Get output from ctPredict
x<-ctPredict(ctstantestfit,subjects=2,timestep=.01)

### Plot with plot.ctKalmanDF
plot(x, subjects=2)

###Single step procedure:
ctPredict(ctstantestfit,subjects=2,
  kalmanvec=c('y','yprior'),
  elementNames=c('Y1','Y2'), 
  plot=TRUE,timestep=.01)

Plot a model improvement search

Description

The search's path: one row per refit, in the order tried, from the log likelihood its round started at to the one the refit reached, both as gained over the starting fit; kept changes filled, rejected ones open. Pruning's decisions are in the subtitle.

Usage

## S3 method for class 'ctModelImprove'
plot(x, ...)

Arguments

x

A ctModelImprove.

...

Unused.

Value

A ggplot.


Plot score tests

Description

Each family's strongest relaxations: their p-values adjusted within the family (Benjamini-Hochberg), on a log scale, against .05.

Usage

## S3 method for class 'ctScoreTest'
plot(x, family = NULL, top = 10, ...)

Arguments

x

A ctScoreTest.

family

Families to show; by default every family tested.

top

Relaxations shown per family, the strongest.

...

Unused.

Value

A ggplot.


plot.ctStanFit

Description

Plots for ctStanFit objects

Usage

## S3 method for class 'ctStanFit'
plot(x, types = "all", wait = TRUE, ...)

Arguments

x

Fit object from ctFit.

types

Vector of character strings defining which plots to create. 'all' plots all possible types, including: 'regression', 'kalman', 'priorcheck', 'trace', 'density','intervals'.

wait

Logical. Pause between plots?

...

Arguments to pass through to the specific plot functions. Bewar of clashes may occur if types='all'. For details see the specific functions generating each type of plot.

Details

This function is just a wrapper calling the necessary functions for plotting - it may be simpler in many cases to access those directly. They are: ctDiscretePars,ctPredict, ctPlotPosterior,stan_trace, stan_dens,stan_plot rstan offers many plotting possibilities not available here, to use that functionality one must simply call the relevant rstan plotting function. Use x$stanfit as the stan fit object (where x is the name of your ctStanFit object). Because a ctStanFit object has many parameters, the additional argument pars=ctRawParnames(x,'pop_') is recommended. This denotes population means, but see ctRawParnames for other options.

Value

Nothing. Generates plots.

Examples

plot(ctstantestfit,types=c('regression','kalman','priorcheck'), wait=FALSE)

Prior plotting

Description

Plots priors for free model parameters in a ctStanModel.

Usage

## S3 method for class 'ctStanModel'
plot(
  x,
  rows = "all",
  wait = FALSE,
  nsamples = 1e+06,
  rawpopsd = "marginalise",
  inddifdevs = c(-1, 1),
  inddifsd = 0.1,
  plot = TRUE,
  ...
)

Arguments

x

ctStanModel object as generated by ctModel with type='ct' or 'dt'.

rows

vector of integers denoting which rows of ctstanmodel$pars to plot priors for. Character string 'all' plots all rows with parameters to be estimated.

wait

If true, user is prompted to continue before plotting next graph.

nsamples

Numeric. Higher values increase fidelity (smoothness / accuracy) of density plots, at cost of speed.

rawpopsd

Either 'marginalise' to sample from the specified (in the ctstanmodel) prior distribution for the raw population standard deviation, or a numeric value to use for the raw population standard deviation for all subject level prior plots - the plots in dotted blue or red.

inddifdevs

numeric vector of length 2, setting the means for the individual differences distributions.

inddifsd

numeric, setting the standard deviation of the population means used to generate individual difference distributions.

plot

If FALSE, ouputs list of GGplot objects that can be further modified.

...

not used.

Details

Plotted in black is the prior for the population mean. In red and blue are the subject level priors that result given that the population mean is estimated as 1 std deviation above the mean of the prior, or 1 std deviation below. The distributions around these two points are then obtained by marginalising over the prior for the raw population std deviation - so the red and blue distributions do not represent any specific subject level prior, but rather characterise the general amount and shape of possible subject level priors at the specific points of the population mean prior.

Examples

model <- ctModel(type='ct',
manifestNames='sunspots', 
latentNames=c('ss_level', 'ss_velocity'),
LAMBDA=matrix(c( 1, 'ma1' ), nrow=1, ncol=2),
DRIFT=matrix(c(0, 1,   'a21', 'a22'), nrow=2, ncol=2, byrow=TRUE),
MANIFESTMEANS=matrix(c('m1'), nrow=1, ncol=1),
# MANIFESTVAR=matrix(0, nrow=1, ncol=1),
CINT=matrix(c(0, 0), nrow=2, ncol=1),
DIFFUSION=matrix(c(
  0, 0,
  0, "diffusion"), ncol=2, nrow=2, byrow=TRUE))
  
plot(model,rows=8)

Plot an approximate continuous-time ACF object from ctACF

Description

Plot an approximate continuous-time ACF object from ctACF

Usage

plotctACF(
  ctacfobj,
  df = "auto",
  quantiles = c(0.025, 0.5, 0.975),
  separateLearnRates = FALSE,
  reducedXlim = 1,
  estimateSpline = TRUE,
  method = c("weighted", "quantile")
)

Arguments

ctacfobj

object

df

basis dimension for the spline. 'auto' uses up to 40 for method='weighted', and 10% of the number of time intervals for method='quantile'.

quantiles

quantiles to plot.

separateLearnRates

if TRUE, estimate the learning rate for the quantile splines for each combination of variables. Slower but theoretically more accurate. Only used with method='quantile'.

reducedXlim

if non-zero, n timesteps are removed from the upper and lower end of the x range where the spline estimates are less likely to be reasonable.

estimateSpline

if TRUE, a spline is estimated according to method, otherwise the samples are simply plotted as lines and the other arguments here are not used.

method

'weighted' (default) fits a spline through the full data estimates, weighting each time interval by the inverse of its bootstrap variance, so that precisely estimated features such as a sharp peak are followed while noisy estimates are smoothed. Needs nboot > 1 in ctACF. 'quantile' fits quantile regression splines to the pooled bootstrap samples, which smooths all time intervals alike and can flatten sharp peaks.

Details

With a spline, the full data estimate at each time interval is shown as a point, the spline as a line, and the band spans the outer quantiles.

Value

a ggplot object

Examples

data.table::setDTthreads(1) #ignore this line
# Example usage:
head(ctstantestdat)
ac=ctACF(ctstantestdat,varnames=c('Y1'),idcol='id',timecol='time',timestep=.5,nboot=5,plot=FALSE)
plotctACF(ac, reducedXlim=0)

Print a profile likelihood

Description

Print a profile likelihood

Usage

## S3 method for class 'ctFitProfile'
print(x, digits = 3, ...)

Arguments

x

a ctFitProfile.

digits

significant digits for the reported values.

...

ignored.

Value

x, invisibly.


Print a ctsem model

Description

Summarises what the model is, what it will estimate, and what its individually varying parameters imply, rather than printing the underlying list.

Usage

## S3 method for class 'ctStanModel'
print(x, matrices = NULL, ...)

Arguments

x

A model from ctModel.

matrices

Whether to print the model matrices. Defaults to TRUE for models small enough to read.

...

Ignored.

Details

x$pars remains the canonical specification and can be printed directly; ctModelMatrices gives the matrix view.

Value

x, invisibly.


Print ctStanFit summaries

Description

Print ctStanFit summaries

Usage

## S3 method for class 'summary.ctStanFit'
print(
  x,
  width = getOption("width"),
  sections = names(x),
  row.names = TRUE,
  ...
)

Arguments

x

Object returned by summary.ctStanFit.

width

Console width to use for wrapping note text and formatting tables.

sections

Optional character vector of summary sections to print.

row.names

Logical. Print row names for data frame sections?

...

Additional arguments passed to print for individual sections.

Value

Invisibly returns x.


sdcor2cov

Description

Converts a lower triangular matrix with standard deviations on the diagonal and partial correlations on lower triangle, to a covariance (or cholesky decomposed covariance)

Usage

sdpcor2cov(mat, coronly = FALSE, cholesky = FALSE)

Arguments

mat

input square matrix with std dev on diagonal and lower tri of partial correlations.

coronly

if TRUE, ignores everything except the lower triangle and outputs correlation.

cholesky

Logical. To return the cholesky decomposition instead of full covariance, set to TRUE.

Value

A covariance matrix; its lower Cholesky factor with cholesky = TRUE; a correlation matrix with coronly = TRUE.

Examples

testmat <- diag(exp(rnorm(5,-3,2)),5) #generate arbitrary std deviations
testmat[row(testmat) > col(testmat)] <- runif((5^2-5)/2, -1, 1) 
print(testmat)
covmat <- sdpcor2cov(testmat) #convert to covariance
cov2cor(covmat) #convert covariance to correlation

Quickly initialise stanfit object from model and data

Description

Quickly initialise stanfit object from model and data

Usage

stan_reinitsf(model, data, fast = FALSE)

Arguments

model

stanmodel

data

standata

fast

Use cut down form for speed

Value

stanfit object

Examples

sf <- stan_reinitsf(ctstantestfit$stanmodel,ctstantestfit$standata)

Convert samples from a stanfit object to the unconstrained scale

Description

Convert samples from a stanfit object to the unconstrained scale

Usage

stan_unconstrainsamples(fit, standata = NA)

Arguments

fit

stanfit object.

standata

only necessary if R session has been restarted since fitting model – used to reinitialize stanfit object.

Value

Matrix containing columns of unconstrained parameters for each post-warmup iteration.

Examples

#get data
sunspots<-sunspot.year
sunspots<-sunspots[50: (length(sunspots) - (1988-1924))]
id <- 1
time <- 1749:1924
datalong <- cbind(id, time, sunspots)

#setup model
ssmodel <- ctModel(type='ct', n.latent=2, n.manifest=1, 
  manifestNames='sunspots', 
  latentNames=c('ss_level', 'ss_velocity'),
  LAMBDA=matrix(c( 1, 'ma1| log(1+(exp(param)))' ), nrow=1, ncol=2),
  DRIFT=matrix(c(0, 'a21 | -log(1+exp(param))', 1, 'a22'), nrow=2, ncol=2),
  MANIFESTMEANS=matrix(c('m1|param * 10 + 44'), nrow=1, ncol=1),
  MANIFESTVAR=diag(0,1), #As per original spec
  CINT=matrix(c(0, 0), nrow=2, ncol=1),
  DIFFUSION=matrix(c(0, 0, 0, "diffusion"), ncol=2, nrow=2))

#fit
ssfit <- ctFit(datalong, ssmodel,
  optimize=FALSE, priors=TRUE,
  sampleControl=list(iter=200, chains=2, max_treedepth=4), backend='stan')
umat <- stan_unconstrainsamples(ssfit$stanfit$stanfit)

Adjust standata from ctsem to only use specific subjects

Description

Adjust standata from ctsem to only use specific subjects

Usage

standatact_specificsubjects(standata, subjects, timestep = NA)

Arguments

standata

standata

subjects

vector of subjects

timestep

ignored at present

Value

list of updated structure

Examples

d <- standatact_specificsubjects(ctstantestfit$standata, 1:2)

Optimize / importance sample a stan or ctStan model.

Description

Optimize / importance sample a stan or ctStan model.

Usage

stanoptimis(
  standata,
  sm,
  init = "random",
  initsd = 0.01,
  estonly = FALSE,
  tol = 1e-08,
  stochastic = TRUE,
  priors = TRUE,
  carefulfit = TRUE,
  stallretries = 2,
  stalltol = 0.01,
  uncertainty = c("hessian", "checked", "is", "bootstrap", "fullbootstrap", "sandwich",
    "opg"),
  uncertaintyDraws = "auto",
  uncertaintyControl = list(),
  subsamplesize = 1,
  parsteps = c(),
  plot = FALSE,
  finishsamples = 1000,
  lproughnesstarget = 0.2,
  verbose = 0,
  cores = 2,
  matsetup = NA,
  nsubsets = 1,
  stochasticTolAdjust = 1000,
  g_tol = NULL,
  x_tol = NULL,
  maxiter = NULL,
  lbfgs_memory = NULL
)

Arguments

standata

list object conforming to rstan data standards.

sm

compiled stan model object.

init

vector of unconstrained parameter values, or character string 'random' to initialise with random values very close to zero.

initsd

positive numeric specifying sd of normal distribution governing random sample of init parameters, if init='random' .

estonly

if TRUE,just return point estimates under $rawest subobject.

tol

objective tolerance – the optimizer stops when the objective changes by less than this between evaluations.

stochastic

Logical. Use stochastic gradient descent as main optimizer. Always finishes (double checks) with mize (bfgs) optimizer.

priors

logical. If TRUE, a priors integer is set to 1 (TRUE) in the standata object – only has an effect if the stan model uses this value.

carefulfit

Logical. If TRUE, priors are always used for a rough first pass to obtain starting values when priors=FALSE

stallretries

Integer. Number of times to restart the optimizer from fresh random values when it finishes somewhere that is not a maximum – see stalltol. Only applies when init='random'; with supplied inits a failed fit is reported but not retried. Set to 0 to disable retrying.

stalltol

Gradient per data point above which the optimizer is taken to have stopped short rather than converged. Rough likelihoods – binary indicators are the case this was built for – can leave both optimizers unable to find an improving step, which they report as convergence, returning the starting values as the estimate. Converged fits sit near 1e-5 per data point, stalled ones at 1 or more.

uncertainty

Character string selecting the optimized-fit uncertainty approximation. Options are 'hessian', 'checked', 'is', 'bootstrap', 'fullbootstrap', 'sandwich', and 'opg'.

uncertaintyDraws

Character string controlling approximate raw-parameter draws from the approximate uncertainty. 'auto' uses empirical draws for uncertainty='bootstrap' or uncertainty='fullbootstrap' and normal draws otherwise. 'normal' draws from a multivariate normal using the selected covariance, 'empirical' uses empirical draws when available, and 'imis' runs importance sampling.

uncertaintyControl

List of method-specific options passed to ctOptimUncertainty internals. Score-based methods use subject-level score contributions when there are at least two subjects; single-subject models warn and use case-level contributions. Score-based methods warn when there are fewer than ten independent subjects or no more score rows than raw parameters. Full bootstrap requires at least two subjects and warns below ten independent subjects. Bootstrap-style methods require at least two returned samples / refits.

subsamplesize

value between 0 and 1 representing proportion of subjects to include in first pass fit.

parsteps

ordered list of vectors of integers denoting which parameters should begin fixed at zero, and freed sequentially (by list order). Useful for complex models, e.g. keep all cross couplings fixed to zero as a first step, free them in second step.

plot

Logical. If TRUE, plot iteration details. Probably slower.

finishsamples

Number of samples to draw (either from hessian based covariance or posterior distribution) for final results computation.

lproughnesstarget

target log posterior roughness for stochastic optimizer (suggest between .05 and .4).

verbose

Integer from 0 to 2. Higher values print more information during model fit – for debugging.

cores

Number of cpu cores to use, should be at least 2.

matsetup

subobject of ctStanFit output. If provided, parameter names instead of numbers are output for any problem indications.

nsubsets

number of subsets for stochastic optimizer. Subsets are further split across cores, but each subjects data remains whole – processed by one core in one subset.

stochasticTolAdjust

Multiplier for stochastic optimizer tolerance.

g_tol

gradient tolerance: stop when the l2 norm of the gradient falls below this. NULL (the default) leaves it off, which is what this optimizer has always done – tol is its criterion. The julia backend reads the same name and defaults it to 1e-8 instead.

x_tol

parameter-step tolerance: stop when the update to the parameter vector is smaller than this. NULL leaves it off.

maxiter

maximum optimizer iterations. NULL leaves the existing caps in place (99999 for the bfgs optimizer, 5000 for the stochastic one).

lbfgs_memory

number of curvature pairs L-BFGS keeps. NULL uses 100.

Value

list containing fit elements


Runs stan, and plots sampling information while sampling.

Description

Runs stan, and plots sampling information while sampling.

Usage

stanWplot(object, iter = 2000, chains = 4, ...)

Arguments

object

stan model object

iter

Number of iterations

chains

Number of chains

...

All the other regular arguments to stan()

Details

On windows, requires Rtools installed and able to be found by pkgbuild::rtools_path()

Value

The stanfit object rstan::sampling returns.

Examples

library(rstan)
#### example 1 
scode <- "
parameters {
  real y[2]; 
} 
model {
  y[1] ~ normal(0, .5);
  y[2] ~ double_exponential(0, 2);
} 
"
#Uncomment the following lines -- launches rscript not compatible with cran check.
#sm <- stan_model(model_code = scode)
#fit1 <- stanWplot(object = sm,iter = 100000,chains=2,cores=1)

Summarise discrete time effect curves

Description

For each curve ctDiscretePars returned – one per subject and pair of processes – the quantities read off it, per draw, and their distribution over draws.

Usage

## S3 method for class 'ctDiscretePars'
summary(object, probs = c(0.025, 0.5, 0.975), ...)

Arguments

object

Output of ctDiscretePars with plot = FALSE.

probs

Quantiles of each quantity over draws.

...

Not used.

Value

A data frame, one row per subject, pair of processes (row the one affected, col the one acting) and quantity, with the mean, sd and quantiles over draws, and where the object carries the fit's estimate (an optimised fit's population effects, attr(object, 'estimate')), the quantity read off the estimate's curve, estimate. The quantities:

peak

The value of largest magnitude, with its sign.

peaktime

The lag at which it occurs.

halftime

The first lag after the peak at which the curve's magnitude has fallen to half the peak's, interpolated between lags. For an autoregression, which peaks at lag zero, its half-life: with impulseType = 'observed', standardise = TRUE, that of the autocorrelation. NA for a draw whose curve does not fall that far within the lags computed; beyond is the share of such draws, and the summary is over the others.

area

The area under the curve over the lags computed: for a unit impulse over long enough lags, the cumulative effect.

The lags are those ctDiscretePars was given, so their spacing bounds the precision of peaktime and halftime, and their range what halftime and area can see.

Examples

dp <- ctDiscretePars(ctstantestfit, times = seq(0, 5, .1), nsamples = 20)
summary(dp)

Summarise empirical Bayes subject-wise ctsem fits

Description

Summarise empirical Bayes subject-wise ctsem fits

Usage

## S3 method for class 'ctEmpiricalBayesFit'
summary(
  object,
  use = c("rawest", "rawposterior"),
  probs = c(0.025, 0.5, 0.975),
  sdscale = c("unit", "rawsd"),
  minsd = 1e-06,
  digits = 4,
  ...
)

Arguments

object

Object returned by ctEmpiricalBayesFit.

use

'rawest' to summarise final-pass subject point estimates, or 'rawposterior' to pool final-pass subject raw posterior samples.

probs

Quantiles to report for transformed parameters.

sdscale

How to set model$pars$sdscale when reconstructing the adjusted single-subject empirical Bayes model from the final empirical raw distribution. 'unit' keeps any later random-effect SDs on the EB-standardised raw scale. 'rawsd' uses the final empirical raw SDs directly. Retained for compatibility; summary() no longer returns a reconstructed model.

minsd

Lower bound used for empirical raw SDs before model adjustment.

digits

Number of digits for printed summary tables.

...

Unused.

Value

Compact list containing fit settings, transformed-parameter summaries, outlier diagnostics, transformed-parameter covariance / correlation matrices, and subjectpars: each subject's final-pass point estimates on the parameters' own scale, one row per subject with its id in subject, unrounded, for comparisons across subjects. The subject fit lists, raw estimate matrices, pass maps, and adjusted model remain on the original ctEmpiricalBayesFit object.


Summarise a ctsem fit

Description

Summarise a fit from ctFit, on either backend: population parameters with their uncertainty, random-effect spreads and correlations, time independent predictor effects, the parameter matrices, and the log likelihood and AIC.

Usage

## S3 method for class 'ctJuliaFit'
summary(
  object,
  timeinterval = 1,
  digits = 3,
  parmatrices = TRUE,
  priorcheck = TRUE,
  residualcov = TRUE,
  ...
)

## S3 method for class 'ctStanFit'
summary(
  object,
  timeinterval = 1,
  digits = 3,
  parmatrices = TRUE,
  priorcheck = TRUE,
  residualcov = TRUE,
  ...
)

Arguments

object

fit object from ctFit, of class ctStanFit or ctJuliaFit.

timeinterval

positive numeric indicating time interval to use for discrete time parameter calculations reported in summary.

digits

integer denoting number of digits to report.

parmatrices

if TRUE, also return additional parameter matrices – can be slow to compute for large models with many samples.

priorcheck

Whether or not to use priorchecking to compare posterior mean and sd to prior mean and sd. backend='stan' only: a julia fit carries no prior block to compare against, and refuses priorcheck = TRUE given explicitly.

residualcov

Whether or not to show standardised residual covariance. Takes a little longer to compute.

...

Additional arguments to pass to priorcheckreport, such as meanlim, or sdlim.

Value

List containing summary items, with a print method for readable console and knitr output. On an optimised backend = 'julia' fit with uncertainty draws, the parameter tables and parmatrices carry a mode column beside the draws' mean, sd and quantiles: the fit's estimate on the same scale, the point a likelihood ratio or an information criterion refers to.

popmeans reports each free parameter. A T0VAR, DIFFUSION or MANIFESTVAR parameter is a standard deviation on the diagonal; off it, a coordinate the model's covmattransform turns into correlations, not a correlation: under the default 'z' it is an entry of the correlation matrix's logarithm, which for two variables is the correlation's Fisher z (tanh() of it is the correlation) and for more is no function of any one correlation. The correlations are in parmatrices, as T0cor, DIFFUSIONcor, MANIFESTcor and asymDIFFUSIONcor beside the covariances, each summarised over the draws' own correlations.

Examples

summary(ctstantestfit)

Tests if 2 values are close to each other

Description

Tests if 2 values are close to each other

Usage

test_isclose(..., tol = 1e-08)

Arguments

...

values to compare. At least two, and none of them empty: comparing a zero-length or NULL value is an error rather than a pass, because there is nothing to compare.

tol

tolerance

Value

Logical or testthat output.

Examples

test_isclose(1,1.0000001, tol=1e-4)