Peter's Homepage on Peter's Homepage

Add Interactions to Regularized Regression

Sat, 13 Oct 2018 00:00:00 +0000

Lasso & glinternet

Every Data Scientist and her dog know linear and logistic regression. The majority will probably also know that these models have regularized versions, which increase predictive performance by reducing variance (at the cost of a small increase in bias). Choosing L1-regularization (Lasso) even gets you variable selection for free. The theory behind these models is covered expertly in The Elements of Statistical Learning (for an easier version, see An Introduction to Statistical Learning), and implemented nicely in the packages glmnet for R and scikitlearn for Python.

Few people, however, have heard about glinternet, which is a powerful extension of the Lasso model:

glinternet adds pairwise variable interactions into the Lasso model, and selects these automatically.

It’s a linear model, and thus easily interpretable. Its computational complexity is the same as penalized regression: $\mathcal O(Nm)$ where there are $N$ rows and $m$ variables¹. Thus it handles problems of the same size as (deep) neural networks, potentially with tens of thousands of variables. It should be stressed here that for a thousand variables there are almost 500,000 pairwise interactions that glinternet checks! For structured data, logistic regression is a very useful benchmark model, which can be on par with sophisticated deep learning models even for huge datasets. The glinternet model is more flexible than logistic regression, and is just as interpretable.

A glinternet tutorial

Let’s look at the car90 dataset in the rpart package. It has 25 categorical and 8 continuous predictors, and a continuous outcome (Price). We fit a glinternet model to it, which is a linear model containing all possible pairwise interactions.

Interactions

A linear model without interactions is usually written as

\[y = \beta_0 + \sum_k \beta_k X_k + \varepsilon\]

where $\beta_0$ is the intercept, $X_k$ are predictor variables, $\beta_k$ the coefficients and $\varepsilon$ a model error. We will use categorical variables, so we need to group the variables and coefficients with their categories:

\[y = \beta_0 + \sum_{i=1}^m \sum_{l=1}^{L_i} \beta_{i,l} X_{i,l} + \varepsilon\] where

$m$ is the number of predictors $X_1, \ldots, X_m$
$L_i$ is the number of levels of $X_i$ if $X_i$ is categorical, and 1 otherwise
$(X_{i,1}, \ldots, X_{i,L_i})$ are 0-1 dummy variables representing the categories of $X_i$ (one-hot encoded)

A pairwise interaction model contains all possible pairwise products of the predictors:

\[y = \beta_0 + \sum_{l=1}^{L_i} \sum_{i=1}^m \beta_{i,l}, X_{i,l} + \sum_{1 \le i<j \le m} \sum_{l=1}^{L_i} \sum_{k=1}^{L_j} \theta_{i,j,l,k} X_{i,l} X_{j,k}\]

Interactions, e.g. between variables $X_i$ and $X_j$, occur when the effect of $X_j$ on $y$ will vary depending on the level (or value) of $X_i$. In that case, some of the values $\theta_{i,j, \cdot, \cdot}$ will be non-zero.

Selection of variables and interactions

The L1 regularization is known as the lasso and produces sparsity. glinternet uses a group lasso for the variables and variable interactions, which introduces the following strong hierarchy: An interaction between $X_i$ and $X_j$ can only be picked by the model if both $X_i$ and $X_j$ are also picked. In other words, interactions between two predictors are not considered unless both predictors have non-zero coefficients in the model. The details are in the original paper.

Fitting glinternet

Unfortunately, glinternet’s “API” is not super user friendly… the first challenge is to bring the data into a form that glinternet accepts: The categorical variables need to be (somewhat un-R-like) coded as integers starting from 0. We also need to construct a numLevels vector containing the number of levels $L_1, \ldots, L_m$ in each column if the column is categorical, and 1 else:

library(dplyr)

# Model2 contains model names, which aren't useful here
df <- rpart::car90 %>% select(-Model2)

# drop rows with empty outcomes
df <- df[!is.na(df$Price), ]
y <- df$Price
df <- df %>% select(-Price)

# impute the median for the continuous variables
i_num <- sapply(df, is.numeric)
df[, i_num] <- apply(df[, i_num], 2, function(x) ifelse(is.na(x), median(x, na.rm=T), x))

# impute empty categories
df[, !i_num] <- apply(df[, !i_num], 2, function(x) {
  x[x==""] <- "empty"
  x[is.na(x)] <- "missing"
  x
})

# get the numLevels vector containing the number of categories
X <- df
X[, !i_num] <- apply(X[, !i_num], 2, factor) %>% as.data.frame()
numLevels <- X %>% sapply(nlevels)
numLevels[numLevels==0] <- 1

# make the categorical variables take integer values starting from 0
X[, !i_num] <- apply(X[, !i_num], 2, function(col) as.integer(as.factor(col)) - 1)

A single glinternet model is fitted with the function glinternet; here, we directly fit a 10-fold Cross-Validated model:

library(glinternet)
set.seed(1001)
cv_fit <- glinternet.cv(X, y, numLevels)
plot(cv_fit)

We see the MSE plotted against the decreasing values of regularization parameter lambda.

i_1Std <- which(cv_fit$lambdaHat1Std == cv_fit$lambda)

It is common not to pick the lambda corresponding to the lowest estimate of cross-validation error, but to err on the side of more regularization. So we are happy with a larger lambda with CV error estimate below the minimum plus one standard deviation. This lambda has index i_1Std = 22. Its coefficients $\beta_{i,l}$ and $\theta_{i,j,l,k}$ can be extracted by running

coefs <- coef(cv_fit$glinternetFit)[[i_1Std]]

coefs is a list of 4 items: mainEffects, mainEffectsCoef, interactions and interactionsCoef.

Let’s look at the part without interactions first: The main effects identified are

coefs$mainEffects

## $cat
## [1] 3
## 
## $cont
## [1]  5  7  9 12 20 22 11

which are the indices of the categorical variables and the continuous variables, in order as they occur in the data. We get the indices via

idx_num <- (1:length(i_num))[i_num]
idx_cat <- (1:length(i_num))[!i_num]
names(numLevels)[idx_cat[coefs$mainEffects$cat]]

## [1] "Rim"

names(numLevels)[idx_num[coefs$mainEffects$cont]]

## [1] "Frt.Leg.Room" "Gear.Ratio"   "HP"           "Length"      
## [5] "Tank"         "Weight"       "Height"

The coefficients for these variables are:

coefs$mainEffectsCoef

## $cat
## $cat[[1]]
## [1]    357.6320  -8221.4669   6589.1535 -36595.3391    118.0856   -920.4584
## 
## 
## $cont
## $cont[[1]]
## [1] 35.48628
## 
## $cont[[2]]
## [1] 314.3327
## 
## $cont[[3]]
## [1] -477.2347
## 
## $cont[[4]]
## [1] 9.059015
## 
## $cont[[5]]
## [1] 507.8227
## 
## $cont[[6]]
## [1] 1.366189
## 
## $cont[[7]]
## [1] 104.6502

(I would have expected all of these to be positive; one should check why HP has a negative coefficient.)

Now for the interaction pairs:

coefs$interactions

## $catcat
## NULL
## 
## $contcont
##      [,1] [,2]
## [1,]    5    9
## [2,]   11   20
## 
## $catcont
##      [,1] [,2]
## [1,]    3    5

The above means:

There is no interaction between the categorical variables (unsurprising because we only identified one. If there was an interaction pair, we would access it via coefs$interactionsCoef$catcat, giving a list of matrices, one matrix of dimension $L_i \times L_j$ for each pair.)
Two pairs of continuous variables have interactions: (5,9) and (11,20), which are (Frt.Leg.Room, HP) and (Height, Tank). The coefficients for these interactions are

coefs$interactionsCoef$contcont

## [[1]]
## [1] 13.66277
## 
## [[2]]
## [1] -6.634724

There is one interaction between categorical variable 3 and continuous variable 5, which are Rim and Frt.Leg.Room. The coefficients for this interaction are:

coefs$interactionsCoef$catcont

## [[1]]
## [1] -164.0751   37.7502 -324.6745  741.9752 -158.0835 -132.8923

Our simple interaction network looks like this:

The model has a root mean squared error (RMSE) on validation data of

sqrt(cv_fit$cvErr[[i_1Std]])

## [1] 3411.904

Compare to linear regression

To quantify how much interactions have contributed to predictive power, we now fit a glmnet model. The best CV mean-squared error for the glmnet model is higher than the mean-squared error for glinternet:

df$Price <- y
X <- model.matrix(Price ~ . - 1, df)
library(glmnet)
cv_glmnet <- cv.glmnet(X,y)
sqrt(min(cv_glmnet$cvm))

## [1] 4238.202

Take-home message

When you’re using logistic or linear regression with Lasso regularization as a baseline, and your next step is to see if adding interactions improve predictive power: try glinternet!

References

Learning Interactions via Hierarchical Group-Lasso Regularization. Michael Lim & Trevor Hastie

Though for this to be true, I believe you have to limit the “interaction search space” via the screenLimit parameter.↩

Variable Order Fractional Fokker-Planck Equations derived from Continuous Time Random Walks

Wed, 01 Aug 2018 00:00:00 +1000

Identification of pollutant source for super-diffusion in aquifers and rivers with bounded domains

Tue, 31 Jul 2018 00:00:00 +1000

Overcoming the data drought: exploring general practice in Australia by network analysis of big data

Sun, 29 Jul 2018 00:00:00 +1000

See my blog post for this publication.

Big Networks in Healthcare

Mon, 16 Jul 2018 00:00:00 +0000

See also the MJA podcast episode accompanying this article.

Our joint work (UNSW CBDRH and Statistics) which analyses Australian patient claim data using big network algorithms is now available on the MJA website. We have processed MBS claims data of 10% of Australians over the years 1994-2014, trying to shed light on the following research questions:

What is the patient sharing behaviour of general practitioners (GPs): are there any meaningful clusters (called “Provider Practice Communities, PPC”) of GPs which collaborate and share patients? How have these clusters changed in the course of 20 years?
How does Continuity of Care (CoC¹) differ between large and small PPCs?

Datasets such as provided by BEACH (Bettering the Evaluation And Care of Health, USYD) would be ideal to tackle these questions, but were discontinued due to lack of funding, and it may take several years until Australia’s My Health Record is adopted by enough Australians to generate useful datasets.

Question 1 really is about the structure of information sharing in the Australian health system. We believe we’ve found evidence of information sharing within clusters of GPs (PPCs) which resemble the typical sizes of practices of GPs. CoC is a proxy for the sharing of important patient information sharing, which is believed to lead to better health outcomes; but information sharing can also mean that GPs have similar behaviour when referring to specialists or pathology services, among which there may be low-value services such as unwarranted surgical interventions and blood tests.

Based on the PPCs which we have identified, it is now possible to discover unwarranted variation of good and bad outcomes on the level of PPCs (practices). There are many interesting questions that can be tackled now, and as a first step into this direction, we redefined “Continuity of Care” on the PPC level, creating an indicator which is high if a patient is loyal to the same practice, rather than the same GP. It seems that the larger a practice, the less loyal patients are to individual GPs; but if loyalty to the same practice is measured, small and large practices seem to fare alike.

Big Networks

The usual approach in network clustering (which was seemingly taken by all previous papers on this topic in the medical literature) is as follows:

Create a network of GPs only², where patients only appear as weights on the edges (i.e. project from bipartite to unipartite): (source: Wikipedia)
Cluster the GP nodes by maximizing³ modularity: (source: sciencedirect)

The main flaws with this approach are:

There is no good way to construct edge weights between GPs.⁴
The resolution limit problem of modularity: The minimum size of discoverable communities grows with the number of nodes $N$ in the network, roughly like $\sqrt N$. Since we study thousands of GPs, this problem is prohibitive.
Modularity assumes that all block structure is assortative, which can be dangerous; moreover even completely random networks can have high modularity.

Using the Python package graph-tool, we were able to:

Avoid the bipartite projection, even though this meant dealing with millions of nodes (patients and doctors) instead of thousands (doctors only).
Fit a hierarchical stochastic blockmodel, which allows detecting much smaller communities. Moreover, the block structure respected bipartiteness, meaning that each block was made up of either doctors only or patients only.

Computations

The main challenges were:

Installing graph-tool: pip and conda are not an option, you have to compile it from source, or install the native Linux / MacOS X packages.
RAM usage: graph-tool needed up to 64GB of RAM. We’ve used the Katana computational cluster at UNSW and got excellent support along the way.
Numerous runs: The hierarchical blockmodelling algorithm converges to a different solution every time; to be somewhat confident to have found a good minimum of the description length, we needed 4 runs of up to 6 hours each, for each of 5 regions.

GitHub Repository

The full Python code used for this project, as well as more details on the blockmodelling approach, can be found on the CBDRH GitHub page.

Conclusion

Using hierarchical stochastic blockmodelling, it is possible to use data which is routinely collected by GPs to monitor characteristics of Australian general practices. Future research will clarify how these characteristics affect patient care.

A patient typically seeing the same GP, or GP clinic, experiences high CoC: $${\rm CoC} = \frac{\max \left\lbrace n_k \right \rbrace}{\sum_k n_k}$$ where $n_k$ are the numbers of visits made to different PPCs.
^{^}
In the graph shown, suppose X are the patients and Y the doctors; then the Y-projection is a network of doctors only, with edge weights computed from the number of shared patients.
^{^}
The picture shows the adjacency matrix, where each row and column corresponds to one node, and a black dot means there is an edge. The red squares represent a grouping of nodes. Modularity, in a nutshell, is the gain in the number of within-group edges achieved by the grouping, compared to the expected number if all edges were reassigned at random.
^{^}
If you go by the number of shared patients, this discards all patients which have only seen one GP, and makes it impossible to find single GPs. If you go by Jaccard index, this leads to weights between 0 and 1, which is not useful for most community detection algorithms.
^{^}

NUS-MIT Datathon

Tue, 10 Jul 2018 00:00:00 +0000

Last week I had the privilege to participate in the NUS-NUH-MIT DATATHON and Workshop on applications of AI in healthcare with the UNSW Centre for Big Data Research in Health (CBDRH) team (Tim Churches, Mark Hanly, Oisin Fitzgerald and Oluwadamisola Sotade).

Thu & Fri: Workshop & Talks

In the workshop “Deploying AI Solutions in Real Clinical Practices” by Dr Ngiam Kee Yuan (CTO, NUHS) we discussed

The large NUHS (National University Health System) databases and their storage structure
Data security and ownership
Applications for access to data
The always changing standards of diagnosis codes (ICD9, ICD10, SNOMED, …) and the problem of matching doctors diagnoses to these codes. In another talk, Hector Yee (@eigenhector) shared some vice ideas on how to use word embeddings as a tool for matching hand-typed diagnoses to ICD10 codes.
The importance of getting doctors to trust AI algorithms. Hector presented an interesting approach which iterates between an AI algorithm proposing codes and a doctor validating these. The goal is of course to augment the doctors, not to replace them.

Fri - Sun: Datathon

Friday 6pm: Pitching Topics & Forming Teams

23 clinicians pitch topics. We were most interested in a topic presented by Dr Lui Pak Ling (Hematology) and Dr Stella Ang (Anesthesiology) at NUH:

Currently-used statistical models predicting the mortality risk of Intensive Care Unit (ICU) patients assume that the importance of prognostic factors does not vary

Research Database, a large multi-centre critical care database made available by Philips Healthcare in partnership with the MIT Laboratory for Computational Physiology, they investigated whether the importance of prognostic factors varied in the course of each patient’s stay in ICU. Currently-used predictive models assume that they don’t vary.

Currently, the mortality risk of ICU (intensive care unit) patients is assessed based on a collection of variables at the time they are presented to the ICU (the so-called Apache score). This score is assumed to be the same during the whole course of stay in the ICU. Is this assumption appropriate, or does the importance of some prognostic variables in fact change over time?

We joined forces with Siqi Liu, a PhD student at Saw Swee Hock School of Public Health and a member of the organising team for the Datathon who represented our team, and Peng Shen (MOH Holdings Pte Ltd).

All of Saturday: Extracting Data

We used the eICU Collaborative Research Database. The organisers have made the data available via Google’s BigQuery, which was very quick and (almost) 100% reliable. Working in pairs we extracted and cleaned various tables of variables that we used as predictors: comorbidity history, lab results, vitalperiodic and vitalaperiodic, coma score, infusion drugs, dialysis, blood culture, mechanical ventilation, …

We queried the database using

the BigQuery web interface,
the Python API and colab notebooks, or
the dbplyr package in R.

Sunday morning: More wrangling

Joining all the extracted tables into one big table. The final SQL query was 306 lines long and extracted of 161,988 rows (a cohort of sepsis patients) and 93 transformed and cleaned variables (categorical and continuous) from tables with 224 million rows with 4-51 columns.

Sunday 12pm-3pm:

Model fitting

As a baseline, we used logistic regression, with L1 penalty for variable selection (using glmnet in R). Virtually all rows had missing data, so we need to impute by median. This didn’t seem to work out very well, as e.g. the variables picked at after 2 days in ICU had almost nothing in common with the variables picked for 3 or 4 days in ICU.

Gradient Boosting (using catboost in R) seemed to work stably with the irregularly spaced, high-dimensional data with many missing values. In our preliminary analysis, it seems that “age” may increase in importance and “SaO2” (oxygen saturation) may decrease in importance.

Presentation writing:

Teams were judged on a three minute presentation only, and the time limit was strictly enforced (our MC made the audience start clapping after exactly 180 seconds). So the presentation had to be practiced to near perfection, which Dami and Siqi succeeded in doing: The judges placed us first in the track “AI for critical care” 🎉🎉🎉

Work to be continued

So far, we have fitted individual models to each point in time (day1, day2, …, day5). This is of course not the ideal way to deal with a longitudinal signal. One ought to try

The takeaway

Our team is walking away with a preliminary analysis on a cleaned dataset which is likely to result in a publication, and 2 domain experts and 1 other data scientist who are invested in continuing our work. I would say these two days were incredibly well spent!

PhD internship in Machine Learning

Tue, 29 May 2018 00:00:00 +0000

In the internship, you’ll explore the latest machine learning methods such as tree ensemble methods, graphical models and neural networks and compare their performance to what is the current industry standard. A great opportunity to improve your machine learning skills and create valuable insights for the credit risk industry.

Applications close 20 June, 2018.

Apply here: https://aprintern.org.au/2018/05/21/int-0431/

Networks of Health Providers

Mon, 14 May 2018 00:00:00 +1000

What is the network structure of healthcare providers and their practices/clinics?
What is their patient sharing pattern?
Does sharing of patient information between providers lead to better health outcomes?
Do practice size and patient sharing behaviour influence the quality of healthcare?

Australian Medicare claims data contains data¹ on tens of thousands of healthcare providers and millions of patients, such as patient characteristics (age, sex, …) and provider characteristics (specialty, type and frequency of services delivered, bulk-billing behaviour,
…).

The natural structure of these data is a network, where providers & patients are connected whenever a service is provided. The structure of this network contains crucial information on the flow of information and the variation of healthcare quality, and thus allows to tackle questions of national importance such as the above.

de-identified, of course. ^{^}

R package CTRE

Mon, 14 May 2018 00:00:00 +1000

R package CTRE: thresholding bursty time series

Fri, 11 May 2018 00:00:00 +0000

The new R package CTRE is now available on CRAN. “CTRE” stands for “Continuous Time Random Exceedances”, which is a model for extreme values of bursty time series. The theory is desribed in another paper (html|pdf).

library(CTRE)
flares_ctre <- ctre(flares)
plot(flares_ctre, log = "y", main = "Solar Flares")

The above time series are measurements of solar flare magnitudes by NASA¹. Which observations are considered extreme is defined by a variable threshold $u$ (dashed line, $< 5\%$ of observations). The exceedances of this threshold (red) follow (asymptotically for high $u$) a Generalized Pareto distribution. This type of modelling of extremes is standard in Extreme Value Theory, and known as the Peaks Over Threshold (POT) method. To date, 18 R packages have been developed for POT!

What’s new about `CTRE`?

CTRE provides a new way to model the bursty timing of the threshold crossings. In complex systems, inter-event times are typically heavy-tailed², and thus highly heterogeneous (think income distributions). The bulk of inter-event times is short, corresponding to Bursts of events. A few very long inter-event times represent quiet periods.

The Mittag-Leffler distribution is universal for times between threshold crossings, see the companion paper. CTRE fits a Mittag-Leffler distribution to a range of threshold values:

flares_ctre <- thin(flares_ctre, k = 700)
par(mfrow = c(1,2))
MLestimates(flares_ctre, tail = 0.9, scale = 3E7)

Above, the threshold is varied from the 5th to the 700th largest magnitude (as you move right along the x-axes) and the parameters estimates are plotted on the y-axis. For both panels, towards the left (exceedances=5), only few data are available, and the estimate has a high variance. Towards the right (exceedances = 700), the threshold is low, and the Mittag-Leffler fit degrades (high bias). In between is where we want to read off our (hopefully stable) parameter estimates. The above plots are hence called “stability plots”.

Important:: The scale parameter is proportional to $k^{-1/\beta}$. The parameter $\beta$ is hence used to rescale the y-values of the right-hand plot, in order to make the estimates constant, and needs to estimated from the left-hand plot first.

`CTRE` applied to data

For the solar flare data, we read off \[ \beta \approx 0.9, \quad \sigma \approx 3 \times 10^7. \] This means that the inter-arrival time $T(k)$ for magnitudes as high as the $k$-th magnitude is distributed as \[ T(k) \sim {\rm ML}\left(0.9, \frac{3 \times 10^7 {\rm sec}}{k^{1/0.9}} \right). \] For example, let’s assume the magnitude (peak rate) $10,000$. How many magnitudes exceed it?

library(magrittr)
(flares_ctre %>% magnitudes() %>% sort(decreasing=TRUE) > 10000) %>% sum()

## [1] 24

So substitute $k = 24$. The model thus predicts inter-arrival times drawn from the distribution \[ {\rm ML}\left(0.9, 8.8\times 10^{5} {\rm sec} \right) = {\rm ML}\left(0.9, 10 {\rm days} \right). \]

Here’s the plot of this distribution, on a logarithmic scale:

Compared to the Exponential distribution, the Mittag-Leffler distribution puts more probability mass on both tiny values (Bursts) and values in the tails (quiet periods).

Shiny app

Running

CTRE::runCTREshiny()

opens a Shiny app that that lets you do the above manipulations via sliders. It is / should be loading below:

What about “clustering of extremes”?

The main other appraoch in Extreme Value Theory which addresses irregular (non-Poissonian) threshold crossing times goes under the name “clustering of extremes”.³ It assumes that the underlying process is stationary, and conflates each excursion of the process above the threshold as one cluster of events.

CTREs and clustering of extremes both generalize the standard assumption of independent events, but in different directions. They are hence complementary, and it needs to be investigated how they should be compared meaningfully.

Feedback

Do you think CTREs are a useful model? Please comment below.

The “complete Hard X Ray Burst Spectrometer event list” is a comprehensive reference for all measurements of the Hard X Ray Burst Spectrometer on NASA’s Solar Maximum Mission from the time of launch on Feb 14, 1980 to the end of the mission in Dec 1989. 12,776 events were detected, with the “vast majority being solar flares”. The list includes the start time, peak time, duration, and peak rate of each event. We have used “start time” as the variable for event times, and “peak rate” as the variable for event magnitudes.↩
See the Introduction of “Modelling bursty time series” by Vajna, Tóth & Kertész, for a list of complex systems with heavy-tailed inter-event times.↩
See e.g. Chapter 3 in the textbook Extremes and Related Properties of Random Sequences and Processes by Leadbetter, Lindgren & Rootzén.↩

A Universal Algorithm for Continuous Time Random Walks Limit Distributions

Sat, 28 Apr 2018 00:00:00 +1000

Peaks Over Threshold for Bursty Time Series

Sat, 28 Apr 2018 00:00:00 +1000

Fractional Diffusion where "fractional" varies in space

Mon, 18 Dec 2017 00:00:00 +0000

The new paper (arxiv.org/abs/1712.06767)

Variable Order Fractional Fokker-Planck Equations derived from Continuous Time Random Walks

is currently under review.

The variable order fractional Fokker-Planck equation (VOFFPE), in simple form¹, is \[\frac{\partial}{\partial t} p(x,t) = \frac{\partial^2}{\partial x^2} \underbrace{D_t^{1-\beta(x)}}_{\text{fractional derivative}} p(x,t).\]

If you take out the fractional derivative operator, you get the Fokker-Planck equation (FPE). It describes the spreading of probability via a random walk:

Adding in the fractional derivative \[D_t^{1-\beta} p(x,t) = \int_0^t p(x,t-s) \frac{s^{-\beta}}{\Gamma(1-\beta)}\,ds\] results in the “fractional FPE”. The fractional derivative corresponds to longs rests of a walker, and slows down the spreading (making the dynamics “sub”diffusive).
Here the order $1-\beta$ of the fractional derivative varies with the location $x$ of the walker, hence the name VOFFPE (variable order fractional FPE). In some regions, e.g. where the intracellular matrix of a cell is dense², the trappingness can be stronger, which is modelled by a smaller $\beta(x)$ in that region.

The paper shows that the equation has a unique solution $p(x,t)$, and that this solution is the probability distribution of a Continuous Time Random Walk (CTRW)³. CTRWs are random walks with random waiting times between jumps:

The fractional derivative in time is tightly linked with Mittag-Leffler distributed waiting times.

Of course CTRWs can be simulated, and $p(x,t)$ may be computed via Monte Carlo simulations (or “particle tracking”). Thus a consequence of the paper is the mathematical certainty that the variable order FFPE (VOFFPE) can be solved by simulations.

The variable order FFPE can also be solved directly, via clever numerical algorithms. The CTRW interpretation of the VOFFPE can be leveraged for such a clever numerical algorithm, as our friends over in applied maths have done with their Discrete Time Random Walk algorithm. This approach has several great benefits⁴, and now we know how to extend these benefits to the variable order setting (to be continued).

Drift and diffusivity are left out here, but they may be any (Lipschitz-) continuous functions.↩
Wong et. al, PRL 2004 ↩
Strictly speaking, a scaling limit of a CTRW.↩
E.g. conservation of mass and non-negative solutions.↩

The Network Structure of General Practice in Australia

Mon, 27 Nov 2017 08:00:00 +1100

R package MittagLeffleR: Using the Mittag-Leffler distributions in R

Mon, 26 Jun 2017 00:00:00 +0000

Ricky Gill and I have created the R package MittagLeffleR, documented at strakaps.github.io/MittagLeffleR/. It is available on CRAN (and the latest development version on GitHub). MittagLeffleR calculates both Mittag-Leffler distributions; i.e. probability density, cumulative distribution function, quantile function and fast random variate generation. It was made possible by the Laplace inversion algorithm for the Mittag-Leffler function by Roberto Garrappa. Install it via

install.packages("MittagLeffleR")

The Two Mittag-Leffler distributions

Various journal articles (including my own) write about “the” Mittag-Leffler distribution, but actually there are two such families of distributions. Both have a shape parameter (tail) taken from the interval $(0,1]$, but the first distribution is very heavy-tailed with infinite mean if tail $\in (0,1)$ (it is equal to the exponential distribution if tail $=1$) whereas the second distribution is light-tailed (has finite moments of all orders). The boolean variable second.type switches between these two distributions.

Examples

First type = waiting times between events

The first type Mittag-Leffler distribution commonly occurs for the inter-arrival times between events in complex systems (e.g. earthquakes, see future post). Suppose that for an event to happen, several “tries” are needed, and that each try has a duration. The number $N$ of tries until a success is geometrically distributed, and the sum of $N$ durations is well approximated by a Mittag-Leffler (type I) distribution (this is the geometric stable property).

As an example, we simulate n=100 Mittag-Leffler (type I) random variables, and estimate their tail and scale parameters via maximum likelihood:

library(MittagLeffleR)
y = rml(n = 100, tail = 0.9, scale = 2)
mlml <- function(X) {
  log_l <- function(theta) {
    #transform parameters so can do optimization unconstrained
    theta[1] <- 1/(1+exp(-theta[1]))
    theta[2] <- exp(theta[2])
    -sum(log(dml(X,theta[1],theta[2])))
  }
  ml_theta <- stats::optim(c(0.5,0.5), fn=log_l)$par
  #transform back
  ml_a <- 1/(1 + exp(-ml_theta[1]))
  ml_d <- exp(ml_theta[2])
  return(list("tail" = ml_a, "scale" = ml_d))
}
mlml(y)

## $tail
## [1] 0.9390633
## 
## $scale
## [1] 1.600294

Second type = number of events

The second type Mittag-Leffler distribution typically occurs for the random number of events in complex systems. For instance, assume a random walk with heavy-tailed waiting times; this has become a widely adopted model for anomalous diffusion. For late times, its number of steps can be well approximated by a Mittag-Leffler (type II) distribution.

The time-change of a random walk by the random number of steps is called subordination; the random number of steps is approximated by the inverse stable subordinator. Below we calculate the probability density of a random walker with step size dx=0.01:

the number of steps up to time $t$ is stored in the vector $h$
the matrix $N$ stores the probability distribution at the lattice sites at consecutive times
the “subordination” happens in the matrix product N %*% h.

library(MittagLeffleR)
tail <- 0.65
dx <- 0.01
x <- seq(-2,5,dx)
umax <- qml(p=0.99, tail=tail, scale=1, second.type=TRUE)
u <- seq(0.01,umax,dx)
h <- dml(x=u, tail=tail, scale=1, second.type=TRUE)
N <- outer(x,u,function(x,u){dnorm(x=x, mean=u, sd=sqrt(u))})
p <- N %*% h * dx
plot(x,p, type='l', main="Fractional diffusion with drift at t=1")

Bugs and Issues

Should you run into any bugs or issues, please let us know. Thanks!