Purpose

forecastML logo

The purpose of forecastML is to provide a series of functions and visualizations that simplify the process of multi-step-ahead direct forecasting with standard machine learning algorithms. It’s aimed at helping the user quickly assess the (a) accuracy, (b) stability, and (c) generalizability of single-outcome forecasts produced from potentially high-dimensional modeling datasets.

This package is inspired by Bergmeir, Hyndman, and Koo’s 2018 paper A note on the validity of cross-validation for evaluating autoregressive time series prediction. In particular, forecastML makes use of

lagged predictors and
nested cross-validation with (a) user-specified standard cross-validation in the inner loop and (b) block-contiguous validation datasets in the outer loop

to build and evaluate high-dimensional forecast models without having to use methods that are time-series specific.

The following quote from Bergmeir et al.’s article nicely sums up the aim of this package:

“When purely (non-linear, nonparametric) autoregressive methods are applied to forecasting problems, as is often the case (e.g., when using Machine Learning methods), the aforementioned problems of CV are largely irrelevant, and CV can and should be used without modification, as in the independent case.”

Direct forecasting

In contrast to the recursive or iterated method for producing multi-step-ahead forecasts used in traditional forecasting methods like ARIMA, direct forecasting involves creating a series of distinct horizon-specific models. Though several hybrid methods exist for producing multi-step forecasts, the simple direct forecasting method with lagged features used in forecastML let’s us avoid the exponentially more difficult problem of having to “predict the predictors” for forecast horizons beyond 1-step-ahead.

Below are some resources for learning more about multi-step forecasting strategies:

The animation below shows how historical data is used to create a 1-to-12-step-ahead forecast for a 12-step-horizon forecast model using lagged features. Though predictor lags greater than 12 steps can be used to make use of additional historical predictive information, a 12-step-horizon direct forecast model requires predictor lags >= 12. This animation is roughly equivalent to how a 12-period seasonal ARIMA(0, 0, 0)(1, 0, 0) model uses historical data to produce forecasts.

Package Features

Transform datasets for modeling by creating various patterns of lagged predictors for user-specified forecast horizons with forecastML::create_lagged_df()
Create datasets for evaluating forecast models using nested cross-validation with forecastML::create_windows()
Train and evaluate machine learning models for forecasting with forecastML::train_model().
Assess forecast accuracy at different forecast horizons with forecastML::return_error().
Assess hyperparameter stability with forecastML::return_hyper().
Create datasets of lagged predictors for direct forecasting.

Example

In this walkthrough of forecastML we’ll compare the forecast performance of two machine learning methods, LASSO and Random Forest, across forecast horizons using the Seatbelts dataset from the dataset package.

Here’s a summary of the problem at hand:

Outcome:
- DriversKilled - car drivers killed per month in the UK.
Predcitors:
- DriversKilled - car drivers killed per month in the UK.
- kms - a measure of distance driven.
- PetrolPrice - the price of gas.
- law - A binary indicator of the presence of a seatbelt law.
Forecast:
- Model training - The first 15 years of the monthly dataset.
- Model testing - The last year of the monthly dataset.

Install forecastML

devtools::install_github("nredell/forecastML")
library(forecastML)

Load packages and data

library(ggplot2)
library(glmnet)
library(randomForest)
library(DT)

data("data_seatbelts", package = "forecastML")
data <- data_seatbelts

data <- data[, c("DriversKilled", "kms", "PetrolPrice", "law")]
DT::datatable(head(data, 5))

Train-test split

ts_frequency <- 12  # monthly time-series

data_train <- data[1:(nrow(data) - ts_frequency), ]
data_test <- data[(nrow(data) - ts_frequency + 1):nrow(data), ]

p <- ggplot(data, aes(x = 1:nrow(data), y = DriversKilled))
p <- p + geom_line()
p <- p + geom_vline(xintercept = nrow(data_train), color = "red", size = 1.1)
p <- p + theme_bw() + xlab("Index")
p

Data Preparation

We’ll create a list of datasets, one for each forecast horizon, with lagged values for each predictor. The lookback argument in forecastML::create_lagged_df() specifies the predictor lags in dataset rows.

horizons <- 1:ts_frequency
lookback <- 1:15

data_list <- forecastML::create_lagged_df(data_train, type = "train",
                                          outcome_cols = 1, lookback = lookback,
                                          horizons = horizons)

Let’s view the modeling dataset for a forecast horion of 6.

DT::datatable(head(data_list[[6]], 10), options = list(scrollX = TRUE))

The plot below illustrates, for a given predictor, the number and position (in dataset rows) of lagged predictors created for each forecast horizon/model. The lookback argument to forecastML::created_lagged_df() was set to create lagged predictors from a minimum of 1 lag to a maximum of 15 lags; however, predictor lags that don’t allow for direct forecasting at a given forecast horizon are removed from the modeling dataset.

plot(data_list)

Nested cross-validation

forecastML::create_windows() creates indices for partitioning the training dataset in the outer loop of a nested cross-validation setup. The validation datasets are created in contiguous blocks of window_length, as opposed to randomly seleted rows, to mimic forecasting over multi-step-ahead forecast horizons. The skip, window_start, and window_stop arguments take dataset indices that allow the user to adjust the number and placement of outer loop validation datasets.

windows <- forecastML::create_windows(lagged_df = data_list, window_length = 12, skip = 0,
                                      window_start = NULL, window_stop = NULL,
                                      include_partial_window = TRUE)
windows

Below is a plot of the nested cross-validation outer loop datasets or windows. In our example, a window_length of 12 resulted in 14 validation windows.

In this nested cross-validation setup, a model is trained with data from 13 windows and forecast accuracy is assessed on the left out window. This means that we’ll need to train 14 models, each selecting different optimal hyperparameters and model coefficients–if available–from the inner validation loop.

plot(windows, data_list, show_labels = TRUE)

Model Training

User-defined forecast function

We’ll compare the forecasting performance of two models: (a) a cross-validated LASSO and (b) a non-tuned Random Forest. The following user-defined functions are needed for each model:

A wrapper function that takes the following positional arguments:
- 1: The input dataset with both target and model predictors. The predictor lags will be created according to the forecastML::create_lagged_df() function.
- 2: The column index of the outcome to be forecasted. Only 1 outcome can be modeled at present.
and returns a fitted model suitable for a predict()-type function.

Any inner loop cross-validation procedure should take place within this function, with the limitation that the inner cross-validation needs to ultimately return() one model.

# Example 1 - LASSO
model_function <- function(data, outcome_cols = 1) {

  x <- data[, -(outcome_cols), drop = FALSE]
  y <- data[, outcome_cols, drop = FALSE]
  x <- as.matrix(x, ncol = ncol(x))
  y <- as.matrix(y, ncol = ncol(y))

  model <- glmnet::cv.glmnet(x, y)
  return(model)
}

# Example 2 - Random Forest
model_function_2 <- function(data, outcome_cols = 1) {

  outcome_names <- names(data)[outcome_cols]
  model_formula <- formula(paste0(outcome_names,  "~ ."))

  model <- randomForest::randomForest(formula = model_formula, data = data, ntree = 200)
  return(model)
}

forecastML::train_model

For each modeling approach, LASSO and Random Forest, a total of N forecast horizons * N validation windows models are trained. In this example, that means training ** models** for each algorithm.

model_results <- forecastML::train_model(lagged_df = data_list, windows, 
                                         model_function, model_name = "LASSO")
model_results_2 <- forecastML::train_model(lagged_df = data_list, windows, 
                                           model_function_2, model_name = "RF")

User-defined prediction function

The following user-defined prediction function is needed for each model:

A wrapper function that takes the following positional arguments:
- 1: The model returned from the user-defined modeling function.
- 2: A data.frame() of the model predictors. Do not manually create the lagged predictors.
and returns a data.frame() of predictions with 1 column for each forecast target (limit 1 at present).

# Example 1 - LASSO
prediction_function <- function(model, data_features) {

  x <- as.matrix(data_features, ncol = ncol(data_features))

  data_pred <- data.frame("y_pred" = predict(model, x, s = "lambda.min"))
  return(data_pred)
}

# Example 2 - Random Forest
prediction_function_2 <- function(model, data_features) {

  data_pred <- data.frame("y_pred" = predict(model, data_features))
  return(data_pred)
}

forecastML::predict

The predict.forecast_model() method takes any number of trained models from forecastML::train_model() and a list of user-defined prediction functions. The list of prediction functions should appear in the same order as the models.

Outer loop nested cross-validation forecasts are returned for each user-defined model, forecast horizon, and validation window.

data_results <- predict(model_results, model_results_2,
                        prediction_function = list(prediction_function, prediction_function_2))

Let’s view the models’ predictions. The data.frame with S3 class training_results contains the following columns:

model: User-defined model name.
horizon: Forecast horizon.
window_length: Number of dataset rows in each validation window (partial windows have the user-specified window_length).
valid_indices: The dataset row indices.
<outcome>: The name of the column being forecasted.
<outcome_pred>: The forecasts.

DT::datatable(head(data_results, 10), options = list(scrollX = TRUE))

Below is a plot of the forecasts for each validation window at select forecast horizons.

plot(data_results, type = "prediction", horizons = c(1, 6, 12))

Below is a plot of the forecast error for select validation windows at select forecast horizons.

plot(data_results, type = "residual", horizons = c(1, 6, 12), windows = 10:14)

The plots below are diagnostic plots to check how forecasts for a target point in time have changed at different forecast horizons. In most cases it would be reasonable to expect shorter-horizon forecasts to be more accurate than longer-horizon forecasts.

Top plot: Rolling origin forecasts for the last validation window in our training data.
Bottom plot: Rolling origin forecasts for the first three points in our training data.

plot(data_results, type = "forecast_stability", windows = max(data_results$window_number))

plot(data_results, type = "forecast_stability", valid_indices = attributes(data_list)$row_indices[1:3])

The forecast_variability plot below is a summary of the forecast_stability plot. It’s a plot of the variability of forecasts for a target point in time collapsed across forecast horizons. A forecast model that produces greater variability across forecast horizons could be the better model provided the forecasts are increasingly accruate at shorter and shorter forecast horizons.

plot(data_results, type = "forecast_variability", valid_indices = 30:80)

Model Performance

forecastML::return_error

Let’s calcuate several common forecast error metrics.

mae: Mean absolute error
mape: Mean absolute percentage error
smape: Symmetrical mean absolute percentage error from (Chen and Yang’s 2004 formula with a 100% multiplier as discussed at https://robjhyndman.com/hyndsight/smape/)

The error for nested cross-validation are returned at 3 levels of granularity:

Error by validation window
Error by forecast horizon, collapsed across validation windows
Golbal error collapsed across validation windows and horizons

data_error <- forecastML::return_error(data_results, metrics = c("mae", "mape", "smape"),
                                       models = NULL)

DT::datatable(data_error$error_global, options = list(scrollX = TRUE))

Below is a plot of error metrics across time for select validation windows and forecast horizons.

plot(data_error, data_results, type = "time", horizons = c(1, 6, 12), windows = 10:14)

Below is a plot of forecast error metrics for each validation window (light) and the average across validation windows (dark).

plot(data_error, data_results, type = "horizon", horizons = c(1, 6, 12))

Below is a plot of error metrics collapsed across validation windows and forecast horizons.

plot(data_error, data_results, type = "global")

Hyperparameters

While it may be reasonable to have distinct models for each forecast horizon or even forecasting model ensembles across horizons, at this point we still have slightly different LASSO and Random Forest models from the outer loop of the nested cross-validation within each horizon-specific model. Here, we’ll take a look at the stability of the hyperparameters for the LASSO model to better understand if we can train one model across forecast horizons or if we need additional predictors or modeling strategies to forecast well under various conditions or time-series dynamics.

User-defined hyperparameter function

The following user-defined hyperparameter function is needed for each model:

A wrapper function that takes the following positional arguments
- 1: The model returned from the user-defined modeling function.
and returns a data.frame() of predictions with 1 column for each forecast outcome.

hyper_function <- function(model) {

  lambda_min <- model$lambda.min
  lambda_1se <- model$lambda.1se

  data_hyper <- data.frame("lambda_min" = lambda_min, "lambda_1se" = lambda_1se)
  return(data_hyper)
}

forecastML::return_hyper

Below are two plots which show (a) univariate hyperparameter variability across the training data and (b) the relationship between each error metric and hyperparameter values.

data_hyper <- forecastML::return_hyper(model_results, hyper_function)

plot(data_hyper, data_results, data_error, type = "stability", horizons = c(1, 6, 12))

plot(data_hyper, data_results, data_error, type = "error", c(1, 6, 12))

Forecast

forecastML::create_lagged_df

To forecast with the direct forecasting method, we need to create another dataset of lagged predictors. We can do this by running create_lagged_df() and setting type = "forecast".

For non-grouped time-series, this function takes the last rows of data_train and creates lagged predictors that allow forecasting from 1-step-ahead to N horizons for each horizon-specific model. Below is the forecast dataset for a 6-step-ahead forecast.

The row_number column gives the row index for the most recent period of actuals. In the example below, the row_number 180 represents the last row in our model-building dataset data_train.

data_forecast_list <- forecastML::create_lagged_df(data_train, type = "forecast", 
                                                  lookback = lookback,  horizon = horizons)

DT::datatable(head(data_forecast_list[[6]]), options = list(scrollX = TRUE))

Forecast results

Running the predict method, predict.forecast_model(), on the lagged predictor dataset created above–with type = "forecast"–and placing it in the data_forecast argument in predict.forecast_model() below, returns a data.frame of forecasts with the following columns:

model: User-defined model name.
model_forecast_horizon: The forecast horizon that the model was trained on.
horizon: Forecast horizon, ranging from 1 to model_forecast_horizon.
window_length: Number of dataset rows in each validation window (partial windows have the user-specified window_length).
window_number: The validation window number.
forecast_period: The dataset row/date for the forecast.
<outcome_pred>: The forecasts.

An S3 object of class, forecast_results, is returned. This object will have different plotting and error methods than the training_results class from earlier.

data_forecast <- predict(model_results, model_results_2,
                         prediction_function = list(prediction_function, prediction_function_2), 
                         data_forecast = data_forecast_list)

DT::datatable(head(data_forecast, 10), options = list(scrollX = TRUE))

Below is a plot of the forecasts vs. the actuals from data_test for each model at select forecast horizons.

It’s clear from the plots that our Random Forest model is producing less accurate forecasts and is more sensitive to the data on which it was trained–producing a handful of erratic forecasts.

plot(data_forecast, data_actual = data_train[-(1:150), ],
     actual_indices = as.numeric(row.names(data_train[-(1:150), ])),
     horizons = c(1, 6, 12), facet_plot = c("model", "model_forecast_horizon"))


plot(data_forecast, data_actual = data_test, 
     actual_indices = as.numeric(row.names(data_test)),
     facet_plot = "model", horizons = c(1, 6, 12))

Forecast error

Finally, we’ll look at the forecast error by forecast horizon for our two models.

If the first argument of forecastML::return_error() is an object of class forecast_results and the data_test argument is a data.frame like data_test from our beginning train-test split, a data.frame of forecast error metrics with the following columns is returned:

model: User-defined model name.
model_forecast_horizon: The forecast horizon that the model was trained on.
horizon: Forecast horizon, ranging from 1 to model_forecast_horizon.
<error_metrics>: Forecast error metrics.

data_error <- forecastML::return_error(data_forecast, data_test = data_test,
                                       test_indices = as.numeric(row.names(data_test)),
                                       metrics = c("mae", "mape", "smape", "mdape"))

DT::datatable(head(data_error$error_by_horizon, 10), options = list(scrollX = TRUE))

Model Selection and Re-training

Because our LASSO model is both stabler and more accurate, we’ll re-train our model across the entire training dataset to get our final 12 models–1 for each forecast horizon. Note that for a real-world forecasting problem this is when we would do additional model tuning to imrpove forecast accuracy across validation windows as well as narrow the hyperparameter search in the user-specified modeling functions.

forecastML::create_lagged_df

data_list <- forecastML::create_lagged_df(data_train, type = "train", lookback = lookback, 
                                          horizon = horizons)

forecastML::create_windows

To create a dataset without nested cross-validation, set window_length = 0 in forecastML::create_windows().

windows <- forecastML::create_windows(data_list, window_length = 0)

plot(windows, data_list, show_labels = TRUE)

forecastML::train_model

Without nested cross-validation and holdout windows, the prediction plot is essnetially a plot of model fit.

model_results <- forecastML::train_model(data_list, windows, model_function, model_name = "LASSO")

data_results <- predict(model_results, prediction_function = list(prediction_function))

DT::datatable(head(data_results, 10), options = list(scrollX = TRUE))

plot(data_results, type = "prediction", horizons = c(1, 6, 12))

plot(data_results, type = "residual", horizons = c(1, 6, 12))

plot(data_results, type = "forecast_stability", valid_indices = 109:120)

forecastML::return_error

data_error <- forecastML::return_error(data_results, metrics = c("mae", "mape", "mdape", "smape"),
                                       models = NULL)

DT::datatable(head(data_error$error_global), options = list(scrollX = TRUE))

plot(data_error, data_results, type = "horizon")

forecastML::return_hyper

data_hyper <- forecastML::return_hyper(model_results, hyper_function)

plot(data_hyper, data_results, data_error, type = "stability", horizons = c(1, 6, 12))

plot(data_hyper, data_results, data_error, type = "error", c(1, 6, 12))

Forecast

data_forecast_list <- forecastML::create_lagged_df(data_train, type = "forecast", 
                                                  lookback = lookback,  horizon = horizons)

data_forecast <- predict(model_results, prediction_function = list(prediction_function), 
                         data_forecast = data_forecast_list)

plot(data_forecast, data_actual = data[-(1:150), ],
     actual_indices = as.numeric(row.names(data[-(1:150), ])),
     horizons = c(1, 6, 12), 
     facet_plot = c("model", "model_forecast_horizon")) + ggplot2::theme(legend.position = "none")


plot(data_forecast, data_actual = data_test, actual_indices = as.numeric(row.names(data_test)),
     facet_plot = NULL, horizons = c(1, 6, 12))

Forecast error

data_error <- forecastML::return_error(data_forecast, data_test = data_test, 
                                       test_indices = as.numeric(row.names(data_test)),
                                       metrics = c("mae", "mape", "mdape", "smape"))

DT::datatable(data_error$error_by_horizon, options = list(scrollX = TRUE))

DT::datatable(data_error$error_global, options = list(scrollX = TRUE))

LS0tDQp0aXRsZTogInBhY2thZ2U6OmZvcmVjYXN0TUwgT3ZlcnZpZXciDQpkYXRlOiAiYHIgbHVicmlkYXRlOjp0b2RheSgpYCINCmF1dGhvcjogIk5pY2sgUmVkZWxsLCBuaWNrcmVkZWxsQGhvdG1haWwuY29tIg0Kb3V0cHV0Og0KICBodG1sX25vdGVib29rOg0KICAgIGNvZGVfZm9sZGluZzogc2hvdw0KICAgIHRvYzogeWVzDQogICAgdG9jX2Zsb2F0OiB5ZXMNCi0tLQ0KDQoqKioNCg0KIyBQdXJwb3NlDQoNCjxpbWcgc3JjPSJmb3JlY2FzdE1MX2xvZ28ucG5nIiBhbHQ9ImZvcmVjYXN0TUwgbG9nbyIgYWxpZ249InJpZ2h0IiB3aWR0aD0iMTUwIiBoZWlnaHQ9IjE1MCIgc3R5bGU9ImRpc3BsYXk6IGlubGluZS1ibG9jazsiPg0KDQpUaGUgcHVycG9zZSBvZiBgZm9yZWNhc3RNTGAgaXMgdG8gcHJvdmlkZSBhIHNlcmllcyBvZiBmdW5jdGlvbnMgYW5kIHZpc3VhbGl6YXRpb25zIHRoYXQgc2ltcGxpZnkgdGhlIHByb2Nlc3Mgb2YgDQptdWx0aS1zdGVwLWFoZWFkIGRpcmVjdCBmb3JlY2FzdGluZyB3aXRoIHN0YW5kYXJkIG1hY2hpbmUgbGVhcm5pbmcgYWxnb3JpdGhtcy4gSXQncyBhaW1lZCBhdCANCmhlbHBpbmcgdGhlIHVzZXIgcXVpY2tseSBhc3Nlc3MgdGhlIChhKSBhY2N1cmFjeSwgKGIpIHN0YWJpbGl0eSwgYW5kIChjKSBnZW5lcmFsaXphYmlsaXR5IG9mIHNpbmdsZS1vdXRjb21lIGZvcmVjYXN0cyANCnByb2R1Y2VkIGZyb20gcG90ZW50aWFsbHkgaGlnaC1kaW1lbnNpb25hbCBtb2RlbGluZyBkYXRhc2V0cy4NCg0KVGhpcyBwYWNrYWdlIGlzIGluc3BpcmVkIGJ5IEJlcmdtZWlyLCBIeW5kbWFuLCBhbmQgS29vJ3MgMjAxOCBwYXBlciANCltBIG5vdGUgb24gdGhlIHZhbGlkaXR5IG9mIGNyb3NzLXZhbGlkYXRpb24gZm9yIGV2YWx1YXRpbmcgYXV0b3JlZ3Jlc3NpdmUgdGltZSBzZXJpZXMgcHJlZGljdGlvbl0oaHR0cHM6Ly9yb2JqaHluZG1hbi5jb20vcGFwZXJzL2N2LXdwLnBkZikuIA0KSW4gcGFydGljdWxhciwgYGZvcmVjYXN0TUxgIG1ha2VzIHVzZSBvZiANCg0KKiAqKmxhZ2dlZCBwcmVkaWN0b3JzKiogYW5kIA0KKiAqKm5lc3RlZCBjcm9zcy12YWxpZGF0aW9uKiogd2l0aCAoYSkgdXNlci1zcGVjaWZpZWQgc3RhbmRhcmQgY3Jvc3MtdmFsaWRhdGlvbiBpbiB0aGUgaW5uZXIgbG9vcCBhbmQgKGIpIGJsb2NrLWNvbnRpZ3VvdXMgdmFsaWRhdGlvbiANCmRhdGFzZXRzIGluIHRoZSBvdXRlciBsb29wDQoNCnRvIGJ1aWxkIGFuZCBldmFsdWF0ZSBoaWdoLWRpbWVuc2lvbmFsIGZvcmVjYXN0IG1vZGVscyAqKndpdGhvdXQgaGF2aW5nIHRvIHVzZSBtZXRob2RzIHRoYXQgYXJlIHRpbWUtc2VyaWVzIHNwZWNpZmljKiouIA0KDQpUaGUgZm9sbG93aW5nIHF1b3RlIGZyb20gQmVyZ21laXIgZXQgYWwuJ3MgYXJ0aWNsZSBuaWNlbHkgc3VtcyB1cCB0aGUgYWltIG9mIHRoaXMgcGFja2FnZToNCg0KPiAiV2hlbiBwdXJlbHkgKG5vbi1saW5lYXIsIG5vbnBhcmFtZXRyaWMpIGF1dG9yZWdyZXNzaXZlIG1ldGhvZHMgYXJlIGFwcGxpZWQgdG8gZm9yZWNhc3RpbmcgcHJvYmxlbXMsIGFzIGlzIG9mdGVuIHRoZSBjYXNlDQo+IChlLmcuLCB3aGVuIHVzaW5nIE1hY2hpbmUgTGVhcm5pbmcgbWV0aG9kcyksIHRoZSBhZm9yZW1lbnRpb25lZCBwcm9ibGVtcyBvZiBDViBhcmUgbGFyZ2VseQ0KPiBpcnJlbGV2YW50LCBhbmQgQ1YgY2FuIGFuZCBzaG91bGQgYmUgdXNlZCB3aXRob3V0IG1vZGlmaWNhdGlvbiwgYXMgaW4gdGhlIGluZGVwZW5kZW50IGNhc2UuIg0KDQojIyBEaXJlY3QgZm9yZWNhc3RpbmcNCg0KSW4gY29udHJhc3QgdG8gdGhlIHJlY3Vyc2l2ZSBvciBpdGVyYXRlZCBtZXRob2QgZm9yIHByb2R1Y2luZyBtdWx0aS1zdGVwLWFoZWFkIGZvcmVjYXN0cyB1c2VkIGluIHRyYWRpdGlvbmFsIGZvcmVjYXN0aW5nIA0KbWV0aG9kcyBsaWtlIEFSSU1BLCBkaXJlY3QgZm9yZWNhc3RpbmcgaW52b2x2ZXMgY3JlYXRpbmcgYSBzZXJpZXMgb2YgZGlzdGluY3QgaG9yaXpvbi1zcGVjaWZpYyBtb2RlbHMuIFRob3VnaCANCnNldmVyYWwgaHlicmlkIG1ldGhvZHMgZXhpc3QgZm9yIHByb2R1Y2luZyBtdWx0aS1zdGVwIGZvcmVjYXN0cywgdGhlIHNpbXBsZSBkaXJlY3QgZm9yZWNhc3RpbmcgbWV0aG9kIA0Kd2l0aCBsYWdnZWQgZmVhdHVyZXMgdXNlZCBpbiBgZm9yZWNhc3RNTGAgbGV0J3MgdXMgYXZvaWQgdGhlIGV4cG9uZW50aWFsbHkgbW9yZSBkaWZmaWN1bHQgcHJvYmxlbSBvZiBoYXZpbmcgdG8gDQoicHJlZGljdCB0aGUgcHJlZGljdG9ycyIgZm9yIGZvcmVjYXN0IGhvcml6b25zIGJleW9uZCAxLXN0ZXAtYWhlYWQuDQoNCkJlbG93IGFyZSBzb21lIHJlc291cmNlcyBmb3IgbGVhcm5pbmcgbW9yZSBhYm91dCBtdWx0aS1zdGVwIGZvcmVjYXN0aW5nIHN0cmF0ZWdpZXM6DQoNCiogW0EgcmV2aWV3IGFuZCBjb21wYXJpc29uIG9mIHN0cmF0ZWdpZXMgZm9yIG11bHRpLXN0ZXAtYWhlYWQgdGltZSBzZXJpZXMgZm9yZWNhc3RpbmcgYmFzZWQgb24gdGhlIE5ONSBmb3JlY2FzdGluZyBjb21wZXRpdGlvbl0oaHR0cHM6Ly9hcnhpdi5vcmcvcGRmLzExMDguMzI1OS5wZGYpDQoqIFtBIGNvbXBhcmlzb24gb2YgZGlyZWN0IGFuZCBpdGVyYXRlZCBtdWx0aXN0ZXAgQVIgbWV0aG9kcyBmb3IgZm9yZWNhc3RpbmcgbWFjcm9lY29ub21pYyB0aW1lIHNlcmllc10oaHR0cHM6Ly93d3cucHJpbmNldG9uLmVkdS9+bXdhdHNvbi9wYXBlcnMvaHN0ZXBfMy5wZGYpDQoNClRoZSAqKmFuaW1hdGlvbiBiZWxvdyoqIHNob3dzIGhvdyBoaXN0b3JpY2FsIGRhdGEgaXMgdXNlZCB0byBjcmVhdGUgYSAxLXRvLTEyLXN0ZXAtYWhlYWQgZm9yZWNhc3QgZm9yIGEgMTItc3RlcC1ob3Jpem9uIA0KZm9yZWNhc3QgbW9kZWwgdXNpbmcgbGFnZ2VkIGZlYXR1cmVzLiBUaG91Z2ggcHJlZGljdG9yIGxhZ3MgZ3JlYXRlciB0aGFuIDEyIHN0ZXBzIGNhbiBiZSB1c2VkIHRvIG1ha2UgdXNlIG9mIA0KYWRkaXRpb25hbCBoaXN0b3JpY2FsIHByZWRpY3RpdmUgaW5mb3JtYXRpb24sIGEgMTItc3RlcC1ob3Jpem9uIGRpcmVjdCBmb3JlY2FzdCBtb2RlbCByZXF1aXJlcyBwcmVkaWN0b3IgbGFncyA+PSAxMi4gDQpUaGlzIGFuaW1hdGlvbiBpcyByb3VnaGx5IGVxdWl2YWxlbnQgdG8gaG93IGEgMTItcGVyaW9kIHNlYXNvbmFsIEFSSU1BKDAsIDAsIDApKDEsIDAsIDApIG1vZGVsIHVzZXMgaGlzdG9yaWNhbCBkYXRhIHRvIHByb2R1Y2UgZm9yZWNhc3RzLg0KDQohW10oLi9kaXJlY3RfZm9yZWNhc3RpbmcuZ2lmKQ0KDQoqKioNCg0KPGJyPg0KDQojIFBhY2thZ2UgRmVhdHVyZXMNCg0KMS4gKipUcmFuc2Zvcm0gZGF0YXNldHMqKiBmb3IgbW9kZWxpbmcgYnkgY3JlYXRpbmcgdmFyaW91cyBwYXR0ZXJucyBvZiBsYWdnZWQgcHJlZGljdG9ycyBmb3IgdXNlci1zcGVjaWZpZWQgDQpmb3JlY2FzdCBob3Jpem9ucyB3aXRoIGBmb3JlY2FzdE1MOjpjcmVhdGVfbGFnZ2VkX2RmKClgDQoNCjIuIENyZWF0ZSBkYXRhc2V0cyBmb3IgZXZhbHVhdGluZyBmb3JlY2FzdCBtb2RlbHMgdXNpbmcgKipuZXN0ZWQgY3Jvc3MtdmFsaWRhdGlvbioqIHdpdGggYGZvcmVjYXN0TUw6OmNyZWF0ZV93aW5kb3dzKClgDQoNCjMuIFRyYWluIGFuZCAqKmV2YWx1YXRlIG1hY2hpbmUgbGVhcm5pbmcgbW9kZWxzIGZvciBmb3JlY2FzdGluZyoqIHdpdGggYGZvcmVjYXN0TUw6OnRyYWluX21vZGVsKClgLg0KDQo0LiBBc3Nlc3MgKipmb3JlY2FzdCBhY2N1cmFjeSoqIGF0IGRpZmZlcmVudCBmb3JlY2FzdCBob3Jpem9ucyB3aXRoIGBmb3JlY2FzdE1MOjpyZXR1cm5fZXJyb3IoKWAuDQoNCjUuIEFzc2VzcyAqKmh5cGVycGFyYW1ldGVyIHN0YWJpbGl0eSoqIHdpdGggYGZvcmVjYXN0TUw6OnJldHVybl9oeXBlcigpYC4NCg0KNi4gQ3JlYXRlIGRhdGFzZXRzIG9mIGxhZ2dlZCBwcmVkaWN0b3JzIGZvciAqKmRpcmVjdCBmb3JlY2FzdGluZyoqLg0KDQoqKioNCg0KYGBge3IsIGluY2x1ZGUgPSBGQUxTRX0NCmtuaXRyOjpvcHRzX2NodW5rJHNldChmaWcud2lkdGggPSA5LCBmaWcuaGVpZ2h0ID0gNikNCmBgYA0KDQpgYGB7ciwgZXZhbCA9IEZBTFNFLCBpbmNsdWRlID0gRkFMU0V9DQpkZXZ0b29sczo6ZG9jdW1lbnQoKQ0KZGV2dG9vbHM6OmxvYWRfYWxsKGV4cG9ydF9hbGwgPSBUUlVFKQ0KYGBgDQoNCjxicj4NCg0KIyBFeGFtcGxlDQoNCkluIHRoaXMgd2Fsa3Rocm91Z2ggb2YgYGZvcmVjYXN0TUxgIHdlJ2xsIGNvbXBhcmUgdGhlIGZvcmVjYXN0IHBlcmZvcm1hbmNlIG9mIHR3byBtYWNoaW5lIGxlYXJuaW5nIA0KbWV0aG9kcywgTEFTU08gYW5kIFJhbmRvbSBGb3Jlc3QsIGFjcm9zcyBmb3JlY2FzdCBob3Jpem9ucyB1c2luZyB0aGUgU2VhdGJlbHRzIGRhdGFzZXQgZnJvbSB0aGUgYGRhdGFzZXRgIHBhY2thZ2UuDQoNCkhlcmUncyBhIHN1bW1hcnkgb2YgdGhlIHByb2JsZW0gYXQgaGFuZDoNCg0KKiAqKk91dGNvbWU6KioNCiAgICAqIGBEcml2ZXJzS2lsbGVkYCAtIGNhciBkcml2ZXJzIGtpbGxlZCBwZXIgbW9udGggaW4gdGhlIFVLLg0KKiAqKlByZWRjaXRvcnM6KioNCiAgICAqIGBEcml2ZXJzS2lsbGVkYCAtIGNhciBkcml2ZXJzIGtpbGxlZCBwZXIgbW9udGggaW4gdGhlIFVLLg0KICAgICogYGttc2AgLSBhIG1lYXN1cmUgb2YgZGlzdGFuY2UgZHJpdmVuLg0KICAgICogYFBldHJvbFByaWNlYCAtIHRoZSBwcmljZSBvZiBnYXMuDQogICAgKiBgbGF3YCAtIEEgYmluYXJ5IGluZGljYXRvciBvZiB0aGUgcHJlc2VuY2Ugb2YgYSBzZWF0YmVsdCBsYXcuDQoqICoqRm9yZWNhc3Q6KioNCiAgICAqIE1vZGVsIHRyYWluaW5nIC0gVGhlIGZpcnN0IDE1IHllYXJzIG9mIHRoZSBtb250aGx5IGRhdGFzZXQuDQogICAgKiBNb2RlbCB0ZXN0aW5nIC0gVGhlIGxhc3QgeWVhciBvZiB0aGUgbW9udGhseSBkYXRhc2V0Lg0KDQojIyBJbnN0YWxsIGZvcmVjYXN0TUwNCg0KYGBge3IsIGV2YWwgPSBGQUxTRX0NCmRldnRvb2xzOjppbnN0YWxsX2dpdGh1YigibnJlZGVsbC9mb3JlY2FzdE1MIikNCmxpYnJhcnkoZm9yZWNhc3RNTCkNCmBgYA0KDQo8YnI+DQoNCiMjIExvYWQgcGFja2FnZXMgYW5kIGRhdGENCg0KYGBge3IsIHdhcm5pbmcgPSBGQUxTRSwgbWVzc2FnZSA9IEZBTFNFfQ0KbGlicmFyeShnZ3Bsb3QyKQ0KbGlicmFyeShnbG1uZXQpDQpsaWJyYXJ5KHJhbmRvbUZvcmVzdCkNCmxpYnJhcnkoRFQpDQoNCmRhdGEoImRhdGFfc2VhdGJlbHRzIiwgcGFja2FnZSA9ICJmb3JlY2FzdE1MIikNCmRhdGEgPC0gZGF0YV9zZWF0YmVsdHMNCg0KZGF0YSA8LSBkYXRhWywgYygiRHJpdmVyc0tpbGxlZCIsICJrbXMiLCAiUGV0cm9sUHJpY2UiLCAibGF3IildDQpEVDo6ZGF0YXRhYmxlKGhlYWQoZGF0YSwgNSkpDQpgYGANCg0KKioqDQoNCjxicj4NCg0KIyMgVHJhaW4tdGVzdCBzcGxpdA0KDQpgYGB7cn0NCnRzX2ZyZXF1ZW5jeSA8LSAxMiAgIyBtb250aGx5IHRpbWUtc2VyaWVzDQoNCmRhdGFfdHJhaW4gPC0gZGF0YVsxOihucm93KGRhdGEpIC0gdHNfZnJlcXVlbmN5KSwgXQ0KZGF0YV90ZXN0IDwtIGRhdGFbKG5yb3coZGF0YSkgLSB0c19mcmVxdWVuY3kgKyAxKTpucm93KGRhdGEpLCBdDQoNCnAgPC0gZ2dwbG90KGRhdGEsIGFlcyh4ID0gMTpucm93KGRhdGEpLCB5ID0gRHJpdmVyc0tpbGxlZCkpDQpwIDwtIHAgKyBnZW9tX2xpbmUoKQ0KcCA8LSBwICsgZ2VvbV92bGluZSh4aW50ZXJjZXB0ID0gbnJvdyhkYXRhX3RyYWluKSwgY29sb3IgPSAicmVkIiwgc2l6ZSA9IDEuMSkNCnAgPC0gcCArIHRoZW1lX2J3KCkgKyB4bGFiKCJJbmRleCIpDQpwDQpgYGANCg0KKioqDQoNCjxicj4NCg0KIyBEYXRhIFByZXBhcmF0aW9uDQoNCldlJ2xsIGNyZWF0ZSBhIGxpc3Qgb2YgZGF0YXNldHMsIG9uZSBmb3IgZWFjaCBmb3JlY2FzdCBob3Jpem9uLCB3aXRoIGxhZ2dlZCB2YWx1ZXMgZm9yIGVhY2ggcHJlZGljdG9yLiANClRoZSBgbG9va2JhY2tgIGFyZ3VtZW50IGluIGBmb3JlY2FzdE1MOjpjcmVhdGVfbGFnZ2VkX2RmKClgIHNwZWNpZmllcyB0aGUgcHJlZGljdG9yIGxhZ3MgaW4gZGF0YXNldCByb3dzLg0KDQpgYGB7cn0NCmhvcml6b25zIDwtIDE6dHNfZnJlcXVlbmN5DQpsb29rYmFjayA8LSAxOjE1DQoNCmRhdGFfbGlzdCA8LSBmb3JlY2FzdE1MOjpjcmVhdGVfbGFnZ2VkX2RmKGRhdGFfdHJhaW4sIHR5cGUgPSAidHJhaW4iLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgb3V0Y29tZV9jb2xzID0gMSwgbG9va2JhY2sgPSBsb29rYmFjaywNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhvcml6b25zID0gaG9yaXpvbnMpDQpgYGANCg0KKioqDQoNCjxicj4NCg0KTGV0J3MgdmlldyB0aGUgbW9kZWxpbmcgZGF0YXNldCBmb3IgYSBmb3JlY2FzdCBob3Jpb24gb2YgNi4NCg0KYGBge3J9DQpEVDo6ZGF0YXRhYmxlKGhlYWQoZGF0YV9saXN0W1s2XV0sIDEwKSwgb3B0aW9ucyA9IGxpc3Qoc2Nyb2xsWCA9IFRSVUUpKQ0KYGBgDQoNCioqKg0KDQo8YnI+DQoNClRoZSBwbG90IGJlbG93IGlsbHVzdHJhdGVzLCBmb3IgYSBnaXZlbiBwcmVkaWN0b3IsIHRoZSBudW1iZXIgYW5kIHBvc2l0aW9uIChpbiBkYXRhc2V0IHJvd3MpIG9mIGxhZ2dlZCBwcmVkaWN0b3JzIA0KY3JlYXRlZCBmb3IgZWFjaCBmb3JlY2FzdCBob3Jpem9uL21vZGVsLiBUaGUgYGxvb2tiYWNrYCBhcmd1bWVudCB0byBgZm9yZWNhc3RNTDo6Y3JlYXRlZF9sYWdnZWRfZGYoKWAgd2FzIHNldCB0byANCmNyZWF0ZSBsYWdnZWQgcHJlZGljdG9ycyBmcm9tIGEgbWluaW11bSBvZiAxIGxhZyB0byBhIG1heGltdW0gb2YgMTUgbGFnczsgaG93ZXZlciwgcHJlZGljdG9yIGxhZ3MgdGhhdCBkb24ndCBhbGxvdyANCmZvciBkaXJlY3QgZm9yZWNhc3RpbmcgYXQgYSBnaXZlbiBmb3JlY2FzdCBob3Jpem9uIGFyZSByZW1vdmVkIGZyb20gdGhlIG1vZGVsaW5nIGRhdGFzZXQuDQoNCmBgYHtyfQ0KcGxvdChkYXRhX2xpc3QpDQpgYGANCg0KKioqDQoNCjxicj4NCg0KIyMgTmVzdGVkIGNyb3NzLXZhbGlkYXRpb24NCg0KYGZvcmVjYXN0TUw6OmNyZWF0ZV93aW5kb3dzKClgIGNyZWF0ZXMgaW5kaWNlcyBmb3IgcGFydGl0aW9uaW5nIHRoZSB0cmFpbmluZyBkYXRhc2V0IGluIHRoZSBvdXRlciBsb29wIG9mIGEgbmVzdGVkIA0KY3Jvc3MtdmFsaWRhdGlvbiBzZXR1cC4gVGhlIHZhbGlkYXRpb24gZGF0YXNldHMgYXJlIGNyZWF0ZWQgaW4gY29udGlndW91cyBibG9ja3Mgb2YgYHdpbmRvd19sZW5ndGhgLCBhcyBvcHBvc2VkIHRvIA0KcmFuZG9tbHkgc2VsZXRlZCByb3dzLCB0byBtaW1pYyBmb3JlY2FzdGluZyBvdmVyIG11bHRpLXN0ZXAtYWhlYWQgZm9yZWNhc3QgaG9yaXpvbnMuIFRoZSBgc2tpcGAsIA0KYHdpbmRvd19zdGFydGAsIGFuZCBgd2luZG93X3N0b3BgIGFyZ3VtZW50cyB0YWtlIGRhdGFzZXQgaW5kaWNlcyB0aGF0IGFsbG93IHRoZSB1c2VyIHRvIGFkanVzdCB0aGUgbnVtYmVyIGFuZCANCnBsYWNlbWVudCBvZiBvdXRlciBsb29wIHZhbGlkYXRpb24gZGF0YXNldHMuDQoNCmBgYHtyLCB3YXJuaW5ncyA9IEZBTFNFLCBtZXNzYWdlID0gRkFMU0V9DQp3aW5kb3dzIDwtIGZvcmVjYXN0TUw6OmNyZWF0ZV93aW5kb3dzKGxhZ2dlZF9kZiA9IGRhdGFfbGlzdCwgd2luZG93X2xlbmd0aCA9IDEyLCBza2lwID0gMCwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd2luZG93X3N0YXJ0ID0gTlVMTCwgd2luZG93X3N0b3AgPSBOVUxMLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlX3BhcnRpYWxfd2luZG93ID0gVFJVRSkNCndpbmRvd3MNCmBgYA0KDQoqKioNCg0KPGJyPg0KDQpCZWxvdyBpcyBhIHBsb3Qgb2YgdGhlIG5lc3RlZCBjcm9zcy12YWxpZGF0aW9uIG91dGVyIGxvb3AgZGF0YXNldHMgb3Igd2luZG93cy4gSW4gb3VyIGV4YW1wbGUsIA0KYSBgd2luZG93X2xlbmd0aGAgb2YgMTIgcmVzdWx0ZWQgaW4gMTQgdmFsaWRhdGlvbiB3aW5kb3dzLg0KDQpJbiB0aGlzIG5lc3RlZCBjcm9zcy12YWxpZGF0aW9uIHNldHVwLCBhIG1vZGVsIGlzIHRyYWluZWQgd2l0aCBkYXRhIGZyb20gMTMgd2luZG93cyBhbmQgDQpmb3JlY2FzdCBhY2N1cmFjeSBpcyBhc3Nlc3NlZCBvbiB0aGUgbGVmdCBvdXQgd2luZG93LiBUaGlzIG1lYW5zIHRoYXQgd2UnbGwgbmVlZCB0byB0cmFpbiAxNCBtb2RlbHMsIA0KZWFjaCBzZWxlY3RpbmcgZGlmZmVyZW50IG9wdGltYWwgaHlwZXJwYXJhbWV0ZXJzIGFuZCBtb2RlbCBjb2VmZmljaWVudHMtLWlmIGF2YWlsYWJsZS0tZnJvbSANCnRoZSBpbm5lciB2YWxpZGF0aW9uIGxvb3AuDQoNCmBgYHtyfQ0KcGxvdCh3aW5kb3dzLCBkYXRhX2xpc3QsIHNob3dfbGFiZWxzID0gVFJVRSkNCmBgYA0KDQoqKioNCg0KPGJyPg0KDQojIE1vZGVsIFRyYWluaW5nDQoNCiMjIFVzZXItZGVmaW5lZCBmb3JlY2FzdCBmdW5jdGlvbg0KDQpXZSdsbCBjb21wYXJlIHRoZSBmb3JlY2FzdGluZyBwZXJmb3JtYW5jZSBvZiB0d28gbW9kZWxzOiAoYSkgYSBjcm9zcy12YWxpZGF0ZWQgTEFTU08gYW5kIChiKSBhIG5vbi10dW5lZCBSYW5kb20gRm9yZXN0LiANClRoZSBmb2xsb3dpbmcgdXNlci1kZWZpbmVkIGZ1bmN0aW9ucyBhcmUgbmVlZGVkIGZvciBlYWNoIG1vZGVsOg0KDQoqIEEgd3JhcHBlciBmdW5jdGlvbiB0aGF0IHRha2VzIHRoZSBmb2xsb3dpbmcgcG9zaXRpb25hbCAqKmFyZ3VtZW50cyoqOg0KICAgICogKioxOioqIFRoZSBpbnB1dCBkYXRhc2V0IHdpdGggYm90aCB0YXJnZXQgYW5kIG1vZGVsIHByZWRpY3RvcnMuIFRoZSBwcmVkaWN0b3IgbGFncyB3aWxsIGJlIGNyZWF0ZWQgYWNjb3JkaW5nIHRvIHRoZSANCiAgICBgZm9yZWNhc3RNTDo6Y3JlYXRlX2xhZ2dlZF9kZigpYCBmdW5jdGlvbi4NCiAgICAqICoqMjoqKiBUaGUgY29sdW1uIGluZGV4IG9mIHRoZSBvdXRjb21lIHRvIGJlIGZvcmVjYXN0ZWQuICpPbmx5IDEgb3V0Y29tZSBjYW4gYmUgbW9kZWxlZCBhdCBwcmVzZW50Ki4NCiogYW5kICoqcmV0dXJucyoqIGEgZml0dGVkIG1vZGVsIHN1aXRhYmxlIGZvciBhIGBwcmVkaWN0KClgLXR5cGUgZnVuY3Rpb24uDQoNCkFueSBpbm5lciBsb29wIGNyb3NzLXZhbGlkYXRpb24gcHJvY2VkdXJlIHNob3VsZCB0YWtlIHBsYWNlIHdpdGhpbiB0aGlzIGZ1bmN0aW9uLCB3aXRoIHRoZSBsaW1pdGF0aW9uIHRoYXQgDQp0aGUgaW5uZXIgY3Jvc3MtdmFsaWRhdGlvbiBuZWVkcyB0byB1bHRpbWF0ZWx5IGByZXR1cm4oKWAgb25lIG1vZGVsLg0KDQpgYGB7cn0NCiMgRXhhbXBsZSAxIC0gTEFTU08NCm1vZGVsX2Z1bmN0aW9uIDwtIGZ1bmN0aW9uKGRhdGEsIG91dGNvbWVfY29scyA9IDEpIHsNCg0KICB4IDwtIGRhdGFbLCAtKG91dGNvbWVfY29scyksIGRyb3AgPSBGQUxTRV0NCiAgeSA8LSBkYXRhWywgb3V0Y29tZV9jb2xzLCBkcm9wID0gRkFMU0VdDQogIHggPC0gYXMubWF0cml4KHgsIG5jb2wgPSBuY29sKHgpKQ0KICB5IDwtIGFzLm1hdHJpeCh5LCBuY29sID0gbmNvbCh5KSkNCg0KICBtb2RlbCA8LSBnbG1uZXQ6OmN2LmdsbW5ldCh4LCB5KQ0KICByZXR1cm4obW9kZWwpDQp9DQoNCiMgRXhhbXBsZSAyIC0gUmFuZG9tIEZvcmVzdA0KbW9kZWxfZnVuY3Rpb25fMiA8LSBmdW5jdGlvbihkYXRhLCBvdXRjb21lX2NvbHMgPSAxKSB7DQoNCiAgb3V0Y29tZV9uYW1lcyA8LSBuYW1lcyhkYXRhKVtvdXRjb21lX2NvbHNdDQogIG1vZGVsX2Zvcm11bGEgPC0gZm9ybXVsYShwYXN0ZTAob3V0Y29tZV9uYW1lcywgICJ+IC4iKSkNCg0KICBtb2RlbCA8LSByYW5kb21Gb3Jlc3Q6OnJhbmRvbUZvcmVzdChmb3JtdWxhID0gbW9kZWxfZm9ybXVsYSwgZGF0YSA9IGRhdGEsIG50cmVlID0gMjAwKQ0KICByZXR1cm4obW9kZWwpDQp9DQpgYGANCg0KPGJyPg0KDQojIyBmb3JlY2FzdE1MOjp0cmFpbl9tb2RlbA0KDQpGb3IgZWFjaCBtb2RlbGluZyBhcHByb2FjaCwgTEFTU08gYW5kIFJhbmRvbSBGb3Jlc3QsIGEgdG90YWwgb2YgYE4gZm9yZWNhc3QgaG9yaXpvbnNgICogYE4gdmFsaWRhdGlvbiB3aW5kb3dzYCANCm1vZGVscyBhcmUgdHJhaW5lZC4gSW4gdGhpcyBleGFtcGxlLCB0aGF0IG1lYW5zIHRyYWluaW5nICoqYHIgbGVuZ3RoKGRhdGFfbGlzdCkgKiBucm93KHdpbmRvd3NbWzFdXSlgIG1vZGVscyoqIA0KZm9yIGVhY2ggYWxnb3JpdGhtLg0KDQpgYGB7cn0NCm1vZGVsX3Jlc3VsdHMgPC0gZm9yZWNhc3RNTDo6dHJhaW5fbW9kZWwobGFnZ2VkX2RmID0gZGF0YV9saXN0LCB3aW5kb3dzLCANCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbW9kZWxfZnVuY3Rpb24sIG1vZGVsX25hbWUgPSAiTEFTU08iKQ0KbW9kZWxfcmVzdWx0c18yIDwtIGZvcmVjYXN0TUw6OnRyYWluX21vZGVsKGxhZ2dlZF9kZiA9IGRhdGFfbGlzdCwgd2luZG93cywgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbW9kZWxfZnVuY3Rpb25fMiwgbW9kZWxfbmFtZSA9ICJSRiIpDQpgYGANCg0KPGJyPg0KDQojIyBVc2VyLWRlZmluZWQgcHJlZGljdGlvbiBmdW5jdGlvbg0KDQpUaGUgZm9sbG93aW5nIHVzZXItZGVmaW5lZCBwcmVkaWN0aW9uIGZ1bmN0aW9uIGlzIG5lZWRlZCBmb3IgZWFjaCBtb2RlbDoNCg0KKiBBIHdyYXBwZXIgZnVuY3Rpb24gdGhhdCB0YWtlcyB0aGUgZm9sbG93aW5nIHBvc2l0aW9uYWwgKiphcmd1bWVudHMqKjoNCiAgICAqICoqMToqKiBUaGUgbW9kZWwgcmV0dXJuZWQgZnJvbSB0aGUgdXNlci1kZWZpbmVkIG1vZGVsaW5nIGZ1bmN0aW9uLg0KICAgICogKioyOioqIEEgYGRhdGEuZnJhbWUoKWAgb2YgdGhlIG1vZGVsIHByZWRpY3RvcnMuIERvIG5vdCBtYW51YWxseSBjcmVhdGUgdGhlIGxhZ2dlZCBwcmVkaWN0b3JzLg0KKiBhbmQgKipyZXR1cm5zKiogYSBgZGF0YS5mcmFtZSgpYCBvZiBwcmVkaWN0aW9ucyB3aXRoIDEgY29sdW1uIGZvciBlYWNoIGZvcmVjYXN0IHRhcmdldCAobGltaXQgMSBhdCBwcmVzZW50KS4NCg0KYGBge3J9DQojIEV4YW1wbGUgMSAtIExBU1NPDQpwcmVkaWN0aW9uX2Z1bmN0aW9uIDwtIGZ1bmN0aW9uKG1vZGVsLCBkYXRhX2ZlYXR1cmVzKSB7DQoNCiAgeCA8LSBhcy5tYXRyaXgoZGF0YV9mZWF0dXJlcywgbmNvbCA9IG5jb2woZGF0YV9mZWF0dXJlcykpDQoNCiAgZGF0YV9wcmVkIDwtIGRhdGEuZnJhbWUoInlfcHJlZCIgPSBwcmVkaWN0KG1vZGVsLCB4LCBzID0gImxhbWJkYS5taW4iKSkNCiAgcmV0dXJuKGRhdGFfcHJlZCkNCn0NCg0KIyBFeGFtcGxlIDIgLSBSYW5kb20gRm9yZXN0DQpwcmVkaWN0aW9uX2Z1bmN0aW9uXzIgPC0gZnVuY3Rpb24obW9kZWwsIGRhdGFfZmVhdHVyZXMpIHsNCg0KICBkYXRhX3ByZWQgPC0gZGF0YS5mcmFtZSgieV9wcmVkIiA9IHByZWRpY3QobW9kZWwsIGRhdGFfZmVhdHVyZXMpKQ0KICByZXR1cm4oZGF0YV9wcmVkKQ0KfQ0KYGBgDQoNCjxicj4NCg0KIyMgZm9yZWNhc3RNTDo6cHJlZGljdA0KDQpUaGUgYHByZWRpY3QuZm9yZWNhc3RfbW9kZWwoKWAgbWV0aG9kIHRha2VzIGFueSBudW1iZXIgb2YgdHJhaW5lZCBtb2RlbHMgZnJvbSBgZm9yZWNhc3RNTDo6dHJhaW5fbW9kZWwoKWAgYW5kIGEgDQpsaXN0IG9mIHVzZXItZGVmaW5lZCBwcmVkaWN0aW9uIGZ1bmN0aW9ucy4gVGhlIGxpc3Qgb2YgcHJlZGljdGlvbiBmdW5jdGlvbnMgc2hvdWxkIGFwcGVhciBpbiB0aGUgc2FtZSBvcmRlciANCmFzIHRoZSBtb2RlbHMuDQoNCk91dGVyIGxvb3AgbmVzdGVkIGNyb3NzLXZhbGlkYXRpb24gZm9yZWNhc3RzIGFyZSByZXR1cm5lZCBmb3IgZWFjaCB1c2VyLWRlZmluZWQgbW9kZWwsIGZvcmVjYXN0IGhvcml6b24sIGFuZCANCnZhbGlkYXRpb24gd2luZG93Lg0KDQpgYGB7cn0NCmRhdGFfcmVzdWx0cyA8LSBwcmVkaWN0KG1vZGVsX3Jlc3VsdHMsIG1vZGVsX3Jlc3VsdHNfMiwNCiAgICAgICAgICAgICAgICAgICAgICAgIHByZWRpY3Rpb25fZnVuY3Rpb24gPSBsaXN0KHByZWRpY3Rpb25fZnVuY3Rpb24sIHByZWRpY3Rpb25fZnVuY3Rpb25fMikpDQoNCmBgYA0KDQo8YnI+DQoNCkxldCdzIHZpZXcgdGhlIG1vZGVscycgcHJlZGljdGlvbnMuIFRoZSBkYXRhLmZyYW1lIHdpdGggUzMgY2xhc3MgYHRyYWluaW5nX3Jlc3VsdHNgIGNvbnRhaW5zIHRoZSBmb2xsb3dpbmcgY29sdW1uczoNCg0KKiAqKm1vZGVsOioqIFVzZXItZGVmaW5lZCBtb2RlbCBuYW1lLg0KKiAqKmhvcml6b246KiogRm9yZWNhc3QgaG9yaXpvbi4NCiogKip3aW5kb3dfbGVuZ3RoOioqIE51bWJlciBvZiBkYXRhc2V0IHJvd3MgaW4gZWFjaCB2YWxpZGF0aW9uIHdpbmRvdyAocGFydGlhbCB3aW5kb3dzIGhhdmUgdGhlIHVzZXItc3BlY2lmaWVkIHdpbmRvd19sZW5ndGgpLg0KKiAqKnZhbGlkX2luZGljZXM6KiogVGhlIGRhdGFzZXQgcm93IGluZGljZXMuDQoqICoqXDxvdXRjb21lPjoqKiBUaGUgbmFtZSBvZiB0aGUgY29sdW1uIGJlaW5nIGZvcmVjYXN0ZWQuDQoqICoqXDxvdXRjb21lX3ByZWQ+OioqIFRoZSBmb3JlY2FzdHMuDQoNCmBgYHtyfQ0KRFQ6OmRhdGF0YWJsZShoZWFkKGRhdGFfcmVzdWx0cywgMTApLCBvcHRpb25zID0gbGlzdChzY3JvbGxYID0gVFJVRSkpDQpgYGANCg0KKioqDQoNCjxicj4NCg0KQmVsb3cgaXMgYSBwbG90IG9mIHRoZSBmb3JlY2FzdHMgZm9yIGVhY2ggdmFsaWRhdGlvbiB3aW5kb3cgYXQgc2VsZWN0IGZvcmVjYXN0IGhvcml6b25zLg0KDQpgYGB7cn0NCnBsb3QoZGF0YV9yZXN1bHRzLCB0eXBlID0gInByZWRpY3Rpb24iLCBob3Jpem9ucyA9IGMoMSwgNiwgMTIpKQ0KYGBgDQoNCioqKg0KDQo8YnI+DQoNCkJlbG93IGlzIGEgcGxvdCBvZiB0aGUgZm9yZWNhc3QgZXJyb3IgZm9yIHNlbGVjdCB2YWxpZGF0aW9uIHdpbmRvd3MgYXQgc2VsZWN0IGZvcmVjYXN0IGhvcml6b25zLg0KDQpgYGB7cn0NCnBsb3QoZGF0YV9yZXN1bHRzLCB0eXBlID0gInJlc2lkdWFsIiwgaG9yaXpvbnMgPSBjKDEsIDYsIDEyKSwgd2luZG93cyA9IDEwOjE0KQ0KYGBgDQoNCioqKg0KDQo8YnI+DQoNClRoZSBwbG90cyBiZWxvdyBhcmUgZGlhZ25vc3RpYyBwbG90cyB0byBjaGVjayBob3cgZm9yZWNhc3RzIGZvciBhIHRhcmdldCBwb2ludCBpbiB0aW1lIGhhdmUgDQpjaGFuZ2VkIGF0IGRpZmZlcmVudCBmb3JlY2FzdCBob3Jpem9ucy4gSW4gbW9zdCBjYXNlcyBpdCB3b3VsZCBiZSByZWFzb25hYmxlIHRvIGV4cGVjdCANCnNob3J0ZXItaG9yaXpvbiBmb3JlY2FzdHMgdG8gYmUgbW9yZSBhY2N1cmF0ZSB0aGFuIGxvbmdlci1ob3Jpem9uIGZvcmVjYXN0cy4NCg0KKiAqKlRvcCBwbG90OioqIFJvbGxpbmcgb3JpZ2luIGZvcmVjYXN0cyBmb3IgdGhlIGxhc3QgdmFsaWRhdGlvbiB3aW5kb3cgaW4gb3VyIHRyYWluaW5nIGRhdGEuDQoqICoqQm90dG9tIHBsb3Q6KiogUm9sbGluZyBvcmlnaW4gZm9yZWNhc3RzIGZvciB0aGUgZmlyc3QgdGhyZWUgcG9pbnRzIGluIG91ciB0cmFpbmluZyBkYXRhLg0KDQpgYGB7cn0NCnBsb3QoZGF0YV9yZXN1bHRzLCB0eXBlID0gImZvcmVjYXN0X3N0YWJpbGl0eSIsIHdpbmRvd3MgPSBtYXgoZGF0YV9yZXN1bHRzJHdpbmRvd19udW1iZXIpKQ0KcGxvdChkYXRhX3Jlc3VsdHMsIHR5cGUgPSAiZm9yZWNhc3Rfc3RhYmlsaXR5IiwgdmFsaWRfaW5kaWNlcyA9IGF0dHJpYnV0ZXMoZGF0YV9saXN0KSRyb3dfaW5kaWNlc1sxOjNdKQ0KYGBgDQoNCioqKg0KDQo8YnI+DQoNClRoZSBgZm9yZWNhc3RfdmFyaWFiaWxpdHlgIHBsb3QgYmVsb3cgaXMgYSBzdW1tYXJ5IG9mIHRoZSBgZm9yZWNhc3Rfc3RhYmlsaXR5YCBwbG90LiBJdCdzIGEgcGxvdCANCm9mIHRoZSB2YXJpYWJpbGl0eSBvZiBmb3JlY2FzdHMgZm9yIGEgdGFyZ2V0IHBvaW50IGluIHRpbWUgY29sbGFwc2VkIGFjcm9zcyBmb3JlY2FzdCBob3Jpem9ucy4gQSANCmZvcmVjYXN0IG1vZGVsIHRoYXQgcHJvZHVjZXMgZ3JlYXRlciB2YXJpYWJpbGl0eSBhY3Jvc3MgZm9yZWNhc3QgaG9yaXpvbnMgY291bGQgYmUgdGhlIGJldHRlciBtb2RlbCANCnByb3ZpZGVkIHRoZSBmb3JlY2FzdHMgYXJlIGluY3JlYXNpbmdseSBhY2NydWF0ZSBhdCBzaG9ydGVyIGFuZCBzaG9ydGVyIGZvcmVjYXN0IGhvcml6b25zLg0KDQpgYGB7cn0NCnBsb3QoZGF0YV9yZXN1bHRzLCB0eXBlID0gImZvcmVjYXN0X3ZhcmlhYmlsaXR5IiwgdmFsaWRfaW5kaWNlcyA9IDMwOjgwKQ0KYGBgDQoNCioqKg0KDQo8YnI+DQoNCiMgTW9kZWwgUGVyZm9ybWFuY2UNCg0KIyMgZm9yZWNhc3RNTDo6cmV0dXJuX2Vycm9yDQoNCkxldCdzIGNhbGN1YXRlIHNldmVyYWwgY29tbW9uIGZvcmVjYXN0IGVycm9yIG1ldHJpY3MuDQoNCiogKiptYWU6KiogTWVhbiBhYnNvbHV0ZSBlcnJvcg0KKiAqKm1hcGU6KiogTWVhbiBhYnNvbHV0ZSBwZXJjZW50YWdlIGVycm9yDQoqICoqc21hcGU6KiogU3ltbWV0cmljYWwgbWVhbiBhYnNvbHV0ZSBwZXJjZW50YWdlIGVycm9yIGZyb20gKENoZW4gYW5kIFlhbmcncyAyMDA0IGZvcm11bGEgd2l0aCBhIDEwMCUgbXVsdGlwbGllciBhcyANCmRpc2N1c3NlZCBhdCBbaHR0cHM6Ly9yb2JqaHluZG1hbi5jb20vaHluZHNpZ2h0L3NtYXBlL10oaHR0cHM6Ly9yb2JqaHluZG1hbi5jb20vaHluZHNpZ2h0L3NtYXBlLykpDQoNClRoZSBlcnJvciBmb3IgbmVzdGVkIGNyb3NzLXZhbGlkYXRpb24gYXJlIHJldHVybmVkIGF0IDMgbGV2ZWxzIG9mIGdyYW51bGFyaXR5Og0KDQoxLiBFcnJvciBieSB2YWxpZGF0aW9uIHdpbmRvdw0KMi4gRXJyb3IgYnkgZm9yZWNhc3QgaG9yaXpvbiwgY29sbGFwc2VkIGFjcm9zcyB2YWxpZGF0aW9uIHdpbmRvd3MNCjMuIEdvbGJhbCBlcnJvciBjb2xsYXBzZWQgYWNyb3NzIHZhbGlkYXRpb24gd2luZG93cyBhbmQgaG9yaXpvbnMNCg0KYGBge3J9DQpkYXRhX2Vycm9yIDwtIGZvcmVjYXN0TUw6OnJldHVybl9lcnJvcihkYXRhX3Jlc3VsdHMsIG1ldHJpY3MgPSBjKCJtYWUiLCAibWFwZSIsICJzbWFwZSIpLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbW9kZWxzID0gTlVMTCkNCg0KRFQ6OmRhdGF0YWJsZShkYXRhX2Vycm9yJGVycm9yX2dsb2JhbCwgb3B0aW9ucyA9IGxpc3Qoc2Nyb2xsWCA9IFRSVUUpKQ0KYGBgDQogDQogKioqDQogDQo8YnI+DQogDQpCZWxvdyBpcyBhIHBsb3Qgb2YgZXJyb3IgbWV0cmljcyBhY3Jvc3MgdGltZSBmb3Igc2VsZWN0IHZhbGlkYXRpb24gd2luZG93cyBhbmQgZm9yZWNhc3QgaG9yaXpvbnMuDQoNCmBgYHtyfQ0KcGxvdChkYXRhX2Vycm9yLCBkYXRhX3Jlc3VsdHMsIHR5cGUgPSAidGltZSIsIGhvcml6b25zID0gYygxLCA2LCAxMiksIHdpbmRvd3MgPSAxMDoxNCkNCmBgYA0KDQoqKioNCg0KPGJyPg0KDQpCZWxvdyBpcyBhIHBsb3Qgb2YgZm9yZWNhc3QgZXJyb3IgbWV0cmljcyBmb3IgZWFjaCB2YWxpZGF0aW9uIHdpbmRvdyAobGlnaHQpIGFuZCB0aGUgYXZlcmFnZSBhY3Jvc3MgDQp2YWxpZGF0aW9uIHdpbmRvd3MgKGRhcmspLg0KDQpgYGB7cn0NCnBsb3QoZGF0YV9lcnJvciwgZGF0YV9yZXN1bHRzLCB0eXBlID0gImhvcml6b24iLCBob3Jpem9ucyA9IGMoMSwgNiwgMTIpKQ0KYGBgDQoNCioqKg0KDQo8YnI+DQoNCkJlbG93IGlzIGEgcGxvdCBvZiBlcnJvciBtZXRyaWNzIGNvbGxhcHNlZCBhY3Jvc3MgdmFsaWRhdGlvbiB3aW5kb3dzIGFuZCBmb3JlY2FzdCBob3Jpem9ucy4NCg0KYGBge3J9DQpwbG90KGRhdGFfZXJyb3IsIGRhdGFfcmVzdWx0cywgdHlwZSA9ICJnbG9iYWwiKQ0KYGBgDQoNCioqKg0KDQo8YnI+DQoNCiMgSHlwZXJwYXJhbWV0ZXJzDQoNCldoaWxlIGl0IG1heSBiZSByZWFzb25hYmxlIHRvIGhhdmUgZGlzdGluY3QgbW9kZWxzIGZvciBlYWNoIGZvcmVjYXN0IGhvcml6b24gb3IgZXZlbiBmb3JlY2FzdGluZyBtb2RlbCANCmVuc2VtYmxlcyBhY3Jvc3MgaG9yaXpvbnMsIGF0IHRoaXMgcG9pbnQgd2Ugc3RpbGwgaGF2ZSBzbGlnaHRseSBkaWZmZXJlbnQgTEFTU08gYW5kIFJhbmRvbSBGb3Jlc3QgbW9kZWxzIA0KZnJvbSB0aGUgb3V0ZXIgbG9vcCBvZiB0aGUgbmVzdGVkIGNyb3NzLXZhbGlkYXRpb24gKndpdGhpbiogZWFjaCBob3Jpem9uLXNwZWNpZmljIG1vZGVsLiBIZXJlLCB3ZSdsbCANCnRha2UgYSBsb29rIGF0IHRoZSBzdGFiaWxpdHkgb2YgdGhlIGh5cGVycGFyYW1ldGVycyBmb3IgdGhlIExBU1NPIG1vZGVsIHRvIGJldHRlciB1bmRlcnN0YW5kIGlmIA0Kd2UgY2FuIHRyYWluIG9uZSBtb2RlbCBhY3Jvc3MgZm9yZWNhc3QgaG9yaXpvbnMgb3IgaWYgd2UgbmVlZCBhZGRpdGlvbmFsIHByZWRpY3RvcnMgb3IgbW9kZWxpbmcgDQpzdHJhdGVnaWVzIHRvIGZvcmVjYXN0IHdlbGwgdW5kZXIgdmFyaW91cyBjb25kaXRpb25zIG9yIHRpbWUtc2VyaWVzIGR5bmFtaWNzLg0KDQojIyMgVXNlci1kZWZpbmVkIGh5cGVycGFyYW1ldGVyIGZ1bmN0aW9uDQoNClRoZSBmb2xsb3dpbmcgdXNlci1kZWZpbmVkIGh5cGVycGFyYW1ldGVyIGZ1bmN0aW9uIGlzIG5lZWRlZCBmb3IgZWFjaCBtb2RlbDoNCg0KKiBBIHdyYXBwZXIgZnVuY3Rpb24gdGhhdCB0YWtlcyB0aGUgZm9sbG93aW5nIHBvc2l0aW9uYWwgKiphcmd1bWVudHMqKg0KICAgICogKioxOioqIFRoZSBtb2RlbCByZXR1cm5lZCBmcm9tIHRoZSB1c2VyLWRlZmluZWQgbW9kZWxpbmcgZnVuY3Rpb24uDQoqIGFuZCAqKnJldHVybnMqKiBhIGBkYXRhLmZyYW1lKClgIG9mIHByZWRpY3Rpb25zIHdpdGggMSBjb2x1bW4gZm9yIGVhY2ggZm9yZWNhc3Qgb3V0Y29tZS4NCg0KYGBge3J9DQpoeXBlcl9mdW5jdGlvbiA8LSBmdW5jdGlvbihtb2RlbCkgew0KDQogIGxhbWJkYV9taW4gPC0gbW9kZWwkbGFtYmRhLm1pbg0KICBsYW1iZGFfMXNlIDwtIG1vZGVsJGxhbWJkYS4xc2UNCg0KICBkYXRhX2h5cGVyIDwtIGRhdGEuZnJhbWUoImxhbWJkYV9taW4iID0gbGFtYmRhX21pbiwgImxhbWJkYV8xc2UiID0gbGFtYmRhXzFzZSkNCiAgcmV0dXJuKGRhdGFfaHlwZXIpDQp9DQpgYGANCg0KPGJyPg0KDQojIyBmb3JlY2FzdE1MOjpyZXR1cm5faHlwZXINCg0KQmVsb3cgYXJlIHR3byBwbG90cyB3aGljaCBzaG93IChhKSB1bml2YXJpYXRlIGh5cGVycGFyYW1ldGVyIHZhcmlhYmlsaXR5IGFjcm9zcyB0aGUgdHJhaW5pbmcgZGF0YSANCmFuZCAoYikgdGhlIHJlbGF0aW9uc2hpcCBiZXR3ZWVuIGVhY2ggZXJyb3IgbWV0cmljIGFuZCBoeXBlcnBhcmFtZXRlciB2YWx1ZXMuDQoNCmBgYHtyfQ0KZGF0YV9oeXBlciA8LSBmb3JlY2FzdE1MOjpyZXR1cm5faHlwZXIobW9kZWxfcmVzdWx0cywgaHlwZXJfZnVuY3Rpb24pDQoNCnBsb3QoZGF0YV9oeXBlciwgZGF0YV9yZXN1bHRzLCBkYXRhX2Vycm9yLCB0eXBlID0gInN0YWJpbGl0eSIsIGhvcml6b25zID0gYygxLCA2LCAxMikpDQpwbG90KGRhdGFfaHlwZXIsIGRhdGFfcmVzdWx0cywgZGF0YV9lcnJvciwgdHlwZSA9ICJlcnJvciIsIGMoMSwgNiwgMTIpKQ0KYGBgDQoNCioqKg0KDQo8YnI+DQoNCiMgRm9yZWNhc3QNCg0KIyMgZm9yZWNhc3RNTDo6Y3JlYXRlX2xhZ2dlZF9kZg0KDQpUbyBmb3JlY2FzdCB3aXRoIHRoZSBkaXJlY3QgZm9yZWNhc3RpbmcgbWV0aG9kLCB3ZSBuZWVkIHRvIGNyZWF0ZSBhbm90aGVyIGRhdGFzZXQgb2YgbGFnZ2VkIHByZWRpY3RvcnMuIA0KV2UgY2FuIGRvIHRoaXMgYnkgcnVubmluZyBgY3JlYXRlX2xhZ2dlZF9kZigpYCBhbmQgc2V0dGluZyBgdHlwZSA9ICJmb3JlY2FzdCJgLg0KDQpGb3Igbm9uLWdyb3VwZWQgdGltZS1zZXJpZXMsIHRoaXMgZnVuY3Rpb24gdGFrZXMgdGhlIGxhc3Qgcm93cyBvZiBkYXRhX3RyYWluIGFuZCBjcmVhdGVzIGxhZ2dlZCBwcmVkaWN0b3JzIHRoYXQgYWxsb3cgZm9yZWNhc3RpbmcgZnJvbSANCjEtc3RlcC1haGVhZCB0byAqTiogaG9yaXpvbnMgZm9yIGVhY2ggaG9yaXpvbi1zcGVjaWZpYyBtb2RlbC4gQmVsb3cgaXMgdGhlIGZvcmVjYXN0IGRhdGFzZXQgZm9yIGEgNi1zdGVwLWFoZWFkIGZvcmVjYXN0Lg0KDQpUaGUgYHJvd19udW1iZXJgIGNvbHVtbiBnaXZlcyB0aGUgcm93IGluZGV4IGZvciB0aGUgbW9zdCByZWNlbnQgcGVyaW9kIG9mIGFjdHVhbHMuIEluIHRoZSBleGFtcGxlIGJlbG93LCB0aGUgYHJvd19udW1iZXJgIDE4MCANCnJlcHJlc2VudHMgdGhlIGxhc3Qgcm93IGluIG91ciBtb2RlbC1idWlsZGluZyBkYXRhc2V0IGBkYXRhX3RyYWluYC4NCg0KYGBge3J9DQpkYXRhX2ZvcmVjYXN0X2xpc3QgPC0gZm9yZWNhc3RNTDo6Y3JlYXRlX2xhZ2dlZF9kZihkYXRhX3RyYWluLCB0eXBlID0gImZvcmVjYXN0IiwgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGxvb2tiYWNrID0gbG9va2JhY2ssICBob3Jpem9uID0gaG9yaXpvbnMpDQoNCkRUOjpkYXRhdGFibGUoaGVhZChkYXRhX2ZvcmVjYXN0X2xpc3RbWzZdXSksIG9wdGlvbnMgPSBsaXN0KHNjcm9sbFggPSBUUlVFKSkNCmBgYA0KDQoqKioNCg0KPGJyPg0KDQojIyBGb3JlY2FzdCByZXN1bHRzDQoNClJ1bm5pbmcgdGhlIHByZWRpY3QgbWV0aG9kLCBgcHJlZGljdC5mb3JlY2FzdF9tb2RlbCgpYCwgb24gdGhlIGxhZ2dlZCBwcmVkaWN0b3IgZGF0YXNldCBjcmVhdGVkIA0KYWJvdmUtLXdpdGggYHR5cGUgPSAiZm9yZWNhc3QiYC0tYW5kIHBsYWNpbmcgaXQgaW4gdGhlIGBkYXRhX2ZvcmVjYXN0YCBhcmd1bWVudCBpbiBgcHJlZGljdC5mb3JlY2FzdF9tb2RlbCgpYCBiZWxvdywgcmV0dXJucyANCmEgZGF0YS5mcmFtZSBvZiBmb3JlY2FzdHMgd2l0aCB0aGUgZm9sbG93aW5nIGNvbHVtbnM6DQoNCiogKiptb2RlbDoqKiBVc2VyLWRlZmluZWQgbW9kZWwgbmFtZS4NCiogKiptb2RlbF9mb3JlY2FzdF9ob3Jpem9uOioqIFRoZSBmb3JlY2FzdCBob3Jpem9uIHRoYXQgdGhlIG1vZGVsIHdhcyB0cmFpbmVkIG9uLg0KKiAqKmhvcml6b246KiogRm9yZWNhc3QgaG9yaXpvbiwgcmFuZ2luZyBmcm9tIDEgdG8gbW9kZWxfZm9yZWNhc3RfaG9yaXpvbi4NCiogKip3aW5kb3dfbGVuZ3RoOioqIE51bWJlciBvZiBkYXRhc2V0IHJvd3MgaW4gZWFjaCB2YWxpZGF0aW9uIHdpbmRvdyAocGFydGlhbCB3aW5kb3dzIGhhdmUgdGhlIHVzZXItc3BlY2lmaWVkIHdpbmRvd19sZW5ndGgpLg0KKiAqKndpbmRvd19udW1iZXI6KiogVGhlIHZhbGlkYXRpb24gd2luZG93IG51bWJlci4NCiogKipmb3JlY2FzdF9wZXJpb2Q6KiogVGhlIGRhdGFzZXQgcm93L2RhdGUgZm9yIHRoZSBmb3JlY2FzdC4NCiogKipcPG91dGNvbWVfcHJlZD46KiogVGhlIGZvcmVjYXN0cy4NCg0KQW4gUzMgb2JqZWN0IG9mIGNsYXNzLCBgZm9yZWNhc3RfcmVzdWx0c2AsIGlzIHJldHVybmVkLiBUaGlzIG9iamVjdCB3aWxsIGhhdmUgZGlmZmVyZW50IHBsb3R0aW5nIGFuZCBlcnJvciBtZXRob2RzIHRoYW4gDQp0aGUgYHRyYWluaW5nX3Jlc3VsdHNgIGNsYXNzIGZyb20gZWFybGllci4NCg0KYGBge3J9DQpkYXRhX2ZvcmVjYXN0IDwtIHByZWRpY3QobW9kZWxfcmVzdWx0cywgbW9kZWxfcmVzdWx0c18yLA0KICAgICAgICAgICAgICAgICAgICAgICAgIHByZWRpY3Rpb25fZnVuY3Rpb24gPSBsaXN0KHByZWRpY3Rpb25fZnVuY3Rpb24sIHByZWRpY3Rpb25fZnVuY3Rpb25fMiksIA0KICAgICAgICAgICAgICAgICAgICAgICAgIGRhdGFfZm9yZWNhc3QgPSBkYXRhX2ZvcmVjYXN0X2xpc3QpDQoNCkRUOjpkYXRhdGFibGUoaGVhZChkYXRhX2ZvcmVjYXN0LCAxMCksIG9wdGlvbnMgPSBsaXN0KHNjcm9sbFggPSBUUlVFKSkNCmBgYA0KDQoqKioNCg0KPGJyPg0KDQpCZWxvdyBpcyBhIHBsb3Qgb2YgdGhlIGZvcmVjYXN0cyB2cy4gdGhlIGFjdHVhbHMgZnJvbSBkYXRhX3Rlc3QgZm9yIGVhY2ggbW9kZWwgYXQgc2VsZWN0IGZvcmVjYXN0IGhvcml6b25zLg0KDQpJdCdzIGNsZWFyIGZyb20gdGhlIHBsb3RzIHRoYXQgb3VyIFJhbmRvbSBGb3Jlc3QgbW9kZWwgaXMgcHJvZHVjaW5nIGxlc3MgYWNjdXJhdGUgZm9yZWNhc3RzIGFuZCBpcyBtb3JlIA0Kc2Vuc2l0aXZlIHRvIHRoZSBkYXRhIG9uIHdoaWNoIGl0IHdhcyB0cmFpbmVkLS1wcm9kdWNpbmcgYSBoYW5kZnVsIG9mIGVycmF0aWMgZm9yZWNhc3RzLg0KDQpgYGB7cn0NCnBsb3QoZGF0YV9mb3JlY2FzdCwgZGF0YV9hY3R1YWwgPSBkYXRhX3RyYWluWy0oMToxNTApLCBdLA0KICAgICBhY3R1YWxfaW5kaWNlcyA9IGFzLm51bWVyaWMocm93Lm5hbWVzKGRhdGFfdHJhaW5bLSgxOjE1MCksIF0pKSwNCiAgICAgaG9yaXpvbnMgPSBjKDEsIDYsIDEyKSwgZmFjZXRfcGxvdCA9IGMoIm1vZGVsIiwgIm1vZGVsX2ZvcmVjYXN0X2hvcml6b24iKSkNCg0KcGxvdChkYXRhX2ZvcmVjYXN0LCBkYXRhX2FjdHVhbCA9IGRhdGFfdGVzdCwgDQogICAgIGFjdHVhbF9pbmRpY2VzID0gYXMubnVtZXJpYyhyb3cubmFtZXMoZGF0YV90ZXN0KSksDQogICAgIGZhY2V0X3Bsb3QgPSAibW9kZWwiLCBob3Jpem9ucyA9IGMoMSwgNiwgMTIpKQ0KYGBgDQoNCioqKg0KDQo8YnI+DQoNCiMjIEZvcmVjYXN0IGVycm9yDQoNCkZpbmFsbHksIHdlJ2xsIGxvb2sgYXQgdGhlIGZvcmVjYXN0IGVycm9yIGJ5IGZvcmVjYXN0IGhvcml6b24gZm9yIG91ciB0d28gbW9kZWxzLg0KDQpJZiB0aGUgZmlyc3QgYXJndW1lbnQgb2YgYGZvcmVjYXN0TUw6OnJldHVybl9lcnJvcigpYCBpcyBhbiBvYmplY3Qgb2YgY2xhc3MgYGZvcmVjYXN0X3Jlc3VsdHNgIGFuZCANCnRoZSBgZGF0YV90ZXN0YCBhcmd1bWVudCBpcyBhIGRhdGEuZnJhbWUgbGlrZSBkYXRhX3Rlc3QgZnJvbSBvdXIgYmVnaW5uaW5nIHRyYWluLXRlc3Qgc3BsaXQsIGEgZGF0YS5mcmFtZSANCm9mIGZvcmVjYXN0IGVycm9yIG1ldHJpY3Mgd2l0aCB0aGUgZm9sbG93aW5nIGNvbHVtbnMgaXMgcmV0dXJuZWQ6DQoNCiogKiptb2RlbDoqKiBVc2VyLWRlZmluZWQgbW9kZWwgbmFtZS4NCiogKiptb2RlbF9mb3JlY2FzdF9ob3Jpem9uOioqIFRoZSBmb3JlY2FzdCBob3Jpem9uIHRoYXQgdGhlIG1vZGVsIHdhcyB0cmFpbmVkIG9uLg0KKiAqKmhvcml6b246KiogRm9yZWNhc3QgaG9yaXpvbiwgcmFuZ2luZyBmcm9tIDEgdG8gbW9kZWxfZm9yZWNhc3RfaG9yaXpvbi4NCiogKipcPGVycm9yX21ldHJpY3M+OioqIEZvcmVjYXN0IGVycm9yIG1ldHJpY3MuDQoNCmBgYHtyfQ0KZGF0YV9lcnJvciA8LSBmb3JlY2FzdE1MOjpyZXR1cm5fZXJyb3IoZGF0YV9mb3JlY2FzdCwgZGF0YV90ZXN0ID0gZGF0YV90ZXN0LA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGVzdF9pbmRpY2VzID0gYXMubnVtZXJpYyhyb3cubmFtZXMoZGF0YV90ZXN0KSksDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtZXRyaWNzID0gYygibWFlIiwgIm1hcGUiLCAic21hcGUiLCAibWRhcGUiKSkNCg0KRFQ6OmRhdGF0YWJsZShoZWFkKGRhdGFfZXJyb3IkZXJyb3JfYnlfaG9yaXpvbiwgMTApLCBvcHRpb25zID0gbGlzdChzY3JvbGxYID0gVFJVRSkpDQpgYGANCg0KKioqDQoNCjxicj4NCg0KIyBNb2RlbCBTZWxlY3Rpb24gYW5kIFJlLXRyYWluaW5nDQoNCkJlY2F1c2Ugb3VyIExBU1NPIG1vZGVsIGlzIGJvdGggc3RhYmxlciBhbmQgbW9yZSBhY2N1cmF0ZSwgd2UnbGwgcmUtdHJhaW4gb3VyIG1vZGVsIGFjcm9zcyANCnRoZSBlbnRpcmUgdHJhaW5pbmcgZGF0YXNldCB0byBnZXQgb3VyIGZpbmFsIGByIGxlbmd0aChob3Jpem9ucylgIG1vZGVscy0tMSBmb3IgZWFjaCBmb3JlY2FzdCBob3Jpem9uLiANCk5vdGUgdGhhdCBmb3IgYSByZWFsLXdvcmxkIGZvcmVjYXN0aW5nIHByb2JsZW0gdGhpcyBpcyB3aGVuIHdlIHdvdWxkIGRvIGFkZGl0aW9uYWwgbW9kZWwgdHVuaW5nIA0KdG8gaW1ycG92ZSBmb3JlY2FzdCBhY2N1cmFjeSBhY3Jvc3MgdmFsaWRhdGlvbiB3aW5kb3dzIGFzIHdlbGwgYXMgbmFycm93IHRoZSBoeXBlcnBhcmFtZXRlciBzZWFyY2ggDQppbiB0aGUgdXNlci1zcGVjaWZpZWQgbW9kZWxpbmcgZnVuY3Rpb25zLg0KDQojIyBmb3JlY2FzdE1MOjpjcmVhdGVfbGFnZ2VkX2RmDQoNCmBgYHtyfQ0KZGF0YV9saXN0IDwtIGZvcmVjYXN0TUw6OmNyZWF0ZV9sYWdnZWRfZGYoZGF0YV90cmFpbiwgdHlwZSA9ICJ0cmFpbiIsIGxvb2tiYWNrID0gbG9va2JhY2ssIA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaG9yaXpvbiA9IGhvcml6b25zKQ0KYGBgDQoNCjxicj4NCg0KIyMgZm9yZWNhc3RNTDo6Y3JlYXRlX3dpbmRvd3MNCg0KVG8gY3JlYXRlIGEgZGF0YXNldCAqd2l0aG91dCBuZXN0ZWQgY3Jvc3MtdmFsaWRhdGlvbiosIHNldCBgd2luZG93X2xlbmd0aCA9IDBgIGluIGBmb3JlY2FzdE1MOjpjcmVhdGVfd2luZG93cygpYC4NCg0KYGBge3J9DQp3aW5kb3dzIDwtIGZvcmVjYXN0TUw6OmNyZWF0ZV93aW5kb3dzKGRhdGFfbGlzdCwgd2luZG93X2xlbmd0aCA9IDApDQoNCnBsb3Qod2luZG93cywgZGF0YV9saXN0LCBzaG93X2xhYmVscyA9IFRSVUUpDQpgYGANCg0KKioqDQoNCjxicj4NCg0KIyMgZm9yZWNhc3RNTDo6dHJhaW5fbW9kZWwNCg0KV2l0aG91dCBuZXN0ZWQgY3Jvc3MtdmFsaWRhdGlvbiBhbmQgaG9sZG91dCB3aW5kb3dzLCB0aGUgcHJlZGljdGlvbiBwbG90IGlzIGVzc25ldGlhbGx5IGEgcGxvdCBvZiBtb2RlbCBmaXQuDQoNCmBgYHtyfQ0KbW9kZWxfcmVzdWx0cyA8LSBmb3JlY2FzdE1MOjp0cmFpbl9tb2RlbChkYXRhX2xpc3QsIHdpbmRvd3MsIG1vZGVsX2Z1bmN0aW9uLCBtb2RlbF9uYW1lID0gIkxBU1NPIikNCg0KZGF0YV9yZXN1bHRzIDwtIHByZWRpY3QobW9kZWxfcmVzdWx0cywgcHJlZGljdGlvbl9mdW5jdGlvbiA9IGxpc3QocHJlZGljdGlvbl9mdW5jdGlvbikpDQoNCkRUOjpkYXRhdGFibGUoaGVhZChkYXRhX3Jlc3VsdHMsIDEwKSwgb3B0aW9ucyA9IGxpc3Qoc2Nyb2xsWCA9IFRSVUUpKQ0KcGxvdChkYXRhX3Jlc3VsdHMsIHR5cGUgPSAicHJlZGljdGlvbiIsIGhvcml6b25zID0gYygxLCA2LCAxMikpDQpwbG90KGRhdGFfcmVzdWx0cywgdHlwZSA9ICJyZXNpZHVhbCIsIGhvcml6b25zID0gYygxLCA2LCAxMikpDQpwbG90KGRhdGFfcmVzdWx0cywgdHlwZSA9ICJmb3JlY2FzdF9zdGFiaWxpdHkiLCB2YWxpZF9pbmRpY2VzID0gMTA5OjEyMCkNCmBgYA0KDQoqKioNCg0KPGJyPg0KDQojIyBmb3JlY2FzdE1MOjpyZXR1cm5fZXJyb3INCg0KYGBge3J9DQpkYXRhX2Vycm9yIDwtIGZvcmVjYXN0TUw6OnJldHVybl9lcnJvcihkYXRhX3Jlc3VsdHMsIG1ldHJpY3MgPSBjKCJtYWUiLCAibWFwZSIsICJtZGFwZSIsICJzbWFwZSIpLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbW9kZWxzID0gTlVMTCkNCg0KRFQ6OmRhdGF0YWJsZShoZWFkKGRhdGFfZXJyb3IkZXJyb3JfZ2xvYmFsKSwgb3B0aW9ucyA9IGxpc3Qoc2Nyb2xsWCA9IFRSVUUpKQ0KcGxvdChkYXRhX2Vycm9yLCBkYXRhX3Jlc3VsdHMsIHR5cGUgPSAiaG9yaXpvbiIpDQpgYGANCg0KKioqDQoNCiMjIGZvcmVjYXN0TUw6OnJldHVybl9oeXBlcg0KDQpgYGB7cn0NCmRhdGFfaHlwZXIgPC0gZm9yZWNhc3RNTDo6cmV0dXJuX2h5cGVyKG1vZGVsX3Jlc3VsdHMsIGh5cGVyX2Z1bmN0aW9uKQ0KDQpwbG90KGRhdGFfaHlwZXIsIGRhdGFfcmVzdWx0cywgZGF0YV9lcnJvciwgdHlwZSA9ICJzdGFiaWxpdHkiLCBob3Jpem9ucyA9IGMoMSwgNiwgMTIpKQ0KcGxvdChkYXRhX2h5cGVyLCBkYXRhX3Jlc3VsdHMsIGRhdGFfZXJyb3IsIHR5cGUgPSAiZXJyb3IiLCBjKDEsIDYsIDEyKSkNCmBgYA0KDQoqKioNCg0KPGJyPg0KDQojIyBGb3JlY2FzdA0KDQpgYGB7cn0NCmRhdGFfZm9yZWNhc3RfbGlzdCA8LSBmb3JlY2FzdE1MOjpjcmVhdGVfbGFnZ2VkX2RmKGRhdGFfdHJhaW4sIHR5cGUgPSAiZm9yZWNhc3QiLCANCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbG9va2JhY2sgPSBsb29rYmFjaywgIGhvcml6b24gPSBob3Jpem9ucykNCg0KZGF0YV9mb3JlY2FzdCA8LSBwcmVkaWN0KG1vZGVsX3Jlc3VsdHMsIHByZWRpY3Rpb25fZnVuY3Rpb24gPSBsaXN0KHByZWRpY3Rpb25fZnVuY3Rpb24pLCANCiAgICAgICAgICAgICAgICAgICAgICAgICBkYXRhX2ZvcmVjYXN0ID0gZGF0YV9mb3JlY2FzdF9saXN0KQ0KDQpwbG90KGRhdGFfZm9yZWNhc3QsIGRhdGFfYWN0dWFsID0gZGF0YVstKDE6MTUwKSwgXSwNCiAgICAgYWN0dWFsX2luZGljZXMgPSBhcy5udW1lcmljKHJvdy5uYW1lcyhkYXRhWy0oMToxNTApLCBdKSksDQogICAgIGhvcml6b25zID0gYygxLCA2LCAxMiksIA0KICAgICBmYWNldF9wbG90ID0gYygibW9kZWwiLCAibW9kZWxfZm9yZWNhc3RfaG9yaXpvbiIpKSArIGdncGxvdDI6OnRoZW1lKGxlZ2VuZC5wb3NpdGlvbiA9ICJub25lIikNCg0KcGxvdChkYXRhX2ZvcmVjYXN0LCBkYXRhX2FjdHVhbCA9IGRhdGFfdGVzdCwgYWN0dWFsX2luZGljZXMgPSBhcy5udW1lcmljKHJvdy5uYW1lcyhkYXRhX3Rlc3QpKSwNCiAgICAgZmFjZXRfcGxvdCA9IE5VTEwsIGhvcml6b25zID0gYygxLCA2LCAxMikpDQpgYGANCg0KKioqDQoNCjxicj4NCg0KIyMgRm9yZWNhc3QgZXJyb3INCg0KYGBge3J9DQpkYXRhX2Vycm9yIDwtIGZvcmVjYXN0TUw6OnJldHVybl9lcnJvcihkYXRhX2ZvcmVjYXN0LCBkYXRhX3Rlc3QgPSBkYXRhX3Rlc3QsIA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGVzdF9pbmRpY2VzID0gYXMubnVtZXJpYyhyb3cubmFtZXMoZGF0YV90ZXN0KSksDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtZXRyaWNzID0gYygibWFlIiwgIm1hcGUiLCAibWRhcGUiLCAic21hcGUiKSkNCg0KRFQ6OmRhdGF0YWJsZShkYXRhX2Vycm9yJGVycm9yX2J5X2hvcml6b24sIG9wdGlvbnMgPSBsaXN0KHNjcm9sbFggPSBUUlVFKSkNCkRUOjpkYXRhdGFibGUoZGF0YV9lcnJvciRlcnJvcl9nbG9iYWwsIG9wdGlvbnMgPSBsaXN0KHNjcm9sbFggPSBUUlVFKSkNCmBgYA0KDQoqKioNCg==

package::forecastML Overview

Nick Redell, nickredell@hotmail.com

2019-06-24

Purpose

Direct forecasting

Package Features

Example

Install forecastML

Load packages and data

Train-test split

Data Preparation

Nested cross-validation

Model Training

User-defined forecast function

forecastML::train_model

User-defined prediction function

forecastML::predict

Model Performance

forecastML::return_error

Hyperparameters

User-defined hyperparameter function

forecastML::return_hyper

Forecast

forecastML::create_lagged_df

Forecast results

Forecast error

Model Selection and Re-training

forecastML::create_lagged_df

forecastML::create_windows

forecastML::train_model

forecastML::return_error

forecastML::return_hyper

Forecast

Forecast error