Share via

rxSummary: Object Summaries

  • Article

Description

Produce univariate summaries of objects in RevoScaleR.

Usage

  rxSummary(formula, data, byGroupOutFile = NULL, 
            summaryStats = c("Mean", "StdDev", "Min", "Max", "ValidObs", "MissingObs"),
            byTerm = TRUE, pweights = NULL, fweights = NULL, rowSelection = NULL,
            transforms = NULL, transformObjects = NULL,
            transformFunc = NULL, transformVars = NULL,
            transformPackages = NULL, transformEnvir = NULL, 
            overwrite = FALSE,
            useSparseCube = rxGetOption("useSparseCube"),
            removeZeroCounts = useSparseCube,         
            blocksPerRead = rxGetOption("blocksPerRead"),
            rowsPerBlock = 100000,
            reportProgress = rxGetOption("reportProgress"), verbose = 0,
            computeContext = rxGetOption("computeContext"), ...)

Arguments

formula

formula, as described in rxFormula. The formula typically does not contain a response variable, i.e. it should be of the form ~ terms. If ~. is used as the formula, summary statistics will be computed for all non-character variables. If a numeric variable is interacted with a factor variable, summary statistics will be computed for each category of the factor.

data

either a data source object, a character string specifying a .xdf file, or a data frame object to summarize.

byGroupOutFile

NULL, a character string or vector of character strings specifying .xdf file names(s), or an RxXdfData object or list of RxXdfData objects. If not NULL, and the formula includes computations by factor, the by-group summary results will be written out to one or more .xdf files. If more than one .xdf file is created and a single character string is specified, an integer will be appended to the base byGroupOutFile name for additional file names. The resulting RxXdfData objects will be listed in the categorical component of the output object. byGroupOutFile is not supported when using distributed compute contexts such as RxHadoopMR.

summaryStats

a character vector containing one or more of the following values: "Mean", "StdDev", "Min", "Max", "ValidObs", "MissingObs", "Sum".

byTerm

logical variable. If TRUE, missings will be removed by term (by variable or by interaction expression) before computing summary statistics. If FALSE, observations with missings in any term will be removed before computations.

pweights

character string specifying the variable to use as probability weights for the observations.

fweights

character string specifying the variable to use as frequency weights for the observations.

rowSelection

name of a logical variable in the data set (in quotes) or a logical expression using variables in the data set to specify row selection. For example, rowSelection = "old" will use only observations in which the value of the variable old is TRUE. rowSelection = (age > 20) & (age < 65) & (log(income) > 10) will use only observations in which the value of the age variable is between 20 and 65 and the value of the log of the income variable is greater than 10. The row selection is performed after processing any data transformations (see the arguments transforms or transformFunc). As with all expressions, rowSelection can be defined outside of the function call using the expression function.

transforms

an expression of the form list(name = expression, ...) representing the first round of variable transformations. As with all expressions, transforms (or rowSelection) can be defined outside of the function call using the expression function.

transformObjects

a named list containing objects that can be referenced by transforms, transformsFunc, and rowSelection.

transformFunc

variable transformation function. See rxTransform for details.

transformVars

character vector of input data set variables needed for the transformation function. See rxTransform for details.

transformPackages

character vector defining additional R packages (outside of those specified in rxGetOption("transformPackages")) to be made available and preloaded for use in variable transformation functions, e.g., those explicitly defined in RevoScaleR functions via their transforms and transformFunc arguments or those defined implicitly via their formula or rowSelection arguments. The transformPackages argument may also be NULL, indicating that no packages outside rxGetOption("transformPackages") will be preloaded.

transformEnvir

user-defined environment to serve as a parent to all environments developed internally and used for variable data transformation. If transformEnvir = NULL, a new "hash" environment with parent baseenv() is used instead.

overwrite

logical value. If TRUE, an existing byGroupOutFile will be overwritten. overwrite is ignored byGroupOutFile is NULL.

useSparseCube

logical value. If TRUE, sparse cube is used.

removeZeroCounts

logical flag. If TRUE, rows with no observations will be removed from the output for counts of categorical data. By default, it has the same value as useSparseCube. For large summary computation, this should be set to TRUE, otherwise R may run out of memory even if the internal C++ computation succeeds.

blocksPerRead

number of blocks to read for each chunk of data read from the data source.

rowsPerBlock

maximum number of rows to write to each block in the byGroupOutFile (if it is not NULL).

reportProgress

integer value with options:

  • 0: no progress is reported.
  • 1: the number of processed rows is printed and updated.
  • 2: rows processed and timings are reported.
  • 3: rows processed and all timings are reported.

verbose

integer value. If 0, no additional output is printed. If 1, additional summary information is printed.

computeContext

a valid RxComputeContext. The RxSpark, and RxHadoopMR compute contexts distribute the computation among the nodes specified by the compute context; for other compute contexts, the computation is distributed if possible on the local computer.

...

additional arguments to be passed directly to the Revolution Compute Engine.

Details

Special function F() can be used in formula to force a variable to be interpreted as factors.

If the formula contains a single dependent or response variable, summary statistics are computed for the interaction between that variable and the first term of the independent variables. (Multiple response variables are not permitted.) For example, using the formula y ~ xfac will give the same results as using the formula ~y:xfac, where y is a continuous variable and xfac is a factor. Summary statistics for y are computed for each factor level of x. This facilitates using the same formula in rxSummary as in, for example, rxCube or rxLinMod.

Value

an rxSummary object containing the following elements:

nobs.valid

number of valid observations.

nobs.missing

number of missing observations.

sDataFrame

data frame containing summaries for continuous variables.

categorical

list of summaries for categorical variables.

categorical.type

types of categorical summaries: can be "counts", or "cube" (for crosstab counts) or "none" (if there is no categorical summaries).

formula

formula used to obtain the summary.

Author(s)

Microsoft Corporation Microsoft Technical Support

See Also

rxTransform

Examples


 # Create a local data frame
 DF <- data.frame(sex = c("Male", "Male", "Female", "Male"),
                  age = c(20, 20, 12, 10), score = 1.1:4.1)

 # get summary of sex variable
 rxSummary(~ sex, DF)

 # obtain within sex-category statistics of the score variable
 rxSummary(score ~ sex, DF)

 # use transforms to create a factor variable and compute
 # summary statistics by each factor level
 rxSummary(~score:ageGroup, data=DF, 
           transforms = list(ageGroup = cut(age, seq(0, 30, 10))))

 # the following will give the same results
 rxSummary(score~ageGroup, data=DF, 
           transforms = list(ageGroup = cut(age, seq(0, 30, 10))))

 # the same formula can be used in rxCube          
 rxCube(score~ageGroup, data=DF, transforms=list(ageGroup = cut(age, seq(0,30,10))))

 # Write summary statistics by group to an .xdf file.  Here the groups 
 # are defined as year of age by sex by state (3 states in CensusWorkers file), 
 # so summary statistics for 46 x 2 x 3 groups are computed. The first term
 # will just compute the Counts for each group, while the second two will
 # compute by-group Means and ValidObs for incwage and wkswork1

 censusWorkers <- file.path(rxGetOption("sampleDataDir"), "CensusWorkers.xdf")
 sumOutFile <-  tempfile(pattern = ".rxTempSumOut", fileext = ".xdf")

 sumOut <- rxSummary(~F(age):sex:state + incwage:F(age):sex:state + wkswork1:F(age):sex:state, 
       data = censusWorkers, blocksPerRead = 3,
       byGroupOutFile = sumOutFile, rowsPerBlock = 10, summaryStats = c("Mean", "ValidObs"))

 rxGetVarInfo(sumOutFile)

 file.remove(sumOutFile)