emacs-orgmode
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [O] Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes w


From: Charles C. Berry
Subject: Re: [O] Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel
Date: Thu, 18 Jun 2015 13:20:18 -0700
User-agent: Alpine 2.11 (OSX 23 2013-08-11)

On Thu, 18 Jun 2015, Rainer M Krug wrote:

"Charles C. Berry" <address@hidden> writes:

On Wed, 17 Jun 2015, William Denton wrote:

On 17 June 2015, Xebar Saram wrote:

I do alot of modeling work that involves using huge datasets and run
process intensive R processes (such as complex mixed models, Gamms etc). in
R studio all works well yet when i use the orgmode eval on R code blocks it
works well for small simple process but 90% of the time when dealing with
complex models and bug data (up to 256GB) it will just freeze emacs/ess.
sometimes i can C-c or C-g it and other times i need to physically kill
emacs.

I've been having the same problem for a while, but wasn't able to
isolate it any more than large data sets, lack of memory, and heavy
CPU usage. Sometimes everything hangs and I need to power cycle the
computer. :(


And you (both) have `ess-eval-visibly' set to nil, right?


[snip: ox-ravel and how it might solve OP's problem]


I have a similar workflow, only that I use a package like
approach, i.e. I tangle function definitions in a folder ./R, data into
./data (which makes it possible to share org defined variables with R
running outside org) and scripts, i.e. the things which do a analysis,
import data, ... i.e. which might take long, into a folder ./scripts/. I
then add the usual R package infrastructure files (DESCRIPTION,
NAMESPACE, ...).
Then I have one file tangled into ./scripts/init.R:

[snip: how and why to structure an analysis as an R package]


I am using this approach at the moment for a paper and which will also
result in a paper. By executing all the scripts, one will be able to do
import the raw data, do the analysis and create all graphs used in the
paper.


And by writing the paper in the form of a vignette that Sweave or knitr can render you have an R package that when installed processes the data and reproduces the paper in pdf format.

ox-ravel will produce that vignette from Org mode.

For example, the bioConductor package geneRxCluster [1] comes from an Org mode document that contains the C and R code as src blocks and a subtree with the vignette (Using geneRxCluster) that analyzes data, produces graphics, etc. tangle-ing the src blocks and exporting the vignette creates the package.

HTH,

Chuck

[1] http://www.bioconductor.org/packages/geneRxCluster



reply via email to

[Prev in Thread] Current Thread [Next in Thread]