|
From: | Markus Appel |
Subject: | Re: Shared memory between parallel threads |
Date: | Mon, 28 Apr 2014 01:54:59 +0200 |
User-agent: | Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 |
On 04/22/2014 04:58 PM, Anton Flugge wrote:
Hi, I am trying to run calls to the kmeans function in parallel. In principal that works well with parcellfun in the parallel toolbox, however, the matrix that I want to use kmeans on is quite large (a couple of GB), and parcellfun creates separate copies of the workspace/variables for each parallel thread, which means I run out of memory quicker than I run out of processor cores. Now kmeans only needs read access, and no write access, to the input data, which means in principal it shouldn't be a problem to share the same matrix/memory between all threads, however, I could not find a solution anywhere on how to do that in Octave... is it possible to have global variables accessed by multiple parallel threads? Thanks, Anton
Hi Anton,parcellfun uses the fork command to create child processes. I am not an expert in this, but from what I find is that fork should use a copy-on-write strategy for the process memory and not blindly copy everything. I ran a small experiment on my Ubuntu 12.04 with the Octave development version:
% create a ~800MB matrix A=rand(1e4); % calculate the sum 32 times using cellfun - no change in memory usage! cellfun(@(a) sum(a(:)) , {A}(ones(1,32)) ) % calculate the sum 32 times using parcellfun % still no change in memory usage! parcellfun(8, @(a) sum(a(:)) , {A}(ones(1,32)) )When executing the line with parcellfun, 8 processes are created and it takes some seconds on my poor old laptop, but I see no change in memory usage, so the copy-on-write is definitely working for me.
So either this is not working for you, or it is not your input matrix but stuff created in the kmeans function (I am not familiar with this particular function) which eats up your memory.
Hope it helps, Markus
[Prev in Thread] | Current Thread | [Next in Thread] |