Hello all,
I'm trying to pickle a very large graph (23 million vertices, 152 million edges, two vertex attributes), but keep getting an `OSError: [Errno 22] Invalid argument` error. However, I think that's erroneous, as if I subsample the graph and save with exact same code I have no problems. Here's the traceback:
g.summary()
Out[8]: 'IGRAPH UN-- 23331862 152099394 -- \n+ attr: name (v), pt (v)'
g.write_pickle(fname='graphs/with_inferred/vz_inferred3_sms{}_voz{}.pkl'.format(sms, voz))
Traceback (most recent call last):
File "<ipython-input-9-6b5409a79251>", line 1, in <module>
g.write_pickle(fname='graphs/with_inferred/vz_inferred3_sms{}_voz{}.pkl'.format(sms, voz))
File "/Users/Nick/anaconda/lib/python3.5/site-packages/igraph/__init__.py", line 1778, in write_pickle
result=pickle.dump(self, fname, version)
OSError: [Errno 22] Invalid argument
g=g.vs[range(3331862)].subgraph()
g.write_pickle(fname='graphs/with_inferred/vz_inferred3_sms{}_voz{}.pkl'.format(sms, voz))
[success]
The graph takes up about 10gb in memory, and the pickle command expands Python's memory footprint to about 35gb before the exception gets thrown, but I'm on a machine with 80gb ram, so that's not the constraint.
Any suggestions as to what might be going on / is there a work around for saving?
Thanks!
Nick