python - What's the best way to serialize a large scipy sparse matrix? -

-

I have a large SPA sparse matrix, which is taking 90% of my total system memory; I would like to save it to disk, because it takes a few hours to create the matrix ...

I have tried cPickle, but it leads to a major memory explosion ... < Pre> as import nppy as np scipy.sparse import lil_matrix import cPickle dim = 10 ** 8m = lil_matrix ((dim, dim), dtype = np.float) with open (file name , 'Wb') as f: cpickle. Dump (M, F) leads to a major memory explosion, is probably being copied to a lot

while HDF 5 did not like datatype: TypeError: Object DTP DTEP ('O')

This is not a native HDF5 equivalent

actually How much data is stored in the matrix? Have you seen the matrix type change before serialization?

LIL matrix is ​​not the most memory efficient sparse matrix available to you. For example: [43]: In dim = 10 ** 6 [44] For example: [49]: m = lil_matrix ((slow, slow), dtype = np.float) In [45]: For category (10000): M [np.random.uniform (0, dim), np.random.uniform (0, dim)] = 1 in [46]: lane (cPickle.dumps ( M.Dodok ()) [46]: 1256302 in [47]: lane (cPickle.dumps (mt) ()) out [47]: compared to # 557691 # [48]: lane (cPickle.dumps (M)) Out [48]: 23018393

These formats do not support the same set of all operations, but the conversion between formats is trivial.

Comments

Post a Comment

Popular posts from this blog

ios - Adding an SKSpriteNode to SKScene from a child SKSpriteNode -

-
I have a SKScene where I am adding an SKSpriteNode I have subclassed the SKSpriteNode class to create this node. In the subclass I am defining some SKActions on Sprite. What do I want to do when the end of this phantom ends the SKAction sequence, then I add a new Prayer node to this scene. How is this possible. The following code is mine: The code for the sequence is that I'm running on the Scansprinitode subclass (TEMissileNode): - SKAction * moveDown = [SKAction moveToY: Self.position.y - 20 Duration: 0.2]; SKAction * animation = [SKAction animateWithTextures: textures timePerFrame: time / 7]; SKAction * moveMissileProjectile = [SKAction Moving: Pointoffscreen Term: Time]; SKAction * group = [SKAction group: @ [animation, movement projectile]]; SKAction * Sequence = [SKAction Sequence: @ [Hilldown, Group, [SKAction removalFromParent]]]; [Self run action: sequence]; From the main scene, I call those actions that execute these tasks TEMissileNode * missile = [tmissil...
Read more

Matlab transpose a table vector -

-
An embarrassing simple question seems to be, but how can I move a Matlab table vector? aTableT = aTable '; I tried a standard syntax for the simple interaction of a line vector to a line vector aTable : ATableT = reshape (aTable, 1, Height (aTable)); and aTableT = rot90 (aTable); According to the head, the last time table should work for table, see. However, I get this error code: Error type in using table / permit (line 396) Undefined function 'permute' for the input arguments. Error in rot 90 (line 29) b = magnitude (b, [2 1 3: ndims (a)]); NB: fliplr is not useful either pretty sure I have covered clear angles - any ideas? Thanks! Try changing your table into an array, move it, then back to a table In other words, try doing this: aTableArray = table2array (aTable); ATableT = array 2table (aTableArray. '); I have also read the document for rot90 , and it says that rot90 is definitely working for tables Do, and I find you...
Read more

c# - Textbox not clickable but editable -

-
I have a small form with 10 text boxes, I have set them in the correct tab order. I want them on my tab service I was wondering if there is a way to set the text box so that they can be selected for editing until they are tabbed. That is ... I do not want the end user to edit the text box to click on them, I just want to make them editable through tabu. this should do the trick public partial category poor text box: Text Box {Safe Override Zero WndProc (Ref: Message Message) {if (m.msg == (Int) WM.LBUTTONDOWN) {Return; // mount down events} base.WndProc (ref m); }} Window message enum can be found. How to do this without anherizing: text box : class EatMouseDown: netwindows {safe override zero WndProc (ref message message) {if (m.msg == (int) WM.LBUTTONDOWN) {return; } Base.WandProc (Ref M); }} Safe Override Zero Onload (EventEurge E) {base.OnLoad (e); How to do this without any part: left a clean part, whatever is important, this can be a buggy but it works R...
Read more