Revised chunking implemented for Issue #67 for improved memory management #106
Conversation
Main change: - Created chunking logic to call the classifier with a maximum number of tests (detailed description as code comment). In addition the following change was made: - Replaced the try/catch with an explicit check for the available function either decision_function or predict_proba.
|
For details of history see PR #105 |
mglearn/plot_2d_separator.py
Outdated
| y_chunk_pos = 0 | ||
| for x_chunk in np.array_split(X, np.arange(chunk_size,X_axis0_size,chunk_size,dtype=np.int32), axis=0): | ||
| Y_result_chunks.append(classifier_pred_or_decide(x_chunk)) | ||
| y_chunk_pos += x_chunk.shape[0] |
There was a problem hiding this comment.
this variable is not used, is it?
There was a problem hiding this comment.
Correct; and I inlined the
X_axis0_size
removing this as well.
mglearn/plot_2d_separator.py
Outdated
| # MLPClassifier(solver='lbfgs', random_state=0, hidden_layer_sizes=[1000,1000,1000]) | ||
| # by reducing the value it is possible to trade in time for memory. | ||
| # It is possible to chunk the array as the calculations are independent of each other. | ||
| # Note: an intermittent version made a distinction between 32- and 64 bit architectures |
There was a problem hiding this comment.
Not sure if the note is necessary but also not opposed.
There was a problem hiding this comment.
I think it is capturing a rationale from the past, so if you do not mind, I leave it.
mglearn/plot_2d_separator.py
Outdated
|
|
||
| # Call the classifier in chunks. | ||
| y_chunk_pos = 0 | ||
| for x_chunk in np.array_split(X, np.arange(chunk_size,X_axis0_size,chunk_size,dtype=np.int32), axis=0): |
There was a problem hiding this comment.
can you please adhere to pep8? So spaces after , and not more than 79 chars per line.
But otherwise looks good, thank you!
There was a problem hiding this comment.
Sure thanks for the patience - I just should have "listened" to my IDE.
There was a problem hiding this comment.
well thanks for being patient with my nit-picks ;)
Minor refinements due PR feedback - Removed unnecessary variables and inlined one-time used variable. - Re-introduced the originally intended solver solver='lbfgs' - Adhered to PEP8, breaking lines and comments accordingly
|
awesome, thanks for your help :) |
This closes issue #67 memory error on 32bit Python
Main change:
In addition the following change was made: