[ENH] Assessing Performance

[samihamdan]

Problem
I noticed that sometimes performance of julearn seems to be not that great. I am not sure whether this is a real trend or just normal frustration with the speed of ml.

Solution
I am not sure whether this is actually a problem, but it would be nice to in general assess performance to keep and eye on how much overhead we add to sklearn. Even if we do not change the speed it is good to make realistic expectations of potential users.

Considerations
How does it change with more data or more transformers.
It could be that each transformation from np.array to pd.DataFrame has a big impact. On the other hand the implementation of confound removal could also be the reason for long computation times in real world observations.

Screenshot
I did one very simple observation with only one transformer.
If I use 4x of the data I still have a similar 3x worse performance of julearn.

[fraimondo]

What about the same test using the numpy version of julearn API?

…

On Fri, 7 May 2021, 14:00 Sami Hamdan, ***@***.***> wrote: *Problem* I noticed that sometimes performance of julearn seems to be not that great. I am not sure whether this is a real trend or just normal frustration with the speed of ml. *Solution* I am not sure whether this is actually a problem, but it would be nice to in general assess performance to keep and eye on how much overhead we add to sklearn. Even if we do not change the speed it is good to make realistic expectations of potential users. *Considerations* How does it change with more data or more transformers. It could be that each transformation from np.array to pd.DataFrame has a big impact. On the other hand the implementation of confound removal could also be the reason for long computation times in real world observations. *Screenshot* I did one very simple observation with only one transformer. If I use 4x of the data I still have a similar 3x worse performance of julearn. [image: image] <https://user-images.githubusercontent.com/44375312/117445735-5477ca00-af3b-11eb-8242-98078bcbb696.png> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#134>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCJDA63NEPVLRM6S6J4CZ3TMPI4NANCNFSM44JVLTPQ> .

[samihamdan]

For this simplistic example no difference between both julearn APIs.
But good point for the general assessment of speed.

[fraimondo]

What about adding some benchmark tests to compare if a PR makes a huge mess with performance? Can it be done using CI? Maybe @synchon can help with this one.

[synchon]

I'll take a look.

[fraimondo]

@synchon will take a look at it soon