Memory error

[ZohrehFattahi]

Hi,

I am trying to run the exome pipeline.
When I run it for 2 or 3 samples it runs correctly.
But, when I run a command for about 20 samples I get the following error.

This is the command:
/data/NGS/Reanalysis-Package/STRetch-master/tools/bin/bpipe run -p ./Twist_Exome_Core_Covered_Targets_hg19_liftover.bed *.fastq.gz

Here is the error:
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:109)
at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:131)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
at java.lang.StringBuilder.append(StringBuilder.java:190)
at org.codehaus.groovy.runtime.IOGroovyMethods.getText(IOGroovyMethods.java:886)
at org.codehaus.groovy.runtime.ResourceGroovyMethods.getText(ResourceGroovyMethods.java:600)
at org.codehaus.groovy.runtime.dgm$962.doMethodInvoke(Unknown Source)
at org.codehaus.groovy.reflection.GeneratedMetaMethod$Proxy.doMethodInvoke(GeneratedMetaMethod.java:73)
at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty.java:76)
at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:64)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:296)
at bpipe.Runner.loadPipelineSrc(Runner.groovy:662)
at bpipe.Runner$loadPipelineSrc.callStatic(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:194)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:222)
at bpipe.Runner.main(Runner.groovy:377)
... 6 more

It seems that the error is related to memory.
I am running on a server with 264g memory. Can you please guide me on how I can solve it?

Zohreh

[hdashnow]

Hi Zohreh,

bpipe (the workflow software running STRetch) should not use substantial memory on 20 samples. I've successfully run it on thousands of samples without issue.

Here are a few things you could try:

• Specify the amount of memory for Java, like this: MAX_JAVA_MEM=24g /data/NGS/Reanalysis-Package/STRetch-master/tools/bin/bpipe run...
• Check that you are running the command in a high-memory environment. For example, many clusters are set up to have a limited memory "head node" where you submit jobs, that are then sent to a higher memory environment. On Linux, check memory with grep MemTotal /proc/meminfo
• Check the latest version of bpipe is being used, and if not, update it. Check the version by running just bpipe without any thing else. In your system: /data/NGS/Reanalysis-Package/STRetch-master/tools/bin/bpipe. You can check/download the latest version here: https://github.com/ssadedin/bpipe/releases/. Simply replace the copy in /data/NGS/Reanalysis-Package/STRetch-master/tools/bin/bpipe.

You can read more about memory usage in bpipe here:
ssadedin/bpipe#97

Let me know if these suggestions worked or not. It could be useful for other users.

Warm regards,
Harriet