Issues with default value - parameter - over 1024 bytes ...

[vangalamaheshh]

Hi @jbingham -

"error": {
"code": 400,
"message": "Invalid value for field "parameter.defaultValue": too long: 1471 bytes, want no longer than 1024 bytes",
"status": "INVALID_ARGUMENT"
}

Just so you know, when the defaultValue exceeds 1024 bytes, it will result in pipeline error. Do you have tips how to overcome this? I can hack for now by passing other parameters and constructing the full paths of the files in the component container, but I am curious if we could fix this in a cleaner way.

Let me know.

Thanks,
Mahesh

[vangalamaheshh]

@jbingham :

Little more info on this:

https://cloud.google.com/genomics/reference/rest/v1alpha2/pipelines#PipelineParameter

The object(PipelineParameter) - defaultValue | string, is where the constraint of len(string) should be less than 1024 bytes. And this is what causing my pipeline to bail.

My use case is that, I have few hundred samples and I am passing in to a component a list of gs://paht/to/certain/file, one for each sample and this is where the issue is raised.

Let me know your thoughts.

Thanks,
Mahesh

[jbingham]

Good catch. Not sure where the limit is coming from.

Can you put all of those hundreds of paths into a fofn (file of file names) and then tell it to load the values from file?

[vangalamaheshh]

@jbingham ,

Yes, I have for now implemented hack of fetching files directly using gsutil with in the container. But, for long term, it would be nice to fix this bug.

Thanks,
Mahesh

[jbingham]

Are you scattering by file or using all in a single task? If the latter, your hack is probably the best solution.

If I knew where the limit came from, it would be easier to fix. Are you using Java or yaml?

[vangalamaheshh]

It is the latter case, @jbingham. Also, I am using Java code not yaml. Here is the dataflow log trace:

https://gist.github.com/vangalamaheshh/5ea26139c8693a62d74969e9ca918e7d

• Mahesh

[jbingham]

Looks to be deep inside of the Dataflow code -- I have no ability to change that.

Maybe Dockerflow could compress and decompress long text strings to stay within the length cap