chore: Add argument to limit file size, put file error to errorCol #2464

levscaut · 2026-01-14T19:26:20Z

Related Issues/PRs

N/A

What changes are proposed in this pull request?

This PR adds file validation parameters to OpenAIPrompt and improves error handling for file preparation:

Graceful error handling for file preparation: Instead of failing the entire process when a file cannot be read (e.g., file not found), errors are now captured in errorCol with null in outputCol, allowing other rows to process normally.
File size limit (setFileSizeLimitMB): New parameter to restrict maximum file size in megabytes. Files exceeding this limit produce an error in errorCol.
SimpleHTTPTransformer enhancement: Modified to preserve existing errorCol values using coalesce, enabling upstream error propagation.

How is this patch tested?

I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

New test cases added:

file preparation error stores error in errorCol instead of failing
file size limit produces error when exceeded

Does this PR change any dependencies?

No. You can skip this section.

github-actions · 2026-01-14T19:26:44Z

Hey @levscaut 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

fix: Fix LightGBM crashes with empty partitions
feat: Make HTTP on Spark back-offs configurable
docs: Update Spark Serving usage
build: Add codecov support
perf: improve LightGBM memory usage
refactor: make python code generation rely on classes
style: Remove nulls from CNTKModel
test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

levscaut · 2026-01-14T19:29:06Z

/azp run

azure-pipelines · 2026-01-14T19:29:19Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

This PR enhances file validation in OpenAIPrompt by adding configurable size and extension limits with graceful error handling for file preparation failures.

Changes:

Added file size limit parameter (fileSizeLimitMB) to restrict maximum file sizes
Added valid file extensions parameter (validFileExtensions) to restrict allowed file types
Modified error handling to capture file preparation errors in errorCol instead of failing the entire process
Enhanced SimpleHTTPTransformer to preserve upstream errors using coalesce

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIPrompt.scala	Added file validation parameters and improved error handling for file operations
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIPromptSuite.scala	Added comprehensive tests for new validation features and error handling
core/src/main/scala/com/microsoft/azure/synapse/ml/io/http/SimpleHTTPTransformer.scala	Modified to preserve existing errors from upstream transformations

Comments suppressed due to low confidence (1)

cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIPrompt.scala:255

The localParamNames sequence is missing the newly added parameters fileSizeLimitMB and validFileExtensions. These should be included if they are meant to be local parameters that are not passed to the underlying services.

  private val localParamNames = Seq(
    "promptTemplate", "outputCol", "postProcessing", "postProcessingOptions", "dropPrompt", "dropMessages",
    "systemPrompt", "apiType", "usageCol", "responseIdCol")

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIPrompt.scala

cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIPromptSuite.scala

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

levscaut · 2026-01-14T19:36:25Z

/azp run

azure-pipelines · 2026-01-14T19:36:37Z

Azure Pipelines successfully started running 1 pipeline(s).

codecov-commenter · 2026-01-14T19:57:09Z

Codecov Report

❌ Patch coverage is 91.11111% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.44%. Comparing base (1405a37) to head (fb8371c).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...zure/synapse/ml/services/openai/OpenAIPrompt.scala	89.47%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2464      +/-   ##
==========================================
+ Coverage   84.42%   84.44%   +0.01%     
==========================================
  Files         335      335              
  Lines       17750    17782      +32     
  Branches     1622     1579      -43     
==========================================
+ Hits        14986    15016      +30     
- Misses       2764     2766       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

levscaut · 2026-01-14T20:05:45Z

/azp run

azure-pipelines · 2026-01-14T20:05:58Z

Azure Pipelines successfully started running 1 pipeline(s).

levscaut · 2026-01-14T20:53:19Z

/azp run

azure-pipelines · 2026-01-14T20:53:31Z

Azure Pipelines successfully started running 1 pipeline(s).

levscaut · 2026-01-14T23:35:14Z

/azp run

azure-pipelines · 2026-01-14T23:35:26Z

Azure Pipelines successfully started running 1 pipeline(s).

cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIPrompt.scala

levscaut · 2026-01-15T20:32:48Z

/azp run

azure-pipelines · 2026-01-15T20:33:00Z

Azure Pipelines successfully started running 1 pipeline(s).

levscaut · 2026-01-15T20:45:55Z

/azp run

azure-pipelines · 2026-01-15T20:46:06Z

Azure Pipelines successfully started running 1 pipeline(s).

levscaut · 2026-01-16T19:12:48Z

/azp run

azure-pipelines · 2026-01-16T19:13:00Z

Azure Pipelines successfully started running 1 pipeline(s).

cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIPrompt.scala

levscaut · 2026-01-16T20:16:40Z

/azp run

azure-pipelines · 2026-01-16T20:16:53Z

Azure Pipelines successfully started running 1 pipeline(s).

levscaut · 2026-01-16T21:02:33Z

/azp run

azure-pipelines · 2026-01-16T21:02:43Z

Azure Pipelines successfully started running 1 pipeline(s).

ranadeepsingh

LGTM, thanks for looking at the feedback!

Wendong Li and others added 7 commits January 12, 2026 21:48

null value passthrough

2a73c6f

real null

8d14f1a

Merge branch 'master' into master

760ae7e

fix style

c5ce0d8

Merge branch 'master' of https://github.com/levscaut/SynapseML

eb9cd27

wrap file prepare error in errorCol

5f23585

file limit

933e1fb

levscaut requested review from Copilot and ranadeepsingh January 14, 2026 19:26

Copilot started reviewing on behalf of levscaut January 14, 2026 19:27 View session

Merge branch 'master' into file_limit

f8670b8

levscaut requested a review from BrendanWalsh January 14, 2026 19:28

Copilot AI reviewed Jan 14, 2026

View reviewed changes

Apply suggestions from code review

845f4c9

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

copilot what are you doing

1fa0a16

funny test error

0b6288b

remove validFileExtensions

a517f30

levscaut changed the title ~~chore: Add argument to limit file sizes and types~~ chore: Add argument to limit file size Jan 14, 2026

levscaut changed the title ~~chore: Add argument to limit file size~~ chore: Add argument to limit file size, put file error to errorCol Jan 14, 2026

ranadeepsingh requested changes Jan 15, 2026

View reviewed changes

address comments

8f471a9

levscaut requested a review from ranadeepsingh January 15, 2026 20:43

levscaut added 2 commits January 15, 2026 20:44

Merge branch 'master' of https://github.com/levscaut/SynapseML

15408e8

Merge branch 'master' into file_limit

f897c48

levscaut enabled auto-merge (squash) January 15, 2026 22:13

levscaut requested a review from smamindl January 16, 2026 18:18

simplify

a3deea9

smamindl reviewed Jan 16, 2026

View reviewed changes

cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIPrompt.scala Show resolved Hide resolved

restore mechanism for Bytes

fb8371c

BrendanWalsh approved these changes Jan 16, 2026

View reviewed changes

ranadeepsingh approved these changes Jan 16, 2026

View reviewed changes

levscaut merged commit 7a3b27c into microsoft:master Jan 16, 2026
68 of 69 checks passed

chore: Add argument to limit file size, put file error to errorCol #2464

chore: Add argument to limit file size, put file error to errorCol #2464

Uh oh!

Conversation

levscaut commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related Issues/PRs

What changes are proposed in this pull request?

How is this patch tested?

Does this PR change any dependencies?

Uh oh!

github-actions bot commented Jan 14, 2026

Uh oh!

levscaut commented Jan 14, 2026

Uh oh!

azure-pipelines bot commented Jan 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

levscaut commented Jan 14, 2026

Uh oh!

azure-pipelines bot commented Jan 14, 2026

Uh oh!

codecov-commenter commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

levscaut commented Jan 14, 2026

Uh oh!

azure-pipelines bot commented Jan 14, 2026

Uh oh!

levscaut commented Jan 14, 2026

Uh oh!

azure-pipelines bot commented Jan 14, 2026

Uh oh!

levscaut commented Jan 14, 2026

Uh oh!

azure-pipelines bot commented Jan 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

levscaut commented Jan 15, 2026

Uh oh!

azure-pipelines bot commented Jan 15, 2026

Uh oh!

levscaut commented Jan 15, 2026

Uh oh!

azure-pipelines bot commented Jan 15, 2026

Uh oh!

levscaut commented Jan 16, 2026

Uh oh!

azure-pipelines bot commented Jan 16, 2026

Uh oh!

Uh oh!

levscaut commented Jan 16, 2026

Uh oh!

azure-pipelines bot commented Jan 16, 2026

Uh oh!

levscaut commented Jan 16, 2026

Uh oh!

azure-pipelines bot commented Jan 16, 2026

Uh oh!

ranadeepsingh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

levscaut commented Jan 14, 2026 •

edited

Loading

codecov-commenter commented Jan 14, 2026 •

edited

Loading