refactor(PLATENG-800): replace platform-sdk-fetch with integration-sdk-http-client#1188
refactor(PLATENG-800): replace platform-sdk-fetch with integration-sdk-http-client#1188tokio-on-jupiter wants to merge 57 commits intomainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR replaces the deprecated @lifeomic/alpha HTTP client with @jupiterone/platform-sdk-fetch canary release to modernize the API client implementation.
Changes:
- Updated dependency from
@lifeomic/alphato@jupiterone/platform-sdk-fetch - Replaced
Alphatype withRequestClientthroughout the codebase - Removed proxy configuration support and deprecated
alphaOptions/proxyUrlparameters
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/integration-sdk-runtime/package.json | Updated dependency to platform-sdk-fetch canary |
| packages/integration-sdk-runtime/src/api/index.ts | Replaced Alpha with RequestClient, removed proxy support |
| packages/integration-sdk-runtime/src/synchronization/index.ts | Updated type imports and error handling |
| packages/integration-sdk-runtime/src/synchronization/events.ts | Updated config type imports |
| packages/integration-sdk-runtime/src/synchronization/error.ts | Replaced AxiosError with custom RequestClientError interface |
| packages/integration-sdk-runtime/tsconfig.dist.json | Added skipLibCheck to handle external type issues |
| packages/cli/src/import/importAssetsFromCsv.ts | Updated type imports |
| packages/integration-sdk-runtime/src/api/tests/index.test.ts | Updated mocks for RequestClient |
| packages/cli/src/tests/cli-import.test.ts | Updated test mocks |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| export const compressRequest: RequestInterceptor = function (config) { | ||
| if ( | ||
| config.method === 'post' && | ||
| config.method === 'POST' && |
There was a problem hiding this comment.
The compressRequest interceptor now only sets the 'Content-Encoding' header but does not actually compress the data. The comment on lines 126-128 notes this issue but doesn't implement actual compression. This means data won't be compressed despite the header claiming it is, which could cause server-side decompression failures.
There was a problem hiding this comment.
This interceptor was removed in a subsequent commit. Compression is now handled directly in `uploadDataChunk()` in `synchronization/index.ts` (lines 537-547), where `gzipData(data)` compresses the payload and sends it via `rawBody` with the `Content-Encoding: gzip` header. The interceptor approach was replaced because `RequestClient` handles request config differently than Alpha.
| @@ -23,7 +35,7 @@ export function synchronizationApiError( | |||
| return new IntegrationError({ | |||
| code: 'UNEXPECTED_SYNCRONIZATION_ERROR', | |||
There was a problem hiding this comment.
Corrected spelling of 'SYNCRONIZATION' to 'SYNCHRONIZATION'.
| code: 'UNEXPECTED_SYNCRONIZATION_ERROR', | |
| code: 'UNEXPECTED_SYNCHRONIZATION_ERROR', |
| 'Content-Encoding': 'gzip', | ||
| }; | ||
| } | ||
| config.data = await gzipData(config.data); |
There was a problem hiding this comment.
Does config.data need to be compressed here?
There was a problem hiding this comment.
This comment is on a deleted line. I'm not sure what action should be taken.
There was a problem hiding this comment.
The compression interceptor was intentionally removed from createApiClient. Compression is now handled directly in uploadDataChunk() (synchronization/index.ts:537-547) where gzipData(data) compresses the payload and sends it via rawBody with the Content-Encoding: gzip header. This approach is more explicit — compression happens at the upload call site rather than as an opaque interceptor.
There was a problem hiding this comment.
No action needed — this was a deleted line from the old Alpha import.
|
/canary-release |
| export const getAccountFromEnvironment = () => | ||
| getFromEnv('JUPITERONE_ACCOUNT', IntegrationAccountRequiredError); | ||
|
|
||
| function parseProxyUrl(proxyUrl: string) { |
There was a problem hiding this comment.
Does the proxy support need to be maintained?
There was a problem hiding this comment.
The comment above suggests that proxy support can still be achieved by setting the HTTPS_PROXY environment variable. I'm not sure if that's true, though. I didn't see any tests that covered that scenario.
There was a problem hiding this comment.
Proxy support was specific to @lifeomic/alpha's internal HTTP client. RequestClient from platform-sdk-fetch does not have built-in proxy configuration — it relies on environment-level proxy settings (e.g. HTTPS_PROXY, HTTP_PROXY) which Node.js and the underlying HTTP client respect natively.
The proxyUrl param is kept in the interface for backward compatibility but now emits a DeprecationWarning when provided, directing users to use environment variables instead. This avoids a breaking change for callers that pass proxyUrl while making it clear the option no longer has any effect.
|
/canary-release |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 11 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| export const compressRequest: RequestInterceptor = function (config) { | ||
| if ( | ||
| config.method === 'post' && | ||
| config.method === 'POST' && | ||
| config.url && | ||
| /\/persister\/synchronization\/jobs\/[0-9a-fA-F-]+\/(entities|relationships)/.test( | ||
| config.url, | ||
| ) | ||
| ) { | ||
| if (config.headers) { | ||
| config.headers['Content-Encoding'] = 'gzip'; | ||
| } else { | ||
| config.headers = { | ||
| // Note: Compression is handled differently in RequestClient | ||
| // The data compression would need to be applied at the request level | ||
| // For now, we mark the headers - actual compression may need additional handling | ||
| return { | ||
| ...config, | ||
| headers: { | ||
| ...config.headers, | ||
| 'Content-Encoding': 'gzip', | ||
| }; | ||
| } | ||
| config.data = await gzipData(config.data); | ||
| }, | ||
| }; | ||
| } | ||
| return config; |
There was a problem hiding this comment.
The compressRequest interceptor sets the 'Content-Encoding: gzip' header but does not actually compress the data. The previous implementation called gzipData(config.data) but this logic has been removed. This will cause the server to expect compressed data but receive uncompressed data, resulting in decompression failures. Either implement the actual data compression using gzipData or remove this interceptor entirely if compression is handled elsewhere.
| code: 'UNEXPECTED_SYNCRONIZATION_ERROR', | ||
| message: errorMessage, | ||
| cause: err, | ||
| cause: err instanceof Error ? err : undefined, |
There was a problem hiding this comment.
Setting cause to undefined when the error is not an instance of Error loses error information. The original implementation passed err directly to preserve all error details. Consider using cause: err as Error or structuring the error differently to retain the original error object for debugging purposes.
| cause: err instanceof Error ? err : undefined, | |
| cause: err as Error, |
There was a problem hiding this comment.
Fixed — now wraps non-Error objects with `new Error(String(err))` instead of discarding them. This preserves the error context for debugging while satisfying the `Error` type requirement for `cause`.
| }); | ||
|
|
||
| describe('getApiKeyFromEnvironment', () => { | ||
| describe('getAccountFromEnvironment', () => { |
There was a problem hiding this comment.
The describe block name 'getAccountFromEnvironment' is a duplicate of line 55's describe block name which should be 'getApiKeyFromEnvironment'. This appears to be a copy-paste error from the original code that should be corrected for clarity.
There was a problem hiding this comment.
This is not a copy-paste error. Line 53 is `getApiKeyFromEnvironment` (tests for `JUPITERONE_API_KEY`) and line 76 is `getAccountFromEnvironment` (tests for `JUPITERONE_ACCOUNT`). They are two separate describe blocks for two different exported functions.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 11 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| /** | ||
| * Request interceptor that compresses upload data for synchronization endpoints | ||
| */ | ||
| export const compressRequest: RequestInterceptor = function (config) { | ||
| if ( | ||
| config.method === 'post' && | ||
| config.method === 'POST' && | ||
| config.url && | ||
| /\/persister\/synchronization\/jobs\/[0-9a-fA-F-]+\/(entities|relationships)/.test( | ||
| config.url, | ||
| ) | ||
| ) { | ||
| if (config.headers) { | ||
| config.headers['Content-Encoding'] = 'gzip'; | ||
| } else { | ||
| config.headers = { | ||
| // Note: Compression is handled differently in RequestClient | ||
| // The data compression would need to be applied at the request level | ||
| // For now, we mark the headers - actual compression may need additional handling | ||
| return { | ||
| ...config, | ||
| headers: { | ||
| ...config.headers, | ||
| 'Content-Encoding': 'gzip', | ||
| }; | ||
| } | ||
| config.data = await gzipData(config.data); | ||
| }, | ||
| }; | ||
| } | ||
| return config; | ||
| }; |
There was a problem hiding this comment.
The compressRequest interceptor sets the 'Content-Encoding: gzip' header but doesn't actually compress the data. The original implementation called gzipData(config.data) to compress the request body. Without actual compression, servers expecting gzipped data will fail to process the request.
| accessToken, | ||
| retryOptions, | ||
| compressUploads, | ||
| alphaOptions, | ||
| proxyUrl, | ||
| }: CreateApiClientInput): ApiClient { |
There was a problem hiding this comment.
The deprecated parameters alphaOptions and proxyUrl are destructured but never used in the function body. Either remove them from the destructuring or explicitly acknowledge their deprecation with a warning log.
| code: 'UNEXPECTED_SYNCRONIZATION_ERROR', | ||
| message: errorMessage, | ||
| cause: err, | ||
| cause: err instanceof Error ? err : undefined, |
There was a problem hiding this comment.
When err is not an Error instance, setting cause to undefined loses error context. Consider converting unknown errors to Error instances or using the original value to preserve debugging information.
| cause: err instanceof Error ? err : undefined, | |
| cause: err, |
|
/canary-release |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 10 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| }, | ||
| ); | ||
| rawBody: compressedData, | ||
| } as any); |
There was a problem hiding this comment.
Using as any type assertion bypasses type safety. Consider defining a proper type for the options parameter that includes the rawBody property, or add a TODO comment explaining this is temporary until the upstream types are updated.
There was a problem hiding this comment.
Already addressed — a `RequestConfigWithRawBody` interface extending `RequestClientRequestConfig` was added (lines 44-47) and is used instead of `as any`. The TODO comment documents that this can be removed once `platform-sdk-fetch` officially exports `rawBody` in its types.
|
🚀 Canary release workflow has been triggered. You can follow the progress here. |
|
❌ Canary release failed. Check the workflow run for details. |
|
/canary-release |
|
🚀 Canary release workflow has been triggered. You can follow the progress here. |
|
❌ Canary release failed. Check the workflow run for details. |
|
/canary-release |
|
🚀 Canary release workflow has been triggered. You can follow the progress here. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 10 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| client.interceptors.response.use( | ||
| (response) => response, | ||
| (error: any) => { | ||
| async (error: any) => { |
There was a problem hiding this comment.
The error parameter is typed as any, which reduces type safety. Consider defining a proper error type that captures the expected error structure from RequestClient.
|
|
||
| export function synchronizationApiError( | ||
| err: AxiosError<SynchronizationApiErrorResponse>, | ||
| err: RequestClientError | Error | unknown, |
There was a problem hiding this comment.
The parameter accepts unknown in addition to specific error types, but then performs type assertions without proper type guards. Consider adding a type guard function to safely narrow the type before accessing properties.
|
❌ Canary release failed. Check the workflow run for details. |
|
/canary-release |
Downstream consumers (e.g. graph-aws) still pass alphaOptions to createApiClient. Throwing breaks them at runtime. Switch to console.warn deprecation notices so migration can proceed incrementally.
ESLint no-console rule blocks console.warn in CI. Switch to process.emitWarning with DeprecationWarning type, which is the Node.js standard for deprecation notices.
Add npm override to force fast-xml-parser@5.3.4, fixing the RangeError DoS vulnerability in numeric entities parsing.
Add package.json exports map for @jupiterone/integration-sdk-testing
with a separate ./mockApiClient entry point. This allows CLI tests
to import createMockApiClient without loading recording.ts, which
uses polly/graceful-fs and conflicts with jest.mock('fs').
Addresses review feedback from @ryanmcafee.
Co-authored-by: M. Scott Ford <scott@mscottford.com>
- Replace lerna@7.1.4 with nx@22.3.3 and @nx/js@22.3.3 - Create nx.json with caching config for build, test, lint targets - Add NX metadata (tags, deploy) to all 11 package.json files - Update root scripts to use npx nx run-many - Update CI/CD workflows (build.yml, canary.yaml) for NX - Update .gitignore with .nx/ and *.tsbuildinfo patterns - Delete lerna.json
The canary workflow runs from the main branch (issue_comment trigger) which still uses lerna commands. Keep lerna.json and lerna dependency until the NX migration is merged to main.
The @sinclair/typebox and fast-xml-parser overrides were not needed: - @sinclair/typebox is already pinned by the lock file, and the override selector didn't match the actual dependency specs - fast-xml-parser is required at an exact version by @aws-sdk/xml-builder, making the override redundant
Rewire createApiClient() to instantiate JupiterOneApiClient instead of calling createRequestClient from platform-sdk-fetch. Remove all imports of RequestClient, RequestClientError, RequestClientRequestConfig, and isRequestClientError across synchronization, events, and test files.
The CLI and integration-sdk-cli packages imported types from @jupiterone/platform-sdk-fetch which is no longer a transitive dependency. Updated to use ApiClient and ApiClientResponse from integration-sdk-runtime. Also widened post() body param to accept string for CSV uploads.
…ild order The tsc -b build command listed integration-sdk-http-client after integration-sdk-runtime, causing CI to fail since runtime now depends on http-client types. Fixed by adding the project reference and reordering the build command.
- Move gzip compression into JupiterOneApiClient.post() for transparent compression (like the old Alpha interceptor) - Remove noopLogger from production code, use createIntegrationLogger - Add optional logger param to CreateApiClientInput - Define ILogger interface for uploadDataChunk to accept simple mocks - Restore config.data assertion and clean it in cleanRequestError - Revert compressUploads: false from all test files - Revert logger eslint-disable prefer-spread overrides back to spread
The nx 22.5.3 bump caused npm to resolve data-model's typebox to 0.32.35 while integration-sdk-core stayed at 0.32.30. TypeScript treats types from different package versions as incompatible, breaking tsc -b with [Kind] symbol mismatches. Pin data-model's typebox to 0.32.30 to match the rest of the workspace.
…ctor ILogger, fix type safety - Restore retryOptions to CreateApiClientInput with original shape, map to http-client fields - Restore proxyUrl with https-proxy-agent@7, support HTTPS_PROXY env fallback - Move ILogger interface to logger/index.ts, IntegrationLogger implements ILogger - Remove blanket eslint-disable from apiClient.test.ts, use real IntegrationLogger - Remove compressUploads: false from uploader.test.ts - Fix child() mock in sync test to return context.logger - Revert skipLibCheck from runtime tsconfig files - Add runtime warnings for deprecated retryCondition and alphaOptions
…om apiClient tests Create TestableApiClient subclass that exposes protected methods as public overrides, eliminating 12 eslint-disable-next-line comments. Change redactAuthHeaders from private to protected with err: unknown.
tokio-on-jupiter
force-pushed
the
feat/PLATENG-800-replace-lifeomic-alpha
branch
from
2d8cded to
6217e7d
Compare
![boostsecurity-io[bot]](https://avatars.githubusercontent.com/in/192984?s=60&v=4){kind=small}
There was a problem hiding this comment.
⚠️ 4 New Security Findings
The latest commit contains 4 new security findings.
| Findings | |||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Dependency: npm / flatted@ 3.3.4 SUMMARY Direct Dependency: flatted Location : package-lock.json OCCURRENCES
|
|||||||||||||||||||||||||||
Remediation :
|
|||||||||||||||||||||||||||
| Dependency: npm / path-to-regexp@ 0.1.12 SUMMARY Direct Dependency: path-to-regexp Location : package-lock.json OCCURRENCE
|
|||||||||||||||||||||||||||
Remediation :
|
|||||||||||||||||||||||||||
| Dependency: npm / picomatch@ 4.0.2 SUMMARY Direct Dependency: picomatch Location : package-lock.json OCCURRENCE
|
|||||||||||||||||||||||||||
Remediation :
|
|||||||||||||||||||||||||||
| Dependency: npm / picomatch@ 4.0.3 SUMMARY Direct Dependency: picomatch Location : package-lock.json OCCURRENCE
|
|||||||||||||||||||||||||||
Remediation :
|
Not a finding? Ignore it by adding a comment on the line with just the word noboost.
Scanner: boostsecurity - OSV-Scanner
Summary
Replaces
@jupiterone/platform-sdk-fetch(private package wrapping@lifeomic/alpha) with@jupiterone/integration-sdk-http-client(already in-repo, already public) as the HTTP client forintegration-sdk-runtime.This eliminates the private transitive dependency that blocked external consumers from installing the public SDK packages, and removes the legacy
@lifeomic/alphadependency chain entirely.Approach
JupiterOneApiClientclass inintegration-sdk-runtimeextendsBaseAPIClientfromintegration-sdk-http-clientpost()/get()API contract returning{ data, status, headers }request()override that passesrawBody: Bufferdirectly tonode-fetch(BaseAPIClient's body serialization doesn't support Buffer)executeRequest()alphaOptionsandproxyUrlemit deprecation warnings instead of throwingChanges
Core (integration-sdk-runtime)
src/api/apiClient.ts—JupiterOneApiClientadapter classcreateApiClient()factory to instantiateJupiterOneApiClient@jupiterone/platform-sdk-fetchimports fromsynchronization/index.ts,synchronization/events.ts,test/util/request.tsplatform-sdk-fetchforintegration-sdk-http-clientinpackage.json@lifeomic/attemptretained for application-level retry infinalizeSynchronization()CLI packages
packages/cli/src/import/importAssetsFromCsv.ts— replacedRequestClienttype import withApiClientfrom runtimepackages/integration-sdk-cli/src/questions/managedQuestionFileValidator.test.ts— replacedRequestClientResponsewithApiClientResponseTesting
packages/integration-sdk-testing/src/logger.ts— removed stale@ts-expect-errordirective (pre-existing issue)Other
createMockApiClientutility inintegration-sdk-testingfast-xml-parserupgrade for CVE-2026-25128Known Items for Follow-up
node-fetchis used directly inapiClient.tsbut not declared as explicit dependency (works via transitive hoisting fromintegration-sdk-http-client)_compressUploadsfield uses underscore convention but is publicly accessedredactAuthHeadersmay be effectively dead code sinceBaseAPIClienterrors don't carry.config.headersTest plan
JupiterOneApiClient(constructor, auth headers, post/get, rawBody, redaction, error handling)nx run-many -t build:dist)jest --changedSince main)platform-sdk-fetchsource imports (only 1 stale comment incli-run.test.ts)17.2.2-canary-1188-22078318628.0published and installed in integrations repojupiterone-devwith canary — job executed successfully with no errors