Can use ETL modules as below by default.
| Step Class Name | Role |
|---|---|
| AzureBlobDownload | Download files from Azure Blob Storage |
| BigQueryRead | Read from bigquery table |
| DynamoDBRead | Read data from Amazon DynamoDB table |
| FirestoreDocumentDownload | Download a document from Firestore |
| FtpDownload | Download a file via ftp |
| FtpDownloadFileDelete | Remove files downloaded via class 'FtpDownload' from FTP server. |
| GcsDownload | Download files from GCS |
| GcsDownloadFileDelete | Remove files downloaded via class 'GcsDownload' from GCS server |
| GcsFileExistsCheck | Check if files exist in GCS |
| HttpDownload | Download a file via http |
| HttpDownloadViaBasicAuth | Download a file via HTTP with basic auth |
| HttpGet | Send HTTP GET request |
| MysqlRead | Execute a query to MySql server and get result as csv file |
| PostgresqlRead | Execute a query to PostgreSQL server and get result as csv file |
| S3Delete | Delete files from S3 |
| S3Download | Download files from S3 |
| S3FileExistsCheck | Check if files exist in S3 |
| SftpDelete | Delete a file via SFTP |
| SftpDownload | Download a file via sftp |
| SftpDownloadFileDelete | Remove files downloaded via class 'SftpDownload' from SFTP server |
| SftpFileExistsCheck | Check if files exist in SFTP server |
| SqliteExport | Export a table data to csv |
| Step Class Name | Role |
|---|---|
| AesDecrypt | Decrypt AES encrypted files |
| AesEncrypt | Encrypt files with AES |
| ColumnLengthAdjust | Adjust columns of a csv file or a tsv file to the specified length. |
| CsvColumnConcat | Concatenate columns of csv files |
| CsvColumnCopy | Copy columns in csv files |
| CsvColumnDelete | Delete specific columns from csv files |
| CsvColumnExtract | Extract specific columns from csv files. |
| CsvColumnHash | Hash columns of a csv file |
| CsvColumnReplace | Replace values in csv columns |
| CsvColumnSelect | Select specific columns from csv files |
| CsvConcat | Concat csv files |
| CsvConvert | Create new csv(tsv) file with given parameters |
| CsvDuplicateRowDelete | Delete duplicate rows from csv files |
| CsvMerge | Merge two csv files to a csv file |
| CsvMergeExclusive | Merge csv files exclusively |
| CsvRowDelete | Delete specific rows from csv files |
| CsvSort | Sort csv files |
| CsvSplit | Split csv files into multiple files |
| CsvToJsonl | Convert csv files to jsonl format |
| CsvTypeConvert | Convert data types of columns in csv files |
| CsvValueExtract | Extract specific values from csv files |
| DateFormatConvert | Convert date format of columns of a csv file to another date format |
| ExcelConvert | Convert a excel file to a csv file |
| ExecuteShellScript | Execute Shell Script |
| FileArchive | Archive files |
| FileCompress | Compress a file |
| FileConvert | Convert file encoding |
| FileCopy | Copy files |
| FileDecompress | Decompress a file |
| FileDivide | Divide a file to plural files |
| FileRename | Change file names with adding either prefix or suffix |
| GpgDecrypt | Decrypt GPG encrypted files |
| GpgEncrypt | Encrypt files with GPG |
| GpgGenerateKey | Generate GPG keys |
| JsonlAddKeyValue | Add key-value pairs to jsonl files |
| JsonlToCsv | Convert jsonl files to csv format |
| Step Class Name | Role |
|---|---|
| AzureBlobUpload | Upload files to Azure Blob Storage |
| BigQueryCopy | Copy data between BigQuery tables |
| BigQueryWrite | Read content from a file and insert it into a table of bigquery |
| DynamoDBWrite | Write data to Amazon DynamoDB table |
| FirestoreDocumentCreate | Create document |
| GcsUpload | Upload files to GCS |
| HttpDelete | Send HTTP DELETE request |
| HttpPut | Send HTTP PUT request |
| PostgresqlWrite | Write data to PostgreSQL database |
| S3Upload | Upload files to S3 |
| SftpUpload | Upload a file via sftp |
| SqliteImport | Read content from csv files and insert them into sqlite table |
| Step Class Name | Role |
|---|---|
| SqliteQueryExecute | Execute query against sqlite table |