Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
134 changes: 3 additions & 131 deletions .github/workflows/build-and-release.yml.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,13 @@ on:
types: [created]

env:
REGISTRY_IMAGE: embucket/embucket
REGISTRY_IMAGE: embucket/embucket-labs

jobs:

# This job builds and pushes a multi-arch Docker image to GitHub Container Registry.
build-docker:
runs-on: ${{ matrix.runner }}
env:
VARIANT: default
strategy:
fail-fast: true
matrix:
Expand Down Expand Up @@ -73,8 +71,6 @@ jobs:
platforms: ${{ matrix.platform }}
labels: ${{ steps.meta.outputs.labels }}
outputs: type=image,name=${{ env.REGISTRY_IMAGE }},push-by-digest=true,name-canonical=true,push=true
build-args: |
ENABLE_EXPERIMENTAL=false
cache-from: type=gha
cache-to: type=gha,mode=max

Expand All @@ -87,87 +83,7 @@ jobs:
- name: Upload digest
uses: actions/upload-artifact@v5
with:
name: digests-${{ env.VARIANT }}-${{ matrix.platform_pair }}
path: ${{ runner.temp }}/digests/*
if-no-files-found: error
retention-days: 1

build-docker-experimental:
runs-on: ${{ matrix.runner }}
env:
VARIANT: experimental
strategy:
fail-fast: true
matrix:
platform:
- linux/amd64
- linux/arm64
include:
- platform: linux/amd64
runner: ubuntu-latest
platform_pair: linux-amd64
- platform: linux/arm64
runner: ubuntu-24.04-arm
platform_pair: linux-arm64
name: Build and Push Experimental Docker Image
steps:
- name: Checkout code
uses: actions/checkout@v5

- name: Prepare
run: |
echo "PLATFORM_PAIR=${{ matrix.platform_pair }}" >> $GITHUB_ENV

- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY_IMAGE }}
tags: |
type=ref,event=branch,suffix=-experimental
type=ref,event=pr,suffix=-experimental
type=semver,pattern={{version}},suffix=-experimental
type=semver,pattern={{major}}.{{minor}},suffix=-experimental
type=raw,value=experimental
type=raw,value=latest-experimental,enable={{is_default_branch}}

- name: Set up QEMU for multi-arch builds
uses: docker/setup-qemu-action@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Log in to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: docker.io
username: ${{ vars.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Build and push experimental Docker image
uses: docker/build-push-action@v6
id: build
with:
context: .
file: ./Dockerfile
platforms: ${{ matrix.platform }}
labels: ${{ steps.meta.outputs.labels }}
build-args: |
ENABLE_EXPERIMENTAL=true
outputs: type=image,name=${{ env.REGISTRY_IMAGE }},push-by-digest=true,name-canonical=true,push=true
cache-from: type=gha
cache-to: type=gha,mode=max

- name: Export digest
run: |
mkdir -p ${{ runner.temp }}/digests
digest="${{ steps.build.outputs.digest }}"
touch "${{ runner.temp }}/digests/${digest#sha256:}"

- name: Upload digest
uses: actions/upload-artifact@v5
with:
name: digests-${{ env.VARIANT }}-${{ matrix.platform_pair }}
name: digests-${{ matrix.platform_pair }}
path: ${{ runner.temp }}/digests/*
if-no-files-found: error
retention-days: 1
Expand All @@ -181,7 +97,7 @@ jobs:
uses: actions/download-artifact@v6
with:
path: ${{ runner.temp }}/digests
pattern: digests-default-*
pattern: digests-*
merge-multiple: true

- name: Login to Docker Hub
Expand Down Expand Up @@ -214,47 +130,3 @@ jobs:
- name: Inspect image
run: |
docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:${{ steps.meta.outputs.version }}

merge-experimental:
runs-on: ubuntu-latest
needs:
- build-docker-experimental
steps:
- name: Download digests
uses: actions/download-artifact@v6
with:
path: ${{ runner.temp }}/digests
pattern: digests-experimental-*
merge-multiple: true

- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ vars.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY_IMAGE }}
tags: |
type=ref,event=branch,suffix=-experimental
type=ref,event=pr,suffix=-experimental
type=semver,pattern={{version}},suffix=-experimental
type=semver,pattern={{major}}.{{minor}},suffix=-experimental
type=raw,value=experimental
type=raw,value=latest-experimental,enable={{is_default_branch}}

- name: Create manifest list and push
working-directory: ${{ runner.temp }}/digests
run: |
docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
$(printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *)

- name: Inspect image
run: |
docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:experimental
12 changes: 6 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ We appreciate every contribution, whether it's fixing bugs, improving documentat

To set up a development environment, please follow these steps:

1. Clone the Embucket/embucket repository
1. Clone the embucket/embucket-labs repository

```sh
git clone https://github.com/Embucket/embucket.git
git clone https://github.com/embucket/embucket-labs.git
cd embucket/
```

Expand All @@ -27,7 +27,7 @@ To set up a development environment, please follow these steps:

## Issues and feature requests

You've found a bug in the source code, a mistake in the documentation or maybe you'd like a new feature? You can help us by. Before you create an issue, make sure to search the issue archive [GithubIssues](https://github.com/Embucket/embucket/issues?q=is%3Aissue) your issue may have already been addressed!
You've found a bug in the source code, a mistake in the documentation or maybe you'd like a new feature? You can help us by. Before you create an issue, make sure to search the issue archive [GithubIssues](https://github.com/embucket/embucket-labs/issues?q=is%3Aissue) your issue may have already been addressed!

Please try to create bug reports that are:

Expand All @@ -41,13 +41,13 @@ Please try to create bug reports that are:
### How to submit a Pull Request

1. Search our repository for open or closed
[Pull Requests](https://github.com/Embucket/embucket/pulls)
[Pull Requests](https://github.com/embucket/embucket-labs/pulls)
that relate to your submission. You don't want to duplicate effort.
2. Fork the project
3. Create your feature branch (`git checkout -b issueID/new_feature`)
4. Commit your changes (`git commit -m 'issue id: New Feature'`)
5. Push to the branch (`git push origin issueID/new_feature`)
6. [Open a Pull Request](https://github.com/Embucket/embucket/compare?expand=1)
6. [Open a Pull Request](https://github.com/embucket/embucket-labs/compare?expand=1)
7. Your pull request build must pass (the build will run automatically).


Expand All @@ -57,4 +57,4 @@ When coding in Rust, please follow the [coding conventions](CODING_CONVENTIONS.m


## Getting Help
For discussions, questions, or collaboration, feel free to reach out via [GitHub Discussions](https://github.com/Embucket/embucket/discussions). We appreciate your contributions and look forward to working with you!
For discussions, questions, or collaboration, feel free to reach out via [GitHub Discussions](https://github.com/embucket/embucket-labs/discussions). We appreciate your contributions and look forward to working with you!
12 changes: 3 additions & 9 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,9 @@ RUN apt-get update && apt-get install -y \
# Copy source code
COPY . .

# Build the application, optionally enabling experimental features
ARG ENABLE_EXPERIMENTAL=false
RUN if [ "$ENABLE_EXPERIMENTAL" = "true" ]; then \
echo "Building embucketd with experimental features enabled"; \
cargo build --release --bin embucketd --features experimental; \
else \
echo "Building embucketd without experimental features"; \
cargo build --release --bin embucketd --no-default-features; \
fi
# Build the application with every feature enabled so the final image ships
# with a single, fully-capable binary.
RUN cargo build --release --bin embucketd --all-features

# Stage 4: Final runtime image
FROM gcr.io/distroless/cc-debian12 AS runtime
Expand Down
11 changes: 5 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@
**Run Snowflake SQL dialect on your data lake in 30 seconds. Zero dependencies.**

[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![SQL Logic Test Coverage](https://raw.githubusercontent.com/Embucket/embucket/assets/assets/badge.svg)](test/README.md)
[![dbt Gitlab run results](https://raw.githubusercontent.com/Embucket/embucket/assets_dbt/assets_dbt/dbt_success_badge.svg)](test/dbt_integration_tests/dbt-gitlab/README.md)
[![SQL Logic Test Coverage](https://raw.githubusercontent.com/embucket/embucket-labs/assets/assets/badge.svg)](test/README.md)
[![dbt Gitlab run results](https://raw.githubusercontent.com/embucket/embucket-labs/assets_dbt/assets_dbt/dbt_success_badge.svg)](test/dbt_integration_tests/dbt-gitlab/README.md)

## Quick start

Start Embucket and run your first query in 30 seconds:

```bash
docker run --name embucket --rm -p 8080:8080 -p 3000:3000 embucket/embucket
docker run --name embucket --rm -p 8080:8080 -p 3000:3000 embucket/embucket-labs
```

Open [localhost:8080](http://localhost:8080)—login: `embucket`/`embucket`—and run:
Expand Down Expand Up @@ -67,8 +67,8 @@ Built on proven open source:

**From source:**
```bash
git clone https://github.com/Embucket/embucket.git
cd embucket && cargo build
git clone https://github.com/embucket/embucket-labs.git
cd embucket-labs && cargo build
./target/debug/embucketd
```

Expand All @@ -85,4 +85,3 @@ For more details, see [CONTRIBUTING.md](CONTRIBUTING.md).
## License

This project uses the **Apache 2.0 License**. See [LICENSE](LICENSE) for details.

2 changes: 1 addition & 1 deletion crates/api-snowflake-rest/src/tests/snow_sql.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ use uuid::Uuid;

pub async fn snow_sql(server_addr: &SocketAddr, user: &str, pass: &str, sql: &str) -> JsonResponse {
// introduce 2ms (to be sure) delay every time running query via "snow sql" as an issue workaround:
// https://github.com/Embucket/embucket/issues/1630
// https://github.com/embucket/embucket-labs/issues/1630
tokio::time::sleep(tokio::time::Duration::from_millis(2)).await;

let client = reqwest::Client::new();
Expand Down
2 changes: 1 addition & 1 deletion crates/api-ui/src/tests/volumes.rs
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ async fn test_s3_volumes_validation() {

#[test]
fn test_serde_roundtrip() {
// https://github.com/Embucket/embucket/issues/1306
// https://github.com/embucket/embucket-labs/issues/1306
let payload: VolumeCreatePayload = create_s3_tables_volume_ok_payload();
let json = serde_json::to_string(&payload).expect("Failed to serialize");
serde_json::from_str::<VolumeCreatePayload>(&json).expect("Failed to parse");
Expand Down
2 changes: 1 addition & 1 deletion crates/core-history/src/entities/result_set.rs
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ impl Row {
}
}

/// `<https://github.com/Embucket/embucket/issues/1662`>
/// `<https://github.com/embucket/embucket-labs/issues/1662`>
/// Custom deserializer for deserializing `RecordBatch` rows having duplicate columns names
/// like this: `[{"col":1,"col":2}, {"col":1,"col":2}]`, into the `Vec<Value>` (omiting keys).
/// It also support deserializng `JsonArray` `[1, 2]`, to the `Vec<Value>`
Expand Down
2 changes: 1 addition & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ services:
\" > /opt/spark/conf/spark-defaults.conf && ./entrypoint.sh notebook
"
embucket:
image: embucket/embucket
image: embucket/embucket-labs
container_name: embucket
depends_on:
- mc
Expand Down
4 changes: 3 additions & 1 deletion docs/astro.config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@ export default defineConfig({
src: './src/assets/logo.svg',
},
favicon: '/favicon.ico',
social: [{ icon: 'github', label: 'GitHub', href: 'https://github.com/Embucket/embucket' }],
social: [
{ icon: 'github', label: 'GitHub', href: 'https://github.com/embucket/embucket-labs' },
],
sidebar: [
{
label: 'Essentials',
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/development/tracing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Embucket uses `tracing::instrument` to instrument code for tracing. You can use

### Tracing span processor experimental async runtime

Embucket uses `BatchSpanProcessor`, which uses a dedicated background thread for collecting and exporting spans. This processor works well in production. Some development environments may hang on startup as reported in [issue #1123](https://github.com/embucket/embucket/issues/1123). In this case, you can switch to `BatchSpanProcessor` using the experimental async runtime.
Embucket uses `BatchSpanProcessor`, which uses a dedicated background thread for collecting and exporting spans. This processor works well in production. Some development environments may hang on startup as reported in [issue #1123](https://github.com/embucket/embucket-labs/issues/1123). In this case, you can switch to `BatchSpanProcessor` using the experimental async runtime.
Use this command-line argument: `--tracing-span-processor=batch-span-processor-experimental-async-runtime`.

### Tracing span processor tuning
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/essentials/quick-start.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ This guide doesn't cover production deployment or advanced configuration options
Start the Embucket server:

```bash
docker run --name embucket --rm -p 8080:8080 -p 3000:3000 embucket/embucket
docker run --name embucket --rm -p 8080:8080 -p 3000:3000 embucket/embucket-labs
```

**Done.** You need no external dependencies, databases, or complex configuration. Embucket runs everything in-memory for this quick start. This zero-disk architecture requires no external storage.
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/essentials/snowflake.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Embucket verifies compatibility using two test methods:
- **SQL Logic Tests**: Verify SQL engine compatibility with Snowflake
- **[dbt](https://docs.getdbt.com/) integration tests**: Verify REST API compatibility with Snowflake

Embucket uses the [dbt](https://docs.getdbt.com/) Gitlab project as a compatibility benchmark. The Embucket [repository](https://github.com/Embucket/embucket) displays current compatibility badges for both SQL logic test suite and dbt Gitlab.
Embucket uses the [dbt](https://docs.getdbt.com/) Gitlab project as a compatibility benchmark. The Embucket [repository](https://github.com/embucket/embucket-labs) displays current compatibility badges for both SQL logic test suite and dbt Gitlab.

## Architecture differences

Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/guides/dbt-snowplow.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ This guide covers local development setup with sample data and basic [dbt](https

<Aside type="note">
For complete example, see [dbt integration
tests](https://github.com/Embucket/embucket/tree/main/test/dbt_integration_tests/dbt-snowplow-web).
tests](https://github.com/embucket/embucket-labs/tree/main/test/dbt_integration_tests/dbt-snowplow-web).
</Aside>

## Prerequisites
Expand Down
4 changes: 2 additions & 2 deletions docs/src/content/docs/guides/load-spark.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ services:

# Embucket server providing Iceberg catalog and web interface
embucket:
image: embucket/embucket
image: embucket/embucket-labs
container_name: embucket
depends_on:
mc:
Expand Down Expand Up @@ -202,7 +202,7 @@ Expected output:

```
NAME IMAGE STATUS PORTS
embucket embucket/embucket Up 0.0.0.0:3000->3000/tcp, 0.0.0.0:8080->8080/tcp
embucket embucket/embucket-labs Up 0.0.0.0:3000->3000/tcp, 0.0.0.0:8080->8080/tcp
mc minio/mc Up (healthy)
minio minio/minio Up 0.0.0.0:9000->9000/tcp, 0.0.0.0:9001->9001/tcp
spark-iceberg tabulario/spark-iceberg Up 0.0.0.0:8888->8888/tcp
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/guides/snowflake-cli.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ pip install snowflake-cli
Launch the Embucket container:

```bash
docker run --name embucket --rm -p 8080:8080 -p 3000:3000 embucket/embucket
docker run --name embucket --rm -p 8080:8080 -p 3000:3000 embucket/embucket-labs
```

<Aside type="note">
Expand Down
6 changes: 3 additions & 3 deletions test/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
[![SQL Logic Test Coverage](https://raw.githubusercontent.com/Embucket/embucket/assets/assets/badge.svg)](test/README.md)
[![SQL Logic Test Coverage](https://raw.githubusercontent.com/embucket/embucket-labs/assets/assets/badge.svg)](test/README.md)

## SLT coverage
![Test Coverage Visualization](https://raw.githubusercontent.com/Embucket/embucket/assets/assets/test_coverage_visualization.png)
![Test Coverage Visualization](https://raw.githubusercontent.com/embucket/embucket-labs/assets/assets/test_coverage_visualization.png)

![Not Implemented Tests Distribution](https://raw.githubusercontent.com/Embucket/embucket/assets/assets/not_implemented_visualization.png)
![Not Implemented Tests Distribution](https://raw.githubusercontent.com/embucket/embucket-labs/assets/assets/not_implemented_visualization.png)

*These visualizations are automatically updated by CI/CD when tests are run.*

Expand Down
Loading
Loading