DEALIoT

DEALIoT is a production-oriented real-time IoT data platform for multimodal telemetry, media metadata, schema governance, stream processing, orchestration, and operational compliance evidence.

The platform targets livestock, precision agriculture, and industrial IoT deployments where device telemetry, GPS data, media objects, and governance evidence must be processed reliably from edge ingestion to curated datasets.

Why Teams Adopt DEALIoT

DEALIoT is designed for teams that need to move beyond isolated IoT demos into governed, reproducible, production-ready data flows. The adoption thesis is simple: prove one field data source, one event contract, one useful output, and one operational decision path before scaling the architecture.

Strong first-fit scenarios include livestock telemetry, WildFi decoding, precision agriculture, industrial IoT hardening, research dataset publication, EU data-sharing readiness, managed hosting evaluation, and device or platform integrations.

Fast Adoption Path

1. Confirm fit with the use case catalog.
2. Compare alternatives with the architecture comparison guide.
3. Run the quick evaluation path and capture the first event reaching Kafka.
4. Help an internal owner make the case with the internal champion kit.
5. Scope one 30-day pilot with the demo pilot playbook.
6. Score the pilot with the validation scorecard.
7. Publish an approved lesson, adopter story, or integration note using the public launch kit.

Platform Scope

DEALIoT provides six runtime planes:

Plane	Responsibility	Primary components
Ingestion	Secure MQTT ingestion and routing to Kafka topics	VerneMQ, MQTT-Kafka bridge
Event backbone	Durable event transport and schema governance	Kafka KRaft, Apicurio Registry
Object storage	Raw and derived media object storage	SeaweedFS S3 locally, managed S3 in production
Processing	Stream processing, feature projection, replay, and backfill	Flink, Beam, Airflow
Storage	Operational SQL state and connection pooling	TimescaleDB, Patroni, HAProxy, PgBouncer
Operations	Observability, control surfaces, and compliance evidence	Prometheus, Grafana, Management Console

Website And Adoption

• Public website: https://smartappli.io/
• Website source repository: https://github.com/Smartappli/DEALWebsite
• Adoption playbook: docs/community/adoption-playbook.md
• Architecture popularity playbook: docs/community/architecture-popularity-playbook.md
• GitHub star growth plan: docs/community/github-star-growth-plan.md
• Use case catalog: docs/community/use-case-catalog.md
• Quick evaluation path: docs/community/quick-evaluation-path.md
• Architecture comparison guide: docs/community/architecture-comparison-guide.md
• Internal champion kit: docs/community/internal-champion-kit.md
• Public launch kit: docs/community/public-launch-kit.md
• First GitHub discussion: docs/community/first-github-discussion.md
• Adoption funnel: docs/community/adoption-funnel.md
• User community launch plan: docs/community/user-community-launch-plan.md
• User onboarding guide: docs/community/user-onboarding-guide.md
• Community rituals: docs/community/community-rituals.md
• User feedback loop: docs/community/user-feedback-loop.md
• Developer community playbook: docs/community/developer-community-playbook.md
• Contributor onboarding: docs/community/contributor-onboarding.md
• Community governance: docs/community/community-governance.md
• Seed discussions: docs/community/seed-discussions.md
• Demo pilot playbook: docs/community/demo-pilot-playbook.md
• Integration partner guide: docs/community/integration-partner-guide.md
• Pilot validation scorecard: docs/community/validation-scorecard.md
• Adopter story template: docs/community/adopter-story-template.md
• Community discussions: https://github.com/Smartappli/DEALIoT/discussions
• Roadmap: ROADMAP.md
• Support policy: SUPPORT.md
• Public references: ADOPTERS.md

The recommended adoption path is: validate the local smoke test, run a 30-day pilot with one data source and one value output, then harden the target runtime with the production deployment guardrails.

The recommended community path is: ask usage questions in GitHub Discussions, open issues only for confirmed bugs or scoped changes, share pilot feedback with the user feedback template, and convert repeated questions into documentation or tests.

Architecture

flowchart LR
  Device[IoT devices] --> MQTT[MQTT broker]
  Media[Media producers] --> S3[S3-compatible object storage]
  MQTT --> Bridge[MQTT-Kafka bridge]
  Bridge --> Kafka[(Kafka)]
  S3 --> Kafka
  Kafka --> Registry[Apicurio Registry]
  Kafka --> Flink[Flink streaming jobs]
  Kafka --> Beam[Beam pipelines]
  Airflow[Airflow orchestration] --> S3
  Airflow --> Kafka
  Flink --> State[(TimescaleDB / state stores)]
  Prometheus[Prometheus] --> Grafana[Grafana]
  Console[Management Console] --> Airflow
  Console --> Registry
  Console --> Flink

Loading

Production Architecture Principles

• Kubernetes is the primary production target.
• Docker Swarm remains available for simpler runtime deployments and smoke validation.
• Stateful dependencies are externalized in production unless managed by a dedicated operator.
• Runtime dependency traffic is encrypted or private: Kafka SASL_SSL, MQTT TLS, S3 TLS, PostgreSQL private connectivity, and Redis private connectivity.
• Kubernetes production uses default-deny NetworkPolicies, Pod Security restricted, immutable image tags, readiness/liveness probes, HPA, PDB, and topology spread constraints.
• Secrets are expected from a secret manager, External Secrets Operator, or equivalent out-of-band mechanism.

Repository Layout

.github/workflows/                         CI, security scans, image build, deployment validation
airflow/dags/                              Airflow DAGs
apicurio/bootstrap/                        Registry schema bootstrap payloads
dealiot_contracts/                         Shared event contract helpers
deploy/kubernetes/base/                    Kubernetes base runtime manifests
deploy/kubernetes/overlays/production/     Production Kustomize overlay
deploy/swarm/                              Docker Swarm runtime and smoke stacks
docs/                                      Architecture, compliance, and runbooks
flink/jobs/                                PyFlink streaming jobs
management-console/                        Internal operational console
mqtt-kafka-bridge/                         MQTT to Kafka ingestion bridge
pipelines/                                 Replay and backfill utilities
scripts/                                   Bootstrap and smoke-test scripts
tests/                                     Unit, integration, and deployment guardrail tests
wildfi-decoder/                            Offline WildFi binary decoder image wrapper

Event Topics

Core runtime topics include:

Topic	Purpose
raw.sensor	Device telemetry and decoded WildFi sensor payloads
raw.gps	GPS and GNSS events
raw.image2d.meta	2D image metadata
raw.image3d.meta	3D image metadata
raw.video2d.meta	2D video metadata
raw.video3d.meta	3D video metadata
media.object.events	Object storage notifications
features.events	Derived feature events
state.latest	Compacted latest state projection
dlq.events	Invalid or unroutable event records

Governance, Data Act, DGA, security, resilience, and compliance evidence topics are defined in docker-compose.yml, apicurio/bootstrap/, and docs/runbooks/security-resilience-compliance.md.

Local Development

Prerequisites

• Docker Engine with Compose v2
• Python 3.12 or newer for local tests
• uv for reproducible Python tooling
• kubectl for rendering Kubernetes overlays

Configure Local Secrets

cp .env.example .env
mkdir -p secrets

Populate the secret files listed in README runbooks and .env.example. Local secrets must stay outside Git; .gitignore and .dockerignore exclude .env and secrets/.

Start The Development Stack

docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build

Useful local endpoints when the development overlay is active:

Service	Endpoint
Airflow	http://localhost:8088
Flink	http://localhost:8081
Apicurio Registry	http://localhost:8082/apis/registry/v3
Management Console	http://localhost:8090
Grafana	http://localhost:3000
Prometheus	http://localhost:9090
SeaweedFS S3	http://localhost:8333

Run The End-To-End Smoke Test

bash scripts/smoke-e2e.sh

The smoke test starts the core event-flow services, submits the minimal Flink job, publishes MQTT fixtures, validates Kafka topics, verifies Apicurio artifacts, and captures diagnostics on failure.

Production Deployment

Kubernetes

The production overlay is located at deploy/kubernetes/overlays/production.

Before deployment:

1. Replace all sha-REPLACE_WITH_RELEASE_SHA image tags with immutable release SHA tags.
2. Replace example dependency endpoints in runtime-config.production.example.env.
3. Provide dealiot-secrets through a secret manager or External Secrets Operator.
4. Narrow NetworkPolicy ipBlock ranges to real private dependency CIDRs.
5. Confirm metrics-server or another HPA metrics provider is installed.

Render locally:

kubectl kustomize deploy/kubernetes/overlays/production >/tmp/dealiot-production.yaml

Apply through your GitOps controller or deployment pipeline after replacing all placeholders.

Docker Swarm

The Swarm stack is located at deploy/swarm/dealiot-stack.yml and expects external Kafka, MQTT, S3, PostgreSQL, and Redis services.

docker stack config -c deploy/swarm/dealiot-stack.yml
docker stack deploy -c deploy/swarm/dealiot-stack.yml dealiot

Create required Swarm secrets before deployment. See deploy/swarm/README.md for the exact contract.

Runtime Security

The production runtime contract requires:

• Kafka SASL_SSL with SCRAM credentials.
• MQTT TLS on port 8883 by default.
• Management Console bearer-token protection for /api/* and mutation routes.
• Kubernetes Pod Security restricted on production and CI smoke namespaces.
• Default-deny Kubernetes NetworkPolicies.
• Immutable image tags and CI checks that reject mutable tags and unresolved placeholders.
• Container resources, readiness/liveness probes, dropped Linux capabilities, and disabled service-account token automounting.

Testing And Quality Gates

Run the same validation layers used by CI:

uv run python -m unittest discover -s tests/unit -p "test_*.py" -v
uv run python -m unittest -v tests/integration/test_platform_integration.py
uv run --with PyYAML python -m unittest -v tests/deployment/test_deployment_readiness.py
uv run python -m unittest -v tests/test_application_smoke.py

Additional CI gates include:

• Pre-commit hooks: YAML, JSON, Ruff, Mypy, djLint.
• CodeQL, Bandit, OSSAR, OSV Scanner, SonarQube, and Codacy coverage upload.
• Docker image builds with SBOM and provenance attestations.
• Kubernetes render and server-side dry-run validation.
• Docker Swarm render and smoke deployment validation.
• kind smoke deployment for the bridge image.

Operations And Runbooks

Primary runbooks:

• Operations
• Backup and restore
• Security hardening
• Security resilience compliance
• WildFi ingestion
• Data Governance Act
• Data Act
• Dataset catalogue and DMP
• Zenodo export
• OpenAIRE export

The GitHub Wiki contains the production architecture handbook, deployment guide, configuration reference, operational runbooks, scaling model, and security checklist.

WildFi Support

DEALIoT supports WildFi telemetry through:

• MQTT subscription to $share/ingestors/wildfi/#.
• Routing decoded GPS/GNSS payloads to raw.gps.
• Routing decoded IMU, environment, proximity, movement, and metadata payloads to raw.sensor.
• Offline binary decoding through the packaged wildfi-decoder image.

References:

• docs/runbooks/wildfi-ingestion.md
• deploy/kubernetes/overlays/production/wildfi-decoder-config.yaml
• deploy/kubernetes/overlays/production/wildfi-decoder-job.yaml

Contribution Workflow

Read CONTRIBUTING.md before opening a pull request.

New contributors should start with the contributor onboarding guide. Maintainers use the developer community playbook and community governance to keep contribution paths small, reviewable, and aligned with production quality.

1. Create changes with tests.
2. Run the validation commands above.
3. Render Kubernetes and Swarm manifests when deployment files change.
4. Keep production placeholders out of rendered manifests.
5. Update README, wiki, runbooks, or community docs when user-facing behavior changes.
6. Commit with a focused message and push to GitHub.

Community expectations are defined in CODE_OF_CONDUCT.md. Security reports must follow SECURITY.md, not public issues.

License

This repository is licensed under the terms in LICENSE.