Scrape Flow

This is a highly customizable web scraper built with Next.js, PostgreSQL, Prisma, and React Flow. It enables users to create, manage, and execute complex web scraping workflows with a visual, no-code/low-code interface. The scraper comes with advanced features such as AI-based data extraction

Screenshots

ScreenShot 1

ScreenShot 2

A modern, full-stack web application for building and executing automated workflows, with a focus on web scraping and data processing. Built with Next.js, TypeScript, and Prisma.

What's included in flow-scrape?

Flow Scrape provides a comprehensive platform for creating, managing, and executing automated workflows. It features a drag-and-drop workflow builder, real-time execution tracking, and integrations with popular services.

Tech Stack

• Next.js: The React framework for production.
• TypeScript: JavaScript with syntax for types.
• Prisma: Next-generation ORM for TypeScript & Node.js.
• PostgreSQL: Advanced open source relational database.
• Stripe: Payment processing for internet businesses.
• Tailwind CSS: A utility-first CSS framework.
• shadcn/ui: Re-usable components built on Radix UI and Tailwind CSS.
• NextAuth.js: Complete open source authentication solution.

Features

• Workflow Builder: Drag-and-drop interface to create custom workflows
• Node-Based Editor: Visual workflow editor with various node types
• Execution Engine: Run workflows with real-time progress tracking
• User Authentication: Secure user management with NextAuth.js
• Billing Integration: Stripe-powered subscription management
• Credential Management: Secure storage of API keys and credentials
• Analytics Dashboard: Track workflow performance and usage
• Responsive Design: Modern UI built with Tailwind CSS and shadcn/ui

Getting Started

Prerequisites

• Node.js 18+
• PostgreSQL database
• Stripe account for payments

Installation

1. Clone the repository:

git clone https://github.com/bjornleonhenry/flow-scrape.git
cd flow-scrape

2. Install dependencies:

npm install
# or
pnpm install

3. Set up environment variables:

cp .env.example .env.local

Fill in the required environment variables:

• DATABASE_URL: Your PostgreSQL connection string
• NEXTAUTH_SECRET: Random secret for NextAuth.js
• NEXTAUTH_URL: Your app's URL
• STRIPE_PUBLISHABLE_KEY: Stripe publishable key
• STRIPE_SECRET_KEY: Stripe secret key
• STRIPE_WEBHOOK_SECRET: Stripe webhook secret

4. Set up the database:

npx prisma generate
npx prisma db push

5. Run the development server:

npm run dev
# or
pnpm run dev

Open http://localhost:3000 to see the application.

Project Structure

├── app/                    # Next.js app directory
│   ├── (auth)/            # Authentication pages
│   ├── (dashboard)/       # Dashboard pages
│   ├── api/               # API routes
│   └── globals.css        # Global styles
├── components/            # Reusable components
│   ├── ui/               # shadcn/ui components
│   └── ...               # Custom components
├── lib/                  # Utility functions and configurations
├── prisma/               # Database schema and migrations
├── public/               # Static assets
└── types/                # TypeScript type definitions

Contributing to flow-scrape

Contributions are welcomed, and appreciated. If you have a feature request, please add it as an issue or make a pull request.