diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md new file mode 100644 index 0000000..24f3f5a --- /dev/null +++ b/DEPLOYMENT.md @@ -0,0 +1,133 @@ +# Cloud Run Deployment Guide + +This guide will help you deploy the webpage replicator application to Google Cloud Run. + +## Prerequisites + +1. **Google Cloud Account**: You need a Google Cloud account with billing enabled +2. **Google Cloud CLI**: Install the `gcloud` CLI tool +3. **Docker**: Ensure Docker is installed and running (for local testing) +4. **Project Setup**: Create a Google Cloud project + +## Quick Deployment + +### Option 1: Using the deployment script (Recommended) + +1. **Update the deployment script**: + ```bash + # Edit deploy.sh and replace 'your-project-id' with your actual project ID + nano deploy.sh + ``` + +2. **Make the script executable and run it**: + ```bash + chmod +x deploy.sh + ./deploy.sh + ``` + +### Option 2: Manual deployment using YAML files + +1. **Set your project ID**: + ```bash + export PROJECT_ID="your-project-id" + gcloud config set project $PROJECT_ID + ``` + +2. **Enable required APIs**: + ```bash + gcloud services enable cloudbuild.googleapis.com + gcloud services enable run.googleapis.com + ``` + +3. **Update YAML files**: + - Replace `PROJECT_ID` in both `frontend/cloudrun.yaml` and `backend/cloudrun.yaml` with your actual project ID + +4. **Build and deploy backend**: + ```bash + cd backend + gcloud builds submit --tag gcr.io/$PROJECT_ID/webpage-replicator-backend + gcloud run services replace cloudrun.yaml --region=us-central1 + cd .. + ``` + +5. **Build and deploy frontend**: + ```bash + cd frontend + gcloud builds submit --tag gcr.io/$PROJECT_ID/webpage-replicator-frontend + gcloud run services replace cloudrun.yaml --region=us-central1 + cd .. + ``` + +## Configuration + +### Backend Configuration + +The backend service may require environment variables for API keys and other configuration. You can set these using: + +```bash +gcloud run services update webpage-replicator-backend \ + --region=us-central1 \ + --set-env-vars="GEMINI_API_KEY=your-api-key-here" +``` + +Or use Google Secret Manager for sensitive data: + +```bash +# Create a secret +gcloud secrets create gemini-api-key --data-file=api-key.txt + +# Update the service to use the secret +gcloud run services update webpage-replicator-backend \ + --region=us-central1 \ + --set-secrets="GEMINI_API_KEY=gemini-api-key:latest" +``` + +### Frontend Configuration + +If your frontend needs to communicate with the backend, update any API endpoint URLs in your frontend code to use the deployed backend URL. + +## Monitoring and Logs + +- **View logs**: `gcloud run logs read webpage-replicator-backend --region=us-central1` +- **Monitor metrics**: Visit the Cloud Console > Cloud Run to view metrics and performance + +## Costs + +Cloud Run pricing is based on: +- CPU and memory allocation +- Number of requests +- Request duration + +The current configuration uses: +- **Frontend**: 1 vCPU, 512Mi memory +- **Backend**: 2 vCPU, 1Gi memory + +Both services scale to zero when not in use, so you only pay for actual usage. + +## Troubleshooting + +### Common Issues + +1. **Build failures**: Check that all dependencies are properly listed in `package.json` +2. **Port issues**: Ensure your application listens on the port specified in the `PORT` environment variable +3. **Health check failures**: Make sure your backend has a `/health` endpoint or update the health check path + +### Useful Commands + +```bash +# View service details +gcloud run services describe webpage-replicator-backend --region=us-central1 + +# View recent deployments +gcloud run revisions list --service=webpage-replicator-backend --region=us-central1 + +# Delete a service +gcloud run services delete webpage-replicator-backend --region=us-central1 +``` + +## Security Considerations + +- Both services are currently configured to allow unauthenticated access +- For production, consider implementing authentication +- Use IAM roles to control access to your services +- Store sensitive configuration in Google Secret Manager \ No newline at end of file diff --git a/backend/.dockerignore b/backend/.dockerignore new file mode 100644 index 0000000..cb0aa94 --- /dev/null +++ b/backend/.dockerignore @@ -0,0 +1,14 @@ +node_modules +npm-debug.log +.git +.gitignore +README.md +.env +.env.local +.env.development.local +.env.test.local +.env.production.local +.DS_Store +*.log +uploads/ +temp/ \ No newline at end of file diff --git a/backend/Dockerfile b/backend/Dockerfile new file mode 100644 index 0000000..242a971 --- /dev/null +++ b/backend/Dockerfile @@ -0,0 +1,20 @@ +# Use Node.js official image +FROM node:18-alpine + +# Set working directory +WORKDIR /app + +# Copy package files +COPY package*.json ./ + +# Install dependencies +RUN npm ci --only=production + +# Copy application files +COPY . . + +# Expose port (assuming Express server runs on port 3001 or process.env.PORT) +EXPOSE 3001 + +# Start the application +CMD ["npm", "start"] \ No newline at end of file diff --git a/backend/cloudrun.yaml b/backend/cloudrun.yaml new file mode 100644 index 0000000..c90ccf5 --- /dev/null +++ b/backend/cloudrun.yaml @@ -0,0 +1,57 @@ +apiVersion: serving.knative.dev/v1 +kind: Service +metadata: + name: webpage-replicator-backend + annotations: + run.googleapis.com/ingress: all + run.googleapis.com/ingress-status: all +spec: + template: + metadata: + annotations: + autoscaling.knative.dev/maxScale: "100" + run.googleapis.com/cpu-throttling: "false" + run.googleapis.com/execution-environment: gen2 + spec: + containerConcurrency: 80 + timeoutSeconds: 300 + containers: + - image: gcr.io/PROJECT_ID/webpage-replicator-backend:latest + ports: + - name: http1 + containerPort: 3001 + env: + - name: PORT + value: "3001" + - name: NODE_ENV + value: "production" + # Add your environment variables here + # - name: GEMINI_API_KEY + # valueFrom: + # secretKeyRef: + # name: gemini-secrets + # key: api-key + resources: + limits: + cpu: 2000m + memory: 1Gi + requests: + cpu: 200m + memory: 256Mi + livenessProbe: + httpGet: + path: /health + port: 3001 + initialDelaySeconds: 30 + periodSeconds: 30 + failureThreshold: 3 + readinessProbe: + httpGet: + path: /health + port: 3001 + initialDelaySeconds: 5 + periodSeconds: 10 + failureThreshold: 3 + traffic: + - percent: 100 + latestRevision: true \ No newline at end of file diff --git a/backend/gcs-service.js b/backend/gcs-service.js new file mode 100644 index 0000000..a107bf1 --- /dev/null +++ b/backend/gcs-service.js @@ -0,0 +1,157 @@ +import { Storage } from '@google-cloud/storage'; + +class GCSService { + constructor() { + this.storage = new Storage(); + this.bucketName = process.env.GCS_BUCKET_NAME; + + if (!this.bucketName) { + throw new Error('GCS_BUCKET_NAME environment variable is required'); + } + + this.bucket = this.storage.bucket(this.bucketName); + } + + /** + * Upload a file to GCS + * @param {string} fileName - The name/path of the file in the bucket + * @param {Buffer} fileBuffer - The file content as a buffer + * @param {string} contentType - The MIME type of the file + * @returns {Promise} - The public URL of the uploaded file + */ + async uploadFile(fileName, fileBuffer, contentType = 'application/octet-stream') { + try { + const file = this.bucket.file(fileName); + + const stream = file.createWriteStream({ + metadata: { + contentType: contentType, + }, + resumable: false, + }); + + return new Promise((resolve, reject) => { + stream.on('error', (error) => { + console.error('Error uploading to GCS:', error); + reject(error); + }); + + stream.on('finish', () => { + // Make the file publicly readable + file.makePublic().then(() => { + const publicUrl = `https://storage.googleapis.com/${this.bucketName}/${fileName}`; + resolve(publicUrl); + }).catch(reject); + }); + + stream.end(fileBuffer); + }); + } catch (error) { + console.error('Error in uploadFile:', error); + throw error; + } + } + + /** + * Download a file from GCS + * @param {string} fileName - The name/path of the file in the bucket + * @returns {Promise} - The file content as a buffer + */ + async downloadFile(fileName) { + try { + const file = this.bucket.file(fileName); + const [fileBuffer] = await file.download(); + return fileBuffer; + } catch (error) { + console.error('Error downloading from GCS:', error); + throw error; + } + } + + /** + * Check if a file exists in GCS + * @param {string} fileName - The name/path of the file in the bucket + * @returns {Promise} - Whether the file exists + */ + async fileExists(fileName) { + try { + const file = this.bucket.file(fileName); + const [exists] = await file.exists(); + return exists; + } catch (error) { + console.error('Error checking file existence:', error); + return false; + } + } + + /** + * List files in a directory (prefix) + * @param {string} prefix - The directory prefix to list + * @returns {Promise} - Array of file objects + */ + async listFiles(prefix = '') { + try { + const [files] = await this.bucket.getFiles({ + prefix: prefix, + }); + + return files.map(file => ({ + name: file.name, + size: file.metadata.size, + created: file.metadata.timeCreated, + contentType: file.metadata.contentType, + publicUrl: `https://storage.googleapis.com/${this.bucketName}/${file.name}` + })); + } catch (error) { + console.error('Error listing files:', error); + throw error; + } + } + + /** + * Delete a file from GCS + * @param {string} fileName - The name/path of the file in the bucket + * @returns {Promise} - Whether the deletion was successful + */ + async deleteFile(fileName) { + try { + const file = this.bucket.file(fileName); + await file.delete(); + return true; + } catch (error) { + console.error('Error deleting file:', error); + return false; + } + } + + /** + * Get a public URL for a file + * @param {string} fileName - The name/path of the file in the bucket + * @returns {string} - The public URL + */ + getPublicUrl(fileName) { + return `https://storage.googleapis.com/${this.bucketName}/${fileName}`; + } + + /** + * Generate a signed URL for temporary access + * @param {string} fileName - The name/path of the file in the bucket + * @param {number} expiresInMinutes - Expiration time in minutes (default: 60) + * @returns {Promise} - The signed URL + */ + async getSignedUrl(fileName, expiresInMinutes = 60) { + try { + const file = this.bucket.file(fileName); + const [signedUrl] = await file.getSignedUrl({ + action: 'read', + expires: Date.now() + (expiresInMinutes * 60 * 1000), + }); + return signedUrl; + } catch (error) { + console.error('Error generating signed URL:', error); + throw error; + } + } +} + +export default GCSService; \ No newline at end of file diff --git a/backend/package.json b/backend/package.json index 6e5d1e7..3712294 100644 --- a/backend/package.json +++ b/backend/package.json @@ -9,6 +9,7 @@ "dev": "node --watch server.js" }, "dependencies": { + "@google-cloud/storage": "^7.13.0", "@google/genai": "^1.15.0", "cors": "^2.8.5", "dotenv": "^17.2.1", diff --git a/backend/server-gcs.js b/backend/server-gcs.js new file mode 100644 index 0000000..62209ed --- /dev/null +++ b/backend/server-gcs.js @@ -0,0 +1,554 @@ +import express from 'express'; +import multer from 'multer'; +import cors from 'cors'; +import { GoogleGenAI } from '@google/genai'; +import { v4 as uuidv4 } from 'uuid'; +import sharp from 'sharp'; +import dotenv from 'dotenv'; +import GCSService from './gcs-service.js'; + +// Load environment variables +dotenv.config(); + +const app = express(); +const PORT = process.env.PORT || 3001; + +// Initialize GCS Service +let gcsService; +try { + gcsService = new GCSService(); + console.log('GCS Service initialized successfully'); +} catch (error) { + console.error('Failed to initialize GCS Service:', error.message); + process.exit(1); +} + +// Middleware +app.use(cors()); +app.use(express.json()); + +// Configure multer for file uploads +const storage = multer.memoryStorage(); +const upload = multer({ + storage: storage, + limits: { fileSize: 10 * 1024 * 1024 } // 10MB limit +}); + +// Initialize Gemini AI +const genAI = new GoogleGenAI({ + apiKey: process.env.GEMINI_API_KEY +}); + +// Function to remove existing terms/conditions elements to avoid duplicates with Clickthrough SDK +function removeExistingTermsElements(html) { + console.log('Scanning for existing terms/conditions elements to remove...'); + + // Patterns to detect and remove existing terms/conditions elements + const termsPatterns = [ + // Checkboxes with terms-related text + /]*type\s*=\s*["']checkbox["'][^>]*[^>]*>\s*[^<]*(?:terms|condition|agree|policy|privacy)[^<]*<\/?\w*>/gi, + /]*>[^<]*]*type\s*=\s*["']checkbox["'][^>]*>\s*[^<]*(?:terms|condition|agree|policy|privacy)[^<]*<\/label>/gi, + + // Divs or paragraphs containing checkbox + terms text + /]*>[^<]*]*type\s*=\s*["']checkbox["'][^>]*>[^<]*(?:terms|condition|agree|policy|privacy)[^<]*<\/div>/gi, + /]*>[^<]*]*type\s*=\s*["']checkbox["'][^>]*>[^<]*(?:terms|condition|agree|policy|privacy)[^<]*<\/p>/gi, + + // Text containing "I agree to" or similar + /<[^>]*>[^<]*(?:I\s+agree\s+to|By\s+clicking|Accept\s+terms|Terms\s+and\s+conditions|Privacy\s+policy)[^<]*<\/[^>]*>/gi, + + // Links to terms/privacy within form contexts + /]*href[^>]*(?:terms|privacy|condition)[^>]*>[^<]*(?:terms|privacy|condition)[^<]*<\/a>/gi, + + // Standalone checkboxes near submit buttons (likely terms acceptance) + /]*type\s*=\s*["']checkbox["'][^>]*>\s*(?=.*(?:submit|sign.up|register))/gi + ]; + + let removedCount = 0; + + // Remove each pattern found + termsPatterns.forEach((pattern, index) => { + const matches = html.match(pattern); + if (matches) { + console.log(`Pattern ${index + 1} found ${matches.length} matches:`, matches); + html = html.replace(pattern, ''); + removedCount += matches.length; + } + }); + + console.log(`Total terms/conditions elements removed: ${removedCount}`); + return html; +} + +// Function to remove image references from generated HTML +function removeImageReferences(html) { + console.log('Removing image references from generated HTML...'); + + // Remove img tags + html = html.replace(/]*>/gi, ''); + + // Remove src attributes from other elements + html = html.replace(/\s+src\s*=\s*["'][^"']*["']/gi, ''); + + // Remove background-image CSS properties + html = html.replace(/background-image\s*:\s*url\([^)]*\);?/gi, ''); + + // Remove other image-related CSS + html = html.replace(/background\s*:\s*url\([^)]*\)[^;]*;?/gi, ''); + + console.log('Image references removed'); + return html; +} + +// Function to add Clickthrough SDK integration +function addClickthroughToHTML(html, clickthroughId, clusterId) { + console.log('Adding Clickthrough integration...'); + console.log('Parameters:', { clickthroughId, clusterId }); + + try { + // Remove existing terms/conditions elements first + html = removeExistingTermsElements(html); + + // Find form elements + const formRegex = /]*>/i; + const formMatch = html.match(formRegex); + + if (!formMatch) { + console.log('No form found for Clickthrough integration'); + return html; + } + + // Find submit button + const submitButtonRegex = /<(button|input)[^>]*(?:type\s*=\s*["']submit["']|class\s*=\s*["'][^"']*submit[^"']*["'])[^>]*>/i; + const submitMatch = html.match(submitButtonRegex); + + if (!submitMatch) { + console.log('No submit button found for Clickthrough integration'); + return html; + } + + // Clickthrough integration code + const clickthroughIntegration = ` + + + `; + + // Insert before closing tag + const bodyCloseIndex = html.lastIndexOf(''); + if (bodyCloseIndex !== -1) { + html = html.slice(0, bodyCloseIndex) + clickthroughIntegration + html.slice(bodyCloseIndex); + console.log('Clickthrough integration added successfully'); + } else { + // If no tag, append to end + html += clickthroughIntegration; + console.log('Clickthrough integration appended to end (no tag found)'); + } + + return html; + + } catch (error) { + console.error('Error adding Clickthrough integration:', error); + // Return original HTML if processing fails + return html; + } +} + +// Endpoint to upload screenshot and generate webpage +app.post('/api/generate-page', upload.single('screenshot'), async (req, res) => { + try { + if (!req.file) { + return res.status(400).json({ error: 'No screenshot uploaded' }); + } + + const pageId = uuidv4(); + const screenshotBuffer = req.file.buffer; + + // Get Clickthrough parameters from form data + const clickthroughId = req.body.clickthroughId; + const clusterId = req.body.clusterId; + + console.log('Received parameters:', { + clickthroughId, + clusterId, + hasFile: !!req.file, + bodyKeys: Object.keys(req.body) + }); + + // Convert image to base64 for Gemini + const base64Image = screenshotBuffer.toString('base64'); + const mimeType = req.file.mimetype; + + // Generate HTML/CSS/JS using Gemini - CLEAN GENERATION WITHOUT CLICKTHROUGH + + // Universal instructions that always apply + const imageHandlingInstructions = ` + + CRITICAL IMAGE HANDLING INSTRUCTIONS (ALWAYS APPLY): + - DO NOT include any tags or image references from the screenshot + - DO NOT attempt to replicate logos, photos, graphics, or any visual images + - REPLACE image areas with appropriate styled elements: + * For logos: Use styled text/typography or CSS-based geometric shapes + * For decorative images: Use CSS backgrounds, gradients, or colored divs + * For photos: Use placeholder colored backgrounds or CSS patterns + * For icons: Use CSS symbols, Unicode characters, or styled elements + - Focus on creating a clean, functional page without broken image links + - Use colors, typography, and CSS styling to maintain visual hierarchy instead of images + `; + + let clickthroughInstructions = ''; + if (clickthroughId && clusterId) { + clickthroughInstructions = ` + + IMPORTANT: Do NOT include any terms and conditions, privacy policy checkboxes, or legal acceptance elements in your generated HTML. These will be added automatically during post-processing. + `; + } + + const prompt = ` + Analyze this screenshot of a webpage and generate complete HTML, CSS, and JavaScript code to replicate it as closely as possible. + + Critical Requirements for Accurate Replication: + + TYPOGRAPHY & FONTS: + - Match exact font families, sizes, and weights + - Replicate line-height, letter-spacing, and text alignment + - Preserve heading hierarchy and text formatting + - Ensure proper font loading and fallbacks + + PAGE FORMATTING & LAYOUT: + - Create pixel-perfect replica of spacing, margins, and padding + - Match exact element positioning and alignment + - Preserve proportions and visual hierarchy + - Implement responsive design with proper breakpoints + + VISUAL DETAILS: + - Match colors exactly (backgrounds, text, borders) + - Replicate shadows, gradients, and visual effects + - Preserve border radius, styling, and decorative elements + - Maintain consistent spacing between all elements + + TECHNICAL REQUIREMENTS: + - Use modern CSS (flexbox, grid) for accurate layout + - Include all interactive elements and form styling + - Implement proper semantic HTML structure + - Add inline CSS and JavaScript in single HTML file + - Ensure full functionality with form validation and interactions + ${imageHandlingInstructions} + ${clickthroughInstructions} + + Focus on maintaining the exact visual appearance and formatting integrity of the original design. + + IMPORTANT: Return ONLY the complete HTML code with embedded CSS and JavaScript. Do not use markdown code blocks, backticks, or any formatting - just return the raw HTML code directly. + `; + + const response = await genAI.models.generateContent({ + model: process.env.GEMINI_MODEL || "gemini-2.5-flash", + contents: [ + { + role: "user", + parts: [ + { text: prompt }, + { + inlineData: { + data: base64Image, + mimeType: mimeType + } + } + ] + } + ] + }); + + let generatedHTML = response.text; + + // Clean up markdown code block formatting if present + generatedHTML = generatedHTML + .replace(/^```html\s*/i, '') // Remove opening ```html + .replace(/^```\s*/gm, '') // Remove any other opening ``` + .replace(/\s*```$/gm, '') // Remove closing ``` + .replace(/```html/gi, '') // Remove any remaining ```html + .replace(/```/g, '') // Remove any remaining ``` + .trim(); + + // POST-PROCESS: Clean up any image references (always apply) + generatedHTML = removeImageReferences(generatedHTML); + + // POST-PROCESS: Add Clickthrough integration if parameters provided + if (clickthroughId && clusterId) { + generatedHTML = addClickthroughToHTML(generatedHTML, clickthroughId, clusterId); + } + + // Save generated files to GCS + const htmlFileName = `pages/${pageId}/index.html`; + const screenshotFileName = `pages/${pageId}/original.png`; + + // Save the HTML file to GCS + const htmlUrl = await gcsService.uploadFile(htmlFileName, Buffer.from(generatedHTML, 'utf8'), 'text/html'); + + // Save the original screenshot to GCS + const screenshotUrl = await gcsService.uploadFile(screenshotFileName, screenshotBuffer, 'image/png'); + + res.json({ + success: true, + pageId: pageId, + url: htmlUrl, + previewUrl: htmlUrl, + screenshotUrl: screenshotUrl + }); + + } catch (error) { + console.error('Error generating page:', error); + res.status(500).json({ + error: 'Failed to generate page', + details: error.message + }); + } +}); + +// Endpoint to compare generated page with original screenshot +app.post('/api/compare-page/:pageId', async (req, res) => { + try { + const { pageId } = req.params; + + // Download files from GCS + const screenshotFileName = `pages/${pageId}/original.png`; + const htmlFileName = `pages/${pageId}/index.html`; + + const originalBuffer = await gcsService.downloadFile(screenshotFileName); + const base64Original = originalBuffer.toString('base64'); + + // Download and read the HTML content + const htmlBuffer = await gcsService.downloadFile(htmlFileName); + const htmlContent = htmlBuffer.toString('utf8'); + + const prompt = ` + Compare this original screenshot with the HTML code that was generated to replicate it. + + Analyze and rate the similarity on a scale of 1-10, paying special attention to: + + 1. LAYOUT ACCURACY: + - Overall page structure and component arrangement + - Spacing, margins, and padding consistency + - Grid/flexbox alignment and distribution + - Responsive design elements + + 2. TYPOGRAPHY & FONT FORMATTING: + - Font family, size, and weight matching + - Line height and letter spacing + - Text alignment and justification + - Heading hierarchy and consistency + - Text color and contrast accuracy + + 3. COLOR MATCHING: + - Background colors and gradients + - Text colors and readability + - Button and interactive element colors + - Border colors and styling + + 4. ELEMENT POSITIONING: + - Precise placement of all UI elements + - Alignment of buttons, inputs, and forms + - Icon and image positioning + - Consistent spacing between elements + + 5. PAGE FORMATTING: + - Overall page dimensions and proportions + - Section breaks and content organization + - Visual hierarchy maintenance + - Brand consistency and styling + + 6. DETAILED FORMATTING: + - Border radius and shadows + - Input field styling and placeholder text + - Button hover states and interactions + - Form validation styling + + HTML Code Analysis: + ${htmlContent.substring(0, 8000)} // Extended for better analysis + + Provide a JSON response with detailed scoring: + { + "similarity_score": number (1-10), + "layout_score": number (1-10), + "color_score": number (1-10), + "typography_score": number (1-10), + "positioning_score": number (1-10), + "formatting_score": number (1-10), + "font_accuracy_score": number (1-10), + "feedback": "detailed feedback focusing on typography, formatting, and layout precision", + "font_issues": ["specific font/typography problems"], + "formatting_issues": ["specific page formatting problems"], + "improvements": ["detailed suggestions for typography and formatting fixes"] + } + `; + + const response = await genAI.models.generateContent({ + model: process.env.GEMINI_MODEL || "gemini-2.5-flash", + contents: [ + { + role: "user", + parts: [ + { text: prompt }, + { + inlineData: { + data: base64Original, + mimeType: 'image/png' + } + } + ] + } + ] + }); + + const comparison = JSON.parse(response.text); + + res.json({ + success: true, + pageId: pageId, + comparison: comparison + }); + + } catch (error) { + console.error('Error comparing page:', error); + res.status(500).json({ + error: 'Failed to compare page', + details: error.message + }); + } +}); + +// Endpoint to list all generated pages +app.get('/api/pages', async (req, res) => { + try { + const pages = await gcsService.listFiles('pages/'); + + // Group files by page ID and create page objects + const pageMap = new Map(); + + pages.forEach(file => { + // Extract pageId from path like 'pages/uuid/index.html' + const pathParts = file.name.split('/'); + if (pathParts.length >= 3 && pathParts[0] === 'pages') { + const pageId = pathParts[1]; + if (!pageMap.has(pageId)) { + pageMap.set(pageId, { + id: pageId, + createdAt: file.created + }); + } + + // Add URL for HTML files + if (pathParts[2] === 'index.html') { + pageMap.get(pageId).url = file.publicUrl; + pageMap.get(pageId).previewUrl = file.publicUrl; + } + // Add screenshot URL for PNG files + if (pathParts[2] === 'original.png') { + pageMap.get(pageId).screenshotUrl = file.publicUrl; + } + } + }); + + const pageList = Array.from(pageMap.values()) + .filter(page => page.url) // Only include pages with HTML files + .sort((a, b) => new Date(b.createdAt) - new Date(a.createdAt)); + + res.json({ + success: true, + pages: pageList + }); + + } catch (error) { + console.error('Error listing pages:', error); + res.status(500).json({ + error: 'Failed to list pages', + details: error.message + }); + } +}); + +// Health check endpoint +app.get('/api/health', (req, res) => { + res.json({ status: 'ok', service: 'webpage-replicator-backend' }); +}); + +// Test Clickthrough integration endpoint +app.post('/api/test-clickthrough', (req, res) => { + const testHTML = ` +Test + +
+ +
`; + + const result = addClickthroughToHTML(testHTML, 'test-clickthrough-id', 'api.in.spotdraft.com'); + + res.json({ + success: true, + original: testHTML, + withClickthrough: result, + hasClickthrough: result.includes('clickthrough-host') + }); +}); + +app.listen(PORT, () => { + console.log(`Backend server running on http://localhost:${PORT}`); + console.log('Make sure to set GEMINI_API_KEY and GCS_BUCKET_NAME environment variables'); +}); \ No newline at end of file diff --git a/backend/server.js b/backend/server-original.js similarity index 100% rename from backend/server.js rename to backend/server-original.js diff --git a/backend/server.js.backup b/backend/server.js.backup new file mode 100644 index 0000000..c1da3f2 --- /dev/null +++ b/backend/server.js.backup @@ -0,0 +1,577 @@ +import express from 'express'; +import multer from 'multer'; +import cors from 'cors'; +import { GoogleGenAI } from '@google/genai'; +import { v4 as uuidv4 } from 'uuid'; +import fs from 'fs/promises'; +import path from 'path'; +import sharp from 'sharp'; +import dotenv from 'dotenv'; + +// Load environment variables +dotenv.config(); + +const app = express(); +const PORT = process.env.PORT || 3001; + +// Middleware +app.use(cors()); +app.use(express.json()); +app.use(express.static(path.join(process.cwd(), '..', 'generated-pages'))); + +// Configure multer for file uploads +const storage = multer.memoryStorage(); +const upload = multer({ + storage: storage, + limits: { fileSize: 10 * 1024 * 1024 } // 10MB limit +}); + +// Initialize Gemini AI +const genAI = new GoogleGenAI({ + apiKey: process.env.GEMINI_API_KEY +}); + +// Function to remove existing terms/conditions elements to avoid duplicates with Clickthrough SDK +function removeExistingTermsElements(html) { + console.log('Scanning for existing terms/conditions elements to remove...'); + + // Patterns to detect and remove existing terms/conditions elements + const termsPatterns = [ + // Checkboxes with terms-related text + /]*type\s*=\s*["']checkbox["'][^>]*[^>]*>\s*[^<]*(?:terms|condition|agree|policy|privacy)[^<]*<\/?\w*>/gi, + /]*>[^<]*]*type\s*=\s*["']checkbox["'][^>]*>\s*[^<]*(?:terms|condition|agree|policy|privacy)[^<]*<\/label>/gi, + + // Divs or paragraphs containing checkbox + terms text + /]*>[^<]*]*type\s*=\s*["']checkbox["'][^>]*>[^<]*(?:terms|condition|agree|policy|privacy)[^<]*<\/div>/gi, + /]*>[^<]*]*type\s*=\s*["']checkbox["'][^>]*>[^<]*(?:terms|condition|agree|policy|privacy)[^<]*<\/p>/gi, + + // Text containing "I agree to" or similar + /<[^>]*>[^<]*(?:I\s+agree\s+to|By\s+clicking|Accept\s+terms|Terms\s+and\s+conditions|Privacy\s+policy)[^<]*<\/[^>]*>/gi, + + // Links to terms/privacy within form contexts + /]*href[^>]*(?:terms|privacy|condition)[^>]*>[^<]*(?:terms|privacy|condition)[^<]*<\/a>/gi, + + // Standalone checkboxes near submit buttons (likely terms acceptance) + /]*type\s*=\s*["']checkbox["'][^>]*>\s*(?=.*(?:submit|sign.up|register))/gi + ]; + + let removedCount = 0; + + // Remove each pattern found + termsPatterns.forEach((pattern, index) => { + const matches = html.match(pattern); + if (matches) { + console.log(`Found ${matches.length} existing terms elements (pattern ${index + 1}):`, matches); + html = html.replace(pattern, ''); + removedCount += matches.length; + } + }); + + // Additional cleanup: Remove empty labels, divs, or paragraphs that might be left behind + html = html.replace(/<(label|div|p)[^>]*>\s*\s*<\/\1>/gi, ''); + html = html.replace(/\s*/gi, ''); + + console.log(`Removed ${removedCount} existing terms/conditions elements`); + + return html; +} + +// Function to remove image references and replace with styled elements +function removeImageReferences(html) { + console.log('Scanning for image references to remove...'); + + let removedCount = 0; + + // Remove all tags and replace with styled placeholders + html = html.replace(/]*>/gi, (match) => { + removedCount++; + + // Extract alt text if available for replacement text + const altMatch = match.match(/alt\s*=\s*["']([^"']*)["']/i); + const altText = altMatch ? altMatch[1] : 'Image'; + + // Extract any classes for styling context + const classMatch = match.match(/class\s*=\s*["']([^"']*)["']/i); + const classes = classMatch ? classMatch[1] : ''; + + // Determine replacement based on context + if (classes.includes('logo') || altText.toLowerCase().includes('logo')) { + // Replace logos with styled text + return `
${altText}
`; + } else if (classes.includes('icon') || altText.toLowerCase().includes('icon')) { + // Replace icons with CSS symbols + return `
●
`; + } else { + // Replace other images with colored placeholders + return `
${altText}
`; + } + }); + + // Remove any background-image references in CSS (they would be broken) + html = html.replace(/background-image\s*:\s*url\([^)]*\)\s*;?/gi, ''); + + // Remove any references to image files in CSS + html = html.replace(/url\(['"]?[^'"]*\.(jpg|jpeg|png|gif|svg|webp)['"]?\)/gi, ''); + + console.log(`Removed ${removedCount} image references and replaced with styled elements`); + + return html; +} + +// Function to add Clickthrough integration to generated HTML +function addClickthroughToHTML(html, clickthroughId, clusterId) { + try { + console.log('Adding Clickthrough integration...', { clickthroughId, clusterId }); + + // STEP 1: Remove existing terms/conditions elements to avoid duplicates + html = removeExistingTermsElements(html); + + // STEP 2: Add SDK script to + const sdkScript = ``; + html = html.replace('', ` ${sdkScript}\n`); + + // STEP 3: Find CTA button patterns and add Clickthrough div above them + const ctaPatterns = [ + /(]*type\s*=\s*["']submit["'][^>]*>)/gi, + /(]*class\s*=\s*["'][^"']*(?:submit|cta|signup|register|join|trial)[^"']*["'][^>]*>)/gi, + /(]*type\s*=\s*["']submit["'][^>]*\/?>)/gi, + /(]*>[^<]*(?:submit|sign.up|register|join|continue|get.started|start.*trial|free.trial)[^<]*<\/button>)/gi, + /(]*class\s*=\s*["']submit-button["'][^>]*>.*?<\/button>)/gi + ]; + + let clickthroughAdded = false; + + for (const pattern of ctaPatterns) { + if (html.match(pattern) && !clickthroughAdded) { + console.log('Found CTA pattern match, adding Clickthrough div'); + html = html.replace(pattern, (match) => { + clickthroughAdded = true; + return ` +
+ + ${match}`; + }); + break; + } + } + + console.log('Clickthrough added:', clickthroughAdded); + + // Fallback: If no CTA found, add before the last closing form tag or div + if (!clickthroughAdded) { + const fallbackPatterns = [ + /(.*<\/form>)/gi, + /(.*<\/div>\s*<\/body>)/gi + ]; + + for (const pattern of fallbackPatterns) { + if (html.match(pattern)) { + html = html.replace(pattern, (match) => { + return match.replace(/(<\/(?:form|div)>)/, ` +
+ + $1`); + }); + clickthroughAdded = true; + break; + } + } + } + + // 3. Add Clickthrough JavaScript before - EXACT implementation per documentation + const clickthroughJS = ` + `; + + html = html.replace('', `${clickthroughJS}\n`); + + return html; + } catch (error) { + console.error('Error adding Clickthrough to HTML:', error); + return html; // Return original HTML if processing fails + } +} + +// Endpoint to upload screenshot and generate webpage +app.post('/api/generate-page', upload.single('screenshot'), async (req, res) => { + try { + if (!req.file) { + return res.status(400).json({ error: 'No screenshot uploaded' }); + } + + const pageId = uuidv4(); + const screenshotBuffer = req.file.buffer; + + // Get Clickthrough parameters from form data + const clickthroughId = req.body.clickthroughId; + const clusterId = req.body.clusterId; + + console.log('Received parameters:', { + clickthroughId, + clusterId, + hasFile: !!req.file, + bodyKeys: Object.keys(req.body) + }); + + // Convert image to base64 for Gemini + const base64Image = screenshotBuffer.toString('base64'); + const mimeType = req.file.mimetype; + + // Generate HTML/CSS/JS using Gemini - CLEAN GENERATION WITHOUT CLICKTHROUGH + + // Universal instructions that always apply + const imageHandlingInstructions = ` + + CRITICAL IMAGE HANDLING INSTRUCTIONS (ALWAYS APPLY): + - DO NOT include any tags or image references from the screenshot + - DO NOT attempt to replicate logos, photos, graphics, or any visual images + - REPLACE image areas with appropriate styled elements: + * For logos: Use styled text/typography or CSS-based geometric shapes + * For decorative images: Use CSS backgrounds, gradients, or colored divs + * For photos: Use placeholder colored backgrounds or CSS patterns + * For icons: Use CSS symbols, Unicode characters, or styled elements + - Focus on creating a clean, functional page without broken image links + - Use colors, typography, and CSS styling to maintain visual hierarchy instead of images + `; + + let clickthroughInstructions = ''; + if (clickthroughId && clusterId) { + clickthroughInstructions = ` + + SPECIAL INSTRUCTION FOR CLICKTHROUGH INTEGRATION: + - If the screenshot contains checkboxes, "I agree to terms", "Terms and Conditions", or privacy policy acceptance elements, DO NOT replicate them + - Skip any terms/conditions/privacy policy checkboxes or acceptance UI elements + - Focus on replicating the form fields and layout, but omit terms acceptance elements + - The terms acceptance will be handled by a separate integration system + `; + } + + const prompt = ` + Analyze this screenshot of a webpage and generate complete HTML, CSS, and JavaScript code to replicate it as closely as possible. + + Critical Requirements for Accurate Replication: + + TYPOGRAPHY & FONTS: + - Match exact font families, sizes, and weights + - Replicate line-height, letter-spacing, and text alignment + - Preserve heading hierarchy and text formatting + - Ensure proper font loading and fallbacks + + PAGE FORMATTING & LAYOUT: + - Create pixel-perfect replica of spacing, margins, and padding + - Match exact element positioning and alignment + - Preserve proportions and visual hierarchy + - Implement responsive design with proper breakpoints + + VISUAL DETAILS: + - Match colors exactly (backgrounds, text, borders) + - Replicate shadows, gradients, and visual effects + - Preserve border radius, styling, and decorative elements + - Maintain consistent spacing between all elements + + TECHNICAL REQUIREMENTS: + - Use modern CSS (flexbox, grid) for accurate layout + - Include all interactive elements and form styling + - Implement proper semantic HTML structure + - Add inline CSS and JavaScript in single HTML file + - Ensure full functionality with form validation and interactions + ${imageHandlingInstructions} + ${clickthroughInstructions} + + Focus on maintaining the exact visual appearance and formatting integrity of the original design. + + IMPORTANT: Return ONLY the complete HTML code with embedded CSS and JavaScript. Do not use markdown code blocks, backticks, or any formatting - just return the raw HTML code directly. + `; + + const response = await genAI.models.generateContent({ + model: process.env.GEMINI_MODEL || "gemini-2.5-flash", + contents: [ + { + role: "user", + parts: [ + { text: prompt }, + { + inlineData: { + data: base64Image, + mimeType: mimeType + } + } + ] + } + ] + }); + + let generatedHTML = response.text; + + // Clean up markdown code block formatting if present + generatedHTML = generatedHTML + .replace(/^```html\s*/i, '') // Remove opening ```html + .replace(/^```\s*/gm, '') // Remove any other opening ``` + .replace(/\s*```$/gm, '') // Remove closing ``` + .replace(/```html/gi, '') // Remove any remaining ```html + .replace(/```/g, '') // Remove any remaining ``` + .trim(); + + // POST-PROCESS: Clean up any image references (always apply) + generatedHTML = removeImageReferences(generatedHTML); + + // POST-PROCESS: Add Clickthrough integration if parameters provided + if (clickthroughId && clusterId) { + generatedHTML = addClickthroughToHTML(generatedHTML, clickthroughId, clusterId); + } + + // Create directory for this page + const pageDir = path.join(process.cwd(), '..', 'generated-pages', pageId); + await fs.mkdir(pageDir, { recursive: true }); + + // Save the generated HTML + const htmlPath = path.join(pageDir, 'index.html'); + await fs.writeFile(htmlPath, generatedHTML); + + // Save the original screenshot for comparison + const screenshotPath = path.join(pageDir, 'original.png'); + await fs.writeFile(screenshotPath, screenshotBuffer); + + res.json({ + success: true, + pageId: pageId, + url: `http://localhost:${PORT}/${pageId}`, + previewUrl: `http://localhost:${PORT}/${pageId}/index.html` + }); + + } catch (error) { + console.error('Error generating page:', error); + res.status(500).json({ + error: 'Failed to generate page', + details: error.message + }); + } +}); + +// Endpoint to compare generated page with original screenshot +app.post('/api/compare-page/:pageId', async (req, res) => { + try { + const { pageId } = req.params; + const pageDir = path.join(process.cwd(), '..', 'generated-pages', pageId); + + // Read the original screenshot + const originalPath = path.join(pageDir, 'original.png'); + const originalBuffer = await fs.readFile(originalPath); + const base64Original = originalBuffer.toString('base64'); + + // For comparison, we'll need a screenshot of the generated page + // This would typically be done with a headless browser like Puppeteer + // For now, we'll analyze based on the HTML structure + + const htmlPath = path.join(pageDir, 'index.html'); + const htmlContent = await fs.readFile(htmlPath, 'utf8'); + + const prompt = ` + Compare this original screenshot with the HTML code that was generated to replicate it. + + Analyze and rate the similarity on a scale of 1-10, paying special attention to: + + 1. LAYOUT ACCURACY: + - Overall page structure and component arrangement + - Spacing, margins, and padding consistency + - Grid/flexbox alignment and distribution + - Responsive design elements + + 2. TYPOGRAPHY & FONT FORMATTING: + - Font family, size, and weight matching + - Line height and letter spacing + - Text alignment and justification + - Heading hierarchy and consistency + - Text color and contrast accuracy + + 3. COLOR MATCHING: + - Background colors and gradients + - Text colors and readability + - Button and interactive element colors + - Border colors and styling + + 4. ELEMENT POSITIONING: + - Precise placement of all UI elements + - Alignment of buttons, inputs, and forms + - Icon and image positioning + - Consistent spacing between elements + + 5. PAGE FORMATTING: + - Overall page dimensions and proportions + - Section breaks and content organization + - Visual hierarchy maintenance + - Brand consistency and styling + + 6. DETAILED FORMATTING: + - Border radius and shadows + - Input field styling and placeholder text + - Button hover states and interactions + - Form validation styling + + HTML Code Analysis: + ${htmlContent.substring(0, 8000)} // Extended for better analysis + + Provide a JSON response with detailed scoring: + { + "similarity_score": number (1-10), + "layout_score": number (1-10), + "color_score": number (1-10), + "typography_score": number (1-10), + "positioning_score": number (1-10), + "formatting_score": number (1-10), + "font_accuracy_score": number (1-10), + "feedback": "detailed feedback focusing on typography, formatting, and layout precision", + "font_issues": ["specific font/typography problems"], + "formatting_issues": ["specific page formatting problems"], + "improvements": ["detailed suggestions for typography and formatting fixes"] + } + `; + + const response = await genAI.models.generateContent({ + model: process.env.GEMINI_MODEL || "gemini-2.5-flash", + contents: [ + { + role: "user", + parts: [ + { text: prompt }, + { + inlineData: { + data: base64Original, + mimeType: 'image/png' + } + } + ] + } + ] + }); + + const comparison = JSON.parse(response.text); + + res.json({ + success: true, + pageId: pageId, + comparison: comparison + }); + + } catch (error) { + console.error('Error comparing page:', error); + res.status(500).json({ + error: 'Failed to compare page', + details: error.message + }); + } +}); + +// Endpoint to list all generated pages +app.get('/api/pages', async (req, res) => { + try { + const pagesDir = path.join(process.cwd(), '..', 'generated-pages'); + const pages = await fs.readdir(pagesDir); + + const pageList = await Promise.all( + pages.map(async (pageId) => { + try { + const pageDir = path.join(pagesDir, pageId); + const stats = await fs.stat(pageDir); + return { + id: pageId, + url: `http://localhost:${PORT}/${pageId}`, + createdAt: stats.birthtime + }; + } catch (error) { + return null; + } + }) + ); + + res.json({ + success: true, + pages: pageList.filter(page => page !== null) + }); + + } catch (error) { + console.error('Error listing pages:', error); + res.status(500).json({ + error: 'Failed to list pages', + details: error.message + }); + } +}); + +// Health check endpoint +app.get('/api/health', (req, res) => { + res.json({ status: 'ok', service: 'webpage-replicator-backend' }); +}); + +// Test Clickthrough integration endpoint +app.post('/api/test-clickthrough', (req, res) => { + const testHTML = ` +Test + +
+ +
`; + + const result = addClickthroughToHTML(testHTML, 'test-clickthrough-id', 'api.in.spotdraft.com'); + + res.json({ + success: true, + original: testHTML, + withClickthrough: result, + hasClickthrough: result.includes('clickthrough-host') + }); +}); + +app.listen(PORT, () => { + console.log(`Backend server running on http://localhost:${PORT}`); + console.log('Make sure to set GEMINI_API_KEY environment variable'); +}); \ No newline at end of file diff --git a/deploy.sh b/deploy.sh new file mode 100644 index 0000000..8e0b887 --- /dev/null +++ b/deploy.sh @@ -0,0 +1,75 @@ +#!/bin/bash + +# Cloud Run Deployment Script for Webpage Replicator +# Make sure to replace PROJECT_ID with your actual Google Cloud Project ID + +set -e + +# Configuration +PROJECT_ID="your-project-id" # Replace with your actual project ID +REGION="us-central1" # Replace with your preferred region +FRONTEND_SERVICE="webpage-replicator-frontend" +BACKEND_SERVICE="webpage-replicator-backend" + +echo "🚀 Starting deployment to Google Cloud Run..." + +# Check if gcloud is installed +if ! command -v gcloud &> /dev/null; then + echo "❌ Error: gcloud CLI is not installed. Please install it first." + exit 1 +fi + +# Check if user is authenticated +if ! gcloud auth list --filter=status:ACTIVE --format="value(account)" | grep -q .; then + echo "❌ Error: You are not authenticated with gcloud. Please run 'gcloud auth login' first." + exit 1 +fi + +# Set the project +gcloud config set project $PROJECT_ID + +# Enable required APIs +echo "📋 Enabling required APIs..." +gcloud services enable cloudbuild.googleapis.com +gcloud services enable run.googleapis.com + +# Build and deploy backend +echo "🔧 Building and deploying backend..." +cd backend +gcloud builds submit --tag gcr.io/$PROJECT_ID/$BACKEND_SERVICE +gcloud run deploy $BACKEND_SERVICE \ + --image gcr.io/$PROJECT_ID/$BACKEND_SERVICE \ + --platform managed \ + --region $REGION \ + --allow-unauthenticated \ + --port 3001 \ + --memory 1Gi \ + --cpu 1 \ + --max-instances 100 +cd .. + +# Build and deploy frontend +echo "🎨 Building and deploying frontend..." +cd frontend +gcloud builds submit --tag gcr.io/$PROJECT_ID/$FRONTEND_SERVICE +gcloud run deploy $FRONTEND_SERVICE \ + --image gcr.io/$PROJECT_ID/$FRONTEND_SERVICE \ + --platform managed \ + --region $REGION \ + --allow-unauthenticated \ + --port 3000 \ + --memory 512Mi \ + --cpu 1 \ + --max-instances 100 +cd .. + +echo "✅ Deployment complete!" +echo "" +echo "📝 Service URLs:" +echo "Backend: $(gcloud run services describe $BACKEND_SERVICE --region=$REGION --format='value(status.url)')" +echo "Frontend: $(gcloud run services describe $FRONTEND_SERVICE --region=$REGION --format='value(status.url)')" +echo "" +echo "💡 Don't forget to:" +echo "1. Update your frontend to use the backend URL" +echo "2. Set up environment variables for the backend (API keys, etc.)" +echo "3. Configure CORS settings if needed" \ No newline at end of file diff --git a/frontend/.dockerignore b/frontend/.dockerignore new file mode 100644 index 0000000..eff8162 --- /dev/null +++ b/frontend/.dockerignore @@ -0,0 +1,12 @@ +node_modules +npm-debug.log +.git +.gitignore +README.md +.env +.env.local +.env.development.local +.env.test.local +.env.production.local +.DS_Store +*.log \ No newline at end of file diff --git a/frontend/Dockerfile b/frontend/Dockerfile new file mode 100644 index 0000000..6339c77 --- /dev/null +++ b/frontend/Dockerfile @@ -0,0 +1,20 @@ +# Use Node.js official image +FROM node:18-alpine + +# Set working directory +WORKDIR /app + +# Copy package files +COPY package*.json ./ + +# Install dependencies +RUN npm ci --only=production + +# Copy application files +COPY . . + +# Expose port +EXPOSE 3000 + +# Start the application +CMD ["npm", "start"] \ No newline at end of file diff --git a/frontend/cloudrun.yaml b/frontend/cloudrun.yaml new file mode 100644 index 0000000..989d25b --- /dev/null +++ b/frontend/cloudrun.yaml @@ -0,0 +1,47 @@ +apiVersion: serving.knative.dev/v1 +kind: Service +metadata: + name: webpage-replicator-frontend + annotations: + run.googleapis.com/ingress: all + run.googleapis.com/ingress-status: all +spec: + template: + metadata: + annotations: + autoscaling.knative.dev/maxScale: "100" + run.googleapis.com/cpu-throttling: "false" + run.googleapis.com/execution-environment: gen2 + spec: + containerConcurrency: 80 + timeoutSeconds: 300 + containers: + - image: gcr.io/PROJECT_ID/webpage-replicator-frontend:latest + ports: + - name: http1 + containerPort: 3000 + env: + - name: PORT + value: "3000" + resources: + limits: + cpu: 1000m + memory: 512Mi + requests: + cpu: 100m + memory: 128Mi + livenessProbe: + httpGet: + path: / + port: 3000 + initialDelaySeconds: 30 + periodSeconds: 30 + readinessProbe: + httpGet: + path: / + port: 3000 + initialDelaySeconds: 5 + periodSeconds: 10 + traffic: + - percent: 100 + latestRevision: true \ No newline at end of file