Overview

ArtTalk is a web application that allows users to create and view comments on a painting. It is designed to be used in a museum setting, where users can view comments from other visitors and add their own. It is also designed to be used by curators, who can view a heatmap of where comments are located on the painting.

This was a project for the MIT class Intelligent Multimodal User Interfaces (6.8510).

Access ArtTalk online at https://arttalk.vercel.app/.

Tech Stack

ArtTalk is built using the following technologies:

• Next.js → React framework for web applications
• Tailwind CSS → CSS framework for styling
• Prisma → Database ORM
• PostgreSQL → Database
• Google Mediapipe → Hand landmark detection and gesture recognition
• Vercel → Web hosting
• Webkit Speech Recognition and Synthesis API → Speech recognition

File Structure

• /prisma
  • schema.prisma → Database schema for storing comments in a database
• /public
  • /models
    • hand_landmarker.task → Model to determine hand landmark positions (from Google)
• src/
  • components/
    • AudioFeedback.tsx → Component for speaking feedback to user
    • CanvasWithGuesture.tsx → Painting component with landmark detection and gesture recognition (main view)
    • Feedback.tsx → Component for text based feedback (at bottom of screen)
    • Heatmap.tsx → Component to show heatmap of where comments are located
    • StringCarousel.tsx → Component to scroll through comments visually
  • constants/
    • constant.tsx → Datatype, graphics, and feedback text string constants
  • pages/
    • api/
      • createComment.tsx → API endpoint to add a new comment to the database
      • getAllPoints.tsx → API endpoint to retrieve all comments from database
      • getPointsAtLocation.tsx → API endpoint to get all comments from the database near a point
    • _app.tsx → Next.js entry point (auto-generated)
    • heatmap.tsx → Web entry point for /heatmap
    • index.tsx → Web entry point for / (main page)
    • test.tsx → Web entry point for /test (finger position graph page)
  • server/
    • db.ts → Database variable setup
  • styles/
    • global.css → Global stylesheet
• .env → Contains DATABASE_URL for connecting to comments database (not available publicly)
• .eslintrc.js → ESLint config file
• .gitignore → Files to ignore for git
• next.config.mjs → Next.js config file
• package-lock.json → Dependency files
• package.json → Package list
• postcss.config.cjs → PostCSS config file
• prettier.config.cjs → Prettier (style formatting) config file
• README.md → This file
• tailwind.config.ts → Tailwind (also style) config file
• tsconfig.json → Typescript config file

Instructions to view ArtTalk

ArtTalk is live on the internet! You do not need to run it locally if you do not want to.

ArtTalk Main Page

Curator Heatmap

Finger Position Graph Page (used to determine algorithm for shake-to-undo)

Please use on Google Chrome, and make sure camera and microphone access is enabled.

Instructions to run code (Created using node v18.12.1 on MacOS)

If you would prefer to run ArtTalk locally, follow these instructions.

First, you need to put the .env file in the main directory. Its location is listed in the table of contents. This file will be emailed separately, as it is insecure to host on GitHub.

npm install
npm run dev

Please use on Google Chrome, and make sure camera and microphone access is enabled.

Credits

This project was created by

• Trudy Painter
• Maria Ascanio
• Andrew Stoddard