🔬 PIDL - Persona in the Loop Research System

Doctoral Research on Competency Transfer in Human-AI Collaboration Models

PIDL (Persona in the Loop) is a research platform that investigates the impact of AI personas at different competency levels on user learning in code generation. The system supports Solidity smart contract development tasks with 10 different personas based on the Dreyfus Model of Skill Acquisition.

---

📋 Table of Contents

• Features
• Project Structure
• Installation
• Usage
• System Architecture
• Research Methodology
• Technical Details
• Database
• Admin Dashboard
• Development
• License

---

✨ Features

🎭 10 Different AI Personas

• 5 Dreyfus Levels: Novice, Advanced Beginner, Competent, Proficient, Expert
• 2 Domains: Education (Pedagogical) and Technology (Technical)
• Each persona has its own unique coding style, philosophy, and system prompt

📊 Competency Assessment System

• Dual Domain Assessment: Technical and pedagogical competencies are measured separately
• 10-Question Survey: 5 technical + 5 pedagogical questions
• Dreyfus Model Mapping: Scores are automatically converted to Dreyfus levels

🤖 Mathematical Recommendation Engine

6-Stage Calculation Process:

1. User Vector Creation: Converting user competency to a 10-dimensional vector
2. Persona Vectors Matching: Defining characteristic vectors for each persona
3. Similarity Score (S): Hybrid Distance (Cosine + Euclidean)
4. Competency Match (C): Gaussian ZPD (Zone of Proximal Development)
5. Performance Prediction (P): Logistic Regression Model
6. Learning Trajectory (L): Exponential Growth Model

Two Recommendation Modes:

• Similar AI: Persona from your strong domain, at your level
• Complementary AI: Persona from your weak domain, to complement you

💻 6 Blockchain Tasks

1. Diploma Verification: Blockchain-based diploma system
2. NFT Creation: NFT minting and transfer
3. Access Control: Role-based access control
4. Loan System: DeFi loan contract
5. Incentive Mechanism: Token incentive system
6. DAO Management: Decentralized autonomous organization

🔬 6-Stage Content Analysis

For each generated code and prompt:

1. Prompt Feature Analysis: Word count, technical terms, clarity, specificity
2. Prompt Similarity Analysis: Cosine Similarity, Jaccard Similarity
3. Code Structure Analysis: Line count, comment ratio, function count
4. Code Complexity Analysis: Cyclomatic Complexity, Nesting Depth
5. Code Quality Assessment: Readability, maintainability, documentation
6. Comparative Analysis: Persona comparison

📈 Data Collection

• Pre-Test / Post-Test: Learning gain measurement
• NASA-TLX: Cognitive load assessment
• AI Code Evaluation: Code quality from user perspective
• Final Survey: General experience evaluation

📊 Admin Dashboard

• Participant analysis and demographic distribution
• Task performance metrics
• Learning gain analysis
• Cognitive load comparison
• AI persona performance comparison
• Detailed data tables and CSV export

---

📁 Project Structure

pidl/
├── research_app.py              # Main research application
├── app.py                       # Demo/test application
├── personas.py                  # 10 persona definitions
├── competency_assessment.py     # Competency assessment
├── recommendation_engine.py     # Mathematical recommendation engine
├── content_analyzer.py          # 6-stage content analysis
├── database/
│   ├── models.py               # SQLAlchemy models
│   ├── database.py             # Database connection
│   └── research_data.db        # SQLite database
├── research_modules/
│   ├── consent_form.py         # Consent form
│   ├── pre_post_test.py        # Pre/post test
│   ├── nasa_tlx.py             # NASA-TLX form
│   ├── ai_evaluation.py        # AI code evaluation
│   ├── final_survey.py         # Final survey
│   └── data_logger.py          # Data logging system
├── tasks/
│   ├── base_task.py            # Task base class
│   ├── task1_diploma.py        # Task 1: Diploma
│   ├── task2_nft.py            # Task 2: NFT
│   ├── task3_access.py         # Task 3: Access Control
│   ├── task4_loan.py           # Task 4: Loan
│   ├── task5_incentive.py      # Task 5: Incentive
│   └── task6_dao.py            # Task 6: DAO
├── pages/
│   └── 1_📊_Yonetim_Paneli.py  # Admin dashboard
├── requirements.txt            # Python dependencies
├── setup.sh                    # Installation script
└── README.md                   # This file

---

🚀 Installation

Requirements

• Python 3.8+
• OpenAI API key (or other LLM providers)

Steps

1. Clone the project:

cd /Users/mustafa/Downloads/pidl-main/Downloads/pidl

2. Run the installation script:

chmod +x setup.sh
./setup.sh

Or manual installation:

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install dependencies
pip install --upgrade pip
pip install -r requirements.txt

3. Create .env file:

# Create .env file
cat > .env << EOF
# OpenAI API Key
OPENAI_API_KEY=your_openai_api_key_here

# Model Settings
DEFAULT_MODEL=gpt-4o-mini
TEMPERATURE=0.7
MAX_TOKENS=2000
EOF

4. Initialize the database:

python -c "from database.database import init_database; init_database()"

---

💻 Usage

Starting the Research Application

source venv/bin/activate
streamlit run research_app.py

Navigate to http://localhost:8501 in your browser.

Accessing the Admin Dashboard

streamlit run pages/1_📊_Yonetim_Paneli.py

Or as a Streamlit multi-page application:

streamlit run research_app.py
# Select "📊 Yönetim Paneli" from the sidebar

Demo/Test Application

streamlit run app.py

---

🏗️ System Architecture

Flow Diagram

1. Consent Form
   ↓
2. Competency Assessment (10 questions)
   ↓
3. Persona Recommendation (6-Stage Mathematical Calculation)
   ├─ Similar AI (Strong domain)
   └─ Complementary AI (Weak domain)
   ↓
4. 6 Task Loop (For each task):
   ├─ Pre-Test (3 questions)
   ├─ Code Generation (with AI Persona)
   ├─ 6-Stage Content Analysis
   ├─ Post-Test (5 questions)
   ├─ NASA-TLX (Cognitive load)
   └─ AI Code Evaluation
   ↓
5. Final Survey
   ↓
6. Completion

Persona Assignment Strategy

• Odd-numbered tasks (1, 3, 5): Similar AI
• Even-numbered tasks (2, 4, 6): Complementary AI

This strategy ensures users gain experience in both their strong and weak domains.

---

🔬 Research Methodology

Theoretical Foundations

1. Dreyfus Model of Skill Acquisition (1980)

   • 5-stage competency model
   • Novice → Expert transition process

2. Cognitive Load Theory (Sweller, 1988)

   • Intrinsic, extraneous, and germane cognitive load
   • Learning optimization

3. Zone of Proximal Development (Vygotsky, 1978)

   • Optimal difficulty level
   • Gaussian ZPD model

4. Multi-Criteria Decision Analysis (MCDA)

   • Multi-criteria decision making
   • Weighted scoring

Research Questions

1. Does Similar AI or Complementary AI provide more effective learning?
2. How does persona level (Dreyfus) affect learning gain?
3. How does cognitive load (NASA-TLX) vary by persona type?
4. What is the relationship between code quality and learning gain?

Data Collection

• Dependent Variables: Learning gain (post-test - pre-test), cognitive load, perceived code quality
• Independent Variables: Persona type (Similar/Complementary), persona level, user competency
• Control Variables: Task difficulty, code generation time, prompt quality

---

🔧 Technical Details

Mathematical Models

1. User Vector Creation

u = [technical_skill, domain_knowledge, ai_experience, learning_goal,
     procedural_knowledge, declarative_knowledge, conditional_knowledge,
     cognitive_capacity, pattern_recognition, abstraction_level]

2. Similarity Score (S)

S = 0.6 × cosine_similarity(u, p) + 0.4 × (1 - euclidean(u, p) / √10)

3. Competency Match (C)

C = exp(-λ × |skill_gap|²) × alignment

4. Performance Prediction (P)

P = σ(β₀ + β₁·skill + β₂·quality + β₃·similarity + β₄·task_complexity)

5. Learning Trajectory (L)

L = L_max × (1 - e^(-k·τ)) × potential

6. Final Recommendation Score

R_similar = 0.30·S + 0.35·C + 0.25·P + 0.10·L
R_complementary = 0.30·(1-S) + 0.35·D + 0.25·P + 0.10·L

Code Analysis Metrics

• Cyclomatic Complexity: McCabe (1976)
• Halstead Metrics: Code volume and difficulty
• Maintainability Index: Maintainability score
• TF-IDF Cosine Similarity: Text similarity

---

🗄️ Database

SQLite Database Structure

Tables:

• participants: Participant information and competency scores
• task_sessions: Task sessions
• pre_post_tests: Pre-test and post-test results
• generated_codes: Generated codes and prompts
• nasa_tlx_responses: NASA-TLX cognitive load measurements
• ai_code_evaluations: AI code evaluations
• final_evaluations: Final survey responses

Data Access

from database.database import DatabaseSession
from database.models import Participant

with DatabaseSession() as session:
    participants = session.query(Participant).all()

---

📊 Admin Dashboard

The admin dashboard provides the following analyses:

1. General Statistics

   • Total number of participants
   • Number of completed tasks
   • Number of generated codes
   • Average cognitive load

2. Participant Analysis

   • Competency level distribution
   • Technical vs Pedagogical scores
   • Demographic distribution

3. Task Performance Analysis

   • Task completion status
   • AI persona usage
   • Task durations

4. Learning Gain Analysis

   • Pre-test vs Post-test comparison
   • Task-based gains

5. Cognitive Load Analysis

   • NASA-TLX dimensions
   • AI type-based cognitive load

6. AI Code Evaluation Analysis

   • Code quality metrics
   • Persona comparison

7. Content Analysis

   • 6-stage mathematical analysis results
   • Persona performance comparison

8. Data Export

   • CSV export
   • Excel export (coming soon)

---

🛠️ Development

Adding a New Persona

Add a new persona to personas.py:

Persona(
    id="new_persona_id",
    name="Persona Name",
    role="Role",
    category="education" or "technology",
    dreyfus_level="novice" | "advanced_beginner" | "competent" | "proficient" | "expert",
    # ... other attributes
)

Adding a New Task

Add a new task class to tasks/ folder:

from tasks.base_task import BaseTask

class Task7New(BaseTask):
    title = "New Task"
    description = "Task description"
    difficulty = "Medium"
    
    def get_pre_test_questions(self):
        # Pre-test questions
        pass
    
    def get_post_test_questions(self):
        # Post-test questions
        pass

Running Tests

python test_system.py

---

📝 Dependencies

Main dependencies:

• streamlit==1.31.0: Web interface
• openai==1.54.0: GPT-4 API
• sqlalchemy>=2.0.44: Database ORM
• scikit-learn==1.4.0: Machine learning models
• numpy==1.26.3: Mathematical calculations
• pandas==2.2.0: Data analysis
• plotly==5.18.0: Visualization

See requirements.txt for the complete list.

---

🔐 Security and Privacy

• All participant data is stored anonymously
• Identity privacy is ensured with UUID
• API keys are stored in .env file (should not be committed to git)
• Database is unencrypted SQLite (PostgreSQL recommended for production)

---

📚 References

Theoretical Foundations

• Dreyfus, S. E., & Dreyfus, H. L. (1980). A five-stage model of the mental activities involved in directed skill acquisition.
• Sweller, J. (1988). Cognitive load during problem solving: Effects on learning.
• Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes.

Code Quality Metrics

• McCabe, T. J. (1976). A complexity measure.
• Halstead, M. H. (1977). Elements of software science.
• ISO/IEC 25010: Software Quality Model

---

🤝 Contributing

This is a doctoral research project. Please contact for contributions.

---

📄 License

This project is for academic research purposes. Permission required for commercial use.

---

🙏 Acknowledgments

• OpenAI GPT-4 API
• Streamlit team
• All participants