# Word Embedding and Clustering with GloVe and KMeans

## Overview

This repository contains a Python script for generating word embeddings using GloVe pre-trained word vectors and performing clustering on a text dataset using KMeans. The main goal is to organize words into clusters based on their semantic similarity.

## Directory Structure

- /content/drive/MyDrive/Colab Notebooks/: The main directory containing the script and necessary data files.
- /content/drive/MyDrive/Colab Notebooks/glove.6B.100d.txt: GloVe pre-trained word vectors file.
- /content/drive/MyDrive/Colab Notebooks/SimLex-999.txt: SimLex999 dataset for evaluating the clustering results.
- /content/drive/MyDrive/Colab Notebooks/brown.csv: Brown Corpus data for training word embeddings.
- /content/drive/MyDrive/Colab Notebooks/Token.csv: Dataset containing preprocessed text of Brown Corpus.

## Dependencies

Ensure the following Python libraries are installed before running the script:
- pandas
- scikit-learn
- gensim
- numpy
- nltk


Approach
1. Data Loading and Preprocessing:
- SimLex999 and Brown Corpus data are loaded into Pandas DataFrames.
- Text preprocessing is applied to tokenize and remove stop words from the Brown Corpus data.

2.Word Embeddings with GloVe:
- GloVe word vectors are loaded and used to create word embeddings for each tokenized sentence in the Brown Corpus.

3.Clustering with KMeans:
- Cosine similarity is calculated between word embeddings in chunks.
- KMeans clustering is performed on the cosine similarity matrix in chunks to handle memory constraints.

4.Evaluation with SimLex999:
- The script evaluates the clustering results using SimLex999 dataset, calculating average similarity within clusters.

5. Spearman Correlation:
- The script calculates Spearman correlation between the predicted similarities from clustering and the actual similarities from SimLex999.











# Word Embedding with GloVe and exploring Cosine Simmilarity in detail

I have thought of doing this implementation and could not due computational restritions. On 19-12-2023(i.e last day of submission) I got access to DGX. So I thought including this code as well. Please go through it as well.

  

This repository contains a Python script for generating word embeddings using GloVe pre-trained word vectors, mapping word similarities, and evaluating the results using the SimLex999 dataset.

## Directory Structure

- /dgxa_home/se20uari151/: The main directory containing the script and necessary data files.
- /dgxa_home/se20uari151/glove.6B.100d.txt : GloVe pre-trained word vectors file.
- /dgxa_home/se20uari151/SimLex-999.txt : SimLex999 dataset for evaluating word similarities.
- /dgxa_home/se20uari151/brown.csv: Brown Corpus data for training word embeddings.
- /dgxa_home/se20uari151/token.csv: Dataset containing preprocessed text of Brown Corpus.



Certainly! Here's a README template for your codebase:

markdown
Copy code
# Word Embedding and Similarity Mapping with GloVe

## Overview

This repository contains a Python script for generating word embeddings using GloVe pre-trained word vectors, mapping word similarities, and evaluating the results using the SimLex999 dataset.

## Directory Structure

- /dgxa_home/se20uari151/: The main directory containing the script and necessary data files.
  - /dgxa_home/se20uari151/glove.6B.100d.txt: GloVe pre-trained word vectors file.
  - /dgxa_home/se20uari151/SimLex-999.txt: SimLex999 dataset for evaluating word similarities.
  - /dgxa_home/se20uari151/brown.csv: Brown Corpus data for training word embeddings.
  - /dgxa_home/se20uari151/token.csv: Dataset containing preprocessed text.

## Dependencies

Ensure the following Python libraries are installed before running the script:

- pandas
- scikit-learn
- gensim
- numpy
- nltk


Approach
1.Data Loading and Preprocessing:
- SimLex999 and Brown Corpus data are loaded into Pandas DataFrames.
- Text preprocessing is applied to tokenize and remove stop words from the Brown Corpus data.

2.Word Embeddings with GloVe:
- GloVe word vectors are loaded and used to create word embeddings for each tokenized sentence in the Brown Corpus.

3.Mapping Similarity:
- Cosine similarity is calculated between word embeddings for a subset of the data.
- A linear transformation is applied to map similarity values from [-1, 1] to [0, 10].

4.Evaluation with SimLex999:
- The script evaluates the mapping results using the SimLex999 dataset.
- The Spearman correlation coefficient is calculated between actual and mapped similarity values.