SCOPe Dataset

[tyler-hayes]

Thanks for your interesting work!

I was wondering if the version of the SCOPe dataset used in the paper is available for download? Or if the list of proteins and their superfamilies is available?

[valerianilucrezia]

Hi! Thank you for your interest in our work. I’m happy to share with you the version of the dataset we used, along with the script for filtering it to obtain the 10,256 sequences and 289 superfamilies.

The dataset can be found here:
https://scop.berkeley.edu/downloads/scopeseq-2.08/astral-scopedom-seqres-gd-sel-gs-bib-40-2.08.fa
and is obtained from the top link in this page: https://scop.berkeley.edu/astral/ver=2.08

The preprocessing script is the one below: it produces a dataframe containing the domain name, sequence, and the corresponding family, class, fold, and superfamily names.

Let me know if you have any doubt!

import pandas as pd
from Bio import SeqIO

# Input dataset file in FASTA format
dataset = 'astral-scopedom-seqres-gd-sel-gs-bib-40-2.08.fa.txt'

# Parse the FASTA file and extract records
records = list(SeqIO.parse(dataset, "fasta"))

# Extract sequence names, sequences, and IDs from the records
names = [seq.name for seq in records]
seqs = [str(s.seq) for s in records]
ids = [str(seq.description).split(' ')[1] for seq in records]

# Create a DataFrame with sequence names
df = pd.DataFrame(names, columns=['name'])
df['sequence'] = seqs  # Add sequences to the DataFrame
df['fam'] = ids  # Add family IDs to the DataFrame

# Split family IDs into hierarchical levels (class, fold, superfamily)
split = [idx.split('.') for idx in ids]
cl = [idx[0] for idx in split]  # Extract class level
df['cl'] = cl
fold = ['.'.join(idx[0:2]) for idx in split]  # Extract fold level
df['fold'] = fold
sp = ['.'.join(idx[0:3]) for idx in split]  # Extract superfamily level
df['sp'] = sp

# Define folds to exclude and filter the DataFrame
delete = ['c.2', 'c.3', 'c.4', 'c.5', 'c.27', 'c.28', 'c.30', 'c.31', 
          'b.66', 'b.67', 'b.68', 'b.69', 'b.70']
tokeep = list(set(df.fold) - set(delete))  # Keep folds not in the delete list
df = df.loc[df['fold'].isin(tokeep)]  # Filter the DataFrame

# Create DataFrames for unique folds, superfamilies, and families
df_fold = pd.DataFrame(set(fold), columns=['fold'])
df_sp = pd.DataFrame(set(sp), columns=['sp'])
df_fam = pd.DataFrame(set(ids), columns=['fam'])

# Count the number of sequences for each superfamily
for sp in df_sp.sp:
    df_sp.loc[df_sp['sp'] == sp, 'num'] = len(df.loc[df['sp'] == sp])
    sp_50 = df_sp.loc[df_sp['num'] >= 10]  # Filter superfamilies with at least 10 sequences
    sp_2 = sp_50.sp.values  # Get the filtered superfamily values
    seq_2 = df.loc[df['sp'].isin(sp_2)]  # Filter sequences for these superfamilies

# Count the number of families in each superfamily
for sp in sp_2:
    sp_50.loc[sp_50['sp'] == sp, 'num_fam'] = len(set(df.loc[df['sp'] == sp].fam))
sp_fam = sp_50.loc[sp_50['num_fam'] > 1].sp.values  # Filter superfamilies with more than 1 family
seq_fam = df.loc[df['sp'].isin(sp_fam)]  # Filter sequences for these superfamilies

# Print the filtered results
#print(seq_fam.name)  # Print domain names
#print(seq_fam.sequence)  # Print domain sequences
#print(seq_fam.sp)  # Print superfamily IDs

print(seq_fam)
	name	sequence	fam	cl	fold	sp
0	d1dlwa_	slfeqlggqaavqavtaqfyaniqadatvatffngidmpnqtnkta...	a.1.1.1	a	a.1	a.1.1
1	d2gkma_	gllsrlrkrepisiydkiggheaievvvedffvrvladdqlsaffs...	a.1.1.1	a	a.1	a.1.1
2	d1ngka_	ksfydavggaktfdaivsrfyaqvaedevlrrvypeddlagaeerl...	a.1.1.1	a	a.1	a.1.1
3	d2bkma_	eqwqtlyeaiggeetvaklveafyrrvaahpdlrpifpddltetah...	a.1.1.1	a	a.1	a.1.1
4	d4i0va_	aslyeklggaaavdlavekfygkvladervnrffvntdmakqkqhq...	a.1.1.1	a	a.1	a.1.1