New Option for ManSpider: Exclude a list of files from the parsing by content

[gokupwn]

Hello!,

First, thank you for this amazing tool.

A little bit of context:

I am using Manspider as part of a tool I am developing to discover and report all sensitive data exposed in the internal network of the company I am working for. We have too many shares to monitor continuously for sensitive data.

When I was testing Manspider (against a pre-prod environment), I found that Manspider enters a blocked state when parsing certain types of files, and sometimes these files don’t have a specific extension which means I can’t exclude them using the --exclude-extensions option.

As I want to use Manspider as part of my monitoring tool and I want my tool to run automatically (as a cronjob), I don’t want it to be blocked when parsing a file (enter a blocking state). I decided to contribute to this project by adding an option to exclude a list of files from the parsing process.

# The new option:
–exclude-files can now be used to exclude a list of files from the parsing process.
If Manspider enters a blocking state due to a file parsing issue, the user can exclude this specific file by specifying its name.

This option will work with both local and remote modes.

Example: Remote mode

python3 -m man_spider.manspider 192.168.56.22 -u 'testuser'  -p 'test123' -d 'north.sevenkingdoms.local' -t 10 -s 5M -m 2 --sharenames 'all' --loot-dir remote_loot -v -c "((secret|password|credentials|.*passe)\s{0,1}[:=]\S*)|(-----BEGIN[A-Z ]* PRIVATE KEY -----)" -q  --exclude-files KIX32.EXE nmap NTUSER.DAT{016888bd-6c6f-11de-8d1d-001e0bcde3ec}.TM.blf setup.rnm SharpHound.exe VERRACF.UPLOAD.XMI
.....

[*] 192.168.56.22: Downloading all\dev\setup.rnm
[*] Skipping 192.168.56.22\all\dev\NTUSER.DAT{016888bd-6c6f-11de-8d1d-001e0bcde3ec}.TM.blf: one of the filenames to skip
[*] 192.168.56.22: Downloading all\dev\SharpHound.exe
[*] Skipping 192.168.56.22\all\dev\setup.rnm: one of the filenames to skip
[*] 192.168.56.22: Downloading all\dev\sqmapi.dll
[*] Skipping 192.168.56.22\all\dev\SharpHound.exe: one of the filenames to skip
[*] 192.168.56.22: Downloading all\dev\test.txt
[*] Skipping 192.168.56.22\all\dev\sqmapi.dll: one of the filenames to skip
...

Example: Local mode

python3 -m man_spider.manspider testingmanspider --loot-dir local_loot -v -c "((secret|password|credentials|.*passe)\s{0,1}[:=]\S*)|(-----BEGIN[A-Z ]* PRIVATE KEY -----)" -q  --exclude-files KIX32.EXE nmap NTUSER.DAT{016888bd-6c6f-11de-8d1d-001e0bcde3ec}.TM.blf setup.rnm SharpHound.exe sqmapi.dll  VERRACF.UPLOAD.XMI
...
[*] ProcessPool: 4 processes finished
[*] Found file: testingmanspider/NTUSER.DAT{016888bd-6c6f-11de-8d1d-001e0bcde3ec}.TM.blf
[*] Skipping testingmanspider/NTUSER.DAT{016888bd-6c6f-11de-8d1d-001e0bcde3ec}.TM.blf: one of the filenames to skip
[*] ProcessPool: 6 processes started
[*] KIX32.EXE
[*] ProcessPool: 5 processes finished
[*] Found file: testingmanspider/report.md
[*] Parsing file: testingmanspider/report.md
[*] ProcessPool: 7 processes started
[*] nmap
[*] ProcessPool: 6 processes finished
[*] Found file: testingmanspider/KIX32.EXE
[*] Skipping testingmanspider/KIX32.EXE: one of the filenames to skip
[*] ProcessPool: 8 processes started
...