How I made a reliable hacking tools and resources search engine in two days (~6500 entries!)

3 min readNov 3, 2022

https://lobuhi.github.io/

In fairness, this is just a fork from Ippsec.Rocks, a great resource which I visited oftenly when I was enrolled for OSCP. It was during the first visit to the IppSec site that I thought this would be a perfect solution to use as a repository for the countless github-hosted resources that we all use on a daily basis.

As time went by, my bookmarks bar started to fill up with dozens of folders and subfolders with different categories, so did my Notion knowledge base and some problems arised.

Notion is a great tool for taking notes and establishing a solid knowledge database structure. One of its main features is the possibility to add code blocks, links or bookmarks, but the integrated search engine fails to find bookmarks easily. Similarly, the browser’s bookmarks bar does not allow a very detailed search beyond the repository name or tool. In addition, the fact that many of these tools are multi-purpose added a degree of difficulty in placing them in a particular category.

With all this, I recovered the idea of using the Ippsec.Rocks solution and started to crawl all the github links that can be found in:

My personal bookmarks
Intigrity Newsletter
TL:DR Sec Newsletter
Kitploit
Hacktricks and Hacktricks-Cloud by Carlos Polop
Pentest Book by six2dez

Once I get and filtered all the github links in a file, I created the bash script below just to check every github page and returns the json structure (kind of) that will fill the dataset.json file:

#!/bin/bash
while read line; do 
    tool=$(echo $line | cut -d "/" -f 5)
    link=$(echo $line|sed 's/\//\\\//g' )
    curl $line | grep  '<p class="f4 my-3">' -A 1 | tail -1 | sed 's/"/\\"/g'| sed "s/^/{\"tool\":\"$tool\",\"link\":\"$link\",\"description\":\"/g" | sed 's/      //g' | sed 's/$/","tag":""},/g' | sed 's/<g-emoji.*<\/g-emoji>//g'
done < $1

The final result is a json with ~6500 entries of tools and resources that can be easily checked at https://lobuhi.github.io/?#

Here you got some examples: