Conversation
Enhance htmlToText function to handle panics and errors safely.
panic: html: open stack of elements exceeds 512 nodes
goroutine 5523922 [running]:
github.com/projectdiscovery/httpx/common/pagetypeclassifier.htmlToText(...)
/home/runner/work/httpx/httpx/common/pagetypeclassifier/pagetypeclassifier.go:36
github.com/projectdiscovery/httpx/common/pagetypeclassifier.(*PageTypeClassifier).Classify(0xc0005164d8, {0xc0ba03a000?, 0xd?})
/home/runner/work/httpx/httpx/common/pagetypeclassifier/pagetypeclassifier.go:26 +0x6f
github.com/projectdiscovery/httpx/runner.(*Runner).analyze(_, _, {_, _}, {{0xc00470c450, 0xb}, {0x0, 0x0}, {0x0, 0x0}}, ...)
/home/runner/work/httpx/httpx/runner/runner.go:2349 +0x7555
github.com/projectdiscovery/httpx/runner.(*Runner).process.func1({{0xc00470c450, 0xb}, {0x0, 0x0}, {0x0, 0x0}}, {0x1686161?, 0x10?}, {0x16ace2d, 0xa})
/home/runner/work/httpx/httpx/runner/runner.go:1444 +0x125
created by github.com/projectdiscovery/httpx/runner.(*Runner).process in goroutine 1
/home/runner/work/httpx/httpx/runner/runner.go:1442 +0x8a6
Bumps [golang.org/x/text](https://github.com/golang/text) from 0.30.0 to 0.31.0. - [Release notes](https://github.com/golang/text/releases) - [Commits](golang/text@v0.30.0...v0.31.0) --- updated-dependencies: - dependency-name: golang.org/x/text dependency-version: 0.31.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.46.0 to 0.47.0. - [Commits](golang/net@v0.46.0...v0.47.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.47.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [github.com/PuerkitoBio/goquery](https://github.com/PuerkitoBio/goquery) from 1.10.3 to 1.11.0. - [Release notes](https://github.com/PuerkitoBio/goquery/releases) - [Commits](PuerkitoBio/goquery@v1.10.3...v1.11.0) --- updated-dependencies: - dependency-name: github.com/PuerkitoBio/goquery dependency-version: 1.11.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps the modules group with 5 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.9` | `1.2.10` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.59` | `1.1.60` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.27` | `0.1.28` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1-0.20251030144701-ce5c4b44e1e6` | `0.6.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.54` | `0.2.55` | Updates `github.com/projectdiscovery/cdncheck` from 1.2.9 to 1.2.10 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](projectdiscovery/cdncheck@v1.2.9...v1.2.10) Updates `github.com/projectdiscovery/gologger` from 1.1.59 to 1.1.60 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](projectdiscovery/gologger@v1.1.59...v1.1.60) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.27 to 0.1.28 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](projectdiscovery/networkpolicy@v0.1.27...v0.1.28) Updates `github.com/projectdiscovery/utils` from 0.6.1-0.20251030144701-ce5c4b44e1e6 to 0.6.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/commits/v0.6.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.54 to 0.2.55 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](projectdiscovery/wappalyzergo@v0.2.54...v0.2.55) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.10 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.60 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.28 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.6.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.55 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com>
…dev/golang.org/x/text-0.31.0 chore(deps): bump golang.org/x/text from 0.30.0 to 0.31.0
Improve error handling in htmlToText function
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.44.0 to 0.45.0. - [Commits](golang/crypto@v0.44.0...v0.45.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.45.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [github.com/weppos/publicsuffix-go](https://github.com/weppos/publicsuffix-go) from 0.50.0 to 0.50.1. - [Changelog](https://github.com/weppos/publicsuffix-go/blob/main/CHANGELOG.md) - [Commits](weppos/publicsuffix-go@v0.50.0...v0.50.1) --- updated-dependencies: - dependency-name: github.com/weppos/publicsuffix-go dependency-version: 0.50.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps the modules group with 11 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.10` | `1.2.11` | | [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.4` | `0.8.5` | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.15` | `0.4.17` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.60` | `1.1.61` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.95` | `0.0.96` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.28` | `0.1.29` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.131` | `1.0.132` | | [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx) | `1.2.1` | `1.2.2` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.102` | `0.0.103` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1` | `0.7.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.55` | `0.2.56` | Updates `github.com/projectdiscovery/cdncheck` from 1.2.10 to 1.2.11 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](projectdiscovery/cdncheck@v1.2.10...v1.2.11) Updates `github.com/projectdiscovery/dsl` from 0.8.4 to 0.8.5 - [Release notes](https://github.com/projectdiscovery/dsl/releases) - [Commits](projectdiscovery/dsl@v0.8.4...v0.8.5) Updates `github.com/projectdiscovery/fastdialer` from 0.4.15 to 0.4.17 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](projectdiscovery/fastdialer@v0.4.15...v0.4.17) Updates `github.com/projectdiscovery/gologger` from 1.1.60 to 1.1.61 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](projectdiscovery/gologger@v1.1.60...v1.1.61) Updates `github.com/projectdiscovery/hmap` from 0.0.95 to 0.0.96 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](projectdiscovery/hmap@v0.0.95...v0.0.96) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.28 to 0.1.29 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](projectdiscovery/networkpolicy@v0.1.28...v0.1.29) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.131 to 1.0.132 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](projectdiscovery/retryablehttp-go@v1.0.131...v1.0.132) Updates `github.com/projectdiscovery/tlsx` from 1.2.1 to 1.2.2 - [Release notes](https://github.com/projectdiscovery/tlsx/releases) - [Changelog](https://github.com/projectdiscovery/tlsx/blob/main/.goreleaser.yml) - [Commits](projectdiscovery/tlsx@v1.2.1...v1.2.2) Updates `github.com/projectdiscovery/useragent` from 0.0.102 to 0.0.103 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](projectdiscovery/useragent@v0.0.102...v0.0.103) Updates `github.com/projectdiscovery/utils` from 0.6.1 to 0.7.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](projectdiscovery/utils@v0.6.1...v0.7.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.55 to 0.2.56 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](projectdiscovery/wappalyzergo@v0.2.55...v0.2.56) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.11 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/dsl dependency-version: 0.8.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.17 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.61 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.96 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.29 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.132 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/tlsx dependency-version: 1.2.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.103 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.7.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.56 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com>
…flow Adding panic guard + Tests
Bumps [github.com/JohannesKaufmann/html-to-markdown/v2](https://github.com/JohannesKaufmann/html-to-markdown) from 2.4.0 to 2.5.0. - [Release notes](https://github.com/JohannesKaufmann/html-to-markdown/releases) - [Commits](JohannesKaufmann/html-to-markdown@v2.4.0...v2.5.0) --- updated-dependencies: - dependency-name: github.com/JohannesKaufmann/html-to-markdown/v2 dependency-version: 2.5.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps the modules group with 10 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.11` | `1.2.12` | | [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.5` | `0.8.6` | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.17` | `0.4.18` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.61` | `1.1.62` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.96` | `0.0.97` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.29` | `0.1.30` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.132` | `1.0.133` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.103` | `0.0.104` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.7.1` | `0.7.3` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.56` | `0.2.57` | Updates `github.com/projectdiscovery/cdncheck` from 1.2.11 to 1.2.12 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](projectdiscovery/cdncheck@v1.2.11...v1.2.12) Updates `github.com/projectdiscovery/dsl` from 0.8.5 to 0.8.6 - [Release notes](https://github.com/projectdiscovery/dsl/releases) - [Commits](projectdiscovery/dsl@v0.8.5...v0.8.6) Updates `github.com/projectdiscovery/fastdialer` from 0.4.17 to 0.4.18 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](projectdiscovery/fastdialer@v0.4.17...v0.4.18) Updates `github.com/projectdiscovery/gologger` from 1.1.61 to 1.1.62 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](projectdiscovery/gologger@v1.1.61...v1.1.62) Updates `github.com/projectdiscovery/hmap` from 0.0.96 to 0.0.97 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](projectdiscovery/hmap@v0.0.96...v0.0.97) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.29 to 0.1.30 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](projectdiscovery/networkpolicy@v0.1.29...v0.1.30) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.132 to 1.0.133 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](projectdiscovery/retryablehttp-go@v1.0.132...v1.0.133) Updates `github.com/projectdiscovery/useragent` from 0.0.103 to 0.0.104 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](projectdiscovery/useragent@v0.0.103...v0.0.104) Updates `github.com/projectdiscovery/utils` from 0.7.1 to 0.7.3 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](projectdiscovery/utils@v0.7.1...v0.7.3) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.56 to 0.2.57 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](projectdiscovery/wappalyzergo@v0.2.56...v0.2.57) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.12 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/dsl dependency-version: 0.8.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.18 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.62 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.97 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.30 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.133 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.104 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.7.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.57 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com>
feat: update -ldp option to show default ports in CLI output - Modified URL formatting in runner.go to respect LeaveDefaultPorts option - Fixed AddURLDefaultPort function to actually add default ports (80/443) - When -ldp is used, CLI output now shows https://example.com:443 instead of https://example.com - Maintains backward compatibility - default behavior unchanged Fixes CLI output inconsistency where -ldp flag only affected Host headers but not the displayed URL in console output.
fix: enhance HTML parser panic protection with multiple fallback strategies - Add ultra-aggressive HTML sanitization to reduce nesting depth - Implement size limiting (1MB) to prevent processing huge documents - Add plain text extraction fallback for complex HTML structures - Enhance panic recovery with comprehensive error handling - Remove deeply nestable elements (div, span, ul, ol, li) from sanitizer - Add comprehensive test coverage for edge cases Resolves HTML parser panic: 'html: open stack of elements exceeds 512 nodes' that occurred after switching to html-to-markdown/v2 library in PR #2255
- Changed 'host' field to return actual hostname (e.g., example.com) - Added new 'host_ip' field for the resolved/dialed IP address - Fixes semantic issue where 'host' was incorrectly returning IP
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the ✨ Finishing touches🧪 Generate unit tests (beta)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.