* Fix issue #3364: web search returns error on non-2xx response and filters search engine self-links
- Add HTTP status code check to execute_web_search: return error if response is not 2xx
instead of proceeding with potentially junk results (e.g., DuckDuckGo challenge pages)
- Filter out links pointing back to the search engine host in extract_search_hits_from_generic_links
to avoid returning search engine self-links when no proper search results are found
- Add regression tests for both scenarios
Refs #3364
* fix: implement registrable domain filtering and error on 202/blocked responses
- Replace exact host matching with registrable domain comparison to properly
filter search engine self-links across subdomains (e.g., html.duckduckgo.com
vs duckduckgo.com)
- Add error handling for 2xx responses with no search results (e.g., 202
challenge pages from blocked search providers)
- Add regression tests for both 202 challenge page and mixed self-link scenarios
- Remove unrelated Cargo.lock version bumps (revert to 0.1.6)
Fixes#3364
* fix(tools): reject DuckDuckGo 202 challenge pages and drop search-engine links
The challenge page arrives as HTTP 202, which is_success() accepts, and its
links point at duckduckgo.com while the request goes to html.duckduckgo.com,
so the previous filter kept them. Only a 200 page is parsed now, and the
generic fallback drops any duckduckgo.com link or link to the configured
search host. Removes the registrable-domain heuristic and the 'no hits is an
error' path (a real query can legitimately return nothing), and replaces
tests that never exercised a self-link with ones that fail on the old code.
Refs #3364
---------
Co-authored-by: clawdbot <clawdbot@ultraworkers.dev>
The test explicit_model_provider_hint_controls_runtime_provider_dispatch
was failing when OPENAI_BASE_URL was set in the environment to a non-standard
URL (e.g., api.layofflabs.com). The test expects the OpenAI provider to use
the default base URL (api.openai.com) when no explicit base_url is provided
in the ProviderConfig.
This fix adds an EnvVarGuard struct to the tests module and uses it to
ensure OPENAI_BASE_URL is unset before running the test, preventing
environment pollution from affecting the test result.
Fixes: https://github.com/ultraworkers/claw-code/issues/3345
Co-authored-by: clawdbot <clawdbot@ultraworkers.dev>
- Updated SECURITY.md with explicit instructions for reporting vulnerabilities via GitHub Security Advisories
- Replaced 'when available' qualifier since Private Vulnerability Reporting is now enabled
- Provided step-by-step guidance for security researchers to submit vulnerability reports
Co-authored-by: clawdbot <clawdbot@ultraworkers.dev>
JSON config output already carries collected config diagnostics in warnings[], so prose stderr emission must be reserved for text/local paths. Lazy permission-mode default resolution prevents an earlier config load from leaking the same deprecation before the JSON renderer runs.\n\nConstraint: ROADMAP #815 requires text mode to keep human stderr warnings while JSON config/list suppresses duplicate app-level config prose.\nRejected: Filtering all stderr in JSON mode | would hide cargo/compiler or unrelated diagnostics outside the app config warning path.\nConfidence: high\nScope-risk: narrow\nDirective: Keep load_collecting_warnings side-effect-free; use load() for human stderr emission.\nTested: cargo fmt; cargo test -p rusty-claude-cli --test output_format_contract config_json_reports_deprecations_structurally_without_stderr_duplicate_815; cargo test -p rusty-claude-cli --test output_format_contract; manual target/debug/claw JSON config fixture.\nNot-tested: cargo clippy -p rusty-claude-cli --all-targets -- -D warnings is blocked by pre-existing runtime dead_code/trident warnings.
Doctor help was already on the local help path in current source, but the exact #702 dogfood surface lacked a focused guard and the JSON help envelope was still too prose-oriented for wrappers. Strengthen the JSON contract while preserving text help.\n\nConstraint: Preserve unrelated dirty rust/Cargo.lock from prior #701 work.\nRejected: Starting runtime/provider/session to inspect doctor semantics | help must be local and credential-free.\nConfidence: high\nScope-risk: narrow\nDirective: Keep doctor help routed through parse_local_help_action and print_help_topic; do not call run_doctor for --help.\nTested: cargo test --manifest-path rust/Cargo.toml -p rusty-claude-cli --test output_format_contract doctor_help -- --nocapture; cargo test --manifest-path rust/Cargo.toml -p rusty-claude-cli --test output_format_contract help -- --nocapture; cargo fmt --manifest-path rust/Cargo.toml --all -- --check; cargo check --manifest-path rust/Cargo.toml -p rusty-claude-cli; timeout 5s cargo run -q --bin claw -- --output-format json doctor --help; timeout 5s cargo run -q --bin claw -- doctor --help.\nNot-tested: full workspace test suite.
Keep malformed diff invocations with trailing JSON format flags on the parser error path and lock the contract with focused output-format regressions.
Constraint: Do not touch tracked .omx state files.
Rejected: Repeating direct binary smoke loops | local auth/provider configuration intercepts those invocations and obscures parser behavior.
Confidence: high
Scope-risk: narrow
Tested: git diff --check; cargo fmt --check; cargo test -p rusty-claude-cli diff_extra_args_have_typed_error_kind_and_hint_766 --test output_format_contract; cargo test -p rusty-claude-cli diff_trailing_json_after_malformed_args_is_bounded_json_3129 --test output_format_contract; cargo test -p rusty-claude-cli diff_non_git_dir_has_error_kind_and_hint_801 --test output_format_contract
Constraint: roadmap-next-id.sh must preserve single-id stdout on success while failing closed if duplicate validation cannot run.
Rejected: Relying only on CI/pre-push duplicate checks | the helper is used immediately before appending and must not certify an already-poisoned file.
Confidence: high
Scope-risk: narrow
Directive: Keep roadmap-next-id.sh stdout machine-clean; route validation failures and checker availability errors to stderr, and keep focused helper behavior coverage in the docs/ROADMAP CI path.
Tested: scripts/roadmap-next-id.sh ROADMAP.md printed 725 before appending #725 and 726 after; temp ROADMAP with duplicate 999 exited nonzero and listed duplicate id; scripts/roadmap-check-ids.sh ROADMAP.md; bash -n scripts/roadmap-next-id.sh scripts/roadmap-check-ids.sh; python -m unittest discover -s tests -p test_roadmap_helpers.py; python -m pytest tests/test_roadmap_helpers.py -q; SKIP_CLAW_PRE_PUSH_BUILD=1 bash .github/hooks/pre-push
Not-tested: full cargo workspace build, unchanged docs/script-only path
Add a lightweight ROADMAP duplicate-id guard and wire it into the low-risk docs/pre-push paths so optimistic append collisions introduced after the next-id helper are caught before merge.
Constraint: Current ROADMAP contains legacy numbered lists and pre-helper duplicate low ids, so the default guard checks helper-era ids >=723 while preserving --min-id 1 for a future strict audit.
Rejected: Fail CI on every numeric duplicate in the whole historical ROADMAP | current main would fail before this PR because old prose/list numbering is already duplicated.
Confidence: high
Scope-risk: narrow
Directive: Keep roadmap-next-id.sh paired with roadmap-check-ids.sh when changing ROADMAP append workflows.
Tested: bash -n scripts/roadmap-check-ids.sh scripts/roadmap-next-id.sh .github/hooks/pre-push; scripts/roadmap-check-ids.sh; temp ROADMAP copy with duplicate 723 failed nonzero and listed id 723; SKIP_CLAW_PRE_PUSH_BUILD=1 .github/hooks/pre-push; git diff --check; python3 .github/scripts/check_doc_source_of_truth.py; python3 .github/scripts/check_release_readiness.py
Not-tested: full cargo workspace build/test because this is docs/scripts-only and the local pre-push cargo build was smoke-tested with its documented skip path.
Constraint: Docs-only ROADMAP pinpoint requested; existing #324/#695 cover adjacent stale-binary and stale-worktree cases but not stale local cargo-run current-main misclassification.
Rejected: Implementing provenance warnings now | scope was to keep implementation out unless trivially obvious and safe.
Confidence: high
Scope-risk: narrow
Directive: Preserve #719 as the cargo-run/local-checkout sibling of #324/#695 when implementing provenance freshness.
Tested: python3 .github/scripts/check_doc_source_of_truth.py; python3 scripts/validate_cc2_board.py --board .omx/cc2/board.json; git diff --check
Not-tested: Runtime provenance warning implementation not changed.
Add a lightweight regression for the documented local pre-push build gate so the skip hatch and lockfile-grade cargo build command stay aligned with operator docs.
Constraint: Issue #696 scope is limited to pre-push hook contract drift after ROADMAP #694.\nRejected: Reworking the hook harness or roadmap board | unnecessary for the focused drift guard.\nConfidence: high\nScope-risk: narrow\nDirective: Keep the hook docs, skip hatch, and cargo --locked command in sync when changing local push gates.\nTested: bash -n .github/hooks/pre-push; python3 tests/test_pre_push_hook_contract.py -v; git diff --check\nNot-tested: Full cargo workspace build; Rust code was not touched.
Use git rev-parse --git-dir so startup preflight follows worktree .git indirections to the real metadata directory, then check directory permission metadata without creating probe files. Add a regression that verifies both the warning kind and structured event path for a read-only external gitdir.
Constraint: ROADMAP #695 requires early startup/worktree diagnostics without destructive writes or broad sandbox redesign.
Rejected: write-probe detection | it mutates git metadata during a diagnostic path.
Confidence: high
Scope-risk: narrow
Directive: Keep startup preflight warnings non-destructive and structured by warning kind/path.
Tested: cargo fmt --manifest-path rust/Cargo.toml --all -- --check; cargo test --manifest-path rust/Cargo.toml -p runtime startup_preflight -- --nocapture; cargo test --manifest-path rust/Cargo.toml -p runtime worker_boot -- --nocapture; cargo check --manifest-path rust/Cargo.toml --workspace
Not-tested: full cargo test --manifest-path rust/Cargo.toml --workspace
Adds test coverage asserting status:ok in skills list, agents list, skills install, skills help, and agents help JSON envelopes. Status fields themselves were already landed in 0581894b and cc1462a7; this PR adds the regression test assertions that were missing. Duplicate status keys in json! macros are benign (serde_json takes last value) but harmless given all tests pass.
Bellman