From d198a861096b0bcba4d2a952b33606fa14080d1f Mon Sep 17 00:00:00 2001 From: rhartuv Date: Thu, 25 Jun 2026 12:11:03 +0300 Subject: [PATCH 1/2] docs: align Configuration File Reference documentation with actual config files --- README.md | 350 +++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 268 insertions(+), 82 deletions(-) diff --git a/README.md b/README.md index 4aff86137..560afa7a7 100644 --- a/README.md +++ b/README.md @@ -531,88 +531,274 @@ Commands: ``` ### Configuration file reference -The configuration defines how the workflow operates, including functions, LLMs, and embedders along with general configuration settings. More details about the NeMo Agent toolkit workflow configuration file can be found [here](https://docs.nvidia.com/aiqtoolkit/latest/workflows/workflow-configuration.html). - -1. **General configuration** (`general`): The `general` section contains general configuration settings for NeMo Agent toolkit which is not specific to any workflow. - - `use_uvloop`: Specifies whether to use `uvloop` event loop which can provide significant speedup. For debugging purposes it is recommended to set this to `false`. - - `telemetry.logging`: Sets log level for logging. - - `telemetry.tracing`: This is used in `config-tracing.yml` where `endpoint` is set to a Phoenix server. Traces of workflow can then be viewed in Phoenix UI. -2. **Functions** (`functions`): The `functions` section contains the tools used in the workflow. - - Preprocessing functions: - - `cve_generate_vdbs`: Generates vector database from code repositories and documentation. - - `agent_name`: Name of the agent executor (`cve_agent_executor`). Used to determine which tools are enabled in the agent to conditionally generate vector databases or indexes. - - `embedder_name`: Name of embedder (`nim-embedder`) configured in `embedders` section. - - `base_vdb_dir`: The directory used for storing vector database files. Default is `.cache/am_cache/vdb`. - - `base_git_dir`: The directory for storing pulled git repositories used for code analysis. Default is `.cache/am_cache/git`. - - `base_code_index_dir`: The directory used for storing code index files. Default is `./cache/am_cache/code_index`. - - `cve_fetch_intel`: Fetches details about CVEs from NIST and CVE Details websites. - - `cve_process_sbom`: Prepares and validates input SBOM. - - `cve_check_vuln_deps`: Cross-references every entry in the SBOM for known vulnerabilities. If none are found, the agent is skipped for that CVE. - - `skip`: If true, skips this check and runs all CVEs through the agent. Useful when the input CVEs to the workflow have been validated by a dedicated vulnerability scanner. Default is false. - - Core LLM engine functions: - - `cve_checklist`: Generates tailored, context-sensitive task checklist for impact analysis. - - `Container Image Code QA System`: Retriever tool used by `cve_agent_executor` to query source code vector database. - - `Container Image Developer Guide QA System`: Retriever tool used by `cve_agent_executor` to query documentation vector database. - - `Lexical Search Container Image Code QA System`: Lexical search tool used by `cve_agent_executor` to search source code. This tool is an alternative to `Container Image Code QA System` and can be useful for very large code bases that take a long time to embed as a vector database. Disabled by default, enable by uncommenting the tool in `cve_agent_executor`. - - `Internet Search`: SerpApi Google search tool used by `cve_agent_executor`. - - `cve_agent_executor`: Iterates through checklist items using provided tools and gathered intel. - - `llm_name`: Name of LLM (`cve_agent_executor_llm`) configured in `llms` section. - - `tool_names`: Container Image Code QA System, Container Image Developer Guide QA System, (Optional) Lexical Search Container Image Code QA System, Internet Search - - `max_concurrency`: Controls the maximum number of concurrent requests to the LLM. Default is `null`, which doesn't limit concurrency. - - `max_iterations`: The maximum number of iterations for the agent. Default is 10. - - `prompt_examples`: Whether to include examples in agent prompt. Default is `false`. - - `replace_exceptions`: Whether to replace exception message with custom message. Default is `true`. - - `replace_exceptions_value`: If `replace_exceptions` is `true`, use this message. Default is `I do not have a definitive answer for this checklist item."` - - `return_intermediate_steps`: Controls whether to return intermediate steps taken by the agent, and include them in the output file. Helpful for troubleshooting agent responses. Default is `false`. - - `verbose`: Set to true for verbose output. Default is `false`. - - `cve_generate_cvss`: generates and calculates the CVSS (Common Vulnerability Scoring System) vector string and score for the vulnerability analysis. - - `llm_name`: Name of LLM (`generate_cvss_llm`) configured in `llms` section. - - `tool_names`: Container Image Code QA System, Container Image Developer Guide QA System, (Optional) Lexical Search Container Image Code QA System, Container Image Analysis Data - - `max_concurrency`: Controls the maximum number of concurrent requests to the LLM. Default is `null`, which doesn't limit concurrency. - - `max_iterations`: The maximum number of iterations for the agent. Default is 10. - - `prompt_examples`: Whether to include examples in agent prompt. Default is `true`. - - `replace_exceptions`: Whether to replace exception message with custom message. Default is `false`. - - `replace_exceptions_value`: If `replace_exceptions` is `true`, use this message. Default is `Failed to generate CVSS for this analysis."` - - `return_intermediate_steps`: Controls whether to return intermediate steps taken by the agent, and include them in the output file. Helpful for troubleshooting agent responses. Default is `false`. - - `verbose`: Set to true for verbose output. Default is `false`. - - `cve_summarize`: Generates concise, human-readable summarization paragraph from agent results. - - `llm_name`: Name of LLM (`summarize_llm`) configured in `llms` section. - - `cve_justify`: Assigns justification label and reason to each CVE based on summary. - - `llm_name`: Name of LLM (`justify_llm`) configured in `llms` section. - - Postprocessing/Output functions - - `cve_file_output`: Outputs workflow results to a file. - - `file_path`: Defines the path to the file where the output will be saved. - - `markdown_dir`: Defines the path to the directory where the output will be saved in individual navigable markdown files per CVE-ID. - - `overwrite`: Indicates whether the output file should be overwritten when the workflow starts if it already exists. Will throw an error if set to `False` and the file already exists. Note that the overwrite behavior only occurs on workflow initialization. For pipelines started in HTTP mode, each new request will append the existing file until the workflow is restarted. -3. **LLMs** (`llms`): The `llms` section contains the LLMs used by the workflow. Functions can reference LLMs in this section to use. The supported LLM API types in NeMo Agent toolkit are `nim` and `openai` The models in this workflow use `nim`. - - Configured models in this workflow: `checklist_llm`, `code_vdb_retriever_llm`, `doc_vdb_retriever_llm`, `cve_agent_executor_llm`, `generate_cvss_llm`, `summarize_llm`, `justify_llm` - - Each `nim` model is configured with the following attributes defined in the NeMo Agent toolkit's [NimModelConfig](https://docs.nvidia.com/aiqtoolkit/latest/api/aiq/llm/nim_llm/index.html#aiq.llm.nim_llm.NIMModelConfig). Use [OpenAIModelConfig](https://docs.nvidia.com/aiqtoolkit/latest/api/aiq/llm/openai_llm/index.html#aiq.llm.openai_llm.OpenAIModelConfig) for `openai` models. - - `base_url`: Optional attribute to override `https://integrate.api.nvidia.com/v1` - - `model_name`: The name of the LLM model used by the node. - - `temperature`: Controls randomness in the output. A lower temperature produces more deterministic results. - - `max_tokens`: Defines the maximum number of tokens that can be generated in one output step. - - `top_p`: Limits the diversity of token sampling based on cumulative probability. -4. **Embedding models** (`embedders`): The `embedders` section contains the embedding models used by the workflow. Functions can reference embedding models in this section to use. The supported embedding model API types in NeMo Agent toolkit are `nim` and `openai`. - - The models uses `nim` model, `nvidia/nv-embedqa-e5-v5`. - - Each `nim` embedding model is configured with the following attributes defined in the NeMo Agent Toolkit\'s [NimEmbedderModelConfig](https://docs.nvidia.com/aiqtoolkit/latest/api/aiq/embedder/nim_embedder/index.html). Use [OpenAIEmbedderModelConfig](https://docs.nvidia.com/aiqtoolkit/#aiq.llm.openai_llm.OpenAIModelConfig) for `openai` embedding models. - - `base_url`: Optional attribute to override `https://integrate.api.nvidia.com/v1` - - `model_name`: The name of the LLM model used by the node. - - `truncate`: Specifies how inputs longer than the maximum token length of the model are handled. Passing `START` discards the start of the input. `END` discards the end of the input. In both cases, input is discarded until the remaining input is exactly the maximum input token length for the model. If `NONE` is selected, when the input exceeds the maximum input token length an error will be returned. - - `max_batch_size`: Specifies the batch size to use when generating embeddings. We recommend setting this to 128 (default) or lower when using the cloud-hosted embedding NIM. When using a local NIM, this value can be tuned based on throughput/memory performance on your hardware. -5. **Workflow** (`workflow`): The `workflow` section ties the previous sections together by defining the tools and LLM models to use in the workflow. - - `_type`: This is set to `cve_agent` indicating to NeMo Agent toolkit to use the function defined in [register.py](src/vuln_analysis/register.py) for the workflow. - - The remaining configuration items correspond to attributes in [CVEWorkflowConfig](src/vuln_analysis/register.py) to specify the registered tools to use in the workflow. -6. **Evaluations and Profiling** (`eval`): The `eval` section contains the evaluation settings for the workflow. Refer to [Evaluating NVIDIA Agent Intelligence Toolkit Workflows](https://docs.nvidia.com/aiqtoolkit/latest/workflows/evaluate.html) for more information about NeMo Agent toolkit built-in evaluators as well as the plugin system to add custom evaluators. The CVE workflow uses the `eval` section to configure a profiler that uses the NeMo Agent toolkit evaluation system to collect usage statistics and store them to the local file system. You can find more information about NeMo Agent toolkit profiling and performance monitoring [here](https://docs.nvidia.com/aiqtoolkit/latest/workflows/profiler.html). - - `general.output_dir`: Defines the path to the directory where profiling results will be saved. - - `general.dataset`: Defines file path and format of dataset used to run profiling. - - `profiler`: The profiler for this workflow is configured with the following options. - - `token_uniqueness_forecast`: Compute inter query token uniqueness - - `workflow_runtime_forecast`: Compute expected workflow runtime - - `compute_llm_metrics`: Compute inference optimization metrics - - `csv_exclude_io_text`: Avoid dumping large text into the output CSV (helpful to not break structure) - - `prompt_caching_prefixes`: Identify common prompt prefixes - - `bottleneck_analysis`: Enable bottleneck analysis - - `concurrency_spike_analysis`: Enable concurrency spike analysis. Set the `spike_threshold` to 7, meaning that any concurrency spike above 7 will be raised to the user specifically. + +The configuration defines how the workflow operates, including pipeline nodes, agent tools, LLMs, embedders, and general settings. More details about the NeMo Agent toolkit workflow configuration file can be found [here](https://docs.nvidia.com/aiqtoolkit/latest/workflows/workflow-configuration.html). + +Reference configuration files: + +| Environment | File | +| ----------- | ---- | +| Local development / Docker Compose | [`src/vuln_analysis/configs/config-http-openai.yml`](src/vuln_analysis/configs/config-http-openai.yml) | +| Cloud-native (Kubernetes / Kustomize) | [`kustomize/base/exploit-iq-config.yml`](kustomize/base/exploit-iq-config.yml) | + +Field meanings for each pipeline node are defined in the corresponding Pydantic config class under [`src/vuln_analysis/functions/`](src/vuln_analysis/functions/) and [`src/vuln_analysis/tools/`](src/vuln_analysis/tools/). The workflow graph is wired in [`src/vuln_analysis/register.py`](src/vuln_analysis/register.py) via `CVEAgentWorkflowConfig`. + +#### Local vs. cloud-native differences + +Where the two reference configs diverge: + +| Area | Local (`config-http-openai.yml`) | Cloud-native (`exploit-iq-config.yml`) | +| ---- | -------------------------------- | -------------------------------------- | +| Cache / data paths | `.cache/am_cache/*` | `${EXPLOIT_IQ_DATA_DIR:-/exploit-iq-data/}*` subdirectories (`git`, `pickle`, `rpms`, `vdb`, `code_index`, `checker`) | +| LLM `_type` | Hard-coded `openai` for all models | `${LLM_TYPE_*:-nim}` per model, with per-model API key and base URL env vars | +| `cve_verify_vuln_package` | Present in `functions` and `workflow` | Not configured (node omitted) | +| `cve_fetch_patches` | Present in `functions` and `workflow` | Not configured | +| `cve_fetch_intel` | Default `retry_on_client_errors` (`true`); HTTP plugin endpoint `http://localhost:8080/...` without K8s auth | `retry_on_client_errors: false`; endpoint uses `CALLBACK_URL_PLACEHOLDER` with `token_path` and `verify_path` for service-account auth | +| `cve_agent_executor` tools | Includes `Configuration Scanner` and `Import Usage Analyzer`; sets `uber_jar_file_threshold: 600` | Omits those tools and `uber_jar_file_threshold` | +| `cve_http_output` | `url: http://localhost:8080`; no top-level auth fields | `url: CALLBACK_URL_PLACEHOLDER`; `auth_type: bearer` with `token_path` / `verify_path`; `mlops_config.enable_verify: true` | +| `cve_source_acquisition` | No `base_checker_dir` | Includes `base_checker_dir` under the data directory | + +#### 1. General configuration (`general`) + +Settings for the NeMo Agent toolkit runtime, not specific to any single pipeline node. + +- `front_end._type`: Front-end type (`fastapi`). +- `front_end.endpoints`: HTTP endpoints exposed by the service. Each entry has `path`, `method`, `description`, and `function_name` (for example, `/health` → `health_check`). +- `use_uvloop`: Use the `uvloop` event loop for performance. Set to `false` when debugging. +- `telemetry.tracing.phoenix._type`: Tracing backend type (`phoenix`). +- `telemetry.tracing.phoenix.endpoint`: OTLP traces endpoint (default `${OTEL_TRACES_ENDPOINT:-http://localhost:6006/v1/traces}`). +- `telemetry.tracing.phoenix.project`: Phoenix project name (default `cve_agent`). + +#### 2. Functions (`functions`) + +Each key is a registered function name; `_type` selects the implementation. Functions are grouped below by pipeline role. + +##### Preprocessing and early pipeline nodes + +- **`cve_clone_and_deps`** (`_type: cve_clone_and_deps`): Clones source repositories and installs dependencies early in the pipeline, before vulnerability verification. VDB/indexing is deferred to `cve_segmentation`. + - `base_git_dir`: Directory for cloned git repositories. + - `base_pickle_dir`: Directory for pickle document cache files. + - `base_rpm_dir`: Directory for downloaded RPM files. + - `ignore_errors` *(optional, default `false`)*: Continue on clone/install errors. + +- **`cve_fetch_intel`** (`_type: cve_fetch_intel`): Fetches CVE intelligence from NVD, CVE Details, and configured intel plugins. + - `rpm_user_type`: Brew/OSIDB profile — `internal` (Red Hat VPN, enables OSIDB intel) or `external`. Default `external`; both reference configs use `${RPM_USER_TYPE:-internal}`. + - `retry_on_client_errors` *(default `true`)*: Retry when the HTTP client cannot connect. Cloud config sets this to `false`. + - `intel_plugin_config`: Plugin wrapper with `plugin_name` and `plugin_config` dict. Both configs use `SimpleHttpIntelPlugin`: + - `plugin_config.source`: Human-readable label for the intel source. + - `plugin_config.endpoint`: URL template with `{vuln_id}` placeholder. + - `plugin_config.api_key` *(optional)*: Bearer token sent as `Authorization` header. + - `plugin_config.token_path` *(optional)*: Path to a file containing a bearer token (used in cloud config for Kubernetes service-account auth). + - `plugin_config.verify_path` *(optional)*: TLS certificate path for HTTPS verification. + +- **`cve_calculate_intel_score`** (`_type: cve_calculate_intel_score`): Scores intel quality and can gate downstream analysis. + - `llm_name`: LLM used for scoring (`intel_source_score_llm`). + - `generate_intel_score` *(default `true`)*: Whether to compute intel scores. + - `intel_low_score` *(default `51`)*: Score threshold below which analysis may be skipped. + - `insist_analysis` *(default `false`)*: Continue analysis even when the score is below `intel_low_score`. + +- **`cve_process_sbom`** (`_type: cve_process_sbom`): Prepares and validates SBOM input (file, HTTP URL, or manual package list). + - `max_retries` *(default `10`)*: Retries when loading an SBOM from a URL. + +- **`cve_verify_vuln_package`** (`_type: cve_verify_vuln_package`): For SOURCE analysis, parses lock files and verifies that the CVE's vulnerable package is present and that the installed version is actually vulnerable. **Present in the local config only.** + - `skip` *(default `false`)*: Skip verification and proceed with all CVEs. + - `base_git_dir`: Base directory for cloned git repositories. + - `llm_name` *(default `checklist_llm`)*: LLM for version comparison and package matching. + +- **`cve_segmentation`** (`_type: cve_segmentation`): Builds vector databases and Tantivy code indexes from already-cloned repositories. Runs after package verification. + - `agent_name`: Agent executor name (`cve_agent_executor`). Used to decide which indexes/VDBs to build based on enabled tools. + - `embedder_name`: Embedder for VDB creation (`nim_embedder`). + - `base_git_dir`, `base_vdb_dir`, `base_code_index_dir`, `base_pickle_dir`: Cache directories for repos, VDBs, code indexes, and pickles. + - `ignore_code_embedding` *(default `false`)*: Skip building the code vector database. Both reference configs set `true`. + - `ignore_errors` *(optional, default `false`)*: Continue on segmentation errors. + - `ignore_code_index` *(optional, default `false`)*: Skip building the Tantivy code index. + +- **`cve_source_acquisition`** (`_type: cve_source_acquisition`): Downloads source containers, extracts layers, and locates package sources for the RPM CVE Checker pipeline. + - `base_git_dir`, `base_pickle_dir`, `base_rpm_dir`: Shared cache directories. + - `base_checker_dir` *(optional)*: Root for checker-specific artifacts. Set in cloud config; defaults to `.cache/am_cache/checker`. + - `rpm_user_type`: Brew profile (`internal` or `external`). + +- **`cve_checker_segmentation`** (`_type: cve_checker_segmentation`): Builds a scoped Tantivy lexical index from extracted RPM source files. + - `base_checker_dir`: Root for checker artifacts. + - `base_code_index_dir`: Tantivy index storage directory. + - `include_extensions` *(optional)*: File extensions to index. + +- **`cve_fetch_patches`** (`_type: cve_fetch_patches`): Fetches vulnerability fix patches from intel references and OSV for the details UI. **Present in the local config only.** + - `llm_name` *(optional)*: LLM for Chromium CL disambiguation. + - `max_concurrency` *(default `5`)*: Concurrent patch fetch operations. + +##### Core analysis nodes + +- **`cve_checklist`** (`_type: cve_checklist`): Generates a tailored, context-sensitive checklist for impact analysis. + - `llm_name`: LLM for checklist generation (`checklist_llm`). + - `agent_name` *(default `cve_agent_executor`)*: Agent whose `tool_names` drive which checklist questions are generated. + - `prompt` *(optional)*: Custom prompt text or path to a prompt file. + +- **`cve_agent_executor`** (`_type: cve_agent_executor`): ReAct agent that iterates through checklist items using the configured tools and gathered intel. + - `llm_name`: LLM for the agent (`cve_agent_executor_llm`). + - `tool_names`: List of tool function names (see [Agent tools](#agent-tools) below). + - `max_concurrency` *(default `null`)*: Maximum concurrent agent invocations; `null` means unlimited. + - `max_iterations` *(default `10`)*: Maximum ReAct iterations per checklist item. + - `prompt` *(optional)*: Custom agent prompt text or file path. + - `prompt_examples` *(default `false`)*: Include few-shot examples in the prompt. + - `replace_exceptions` *(default `false` in model; `true` in both reference configs)*: Replace tool exceptions with a custom message. + - `replace_exceptions_value`: Message used when `replace_exceptions` is `true`. + - `return_intermediate_steps` *(default `false`)*: Include intermediate agent steps in output. + - `transitive_search_tool_enabled` *(default `true`)*: Enable the Call Chain Analyzer tool. + - `cve_web_search_enabled` *(default `true`)*: Enable the CVE Web Search tool. + - `uber_jar_file_threshold` *(default `600`)*: Source file count above which a JAR is treated as an uber-JAR, disabling same-artifact optimizations. Set in local config only. + - `verbose` *(default `false`)*: Verbose agent logging. + - `context_window_token_limit` *(default `5000`)*: Token threshold for pruning old messages. + +- **`cve_package_code_agent`** (`_type: cve_package_code_agent`): Level 1 Package Code Agent for the RPM CVE Checker — investigates CVEs using extracted source and the scoped code index. + - `llm_name`, `tool_names`, `max_iterations` *(default `10`)*: Agent LLM, tools, and iteration limit. + - `base_checker_dir`, `base_code_index_dir`, `base_rpm_dir`: Artifact and cache directories. + - `rpm_user_type`: Brew profile for reference-patch SRPM lookup. + - `context_window_token_limit` *(default `5000`)*: Context pruning threshold. + - Additional optional fields (reference mining, git search, repo resolution) are defined in [`cve_package_code_agent.py`](src/vuln_analysis/functions/cve_package_code_agent.py). + +- **`cve_build_agent`** (`_type: cve_build_agent`): Level 2 Build Agent — checks whether vulnerable code is compiled into binaries and whether hardening flags mitigate the CVE. + - `llm_name`, `tool_names`, `max_iterations` *(default `5` in model; `10` in both reference configs)*. + - `base_checker_dir`: Checker artifact root. + - `context_window_token_limit` *(default `5000`)*. + +- **`cve_checker_report`** (`_type: cve_checker_report`): Synthesizes the final L1/L2 checker report. + - `llm_name`: LLM for report generation. + - `base_checker_dir`: Checker artifact root. + +- **`cve_generate_cvss`** (`_type: cve_generate_cvss`): Agent that generates CVSS vector strings and scores. + - `skip` *(default `false`; both reference configs set `true`)*: Skip CVSS generation entirely. + - `llm_name`, `tool_names`, `max_concurrency`, `max_iterations`, `prompt`, `prompt_examples` *(default `true`)*, `replace_exceptions` *(default `false`)*, `replace_exceptions_value`, `return_intermediate_steps`, `verbose`: Same semantics as `cve_agent_executor`. + +- **`cve_summarize`** (`_type: cve_summarize`): Produces a concise summary paragraph from agent results. + - `llm_name`: LLM for summarization (`summarize_llm`). + +- **`cve_justify`** (`_type: cve_justify`): Assigns a VEX justification label and reason to each CVE. + - `llm_name`: LLM for justification (`justify_llm`). + +- **`cve_generate_vex`** (`_type: cve_generate_vex`): Generates a VEX document for vulnerable components. + - `skip` *(default `false`)*: Skip VEX generation. + - `vex_format` *(default `csaf`)*: Output format (cloud config shows a commented `# vex_format: csaf` example). + +##### Output and health + +- **`cve_http_output`** (`_type: cve_http_output`): Posts workflow results to an HTTP endpoint. + - `url`: Base URL for the callback service. + - `endpoint` *(default `/api/v1/reports`)*: Path appended to `url` for successful reports. + - `auth_type` *(default `disabled`)*: `bearer`, `basic`, or `disabled`. Cloud config uses `bearer`. + - `token` *(optional)*: Inline auth token. + - `token_path` *(optional)*: File path to a bearer token (Kubernetes service account in cloud config). + - `verify_path` *(optional)*: TLS certificate for HTTPS verification. + - `username`, `password` *(optional)*: Basic-auth credentials. + - `failure_endpoint` *(default `/api/v1/reports/failed`)*: Endpoint for failure reports when code indexing fails. + - `enable_mlops` *(default `false`)*: Forward output to an MLOps endpoint. + - `mlops_config`: MLOps sub-configuration: + - `mlops_url`: MLOps service URL. + - `auth_type` *(default `None`)*: `bearer`, `basic`, or `None`. + - `token_path`, `verify_path`: Auth token and TLS cert paths. + - `enable_verify` *(default `false`)*: Enable TLS verification for MLOps. + +- **`health_check`** (`_type: health_check`): Health-check function wired to `GET /health`. + +##### Agent tools + +Tool entries in `functions` are referenced by name from agent `tool_names` lists. Display names in the reference configs map to `_type` values as follows: + +| Config key | `_type` | Key fields | +| ---------- | ------- | ---------- | +| `Code Semantic Search` | `local_vdb_retriever` | `embedder_name`, `llm_name`, `vdb_type` (`code` or `doc`), `return_source_documents` *(default `false`)* | +| `Docs Semantic Search` | `local_vdb_retriever` | Same as above with `vdb_type: doc` | +| `Code Keyword Search` | `lexical_code_search` | `top_k` *(default `5`)*; `base_code_index_dir` *(optional)* | +| `CVE Web Search` | `serp_wrapper` | `max_retries` | +| `Call Chain Analyzer` | `transitive_code_search` | No additional fields in the config model | +| `Function Caller Finder` | `calling_function_name_extractor` | No additional fields in the config model | +| `Function Locator` | `package_and_function_locator` | No additional fields in the config model | +| `Function Library Version Finder` | `calling_function_library_version_finder` | No additional fields in the config model | +| `Configuration Scanner` | `configuration_scanner` | `max_results` *(default `15`)*, `context_lines` *(default `5`)*. Local config only. | +| `Import Usage Analyzer` | `import_usage_analyzer` | `max_files` *(default `20`)*. Local config only. | +| `Container Analysis Data` | `container_image_analysis_data` | No additional fields; returns pre-analyzed container data from earlier workflow stages | +| `Source Grep` | `source_grep` | `base_checker_dir`, `max_results` *(default `50`)*, `context_lines` *(default `3`)* | + +> **Note:** `enable_transitive_search` and `enable_functions_usage_search` appear in the YAML configs under tool entries but are not defined in the tool config models; tool availability is controlled via `cve_agent_executor.tool_names` and the `transitive_search_tool_enabled` / `cve_web_search_enabled` flags on the agent. + +##### Alternate / optional functions (not in reference configs) + +These functions are supported by the codebase but not wired in the two reference configs above: + +- **`cve_generate_vdbs`**: Legacy combined clone + VDB/index node (superseded by `cve_clone_and_deps` + `cve_segmentation`). Fields: `agent_name`, `embedder_name`, `base_git_dir`, `base_vdb_dir`, `base_code_index_dir`, `base_pickle_dir`, `base_rpm_dir`, `ignore_errors`, `ignore_code_embedding`, `ignore_code_index`. +- **`cve_check_vuln_deps`**: Cross-references SBOM packages against known vulnerabilities for IMAGE analysis. Fields: `skip` *(default `false`)*, `retry_on_client_errors` *(default `true`)*. +- **`cve_file_output`**: Writes results to a local JSON file and per-CVE markdown reports. Fields: `file_path`, `markdown_dir`, `overwrite` *(default `false`)*. Use as `cve_output_config_name` instead of `cve_http_output` for file-based output. + +#### 3. LLMs (`llms`) + +Named LLM configurations referenced by `llm_name` in pipeline nodes. + +Configured models: `checklist_llm`, `code_vdb_retriever_llm`, `doc_vdb_retriever_llm`, `cve_agent_executor_llm`, `generate_cvss_llm`, `summarize_llm`, `justify_llm`, `intel_source_score_llm`. + +Each entry supports: + +- `_type`: LLM API type — `nim` or `openai`. Local config hard-codes `openai`; cloud config uses per-model `${LLM_TYPE_*:-nim}` env vars. +- `api_key`: API key (local config uses `"EMPTY"` with NVIDIA API; cloud config uses `${LLM_API_KEY_*}` env vars). +- `base_url`: API base URL. Local config uses `${NVIDIA_API_BASE:-...}` for all models; cloud config uses per-model base URL env vars (for example `CHECKLIST_LLM_API_BASE`, `AGENT_EXECUTOR_LLM_API_BASE`). +- `model_name`: Model identifier, overridable via env vars (for example `${CHECKLIST_MODEL_NAME:-meta/llama-3.1-70b-instruct}`). +- `temperature`, `max_tokens`, `top_p`: Standard generation parameters. + +See [NimModelConfig](https://docs.nvidia.com/aiqtoolkit/latest/api/aiq/llm/nim_llm/index.html) and [OpenAIModelConfig](https://docs.nvidia.com/aiqtoolkit/latest/api/aiq/llm/openai_llm/index.html) for the full parameter set. + +#### 4. Embedding models (`embedders`) + +- **`nim_embedder`** (`_type: nim`): + - `base_url`: Embedding API URL (default `${NIM_EMBED_BASE_URL:-https://integrate.api.nvidia.com/v1}`). + - `model_name`: Embedding model (default `${EMBEDDER_MODEL_NAME:-nvidia/nv-embedqa-e5-v5}`). + - `truncate`: How to handle over-length inputs — `START`, `END`, or `NONE` (default `END`). + - `max_batch_size`: Batch size for embedding requests (default `128`). + +See [NimEmbedderModelConfig](https://docs.nvidia.com/aiqtoolkit/latest/api/aiq/embedder/nim_embedder/index.html) for details. + +#### 5. Workflow (`workflow`) + +Ties pipeline nodes together. `_type` is `cve_agent`, registering the workflow in [`register.py`](src/vuln_analysis/register.py). + +Each `*_name` field references a function key from the `functions` section: + +| Field | Function | Required | +| ----- | -------- | -------- | +| `cve_clone_and_deps_name` | `cve_clone_and_deps` | Yes | +| `cve_segmentation_name` | `cve_segmentation` | Yes | +| `cve_fetch_intel_name` | `cve_fetch_intel` | Yes | +| `cve_calculate_intel_score_name` | `cve_calculate_intel_score` | Yes | +| `cve_process_sbom_name` | `cve_process_sbom` | Yes | +| `cve_verify_vuln_package_name` | `cve_verify_vuln_package` | Optional (local config only) | +| `cve_checklist_name` | `cve_checklist` | Yes | +| `cve_agent_executor_name` | `cve_agent_executor` | Yes | +| `cve_generate_cvss_name` | `cve_generate_cvss` | Yes | +| `cve_generate_vex_name` | `cve_generate_vex` | Yes | +| `cve_summarize_name` | `cve_summarize` | Yes | +| `cve_justify_name` | `cve_justify` | Yes | +| `cve_output_config_name` | `cve_http_output` (or `cve_file_output`) | Optional (`null` prints to console) | +| `cve_source_acquisition_name` | `cve_source_acquisition` | Optional (RPM CVE Checker) | +| `cve_checker_segmentation_name` | `cve_checker_segmentation` | Optional (RPM CVE Checker) | +| `cve_package_code_agent_name` | `cve_package_code_agent` | Optional (RPM CVE Checker) | +| `cve_checker_report_name` | `cve_checker_report` | Optional (RPM CVE Checker) | +| `cve_build_agent_name` | `cve_build_agent` | Optional (RPM CVE Checker) | +| `cve_fetch_patches_name` | `cve_fetch_patches` | Optional (local config only) | + +#### 6. Evaluations and profiling (`eval`) + +Configures the NeMo Agent toolkit profiler used by `aiq eval`. See [Evaluating Workflows](https://docs.nvidia.com/aiqtoolkit/latest/workflows/evaluate.html) and [Profiler](https://docs.nvidia.com/aiqtoolkit/latest/workflows/profiler.html). + +- `general.output_dir`: Directory for profiling output (default `./.tmp/eval/cve_agent`). +- `general.dataset._type`: Dataset format (`json`). +- `general.dataset.file_path`: Path to evaluation dataset (default `data/eval_datasets/eval_dataset.json`). +- `profiler.token_uniqueness_forecast`: Compute inter-query token uniqueness. +- `profiler.workflow_runtime_forecast`: Estimate expected workflow runtime. +- `profiler.compute_llm_metrics`: Compute inference optimization metrics. +- `profiler.csv_exclude_io_text`: Omit large I/O text from CSV output. +- `profiler.prompt_caching_prefixes.enable`: Identify common prompt prefixes. +- `profiler.prompt_caching_prefixes.min_frequency`: Minimum prefix frequency threshold (default `0.1`). +- `profiler.bottleneck_analysis.enable_nested_stack`: Enable nested-stack bottleneck analysis. +- `profiler.concurrency_spike_analysis.enable`: Enable concurrency spike detection. +- `profiler.concurrency_spike_analysis.spike_threshold`: Spike threshold (default `7`). ### NGINX caching server From d16061a6a8f5a14e8126398d7c18a25483881ab7 Mon Sep 17 00:00:00 2001 From: rhartuv Date: Sun, 28 Jun 2026 13:52:40 +0300 Subject: [PATCH 2/2] fix: configfile --- README.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 560afa7a7..9d8040584 100644 --- a/README.md +++ b/README.md @@ -545,18 +545,19 @@ Field meanings for each pipeline node are defined in the corresponding Pydantic #### Local vs. cloud-native differences -Where the two reference configs diverge: +The table below lists deliberate differences between the two reference files. Shared nodes (for example, `cve_checklist`, semantic search tools, and `cve_generate_cvss`) are not repeated here even when their parameter values differ. | Area | Local (`config-http-openai.yml`) | Cloud-native (`exploit-iq-config.yml`) | | ---- | -------------------------------- | -------------------------------------- | | Cache / data paths | `.cache/am_cache/*` | `${EXPLOIT_IQ_DATA_DIR:-/exploit-iq-data/}*` subdirectories (`git`, `pickle`, `rpms`, `vdb`, `code_index`, `checker`) | -| LLM `_type` | Hard-coded `openai` for all models | `${LLM_TYPE_*:-nim}` per model, with per-model API key and base URL env vars | -| `cve_verify_vuln_package` | Present in `functions` and `workflow` | Not configured (node omitted) | -| `cve_fetch_patches` | Present in `functions` and `workflow` | Not configured | -| `cve_fetch_intel` | Default `retry_on_client_errors` (`true`); HTTP plugin endpoint `http://localhost:8080/...` without K8s auth | `retry_on_client_errors: false`; endpoint uses `CALLBACK_URL_PLACEHOLDER` with `token_path` and `verify_path` for service-account auth | -| `cve_agent_executor` tools | Includes `Configuration Scanner` and `Import Usage Analyzer`; sets `uber_jar_file_threshold: 600` | Omits those tools and `uber_jar_file_threshold` | +| LLM `_type` | Hard-coded `openai` for all models | `${LLM_TYPE_*:-nim}` per model, with per-model `api_key` and `base_url` env vars (for example, `code_vdb_retriever_llm` at lines 210–217) | +| `cve_verify_vuln_package` | Defined in `functions` and wired in `workflow` between `cve_process_sbom` and `cve_checklist` | No `cve_verify_vuln_package` entry; after `cve_process_sbom`, the next node is `cve_checklist` (lines 59–63) | +| `cve_fetch_patches` | Defined in `functions` and wired in `workflow` | Not defined in `functions` or `workflow` | +| `cve_fetch_intel.intel_plugin_config` | `endpoint: http://localhost:8080/...`; no `token_path` / `verify_path` | `endpoint: CALLBACK_URL_PLACEHOLDER/...`; `token_path` and `verify_path` for Kubernetes service-account auth (lines 47–57) | +| `cve_fetch_intel.retry_on_client_errors` | Omitted (defaults to `true`) | Explicitly set to `false` | +| `cve_agent_executor.tool_names` | Includes `Configuration Scanner` and `Import Usage Analyzer` (10 tools total) | Same eight core tools only; no `Configuration Scanner` or `Import Usage Analyzer` in the agent's `tool_names` list (lines 99–110). Both configs still define semantic search tools and reference them from `cve_generate_cvss.tool_names` (lines 120–128) | | `cve_http_output` | `url: http://localhost:8080`; no top-level auth fields | `url: CALLBACK_URL_PLACEHOLDER`; `auth_type: bearer` with `token_path` / `verify_path`; `mlops_config.enable_verify: true` | -| `cve_source_acquisition` | No `base_checker_dir` | Includes `base_checker_dir` under the data directory | +| `cve_source_acquisition.base_checker_dir` | Not set (uses model default) | Set to `${EXPLOIT_IQ_DATA_DIR:-/exploit-iq-data/}checker` | #### 1. General configuration (`general`) @@ -646,7 +647,7 @@ Each key is a registered function name; `_type` selects the implementation. Func - `return_intermediate_steps` *(default `false`)*: Include intermediate agent steps in output. - `transitive_search_tool_enabled` *(default `true`)*: Enable the Call Chain Analyzer tool. - `cve_web_search_enabled` *(default `true`)*: Enable the CVE Web Search tool. - - `uber_jar_file_threshold` *(default `600`)*: Source file count above which a JAR is treated as an uber-JAR, disabling same-artifact optimizations. Set in local config only. + - `uber_jar_file_threshold` *(default `600`)*: Source file count above which a JAR is treated as an uber-JAR, disabling same-artifact optimizations. - `verbose` *(default `false`)*: Verbose agent logging. - `context_window_token_limit` *(default `5000`)*: Token threshold for pruning old messages. @@ -714,8 +715,8 @@ Tool entries in `functions` are referenced by name from agent `tool_names` lists | `Function Caller Finder` | `calling_function_name_extractor` | No additional fields in the config model | | `Function Locator` | `package_and_function_locator` | No additional fields in the config model | | `Function Library Version Finder` | `calling_function_library_version_finder` | No additional fields in the config model | -| `Configuration Scanner` | `configuration_scanner` | `max_results` *(default `15`)*, `context_lines` *(default `5`)*. Local config only. | -| `Import Usage Analyzer` | `import_usage_analyzer` | `max_files` *(default `20`)*. Local config only. | +| `Configuration Scanner` | `configuration_scanner` | `max_results` *(default `15`)*, `context_lines` *(default `5`)*. Defined in the local reference config only; wired into `cve_agent_executor.tool_names` locally but not present in `exploit-iq-config.yml`. | +| `Import Usage Analyzer` | `import_usage_analyzer` | `max_files` *(default `20`)*. Defined in the local reference config only; wired into `cve_agent_executor.tool_names` locally but not present in `exploit-iq-config.yml`. | | `Container Analysis Data` | `container_image_analysis_data` | No additional fields; returns pre-analyzed container data from earlier workflow stages | | `Source Grep` | `source_grep` | `base_checker_dir`, `max_results` *(default `50`)*, `context_lines` *(default `3`)* |