[tanishqbaweja]

GenerateAnswerNode has a fallback for model replies that aren't valid JSON. It's meant to put {"error": "Invalid JSON response format", "raw_response": ...} in the state instead of raising, with "in chunk processing" and "during merge" variants for the multi-chunk path. That fallback never runs for a reply that isn't JSON: graph.run() raises OutputParserException instead. Models do reply like this: there are several closed OutputParserException: Invalid json output reports, mostly with small models such as Llama 3 8B and Gemma 7B.

Reproduction on pre/beta, no API key needed:

from langchain_core.language_models import FakeListChatModel

from scrapegraphai.graphs import SmartScraperGraph

llm = FakeListChatModel(responses=["Sorry, I cannot answer that in JSON."])
graph = SmartScraperGraph(
    prompt="What is the product name?",
    source="<html><body><h1>Product name: Widget</h1></body></html>",
    config={"llm": {"model_instance": llm, "model_tokens": 8192}},
)
print(graph.run())

On pre/beta it raises:

langchain_core.exceptions.OutputParserException: Invalid json output: Sorry, I cannot answer that in JSON.

With this PR it returns:

{'error': 'Invalid JSON response format', 'raw_response': 'Invalid json output: Sorry, I cannot answer that in JSON.\nFor troubleshooting, visit: https://docs.langchain.com/oss/python/langchain/errors/OUTPUT_PARSING_FAILURE '}

Root cause

In 5c0bc46 the three except Timeout clauses (single chunk, chunk stage, merge) became except (Timeout, json.JSONDecodeError), with the "Invalid JSON response format" messages. Unless the model is ChatBedrock, each chain ends in an output parser: TolerantJsonOutputParser without a schema, JsonOutputParser(pydantic_object=...) with one. For a reply that isn't JSON, both raise langchain_core.exceptions.OutputParserException. That exception wraps the JSONDecodeError but isn't a subclass of it, so none of the except clauses match.

The existing test_execute_merge_json_decode_error didn't catch this, because it replaces invoke_with_timeout with a function that raises json.JSONDecodeError directly.

Fix

Import OutputParserException and add it to the three except tuples. The error messages don't change, raw_response is still str(e) (for this exception it includes the model's reply), and Timeout and json.JSONDecodeError are still caught.

Tests

New tests/test_generate_answer_node_invalid_json.py runs the node with FakeListChatModel and the node's real parsers. It doesn't mock invoke_with_timeout.

• Single chunk, no schema: a non-JSON reply gives "Invalid JSON response format".
• Single chunk with a pydantic schema: same.
• Two chunks whose replies aren't JSON: "Invalid JSON response format in chunk processing".
• Two chunks with valid JSON replies and a merge reply that isn't JSON: "Invalid JSON response format during merge".
• Valid JSON replies are still parsed in all three paths.

The fake model takes a lock in _call, because the chunk chains run in parallel threads and the merge test needs the replies handed out in order.

On pre/beta (b4a992f) with only the new test file added, the four error cases fail and the three valid JSON cases pass:

$ uv run --frozen pytest tests/test_generate_answer_node_invalid_json.py
FAILED ...::test_single_chunk_non_json_reply_returns_error - langchain_core.exceptions.OutputParserException: Invalid json output: Sorry, I cannot answer...
FAILED ...::test_single_chunk_non_json_reply_with_schema_returns_error - langchain_core.exceptions.OutputParserException: Invalid json output: Sorry, I c...
FAILED ...::test_chunk_stage_non_json_reply_returns_error - langchain_core.exceptions.OutputParserException: Invalid json output: Sorry, I cannot answer ...
FAILED ...::test_merge_stage_non_json_reply_returns_error - langchain_core.exceptions.OutputParserException: Invalid json output: Sorry, I cannot answer ...
4 failed, 3 passed

With the fix: 7 passed. The new file isn't on the list of test files in .github/workflows/test-suite.yml, so CI won't run it unless it's added there.

Other checks, on Windows 11 with Python 3.12.15 and dependencies from uv sync --frozen, each compared with pre/beta:

uv run --frozen pytest tests/test_atlascloud_model.py tests/test_batch_api.py tests/test_csv_scraper_multi_graph.py tests/test_depth_search_graph.py tests/test_error_page_detection.py tests/test_json_scraper_graph.py tests/test_minimax_models.py tests/test_scrape_do.py tests/test_search_graph.py tests/utils/convert_to_md_test.py tests/utils/output_parser_test.py tests/utils/parse_state_keys_test.py
uv run --frozen pytest tests/test_generate_answer_node.py
uv run --frozen ruff check scrapegraphai/nodes/generate_answer_node.py tests/test_generate_answer_node_invalid_json.py
uv run --frozen black --check scrapegraphai/nodes/generate_answer_node.py tests/test_generate_answer_node_invalid_json.py
uv run --frozen isort --check-only scrapegraphai/nodes/generate_answer_node.py tests/test_generate_answer_node_invalid_json.py

• The unit suite from .github/workflows/test-suite.yml (first command): 95 passed, same as pre/beta.
• tests/test_generate_answer_node.py: 3 failed, 5 passed, same as pre/beta. These failures are already there and this PR leaves them alone:
  • test_execute_multiple_chunks and test_execute_merge_json_decode_error: the fake invoke_with_timeout looks for a "context" key, but the node passes "content" to the merge chain.
  • test_init_chat_ollama_format: the test builds ChatOllama from langchain_community, but the node checks for the langchain_ollama class.
• ruff check and black --check pass on both changed files.
• isort --check-only passes on the new test file. It flags generate_answer_node.py for the existing order of the langchain_aws and langchain_core.runnables imports, the same as on pre/beta. The new import is where isort puts it, and the other imports aren't reordered.
• Repo-wide, the make lint commands give the same results as on pre/beta: ruff reports 6 unused imports in nodes/batch_generate_answer_node.py and utils/batch_api.py, black would reformat 23 files and isort flags 26.
• The repo-wide make type-check wasn't run. mypy on the two changed files reports nothing on the changed lines. The other errors are strict-mode errors already in the node, plus missing annotations in the new test file (the existing tests aren't annotated either).
• I didn't run the full make test (the graph, integration and browser suites need API keys or network access) or make pre-commit.