Reject the invalid 'llm.model' config key with an actionable error

[akushonkamen]

Fixes #427

Problem

LLMModelConfig has no model field — models are configured via the llm.models array or llm.primary_model. But dacite silently drops dict keys that don't match a dataclass field, so a config carrying llm.model passed Config.from_dict without a whisper and produced llm.models == []. Downstream, every LLM call then died in LLMEnsemble._sample_model with IndexError: list index out of range, was retried, and flooded the log — exactly the reporter's "heaps of failed attempts: ERROR - LLM generation failed: list index out of range". The reporter's self-diagnosis in #427 is precise: without primary_model, "the model is not recognized and an empty list is loaded as the model ensemble".

The kicker: this is not a user reading carefully enough. The README itself teaches the broken spelling — the Google Gemini, Local Models (Ollama/vLLM), and OptiLLM provider examples all wrote model: under llm:. Anyone copying an official example hit the crash; the reporter did.

What

• openevolve/config.py (Config.from_dict): raises ValueError("Invalid config key 'llm.model'. Use 'llm.primary_model' or the 'llm.models' array instead (see README configuration section).") when llm.model is present. Deliberate choices:
  • Explicit error, not a silent alias: from_dict already has three explicit ValueErrors (diff_pattern regex, island_system_messages count, programs_as_changes_description requires diff mode), so this matches the house style. model was never a valid field, so there is no backward-compatibility burden (unlike the primary_model alias precedent in __post_init__, which migrated a field that used to exist). A silent mapping would also leave model-plus-models semantics ambiguous.
  • In from_dict, not __post_init__: internally constructed Config() objects never pass through a dict and legitimately start with an empty ensemble (that existing convention is documented in the __post_init__ validation comment), so the dataclass path must keep accepting models == [].
  • Guarded with isinstance(config_dict["llm"], dict) so a malformed llm: value still fails in dacite's existing type check instead of a confusing error from the new check.
• README.md: the three provider examples (Gemini ~line 359, Ollama/vLLM ~line 375, OptiLLM ~line 399) corrected from model: to primary_model:, consistent with examples/*/config.yaml and the rest of the configuration docs. A sweep found no other bare model: keys in README or example/config YAMLs (embedding_model: is a different, valid key).
• tests/test_llm_model_key_rejected.py: new unittest module (no LLM, ~0.001s):
  • test_model_key_raises_helpful_error: Config.from_dict({"llm": {"model": ...}}) raises ValueError whose message names primary_model and llm.models;
  • test_model_key_with_other_keys_still_raises: still raises when model: is mixed into an otherwise valid config (guards against a "only check bare dicts" false negative);
  • test_primary_model_config_still_valid and test_models_array_config_still_valid: both documented spellings keep working (regression guards);
  • test_code_constructed_config_unaffected: Config() keeps its empty-default ensemble — the internal initialization path is untouched.

Red -> green evidence

• Before the fix:

  $ python -m unittest tests.test_llm_model_key_rejected -v
  ...
  FAILED (failures=2)
  AssertionError: ValueError not raised   (x2: bare llm.model, llm.model mixed into valid config)
  

  and the 3-line repro printed BUG: silently accepted, models = []:

  $ python -c "from openevolve.config import Config; \
    cfg = Config.from_dict({'llm': {'api_base': 'https://x/v1', 'model': 'glm-ocr'}}); \
    print('BUG: silently accepted, models =', cfg.llm.models)"
  BUG: silently accepted, models = []
  

• After the fix: Ran 5 tests in 0.001s ... OK, and the same repro now raises
  ValueError: Invalid config key 'llm.model'. Use 'llm.primary_model' or the 'llm.models' array instead (see README configuration section).

Full suite: OPENAI_API_KEY=test-key-for-unit-tests python -m unittest discover tests -> Ran 579 tests in 33.177s ... OK (574 baseline + 5 new).

Notes

• Open WebUI itself (the reporter's setup) works fine through its OpenAI-compatible endpoint once the config uses primary_model; this PR makes that mistake impossible to miss instead of silently fatal.
• Open PR use primary_model instead of model for ollama config #451 fixes the same model: → primary_model: mistake for the Ollama README example only; this PR fixes all three examples plus adds the config-layer validation, so use primary_model instead of model for ollama config #451 would become redundant (happy to drop the Ollama hunk here if use primary_model instead of model for ollama config #451 lands first — at most a 1-line textual conflict).
• Scope deliberately excludes a generic unknown-key warning (e.g. diffing user keys against dataclass fields) — happy to discuss that as a follow-up, but it would flag legitimately ignored keys and balloon this change.

[CLAassistant]

All committers have signed the CLA.