Problem statement
New Home Assistant users frequently struggle to find the documentation they need to complete setup and configuration tasks. When users cannot find instructions, they turn to community forums, Discord, or third-party sources - increasing support burden and reducing confidence in the product. Poor documentation discoverability is a barrier to successful onboarding and undermines OHF's goal of making home automation accessible to everyone.
We currently lack structured, evidence-based data on where and why users fail to find documentation. Without this, documentation improvements are driven by assumption rather than observed user behavior.
Community signals
From the community forum (WTH series):
-
WTH is there no consistent documentation concept? (2022) - A user described the experience of setting up HA as "a painful search through official documentation, the forum, external blog posts, Amazon reviews, YouTube videos" before getting things working. They noted that official docs sometimes point to other docs that are incomplete or missing.
-
More/better information for beginners (December 2024) - A beginner raised that documentation lacks a consistent structure that would help new users orient themselves, specifically missing summary blocks at the top of pages that explain what an object is, its dependencies, and how to use it.
Scope & Boundaries
In scope
- Defining a set of core user workflows that require documentation (for example, adding a device, setting up an automation, troubleshooting a failed integration)
- Recruiting participants and conducting moderated or unmoderated usability tests focused on documentation findability
- Collecting and analyzing participant success rates and difficulty ratings
- An LLM-based alternative: querying documentation through LLM interfaces and evaluating response accuracy and completeness as a proxy for discoverability
Not in scope
- Redesigning the documentation site or navigation (that follows from findings)
- Testing documentation quality or technical accuracy in depth (focus is on findability, not content correctness)
- Full end-to-end task performance testing in a Home Assistant instance (optional extension of the research, not part of the core study)
Foreseen solution
Conduct a round of recorded user research sessions using a tool such as usertesting.com or equivalent. Participants are given a set of predefined tasks representing core new-user workflows and asked to find the relevant documentation on the Home Assistant website. They rate ease of finding instructions and whether they succeeded.
Optionally, a test Home Assistant instance can be provided so participants can also attempt to complete the tasks, validating whether the documentation they found was sufficient.
As an alternative or complement: run the same task set through LLM interfaces (for example, asking an LLM trained or retrieval-augmented on HA docs) and evaluate the accuracy and completeness of responses as a signal of how well the documentation structure supports machine and human discoverability.
Outputs: a structured report of findings with identified failure points, usability friction areas, and a prioritized list of documentation gaps or navigation issues to address.
Risks & open questions
- Recruiting representative new users (not power users or existing community members) may be difficult and costly.
- Small sample sizes may not generalize across the diverse HA user base.
- The LLM-based alternative measures a different thing than human findability - results would need careful interpretation.
- Open question: what constitutes "core new-user workflows"? This needs alignment before research begins.
- Open question: do we run this as a one-time study or establish it as a recurring benchmark?
Appetite
Small to medium, roughly 1 cycle, primarily research/docs team effort with no core/frontend changes required at this stage.
Execution issues
No response
Decision log
Problem statement
New Home Assistant users frequently struggle to find the documentation they need to complete setup and configuration tasks. When users cannot find instructions, they turn to community forums, Discord, or third-party sources - increasing support burden and reducing confidence in the product. Poor documentation discoverability is a barrier to successful onboarding and undermines OHF's goal of making home automation accessible to everyone.
We currently lack structured, evidence-based data on where and why users fail to find documentation. Without this, documentation improvements are driven by assumption rather than observed user behavior.
Community signals
From the community forum (WTH series):
WTH is there no consistent documentation concept? (2022) - A user described the experience of setting up HA as "a painful search through official documentation, the forum, external blog posts, Amazon reviews, YouTube videos" before getting things working. They noted that official docs sometimes point to other docs that are incomplete or missing.
More/better information for beginners (December 2024) - A beginner raised that documentation lacks a consistent structure that would help new users orient themselves, specifically missing summary blocks at the top of pages that explain what an object is, its dependencies, and how to use it.
Scope & Boundaries
In scope
Not in scope
Foreseen solution
Conduct a round of recorded user research sessions using a tool such as usertesting.com or equivalent. Participants are given a set of predefined tasks representing core new-user workflows and asked to find the relevant documentation on the Home Assistant website. They rate ease of finding instructions and whether they succeeded.
Optionally, a test Home Assistant instance can be provided so participants can also attempt to complete the tasks, validating whether the documentation they found was sufficient.
As an alternative or complement: run the same task set through LLM interfaces (for example, asking an LLM trained or retrieval-augmented on HA docs) and evaluate the accuracy and completeness of responses as a signal of how well the documentation structure supports machine and human discoverability.
Outputs: a structured report of findings with identified failure points, usability friction areas, and a prioritized list of documentation gaps or navigation issues to address.
Risks & open questions
Appetite
Small to medium, roughly 1 cycle, primarily research/docs team effort with no core/frontend changes required at this stage.
Execution issues
No response
Decision log