-
Notifications
You must be signed in to change notification settings - Fork 51
Automate alt text generation #679
Copy link
Copy link
Open
Labels
featureNew functionality, or change in existing functionalityNew functionality, or change in existing functionalityinfrastructureHas to do with changes to the development process, e.g., build scripts, CI, testing utilitiesHas to do with changes to the development process, e.g., build scripts, CI, testing utilitieswolfram languageRequires Wolfram Language implementationRequires Wolfram Language implementation
Description
Metadata
Metadata
Assignees
Labels
featureNew functionality, or change in existing functionalityNew functionality, or change in existing functionalityinfrastructureHas to do with changes to the development process, e.g., build scripts, CI, testing utilitiesHas to do with changes to the development process, e.g., build scripts, CI, testing utilitieswolfram languageRequires Wolfram Language implementationRequires Wolfram Language implementation
The problem
A lot of documentation contains images of outputs in case the outputs are graphs, hypergraphs, plots, etc. However, the images also need alt text. Currently, the alt text needs to be written manually (e.g., #673), which is error prone and takes time.
Possible solution
We could try to generate alt text automatically. This has to depend on the head of the output and maybe needs additional hints beyond the head (e.g., it might be necessary to know that a
Graphis actually a token-event graph). Further, this can never be perfect as some parts of the output will often need to be omitted and potentially described with words instead (e.g., if a graph has 10k edges). However, we can still generate the starting point automatically. This likely needs to be implemented in Rasterization.m which already generates code for including images.Alternative solutions
We can also try to just use LLMs to generate alt text, however, it doesn't look like LLMs (at least OpenAI o1 and 4o) are any good at understanding SetReplace at this point.