Yes, the problem is real. I'd try your solution if it existed.
Optimal for me would be emacs or vscode keybindings, not the 4-fingers of tablet computing.
I've never encountered this issue with figures and drawings. Maybe I have not read enough figure-packed textbooks, but I don't remember ever losing the reference to a figure. I do remember losing the reference to an equation when reading linear programming models or the like, but we are talking about ~30 equations stacked one above the other. But since I mostly read these from academic papers in pdf which already include suitable hyperlinks, I would say that the overall issue is almost nonexistent.
I definitely recall doing quite a bit of flipping pages back and forth when going through STEM textbooks in real life - often the text describing a graph is on the page preceding or following the graph, due to the realities of how the page flows. I could see an augmented reality app for physical textbooks, where you could take a picture of figure 6.3 and then pointing the camera at "Fig. 6.3" superimposes the figure over the text on your phone, or something.
Aside from AR, this doesn't seem like it would be super useful for physical textbooks, though. Are e-textbooks commonly used (more common than physical)?
What format are most e-textbooks stored as? Is there an open-source e-reader for which a plugin could be made to do this?
I don't know - but it's a cool idea.
I have been consistently successful at finding PDF versions of the textbooks for my classes. Kindle e-books are sometimes available, and the software prototype I’m making would work just as well with them. Potentially it could even be used via a phone camera for a hard copy of a book - that is a great idea, thanks!
Something like this would be great. Also adding links to the pdf (sci-hub / libgen / archive.org if necessary) for references given would be a huge plus; for many kinds of scholarship, the minute or two to track down a pdf (going from the short reference given inline, to the bibliography, then to googlescholar or libgen, then finding the right page...) really adds up and becomes tiring if you're not sure exactly what you're looking for but are getting benefit from following a gradient given by crawling the web manually (i.e. using the context of the reference as a sort of annotated bibliography).
Also there should be programmatic ways to generate some of this for stuff written with LaTeX, e.g. most arxiv papers and modern math/CS textbooks. Maybe someone's already done this for LaTeX for all I know.
Problem: STEM textbooks often reference figures and equations from earlier in the textbook. They usually do this with statements like "the shearing stress τ... may be obtained from the shearing-stress-strain diagram of Fig. 3.30." But Fig. 3.30 might be from 5 pages ago, or even earlier. Many textbooks don't have a list or index of equations, so the only way to find the referenced figure is to search page by page.
How this would ideally be solved: Every textbook would have an e-book version that links to, and previews, referenced equations every time they are mentioned.
Why not do this: Different textbooks identify and reference figures and equations different ways. Many readers of e-books use DRM-protected apps. These make it hard to create a universal solution that automatically identifies and annotates texts to provide this functionality. In addition, building an e-reader with this functionality requires building an e-reader to display PDFs and other formats, which seems very hard.
How I suspect most people solve the problem: Most people probably deal with this more or less like I do. The struggle to be diligent and actually look up the referenced figure. They might try to memorize the information. They also might put it into a list of reference notes containing the equations.
Why these solutions are suboptimal: Trying to look up or memorize information is cognitively and motivationally challenging. Learning dense material is already hard, and this adds to the burden. Taking notes can help. However, this leads to visual clutter when most equations aren't necessary at a given time, or when material that needs to be looked at simultaneously is distributed across different pages of notes. It's not always easy to get the required information into the note-taking system. Some note-taking systems also have trouble managing the sort of information a STEM student might need.
How this new app would solve the problem: This app depends on a "snip" feature, where we take a screenshot of part of the screen. Students would read an e-textbook, and specify in the app a collection of tags under which all new "infosnips" will be categorized (i.e. "Mechanics of Materials"). When it presents a new equation or figure, they take a "snip" of it, including its caption. Either instantly, or at the press of a hotkey, the app uses image-to-text to identify the reference number (i.e. the "3.30" in "Fig. 3.30"), as well as determines whether it's a figure, equation, or table. It checks for conflicts with other "infosnips" sharing the same ID. If no conflicts exist, it then tags and stores the new "infosnip" as an image.
Users can then define and display collections of "infosnips" on boards, akin to Pinterest, in a grid. For example, imagine a user on page 392 of their textbook. In addition to presenting 7 new "infosnips," this page references 3 "infosnips" from previous parts of the textbook that the user already entered into the system during their earlier reading. The user can call up those "infosnips" at the press of a button and add them to the current board, which contains just the material under active discussion.
Infosnips keep track of which other infosnips they're associated with on different boards. This allows users to explore them via a web of relationships. There could be lots of additional functionality on top of this basic image-based system, such as the ability to convert images of equations to text, to annotate "infosnips" with text from the user, to share "infosnips" for a certain textbook with other users, etc. But the core functionality is just to conveniently store "infosnips" and view them in "boards."
My questions for you:
Thank you for your feedback!