We’re both physicists, and one of our biggest frustrations during grad school was finding research — There were a lot of times when we had to sit down to scope out new ideas for a project and quickly become a deep expert, or we had to find solutions to really complex technical problems, but the only way to do that was manually dig through papers on Google Scholar for hours. It was very tedious, to the point where we would often just skip the careful research and hope for the best. Sometimes you’d get burned a few months later because someone already solved the problem you thought was novel and important, or you’d waste your time inventing/building a solution for something when one already existed.
The problem was there’s just no easy way to figure out what others have done in research, and load it into your brain. It’s one of the biggest bottlenecks for doing truly good, important research.
We wanted to fix that. LLMs clearly help, but are mostly limited to general knowledge. Instead, we needed something that would pull in research papers, and give you exactly what you need to know, even for very complex ideas and topics. We realized the way to do this is to mimic the research strategies we already know work, because we do them ourselves, and so we built an agent-like LLM pipeline to carefully search in a way that mimics human research strategies.
Our search system works a bit differently from casual search engines. First, we have you chat back and forth with an LLM to make sure we actually understand your really complex research goals up front, like you’re talking to a colleague. Then the system carefully searches for you for ~3 minutes. At a high level, it does something similar to tree search, following citation rabbit holes and adapting based on what it discovers to look for more content over multiple iterations (the same way you would if you decided to spend a few hours). The 3 minute delay is annoying, but we’re optimizing for quality of results rather than latency right now. At the end there’s a report.
We’re trying to achieve two things with this careful, systematic agent-like discovery process:
1. We want to be very accurate, and only recommend very specific results if you ask for a specific topic. To do this, we carefully read and evaluate content from papers with the highest quality LLMs (we’re just reading abstracts and citations for now, because they’re more widely accessible - but also working on adding full texts).
2. We want to find everything relevant to your search, because in research it’s crucial to know if something exists or not. The key to being exhaustive is the adaptive algorithms we’ve developed (following citations, changing strategy based on what we find, etc). However, one cool feature of the automated pipeline is we can track the discovery process as the search proceeds. Early on, we find many good results, and later on they get more sparse, until all the good leads are exhausted and we stop finding anything helpful. We can statistically model that process, and figure out when we’ve found everything (it actually has an interesting exponential saturation behavior, which you can read a bit more about in our whitepaper (https://www.undermind.ai/static/Undermind_whitepaper.pdf), which we wrote for a previous prototype.)
You can try searching yourself here: https://www.undermind.ai/query_app/promotion/. This is a special HN link where, for today, we’ve dropped the signup gate for your first few searches. Usually we require login so you can save searches.
We’re excited to share this with you! We’d love to hear about your experiences searching, what’s clear or not, and any feedback. We’ll be here to answer any questions or comments.
I'd love to see limitations like this quantified and clearly flagged. Otherwise there's a danger that people may the assume results are definitive, and this could have the opposite outcome to that intended (much time spent working on something only to disocver it's been done already).
If you can get this to work for patent searches across multiple languages, you'd really have a killer product. Patent searches, via an attorney, cost thousands of dollars each and are nevertheless frequently imperfect. (I had a patent denied because somewhere, in a throwaway paragraph buried in the 100-page description of an entirely different invention, something similar was briefly mentioned and never referred to again.)
I'd gladly pay $100/month for "Deep Patent Search," and more than that if it's really good.
Claude 3.5 is reluctant to provide references—-although it will if coaxed by prompting.
Undermind solves this particular problem. A great complement for my research question —- the evidence that brain volume is reduced as a function of age in healthy cognitively normal humans. In mice we see a steady slow increase that averages out to a gain of 5% between the human equivalents of 20 to 65 years of age. This increase is almost linear as a function of log of age.
Here is the question that was refined with Undermind’s help:
>I want to find studies on adult humans (ages 20-100+) that have used true longitudinal repeated measures designs to study variations in brain volume over several years, focusing on individuals who are relatively healthy and cognitively functional.
I received 100 ranked and lightly annotated set of 100 citations in this format:
>[1] Characterization of Brain Volume Changes in Aging Individuals With Normal Cognition Using Serial Magnetic Resonance Imaging S. Fujita, ..., and O. Abe JAMA Network Open 2023 - 21 citations - Show abstract - Cite - PDF 99.6% topic match Provides longitudinal data on brain volume changes in aging individuals with normal cognition. Analyzes annual MRI data from 653 adults over 10 years to observe brain volume trajectories. Excludes populations with neurodegenerative diseases; employs true longitudinal design with robust MRI techniques.
1. It's on adults.
2. It's longitudinal over multiple years.
3. It studies variations in brain volume.
4. It focuses on healthy individuals.
You can see the full results for that search text here: https://undermind.ai/query_app/display_one_search/e1a3805d35...
One thing I'd improve is how it asks followup questions.
For example, I asked "What are the best types of videogames that improve cognition?"
The followup response was
```
Are you specifically looking for studies that focus on cognitive improvements as measured by standardized tests, or are you more interested in everyday functional improvements observed in real-world settings?
Could you be more precise about which cognitive domains you're interested in? For example, are you focusing on memory, problem-solving, attention, or something else? Additionally, are you looking for papers that compare different genres of videogames (e.g., action, puzzle, strategy) or studies targeting specific age groups or populations (e.g., children, older adults)?
Lastly, are you interested in experimental studies where the cognitive benefits were directly measured pre- and post-intervention, or are observational studies that report correlations between videogame use and cognitive performance also of interest to you? Understanding this will help determine the type of articles we should prioritize in the search.
```
It would be great if it turned those into multiple choice. For example:
```
Could you be more precise about which cognitive domains you're interested in?
[] memory
[] problem-solving
[] attention
[] something else (please specify)
```
Would save a ton of time having to reply/ reread everything.
Is exactly how you gain expertise in a field and/or find those subtle gaps in knowledge that are the seeds of real breakthrough
I think this is somewhat equivalent to how we used to have to learn 100 different integrals and derivatives in calculus. That's somewhat helpful. I learned to see patterns in math like that, the same way I learned a decent bit by browsing irrelevant abstracts and follow citation trails. But physically memorizing 100s of integrals is mostly a waste, and so are the irrelevant abstracts. You'll be much better at math (and hopefully science) if you can learn ~10 key integrals or read ~10 abstracts, and then spend the rest of your time understanding the high level patterns and implications by talking with an expert. Just like I can now ask GPT-4 to explain why some integral formula is true, which ones are related, and so on.
And that's the last point - these literature search tools aren't developing in isolation. We will get to have a local "expert" to discuss what we find with. That changes the cost-benefit analysis too.
The old dinosaurs on HN are snickering about the word "manual" here, having blown days pawing through the library stacks.
There is also the real fact that the greatest discoveries usually come from obscure corners and reading as much as you can is the only way to explore those corners. Otherwise, you're just refining what was done before you
Arbitrary numbers are really the least information. At least use last names and years, so I can have some idea which paper you are talking about without scrolling back and forth.
There's a 50/50 false positive rate, but I can deal with that. It means looking at 10 papers to find 5 useful ones instead of looking at 1000 papers to also find 5 useful ones.
I'm impressed.
I tried a search on my previous research area (https://www.undermind.ai/query_app/display_one_search/5408b4...) and it missed some key theoretical papers. At the same time, it picked up the three or four papers I’d expected it to find plus a PhD thesis I expected it to find. The results at the top of the list though are very recent and one of them is on something totally different to what I asked it for (“Skyrmion bubbles” != “Skyrmions”). The 7th result is an absolutely core paper, would be the one I’d give to a new PhD student going into this area and the one I’d have expected it to push up to the top of the list.
But the fact I wanted to save a result is a good sign. Nice work!
Also, on the usual system if you're logged in (instead of the HN free link), all your searches automatically get saved on a "history" page.
Anyway, I do not mean to detract from the accomplishment -- and I liked the product! So I hope you take the above feedback/nitpicking in the spirit it is intended.
One anecdote that I heard from the team developing it: turned out that researchers more readily sourced material from their social networks, notably twitter at the time. Meta's search functionality didn't receive enough traffic and eventually was shut down.
Perhaps LLMs will make the search capability more compelling. I guess we'll see.
this was the search <https://www.undermind.ai/query_app/display_one_search/cba773...> if you need a reference too it, ie bugs or performance monitoring...
You should have seen what it used to be like a few decades ago :)
One suggestion: The back-and-forth chat in the beginning could be improved with a more extensive interaction. So, the final prompt could be more fine-grained into a specific area/context/anything one would aim for.
This is obnoxious. Please remove this unnecessary roadblock.
Their core feature now is a fast keyword search engine, but they also have a few advanced search features through their API (https://api.semanticscholar.org/api-docs/) like recommendations from positive/negative examples, but neither KW search nor these other systems are currently high enough quality to be very useful for us.
FYI our core dataset for now is provided by Semantic Scholar, so hugely thankful for their data aggregation pipeline and open access/API.
And yes, Semantic Scholar is a wonderful part of the academic commons. Fingers crossed they don't go down the jstor/oclc path.
Are you breakdown the question into subtopics, doing a broad search and then doing some sort of dim reduction -> topical clustering to get it in the format?
In my opinion elicit has better looking UI and much more features and further along
Feature-wise, we definitely have a lot of work to do :) What crucial pieces do you think we're missing?
Are there any plans on releasing any sort of API integration? I work in Technology Transfer consultancy for research institutes in Europe, and often we have to do manual evaluation of publications for novelty check and similar developments. Since most of the projects we work on were developed by leading researchers in academic institutions, it is important for us to quickly assess if a certain topic has been studied already.
Currently, one of my company's internal projects is a LLM-powered software to automate much of the manual search, together with other features related to the industry.
I think it be very beneficial for us to implement academic papers search function, but for that an API system would be required.
Great work nonetheless, good luck on the journey
let’s say i am interested in coffee and i’d like to get new research papers on it. would this work?
From a strategy perspective, we designed it for topic search because it makes more sense to find everything on a topic first, then filter for the most recent, if recent is what you want. That's because there is a lot of useful information in older articles (citation connections, what people discuss, and how), and gathering all that helps uncover the most relevant results. Conversely if you only ever filtered on articles in the last year, you might discover a few things, but you wouldn't have as much information to adapt to help the search work better.
So, you can ask for articles on coffee (though ideally it should be something a bit more specific, or there will be thousands of results). Our system will carefully find all articles, then you can filter for 2024 articles or look at the timeline.
I did 2 searches.
First I asked about a very specific niche thing. I gave me results but none I wanted. It looked like I missed a crucial piece of information.
So I did the second search. I started with the final request it written for the previous search and added the information I though I missed. It gave me virtually the same results with a little sprinkle of what I was actually after.
A few observations:
1. I'm not sure but it seems like it relies too much on citation count. Or maybe citations in papers make it think that the paper is absolutely a must read. I specifically said I'm not interested in what's in that paper and I still got those results.
2. I don't see much dissertations/theses in the result. I know for sure that there a good results for my request in a few dissertations. None of them are in the results.
That said, while I didn't get exactly what I want I've found a few interesting papers even if they're tangential to the actual request.
Regarding citations: we use them, but only for figuring out which papers to look at next in the iterative discovery process, not for choosing what to rank higher or lower at the end (unless you explicitly ask for citations). It's ranking based on topic match.
If you're comfortable, posting the report URLs here can let us debug.
What use case are you thinking of?
And post.harvard.edu has been sunsetted for alums, so I don't have that email either.
1. https://answerthis.io/
The results from their website aren't sharable, but their lists of references do not seem relevant (ie. they miss the fact that shuttling needs to be in 3D, and the list of experiments for laser cooling to BEC is missing all of the relevant papers).
I think, like other research tools, they're more focused on the summarization/extraction of information, rather than the discovery process (though they are similar to us in the way they say they do multi-stage retrieval and it takes some time).
I love all the work that's being done in the field.
Can't wait for science to be faster.
It doesn’t recognize my university.
With the current attitudes to AI, the name feels a little tone deaf being so easily mistaken for AI undermining people.
I gave it a version of my question, it asked me reasonable follow-ups, and we refined the search to:
> I want to find randomized controlled trials published by December 2023, investigating interventions to reduce consumption of meat and animal products with control groups receiving no treatment, measuring direct consumption (self-reported outcomes are acceptable), with at least 25 subjects in treatment and control groups (or at least 10 clusters for cluster-assigned studies), and with outcomes measured at least one day after treatment begins.
I just got the results back: https://www.undermind.ai/query_app/display_one_search/e5d964....
It certainly didn't find everything in my dataset, but:
* the first result is in the dataset.
* The second one is a study I excluded for something buried deep in the text.
* The third is in our dataset.
* The fourth is excluded for something the machine should have caught (32 subjects in total), but perhaps I needed to clarify 25 subjects in treatment and control each.
* The fifth result is a protocol for the study in result 3, so a more sophisticated search would have identified that these were related.
* The sixth study was entirely new to me, and though it didn't qualify because of the way the control group received some aspect of treatment, it's still something that my existing search processes missed, so right away I see real value.
So, overall, I am impressed, and I can easily imagine my lab paying for this. It would have to advance substantially before it was my only search method for a meta-analysis -- it seems to have missed a lot of the gray literature, particularly those studies published on animal advocacy websites -- but that's a much higher bar than I need for it to be part of my research toolkit.
(Also, if you want, you can share your report URL here, others will be able to take a look.)
My wife does that for her day job (in the U.K. national healthcare system) and the systematic reviews have to be super well documented and even pre-registered on a system called PROSPERO. The published papers always have the full search strategy at the end.
However, because I'm writing a methods-focused review -- we only look at RCTs meeting certain (pretty minimal) criteria relating to statistical power and measurement validity -- what I'm doing is closer to a combination of review of previous reviews (there have been dozens in my field) and a snowball search (searching bibliographies of papers that are relevant). I also consulted with experts in the field. however, finding bachelor's theses has been challenging, but many are actually relevant, so undermind was helpful there.