about

medifreek is a PubMed search tool and a personal project. The idea was to improve existing PubMed search by using modern search techniques, i.e. semantic search (also called vector search). Because of this, medifreek is flexible when it comes to keywords, synonyms, and common misspellings as well: cholecalciferol will bring up pages related to vitamin D and cacner will give you results close to cancer, because they're semantically similar.

You can bookmark articles. Bookmarks persist until you unbookmark them. They cannot be shared between devices.

The data is up-to-date as of mid-February 2025. medifreek only stores English articles.

Technical details

medifreek started off as a simple vector search engine for PubMed. There were others out there, but I thought this one could be better. The goal of medifreek was to be fast enough, but especially complete: better to wait a little longer than miss a result. If someone was looking for information on, say, "chronic serious chorioretinopathy", I wanted medifreek to pull up everything they could realistically want (at least that's on PubMeD) and put it all in one place.

And to be honest, PubMed itself already has a good search engine, so no shade do I throw. It's not surprising: they have millions of dollars and a team of geniuses. I have dozens of dollars and, let's say... casual experience with computers. But I like making things and the idea amused me, so I ran with it.

In building medifreek I quickly ran into design problems like:

1. Which model do I use for encoding?

2. What parts of the text do I even encode

3. How do I compute embeddings cheaply?

4. How much space and RAM do I need?

5. What kind of index should I use?

Initially I went with the bog-standard sentence-transformers/all-MiniLM-L6-v2 model for encoding. I tried other models with slightly more success, but with ~40M articles, I had to be careful about space and compute as I'm on a rather tight financial leash and I'm also a bit of a nickelpinch. Ultimately, in reading about someone else's failed vector search project, I got tipped off to the existence of matryoshka (nesting-doll) models.

During training, Matryoshka models are encouraged to front-load the most important information by computing a loss against the full embedding but also against half, quarter, eighth embeddings, and so on. In trying to maximize the loss, the model tends to put the most important information in the first components of the vector. Once trained, when you compute an encoding and truncate it, you tend to truncate the least important dimensions. It's a bit like compressing music by cutting off the highest and lowest frequencies.

So this is nice for two reasons. The first is that whenever you have a large vector encoding, you get smaller ones for free. You can dynamically tailor the vectors as you need to to balance cost/performance. On the other hand, you can get better performance in a smaller vector from the outset. The model I chose outperforms most all-MiniLM-L6-v2 -- a 384 dimensional model -- at just 64 dimensions -- because it's trained on biomedical texts and puts all the important stuff up front.

To put this into context: 40M float32 vectors of 64 dimensions translates to ~10GB worth of storage. Hetzner puts that amount of storage at around $1 a month. If these were 384 dimensional vectors (like they are for all-MiniLM) that would be 60GB and they'd be 6x slower to scan. So this is a major improvement right from the get-go.

So now -- what to encode? Encoding the titles is straightforward enough: models do best at encoding text that's semantically similar, and titles generally express one main idea.

But encode too many different things and when you lump them together you end up with a vector which doesn't encode much of anything specific. So although I wanted to encode the entire abstracts, they tend to be, well, too abstract. Encoding by sentence would be nice, but that would be terabytes worth of vectors -- impossible to work with on a budget. So I went back and forth about the best way to do this, since a good PubMed search engine should index the information in the abstracts in one way or another.

It eventually occurred to me that abstracts are, most of the time, grouped into semantically-related chunks: background, materials and methods, conclusion, etc. And when they're not, the tend to be more succinct. So I decided to go in-between and chunk those pieces to compute the vector embeddings. This seems to work acceptably well. So those different pieces of the abstract are included in the search too.

At the beginning I looked into to renting GPUs from Hetzner and elsewhere, but couldn't justify the cost. So I sucked it up and had my little Hetzner machine with its 16vCPUs patiently churn out the embeddings. I honestly didn't know any better! But I got curious and experimented with the free GPUs on Google Colab notebooks. WOW! So that's why people talk about GPUs all the time. Seriously, I didn't know! That's when I found vast.ai and saw I could get a week of training done in just a few hours by renting some cheap GPUs in third-world countries and squeezing them until they bled. So I spent ten bucks for some GPU time instead of waiting two weeks and it was totally worth it.

And that's when ten bucks became twenty became thirty became... because you know, you really have no idea what you're doing when you're just starting out. It turns out that it's not so easy to just "compute" embeddings for 100M vectors! You have to send the data to some guy's machine in Bulgaria, process it, send it back, insert it -- all while milking the most out of your GPUs and, of course, picking the right embedding model. And you can never know if the embedding model you picked will actually work on your full, 100M-vector dataset until you try it out, and by then, you're down a fat wad and the guy in Bulgaria is already living large. So it's helpful to try to get things right the first time!

Back to what I was talking about: search. To make a long story short, we have the vectors. But even vector search doesn't get all the way there, so we use postgres's built-in text search with indexes just to make sure we'll get everything we could possibly want. Postgres handles this very nicely. It converts strings of texts into lists of tokens, then builds an inverted index: a map from each token to all the texts that contain it. (This is in contrast to scanning each document for a list of words). Then finding those texts is super fast -- you just hash the tokens in the search to find all the associated documents. In other words, it's O(1).

From there, once I have my candidates, I whip out the cross-encoder to finish the job. It ranks matches and does a better job than bi-encoders alone. But it's more costly, so it helps to do it after you've already narrowed down your results.

There are several good choices of index. Initially I preferred HNSW, but it loves RAM, and I can't afford that. So instead I went with DiskANN, which leverages the SDD to do a lot of the heavy lifting.