Finding Magic cards by what they do, not what they say
Pick a Magic: The Gathering card, and Cognate lists the cards its model rates as most similar in effect, often even when the words differ. The model is e5-base-v2, a small open text-embedding model that I fine-tuned on community tags. This page shows how I built and tested it, and where it still fails.
- Cards you can search
- 27,866
- Dev nDCG@10, TF-IDF baseline
- 0.604
- Dev nDCG@10, deployed
- 0.714
- Dev nDCG@10, untuned
- 0.565
Keyword search matches words, not effects
Players often want "a card that does what this card does". It might be a cheaper copy, the same effect in another colour, or a card for a deck theme. Search engines match words. But the first two cards below share almost every word, and they do opposite things.
If a source would deal damage to a permanent or player, it deals half that damage, rounded down, to that permanent or player instead.
If a source would deal damage to a permanent or player, it deals double that damage to that permanent or player instead.
{T}: Prevent the next 1 damage that would be dealt to each creature and each player this turn.
A model that reads for meaning should put Ghosts next to damage-prevention cards such as Kitsune Palliator, not next to Furnace. But shared words are still a strong signal. On the development cards, a baseline that only matches wording (character n-gram TF-IDF, no learning) scores 0.604. The untuned embedding model scores lower, at 0.565.
1 Data
Rules text from Scryfall
The card text comes from Scryfall's bulk data, snapshot 2026-10-16. That gives 31,066 cards. I remove all text inside matched parentheses. Most of it is reminder text, but a few parentheticals matter to the rules, and they go too. For the 847 cards with more than one face, I join the faces into one text. I use only English rules text.
2 Labels
Community tags stand in for "same effect"
The labels come from Scryfall Tagger, where players tag cards with what they do, such as "removal", "card draw" and "change target". I dropped the tags about cycles, names, errata, and printing or set trivia. That leaves 2,709 effect tags and 175,457 card–tag links.
Two cards count as similar when they share tags, and rare tags count for more. This makes the tags a proxy for "same effect". They reward any shared tag, even an incidental one. Also, I train and score the model on the same source. The benchmark scores and promotion tests use this proxy, not a human judgement. The neighbour lists use the model's cosine similarity. The trap exam below uses matches that I accepted myself.
3 Splits
Three splits, one held out for a single final test
- Fit 24,817Training pairs come only from these.
- Dev 3,049Used to compare runs and pick the recipe.
- Sealed 3,200Locked away. One authorised test used it, on the earlier 2000-update model. That test restarted once after a crash, before any model was scored. Not in the explorer.
Cards with identical text, cards that differ only by their own name, and melded pairs form 28,999 families. Most families hold one card. Each family stays in one split. Near-identical cards can still land in different splits, so the hard-lexical view below controls for that.
4 Training
Contrastive fine-tuning of a small encoder
- Input"query: " + rules textup to 94 tokens
- Encoderintfloat/e5-base-v2mean pooling
- Output768-number vectorcompared by cosine
Each card becomes one vector. A card's neighbours are the cards with the most similar vectors. The pretrained model already finds some related effects, but it scores below the wording baseline on this benchmark. Fine-tuning moves its rankings toward the community tags. It learns from pairs of cards that share an effect tag.
The loss is contrastive (InfoNCE). Each batch has 32 pairs. The loss pushes the model to score a card's partner above every other card in the batch. Cards from one family never share a batch. The loss ignores any batch card that shares a training tag with the card, so related cards do not get pushed apart. (A training tag is one that 2 to 500 training cards carry.) Each pair also gets 1 hard negative: a card with similar wording but no shared tag. I pick it at random from the card's 30 closest training cards by character n-gram TF-IDF.
Deployed run: 4,000 updates, learning rate 3e-5, seed 1 (runs/confirm-x1-s1/manifest.json).
5 Results
What changed, and how sure I can be
The same card, before and after
Each example shows the top 8 neighbours from the pretrained model and from the deployed fine-tuned model. The first six are familiar cards from the training split. They show behaviour, not performance on unseen cards. The other six are development cards, picked with a random seed (20261026) before I looked at their lists. I did not choose them, and I did not remove any. Six is too few to measure anything. They only give a rough feel for unseen cards: help on some, little change on others, and misses. You can also type the name of any card. The explorer hides 1,099 keyword-only cards, but this view does not.
Before pretrained e5-base-v2
After fine-tuned on card tags
Scores against baselines
The score is nDCG@10, averaged over 3,016 development cards (3,049 cards in the split, minus those with no rules text or no tag seen in training). Each card is ranked against 27,573 training and development cards that have rules text. The score rates the top 10 neighbours. A neighbour earns the summed weight of the tags it shares, and rare tags weigh more. Higher ranks count for more, and 1.0 is perfect. The hard-lexical view also removes cards from the same family and cards with at least 80% word overlap with the query. That way, a model cannot score points by finding copies.
Is the gain more than noise?
I resample card families and the three seeds together, 12,000 times. That gives each difference a 95% interval and a one-sided lower bound. I adjust the lower bounds for the number of comparisons (a Bonferroni correction). With three seeds, the intervals mostly reflect which cards were sampled, not how much training varies. A model passes a comparison only if it meets both rules. In the full view, the gain must be at least 0.005 with an adjusted lower bound above 0. In the hard-lexical view, the gain must be at least 0 with an adjusted lower bound above −0.010. Every adjusted lower bound is above zero.
Development cards: the deployed recipe3 seeds, 6 comparisons corrected together. Decision: promoted. The decision is provisional. It uses the tag proxy on development cards, and those cards also helped me choose the model.
Sealed cards: the earlier 2000-update model, one test4 comparisons. Gate cleared.
| Comparison | View | Difference | 95% interval | Adjusted lower bound |
|---|---|---|---|---|
| vs pretrained E5 | full | +0.151 | 0.142 to 0.159 | 0.141 |
| vs pretrained E5 | hard-lexical | +0.162 | 0.153 to 0.171 | 0.151 |
| vs fine-tuned BGE-small | full | +0.053 | 0.047 to 0.058 | 0.046 |
| vs fine-tuned BGE-small | hard-lexical | +0.055 | 0.049 to 0.061 | 0.048 |
| vs the 2000-update recipe | full | +0.010 | 0.007 to 0.013 | 0.006 |
| vs the 2000-update recipe | hard-lexical | +0.010 | 0.007 to 0.013 | 0.006 |
| Comparison | View | Difference | 95% interval | Adjusted lower bound |
|---|---|---|---|---|
| vs pretrained E5 | full | +0.133 | 0.125 to 0.141 | 0.124 |
| vs pretrained E5 | hard-lexical | +0.144 | 0.134 to 0.154 | 0.133 |
| vs fine-tuned BGE-small | full | +0.041 | 0.037 to 0.045 | 0.036 |
| vs fine-tuned BGE-small | hard-lexical | +0.045 | 0.039 to 0.052 | 0.039 |
The development results are selection-exposed. Those cards helped decide which recipe and checkpoint to keep. The sealed numbers are for the earlier 2000-update model (confirm-e5-s1: 0.705 full, 0.683 hard-lexical). The sealed split is now used up, so the deployed 4000-update model has no sealed result.
Training longer: 2000 vs 4000 updates
Doubling the training length raised the dev score on every seed that I tried. Against the 2000-update recipe, the gain is +0.010 full and +0.010 hard-lexical. The gain is small, and I measured it only on the development tag proxy.
| Run | 2000 updates | 4000 updates | Change |
|---|---|---|---|
| seed 1 | 0.706 | 0.714 | +0.009 |
| seed 2 | 0.707 | 0.717 | +0.010 |
| seed 3 | 0.707 | 0.718 | +0.011 |
Trap exam
The exam has 51 items. Two come from my own review of the app. Simple text rules in my code proposed the other 49, and I checked and accepted them. Each item names a card or line, the match I accept, and a lookalike that must rank lower. The lookalikes use shared rider clauses, one-word flips and changed numbers. The rules chose the matches and lookalikes from the same Tagger tags that the model trains on, so the exam is not independent of the training labels. Its fit cards also appear in the training pairs.
The deployed model fails 4 (T001, T021, T022, T029), and two of them are below. The other two seeds of this recipe pass 49 and 48, so a difference of one or two items is within seed noise. The exam is not part of the final promotion test. But it was one of the rules for keeping the 4000-update recipe and for judging the follow-up runs, so it is selection-exposed.
6 Where it fails
Where it still goes wrong
Deflecting Swat: the labels favour the wrong neighbour
Deflecting Swat redirects a spell, and it is free if you control a commander. The ideal neighbour is Redirect. But the fine-tuned model ranks Obscuring Haze higher, and Haze shares only the free-with-a-commander line. The tags may be the reason. Swat shares three tags with Obscuring Haze and only two with Redirect, so training on tags may pull the wrong card closer. I did not isolate the cause, and my one follow-up run on effect-only labels did not fix it. On the trap exam, the margin went from -0.064 before tuning to -0.469 after. A margin below zero is a fail. In the explorer, pick Swat's redirect line under "Similar cards", and Redirect does show up.
Lexical twins still fool it
For Ghosts of the Innocent, the model still ranks Furnace of Rath above Kitsune Palliator (trap-exam margin -0.223). I ran three follow-up runs, each with one intended change: more hard negatives, effect-only labels, and rare-tag weighting. They also used gradient checkpointing to save memory, so they are not fully controlled tests. None of them cleared its rule. Each also failed at least one exam item that its comparator passes. The rider problem is not solved.
- Proxy, not people. The scores and promotion tests measure how well the model agrees with community tags. I accepted the trap-exam items myself; a panel did not. No human relevance study has been run. I built a blinded pool of 1,256 card pairs from an earlier model. Nobody has labelled it, and its labelling tool is not in this repository.
- Selection-exposed. The development scores helped me choose models. The only sealed result is for the earlier model. I fixed the seeds, thresholds and statistical test before the first training run, and recorded them in a private log that is not in this repository. The promotion test for the 4000-update recipe reuses those rules, but I wrote its design down after two of its three seeds had finished.
- Card to card only. You always start from a card. Cognate is not a natural-language search model, and I never measured how well it handles typed queries.
- Exploratory views. The explorer marks three views as exploratory: matches for one chosen ability, Line match and Face match. I do not claim that any of them is an improvement. With the earlier 2000-update model, Line match scored below Whole card on the development tag proxy (0.687 against 0.706). Face match scored about the same.
- Few seeds. The three recipes in the main chart each ran with 3 seeds. The follow-up runs used one seed each. Three seeds say little about run-to-run variation.
- Long and rare cards. The model reads at most 94 tokens, including its prefix. That cuts 260 of the 27,573 benchmark cards. Scores are lower for rare effects (0.636), long cards (0.628) and two-faced cards (0.650), against 0.714 overall.
- Lenient metric. The top result almost always shares at least one tag with the card (MRR@10 is 0.985). So the metric mostly rewards putting good neighbours in the right order.
- Narrow comparison. I compared only with the base model, a small BGE model and a TF-IDF baseline. I did not compare with larger embedding models.
- Display filter. The explorer hides 1,099 keyword-only cards, such as plain flyers, from the map and the lists. All scores include them.
- Snapshot. Cards printed after the data snapshot are missing.
7 Built with
Built with
Training and evaluation use Python, PyTorch and Hugging Face Transformers. The map uses scikit-learn (t-SNE). This is a static site built with SvelteKit, with no server of its own and no tracking. Card images load from Scryfall. A script fills in every result on this page from my local run and evaluation records. Those records are not in this repository.
Card data and images come from Scryfall and tags from Scryfall Tagger. The idea of a trap exam, and several ideas for line-level labels and training, come from Delvefall. I used the ideas only. I did not copy any code, data or exam items.