How IrishTalents uses Jev, a typed-decision model

IrishTalents helps international candidates find work in Ireland, and most of the hard parts are small labelling decisions. Which official occupation does this job title belong to? Does this advert offer visa sponsorship, refuse it, or say nothing? Is this job a realistic match for this person? Each answer changes what someone sees, and a wrong one can point a nurse to a permit route that does not fit their job, or hide a role that would have sponsored them.

Until September these decisions were made by rules: keyword lists, aliases and scoring. The rules were cheap and fast, and we had no way to see how often they were wrong. In September we put Jev, a typed-decision model from TypeSafe AI, behind three of them. Below are the numbers two weeks in, including the place where it did not help.

What a typed-decision model is

Jev returns a choice from a list you give it. You send it a description of the situation and one or more questions, and each question comes with a closed set of options. Jev answers with one option per question and a probability for every option. There is no prose to parse and no answer outside the list.

That shape suits our problems better than a general model. For a job title we need one of 369 occupation codes, or "none", and nothing else. We call Jev through OpenRouter, where at the time of our tests input cost US$0.042 per million tokens, output was free, and a call took about a third of a second (329 ms at the median, 467 ms at the 95th percentile, over 600 calls).

What a call looks like

This is the request our ingestion sends for one ambiguous job title. The criteria are the shortlist our rules produced, each described as "title (group)", plus an explicit none.

POST https://openrouter.ai/api/alpha/decisions
Authorization: Bearer $OPENROUTER_API_KEY

{
  "model": "~typesafe/jev-latest",
  "state": "IT Systems Administrator",
  "questions": {
    "soc": {
      "type": "choice",
      "instructions": "You are given a job title from an Irish job board. Pick the SOC 2010 occupation code whose description best matches the occupation of the role, or 'none' if no option fits.",
      "criteria": {
        "4159": "Other administrative occupations n.e.c. (Other administrative occupations)",
        "3131": "IT operations technicians (Information technology technicians)",
        "2139": "Information technology and telecommunications professionals n.e.c. (Information technology and telecommunications professionals)",
        "2136": "Programmers and software development professionals (Information technology and telecommunications professionals)",
        "none": "None of these SOC occupation codes fits this job title"
      }
    }
  }
}

The answer comes back keyed by question. Several questions about the same state can ride in one request, which is how the CV reader asks about every empty profile field in a single call. The values below are illustrative and trimmed to the two fields this gate reads.

{
  "answers": {
    "soc": { "choice": "3131", "confidence": 0.97 }
  }
}

Our side treats that response as untrusted input. A choice is stored only if all three checks pass:

import math

def accept(answer, offered_codes, gate=0.9):
    if not isinstance(answer, dict):
        return None
    choice, confidence = answer.get("choice"), answer.get("confidence")
    if not isinstance(choice, str) or choice == "none":
        return None          # Jev declined, or hedged
    if isinstance(confidence, bool) or not isinstance(confidence, (int, float)):
        return None          # malformed, e.g. true or null
    if math.isnan(confidence) or confidence < gate:
        return None          # below the gate: the rule-based answer stands
    if choice not in offered_codes:
        return None          # never trust an API to enforce its own contract
    return choice

Any HTTP error, timeout or unparseable body returns None from the one function that makes the call, so every caller gets the same fail-open behaviour for free.

The pattern behind every call

Every place we use Jev follows the same four steps.

Three rules apply to every call:

• Jev only chooses from a list we built. It never sees the whole taxonomy and never invents a label.
• A confidence gate, and fail open. We accept Jev's answer only when its top option reaches 0.9. Below that, or when a call fails or times out, the rule-based answer stands, so a model outage never blocks a page.
• Provenance on every value. Anything Jev decided is stored as model-decided, which means we can audit it, measure it and roll it back in one query. A value a person typed into their own profile is never overwritten.

Why 0.9

The 0.9 came from the data. On the 300 titles where our rules were already confident, and so had ground truth to compare against, 0.9 is roughly where Jev's wrong answers stopped showing up in the sample. We did not hunt for a more precise floor, because the top of the scale was the only part we would ever let write to production. Jev's probabilities also cluster near the top whether or not it is sure, so we read 0.9 as a threshold on a score, never as "nine in ten of these are right". Each task gets its own gate from its own labelled examples. The classifier that tags recruitment agencies by sector uses 0.95, because its answers appear on public directory pages.

What we send, and what we never send

For a job posting's occupation, the state is the title alone, because advert descriptions mostly describe the client's industry rather than the role. For a candidate's occupation, the stated occupation comes first, followed by field of study, up to 20 skills and the first 500 characters of their bio when they have one. Raw CV text is left out of that call: its first lines usually carry a name, email, phone number and address, none of which helps choose between five codes, and it was not shown to improve acceptance. The CV reader is the one call that does see the CV, an excerpt of up to 4,000 characters, because filling empty profile fields from it is the whole job, and a field the candidate already filled is never asked about.

Job titles to official occupation codes

Irish employment permits are decided occupation by occupation, using the SOC 2010 codes the Department of Enterprise lists. Every job we show needs one of those codes before we can say which permit route it points to. Our rules matched titles against curated aliases and scored the closest codes. For a clean title like "Staff Nurse" that works. "Senior Database Administrator - Oracle" got an office administration code, and about 7,300 live postings sat in an "ambiguous" bucket with no code at all.

We gave Jev each title with the top five codes our rules proposed, plus "none", and compared its answers with codes we had curated by hand on 600 production titles. Overall it agreed 90.3% of the time. At the 0.9 gate, which three in four titles passed, it agreed 96.4% of the time.

Then we ran it once over the 4,931 distinct titles our rules could not settle. It cost US$0.098 in total. 1,159 titles cleared the gate and gave a code to 1,735 live postings, taking postings with an occupation code from 6,274 to 8,010 (up 27.7%). We checked a sample of 15 by hand and all were right.

The gate now runs on every new job we ingest. As of 30 September, 1,947 postings have a code that Jev decided, 819 of them still live.

The disagreements were mostly ours

When Jev disagreed with our stored code at high confidence, we expected Jev to be wrong, and most of the time it was right. Our alias for "administrator" mapped database and systems administrators to an office administration code, and an industrial radiographer had been filed as a medical one. We audited the 138 titles behind that one alias. Jev flagged 16 as wrong at high confidence, and the corrections were obvious once we looked. We split that alias for specialised roles, which fixed all 16.

Where it did not transfer

Encouraged, we tried the same gate on candidates. People describe their own occupation in their own words ("Administração", "Academic background in economics"), and our alias rules were built for standardised job titles. Only 14% of candidate occupations cleared the gate. In 109 of 120 cases the shortlist we gave Jev was empty or noise.

So we tried letting Jev choose across the whole taxonomy in two steps, 25 groups and then the codes inside one. The picks were sensible, but the confidence spread out across so many options that only 12% reached 0.9.

Two lessons came out of this. First, the shortlist matters more than the model, and a threshold tuned on five close options means nothing on 369. Second, the probabilities are not calibrated, so we treat them as a score to rank and gate on, and set each gate against labelled examples for that specific task.

Reading sponsorship wording in adverts

About one in twelve adverts says something about sponsorship, and the wording is slippery. "We are unable to offer sponsorship" is clear. "Sponsorship available for the right candidate" is a maybe. "All offers are subject to background checks and right to work verification" says nothing about sponsorship at all, and our rules read it as a refusal.

We labelled 178 live adverts by hand and measured both approaches. The rules alone were right 89.6% of the time. With Jev deciding at the gate and the rules handling the rest, accuracy reached 99.4%.

Jev reads "available for the right candidate" as possible, with confidence below 0.75, so the gate hands it back to the rules instead of forcing a yes or no. When we ran it over the live adverts, 53 readings changed, 39 of them boilerplate background-check lines that had been counted as refusals. Today Jev has the final word on 558 of the 6,850 live adverts, and the rules handle the rest. At about US$0.000025 an advert, the monthly bill is around two cents.

Checking whether a job fits a person

The third use is the one candidates see. When someone finishes onboarding, we suggest open jobs that fit their profile. We labelled the top five suggestions for 30 real candidates by hand, and only 24% were roles the person could realistically apply for.

The first fix did not involve Jev. Most of the damage came from our retrieval. It only looked at the 500 newest jobs out of about 7,400, grouped every software role under one code (a front-end developer was offered DevOps lead), ignored seniority (ten years of experience brought back graduate schemes) and let duplicate adverts fill the list. Fixing those rules alone took us to 72%.

Then Jev judges each remaining pair of candidate and job: strong fit, partial fit, or no fit.

We cache each judgement by a fingerprint of the profile and of the job text, so the same pair is never judged twice until one of them changes. In production since 24 September, Jev has judged 1,951 candidate and job pairs across 98 candidates and ruled 46% of them no fit. A role ruled no fit does not appear in that candidate's Recommended list. The whole bill so far is under five US cents.

Its remaining mistakes follow a few patterns. It sometimes misses a language requirement (a role needing fluent German offered to someone who lists only English), confuses physical security with cybersecurity, and treats a student co-op as fine for a graduate. Languages are the next thing we are adding to what Jev sees.

What it costs

Inference has cost less than a dollar in total. Most of the cost was engineering time, spent building shortlists worth choosing from, labelling examples to set each gate, and wiring provenance, caching and fallbacks.

What we would tell another team

• Put the model at the end of a pipeline you already trust. Let rules and data narrow the question, then ask the model to pick.
• Make every answer reversible. Store where each value came from, so you can measure and undo a model's work without a migration.
• Set one gate per task, from labelled examples. A threshold does not travel between tasks with different option sets.
• Fail open. A missing model answer should leave you where you were, never stuck.
• Label a small set by hand before you trust any number. Every figure in this post comes from a set of 30 to 600 examples that we labelled ourselves, and we will keep measuring as those sets grow.

The honest caveats

These are our own measurements on our own data. The labelled sets are small, and the labels are ours. Jev was a few days old when we started, so model versions and prices may have changed since. IrishTalents gives information to help people search, and the Department of Enterprise decides every permit application, whatever label we show.

If you want to see what these decisions add up to, browse the jobs we list or read what changed on IrishTalents each month. How we use AI on candidate data, and the right to ask for a human review, is in our privacy policy.