To measure AI literacy on your team, test what people can actually do with AI on realistic work tasks, not how confident they feel or how often they open a tool. Focus on a few observable skills, such as giving clear instructions, catching wrong output and handling data safely, and score them with a short rubric. Run a baseline, train to the gaps, and measure again with the same method a few weeks later.
Key takeaways
- Usage numbers and self-ratings tell you who is enthusiastic, not who is skilled. Measure performance on real tasks instead.
- Five skill areas cover most office work. They are understanding the basics, choosing good uses, directing the tool, checking output and using it responsibly.
- A mix of a short knowledge check, one hands-on task and a manager observation gives a fair picture without a heavy process.
- Make the baseline low stakes. People who fear the results will hide what they do not know.
- Re-measure with the same method after training, and track errors caught and time saved, not just scores.
Why measure AI literacy at all?
Most teams already use AI in some form, but few have a clear view of how well. Gallup reported in June 2025 that the share of US employees using AI at work a few times a year or more had nearly doubled in two years, to 40%, while only 30% said their organization had general guidelines or formal policies for it. That gap means many people are learning on their own, with uneven results.
Uneven skill shows up in predictable ways. One analyst writes a precise request and checks every number. Another pastes a client file into an unapproved tool and forwards the summary without reading it. Both count as "using AI." Measuring literacy helps you see the difference, target training where it will matter, and avoid spending a training budget teaching people things they already know.
What should you measure?
In February 2026 the US Department of Labor published a voluntary AI literacy framework for workforce programs and employers. According to a summary by the law firm Fisher Phillips, it names five core areas, which are understanding AI principles, exploring AI uses, directing AI effectively, evaluating AI outputs and using AI responsibly. Those areas translate well into skills a manager can observe.
| Skill area | What good looks like at work | How you can see it |
|---|---|---|
| Understanding the basics | Knows AI can be confidently wrong and has no access to internal facts unless given them | Short knowledge questions |
| Choosing good uses | Picks drafting, summarizing or sorting, and avoids using AI for final judgment on high-stakes calls | Scenario questions about which tasks fit |
| Directing the tool | Gives goal, audience, context and format, and iterates when the first result misses | A hands-on task with a scored prompt |
| Checking output | Verifies facts, numbers and quotes against the source before using them | Spotting planted errors in an AI draft |
| Using it responsibly | Keeps confidential data out of unapproved tools and discloses AI use where required | Scenario questions tied to your policy |
You do not need all five on day one. If your biggest worry is data leaks, start with responsible use. If people complain that AI output is useless, start with directing the tool. Pick the two or three areas tied to real problems on your team.
How to assess skills without a big program
The most reliable signal is a person doing a task that looks like their actual job. A combination of three light methods works for most teams.
- A short knowledge check of 10 to 15 scenario questions, built around situations your team faces. Ask "a colleague wants to paste a customer contract into a free tool to summarize it, what should they do?" rather than definitions.
- One hands-on task of about 20 minutes. Give everyone the same input, such as a messy meeting transcript, and ask for a summary with decisions, owners and dates. Score the result and the request they wrote.
- A planted-error exercise. Hand people an AI-written draft with three or four mistakes, such as a wrong total, an invented source or a dropped caveat, and see how many they catch.
A manager observation can round this out. Over two weeks, note whether people check output before sharing it and whether they ask when unsure about data. Keep notes factual and tied to behavior.
How to score fairly
A simple rubric keeps scoring consistent across people and graders. Use three levels per skill rather than a ten-point scale, because finer scales create false precision.
- A score of 1 means the person needs support. They accept output as is, give vague requests or share data without thinking.
- A score of 2 means they are working independently. They give clear instructions, check the important facts and follow policy.
- A score of 3 means they could coach others. They build reusable templates, catch subtle errors and know when not to use AI at all.
Have two people score a handful of the same submissions and compare. If they disagree often, tighten the rubric wording until they agree. Report results by skill area for the team, and share individual results only with the person and their manager. The goal is a training plan, not a ranking.
Which metrics are misleading?
Some numbers are easy to collect and tempting to report, but they say little about skill.
- Login counts and messages sent measure activity. Heavy use can mean someone is skilled or that they lean on the tool for things it does badly.
- Self-assessment surveys measure confidence. Two people with the same skill can rate themselves very differently, so the numbers are hard to compare.
- Course completion measures attendance. Finishing a video series does not show that someone can catch a wrong figure in a report.
Use these as context, alongside the task-based results. A team with high usage and low scores on checking output is a team that needs help soon.
A simple four-week rollout plan
You can run a first cycle in a month with a small group, then expand.
- In week one, pick the two or three skill areas that matter most, write the scenario questions and the hands-on task, and agree on the rubric with one other reviewer.
- In week two, run the baseline with a pilot group of 8 to 15 people. Tell them clearly that results shape training and do not affect performance reviews.
- In week three, score the results, find the common gaps and run short, targeted sessions on those gaps using your own documents and tasks.
- In week four, repeat a parallel version of the assessment, compare results by skill area and decide what to change before rolling out to the next group.
After the first cycle, repeat the measurement every six months or whenever your tools or policies change. Add a practical outcome or two, such as how many AI errors reviewers catch before client delivery, so you can show the training is changing work, not just scores.
The Department of Labor framework also stresses learning in context and building human skills alongside AI ones, per the same summary. In practice that means training on your real files and workflows, and keeping judgment, writing and domain knowledge in the picture.
Frequently asked questions
How long should an AI skills assessment take?
About 30 to 45 minutes per person is enough for a useful baseline. That covers a short scenario quiz and one hands-on task. Longer tests tend to measure stamina and goodwill more than skill.
Should AI literacy results affect performance reviews?
For the first round, it is usually better to keep them separate. People answer honestly when the stakes are low, and honest baselines produce better training. Once expectations and training are in place, some teams add AI skills to role expectations, which is a decision to make with HR.
Do we need different assessments for different roles?
Keep the core the same and change the scenarios. Everyone needs to check output and handle data safely, but a sales rep's task might be a client email, while a finance analyst's task might involve a spreadsheet formula. Shared core areas let you compare results across the team.
Going further
If you want an outside, scenario-based check of everyday AI skills, the VeriLevel Practitioner certification covers instructions, evaluating output, everyday workflows, data and privacy, and responsible use in 35 work scenarios. It is an independent, unproctored online exam, not an accredited credential, and some managers use it as one input alongside their own task-based assessment.
