Agents for research & patient simulation benchmarking

There’s a lot of exciting stuff to go through this month. It was way better than the gloomy month that was september. Kicking off the month with D3A helped tremendously with motivation.

Participation in D3A

I had prepared a small project and poster to D3A which was accepted. Apparently there was only a ~65% acceptance rate compared to the 95-100% of earlier years. I was a little bit nervous about the project as it was done somewhat hastily just to have something to show. But it turned out to be a really good experience!

There was genuine interest in what I had worked on which essentially just opened the conversation to a bunch more questions.

The conference itself was also just very nice. I got the sense that it’s many of the same people attending each year which makes it a very good place to keep up with those specific people. This year I also took the opportunity to skip a few lecture sessions to have more informal talks in the hallways which, as we know, is where the real action is.

It is said that there are three stages to a PhD. In the first stage you look at a related paper’s reference section and you haven’t read most of the papers. In the second stage you recognize all the papers. In the third stage you’ve shared a beer with all the first authors of all the papers.

Agents for research projects

I’ve been trying to lean more into using agents to solve small problems I have here and there. A core challenge I have noticed is that “vibe coded” research projects can very easily spiral out of control and it can be a struggle to point the agent back to the core of the project. In an attempt to make it easier for me to work with agents as a collaborator I set up a research agent kit that contains all the bits and pieces I would like an agent to have while it helps me conduct research projects.

A few of its core features include

Dogfooding this research agent kit project in actual research projects allows me to iterate and optimize the agentic research assistant experience.

I find it really pleasant to work with so far. I find the “free” documentation especially beneficial. By having the agent keep a decisions.md and write thorough issues, PRs and comments I can easily track work throughout the project. This is way better for me as well as for parallelizing work as all context is saved within the GitHub repo and not enclosed to a single agent’s chat history.

Patient simulation benchmarking

One of the first features I worked on after joining Laerdal was the Conversational AI in one of our virtual simulation solutions. Simply put, it allows you to have a voice-first experience wherein you talk to a simulated virtual patient instead of navigating options in a menu. This is in our Virtual Reality solution, vrClinicals for Nursing, allowing for higher fidelity conversations and better learning outcomes in terms of patient communication.

When I joined we had early versions of this with GPT-3.5 which demonstrated the potential but failed in weird ways. For example, it was incredibly easy to switch the persona such that the LLM all of the sudden switched to being the healthcare worker and not the patient. This was not acceptable for the product and we could not ship it but it was fine for development purposes. The launch of GPT-4o fixed this from one day to the other. It was way better at instruction following and stayed in character as the given patient much better than GPT-3.5. This was my first encounter of “strategic procrastination” working out when working with LLMs for products.

Employ some “strategic procrastination” here, build what you absolutely need, and await the obvious expansions to capabilities from providers. (https://applied-llms.org/)

Now that I’m working with on-device deployment of AI a natural continuation would be to implement this conversational AI system so it can work offline. This would be good to avoid per-session costs and ensuring users’ privacy. A few use-cases also require that is works without internet access for which local AI is the only solution.

I built a prototype rather quickly with Claude Code (Opus 5.5) using Transformers.js as I knew this would be quick and easy. It was promising but one requirement made the project halt for a bit: It had to run on an iPad. I tested the prototype on Safari on my M2 iPad Air and it simply didn’t load. Why was that? Turns out there is a limit for how much memory a single tab can load in iPadOS. This significantly limits which models I’m able to use for the application and their capabilities. Trying out those smaller models on this “patient roleplaying”-task was highly reminiscent of the GPT-3.5 days as they all failed in odd ways. So I decided to pose the question: What’s the smallest model that can deliver acceptable performance as a patient persona?. This turned out to be quite hard to answer - which is good! It makes for interesting research in capabilities of small model.

On-prem Qwen 3.8 27B

I continued to work on the deployment of Qwen 3.8 27B on our desktop PC with an RTX Pro 4000 Blackwell GPU. That being said, I practically just instructed Claude to “Set up vLLM with Qwen 3.8 27B. Be sure to use speculative decoding. Report back tokens per second and benchmark results.” and after a while it reported a few tables and had configured the whole thing.

Next step is to onboard developers and designers. We have just changed office so getting things set up takes a little longer than anticipated but it’s getting there. I had the pleasure of presenting what I had done in an internal “AI Coding Guild” meeting that is held regularly.