A girl in front of a library.

Thumbs up by Vinnova for Maria Ulan’s method for quality assessment

The doctoral student Maria Ulan has developed a method for measuring the quality of decisions based on complex data sets. On a doctoral student course, she pitched her method to an expert panel at Vinnova, Sweden’s innovation agency. The panel gave her two thumbs up and also wanted to test her tool in their own activities.

“It came as a surprise to me when one of the members of the Vinnova panel told me that they want to use my method. I wasn’t expecting that”, says Maria Ulan, doctoral student in computer and information science at Linnaeus University.

Doctoral student course with pitch

Ulan was one of the participants on the doctoral student course “Innovative applications of research and science”. The course deals with, among other things, development of ideas, research impact, law relating to agreements, and how to communicate your idea. To conclude the course, the participants got to pitch their idea to Vinnova.

“The idea of the pitching is that the doctoral students should get high-quality feedback from a financier who has an understanding of impact and utilisation”, says Mathias Gaunitz, innovation adviser at Linnaeus University.

“I really liked the course. It was great. My goal was to find new collaborators, and it went very well”, says Maria.

Multi-Criteria Decision Making

Ulan carries out research on software quality. More specifically, she studies multi-criteria decision making; decision making that involves many factors or properties to consider. An everyday example can be, for instance, that you want to buy a house and have many different options to choose between. What separates the alternatives from each other are properties like price, area, number of rooms, vicinity to shops and school, etc. The number of properties can be very large.

Ulan has developed a statistical method that aggregates all properties in a single measure, which results in a ranking of the alternatives, that is to say, all alternatives are given a number. This makes it possible to measure the quality of the different alternatives and the quality of your decision.

“This method is good if you want to reduce the number of properties within a certain field, in order to be able to make a good decision. It’s also good if you want to interpret your results”, says Ulan.

Vinnova gave two thumbs up

“When Ulan pitched her idea, I thought ‘we could make use of this in our activities’. We make a lot decisions on whether applications should be approved or not. We could use this to see if our assessments are of an adequately high quality”, says Jan Sandred, administrator at Vinnova.

Vinnova came to an agreement with Ulan that they will test her method on their call for proposals Innovativa Startups, where small, newly-established companies can apply for funding. For each round, Vinnova receives 600 applications that are to be assessed and graded based on different criteria, for instance, potential and feasibility.

"Naturally, all applicants should be treated the same way. However, you may be unlucky and end up with an assessor who values the grade 5 much higher than the grade 4, which will mean that you get a lower ranking. Or that an assessor gives low grades in general. Ulan’s tool will pay attention to this, and help us calibrate our tools”, says Sandred.

Great potential

Ulan was given all data from the grading assessment by Sandred and tested the data in her tool. The result showed that Vinnova’s assessment had been fair.

“We now know for sure; our assessments are of a high quality. This idea has great potential for anyone with complex data sets, where it is suspected that there are variations of which one is unaware. In such cases, this type of method is good to review the quality”, says Sandred.

Ulan’s collaboration with Vinnova has also resulted in a research report. Ulan sees more future areas of use for her method, for instance, in sports statistics or for the measuring of wine quality.

“When I talk to people from the trade and industry, I quite often hear that my research is too complicated. But not here! Vinnova realised the potential straight away. That feels great”, Ulan concludes.


The doctoral student course “Innovative applications of research and science” belongs to Inovationskontoret Fyrklövern and is offered to doctoral students at fur higher education institutions; Linnaeus University, Karlstad University, Mid Sweden University, and Örebro University.