Unlocking the Potential of Assessments
Join Questionmark on our quest to discover and examine the latest in best practice guidance with a wide array of guests - including assessment luminaries, industry influencers, SMEs and customers - and discuss "all things assessment."
Unlocking the Potential of AssessmentsNov 29, 2023
EdTech Revolution: Rethinking Learning Models with Carla Aerts
Join host John Kleeman in an insightful conversation with Carla Aerts, a trailblazer in the reinvention of education. Carla, with an unconventional journey from the performing arts to education technology, shares her accidental foray into the sector and delves into the importance of making impactful contributions to teaching and learning.
Carla explains what "perfect EdTech" looks like to her and critiques traditional exam-style assessments, advocating for a shift towards formative assessments and diverse models that prioritize skills, collaboration, and social engagement.
The discussion extends to the role of AI in education and Carla's predictions of how it will impact the future. The episode concludes with three key takeaways: the significance of social learning, the need for skills development in reflection and metacognitive thinking, and the promotion of learner agency for effective lifelong learning.
Tune in for a thought-provoking exploration of education, technology, and assessments, guided by Carla Aerts' forward-thinking perspectives.
Workday: Being Ready for What’s Next - Skills as Strategy
John Kleeman welcomes Sonny Yuen, Senior Director of Product Strategy at Workday, to discuss the value of assessments in the world of e-learning and corporate learning. Sonny's impressive background includes roles at major e-learning companies, and he shares insights into the field.
Together, they delve into the significance of assessments in the learning process and how assessments are integral to verifying competence and capabilities. The discussion covers Questionmark's integration with Workday, AI's role in learning, recommendations for setting up successful learning programs, and the importance of diversity, equity, and inclusion in learning.
Sonny emphasizes the need to involve stakeholders and suggests that organizations allocate dedicated time for self-elective learning to foster professional development. Listen now to explore the evolving landscape of corporate learning and how technology plays a crucial role in optimizing the learning experience.
Interested in learning more about us? Talk to us at questionmark.com
Ep. 39 – Assessment Luminary, John Weiner, Lifelong Learner Holdings
Host John Kleeman is joined by John Weiner, an expert in the assessment industry with four decades of experience. Until recently, he was Chief Science Officer at PSI, and now has the same role at its parent company, Lifelong Learner Holdings.
They explore the evolution of assessments from paper-based to technology-driven, highlighting shifts in workplace demands and educational needs. John shares his thoughts on fairness and its multiple dimensions, along with the challenges in measuring and ensuring it. The conversation dives into the impact of generative AI on assessments and the importance of critical thinking.
He also shares his predictions of the future of the workforce as technology reshapes roles. Tune in for insights into the changing landscape of assessments and the exciting possibilities that lie ahead!
Interested in learning more about us? Talk to us at questionmark.com
Ep. 38 – Assessment Luminary, Mark Lynch, Learnosity
Host John Kleeman is joined by Mark Lynch, co-founder of Learnosity, a leading assessment technology company. Mark shares his accidental entry into the field of education and assessment, starting with his work on an assessment engine for the New South Wales Board of Education. He discusses the growth of Learnosity, highlighting their focus on solving hard problems and their early success in delivering highly accessible assessments.
Mark emphasizes the importance of tackling the hardest problems in assessment, as solving them paves the way for addressing other challenges. He also talks about Learnosity's evolution and their work with governments and schools worldwide to deliver equitable assessments.
The conversation delves into the future of assessment, particularly the role of AI in personalization and content creation. Mark acknowledges the potential of AI to improve accessibility and productivity in assessment, but also highlights the challenges associated with evolving standards and conflicting accessibility needs.
He offers advice for those starting out in ed tech and much more!
Interested in learning more about us? Talk to us at questionmark.com
Ep. 37 – Assessment Luminary, Matt Wingfield, e-Assessment Association
We had the privilege of hosting Matt Wingfield, a renowned expert in the field of assessments and a Chairman of the e-Assessment Association. Speaking with our host, John Kleeman, Matt shares valuable insights into the world of e-assessment and its implications for education, certification and professional development.
From an overview of the e-Assessment Association and its mission to promote and advocate for high-quality e-assessment practices worldwide to the growing importance of e-assessment in various sectors, he highlights its ability to enhance learning outcomes, improve efficiency and provide reliable and timely results. While privy to the potential concerns related to e-assessment- including issues of security, fairness and accessibility- Matt emphasizes the importance of a robust infrastructure, data protection and ethical considerations to ensure a trustworthy and inclusive assessment environment.
He delves into the future of e-assessment, exploring the potential of artificial intelligence and machine learning in enhancing assessment practices. Matt envisions a future where e-assessment plays a central role in continuous learning and professional development, enabling individuals to showcase their skills and knowledge throughout their careers.
Interested in learning more about us? Talk to us at questionmark.com
Ep. 36 - Assessment Luminary, Maria Hamdani, Center for Measurement Justice
Maria Hamdani, Vice President of Assessment and Research at the Center for Measurement Justice, joins John Kleeman to discuss the role of assessments in promoting equity and justice in education and employment. She shares her insights on how assessments can be designed and administered in a way that is fair and unbiased, and how they can be used to identify and address systemic barriers to educational and professional success.
Maria highlights the importance of including diverse perspectives and voices in the development of assessments, and the need to continuously review and update them to ensure they remain relevant and effective. She also discusses the challenges of balancing the need for standardization and consistency in assessments with the need to accommodate individual differences and contextual factors.
Maria emphasizes the critical role that assessments can play in advancing social justice and equity, particularly in the context of medical education and licensure, and encourages educators and employers to embrace a holistic and collaborative approach to assessment design and implementation, one that is grounded in principles of fairness, transparency, and inclusivity.
Want to know more about us? Talk to us at questionmark.com
Ep. 35 - Assessment Luminary, Norihisa Wada, EduLab
John Kleeman is joined by assessment luminary, Norihisa Wada, executive adviser of EduLab in Japan, and director of the Japan Institute for Educational Measurement. Nori has experience with AI-driven voice recognition and has worked with Carnegie Mellon University and Kyoto University. He is also involved in EdTech investment. Delve into his insights on assessment and his background in the industry. Nori shares how he transitioned from his work on game development for consumer educational games and fitness games for Nintendo to the assessment industry, the future of assessments and gives advice on how assessment organizations can be more inclusive.
Ep. 34 - Assessment Luminary, Dr. Gary A. Gates, Pearson VUE
John Kleeman is joined by assessment luminary, Dr. Gary A. Gates, the Managing Director of Pearson VUE. Originally a nuclear physicist, Dr. Gates details his journey into the world of assessments. He discusses his key priorities going into his second year leading Pearson, the future of test centers during the rise of online proctoring and whether new AI technology is harmful or helpful to proctoring and assessments.
Ep. 33 - Assessment Luminary, Mike Olsen, Proctorio
John Kleeman is joined by assessment luminary, Mike Olsen, the CEO and Founder of Proctorio. Mike discusses what drove him to create a company focused on proctoring, how it has expanded beyond the education sector, and how the tool can be used to address test fraud.
Ep. 32 - Assessment Luminary, Phillip Dawson, Deakin University
John Kleeman is joined by assessment luminary, Professor Phillip Dawson, the Associate Director of the Centre for Research in Assessment and Digital Learning (CRADLE) at Australia's Deakin University. With an academic background in education, artificial intelligence and cybersecurity, he leads CRADLE's work on cheating, academic integrity and assessment security. This work spans hacking and cheating on online exams, training academics to detect contract cheating, student use of study drugs, the effectiveness of legislation in stopping cheating and the evaluation of new assessment security technologies. Phillip shares his thoughts on constructing an assessment that gives feedback, reducing test cheating and the privacy concerns associated with remote proctoring.
Ep. 31 - Assessment Luminary, Thomas Goettsche, SAP
John Kleeman is joined by assessment luminary, Thomas Goettsche, SAP Certification Principal Production and Psychometrics Consultant. A trained psychologist, he has designed key performance indicators, e-learning programs, best practice standards and much more over the course of his career. Thomas identifies the intrinsic value of certifications, his contributions as a member of SAP's development team and explains the ins and outs of standard setting.
Ep. 30 - Assessment Luminary, Dylan Wiliam, University College London
John Kleeman is joined by assessment luminary, Dylan Wiliam, Emeritus Professor of Educational Assessment at University College London. Dylan's academic work is focused on the use of assessment to support learning. A leader in formative assessment, he explains that testing can go beyond just measuring a student's learning as it can also be used to improve it. Dylan also shares his thoughts on test validity, online learning and the college admissions testing controversy.
Ep. 29 - Assessment Luminary, Isabel Nisbet, University of Cambridge
John Kleeman is joined by assessment luminary, Isabel Nisbet, Affiliated Lecturer at the University of Cambridge. Isabel discusses the two fundamentals of fairness, the importance of public confidence in educational assessment and shares her thoughts on the steady push for test-optional college admissions in the US.
Ep. 28 - Assessment Luminary, Will Thalheimer, TiER1
John Kleeman is joined by assessment luminary, Will Thalheimer, Principal at TiER1. Will details his 30-year journey to becoming a learning expert, which started as an instructional designer, trainer, and simulation architect. He explains which key findings from learning science we should put into practice, the four pillars of learning effectiveness, and the best way to evaluate how effective learning is.
Ep. 27 - Assessment Luminary, Siddhartha Gupta, Mercer | Mettl
John Kleeman is joined by assessment luminary Siddhartha Gupta, CEO of Mercer | Mettl. Over the course of his career, Sid held key leadership positions at some of the most prestigious global IT companies. It was during that time that he discovered the niche of assessments and eventually moved to Mercer | Mettl. Sid shares his opinion on technology and bias, assessment innovation, and Mercer | Mettl's partnership with Questionmark for the upcoming Bring Your Own Proctor feature.
Ep. 26 - Assessment Luminary, Suzanne Heiob, SAP
John Kleeman is joined by assessment luminary Suzanne Heiob, the Global Certification Product Owner of SAP, who began her journey with the company as a student. Among her responsibilities, she oversees the product roadmap and keeps an eye on market trends to ensure that they are adopted into the program. Suzanne shares her thoughts on the best practices and future of certification programs, along with discussing international challenges, the computer translation capability co-innovation with Questionmark, and the ITCC award it recently won.
Ep. 25 - Assessment Luminary, Andre Allen, FifthTheory
John Kleeman is joined by assessment luminary, Andre Allen, Managing Member and Executive Vice President of FifthTheory, who has over 23 years of continuous experience in the test publishing industry. Andre has pioneered many of the technologies that are the basis for many of the assessment delivery and scoring options for today. Andre shares how he got into assessment and discusses how you can assess leadership, admissions testing and how we can do better around diversity in testing.
Ep. 24 - Assessment Luminary, Linda Anguish, SPHR, Institute for Credentialing Excellence (ICE)
John Kleeman is joined by assessment luminary, Linda Anguish, SPHR, Director of Accreditation Services at the Institute for Credentialing Excellence (ICE). She is responsible for the accreditation programs at ICE including the NCCA, ACAP and ICE’s newest accreditation program for ISO 17024 compliance. Linda shares how she got into assessment and credentialing and discusses the advantages of credentialing, the ICE certification program, diversity and the future of certification.
Ep. 23 - Assessment Luminary, Stephen Sireci, UMass Amherst
John Kleeman is joined by assessment luminary, Stephen Sireci, Professor in the Psychometrics Program and Director, Center for Educational Assessment at University of Massachusetts Amherst. His work is well known in the world of assessment. Steve shares how he got into assessment and discusses his research around validity, the five sources for validity evidence and his good practice advice.
Ep. 22 - Assessment Luminary, Jarrod Morgan, Meazure Learning
John Kleeman is joined by assessment luminary, Jarrod Morgan, Founder of ProctorU (now known as Meazure Learning) and Chief Strategy Officer. Jarrod has been a leader in the educational technology field for over 16 years. Jarrod shares how he got started in assessment and discusses all things proctoring, including where he sees proctoring going, good practices in remote proctoring and the human first approach.
Ep. 21 - Assessment Luminary, Marten Roorda, Riiid Labs
John Kleeman is joined by assessment luminary, Marten Roorda, Chief Measurement and Learning Officer at AI and education company, Riiid Labs. Marten is a pioneer in educational technology, a leading innovator in computer-based testing, e-learning and adaptive learning and has an extensive background, which includes ACT, CITO and more. Marten shares how he got started in assessment and discusses how to use AI in assessment, adaptive learning and his best practice advice.
Ep. 20 - Assessment Luminary, Alina von Davier, Duolingo
John Kleeman is joined by assessment luminary, Alina von Davier, Ph.D, Chief of Assessment at Duolingo. At Duolingo, Alina and her team operate at the forefront of computational psychometrics. Alina shares how she got into assessment and discusses how to make a high stakes test equitable with multiple instances, her experience as a female leader in the assessment space and her advice to people who want to progress in assessment.
Ep. 19 - Assessment Luminary, Riz Khan, SAP
John Kleeman is joined by assessment luminary and "Learning Enthusiast," Riz Khan of SAP. Riz is Global COO and Vice President of Training and Adoption at SAP and frequently partners with CxOs, executives and leadership groups to help them understand the importance of corporate education and the power of user adoption. Riz shares his background in assessment and learning and discusses the value of investing in learning and training and how assessments fit in, his best practice advice and the future of learning and adoption.
Ep. 18 - Assessment Luminary, Gavin Cooney, Learnosity
John Kleeman is joined by assessment luminary, Gavin Cooney, CEO and co-Founder of Learnosity, the leading provider of assessment infrastructure for the learning industry. As you may have heard Learnosity recently acquired Questionmark. Gavin discusses how he got into assessment and dives into various topics, including the importance of accessibility and inclusivity in education, the trends he's noticed and how Questionmark and Leanosity fit together.
Ep. 17 - Assessment Luminaries, Sharon Shrock and William Coscarelli
John Kleeman is joined by assessment luminaries, Sharon Shrock and Bill Coscarelli, co-authors of award winning book "Criterion-Referenced Test Development." Sharon and Bill discuss how they got into assessment and tell the story of their book. The pair also shares patterns in testing they've seen working with various organizations and their advice.
Ep. 16 - Assessment Luminary, Dr. Leanne Furby, National League for Nursing
John Kleeman is joined by assessment luminary, Dr. Leanne Furby, Director of the Assessment Services Division at National League for Nursing. Leanne discusses how she got into assessments, talks about the assessment program at NLN and remote proctoring during COVID-19 and explains spaced learning and how it helps.
Ep. 15 - Assessment Luminary, Akil Bello, FairTest
John Kleeman is joined by assessment luminary, Akil Bello, who is an expert in assessment reform. He currently serves as the Senior Director of Advocacy and Advancement at FairTest, the National Center for Fair and Open Testing. Akil discusses how he got interested in assessments and talks about his work with admissions tests, whether recruitment tests are fair and his thoughts on what the testing industry should be doing better.
Ep. 14 - Assessment Luminary, Helen Baron, Psychometric Test Development
John Kleeman is joined by assessment luminary, Helen Baron, who is an expert in psychometric measurement and assessment. Helen works with test publishers and employers on developing tests and assessment processes. Helen discusses how she got into assessment, and dives into her time at SHL, the different types of assessments used in recruiting and where personality tests come in and her work on assessment standards.
Ep. 13 - Assessment Luminary, Wendy Kirkpatrick, Kirkpatrick Partners
John Kleeman is joined by assessment luminary, Wendy Kirkpatrick, President and Founder of Kirkpatrick Partners. As a global driving force of the use and implementation of the Kirkpatrick Model, Wendy leads companies to measurable success through training and evaluation. Wendy discusses how she got into training and assessments, and dives into the four levels of evaluation, why level three is the most important and how to measure behavior and her tips for evaluating training while remote working.
Ep. 12 - Assessment Luminary, Chris Norton, Mentor Group
John Kleeman is joined by assessment luminary, Chris Norton, Director at Mentor Group, a global sales transformation company based in UK. Chris shares his background and experience with assessments and discusses the value and best practice advice on using assessments in the sales space, the new Questionmark B2B Sales Qualification by Mentor Group and the future of digital selling.
Ep. 11 - Assessment Luminary, Bernt Nilsen, Datakortet
John Kleeman is joined by assessment luminary, Bernt Nilsen, founder of Datakoret (ECDL Norway) and Norsk Test, the leading test developer and supplier of national certification programs in Norway. Bernt discusses how he got into the assessment world and shares what's important when running a testing service business, his best practice advice for creating high-stakes assessments and the value in anonymous testing.
Ep. 10 - Assessment Luminary, Alan Thiemann, Attorney and General Counsel to ATP
John Kleeman is joined by renowned attorney, Alan Thiemann of Hans Santos, PLLC and General Counsel to the Assocation of Test Publishers (ATP). Alan discusses how he got into testing and his role at ATP and dives into topics such as privacy and remote proctoring, how to avoid being taken to court for a bad test and testing controversies in the US.
Ep. 9 - Assessment Luminary, Liberty Munson, Microsoft
John Kleeman is joined by assessment luminary, Liberty Munson, who is the Senior Psychometrician for Microsoft. Liberty discussess how she got into assessments, why people stay in jobs, the value of certifications and the future assessments, among other topics. Liberty also shares her aha! moment - listen to hear more!
Ep. 8 - Assessment Luminary, Jim Holm, Examity
John Kleeman is joined by assessment luminary, Jim Holm, CEO of Examity, the leading solution for learning validation and test security. Jim discusses how he got into his current role and the challenges he's faced with COVID-19, and dives into remote proctoring, including the value, his best practice advice, the future of proctoring and his views on artificial intelligence and proctoring.
Ep. 7 - Assessment Luminary, Paul Crump, Cambridge Assessment
John Kleeman is joined by assessment luminary, Paul Crump of Cambridge Assessment, the non-teaching department of the University of Cambridge that develops and delivers examinations and tests around the world through their three exam boards. Crump discusses his background in assessments and dives into the Thinking Skills testing initiated by Cambridge Assessment and his best practice advice for those creating questions and assessment.
Ep. 6 - Assessment Luminary, Sue Martin, Certification Expert
Host, John Kleeman is joined by assessment luminary, Sue Martin, a pioneer in the assessment industry responsible for implementing the first test center network in Europe. Sue shares her story and advice on transforming certification programs. Discover the five most important considerations when setting up your certification program.
Ep. 5 - Assessment Luminary, William G Harris, ATP
John Kleeman is joined by assessment luminary, William G Harris, Chief Executive Officer of the Association of Test Publishers (ATP), the leading international organization for publishers of assessment programs. G Harris discusses his background in assessments and dives into his advice for those starting out in assessments, the importance of validity in assessments and the different sets of standards for testing, among other topics.
Ep. 4 - Assessment Luminary, Dave Foster, Caveon
John Kleeman is joined by assessment luminary, Dave Foster, CEO of Caveon, the first test security company. Dave is a psychologist, psychometrician and recognized luminary in the measurement industry. In this episode, Dave discusses his background in assessments and dives into the main risks and threats of testing, his recommendations on the key measures you should take with test security and how statistical analysis fits into this, among other topics.
Ep. 3 - Assessment Luminary, Steve Dept, cApStAn
John Kleeman is joined by assessment luminary, Steve Dept, CEO and founding partner of cApStAn, a high-profile language service provider. Steve received his education in English, Dutch, French and German but he is essentially an autodidact and a field practitioner. In this episode, Steve discusses his background in assessments and dives into good practice advice for translating tests, including the translation process, translation technology and other translation resources.
Ep. 2 - Assessment Luminary, Steve Lay, Questionmark
John Kleeman is joined by one of Questionmark's product managers, Steve Lay. Steve is an expert on scalable, computerized assessment and integration between systems, and has ensured that the million of assessments delivered by Questionmark are robust and scalable. He is also active in the IMS and other e-learning standards groups. In this episode, John dives into many topics with Steve including the impact of technology on the assessment process, QTI and LTI and proctoring.
Ep. 1 - Assessment Luminary, Jim Parry, Compass Consultants
Host, John Kleeman is joined by assessment luminary, Jim Parry, who has over 40 years of experience as a course designer, developer and instructor. Jim discusses his 12-year tenure at the Coast Guard as a civilian employee as the Test Development and e-testing Manager, during which he guided the move from paper to online testing for the entire Coast Guard.