Degreed Experiments with Emerging Technologies

Taylor Blake

Thursday, April 18th, 2024

Share

Honestly, I don’t care much for change. I’d rather be at Blockbuster on a Friday night than sifting through Netflix recommendations. But I do care a lot about how we learn at work. So, for the past year, my mind has been racing thinking about the impact of all these emerging technologies for L&D. What will survive past the hype? How do we use it to make a real impact and not just create cheap imitations? 

We’ve been testing and discussing these topics inside and outside of Degreed. But, we’d like to open up the conversation. 

That’s why we’re excited to announce a new initiative, Degreed Experiments. This will be a blog series (and hopefully a two-way conversation) geared toward exploring the suitability of emerging technologies for the challenges in L&D. Through hands-on prototyping, we’ll share with you what works, what doesn’t, and new questions we encounter along the way. Through it all you’ll become a more informed, and credible partner who can help your business evaluate emerging technologies and opportunities. 

Why Degreed Experiments?

We’ve been building learning products for a long time, but the way we build products is changing. With the incorporation of AI, product experiences are less predictable and deterministic. You don’t really know what an AI-powered product will be like until you can play around with it.

Combine that unpredictability with the future of work and the evolving role of L&D and we have a dynamic that requires rapid experimentation and iteration.

Now, we see all the AI chatter and don’t want to contribute to the noise. We’re determined to give you actual data and examples that can only come from trial and error.

What You Can Expect: Hands-on Prototypes

First, we’re builders. That means there will be more showing as we demo hands-on prototypes, critically evaluate the outputs, and let you judge the results for yourself.

In future posts, we’ll deep dive into emerging technology use cases like:

Upgrading Practice Scenarios from Chat to Live-speech Conversations

Our Hypothesis: Practice scenarios are a powerful way to develop skills in context. Having the option to not only practice in a chat-based scenario but also in a live conversation (like you’re really talking to someone) will make the scenario feel more real. This could be more effective for use cases like sales calls, interviewing, and executive coaching.

Skill Inference from Employee Data

Our Hypothesis: Leveraging existing employee data and activity can help you quickly identify skill strengths and gaps resulting in better data coverage, more up-to-date profiles, and should be good enough for some use cases (though not all). We also expect the details on how this is done to matter a lot.

Dynamic Taxonomies for Skills, Tasks, or Other Requirements

Our Hypothesis: Organizing skills, people, and work quickly and flexibly (versus rigid and traditional taxonomies) can speed up building learning experiences, identifying talent, and mapping career paths.

Smart, Mobile Nudges and Content Delivery

Our Hypothesis: Lots of your existing content could be reformatted into mobile-friendly nudges to better optimize engagement and retention. The micro-learning concept has been overplayed, but we think there are new ways to approach this with emerging technologies.

Identifying Internal Talent

Our Hypothesis: We need dynamic ways to match people to internal opportunities. This may include looking at someone’s skills, experience, working relationships, or other key attributes. Those doing the evaluation should be able to change the criteria used for matching as needed to find the best fit.

Building Personalized Learning Experiences

Our Hypothesis: Learning is more than content. We can design and personalize experiences that are optimized for closing performance gaps.

Side-by-side Model Evaluation

Our Hypothesis: Comparing models and methodologies side-by-side will give us the ability to identify the best fit for any use case.

Reviews of the Latest Research on Emerging Technologies

AI is advancing rapidly and chaotically. We’re going to explore the latest technical research and evaluate its impact on L&D. 

Recent and dramatic advancements in emerging technologies will open new use cases for learning and development. 

• Multimodal: Images, audio, video, and avatars will open new doors (and traps) for content creation and delivery.
• Evolution of RAG (Retrieval Augmented Generation): The ability to search, summarize, or ask questions from your own source documents are huge use cases. We’re seeing rapid advancements in RAG methodologies, longer context windows, methods for extending memory, and new architectures (beyond transformers used in models like ChatGPT) that will empower knowledge management.

• Agents: As AI models get more similar and competitive, the next frontier will be the use of agents that can perform chains of actions (including planning and evaluating the work done) using various models. This will open the door for even more automation.

• Screen vision: Researchers are exploring AI assistants that can see and act on what’s on a screen, regardless of whether any formal integration between applications exists. This could transform on-the-job training and performance support.
• Wearables: New AI-wearables are just emerging but could inspire more ways to support deskless workers.

In addition, non-obvious but important limitations exist. While these can be managed or mitigated, it’s still crucial to understand them.

• Factuality: AI models still struggle with factuality when creating long-form content and providing accurate citations.
• Long-form reasoning: While AI models do a great job at finding self-contained pieces of information, they struggle to reason across long contexts.
• Following instructions: Even the most advanced models still don’t fully follow instructions a third of the time.
• Unpredictability: Not all models are well suited to all tasks. Outputs from models change, and the models themselves are frequently updated in unpredictable ways.
• Cost: As usage scales, cost will be an important factor. Vendors will look to make tradeoffs between capabilities and cost or pass the cost onto clients.

Finally, there are more strategic implications.

• Are you prepared to help navigate career mobility when AI disrupts the viability or capacity of certain roles?
• What functionality should you get from new vendors vs. waiting for it to be incorporated into your existing applications?
• How will new regulations and audits affect which capabilities you can take advantage of?
• Which data will AI have access to, what is the data quality, and will it increase visibility to sensitive or misclassified data?
• Will there be responsibilities that AI displaces from L&D?
• What is required to build AI literacy in your organization?

See, I told you there’d be lots of questions. So, follow along and we’ll see if we can answer them together.

What’s In It For You

Hopefully, a lot. We’ll let you judge for yourself the suitability of the emerging technologies for your use cases and organization. We’ll make it easy to stay up to date with all the advancements. We’ll help you become a better partner for your business as your colleagues ask similar questions internally. And hopefully we can even have some fun along the way!

How to Get Involved

Send us ideas or questions. We’d love to hear what’s on your mind. Email tblake@degreed.com.

Volunteer to take our experiments for a test run. We’ll pick a few partners to help test and provide feedback on each prototype. Let us know which topic has piqued your interest by emailing tblake@degreed.com

Follow us on LinkedIn to catch our next posts, which will be coming soon.

Thanks, all! We’re excited to see what we learn together.

Watch an on-demand session with AI Guru, Noelle Russell, discussing the future of AI.