In this cutting-edge episode of The Prompt Desk, your hosts Justin Macorin and Bradley Arsenault explore innovative techniques for testing AI agents using synthetic data and AI-generated test cases. Discover how to create testing agents that validate the performance of your main AI agent, handle edge cases, ensure proper conversation realignment, and improve overall reliability and robustness.---Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai (http://PromptDesk.ai) for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.me (http://bradleyarsenault.me)Add Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_link Hosted by Ausha. See ausha.co/privacy-policy for more information.

Explore the current landscape and future possibilities of AI agents with machine learning experts Justin and Brad. They discuss the trade-offs between specialized and general-purpose agents, the importance of domain expertise, and the potential for agent aggregators. The conversation also covers the standardization of agent-to-agent communication and how organizations may adopt and integrate these technologies.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai (http://PromptDesk.ai) for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.me (http://bradleyarsenault.me)Add Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_link Hosted by Ausha. See ausha.co/privacy-policy for more information.

As AI agents powered by large language models become more complex, developers often encounter erratic and unexpected behaviors during testing. From agents falling into infinite loops to models struggling with certain data formats, these issues can be tricky to diagnose and resolve. In this episode, Bradley Arsenault and Justin Macorin explore real-world examples of AI agents going off the rails. They discuss practical techniques like action governors, confusion matrix analysis, minimum task requirements, and targeted fine-tuning to create more robust and reliable agents. Tune in for valuable insights on taming unruly AI from two experienced practitioners at the forefront of prompt engineering and AI product development.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai (http://PromptDesk.ai) for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.me (http://bradleyarsenault.me)Add Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_link Hosted by Ausha. See ausha.co/privacy-policy for more information.

[Bonus] This is a bonus episode. We had too many unreleased episodes in our backlog, so we decided to give you an extra treat this week! Hope you enjoy it!Join the discussion on how AI agents can intelligently browse websites to retrieve specific information across links and pages. The hosts dive deep into techniques like embedding link text, URL paths, and accessibility attributes to compare against the desired information goal. They explore methods to filter and re-rank links through this embedding approach, accounting for factors like the total number of links per page and whether the target requires navigating across multiple sites.The conversation covers setting reasonable boundaries on recursive browsing, using a large language model for final relevance assessment, and the broader considerations of efficient agent design for web information retrieval tasks. With their combined 25+ years of AI engineering experience, the hosts provide insights into this emerging and complex challenge.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai (http://PromptDesk.ai) for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.me (http://bradleyarsenault.me)Add Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_link Hosted by Ausha. See ausha.co/privacy-policy for more information.

Gain a better understanding of completion vs. chat completion methods for large language models. Get the scoop on why chat completions are great for building chatbots quick, but can sometimes go off the rails. Learn how to keep your model on track with slick moves like tool completions and throwing a "governor" into the mix.---Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai (http://PromptDesk.ai) for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.me (http://bradleyarsenault.me)Add Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_link Hosted by Ausha. See ausha.co/privacy-policy for more information.

[Bonus] This is a bonus episode. We had too many unreleased episodes in our backlog, so we decided to give you an extra treat this week! Hope you enjoy it!Dive into the world of prompt engineering for large language models like GPT. This episode explores the important role that non-technical subject matter experts play in developing effective AI prompts and systems. The hosts discuss how domain experts across industries like food, real estate, and medicine are uniquely qualified to label training data, provide contextual knowledge, and design prompts that capture the right tone and style for their field. The hosts argue that while engineering expertise is needed for certain technical aspects, most prompts should be designed collaboratively with non-technical experts leading the way. With over 25 years of combined AI and product experience, the hosts share insights on bridging the gap between technical and non-technical teams for successful prompt engineering.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai (http://PromptDesk.ai) for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.me (http://bradleyarsenault.me)Add Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_link Hosted by Ausha. See ausha.co/privacy-policy for more information.

In this insightful episode, hosts Justin Macorin and Bradley Arsenault chat with AI expert Abi Aryan to dive deep into the challenges of measuring the business impact of large language model applications. Abby draws from her 8 years of experience deploying and productizing LLMs to provide a comprehensive framework for defining relevant metrics.She emphasizes the importance of bridging the gap between engineering and business objectives, highlighting that 76% of machine learning projects fail due to this disconnect. Abby shares practical advice on evaluating LLM performance through a combination of language skills assessment, task-specific metrics, human evaluation, and aligning with overarching business goals.The conversation covers key topics such as measuring the efficacy of retrieval versus generation components, incorporating user feedback beyond simple thumbs-up/thumbs-downs, detecting and mitigating hallucinations, and tying metrics to concrete business KPIs like sales funnels and revenue generation.---Please check out Abi on LinkedIn at https://www.linkedin.com/in/goabiaryan/Visit her website here: https://abiaryan.com/Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai (http://PromptDesk.ai) for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.me (http://bradleyarsenault.me)Add Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_link Hosted by Ausha. See ausha.co/privacy-policy for more information.

[Bonus] This is a bonus episode. We had too many unreleased episodes in our backlog, so we decided to give you an extra treat this week! Hope you enjoy!This episode explores the idea of making large language model interactions more transparent and immersive for users. The hosts discuss the potential of showing intermediate steps and outputs instead of just providing a final result. They envision an interface where the AI agent's process is visualized, such as displaying the generated search terms, rendered search results, visited web pages, and other intermediary steps. This approach aims to build user trust, provide insights for engineers to optimize the systems, and allow varying degrees of user control akin to management styles.The conversation highlights the need to move beyond the current "magic" of inputting text and receiving an output. By presenting the AI's workflow in a visually intuitive manner, users can better understand and even interact with the process. The hosts propose combining traditional UI elements like search bars and web page renderings with conversational AI interfaces. This transparency could pave the way for more advanced and reliable AI applications that meet users' needs while fostering trust through insight into the underlying mechanisms.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai (http://PromptDesk.ai) for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.me (http://bradleyarsenault.me)Add Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_link Hosted by Ausha. See ausha.co/privacy-policy for more information.

Did you immediately try out the GPT-4o model as soon as it came splashing across your news feed? If you were anything like your show hosts, you quickly realized that it's just one step in the ever-evolving world of AI. This realization might have sparked your curiosity about the future potential of language models and their impact on various industries.In the latest episode of The Prompt Desk, your hosts discuss how our expectations need to be recalibrated —Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai (http://PromptDesk.ai) for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.me (http://bradleyarsenault.me)Add Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_link Hosted by Ausha. See ausha.co/privacy-policy for more information.

Note! This is an episode that we found in our archive unreleased. It is one of the earliest episodes we ever recorded, but has so far gone not been listened to. It's not quite the same style as our current episodes, but we thought we'd release it anyway as a special treat for interested listeners!The podcast episode explores the realm of large language models, prompt engineering, and best practices, with a emphasis on discussing word embeddings and their applications. The hosts, Bradley and Justin, reminisce about their initial encounters with word embeddings and how they transformed the field of natural language processing.They then examine the advantages and potential drawbacks of word embeddings, highlighting their usefulness in various tasks such as retrieval, named entity recognition, and deduplication. The conversation also touches on practical tips and tricks for working with word embeddings, stressing the importance of vectorizing the right information and choosing the appropriate model for the task at hand.Throughout the discussion, the hosts underscore the need for product involvement and a thorough understanding of the problem at hand when deciding how to best utilize word embeddings.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai (http://PromptDesk.ai) for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.me (http://bradleyarsenault.me)Add Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_link Hosted by Ausha. See ausha.co/privacy-policy for more information.