AI Test Engineer- BU

At Lilly, we unite caring with discovery to make life better for people around the world. We are a global healthcare leader headquartered in Indianapolis, Indiana. Our employees around the world work to discover and bring life-changing medicines to those who need them, improve the understanding and management of disease, and give back to our communities through philanthropy and volunteerism. We give our best effort to our work, and we put people first. We’re looking for people who are determined to make life better for people around the world.

Department: International Business Unit (IBU) IBU Testing Center of Excellence (CoE)

Job Overview:

Eli Lilly and Company is seeking a highly skilled and motivated AI Tester to join the IBU Testing Center of Excellence (CoE) team under IBU. As part of our commitment to advancing innovative AI technologies within the pharmaceutical industry, this role will be integral to the testing and quality assurance of AI-based solutions and platforms that will be deployed across various business functions in 2025. The AI Tester will work collaboratively with data scientists, engineers, product teams and IBU Test Manager to ensure that all AI models, algorithms, and associated software meet the highest quality standards and perform as expected in real-world applications.

This is a unique opportunity for a candidate passionate about AI technologies to shape the future of testing within the pharma industry while contributing to AI-driven solutions that impact the health and well-being of people worldwide.

Key Responsibilities:

AI Testing Strategy and Planning

• Collaborate with cross-functional teams to develop comprehensive AI testing strategies and plans for AI-powered applications.
• Work closely with product managers, data scientists, and developers to understand AI model requirements, use cases, and project goals.
• Define the scope and objectives of AI testing efforts, including performance, accuracy, bias detection, and robustness of AI models.

Test Execution for AI Models and Algorithms

• Design, develop, and execute test cases for AI systems and models (including machine learning and deep learning algorithms).
• Test and validate AI solutions across various stages of the development lifecycle, including model training, testing, and deployment.
• Ensure that AI models meet business requirements and perform accurately under various real-world conditions.
• Evaluate the performance of AI models by assessing speed, efficiency, scalability, and resource utilization.
• Perform manual and automated testing on AI-based applications, platforms, and solutions.

AI Model Accuracy and Validation

• Test AI models for accuracy, precision, recall, F1 score, and other performance metrics.
• Ensure AI models' fairness by conducting tests for potential bias in decision-making processes, especially in clinical or medical applications.
• Validate AI model predictions against real-world data, ensuring that results are consistent, reliable, and actionable.

Collaboration and Knowledge Sharing

• Work with data scientists, AI engineers and Test Manager to improve testing methodologies and continuously optimize AI model testing processes.
• Provide feedback on AI models, pointing out any potential improvements in testing coverage or areas for model retraining.
• Communicate findings, bugs, and issues related to AI models to technical teams, ensuring prompt resolution.

Test Automation for AI Projects

• Develop and implement automated testing scripts and frameworks specifically designed for AI applications.
• Utilize AI testing tools and frameworks (RAGAS etc.) to automate the validation of AI models and algorithms.
• Integrate automated AI testing within continuous integration and continuous deployment (CI/CD) pipelines.

Compliance and Regulatory Testing

• Ensure that AI applications comply with industry-specific regulations, especially in the pharma and healthcare sectors (e.g., FDA regulations, HIPAA compliance).
• Verify that all AI-driven processes adhere to ethical standards and data privacy laws.

Continuous Improvement and Research

• Stay up-to-date with the latest trends, tools, and techniques in AI testing and apply these advancements to optimize the testing process.
• Participate in AI testing forums and workshops, contributing insights to improve best practices within the team.

Reporting and Documentation

• Document test results, methodologies, and issues clearly, providing insights into test coverage, risk analysis, and performance benchmarks.
• Prepare detailed reports for both technical and non-technical stakeholders, summarizing testing outcomes and potential risks associated with AI implementations.
• Assist in the creation and maintenance of knowledge-sharing platforms related to AI testing best practices.

Key Skills and Qualifications:

Technical Expertise

• Strong knowledge of AI/ML testing methodologies and best practices.
• Experience with AI development frameworks and libraries such as TensorFlow, Keras, PyTorch, scikit-learn, RAGAS and MLlib.
• Proficient in testing tools and environments for AI-based systems (e.g., Jupyter Notebooks, Apache Spark, and DataRobot).
• Experience with performance testing tools like Grafana and JMeter for AI solutions.
• Knowledge of Python, R, JavaScript or other programming languages frequently used in AI/ML.
• Understanding of test automation frameworks and experience in tools like  Cypress, and JUnit for automating AI tests.

AI Model Evaluation

• Solid understanding of machine learning and deep learning models, including supervised and unsupervised learning techniques.
• Familiarity with evaluating AI models on metrics such as accuracy, precision, recall, F1 score, confusion matrices, and AUC.
• Ability to identify and test for model biases, fairness, and ethical implications, especially in sensitive applications like healthcare and pharma.

Analytical and Problem-Solving Skills

• Strong problem-solving abilities and keen attention to detail, with a systematic approach to diagnosing and resolving AI-related issues.
• Ability to perform root cause analysis of issues in AI algorithms and suggest actionable fixes.

Collaboration and Communication

• Excellent teamwork and communication skills, with the ability to collaborate with cross-functional teams, including data scientists, engineers, and product managers.
• Strong verbal and written communication skills to convey technical information clearly and concisely to both technical and non-technical stakeholders.

Experience

• Minimum of 8 years experience in software testing, with at least 2 years focused on testing AI/ML models or AI-based applications.
• Proven experience in testing AI/ML algorithms in production or staging environments.
• Experience working in a regulated industry (such as pharmaceuticals or healthcare) is a plus.

Preferred Qualifications:

• Experience with cloud platforms (e.g., AWS, Azure) for deploying AI applications and models.
• Familiarity with DevOps practices and integrating AI testing into CI/CD pipelines.
• Certification in AI/ML or related testing frameworks (e.g., Certified Software Test Professional (CSTP) or Certified AI Engineer).
• Knowledge of big data technologies (e.g., Hadoop, Spark) and how they integrate with AI models.

This AI Tester role is a unique opportunity to shape the future of AI in the pharmaceutical industry. If you’re passionate about AI, testing, and making a difference in healthcare, we encourage you to apply.

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly