ServiceNow Research

We're hiring! Come help us crack hard problems related to evaluation and governance of large language models, agents and deployed AI systems. No shortage of hard and emerging challenges to sink your teeth into! ServiceNow Research's Trust & Governance Lab is looking for a Senior Applied Research Scientist. You will join a multi-disciplinary team of applied AI researchers and developers who work on test & eval challenges up and down the stack, from figuring out what capabilities and vulnerabilities foundation models have, to researching the strengths and weaknesses of using models (and juries and human-AI collaborative arrangements and...) to judge other models, to building out risk-aware test and eval methodologies and tooling for teams deploying to production, and more. Link to the job posting in comments. #artificialintelligence #trustworthyai #aisafety ServiceNow ServiceNow Research

Are you passionate about pushing the boundaries of AI safety and security research and having an impact on enterprise AI? Join our team! **About the Role** The Reliable and Secure AI (ResAI) team at ServiceNow Research is seeking a talented Senior Research Engineer to tech lead our efforts. We're at the forefront of adapting frontier AI technologies for customers with sensitive data and strict security requirements. **What We Offer** * A chance to be part of a 0 to 1 initiative * Work on cutting-edge AI security and safety research, and help over 8000 ServiceNow customers put AI to work in a secure fashion * Collaborate with a team of top researchers and engineers **How to Apply** If you're excited about this opportunity and have relevant experience, we want to hear from you! * Apply online: https://lnkd.in/gPBKs_r3 * Reach out directly to discuss the role (Note: While we may not be able to respond to everyone individually, we seriously consider all qualified applications) Join us in shaping the future of secure AI in enterprise software! #AIResearch #JobOpportunity #ServiceNow #EnterpriseAI

💡Today we're spotlighting Sean Hughes AI Ecosystem Director at ServiceNow! For Sean, open scientific collaboration and open-source AI provides fuel to the virtuous cycle of open innovation. Check out the post to learn more about Sean, his contributions to the open-source AI ecosystem and his work co-leading the AI Alliance Foundation Models working group and community efforts.

📢 Call for Papers: Time Series in the Age of Large Models We are organizing a workshop on "Time Series in the Age of Large Models" at NeurIPS 2024. We invite submissions related to the theme of time series in the age of large models. Key topics include, but are not limited to: 📌 Building Time Series Foundation Models 📌 Analysis of Pretrained Time Series Models 📌 Critiques on Time Series Foundation Models 📌 Faster and Better Inference Schemes for Autoregressive Time Series Models 📌 Leveraging Pretrained Models of Other Modalities for Time Series 📌 Multimodal Time Series Models 📌 Large-Scale Time Series Datasets and Benchmarks 📌 Time Series Evaluation 📌 Real-World Applications of Large Time Series Models Please consider submitting your latest research on these topics to our workshop. Submission deadline: Sep 15, 2024 (11:59 PM AoE) Acceptance notification: Oct 14, 2024 Workshop Date: Dec 14/15, 2024 We have an excellent lineup of keynote speakers and panelists from diverse backgrounds: 💎 Andrew Gordon Wilson 💎 Christoph Bergmeir 💎 Mihaela van der Schaar 💎 Tomas Pfister 💎 Qingsong Wen 💎 Valentina Zantedeschi Further details can be found on the workshop's website: https://lnkd.in/ePC26dUa We are also looking for reviewers for the workshop. If you would like to nominate someone (or yourself), please fill this form: https://lnkd.in/evzeNxst Organizers: Abdul Fatir Ansari, Arjun Ashok, Imry Kissos, Moshe Unger, Kashif Rasul, Pedro Mercado, Laurent Callot, Johannes Stephan. Please help us spread the word!

The AI Alliance welcomes the U.S. Department of Commerce and the National Telecommunications and Information Administration (NTIA) for its forward-thinking report on open model weights, which recognizes the profound benefits of open foundation models for fostering innovation, economic growth, and AI safety. By concluding that "the government should not restrict the wide availability of model weights for dual-use foundation models,” the Commerce Department has taken a crucial step towards ensuring that the U.S. remains at the forefront of AI development. Read more about the AI Alliance’s input here. https://lnkd.in/g_B2-ssW

Thrilled to announce that ServiceNow has won VentureBeat's AI Innovation Award for best enterprise software implementation of GenAI! This award recognizes our work with Hugging Face and the BigCode community on StarCoder, a unique family of code LLMs built following open scientific development principles. StarCoder empowers organizations to customize LLMs for their specific needs, fostering a truly collaborative and open AI ecosystem. Huge congrats to the Hugging Face and ServiceNow Research teams for their collaboration and stewardship of BigCode. We look forward to many more exciting collaborations that push the boundaries of open innovation and open-source AI. #AI #OpenSource #GenAI #StarCoder #BigCode #ServiceNow #HuggingFace

𝗠𝟮𝗟𝗶𝗻𝗴𝘂𝗮𝗹 🔈 🔥 𝗡𝗲𝘄 𝗗𝗮𝘁𝗮𝘀𝗲𝘁 𝗔𝗹𝗲𝗿𝘁 ❗❗❗ ⭐ 𝗠𝗨𝗟𝗧𝗜 𝗟𝗜𝗡𝗚𝗨𝗔𝗟 + 𝗠𝗨𝗟𝗧𝗜 𝗧𝗨𝗥𝗡 + 𝗦𝗬𝗡𝗧𝗛𝗘𝗧𝗜𝗖 + 𝗧𝗔𝗫𝗢𝗡𝗢𝗠𝗬 𝗚𝗨𝗜𝗗𝗘𝗗 + 𝗘𝗩𝗢𝗟𝗦 ⭐ 🥇 𝟏𝟖𝟐𝐊 𝐦𝐮𝐥𝐭𝐢𝐥𝐢𝐧𝐠𝐮𝐚𝐥 𝐢𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧𝐬 🗣 𝟕𝟏 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞𝐬 🌐 𝟯𝟳 𝗹𝗼𝘄 𝗿𝗲𝘀𝗼𝘂𝗿𝗰𝗲 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲𝘀 🤹♂️ 𝟭𝟵 𝗱𝗶𝘃𝗲𝗿𝘀𝗲 𝗺𝘂𝗹𝘁𝗶𝗹𝗶𝗻𝗴𝘂𝗮𝗹 𝗡𝗟𝗣 𝘁𝗮𝘀𝗸𝘀 🏆 𝗜𝗺𝗽𝗿𝗼𝘃𝗲𝘀 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗯𝘆 𝘂𝗽𝘁𝗼 𝟯𝟬% 𝗼𝗻 𝗺𝘂𝗹𝘁𝗶-𝗹𝗶𝗻𝗴𝘂𝗮𝗹 𝘁𝗮𝘀𝗸𝘀 🔍 𝗗𝗲𝗺𝗼𝗻𝘀𝘁𝗿𝗮𝘁𝗲𝘀 𝗠𝘂𝗹𝘁𝗶-𝗹𝗶𝗻𝗴𝘂𝗮𝗹 𝗦𝗙𝗧 𝗶𝗺𝗽𝗿𝗼𝘃𝗲𝘀 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗮𝗯𝗶𝗹𝗶𝘁𝗶𝘁𝗲𝘀 𝘀𝗶𝗴𝗻𝗶𝗳𝗶𝗰𝗮𝗻𝘁𝗹𝘆 Evol mutations to generate more IFT pairs are guided by: - 𝐍𝐋𝐏 𝐭𝐚𝐬𝐤 𝐬𝐩𝐞𝐜𝐢𝐟𝐢𝐜 𝐞𝐯𝐨𝐥 𝐭𝐚𝐱𝐨𝐧𝐨𝐦𝐲. - Reasoning, slang, dialect, transliteration, etc for general instructions. - 𝗠𝐮𝐥𝐭𝐢-𝐭𝐮𝐫𝐧 𝐞𝐯𝐨𝐥 𝐭𝐚𝐱𝐨𝐧𝐨𝐦𝐲 for generating different types of - {follow-up, refinement, expansion, recollection} conversational follow ups 🔄 For more, checkout our paper below! Authors - Rishabh Maheshwary Vikas Yadav, PhD Hoang Nguyen Khyati Mahajan Sathwik Tejaswi Madhusudan 🌟 We are also actively hiring if you want to join us on our journey 🚀 Dataset: https://lnkd.in/e5E-VrN8 #genai #LLMs #research #machinelearning #deeplearning #nlp #huggingface #dataset #multilingual #sft #instructionfollowing #ai #chatgpt #gpt4 #servicenow #hiring #jobs

🚨 It is time for foundation models for Embodied AI. Excited to Introduce GenRL! An AI agent that learns multimodal foundation world models 🌍 By connecting the multimodal knowledge of foundation models with the embodied knowledge of world models for RL, GenRL enables turning vision and language prompts into actions! We release paper, code, datasets, models, and a Gradio interactive demo! Visit the project website to know more: https://lnkd.in/e8TenzbP Pietro Mazzaglia, Aaron Courville, Tim Verbelen, ServiceNow Research, Mila - Quebec Artificial Intelligence Institute Bonus: we also tested GenRL on the Minecraft learning environment. The agent can identify biomes easily but struggles with accurate tasks, e.g. crafting objects. In the future, we will extend GenRL to more complex Minecraft tasks, e.g. using MineCLIP from Jim Fan

📢 New paper out! We release RepLiQA: a Q&A dataset designed to evaluate language models in a truly unseen context. RepLiQA covers made-up entities so that knowledge obtained during training cannot be re-used to answer its questions. #LLM #Evaluation @ServiceNow @ServiceNowRSRCH 🚀 RepLiQA comprises 90k question-answer pairs and 18k reference documents across 17 categories. The data is partitioned into 5 splits to limit the exposure of our documents and annotations to potential scraping. We've just released the first split! 🔍 Current evaluation benchmarks for LLMs may be compromised due to potential data leakage into training or due to them covering well documented facts. Evaluation may be confounded by knowledge from pre-training, posing challenges in assessing performance. #RepLiQA #DataIntegrity 🔎 We conducted a large-scale benchmarking set of experiments with 18 state-of-the-art LLMs. Our study reveals insights into how LLMs rely on internal memory versus external prompts. We relied on @OpenRouterAI to easily evaluate multiple models. See https://lnkd.in/g5pkqvtx 🔗 Access RepLiQA: Available splits can be accessed via @huggingface and found at https://lnkd.in/gznfmHN8. Stay tuned for upcoming releases of new test splits every two months starting December 2024! 📆📥 🔗 Find more details and results on our paper or blog post: - https://lnkd.in/gdHCT8Y4 - https://lnkd.in/gbJZmfNq We also provide code for getting the data and running inference at: - https://lnkd.in/g5pkqvtx