Generative AI tools, particularly Large Language Models (LLMs) such as ChatGPT, offer immense potential for solving all kinds of business problems, from creating documents to generating code. They can also introduce security risks in two novel ways: leaking information and introducing code vulnerabilities. This article explores the ways these challenges often arise across organizations and provides mitigation strategies to minimize negative outcomes.