Why an AI installer?
In the dynamic field of software development, automation is crucial. AppMap, amidst its product-led growth, strives to meet evolving user feature demands with limited resources. With a global installed base using AppMap for runtime code analysis, we recognized the need to expand from code editors to CI systems for broader team collaboration.
Facing diverse CI environments to support, we turned to AI for a solution. Our trials showed that AI significantly accelerates the CI setup process in our new application from 28 minutes to 6 minutes, offloading much of the configuration work from users to AI and enhancing user experience while reducing errors. This AI-led approach is a potentially valuable path for open-source software teams managing various environments and configurations.
An on-demand, developer, and code-centric observability platform running “offline” inside a user’s own CI environment? Sure - let’s build that!
AppMap’s GitHub Action is a runtime analysis tool for CI. It instruments code during automated testing to provide insights into how code changes affect application behavior. Using libraries for Java, Ruby, Python, and Node, it traces backend activities, creating visual “maps” of code behavior changes for each pull request. This data, stored in the user’s build environment, is analyzed and reported in pull request comments.
AppMap configuration is different because… AppMap is not a hosted SaaS. Users configure the product to run “offline, on-prem” in CI.
Scope creep: CI support is a big lift for a small team
When we surveyed users about their CI systems, we realized the challenge of supporting their diverse configurations for running tests. For our initial release, we partnered with GitHub due to both its rising CI platform popularity and our use of GitHub Actions. Our initial experience with design partners revealed a wide range of configurations in GitHub Actions and workflows.
Initially, we created a standard installer with instructions for users. Configuring AppMap for CI involves a few manageable steps: two PRs to merge and five copy-pastes. This process sets up a fully functional observability platform within your CI system, operating as a GitHub Action for every code change.
Was installation easy?
The ease of our initial setup depended on familiarity with GitHub Actions and workflows. For those new to it, integrating AppMap could be error-prone and require additional support, particularly in placing YAML updates correctly in the config file. Despite thorough instructions, unforeseen user environment variations posed challenges. Faced with this, we had two options: refine our manual installer and documentation or innovate with AI to assist in the installation process.
Two PRs and five copy-paste steps isn’t a lot of work….but AIs are happy to do it for you.
We explored using GPT, enhanced by our prompts, for more reliable installation outcomes. Our updated installer utilizes GPT’s language processing to understand and translate a user’s existing workflow requirements and appropriately insert AppMap’s code and configuration steps into the user’s YAML file.
A significant change for data security involves users granting read and write access to their GitHub workflow repository, allowing the AI to suggest code modifications on a branch. To maintain user control, we implemented two sanity checks requiring the user to review and push changes.
AI is the installer of the future today for complex integrations
The AI installer’s main benefits are its efficiency and high user acceptance, with many users preferring it over manual installation. This preference may be due to the AI’s limited access to user workflow files, both in time and information, just the workflow files. We ensure users revoke our installer’s repository access post-installation, limiting its function to a single job and file. AI installation cut our setup time from over 28 minutes to just 6 while also minimizing human error risks. We anticipate even faster installations as OpenAI improves response times.
The AI process doesn’t abandon - it iterates
We designed the AI to refine a commit until it meets our success criteria, then presents the code for user review. Users must approve each change before proceeding, maintaining control over their workflow configurations. This ensures users stay engaged until successful installation completion without any “cost.”
The AI installer saved us development time. Yes… but…
Our experience with the manual version informed the AI integration. It guided our prompt creation and understanding of user-testing pitfalls, enabling a swift integration of the GPT-powered installer. Without this knowledge, it is difficult to estimate how much development time we might save.
Pros and Cons of AI-Guided Installations for CI/GitHub Action Workflows
| Pros | Cons |
|---|---|
| Efficiency: Reduces setup time, streamlines installation, and eliminates incomplete setups. | Limited Flexibility: May struggle with unique or specialized setups. |
| User-Friendly: Intuitive, simplifying CI/GitHub Action configurations for all users. | Unanticipated Behavior: Risk of errors if AI misinterprets user inputs or requirements. |
| Error Reduction: Minimizes human error, particularly in repetitive or complex steps. | Security Risks: Potential issues if AI access to workflow files is not properly managed. |
| Time-Saving: Cuts down on tasks like interpreting and modifying files. | Black-Box Process: Can reduce understanding of processes, complicating troubleshooting. |
| Scalability: Handles multiple installations, aiding software deployment across different environments. | Learning Curve: Users may need time to trust and understand AI’s decisions or accept proposed changes. |
| Customization: Capable of learning and adapting to user preferences for tailored experiences. | Interpretation Errors: Possible inaccuracies in handling complex or unclear requirements. |
Downsides? A few
This method is in its early days, and we are still learning. There may be scenarios where it will not perform well or hallucinate, which will negatively impact users. Dependence on AI for configuration means that highly specialized or unconventional setups might not be as easily interpreted by the installer or the prompts we provided.
Additionally, there’s always a concern with AI that unexpected behavior will be introduced. To ensure this isn’t an unmonitored event, we require user review and approval for specific setup commit steps. Users may not feel comfortable evaluating these workflow changes without further explanation by the AI, which may not provide satisfactory answers during the review process.
Finally, security concerns with using GPT regarding workflow code safety and security might initially deter some users, but this has not been borne out in installation preference acceptance rates. GitHub workflow files that execute test cases can be considered sensitive, albeit perhaps less sensitive than the source code itself, but it is in the software security supply chain of risk. As mentioned above, we instruct users to revoke permissions to the AI-enhanced installer at the end of the workflow, but it could be considered another threat vector if these instructions are not adhered to. However, given the small scope of the AI process and the user preference for this workflow, perhaps the concern over this time-limited risk is small.
Next stop on the AppMap AI installer journey
We plan to enhance AI capabilities for broader CI system compatibility based on user feedback. Prioritizing data privacy, this step initiates our AI integration for AppMap. As an open-source project, we welcome community contributions and suggestions for AppMap’s improvement.
AppMap is continuing to lean in on AI
This project highlights AI’s role in automating and simplifying installations for open-source projects and complex CI workflows. We’re eager to further explore AI’s potential in our installer flow to better support AppMap’s user base.
Try AppMap!
We invite you to explore our new AppMap CI integration and our new AI installer. To learn more about our open source project or share your thoughts and feedback, check us out on GitHub or join our community.
Your feedback on AppMap’s approach to AI is not just welcome; it is essential to the evolution of the AppMap project.
Get AppMap