Every company is jumping on top of the AI train, and so are we. Not because of the hype, but we seriously think this could improve our Attic Security platform and make lives easier. Using this technical blog, we share our lessons learned and insights in how it was built.
What is Attic?
Attic Security is a cybersecurity platform that allows companies to improve the security of their Microsoft 365 environment. It shares recommendations to improve your security configuration and alerts you whenever malicious activity is detected in your tenant.
Helping our customers
We believe it’s important to help our customers in understanding the importance of security incidents. Therefore, we offer a feature in Attic that allows you communicate with one of our operators. If there are any questions regarding a security incident, our operators answer those questions.
Quality and time-consuming
Even though we are trying to minimize the number of questions by optimizing our content, there always remain questions. Depending on the number of questions this can be a time-consuming task for our operator. We expect AI can assist us to improve the quality of our communication and to decrease that required time. But there is always a human operator that reviews the proposed message and finalizes it.
Hooking into the communication
Under the hood, Attic uses tickets to provide the communication with our customers. We use Zendesk as a ticketing system. Zendesk offers webhooks and an API, which allows us to easily communicate with the system. This allows us to hook into the communication between our operator and the customer. As the AI engine we are going to use OpenAI.
Creating a webhook trigger
Zendesk offers webhooks to get notified whenever a change occurs within Zendesk. This can be configured using triggers. We configured a trigger to send a webhook to a URL whenever a ticket is updated. This means that every comment on tickets will be posted to our AI-backend.
AI-backend software decision
At the first attempt in creating our AI-backend we used the Zapier cloud service because it offered Zendesk and OpenAI integrations. We succeeded in building the system there, but we noticed quickly that the costs were too high to keep this running.
Therefor we decided to shift to Azure Functions combined with Python. We regularly use Functions because it’s a great way to run your code ‘serverless’, some advantages:
- low maintenance (only the code)
- Low costs
- Automatically scales on demand
Below images shows a summary of the system that we have built. Under the image we explain the different flows.
Flow 1: OpenAI opt-in
We understand that not everyone likes to share content with OpenAI. Therefore, we use an opt-in approach. In our Attic platform, users will get a ticket to enable their AI assistant. By simply running the fix, they opt into the AI assistant. Only then, data is shared with OpenAI.
Flow 2: new incident
Whenever a new ticket (incident) is created, Zendesk sends a webhook to our Azure Function. We will store the following information into an Azure Table:
- the ticket id as a reference
- the ticket title containing the title of an incident
- the first comment containing a description of the incident
- the second comment containing the second part of the description
- external content related to the incident, such as code
- boolean thread_created, whether the OpenAI is created (set to false)
- boolean ready, whether all incident information has been gathered successfully
This information is stored in the database so we can look it up if a user asks a question. This allows us to keep all the data in-house, until a user asks a question. It also reduces the amount of OpenAI tokens, and so the costs.
Flow 3: new question
If a comment is added to a ticket, Zendesk sends a webhook to our Azure Function. In this flow we will lookup all the content that was stored about the incident combined with the question.
On the first question, we will create a new OpenAI thread. If its question number two or more, the question will be added to the existing OpenAI thread. This allows the AI assistant to know all the previous content and reply accordingly to the conversation.
The reply by OpenAI is obtained and send to a Microsoft Teams channel. Our operator gets a Teams notification whenever a new proposed reply occurs.
Flow 4: operator review
This brings us to the last flow; the operator will review the message via a Teams notification. Allowing her to send or ignore the proposed AI message. Teams message example:
Fine-tuning the AI bot
A nice feature of the OpenAI platform is that you can create your own customized Assistant. This allows you to set some settings and specific instructions the bot must follow. Some example instruction we pass onto our bot:
You are a cybersecurity assistant. Assist our customers by explaining what security incident is going on and how they can mitigate the incident.
Context
You are a cybersecurity assistant embedded in the app called Attic. Attic is a solution to help companies harden and monitor their Office 365 environment.
Instructions
- Always end your message with ^Joyce AI
- Keep your answers as short as possible and to the point
- The first messages of the conversation contain the incident information related to the client question. You can use this information to answer the customer question.
Fine-tuning the AI assistant bot does not require technical skills. By just supplying ‘normal human language’ the bot will change its behavior in the future. This doesn’t require any code changes, which is perfect to improve the bot’s behavior daily.
The result
After implementing the AI Assistant in the Attic platform, the following screenshots shows the result on the user experience:
Conclusion – our experience up until now
We are still in the process of bringing the AI assistant to production to our customers. Up until now, the replies generated by the AI assistant looks promising. Of course, the AI assistant will not always generate usable replies. But we believe this can be a very effective tool to support our operator – to have more time for fun stuff. Finetuning the incident input and AI bot instructions will make the AI assistant more effective by the day. Maybe, one day, the operator review flow can be completely removed.
We published the source code of our implementation on Github. Feel free to use our code as an inspiration to build your own AI Assistant.
This project was made possible by SIDNfonds.