Lately, I’ve been struggling with my addiction to binge-watching Naruto. While it’s enjoyable, it definitely doesn’t help me deliver shareholder value. 😄
So, why not build an AI personal assistant that monitors my screen and lets me know if I am overdoing something I should not do, like watching anime? 😅
Considering the rapid development in AI in the past year, I decided to use a multi-modal language model to monitor my screen and let me know when I’m spending too much time on non-productive activities.
So, here’s how I did it.
Configure OpenAI GPT-4o, multi-modal AI model.
Use a screen analyzer tool from Composio to monitor the screen.
Pass the screenshots to GPT at regular intervals.
Rendering the message from GPT as a notification in the system.
In this article, I will also explain how you can build your personal AI friend using OpenAI and Composio.
Composio – Your AI Agent Tooling Platform
Composio is an open-source platform that equips your AI agents with tools and integrations. It lets you extend the ability and versatility of your AI agents through integration tools like code interpreter, RAG, Embedding and integrations like GitHub, Slack, Jira, etc.
Prerequisites for Building AI Friend
To successfully complete the project, you will need the following.
OpenAI SDK and API key: To interact with the LLM.
Composio: For accessing image analysing tool.
PyAutoGUI: To automate interactions on the screen.
Osascript: To execute AppleScript commands for controlling macOS applications.
So, let’s get started.
Let’s Get Started 🔥
Begin by creating a Python virtual environment.
cd ai-friend
source bin/activate
Now, install the following dependencies.
pip install composio-openai openai
pip install pyautogui
Next, Create a .env file and add environment variables for the OpenAI API key.
Set Up Composio
You can use the CLI to set up Composio easily.
First, log in to your account by running the following command.
Finish the login flow to proceed further.
Now, update apps.
Now, you are ready to move to the coding part.
Building the AI Friend
Now that you have set up the environment, let’s hop on to the coding part.
First, import the libraries and initialize the toolsets.
from openai import OpenAI
from composio_openai import App, ComposioToolSet
from composio.utils.logging import get as get_logger
logger = get_logger(__name__)
# Load environment variables from .env
dotenv.load_dotenv()
# Initialize tools.
openai_client = OpenAI()
composio_toolset = ComposioToolSet()
# Retrieve actions
actions = composio_toolset.get_tools(apps=[App.SYSTEMTOOLS, App.IMAGEANALYSERTOOL])
So, in the above code block,
We imported all the required libraries and modules.
Loaded the variables defined in the .env file.
Created an instance of OpenAI() and ComposioToolSet.
Retrieved the Actions from SYSTEMTOOLS and IMAGEANALYSERTOO.
So, here is what these tools do.
SYSTEM TOOLS: The system tools have two Actions: push notifications and screen capture.
IMAGEANALYSERTOOL: This tool has only one Action: analyzes images using multi-modal LLMs like GPT-4o and Claude Sonnet, etc.
If you want to examine the code and how it works, check the code files for system tools and the image analyser tool.
Note: Actions in Composio are tasks that your agent can perform, such as clicking a screenshot, sending a notification, or sending a mail.
Set Up OpenAI Assistant
Now, define a clear and concise prompt for the agent. This is crucial for agent performance. You can alter the prompts based on your requirements.
“””You are an intelligent and proactive personal productivity assistant.
Your primary tasks are:
1. Regularly capture and analyze screenshots of the user‘s screen.
2. Monitor user activity and provide timely, helpful interventions.
Specific responsibilities:
– Every few seconds, take a screenshot and analyze its content.
– Compare recent screenshots to identify potential issues or patterns.
– If you detect that the user is facing a technical or workflow problem:
– Notify them with concise, actionable solutions.
– Prioritize non-intrusive suggestions that can be quickly implemented.
– If you notice extended use of potentially distracting websites or applications (e.g., social media, video streaming):
– Gently remind the user about their productivity goals.
– Suggest a brief break or a transition to a more focused task.
– Maintain a balance between being helpful and not overly disruptive.
– Tailor your interventions based on the time of day and the user
Operational instructions:
– You will receive a
1. Take a screenshot using the screenshot tool.
2. Then, analyse that screenshot using the image analyser tool.
3. Then, check if the user uses distracting websites or applications.
4. If they are, remind them to do something productive.
5. If they are not, check if the user is facing a technical or workflow problem based on previous history.
6. If they are, notify them with concise, actionable solutions.
7. Try to maintain a history of the user‘s activity and notify them if they are doing something wrong.
Remember: Your goal is to enhance productivity while respecting the user
‘s autonomy and work style.“””)
assistant = openai_client.beta.assistants.create(
name=“Personal Productivity Assistant“,
instructions=assistant_instruction,
model=“gpt-4-turbo“,
tools=actions, # type: ignore
)
# create a thread
thread = openai_client.beta.threads.create()
print(“Thread ID: “, thread.id)
print(“Assistant ID: “, assistant.id)
In the above code block,
A detailed assistant instruction is provided.
Created a new assistant instance with the previously defined instruction, model name, and previously defined actions.
Finally, create a thread for interaction with the models.
Define and Run the Assistant
Now, define a function for running the assistants.
logger.info(“Checking and running assistant“)
# Send ‘CHECK’ message to the assistant
message = openai_client.beta.threads.messages.create(
thread_id=thread.id,
role=“user“,
content=“CHECK“,
)
# Execute Agent
run = openai_client.beta.threads.runs.create(
thread_id=thread.id,
assistant_id=assistant.id,
)
# Execute function calls
run_after_tool_calls = composio_toolset.wait_and_handle_assistant_tool_calls(
client=openai_client,
run=run,
thread=thread,
)
# Run the assistant check every 10 seconds
while True:
check_and_run_assistant()
Here’s what is going on in the above code.
Send a ‘CHECK’ Message: This sends a “CHECK” message to the assistant in the specified thread to ensure the model is responsive.
Execute Agent: Creates a run for the assistant using the specified thread and assistant IDs.
Handle Tool Calls: Waits for and handles tool calls made by the assistant using the Composio toolset.
Loop the Agent: Loop the agent so it runs and monitors your workflow continuously.
Finally, execute the file by running the Python file and letting your new AI friend keep you focused on your goals.
The agent monitors your screen and sends a notification when it sees you doing something you should not.
The full code can be found here
Here is an example of the agent in action.
// Detect dark theme
var iframe = document.getElementById(‘tweet-1820129229683454160-625’);
if (document.body.className.includes(‘dark-theme’)) {
iframe.src = “https://platform.twitter.com/embed/Tweet.html?id=1820129229683454160&theme=dark”
}
Next Steps
In this article, you built your personalised AI friend that monitors your activity. However, adding external integrations such as a Calendar or Gmail tool can make it even more useful. This lets you know if you have some events to attend or important emails to respond to.
You can do it easily with Composio’s wide array of integrations, from GitHub and Calendar to Slack, Discord, and more.
If you want to see more on AI, friend, let me know in the comments and give us a star on GitHub.
Star the Composio.dev repository ⭐
