Screenpilot

MCP server to let LLM take full control on your device by providing screen automation toolkit for controlling and interacting with graphical user interfaces. Good for automation, education and having fun.

Main Features

📷 Screen capture and analysis
🖱️ Mouse control (clicking, positioning)
⌨️ Keyboard input (typing, key presses, hotkeys)

watch demo

https://github.com/user-attachments/assets/c18380c0-b3dd-4b7c-925d-28ef205ca11f

Installation

Install python 3.12

Clone the repository:

git clone https://github.com/Mtehabsim/ScreenPilot.git

create virtiual environment


python -m venv venv

activate the env

venv\Scripts\activate

Install the required packages:
```
pip install -r requirements.txt
```
Open Claude AI desktop
file -> settings -> developer -> edit config
open config file and paste this

{
    "mcpServers": {
        "device-controll": {
            "command": "pathToEnv\\venv\\Scripts\\python.exe",
            "args": [
                "pathToProject\\ScreenPilot\\main.py"
            ]
        }
    }
}

Replace "pathToEnv\venv\Scripts\python.exe" → with the full path to your python.exe "pathToProject\ScreenPilot\main.py" → with the full path to your main.py file
Save the config file.
Open Claude AI Desktop.
Go to File → Exit
You can now open Claude AI Desktop and enjoy ScreenPilot.

Available Tools

Screen Capture: Take screenshots and get screen information
Mouse Control: Move the mouse and perform clicks
Keyboard Actions: Type text, press keys, and use hotkey combinations
Scrolling: Scroll in different directions and to specific positions
Element Detection: Check if elements exist on screen and wait for them to appear
Action Sequences: Perform multiple actions in sequence

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.