Claude's New Best Practices Boost Browser Automation Accuracy

TL;DR

• Anthropic has released new best practices for using Claude models in computer and browser automation.
• The key to improved click accuracy is pre-downscaling screenshots to fit within API limits.
• Claude 4.6 models have image limits of 1568 pixels (long edge) and 1.15 megapixels, while Claude Opus 4.7 supports higher resolutions up to 2576 pixels and 3.75 megapixels.
• A recommended starting resolution for integrations is 1280x720.

Accurate clicking is fundamental for any computer automation task powered by AI. When your AI agent needs to interact with a graphical interface, misaligned clicks can lead to failed workflows, incorrect data entry, and general ineffectiveness. Anthropic's latest guidance highlights a crucial optimization for improving this click accuracy when using their Claude models: the careful management of image resolution sent to the Computer Use API.

When you send a screenshot to Claude's Computer Use API, the model processes it and returns coordinates relative to the display dimensions you specify. However, there are inherent limitations within the API regarding image size. Images exceeding these internal processing limits are automatically downscaled before the model analyzes them. This creates a discrepancy: the model clicks based on a potentially degraded image, while your automation harness expects coordinates mapped to the original, higher-resolution image. This mismatch is a primary driver of click inaccuracy, especially at higher resolutions.

The Claude 4.6 model family, which includes models like Claude 4.6 Sonnet and Haiku 4.5, adheres to specific image size constraints for the Computer Use API. The maximum allowable long edge is 1568 pixels, and the total pixel count is capped at 1.15 megapixels. For users leveraging the more advanced Claude Opus 4.7, the limits are more generous, supporting a maximum long edge of 2576 pixels and a total pixel count of 3.75 megapixels. In both cases, images that surpass these thresholds will be internally downscaled by the API.

The most impactful solution to this problem is elegantly simple: pre-downscale your screenshots to ensure they fall within the specified API limits before transmitting them to Claude. By doing so, you guarantee that the model is operating on an image that matches the coordinate space your harness is using, thereby significantly improving the precision of its actions. This practice is more effective than many other complex optimizations and is a cornerstone of reliable computer use integrations.

For most applications, a recommended starting resolution is 1280x720. This resolution is a practical and safe default, consuming approximately 80% of the pixel budget and comfortably fitting within both the long edge and total pixel limits for all supported Claude models. It represents a standard resolution that models are well-trained on and works effectively across a wide range of modern web interfaces and older desktop applications. If you are using the more capable Claude Opus 4.7, you might consider 1080p resolutions, which offer a tangible quality improvement over 720p while maintaining a good balance between token consumption and performance.

For developers aiming to provide the model with the maximum possible visual information without distortion, an advanced technique involves calculating the optimal resolution per image based on its native aspect ratio. This "max API fit" approach ensures that the full pixel budget is utilized while strictly adhering to the API's constraints and preserving the image's original proportions. While slightly more complex to implement, this method can yield modest but noticeable accuracy improvements by avoiding aspect ratio distortions that can occur when forcing different aspect ratios into fixed display resolutions. You can find more detailed information and implementation guidance in the Computer Use Tool Documentation.

Summary

• To enhance click accuracy in Claude-powered automation, developers should pre-downscale screenshots before sending them to the Computer Use API.
• Claude 4.6 models have limits of 1568px (long edge) and 1.15MP, while Claude Opus 4.7 supports up to 2576px and 3.75MP.
• A 1280x720 resolution is a recommended starting point for most computer use integrations.
• Further details and advanced techniques are available in the Computer Use Tool Documentation.

Source: Best practices for computer and browser use with Claude | Claude