Claude Sonnet 3.5
VS
ChatGPT 4o

Additional Features

This time, both models have vision capabilities. Performance is complex and to be tested separately.

Anthropic keeps making strides with its safety research - Sonnet 3.5 is your go-to if you want to push for a change in the state of current AI development.

ChatGPT 4o gives around 100 tokens/s, whilst Sonnet provides 80 - both are impressive feats. Moreover, GPT 4o still has an upper hand in the multimodal aspect with its speech recognition.

Conclusion

Our testing showed that Sonnet is stronger in maths - most likely due to the low sample size. The rest of the tasks show comparable results.

If Sonnet 3.5 can come out on top in more complicated tasks, most likely Claude 3.5 will be an absolute beast. With recent advances of Gemini 1.5 to the top of the LLM arenas - we have an incredibly competitive landscape on our hands.

Pricing

The Pricing model is given in AI/ML API tokens. GPT 4o and Sonnet 3.5 have the same output prices with Sonnet having the cheaper input price.

Compare for yourself

You've seen what these models can do - now try them for your use case. Plug the code below into Google Colab or any IDE, use your API Key, and get testing!

import openai
import requests

def main():
    client = OpenAI(
      api_key='<YOUR_API_KEY>',
      base_url="https://api.aimlapi.com",
    )

    # Specify the two models you want to compare
    model1 = 'gpt-4o'
    model2 = 'claude-3-5-sonnet-20240620' 
    
    response1="no response received"
    response2="no response received"

    system_prompt = 'You are an AI assistant that only responds with jokes.'
    user_prompt = 'Why is the sky blue?'


    try:
        response = client.chat.completions.create(
            model=model1,
            messages=[
                {'role': 'system', 'content': system_prompt},
                {'role': 'user', 'content': user_prompt}
            ],
        )
        response1 = response.choices[0].message.content
    except Exception as error:
        print(f"Error with model {model1}", error)

    try:
        headers = {
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json"
        }

        data = {
            "model": model1,
            "system": system_prompt,
            "messages": [
                {
                    "role": "user",
                    "content": user_prompt
                }
            ]
        }

        response = requests.post(
            "https://api.aimlapi.com/chat/completions",
            headers=headers,
            json=data
        )

        response_json = response.json()
        response2 = response_json['choices'][0]['message']['content']
    except Exception as error:
        print(f"Error with model sonnet-3.5:", error)

    # Compare the results
    print('Comparison of models:\n')
    print(f"\n{model1}: {response1}")
    print(f"\n{model2}: {response2}")

if __name__ == "__main__":
    main()

‍

Conclusion

Claude Sonnet 3.5 is a worthy competitor, proudly taking its place amongst the best models on the market. The tests are stacked against GPT 4o, and even the pricing is very welcoming. Pick Sonnet 3.5 if our tests have convinced you - or stick to the old reliable ChatGPT 4o.

You can check our model lineup here - try any of them for yourself with our API Key.