Title is a bit dramatic, but yes, Claude 3 claims to be better than GPT 4 in most ways.

  • kromem@lemmy.world
    link
    fedilink
    English
    arrow-up
    8
    ·
    8 months ago

    The sonnet model is decent, but something weird is going on with their opus model as it just terribly sucks.

    Mistral-large is probably the best large model for practical purposes at this point.

    • bugsmith@programming.dev
      link
      fedilink
      English
      arrow-up
      4
      ·
      8 months ago

      Mistral-large is probably the best large model for practical purposes at this point.

      What makes you say that? I have not performed my own comparison, but everything I have seen and read suggests that GPT4 is king, currently.

      • kromem@lemmy.world
        link
        fedilink
        English
        arrow-up
        12
        arrow-down
        1
        ·
        edit-2
        8 months ago

        It depends on the task, but in general a lot of the models have fallen into a dark pattern of Goodhart’s Law, targeting the benchmarks but suffering at other things.

        So as an example, while GPT-4 used to correctly model variations of the wolf, goat, cabbage problem with token similarity hacks (i.e. using emojis instead of nouns to break pattern similarity with the standard form of the question), now it even fails for that with the most recent updates, whereas mistral-large is the only one that doesn’t need the hack at all.

        • bugsmith@programming.dev
          link
          fedilink
          English
          arrow-up
          3
          ·
          8 months ago

          Interesting. That’s not something I’ve heard about until now, but something I’ll surely look into.