Dylan Patel 称拥有大量 TPU 的实验室会为 TPU 协同设计模型注意力形状,因此 GPU 优化的 Muse Spark Open 低估了 TPU 的潜力。
注意,拥有大量 TPU 算力的 AI 实验室,如 Anthropic、Google DeepMind 以及即将加入的 OpenAI,通常会协同设计模型,使注意力形状适配 TPU 以避免这一问题。原文 (en)
Note that AI labs with lots of TPU capacity, such as Anthropic and Google Deepmind and soon OpenAI, typically co-design their models such that the attention shapes would fit nicely on TPUs to avoid this issue.