多模态基础模型，如 GPT-4 和 Gemini，能够处理文本以外的数据格式，例如图像，因此在各种应用中都非常有效。然而，这些模型在评估大量多维时间序列数据方面却未得到充分利用，而这在医疗保健、金融和社会科学等行业至关重要。随着时间的推移而进行的顺序测量，即时间序列数据，是一个丰富的的信息来源，但当前模型并未充分利用。这表明错失了在这些领域推动数据驱动决策的更深入、更复杂见解的机会。 为了通过绘图查看时间序列数据，来自 Google AI 的最新研究提出了一种独特而简单的解决方案，即利用多模态模型中已有的视觉编码器。此方法将时间序列数据转换为可视化绘图，并将其馈送到模型的视觉组件中，而不是将原始数值序列提供给模型，这通常会导致性能不佳。这消除了对进一步模型训练的需求，这可能会既昂贵又费时。 研究表明，通过实证评估，以文本格式提供原始时间序列数据的效果不如使用这种可视化技术。使用时间序列数据的可视化表示的主要优势之一是与使用模型 API 相关的显著成本节省。与相同数据的基于文本的序列相比，当数据表示为绘图时，视觉输入需要的令牌（模型处理的信息单位）要少得多，导致模型成本降低高达 90%。 在时间序列数据通常由数千个文本令牌表示的情况下，单个绘图可以更少的视觉令牌传达相同的信息，这不仅使过程更高效，而且更具成本效益。 合成数据试验已被用于验证使用绘图可视化时间序列数据将提高模型性能的假设。这些实验从确定干净数据的函数形式等简单任务开始，然后转向更具挑战性的任务，例如从噪声散点图中推导出重要趋势。模型在这些受控研究中的表现证明了这种技术的稳健性。 研究人员将该技术用于现实世界的消费者健康活动，如跌倒检测、活动识别和准备评估，以进一步验证其超越合成数据的泛化能力。为了使模型能够在这些任务上得出正确的结论，它必须对异构和噪声数据进行多步骤推理。即使在这些要求苛刻的任务中，基于视觉绘图的策略也保持着比基于文本的策略更好的性能。 结果表明，采用时间序列数据的可视化表示显著提高了合成和现实世界任务的性能。在称为零样本任务的合成任务中，模型没有得到任何先验知识，性能提高了高达 120%。结果表明，在现实世界任务中，性能提高了更多，相对于使用原始文本数据，如活动识别和跌倒检测，性能提高了高达 150%。 总之，这些结果证明了利用 GPT 和 Gemini 等多模态模型的固有视觉能力来处理复杂时间序列数据的可能性。绘图已被用来描绘这些数据，这种方法不仅降低了成本，而且提高了性能，使其成为各种应用的可行且可扩展的选项。这种方法使基础模型能够在时间序列数据至关重要的领域以新的方式应用，从而实现更有效和更高效的数据驱动见解。

🤔 为了提升多模态基础模型在时间序列分析方面的应用效果，研究人员提出了一种新的方案：将时间序列数据转换为可视化绘图，并将其输入模型的视觉组件，而不是直接输入原始数值序列。

💡 这种方案能够有效地利用多模态模型中已有的视觉编码器，避免了对模型进行额外训练，从而节省了大量成本和时间。

📈 通过实证评估，研究发现使用可视化绘图表示的时间序列数据，能够显著提高模型性能，在合成数据和现实世界任务中都取得了显著的成果，例如在零样本任务中性能提升高达 120%，在活动识别和跌倒检测等现实世界任务中性能提升高达 150%。

Multimodal foundation models, like GPT-4 and Gemini, are effective tools for a variety of applications because they can handle data formats other than text, such as images. However, these models are underutilized when it comes to evaluating massive amounts of multidimensional time-series data, which is essential in industries like healthcare, finance, and the social sciences. Sequential measurements made over time, or time-series data, are a rich source of information that current models don’t fully utilize. This indicates a squandered chance to glean deeper, more complex insights that might propel data-driven decision-making in these domains.

In order to see time-series data through plots, recent research from Google AI has suggested a unique yet simple solution to this challenge by utilizing the vision encoders already present in multimodal models. This method transforms time-series data into visual plots and feeds them into the model’s vision component instead of giving raw numerical sequences to the models, which frequently results in subpar performance. This removes the requirement for further model training, which could be costly and time-consuming.

The research has shown through empirical evaluations that supplying raw time-series data in text format is not as effective as using this visual technique. The significant cost savings associated with using model APIs is one of the main benefits of employing visual representations of time-series data. Compared to text-based sequences of the same data, much fewer tokens, which are units of information processed by the model, are needed for visual input when the data is represented as plots, resulting in up to a 90% decrease in model costs. 

A single plot may convey the same information with significantly fewer visual tokens in instances where time-series data would normally be represented by thousands of text tokens, which not only makes the process more efficient but also more cost-effective.

Synthetic data trials have been used to validate the premise that using plots to visualize time-series data would improve model performance. Simple tasks like determining the functional form of clean data were the starting point for these experiments, which then moved on to more difficult challenges like deriving significant trends from noisy scatter plots. The resilience of this technique has been proved by the model’s performance in these controlled studies.

The researchers used the technique for real-world consumer health activities like fall detection, activity recognition, and preparedness evaluation to further verify its generalisability beyond synthetic data. In order for the model to reach the right conclusions on these tasks, it must do multi-step reasoning on heterogeneous and noisy data. The visual plot-based strategy was maintained to perform better than the text-based one, even with these demanding jobs.

The results demonstrated that adopting visual representations of time-series data significantly improved performance on both synthetic and real-world tasks. The performance increased by up to 120% in synthetic tasks known as zero-shot tasks, in which the models were given no prior knowledge. The results showed significantly more improvement in real-world tasks, with up to 150% performance increase over using raw text data, such as activity recognition and fall detection.

In conclusion, these results have demonstrated the possibility of handling complex time-series data by utilizing the innate visual capabilities of multimodal models such as GPT and Gemini. Plots have been used to depict this data, and this method not only lowers costs but also improves performance, making it a workable and scalable option for a variety of applications. This approach makes it possible to apply foundation models in new ways in fields where time-series data is essential, enabling more effective and efficient data-driven insights.

---

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. If you like our work, you will love our newsletter.. Don’t Forget to join our 50k+ ML SubReddit

[Upcoming Event- Oct 17 202] RetrieveX – The GenAI Data Retrieval Conference (Promoted)

The post Enhancing Time-Series Analysis in Multimodal Models through Visual Representations for Richer Insights and Cost Efficiency appeared first on MarkTechPost.