Stabilization and standardization of data from +100 APIs at InCommodities

InCommodities

twoday was tasked with retrieving, stabilizing, and standardizing large amounts of data from several hundred APIs and websites. Initially driven by an urgent need, it has since evolved into a close collaboration spanning several years.

Background

In 2017, InCommodities, a tech company specializing in trading electricity and gas in the European and North American markets, was founded with the ambition to do things differently in the industry. One of the goals was to be more agile, people-focused, and trust-based.

Since then, there has been only one direction - up! This has resulted in a growth of more than 3000% and a position among the largest revenue-generating companies in Denmark. However, as an energy trading company, they are extremely dependent on data and the speed at which data is available. More horsepower was needed to standardize the many data sources that algorithms, analysts, and traders rely on for their transactions.

The task

The tech-based approach, combining deep data and technology to create forecasts upon which InCommodities bases their trades, was a perfect foundation for collaboration with twoday.

twoday was tasked with retrieving, stabilizing, and standardizing large amounts of data from several hundred APIs and websites. Initially, it was driven by an urgent need, but it has since evolved into a close collaboration spanning several years.

At InCommodities, high demands are placed on both the expertise and personality of employees and external consultants. Therefore, presenting the right consultants for the job was an essential part of the project.

"Lars has been extremely skilled at presenting us with the right consultants for the tasks, both reactively and proactively. Our values and culture are essential to us, which places great demands on the presentation of consultants for the tasks, both professionally and personally."

Jackob Hestbæk Head of Software Development at InCommodities

How twoday Solved the Task

twoday's consultants developed data scrapers in JavaScript to achieve high development speed. These scrapers are supported by a range of scheduling and monitoring tools, as well as a home-built - yet mature - SDK that makes it easy to perform the most common tasks.

A few APIs fetch large amounts of data every day, and these are developed in F#. This is partly because it is the primary programming language used in InCommodities' IT department, but also because it provides robustness and execution speed that JavaScript does not have.

The Value of Collaboration

twoday now constitutes 50% of InCommodities' data scrape team, internally known as the "gold diggers," playing a crucial role in forming the overall market picture.

An otherwise trivial software development task has been elevated to a level where it is now considered an essential piece in a larger setup, integrated into the operational excellence setup necessary for success in the energy trading space.

Especially the large and diverse selection of social, outgoing, and extremely competent software developers who can also act as critical consultants has been a decisive factor for InCommodities in choosing twoday as a project partner.