This article today is from data scientist and Delphi developer Max Kleiner. Max is the author of the MaXbox which Max’s site describes as “a precompiled object based scripting tool”. Max describes his use of RAD Studio with Delphi to develop Windows software and how he uses JSON objects from HTTP requests to visualize aggregated gas storage inventory data into graphs and charts.
Table of Contents
Data representation of gas in storage as a timeline AGSI dataset
“There are many kinds of data scientists: a) Those who dive in data are the best.“
This data science tutorial explains the so called AGSI data storage and his visualization of the timeline. AGSI is the Aggregated Gas Storage Inventory and offers you the possibility to be kept up to date whenever a new service announcement or update from one of our data providers is posted on the website.
The Gas Infrastructure Europe (GIE) is also providing related data such as the Storage Map and Storage Investment Database at https://www.gie.eu/publications/maps/
What will the result of the data chart look like?
The result of the data will be the chart below.
What components do you use to download and chart the data?
We use WinHttp.WinHttpRequest, JSONObjects and TEECharts library with loading and testing the plot. Also an API-key is needed, get the key first at: https://agsi.gie.eu/account
What does the data represent?
The data represents gas in storage at the end of the previous gas day. Data is updated every day at 19:30 CET and a second time at 23:00. Before we dive into code this is the main part of the script:
What does the code do?
The main part generates a form, invokes the API and plots the data.
GIE is providing an API (Application Programming Interface) service on its AGSI and ALSI transparency publication platforms.
Using API access, consumer can bypass the AGSI and ALSI website and get hold of the data directly and continuously. It enables to extract, filter, aggregate the data and create any subset as required, without having to download each dataset separately from the website. The API export format is JSON. For example a subset of 150 days:
What is in the data you are charting?
The published datasets are based on the EIC code mapping table provided to ACER. Storage data is aggregated by company, and country.
With the call I pass country, start-date and number of days:
All available datasets can be downloaded also in Excel, CSV and JSON format. The data in this report shows an aggregated view – individual datasets by company and storage facility are also accessible.
How to call the API to get some data
So, let’s start with the API call:
This command and script runs WinHttp.WinHttpRequest. When you fail you get a bunch of exceptions like the following:
The funny thing is the JSON formatted exception. Be careful, also, to expose your key as I get from Git: GitGuardian has detected the following Google API Key exposed within your GitHub account.
How to get JSON data from the API
Next is the formatting of the get-call with a valid API-key request in the function energyStream()
And where is the fabulous content-type? As far as I understood there are only two places in a web-request where to set a content-type:
- The client sets a content type for the body he is sending to the server (e.g. for get and post).
- The server sets a content type for the response.
Making sure there is a Content-Type available
A sender that generates a message containing a payload body has to generate a Content-Type header field in that message unless the intended media type of the enclosed representation is unknown to the sender; otherwise, we failed at getting response: 503503 - Service Unavailable.
It means that the content-type HTTP header should be set only for PUT and POST requests. GET requests can have “Accept” headers, which say which types of content the client understands. The server can then use that to decide which content type to send back.
As an option you can also use TALWinInetHttpClient. It is a easy to use WinInet-based protocol and supports HTTPs. HTTP client component which allows to post and get any data from the Web via HTTP protocol.
How do we convert a JSON response into usable data for the chart?
Any missing or incomplete data is also visible on AGSI. Next, we step to the conversion of our JSON response for the plot with TJSONObject:
And this JSON Array as above function returns, we pass to the next plot:
What is in the chart series data?
As we can see we have 4 series to plot (including timeline):
- Injection (Injection during gas day)
- Full (Storage / WGV (in%))
- Withdrawal (Withdrawal during gas day (2 digits accuracy)).
- GasDayStart (The start of the gas day reported)
The time series is a result of the gas day and a trend is available.
“Gas day” means the period from 5:00 to 5:00 UTC the following day for Wintertime and from 4:00 to 4:00 UTC the following day when daylight saving is applied. Gas day is to be interpreted as UTC+1 for CET or UTC+2 in Summertime for CEST. (Definition: see CAM Network Code specifications).
How do we access the API for the data?
API access is provided in a REST-like interface (Representational State Transfer) exposing database resources in a JSON format with content-type in the mentioned Response Header.
Where can other developers access this data?
The code of the data science vision contains example usage, and runs under Python3, Delphi, Jupyter-Notebook, object Pascal and maXbox4. Note that the API service is made available to the public free of charge. ONLY the data as currently available on the platforms is made available.
Tip: To extract data direct from the system, you can click on one of these links in a browser (web traffic versus API traffic) AGSI+ https://agsi.gie.eu/api?type=eu
What have we learned about the GIE data API?
GIE is providing an API (Application Programming Interface) service on its AGSI and AGSI+ storage data. The API documentation is on progress and provides examples and guidance on how to use the service and is available after registration to get an API-key. Below zoom of the past half year:
Where can people find your example scripts and code?
The scripts and images can be found at https://github.com/maxkleiner/agsi-data
Do you have an example showing the download code?
Yes, here’s a WinAPIDownload class from maXbox4 integration.
References and additional links
- https://agsi.gie.eu/api
- https://www.gie.eu/transparency-platform/GIE_API_documentation_v006.pdf
- https://svn.code.sf.net/p/alcinoe/code/demos/ALWinInetHTTPClient/_source/Unit1.pas
- https://docwiki.embarcadero.com/Libraries/Sydney/en/System.Net.HttpClient.THTTPClient.Post
- Doc and Tool: https://maxbox4.wordpress.com (Script Ref: 1154_energy_api_agsi_plot14.txt)
This article was submitted as part of our Enterprise Article Showcase. If you have a success story to tell about a project which makes use of RAD Studio with Delphi or C++ Builder, or any of our other great enterprise products please get in touch, you could be featured too and win a prize! Read all about it here: Enterprise Article Showcase