Cooperating with ENTSO-E on data


#1

The European Network of Transmission System Operators for Electricity (ENTSO-E) is sending two people to the Openmod Munich Workshop to discuss data with the open modelling community. Topics will include the Transparency Platform, grid data and data licencing.

This is a great opportunity to start a dialogue with ENTSO-E on improving cooperation between ENTSO-E and the rest of the modelling community. Rather than just presenting ENTSO-E with a long wish list, in this forum topic please make suggestions for how the Openmod community can help ENTSO-E. Please post ideas of how a collaboration could be constructive and useful to both sides.


#2

Here are some suggestions:

  • Start a forum for providing feedback on data gaps and inconsistencies in the ENTSO-E Transparency Platform data. Prototypical examples could be this forum, Stack Exchange or the Central-Western-Europe Flow Based Market Coupling Q&A forum set up by the TSOs.
  • Develop ideas for visualising mash-ups of the available data, which would be interesting to users (cf. electricitymap.org or energinet.dk).
  • Develop ideas to visualise data quality issues (e.g. missing data, total generation being lower than load+exports, PV producing at night, etc.) so that it’s easy to identify problems and publicise them (i.e. so that the primary data owners can see where things can be improved)
  • Work with ENTSO-E to share tools for cleaning, processing and standardising data. The tools developed by the Open Power System Data (OPSD) project would be a good place to start.
  • Work with ENTSO-E on metadata standards and standards for presenting data (e.g. the data availability table from OPSD).
  • Provide advice to ENTSO-E on data licencing issues. Gold standard would be Create Commons Attribution 4.0 International, like BNetzA SMARD platform.
  • Coordinate on useful grid data for the community, cf. the list compiled by Eva Schmid.
  • If ENTSO-E’s original datasets are difficult to make openly available, ENTSO-E could validate existing open datasets against the originals and given them an ENTSO-E seal of approval.
  • If ENTSO-E’s original datasets are difficult to make openly available, ENTSO-E could make reduced datasets available, e.g. substation coordinates with small random noise, or clustered/reduced datasets of bigger datasets (e.g. a European transmission network dataset with 500 buses rather than 5000).

#3

An important keystone in this discussion is just how debilitating standard copyright is for energy modelers. I searched extensively for legal analysis on this topic and found nothing concrete. But I have one “off the record” email from an open source lawyer that suggests that, under German law, the machine use of copyrighted datasets probably breaches copyright. What is legally clear is that modeling projects cannot optionally reformat and modify and then republish copyrighted datasets to assist their modeling communities. Ultimately, what is needed is a legal opinion (Rechtsgutachten) on this matter — namely the machine use of copyrighted datasets under German and European law.

Another issue is data longevity. European Commission Regulation 543/2013 states (3 § 1): “The data shall be up to date, easily accessible, downloadable and available for at least five years. Data updates shall be time-stamped, archived and made available to the public.” (emphasis added) So after five years the (third party) copyrighted datasets hosted by ENTSO-E can go dark and there is nothing we can do about it.

Some caucusing ahead of the panel discussion is a good idea, Thanks Tom. Robbie

References

European Commission (15 June 2013). “Commission Regulation (EU) No 543/2013 of 14 June 2013 on submission and publication of data in electricity markets and amending Annex~I to Regulation (EC) No 714/2009 of the European Parliament and of the Council”. Official Journal of the European Union. L 163: 1–12.


#4

Thanks @robbie.morrison, these are very good points.

The current plan for the format of the hour-long panel is:

  • 20 mins: ENTSO-E presentation on data availability
  • 10 mins: Openmod view/suggestions on data availability and licensing collected from this forum (presented by me, I’ll try to share the slides at least a few days in advance for feedback)
  • 30 mins: Discussion, questions from the audience

Then we’d follow up in a dedicated breakout group on Friday morning.


#5

Another thread (that others are more involved in than me) is the question of data and metadata standards. Perhaps those working more directly with data (OPSD project, OpenEnergy Database/Platform project) can contribute here.


#6

It’s probably quite late for this but I would like to express my support for a forum/stackexchange idea. It would be useful to avoid duplicate work and also increase the validity of the data.


#7

Examples of good practice

https://open-power-system-data.org (includes clear visualisation of data availability, scripts for post-processing (for example, UTC issues, missing data, country-specific quirks), and nicely-prepared consolidated data to download)

https://github.com/open-power-system-data (OPSD processing scripts online)

https://open-power-system-data.org/data-projects (a collection of other data projects)

https://wiki.openmod-initiative.org/wiki/Data (collection of other data projects)

http://oep.iks.cs.ovgu.de/ (database presentation, organisation, versioning)

https://github.com/FRESNA/powerplantmatching (tool for matching and resolving different power plant databases)

Lists of missing data/points to change

Overview: see @tom_brown’s presentation at openmod October: https://www.nworbmot.org/energy/brown-entsoe.pdf

TODO: More detailed list: crowd-source on forum, see “ENTSO-E Transparency Platform” category https://forum.openmod-initiative.org/c/entso-e-transparency-platform.

Code for automatic checking, visualising problems

TODO: Look for funding for automated scripts to look for problems and fix them, or report automatically to Primary Data Owner.

ENTSO-E is looking for secondments!

ENTSO-E offers six-month secondments for people interested in working on data and modelling.

Legal opinion on European energy data licensing

Jaeger, Till (24 July 2017). Legal aspects of European energy data — Legal opinion. Berlin, Germany: JBB Rechtsanwälte.


#8

Personal report from the openmod Munich workshop panel discussion

A panel discussion was held on Friday 13 October 2017 with two representatives from ENTSO‑E as part of the 7th openmod workshop. Some personal impressions of that discussion and the subsequent breakout group follow. Tom Brown arranged the session, so kudos to Tom.

ENTSO‑E stands for European Network of Transmission System Operators for Electricity and is the umbrella organization for national and sub-national transmission system operators (TSO) across Europe. ENTSO‑E coordinates the Transparency Platform (TP), operated under European Commission Regulation 543/2013 (a full reference is given elsewhere on this thread). Although ENTSO‑E is cited throughout this post without qualification, all remarks derive from the two representatives present and may not necessarily reflect official ENTSO‑E positions.

ENTSO‑E is guided by its member TSOs and its policy positions normally require unanimous agreement. Most ENTSO‑E data is sourced from the TSOs who, in turn, may collect it from various power exchanges (PX) and market participants within their jurisdictions. Hence, ENTSO‑E sometimes faces an uphill battle to obtain good primary data and encounters many of the same issues that our community does when trying to create consistent and complete datasets. Open licensing can only take place with the consent of the primary data owner. In other words, the licensing web starts with the copyright holders.

Thanks to concerted efforts within our community, including by the OPSD project (Lion Hirth), the OpenEnergy Platform (Ludwig Hülk), and the PyPSA project (Tom Brown), ENTSO‑E are now aware that, in relation to the Transparency Platform and ENTSO‑E public data more generally:

  • the open licensing of public datasets is essential for both scientific research and open development
  • data quality remains an issue and some means by which missing and spurious data points can be flagged and fixed at the point of publication would be to everyones’ advantage

ENTSO‑E noted that the Transparency Platform was conceived for transparency and not as a source of modeling data. However the Transparency Platform does get regularly interrogated and scraped, leading to degraded service. Anecdotally, some TSOs source modeling data from the platform rather than using internal routes.

ENTSO‑E is working on its Pan-European Market Model Database 3.0 (PEMMD3), an extension of version 2.0 used internally, which is distinct from the Transparency Platform. The extent to which this database will be made public and under what licensing terms has not been determined.

The European Commission Integrated Database of the European Energy Sector (IDEES) project was mentioned. But similarly the extent and terms governing its public usage have yet to be notified.

In contrast, the BNetzA (Bundesnetzagentur or German Federal Network Agency) SMARD (Strommarktdaten und zielt auf mehr Transparenz im Strommarkt) energy market platform releases its information under a Creative Commons CC BY 4.0 permissive license (congratulations to BNetzA).

During the breakout group, Frauke Wiese described the OPSD project, Ludwig Hülk the OpenEnergy Platform, and Jonas Hörsch work he and Fabian Hoffmann had undertaken on power plant fleet comparisons. A python module named powerplantmatching can be used to combine disparate power plant fleet databases (vertical matching) and then compare and flag discrepancies (horizontal matching). The module will also optionally select the median value when faced with conflicting plant attributes (field values). I think it would be fair to say that the ENTSO‑E representatives were not much aware of the scope and quality of the work on database support and data sourcing and cleansing taking place within the openmod community.

Information categorization was a recurring theme. For example, some power plant fleet databases distinguish between lignite and hard coal firing while others do not. Simple aggregation is not a difficult issue, but misaligned definitions are. This means that a broadly agreed energy system glossary would be advisable. The openmod community has made a start on their wiki, but this work remains highly incomplete. Database ontologies were also discussed. Wiktionary defines an ontology as “a structure of concepts or entities within a domain, organized by relationships; a system model”. A glossary can be seen as a precursor to a coherent ontology.

Obtaining grid information presents problems for energy modelers, particular geo-location data for engineering assets such as substations. TSOs will not allow too detailed information at the substation level to be made public for security reasons, so there was some discussion on adding white noise (say ±10km) to latlon information to address this concern. It was pointed out that such information will doubtless be available on OpenStreetMap or related websites in due course.

Finally, data and metadata standards were not much discussed but remain an important issue on which to seek consensus and adoption. The Common Grid Model Exchange Specification (CGMES) technical standard for network modeling was mentioned.

My personal thanks to the two ENTSO‑E representatives for contributing to the discussions on open energy system data and to ENTSO‑E more generally for entering into a dialog with the openmod community regarding such data.