Do-a-thon: Draft for 'energy' datapackage standard (?)

zurich-2018
do-a-thon
oki-data-package
#1

Hi all,

here we would like to discuss the idea of a new datapackage standard based on the specific example of frictionless datapackages. This one is related to the common-ontology. However, the goal is not to find common terminology or ontology but rather a logical model that allows for a adequate description of energy (system) data in the datapackage format.

Besides the Tabular-Datapackage Standard a Fiscal Datapackage Standard exists. I think the basic idea and also a part of the specification of the Fiscal Datapackage is a very good starting point. But as it is specifically designed for budget data it will most likely not be suitable to use it for energy modelling related data without ‘misusing’ it.

What do you think? Does it make sense to develop a profile for an “energy datapackage” similar to the fiscal data package?

Do-A-Thon: Using Julia's JuMP for energy system optimisation
Standard open common data format (CDF) for power systems
#2

Hi @simnh. This is certainly an idea worth exploring. The physical/logical mapping that the fiscal data package (FDP) employs is one that could be adopted here — you indicate that in any case. My one concern is making such a package general enough to be applicable and specific enough to be useful. HTH, Robbie.

#3

Hello @simnh ,
your ideas and your motivation are perfectly in line with the goals of the “SzenarienDB” project and the current developments. I’m more than happy to contribute to your draft.

Here are some links and comments in random order:

Wow, that’s a bunch of scattered information… No doubt that it is too hard to find already discussed topics!

As @robbie.morrison already pointed out, the main work is the logical model.
This is where I also see a strong connection to the ontology. Because it will not be just an extended dictionary/glossary but a conceptual representation of the energy domain.
If we bring these ideas together, the resulting “energy data package” would be of great value for the community.

Can you outline what we can Do during the “Do-a-thon” to bring this forward?
Who is also interested from other projects or modelling teams?
What about the “model experiments/ comparison” people?

#4

I would be interested in the do-a-thon as well. Since some energy projects are using tabular data packages already, some simple rules for the energy domain for a datapackage profile/resource could already be enough and be used directly (our team in Flensburg included). Meaning let’s try to follow KISS. Just to address Robbies concern.

#5

perhaps we could find a way for a shared meta-documentation specification (or standard) to combine

and learn a little from the climate sciences.

#6

OPSD has made good experience with following the data package standard from frictionless data :slight_smile:

#7

Since you already mention the learning from climate sciences. Here is one of the main sources for the meta information used in the netcdf files: http://cfconventions.org/

Maybe it’s worth having a look at to gain some inspiration on how this may apply to energy system models.

#8

There is an ongoing developing from the OEP community:

#9

Please use the wiki page:

https://wiki.openmod-initiative.org/wiki/Common_Data_Format_Energy

to upload files for now. Hopefully we will have a better structure in the next days…

#10

for IAMC format:
IAMC.ods (14.6 KB)

#11

The “criteria group” has found that the design philosophy of the frictionless data specifications already matches our “generic” data-related criteria and could be a good/mandatory starter. Moreover, we agreed that there is already a lot of literature about scenario factsheets, etc. and we should gather all information in a new wiki page or add it to an existing one. This might help in the process of finding a common standard.

#12

List of information, papers and standards of criteria :

#13

datapackage_in_OEP_metadata_v1.3.txt (1.5 KB)

#14

Should we have an other time slot on Thursday on this topic? What time slot would be best?

#15

see zip file from datapackage group below…

#16

I think we should have another slot, also for looking at the next steps (Wiki-page, etc) …

#17

I added the datapackage from yesterday:

datapackage.zip (1.3 KB)

#18

More information on the IAMC timeseries data standard can be found here: data.ene.iiasa.ac.at/databases

This standard has been used, amongst others, for the integrated-assessment scenario database compiled by IPCC Fifth Assessment Report (AR5), Working Group 3, see tntcat.iiasa.ac.at/AR5DB, and will also be used for the next IPCC reports.

#19

We now have a slot from 14:00 - 15:30 to continue on this one. We meet at Room D44 as listed in the schedule for today.

#20

Here is the netcdf4 version (jupyter notebook and datapackage as netcdf file) from yesterday.

netcdf4_datapackage.zip (2.8 KB)