Quantifying the benefits of open energy modelling?

Dear openmod community,

This is kind of a naive, and most certainly a wide question, but I was wondering: how can we qualify and quantify the benefits of open energy modelling ? In other words: how to assess that the open energy modelling we put in practice is usable and useful ? Do you have methods or examples of such work ? For instance:

  • Surveys towards the models developers / users / results recipients ?
  • Criteria on the tools themselves, such as the work of Cao et al. [1] or Groissbock [2] ?
  • …?

To be precise, I am not asking about open energy modelling interests, limits or good practices: I think I now have a pretty good understanding of these after the truly interesting presentations and exchanges we had in Berlin at the openmod workshop and on the forum; plus thanks to the reading of articles such as [3-6]. To rephrase it once again, my question would be: how to quantify the identified interests of such good practices ?

Thanks a lot !


  1. Cao, Karl-Kiên, Felix Cebulla, Jonatan J Gómez Vilchez, Babak Mousavi, and Sigrid Prehofer (28 September 2016). “Raising awareness in model-based energy scenario studies — a transparency checklist”. Energy, Sustainability and Society. 6 (1): 28. ISSN 2192-0567. doi:10.1186/s13705-016-0090-z. Open access.

  2. Groissböck, Markus (1 March 2019). “Are open source energy system optimization tools mature enough for serious use?”. Renewable and Sustainable Energy Reviews. 102: 234–248. ISSN 1364-0321. doi:10.1016/j.rser.2018.11.020. Closed access.

  3. Pfenninger, Stefan, Joseph F DeCarolis, Lion Hirth, Sylvain Quoilin, and Iain Staffell (February 2017). “The importance of open data and software: is energy research lagging behind?”. Energy Policy. 101: 211–215. ISSN 0301-4215. doi:10.1016/j.enpol.2016.11.046. Open access.

  4. Bazilian, Morgan, Andrew Rice, Juliana Rotich, Mark Howells, Joseph DeCarolis, Stuart Macmillan, Cameron Brooks, Florian Bauer, and Michael Liebreich (1 October 2012). “Open source software and crowdsourcing for energy analysis”. Energy Policy. 49: 149–153. ISSN 0301-4215. doi:10.1016/j.enpol.2012.06.032. Working draft.

  5. Morrison, Robbie (April 2018). “Energy system modeling: public transparency, scientific reproducibility, and open development”. Energy Strategy Reviews. 20: 49–63. ISSN 2211-467X. doi:10.1016/j.esr.2017.12.010. Open access.

  6. Pfenninger, Stefan, Lion Hirth, Ingmar Schlecht, Eva Schmid, Frauke Wiese, Tom Brown, Chris Davis, Matthew Gidden, Heidi Heinrichs, Clara Heuberger, Simon Hilpert, Uwe Krien, Carsten Matke, Arjuna Nebel, Robbie Morrison, Berit Müller, Guido Pleßmann, Matthias Reeg, Jörn C Richstein, Abhishek Shivakumar, Iain Staffell, Tim Tröndle, and Clemens Wingenbach (2017). “Opening the black box of energy modelling: strategies and lessons learned”. Energy Strategy Reviews. 19: 63–71. ISSN 2211-467X. doi:10.1016/j.esr.2017.12.002. Open access.

PS : do not hesitate to ask me precision if this is not clear, or to ask me to change tags / category if necessary (I was not too sure about where to ask this).

1 Like

The jury is still out but not for long?

First I would like to throw energy system data into the mix. The gains from sharing and curating open data collectively are self‑evident. And, as will be seen, it is often hard to split models from data in any case.

This blog looks the relative merits of open versus non‑open energy system modeling in the context of public policy development. It sidesteps other considerations to focus on the “efficacy” of the various approaches in use today.1 And I would say that, under current practices, there is no clear conclusion as to which approach performs best.

Established policy models are often constituted as excludable economic clubs, typically with five figure annual membership fees, very significant contracts from public institutions, and dedicated inputs from publicly-funded science. Examples include the IEA World Energy Model (WEM), MARKAL/TIMES models, WEC models, and the GTAP database and software. And these clubs usually cover code and data in roughly equal measure.

There are relatively few single institution closed models that are also influential. The key example would be PRIMES from NTUA, used by the European Commission.2,3,4 These projects are more akin to proprietary software development than the consortiums just listed. And their documentation and drill‑down transparency tends to be patchy at best. The German government is unusual in that it makes wide use of single institution models from both research agencies and dedicated policy consultancies.

The US government National Energy Modeling System (NEMS) is one of the few examples of a national government maintaining a substantial in‑house project. And although technically the core code is public domain, it is not possible for external parties to build and run NEMS — nor is that use case intended or supported.

On the other side of the fence, one sees the emergence of robust open modeling projects making substantial efforts to engage in good software practices. Retrospectively in the case of OSeMOSYS — which has forked all over the place since its early beginnings. And from the outset in the case of oemof. These are the only two open projects I am going to mention by name, so don’t treat their appearance here as an endorsement. See Muschner (2020) for a comparison of the two frameworks.

Another consideration is the uptake of open models by small or less developed countries that cannot easily sign on for the club or closed approaches. See my comments here for some specific examples.

The one area where open source development is lagging well behind is optimization solvers. The open source GLPK and cbc solvers are clearly inferior to their commercial counterparts, such as Gurobi and CPLEX, by a wide margin.

Also interesting to note that many of the previous club projects are becoming open incrementally. Knowledge bases from the World Bank Group provide one example, for instance energydata.info. In addition, the European Commission is steadily migrating from closed models like PRIMES to open models like Dispa‑SET. But the Commission is also funding the “open book” METIS suite of models whereby external parties undertake the core development and provide the Commission with the legally encumbered source code to run in‑house 5 — not a practice I warm to.

That trend toward openness is often tempered by withholding or overpricing necessary workflow tooling, post-processing software, databases, and other essential components. Which raises questions about the underlying motivations for publishing the core code in the first place — is it merely a response to public pressure for better optics?

The bulk of open energy system model development occurs within universities and there are still unmet challenges to improve the quality of academic software. The Research Software Engineering (RSE) initiative is one entity pursuing this issue and the adoption of open source development practices is a key theme.

To conclude, I deliberately replaced “benefits” with “efficacy” to narrow the discussion. And even so, it is hard to say whether club and closed development outperforms open development or vice versa. But what is clear is that energy modeling for policy support is heading open for many reasons, most of which do not involve development efficacy. Of course I am happy with that shift and the underlying logic. And I believe the gains from open development and community curation more generally — as evidenced by projects like the Linux kernel and OpenStreetMap — will arrive quite soon. And when they do, there will be no turning back from the advantages that accrue from collective open development beyond some indeterminate tipping point.

Feel free to add your views and reactions! R


  1. Wiktionary defines “efficacy” as the “ability to produce a desired effect under ideal testing conditions”.

  2. European Commission (23 November 2016). Modelling tools for EU analysis. Climate Action — European Commission. Accessed 14 November 2020.

  3. The last model description of PRIMES to my knowledge is: E3MLab (April 2015). PRIMES model 2013‑2014: detailed model description. Athens, Greece: E3MLab/ICCS at National Technical University of Athens. Publication date from PDF metadata.

  4. Capros, Pantelis, Maria Kannavou, Stavroula Evangelopoulou, Apostolos Petropoulos, Pelopidas Siskos, Nikolaos Tasios, Georgios Zazias, and Alessia DeVita (1 November 2018). “Outlook of the EU energy system up to 2050: the case of scenarios prepared for European Commission’s “clean energy for all Europeans” package using the PRIMES model”. Energy Strategy Reviews. 22: 255–263. ISSN 2211-467X. doi:10.1016/j.esr.2018.06.009. Closed access. USD 31.50 to purchase.

  5. Barberi, Paul, Paul Khallouf, Tobias Bossmann, and Laurent Fournié (May 2020). Introduction to METIS models. Brussels, Belgium: European Commission. Directorate-General for Energy.


Muschner, Christoph (2020). An Open Source Energy Modelling Framework Comparison of OSeMOSYS and oemof (MSc). Stockholm: KTH.


For those who need numbers, an earlier posting on code resue within our community:

Thank you very much for this thorough and interesting overview of the current situation regarding the open energy modelling and data initiatives efficacy.

I agree that the “gains from sharing and curating open data collectively are self‑evident”, as well as other gains from open energy modelling practices: long-term availability of the source code, … But from another standing point (probably the one of the devil’s advocate), if the improved reproducibility / credibility / cooperation enabled by open practices seems obvious, has it been proven ?
I think we have plenty of arguments and experiences underlining the gains from open practices compared to private ones : for instance if the IEA reports are available resources, its data and if I am not mistaken, its tools are not open and rarely presented, which do not enable a full understanding of the results and interpretations without any access to the detailed assumptions.

Going further with the idea of qualifying and quantifying the benefits of open energy modelling, we could imagine (or has it already been undertaken ?) sort of a social experiment where we present the results from private and open processes to an uninvolved third party and gather its feedback. The third part could be an energy modeller or a policy maker for instance (expert and non-expert reader in the sense of Cao et al. [1]).

Finally regarding the efficacy, perhaps it can be interesting to differentiate:

  • the performance of the tool or the method, i.e. the efficacy in the run time, with a clear advantage for private optimization solvers,
  • the efficacy for the reader, i.e. the efficacy in the design time (scenario construction) and in the results understanding, where open practices seem relevant and useful.

Here as well, feel free to add your views and reactions!

The International Energy Agency (IEA) executive summary (IEA November 2020) is free‑of‑charge to download but remains under full copyright. The underlying report costs €120 per PDF (and making and distributing copies is breach of copyright and actionable). The World Energy Model (WEM) is closed source but documented (IEA January 2020). At least some of the data used remains subject to IEA Policies and Measures Databases (PAMS) terms and conditions and is available to participating government agencies on those terms but not to citizens.

The IEA has certainly been strong in advocating for rapid decarbonization (after earlier issues involving US political influence, see wikipedia). For instance, IEA’s executive director, Fatih Birol recently opined in April this year (Ambrose 2020):

The plunge in demand for nearly all major fuels is staggering, especially for coal, oil and gas. Only renewables are holding up during the previously unheard of slump in electricity use.

The point is not accuracy or even efficacy, but rather that public interest analysis of this nature should not be undertaken behind closed doors using private information.


Ambrose, Jillian (30 April 2020). “Covid-19 crisis will wipe out demand for fossil fuels, says IEA”. The Guardian. London, United Kingdom. ISSN 0261-3077.

International Energy Agency (November 2020). World energy outlook — Executive summary. Paris, France: IEA Publications.

IEA (16 January 2020). World Energy Model documentation — 2019 version. Paris, France: International Energy Agency (IEA). Publication date from PDF metadata.

A very interesting topic. Thank you for a lively discussion! A few thoughts related to development in general.

Going further with the idea of qualifying and quantifying the benefits of open energy modelling, we could imagine (or has it already been undertaken ?) sort of a social experiment where we present the results from private and open processes to an uninvolved third party and gather its feedback.

It was a brilliant talk during FOSDEM 2020 about benefits of the open source in general and General Public Licenses in particular for business. Frank Karlitschek has mentioned something similar to such an experiment comparing development of the proprietary ownCloud and an open source Nextcloud (40:00 to 42:00 of the talk recording). One of the key benefits revealed was the developers’ attitude. They were clearly more happy working in the open source project.

I have checked ScienceDirect to find that the open source continues to gain popularity among researchers:

Joined with developers’ happiness, we can hope that this quantity will sooner or later result in quality, can’t we? :wink:

1 Like