“Estimating uncertainty of model parameters obtained using numerical optimisation”
Authors: Ole Magnus Brastein, Bernt Lie, Carlos F. Pfeiffer and Nils-Olav Skeie,Affiliation: University of South-Eastern Norway
Reference: 2019, Vol 40, No 4, pp. 213-243.
Keywords: Parameter estimation, Uncertainty analysis, Bootstrapping, Profile Likelihood
Abstract: Obtaining accurate models that can predict the behaviour of dynamic systems is important for a variety of applications. Often, models contain parameters that are difficult to calculate from system descriptions. Hence, parameter estimation methods are important tools for creating dynamic system models. Almost all dynamic system models contain uncertainty, either epistemic, due to simplifications in the model, or aleatoric, due to inherent randomness in physical effects such as measurement noise. Hence, obtaining an estimate for the uncertainty of the estimated parameters, typically in the form of confidence limits, is an important part of any statistically solid estimation procedure. Some uncertainty estimation methods can also be used to analyse the practical and structural identifiability of the parameters, as well as parameter inter-dependency and the presence of local minima in the objective function. In this paper, selected methods for estimation and analysis of parameters are reviewed. The methods are compared and demonstrated on the basis of both simulated and real world calibration data for two different case models. Recommendations are given for what applications each of the methods are suitable for. Further, differences in requirements for system excitation are discussed for each of the methods. Finally, a novel adaption of the Profile Likelihood method applied to a moving window is used to test the consistency of dynamic information in the calibration data for a particular model structure.
PDF (19429 Kb)
DOI: 10.4173/mic.2019.4.3DOI forward links to this article:
| [1] O.M. Brastein, A. Ghaderi, C.F. Pfeiffer and N.-O. Skeie (2020), doi:10.1016/j.enbuild.2020.110236 |
| [2] Daniele Ronzani, Joris Gillis, Goele Pipeleers and Jan Swevers (2022), doi:10.1109/AMC51637.2022.9729282 |
| [3] Sarah Juricic, Jeanne Goffart, Simon Rouchier, Arnaud Jay and Pierre Oberl (2022), doi:10.1016/j.enbuild.2022.112211 |
| [4] Gaelle Faure, Sarah Juricic, Mickael Rabouille, Simon Rouchier, Arnaud Challansonnex and Arnaud Jay (2023), doi:10.1088/1742-6596/2654/1/012048 |
[1] Akaike, H. (1998). Akaike, H, Information theory and an extension of the maximum likelihood principle. In Selected papers of Hirotugu Akaike, pages 199--213. Springer. doi:10.1007/978-1-4612-1694-0_15
[2] Bacher, P. and Madsen, H. (2011). Bacher, P, and Madsen, H. Identifying suitable models for the heat dynamics of buildings. Energy and Buildings. 43(7):1511 -- 1522. doi:10.1016/j.enbuild.2011.02.005
[3] Bentley, J.P. (2005). Bentley, J, P. Principles of measurement systems. Pearson education. .
[4] Berthou, T., Stabat, P., Salvazet, R., and Marchio, D. (2014). Berthou, T, , Stabat, P., Salvazet, R., and Marchio, D. Development and validation of a gray box model to predict thermal behavior of occupied office buildings. Energy and Buildings. 74:91--100. doi:10.1016/j.enbuild.2014.01.038
[5] Bohlin, T. and Graebe, S.F. (1995). Bohlin, T, and Graebe, S.F. Issues in nonlinear stochastic grey box identification. International journal of adaptive control and signal processing. 9(6):465--490. doi:10.1002/acs.4480090603
[6] Brastein, O., Perera, D., Pfeiffer, C., and Skeie, N.-O. (2018). Brastein, O, , Perera, D., Pfeiffer, C., and Skeie, N.-O. Parameter estimation for grey-box models of building thermal behaviour. Energy and Buildings. 169:58 -- 68. doi:10.1016/j.enbuild.2018.03.057
[7] Brastein, O.M., Lie, B., Sharma, R., and Skeie, N.-O. (2019). Brastein, O, M., Lie, B., Sharma, R., and Skeie, N.-O. Parameter estimation for externally simulated thermal network models. Energy and Buildings, 2019. 191:200--210. doi:10.1016/j.enbuild.2019.03.018
[8] Brastein, O.M., Sharma, R., and Skeie, N.-O. (2019). Brastein, O, M., Sharma, R., and Skeie, N.-O. Sensor placement and parameter identifiability in grey-box models of building thermal behavior. In Proceedings of The 60th Conference on Simulation and Modelling (SIMS 60), 13-16 August 2019, Vaesteras, Sweeden. Linkoeping University Electronic Press, 2019. .
[9] Deconinck, A.-H. and Roels, S. (2017). Deconinck, A, -H. and Roels, S. Is stochastic grey-box modelling suited for physical properties estimation of building components from on-site measurements? Journal of Building Physics. 40(5):444--471. doi:10.1177/1744259116688384
[10] Efron, B. (1979). Efron, B, Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics. 7(1):1--26. doi:10.1007/978-1-4612-4380-9_41
[11] Ergon, R. and DiRuscio, D. (1997). Ergon, R, and DiRuscio, D. Dynamic system calibration by system identification methods. In European Control Conference (ECC), 1997. IEEE, pages 1556--1561. doi:10.23919/ECC.1997.7082324
[12] Farrell, J.A. and Polycarpou, M.M. (2006). Farrell, J, A. and Polycarpou, M.M. Adaptive approximation based control: unifying neural, fuzzy and traditional adaptive approximation approaches, volume48. John Wiley & Sons. .
[13] Ferrero, C.S., Chai, Q., DuenasDiez, M., Amrani, S.H., and Lie, B. (2006). Ferrero, C, S., Chai, Q., DuenasDiez, M., Amrani, S.H., and Lie, B. Systematic analysis of parameter identifiability for improved fitting of a biological wastewater model to experimental data. Modeling, Identification and Control. 27(4):219. doi:10.4173/mic.2006.4.2
[14] Fux, S.F., Ashouri, A., Benz, M.J., and Guzzella, L. (2014). Fux, S, F., Ashouri, A., Benz, M.J., and Guzzella, L. EKF based self-adaptive thermal model for a passive house. Energy and Buildings. 68:811--817. doi:10.1016/j.enbuild.2012.06.016
[15] Hoos, H.H. and Stutzle, T. (2004). Hoos, H, H. and Stutzle, T. Stochastic local search: Foundations and applications. Elsevier. .
[16] Jazwinski, A.H. (1970). Jazwinski, A, H. Stochastic processes and filtering theory. Dover Publications, Inc. .
[17] Johansson, R. (1993). Johansson, R, System Modeling and Identification. Information and system sciences series. Prentice Hall. .
[18] Johnson, R. and Wichern, D. (2007). Johnson, R, and Wichern, D. Applied Multivariate Statistical Analysis. Applied Multivariate Statistical Analysis. Pearson Prentice Hall, 2007. .
[19] Juhl, R., Moller, J.K., Jorgensen, J.B., and Madsen, H. (2016). Juhl, R, , Moller, J.K., Jorgensen, J.B., and Madsen, H. Modeling and prediction using stochastic differential equations. In Prediction Methods for Blood Glucose Concentration, pages 183--209. Springer, 2016. doi:10.1007/978-3-319-25913-0_10
[20] Juhl, R., Moller, J.K., and Madsen, H. (2016). Juhl, R, , Moller, J.K., and Madsen, H. ctsmr-Continuous Time Stochastic Modeling in R. arXiv preprint arXiv:1606.00242, 2016. .
[21] Killian, M. and Kozek, M. (2016). Killian, M, and Kozek, M. Ten questions concerning model predictive control for energy efficient buildings. Building and Environment. 105:403--412. doi:10.1016/j.buildenv.2016.05.034
[22] Kristensen, N.R. and Madsen, H. (2003). Kristensen, N, R. and Madsen, H. Continuous time stochastic modelling. Mathematics Guide. pages 1--32. .
[23] Kristensen, N.R., Madsen, H., and Jorgensen, S.B. (2004). Kristensen, N, R., Madsen, H., and Jorgensen, S.B. Parameter estimation in stochastic grey-box models. Automatica. 40(2):225--237. doi:10.1016/j.automatica.2003.10.001
[24] Kullback, S. (1939). Kullback, S, A Note on Neyman's Theory of Statistical Estimation. The Annals of Mathematical Statistics. 10(4):388--390. https://www.jstor.org/stable/2235617. .
[25] Kunsch, H.R. (1989). Kunsch, H, R. The jackknife and the bootstrap for general stationary observations. The Annals of Statistics. pages 1217--1241. https://www.jstor.org/stable/2241719. .
[26] Lie, B. (2009). Lie, B, Model uncertainty and control consequences: a paper machine study. Mathematical and Computer Modelling of Dynamical Systems, 2009. 15(5):463--477. doi:10.1080/13873950903375452
[27] Ljung, L. (1999). Ljung, L, System Identification: Theory for the User. Prentice Hall information and system sciences series. Prentice Hall PTR. .
[28] Lodhi, H. and Gilbert, D. (2011). Lodhi, H, and Gilbert, D. Bootstrapping parameter estimation in dynamic systems. In International Conference on Discovery Science. Springer, pages 194--208. doi:10.1007/978-3-642-24477-3_17
[29] Madsen, H. (2007). Madsen, H, Time series analysis. Chapman and Hall/CRC. .
[30] Madsen, H. and Holst, J. (1995). Madsen, H, and Holst, J. Estimation of continuous-time models for the heat dynamics of a building. Energy and buildings. 22(1):67--79. doi:10.1016/0378-7788(94)00904-X
[31] Maiwald, T. and Timmer, J. (2008). Maiwald, T, and Timmer, J. Dynamical modeling and multi-experiment fitting with PottersWheel. Bioinformatics. 24(18):2037--2043. doi:10.1093/bioinformatics/btn350
[32] Meeker, W.Q. and Escobar, L.A. (1995). Meeker, W, Q. and Escobar, L.A. Teaching about approximate confidence regions based on maximum likelihood estimation. The American Statistician. 49(1):48--53. doi:10.1080/00031305.1995.10476112
[33] Murphy, S.A. and Vander Vaart, A.W. (2000). Murphy, S, A. and Vander Vaart, A.W. On profile likelihood. Journal of the American Statistical Association. 95(450):449--465. doi:10.1080/01621459.2000.10474219
[34] Neyman, J. (1937). Neyman, J, Outline of a theory of statistical estimation based on the classical theory of probability. Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences. 236(767):333--380. doi:10.1098/rsta.1937.0005
[35] Nocedal, J. and Wright, S. (2006). Nocedal, J, and Wright, S. Numerical optimization. Springer Science & Business Media. .
[36] Pohjanpalo, H. (1978). Pohjanpalo, H, System identifiability based on the power series expansion of the solution. Mathematical Biosciences. 41(1):21--33. doi:10.1016/0025-5564(78)90063-9
[37] Politis, D.N. (2003). Politis, D, N. The impact of bootstrap methods on time series analysis. Statistical Science. pages 219--230. https://www.jstor.org/stable/3182852. .
[38] Politis, D.N. and Romano, J.P. (1994). Politis, D, N. and Romano, J.P. The stationary bootstrap. Journal of the American Statistical association. 89(428):1303--1313. doi:10.1080/01621459.1994.10476870
[39] Press, W.H., Teukolsky, S.A., Vetterling, W.T., and Flannery, B.P. (2007). Press, W, H., Teukolsky, S.A., Vetterling, W.T., and Flannery, B.P. Numerical recipes in C++, volume3. Cambridge University Press. .
[40] Raue, A., Kreutz, C., Maiwald, T., Bachmann, J., Schilling, M., Klingmuller, U., and Timmer, J. (2009). Raue, A, , Kreutz, C., Maiwald, T., Bachmann, J., Schilling, M., Klingmuller, U., and Timmer, J. Structural and practical identifiability analysis of partially observed dynamical models by exploiting the profile likelihood. Bioinformatics. 25(15):1923--1929. doi:10.1093/bioinformatics/btp358
[41] Rosen, R., Wichert, G.v., Lo, G., and Bettenhausen, K.D. (2015). Rosen, R, , Wichert, G.v., Lo, G., and Bettenhausen, K.D. About the importance of autonomy and digital twins for the future of manufacturing. 15th IFAC Symposium onInformation Control Problems in Manufacturing. IFAC-PapersOnLine. 48(3):567 -- 572. doi:10.1016/j.ifacol.2015.06.141
[42] Rossi, R.J. (2018). Rossi, R, J. Mathematical Statistics: An Introduction to Likelihood Based Inference. John Wiley & Sons. .
[43] Runge, C. (1895). Runge, C, Ueber die numerische Aufloesung von Differentialgleichungen. Mathematische Annalen. 46(2):167--178. doi:10.1007/BF01446807
[44] Simon, D. (2006). Simon, D, Optimal state estimation: Kalman, H infinity, and nonlinear approaches. John Wiley & Sons. .
[45] Venzon, D. and Moolgavkar, S. (1988). Venzon, D, and Moolgavkar, S. A method for computing profile-likelihood-based confidence intervals. Applied statistics. pages 87--94. doi:10.2307/2347496
[46] Wang, L. (2009). Wang, L, Model predictive control system design and implementation using MATLAB. Springer Science & Business Media. .
[47] Wilks, S.S. (1938). Wilks, S, S. The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses. The Annals of Mathematical Statistics. 9(1):60--62. https://www.jstor.org/stable/2957648. .
BibTeX:
@article{MIC-2019-4-3,
title={{Estimating uncertainty of model parameters obtained using numerical optimisation}},
author={Brastein, Ole Magnus and Lie, Bernt and Pfeiffer, Carlos F. and Skeie, Nils-Olav},
journal={Modeling, Identification and Control},
volume={40},
number={4},
pages={213--243},
year={2019},
doi={10.4173/mic.2019.4.3},
publisher={Norwegian Society of Automatic Control}
};