Chat over at Uncertain Principles has a post about the Status of Simulations in science. He focuses primarily on physics, his own field, but his jumping off point is from a question that was raised in the context of geology about whether the results of simulations can be considered “data”.
Before I get to the stuff Chad was talking about, I do want to note that the word “data”, much like “law”, has a number of different meanings. As we all have to be very careful about how we talk, as people are out there looking to misuse our words to attack the facts of evolution or climate change, we sometimes overdefine things. At one level, “data” just means a collection of numbers or facts, wherever it came from. If you have code, for instance, that is going to do calculations on some input, you might call the file you get it a data file.
If you want to be more precise and talk about the things out there in the natural world, as opposed to things we just calculated based on some model, you might use the terms “experimental data” or “observational data”. Even there, though, sometimes (as Chad notes) you include simulations in your basic processing of the data. For instance, when analyzing photometry from the Hubble Space Telescope (HST), in order to extract the highest precision possible results you might use simulations of the diffraction pattern of the telescope as part of your data processing.
So what is the status of simulations in astronomy? There’s quite a range. Typically, in astronomy and astrophysics, the theorists are more tightly coupled with the observers than is the case in many fields. Indeed, theorists often cross over and get involved in observing projects. In Chad’s field— roughly speaking, atomic, molecular, and optical (AMO) physics— sometimes the systems are simple enough that you can compare pencil-and-paper theory predictions to experiment. Even there, however, that’s not always the case. Even with single atoms more complicated than Hydrogen, it takes numerical approximations and intense computer calculations to figure out levels and transitions of the more excited states. In astronomy, however, you’re almost always dealing with a big complicated many-particle system. This isn’t always true; Kepler was able to compare data to very simple laws, as he was effectively dealing with two-particle systems.
Consider stellar evolution. We have a great theory of the structure and evolution of stars, that’s confirmed by a wide range of observations. However, in many cases it takes intense simulations to produce the values that are to be compared with experiment. This includes not only of the nuclear reactions at the core, but also of the transfer of energy from the core to the surface. Different theorists using different models will produce subtly different predictions as to the surface temperature and luminosity of a star. They agree broadly, but sometimes nowadays, especially with dimmer stars, the data is good enough that we’re pushing the limitations of the models. (The disagreement, however, is not at the level that we question the underlying theory. Rather, it means that we can’t be sure about, for example, the exact age of a given pre-main-sequence star given its color and luminosity, as different models give different values for the age.)
I do want to correct one thing that Chad says. He says that with observational sciences like astronomy, we’ve got just one system to look at. While it’s true that we can’t run controlled experiments in the same way that a laboratory science can, only in a few cases in astronomy do we only have one system to look at. For instance, if you’re modeling the Cosmic Microwave Background, or anything else to do with the Universe as a whole, we’ve only got one system to look at. However, if you’re looking at supernovae, or stars, or nebulae, or galaxy clusters, there are a lot of systems out there. You can do things such as divide your sample of observational targets randomly into two subsets. Use one subset of objects to determine any free parameters in your model, and then test the now-fixed model against another subset of the targets.
When a theory, or simulations based on a theory, made a range of predictions that are confirmed by observation or experiment, we start to take that theory and those simulations seriously even where we haven’t been able to directly confirm the predictions. We haven’t directly observed gravitational waves. (We’ve observed them indirectly in the orbital decay of a binary pulsar.) However, until we have convincing observational limits otherwise, we believe they exist. Why? Because they’re a prediction of General Relativity (GR), and GR is an extremely robust theory that’s stood up to a wide range of other tests.
I suspect that the motivation to accord some “status” to “data” as produced by simulations comes from scientists reacting to misplaced pedants who want to latch on to single words as a way of undermining science. This is what creationists do when saying “evolution is a theory, not a fact”. Yes, it’s true that evolution is a theory. This, however, does not undermine it in any way, if you understand what a theory is, and if you understand how well supported by observational evidence that particular theory is. Likewise, people might argue against the fact of anthropogenic climate change based on there “not being real data”, as some of the “data” comes out of models. I don’t think any scientist is at all unclear on the difference. Models produce predictions, and may produce large amounts of “data”. However, even if we use the term to describe those values, we don’t confuse this with the results of observations an experiments (either raw, or processed through models similar to the HST diffraction model mentioned above). So, yes, it’s “just” the results of models that if we continue on our current path, the global average temperature is going to increase by at least a certain amount in the next decades. However, that prediction is based on multiple different models of the climate, all of which have withstood some tests to indicate that we ought to take them seriously.