Gfrktyl

Do you have reason to doubt their veracity or good faith? Are their claims somehow not believable or highly questionable based on your knowledge of the field? If it isn't standard in your field to release code I don't think a reviewer should necessarily demand it, regardless of your feelings about making code public.

The authors should describe their methodology sufficiently for someone else to replicate it; in that way, they are putting their reputations at risk that were someone to duplicate their approach they would find the same results. Fabricating results is a very serious accusation. There are some statistical approaches to test whether data are likely to be fabricated, but the efficacy of this approach depends on the sophistication of the fabrication, and that question is better suited to CrossValidated.

If their work is meaningful in the field, at some point someone will implement their approach again. There is necessarily a bit of trust in science that people do what they say they've done. A reviewer should check that a paper's conclusions follow logically from their results, and that their methodology is described and sound, but they need not verify or replicate the results. As David Thornley points out in a comment on another answer, even simply running code tells you very little about those key factors, it only tells you if there is a blatant lie or mistake.

You can't really judge if the simulation was really performed. That's why we've had things such as the Schön scandal - the reviewers of those manuscripts didn't detect the fraud either.

What you can do is implement the "smell test". Is this approach feasible? Are the results reasonable? Were there any glaring omissions? If you can't see any obvious problems with the simulation, that's good enough: the real peer review happens after publication.

After all these experiences, I can tell you there is no way to confirm the correctness of simulation results.

That is not necessarily true. In some cases, it is very easy to discern that a graph cannot possibly be correct or at the least has been badly misconstrued or misinterpreted. I had such a mistake caught in one of my early papers and have caught them in several papers I have reviewed.

It is not easy to prove that the simulations have actually been performed. However, the Open Science framework is designed to make it easier to verify results of both computational and experimental work.

there is no way to confirm the correctness of simulation results.

Simulations should be repeatable, hence, correctness can be checked by re-running the simulation. Of course, the authors might not provide the necessary code, but then you can request the code as a part of the review process.

If I were reviewing a paper which relied to a heavy degree on some computational analysis, but didn't supply the code, I would reject that paper unless the authors could give a good reason: even if they'd used proprietary libraries, I'd want to see the calls they made to those libraries. This is required by many journals in my field (genomics/bioinformatics).

That said, if the simulations ran for 2 months on 10,000 cores (or even a week on 200 cores), there is not much chance of me reproducing even with the code. And I almost certainly have neither the time, moneny nor expertise to repeat lab experiments in any paper I read.

I don't think that providing code, although a good practice, is a protection against dis-honesty. In the end there is very little protection against outright fraud, and the review process is not primarily there for that purpose.

I end up reviewing data science type papers where the underlying code is critical. I've started being that guy during reviews, and this is what I ask for:

1. Code must be available to me as a reviewer. Full stop.


2. The code needs to have tests that I as a reviewer can run. I can't go through the code to check and make sure it works exactly as advertised, but I can see if the tests are appropriate and that they pass when I run it.


3. The code should have reasonably good test coverage around the scientific parts, and regression tests should be reasonably justified (i.e. We expect to see this result because of reason X).


4. If you've re-invented a wheel you have to explain why (even if that explanation is that you didn't know someone else already made it)

These all seem pretty reasonable to me and address a lot of the concerns about code quality. Most people with well designed code will hit them all just by virtue of having continuous integration tests on a GitHub repository, and I think it's no longer necessary to coddle people with poorly designed code (which is too many people).

In order for simulation work to be precisely reproducible, it would be necessary to have (a) the code for the simulation and (b) the seed for the pseudoramdon generator used to run the code. Unless the code is proprietary, there is no good reason for authors to withhold this information from reviewers, even if the code will not be published as part of the paper. However, publishable simulation studies may be so extensive that it is not feasible even for an energetic skeptical reviewer to repeat the simulations.

To a degree many simulation studies can be self-verifying. When this is possible reviewers should insist on feasible inherent verification. In non-technical language here are a couple of examples of what I mean.

(1) Often a simulation will produce several results of which some are novel and some are easily obtained or generally known without resorting to simulation. Then at the very least a reviewer can can confirm that the latter results are valid. Somewhat similarly, simulations may refine results that can only be approximated by probabilistic or other mathematical computations. Then the reviewer can confirm that the results
are at least approximately correct.

(2) Very frequently, an important part of a simulation study may be to obtain approximate bounds within which the simulated results are likely (perhaps 95% likely) to lie. If it seems feasible to have obtained such bounds and the paper under review lacks them, then the reviewer should ask for them or for an explanation of their absence.

Addendum: This is a trivial example illustrating some concepts in (1) and (2). Suppose five dice are loaded so that faces 1 through 6 have respective probabilities (1/12, 1/6, 1/6, 1/6, 1/6, 1/4) of occurring. If all five are rolled, what is the probability that the total is at least 25? A simulation in R statistical software of a million such 5-die
experiments shows that the fraction of outcomes with totals 25 or more was 0.092903. Is this result believable? The answer is Yes, to about three places.

The simulated 95% margin of simulation error is "within 0.0006." It is easy to see that the average total is 19.583 and the corresponding simulated result is 19.580. A reasonable 2-place normal approximation is 0.0922.
This particular example is rich in corroborative
possibilities, but those are a few.

Note: Another issue is that, using various kinds of mathematical software, this problem could be solved exactly by advanced combinatorial methods. One exact method is based on this page, except that our dice are biased and outcomes are not equally likely. It is questionable whether simulations should be published if there is a tractable exact solution. One job of a reviewer is to identify papers that should not be published for this reason.

It is an unfortunate artifact of history and culture but academia is still in the dark ages with regards to sharing source code. Serious computational researchers will often provide at least a detailed algorithm, and often the code as well. But dabblers who had to learn how to code "on the job", or people very set in their ways and traditions, will often neglect to do so and publications standard rarely require it, even though you would think that anyone in their right mind would agree that all scientific code should be open source (or at least shared with reviewers) if it is to be worth discussion.

Unfortunately, the established culture does not expect sharing of code, even though the in the analogous case of physical experiments there is an expectation of sharing the exact process down to every detail of method and material such that it may be exactly reproduced by other researchers. I would suspect that the reason for this is that in the grand scheme of things computers are a relatively recent tool of science, and the ability to easily share code is still more recent. That said, we've had ubiquitous internet and zero effort code hosting like Github for over a decade now, so if you ask me, it's about damn time. But it looks like there is still quite a bit of inertia.

I have been reviewing the articles for various top-rank journals and conferences for the last few years. After all these experiences, I can tell you there is no way to confirm the correctness of simulation results. Therefore, I usually make comments on design, procedure, mathematical and analytical analysis.

That's about the best you can do. You can also try to intuitively conjecture based on the rough description (if any) of the computational approach, whether the results achieved are credible or not. But ultimately it is impossible to know for sure.

I try to add a little nag at the end of my reviews about releasing the source code, although I don't think it gets taken seriously very often.

In the results section, I can ask some questions about why this or that is so, but how can I judge if the simulation was really performed or these are just fabricated graphs?

Well, the way you phrase it, you can't really know if any graph or result is fabricated, unless maybe you were personally present while the research was done. There is inevitably an element of trust. But without source code, even if you do trust, you cannot offer meaningful critique about some computational parts of the paper. Obviously you can still comment on initial assumptions and the approach chosen. You can comment on how the results are interpreted. But the implementation itself is out of reach until you can see the code. Actually, even providing a detailed algorithm would not be sufficient: The authors' implementation may not necessarily be an exact match for the algorithm they intended.

This question came in my mind because I observed on few occasions, during review process, a reviewer ask for including new results, which in my opinion required a lot of coding and effort to implement, but author responded within 7-10 days with new results and improved article.

I don't think it's fair to be suspicious just because they did it a little too quickly. They may just be very good at coding. Personally, my development rate is very variable: Sometimes things just click and I can write code really fast, sometimes simple things take forever. They may know of easier ways to implement the change than you are aware. They may have already coded something similar in separate work and been able to repurpose it quickly.

If someone were to falsify results, I think they would either respond right away because they don't care, or wait "long enough" to avoid suspicion. If they bothered to wait at all, I don't think they would jeopardize the whole enterprise by waiting too little.

Usually known examples are reproduced to gain confidence that the simulation is doing what it claims. Then further simulations show the new results, which might not be reproduced by an other group because the did not use the described method or approach described in the paper/talk.

Due to brevity, often the first part is omitted in journal articles and conference talks. PhD theses contain them more often.

One of my first publications, it does not contain any breakthrough, is only cited from others to show that they match my results.