View Issue Details

IDProjectCategoryView StatusLast Update
0001355OpenFOAMBugpublic2015-03-06 16:20
Reporteruser632Assigned Tohenry  
PrioritynormalSeveritymajorReproducibilityalways
Status resolvedResolutionno change required 
Summary0001355: Openfoam not writing time steps
DescriptionHi
It happen sometimes that when my time on supercomputer finishes and I decide I still have to run my simulations, some processors have one time step more than the the remaining processors. In order to restart, if I use start from latestTime, it gives error, so I remove the last time step from the processors having the additional time step so that all processors can have same time step. When simulations start running, Openfoam does not write time steps data. The controlDict remain same as earlier when it wrote data. This has happened numerous times and I have to delete my simulations and start from time step 0. This is a source of great frustration and a lot of time wastage as to complete one simulation, it takes more than 2 weeks. Kindly advise the reason.

Regards
Imad
Steps To ReproduceRun any simulation and if it stops with different time steps and you make all time steps same by removing the extra time step, data stops writing.
TagsNo tags attached.

Activities

wyldckat

2014-07-27 11:59

updater   ~0003176

Hi Imad,

Disclaimer: I'm just a fellow bug reporter.

If I understand you correctly, the problem is that the job scheduler on the supercomputer will terminate your simulation at an unexpected point in the simulation. To make things worse, it stops exactly when OpenFOAM was taking care of saving another time step, which would explain why the last time folder isn't saved properly for all processors.

OpenFOAM does provide a few ways to avoid such a situation in the first place, but it depends on the version you're using and what type of warnings the job scheduler on the supercomputer provides.

For example, as of OpenFOAM 2.2.0, it's possible to have a second write interval, which you can base on "cpuTime" or "clockTime": http://www.openfoam.org/version2.2.0/runtime-control.php
If the job scheduler supports a warning mechanism, where it writes a file indicating that the case should save and stop as soon as possible (preferably with a 60s or more warning in advance), then the function object "abortCalculation" can help you with that. This one is available since OpenFOAM 2.0.0.

As for not being able to continue a simulation from a previous time step, you'll need to provide more details about the problem you're having, namely which OpenFOAM version you're using and specific details on what kind of simulation you're doing.
Because if this is what I'm thinking about, it's possibly a bug that has already been fixed in the more recent OpenFOAM versions. For example, bug #382, which was fixed in OpenFOAM 2.1.x back in January 2012: http://www.openfoam.org/mantisbt/view.php?id=382

Best regards,
Bruno

user632

2014-07-28 04:15

  ~0003178

Last edited: 2014-07-28 04:17

Dear Bruno

Thanks for your comment. I know you from cfdonline. You have solved one key problem for me earlier. The problem is when my simulation stops at different time steps, I delete the last different time step from all processors and restart from latestTime as now all processors have the same time step. After this, my simulations continues to run for 10 or 11 hours without writing any time step whereas previously time steps appeared every 1 hour. The configurations are all same as before simulation stopped. Previously it was writing time steps. Now it does not. The disk has space. This has happened numerous times and when I have tried it sometimes on my own to resolve it, I have to delete the complete simulation and then re-run it from time 0 and then all time steps appear.

Best Regards
Imad

wyldckat

2014-12-29 11:34

updater   ~0003379

Dear Imad,

Only now did I manage to take a look into this again. A very critical detail is missing: Which exact OpenFOAM version did you use?

Best regards,
Bruno

wyldckat

2015-03-06 14:21

updater   ~0003991

From what I can remember and found again, Imad was using OpenFOAM 2.1.1, as indicated here: http://www.cfd-online.com/Forums/openfoam-solving/134568-foam-fatal-io-error-cannot-find-file.html#post494017 - post #14

As for the this exact report, I'm not certain how to reproduce this, but it seems related to this issue: http://www.openfoam.org/mantisbt/view.php?id=1557


For reference, Imad's previous issue (not exactly the one reported here, but related nonetheless) was solved with the instruction given here: http://www.cfd-online.com/Forums/openfoam-solving/134568-foam-fatal-io-error-cannot-find-file-2.html#post495259 post #22
Namely, the issue was that the time step was saved after OpenFOAM upped the time precision due to a need for balancing time steps with the Courant Number. The solution was to edit "controlDict" and increase the time precision.

This was why Imad was deleting folders, in an attempt to overcome the problem. Unfortunately back then I didn't manage to have enough time to gather all of the necessary information for reproducing this problem.

Issue History

Date Modified Username Field Change
2014-07-26 00:48 user632 New Issue
2014-07-27 11:59 wyldckat Note Added: 0003176
2014-07-28 04:15 user632 Note Added: 0003178
2014-07-28 04:17 user632 Note Edited: 0003178
2014-12-29 11:34 wyldckat Note Added: 0003379
2015-03-06 14:21 wyldckat Note Added: 0003991
2015-03-06 16:20 henry Status new => resolved
2015-03-06 16:20 henry Resolution open => no change required
2015-03-06 16:20 henry Assigned To => henry