View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000223OpenFOAM[All Projects] Bugpublic2011-06-20 15:412012-02-28 20:04
Assigned Toandy 
PlatformLinuxOSArch LinuxOS Versionx86_64
Product Version 
Target VersionFixed in Version 
Summary0000223: Multi processor runs return different results than single processor runs
DescriptionRunning multiprocessor simulations with cyclic boundary conditions return different results than the same case run on a single processor.

The problem occurs when two cyclics linked together are decomposed to run on different processor.

For example, taking the channel flow test case joined as attachment, when inout1 has its "inlet" residing on one processor, and its "outlet" on an other one. So this could be a simple (2 1 1) decomposition.
Taking a simple (1 2 1) decomposition, so having corresponding cyclics on the same processor, yields much more accurate results.

The variation (U_1 - U_2) between the two velocity fields at the same time steps are quite significant:
- 0.2% of Umax for cases where corresponding cyclics are on different processors
- compared to 0.0002% for the cases where corresponding cyclics are on the same processor.

It's interesting to note that the mean Courant number returned from the very first timeStep (and onwards of course) is different between parallel and single processor runs.
Steps To Reproduce- Extract test case
- Run blockMesh
- Compare parallel results to single processor results

Additional InformationThis problem was discussed and acknowledged by Hrvoje a bit less than a month ago when he was on visit at our university.

On his advice, the tests were run using the "diagonal" preconditioner, because it was the only preconditioner which could guarantee identical solution between single and parallel runs.

I haven't be able to join him since then, so you might want to contact him to know whether he has made any progress on the issue.

TagsNo tags attached.
Attached Filestgz file icon testCase.tgz [^] (796,516 bytes) 2011-06-20 15:41
pdf file icon update-28-02-2012.pdf [^] (97,569 bytes) 2012-02-28 14:35

- Relationships

-  Notes
henry (manager)
2011-06-20 15:47

Can you reproduce the same behavior with OpenFOAM-2.0.x? If so send the test case and we will investigate.
ftpronk (reporter)
2011-06-20 21:17
edited on: 2011-06-20 21:18

Dear Henry,

Thank you for your swift reply!

I will give it a try, but haven't installed OpenFOAM-2.0.x yet, so it might take a day or two. I'll come back to you as soon as possible.

Kind regards,


ipopov (reporter)
2011-06-21 11:43

I've just tried Francois's test case with OpenFoam 2.0.x using DIC preconditioner
(i've uncommented corresponding lines in fvSolution) on one and two processors and got the relative difference in velocity up to 0.37% at time 1 (mag(U-Ua)/mag(U))

I am very interested in this issue too.

Kind regards,
ftpronk (reporter)
2011-06-21 22:11

Dear Henry,

After compilation and runs with OpenFOAM-2.0.x, the errors returned are exactly the same.
This said, I used foamUpgradeCyclics to convert my 1.7.x test cases to 2.0.x cases, and I haven't had time to investigate whether that was enough to make full use of the new cyclic implementation in 2.0.x.

Just for information, here are the difference max( mag( U1 - U2)/mag(U1) ) for different decompositions on 2 processors, compared to a single processor run, after a 1 second run:
- simple( 2 1 1 ) : 0.538 %
- simple( 1 2 1 ) : 0.0009 %
- simple( 1 1 2 ) : 0.1858 %

Kind regards,

ftpronk (reporter)
2011-07-02 15:14

Dear Henry,

Further changing the Cyclics definition to the pure 2.0.x standard using matchTolerance 0.0001 instead of featureCos 0.9 yields almost exactly the same results:
- simple( 2 1 1 ) : 0.5345 %
- simple( 1 2 1 ) : 0.0009 %
- simple( 1 1 2 ) : 0.1855 %

I was therefore wondering whether any progress was made towards solving this issue?

Kind regards,

henry (manager)
2012-01-03 18:45

The case you are running is chaotic at the small scales and the evolution of the turbulent structures is dependent on the details of the numerics, round-off errors etc. and indeed the way in which the domain is decomposed. What is important is that the statistics generated are independent of the decomposition, i.e. the means and variances. However, the case as provided runs for far too short at time to produce reliable means; have you tried to run for much longer and compared the mean velocity field?
andy (reporter)
2012-02-28 12:01

orphaned report - closing
ftpronk (reporter)
2012-02-28 14:35
edited on: 2012-02-28 14:36

Mm.. I had almost forgotten I finally got feedback om my bug post.. Sorry about that.

But as a reply, I should say that yes, I did take longer time averages, anticipating that the problem might come from slight differences in initial conditions.. However, after 30 seconds averaging and 150'000 iterations with a CFLmax of 0.1, there were still differences between single and multi-processor runs. See attachment.

Kind regards,

henry (manager)
2012-02-28 15:40

How big are the differences? Are they getting smaller with averaging? Where in the domain are they?
ftpronk (reporter)
2012-02-28 18:50

The differences are in the order of a few percent. Somewhere between 0 and 4% away from the wall, and around 10% close to the wall.
The differences to tend to get smaller with averaging, in the order of 0.1 to 0.2% per 5 seconds averaging window, although I don't know what the trend becomes after 30 seconds averaging. This because it already took me more then a month of computation time to get that far on 1 processor, so I stopped the computation at that point.

Kind regards, Francois.
henry (manager)
2012-02-28 20:04

It would appear that the difference tends to zero with averaging as expected.

- Issue History
Date Modified Username Field Change
2011-06-20 15:41 ftpronk New Issue
2011-06-20 15:41 ftpronk File Added: testCase.tgz
2011-06-20 15:47 henry Note Added: 0000464
2011-06-20 21:17 ftpronk Note Added: 0000465
2011-06-20 21:18 ftpronk Note Edited: 0000465 View Revisions
2011-06-21 11:43 ipopov Note Added: 0000470
2011-06-21 22:11 ftpronk Note Added: 0000472
2011-07-02 15:14 ftpronk Note Added: 0000504
2012-01-03 18:45 henry Note Added: 0000894
2012-02-28 12:01 andy Note Added: 0001079
2012-02-28 12:01 andy Status new => closed
2012-02-28 12:01 andy Assigned To => andy
2012-02-28 12:01 andy Resolution open => no change required
2012-02-28 14:35 ftpronk Note Added: 0001084
2012-02-28 14:35 ftpronk Status closed => feedback
2012-02-28 14:35 ftpronk Resolution no change required => reopened
2012-02-28 14:35 ftpronk File Added: update-28-02-2012.pdf
2012-02-28 14:36 ftpronk Note Edited: 0001084 View Revisions
2012-02-28 15:40 henry Note Added: 0001087
2012-02-28 18:50 ftpronk Note Added: 0001094
2012-02-28 18:50 ftpronk Status feedback => assigned
2012-02-28 20:04 henry Note Added: 0001095
2012-02-28 20:04 henry Status assigned => resolved
2012-02-28 20:04 henry Resolution reopened => fixed