View Issue Details

IDProjectCategoryView StatusLast Update
0000223OpenFOAMBugpublic2012-02-28 20:04
Reporteruser47Assigned Touser2 
PriorityurgentSeveritymajorReproducibilityalways
Status resolvedResolutionfixed 
PlatformLinuxOSArch LinuxOS Versionx86_64
Summary0000223: Multi processor runs return different results than single processor runs
DescriptionRunning multiprocessor simulations with cyclic boundary conditions return different results than the same case run on a single processor.

The problem occurs when two cyclics linked together are decomposed to run on different processor.

For example, taking the channel flow test case joined as attachment, when inout1 has its "inlet" residing on one processor, and its "outlet" on an other one. So this could be a simple (2 1 1) decomposition.
Taking a simple (1 2 1) decomposition, so having corresponding cyclics on the same processor, yields much more accurate results.

The variation (U_1 - U_2) between the two velocity fields at the same time steps are quite significant:
- 0.2% of Umax for cases where corresponding cyclics are on different processors
- compared to 0.0002% for the cases where corresponding cyclics are on the same processor.

It's interesting to note that the mean Courant number returned from the very first timeStep (and onwards of course) is different between parallel and single processor runs.
Steps To Reproduce- Extract test case
- Run blockMesh
- Compare parallel results to single processor results

Additional InformationThis problem was discussed and acknowledged by Hrvoje a bit less than a month ago when he was on visit at our university.

On his advice, the tests were run using the "diagonal" preconditioner, because it was the only preconditioner which could guarantee identical solution between single and parallel runs.

I haven't be able to join him since then, so you might want to contact him to know whether he has made any progress on the issue.

TagsNo tags attached.

Activities

user47

2011-06-20 16:41

 

testCase.tgz (796,516 bytes)

henry

2011-06-20 16:47

manager   ~0000464

Can you reproduce the same behavior with OpenFOAM-2.0.x? If so send the test case and we will investigate.

user47

2011-06-20 22:17

  ~0000465

Last edited: 2011-06-20 22:18

Dear Henry,

Thank you for your swift reply!

I will give it a try, but haven't installed OpenFOAM-2.0.x yet, so it might take a day or two. I'll come back to you as soon as possible.

Kind regards,

Francois.

user71

2011-06-21 12:43

  ~0000470

I've just tried Francois's test case with OpenFoam 2.0.x using DIC preconditioner
(i've uncommented corresponding lines in fvSolution) on one and two processors and got the relative difference in velocity up to 0.37% at time 1 (mag(U-Ua)/mag(U))

I am very interested in this issue too.

Kind regards,
Ilya.

user47

2011-06-21 23:11

  ~0000472

Dear Henry,

After compilation and runs with OpenFOAM-2.0.x, the errors returned are exactly the same.
This said, I used foamUpgradeCyclics to convert my 1.7.x test cases to 2.0.x cases, and I haven't had time to investigate whether that was enough to make full use of the new cyclic implementation in 2.0.x.

Just for information, here are the difference max( mag( U1 - U2)/mag(U1) ) for different decompositions on 2 processors, compared to a single processor run, after a 1 second run:
- simple( 2 1 1 ) : 0.538 %
- simple( 1 2 1 ) : 0.0009 %
- simple( 1 1 2 ) : 0.1858 %

Kind regards,

Francois.

user47

2011-07-02 16:14

  ~0000504

Dear Henry,

Further changing the Cyclics definition to the pure 2.0.x standard using matchTolerance 0.0001 instead of featureCos 0.9 yields almost exactly the same results:
- simple( 2 1 1 ) : 0.5345 %
- simple( 1 2 1 ) : 0.0009 %
- simple( 1 1 2 ) : 0.1855 %

I was therefore wondering whether any progress was made towards solving this issue?

Kind regards,

Francois.

henry

2012-01-03 18:45

manager   ~0000894

The case you are running is chaotic at the small scales and the evolution of the turbulent structures is dependent on the details of the numerics, round-off errors etc. and indeed the way in which the domain is decomposed. What is important is that the statistics generated are independent of the decomposition, i.e. the means and variances. However, the case as provided runs for far too short at time to produce reliable means; have you tried to run for much longer and compared the mean velocity field?

user2

2012-02-28 12:01

  ~0001079

orphaned report - closing

user47

2012-02-28 14:35

  ~0001084

Last edited: 2012-02-28 14:36

Mm.. I had almost forgotten I finally got feedback om my bug post.. Sorry about that.

But as a reply, I should say that yes, I did take longer time averages, anticipating that the problem might come from slight differences in initial conditions.. However, after 30 seconds averaging and 150'000 iterations with a CFLmax of 0.1, there were still differences between single and multi-processor runs. See attachment.

Kind regards,
Francois.

user47

2012-02-28 14:35

 

update-28-02-2012.pdf (97,569 bytes)

henry

2012-02-28 15:40

manager   ~0001087

How big are the differences? Are they getting smaller with averaging? Where in the domain are they?

user47

2012-02-28 18:50

  ~0001094

The differences are in the order of a few percent. Somewhere between 0 and 4% away from the wall, and around 10% close to the wall.
The differences to tend to get smaller with averaging, in the order of 0.1 to 0.2% per 5 seconds averaging window, although I don't know what the trend becomes after 30 seconds averaging. This because it already took me more then a month of computation time to get that far on 1 processor, so I stopped the computation at that point.

Kind regards, Francois.

henry

2012-02-28 20:04

manager   ~0001095

It would appear that the difference tends to zero with averaging as expected.

Issue History

Date Modified Username Field Change
2011-06-20 16:41 user47 New Issue
2011-06-20 16:41 user47 File Added: testCase.tgz
2011-06-20 16:47 henry Note Added: 0000464
2011-06-20 22:17 user47 Note Added: 0000465
2011-06-20 22:18 user47 Note Edited: 0000465
2011-06-21 12:43 user71 Note Added: 0000470
2011-06-21 23:11 user47 Note Added: 0000472
2011-07-02 16:14 user47 Note Added: 0000504
2012-01-03 18:45 henry Note Added: 0000894
2012-02-28 12:01 user2 Note Added: 0001079
2012-02-28 12:01 user2 Status new => closed
2012-02-28 12:01 user2 Assigned To => user2
2012-02-28 12:01 user2 Resolution open => no change required
2012-02-28 14:35 user47 Note Added: 0001084
2012-02-28 14:35 user47 Status closed => feedback
2012-02-28 14:35 user47 Resolution no change required => reopened
2012-02-28 14:35 user47 File Added: update-28-02-2012.pdf
2012-02-28 14:36 user47 Note Edited: 0001084
2012-02-28 15:40 henry Note Added: 0001087
2012-02-28 18:50 user47 Note Added: 0001094
2012-02-28 18:50 user47 Status feedback => assigned
2012-02-28 20:04 henry Note Added: 0001095
2012-02-28 20:04 henry Status assigned => resolved
2012-02-28 20:04 henry Resolution reopened => fixed