View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0001224OpenFOAMBugpublic2014-03-16 12:452015-03-05 19:06
Reporterzordiack Assigned Tohenry  
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionno change required 
PlatformLinux 64bitOSRed HatOS Version(please specify)
Summary0001224: Number of decomposition domains is affecting pimpleDyMFoam solution with cyclicAMI and scotch
DescriptionI have a turbomachinery case which has two cyclicAMI patches and a rotating mesh. I have run the case with adjustableTimeStep and maxCo 1.0 with decomposition method scotch and n=12 successfully for four days and the timestep seems to be 2..3e-6 seconds.

I uploaded the exact same case to a cluster to speed up the calculation and found out that the calculation crashes if I use more than n=32 domains. Nothing else has been changed in the case. OpenFOAM version is the same, but cluster is using IntelMPI instead of OpenMPI. The case is running now in the cluster with n=32 (I also tested with n=16) and the timestep seems to be around 1.4e-6 seconds.

Either scotch or cyclicAMI is breaking the solution if try to increase the domain count, n=48 leads to a crash and n=64 leads to a crash even sooner. All the other settings are the same, only the number of domains is being altered.

Using preservePatches in decomposeParDict didn't help. I strongly believe that the number of decomposition domains shouldn't affect the solution in any way.
Steps To ReproduceTry to use scotch decomposition method and cyclicAMI patches for a parallel case with n > 32.

The case is quite big (5 million cells), but I can upload it if needed.
Additional InformationCluster OpenFOAM version is 2.3.x-c51d53a9eadd and it was build with IntelMPI 4.1.3 and ThirdParty Gcc 4.8.2.

AMI patches are:

    rotor_inlet
    {
        nFaces 4992;
        startFace 16303989;
        type cyclicAMI;
        inGroups 1 ( cyclicAMI );
        matchTolerance 0.0001;
        transform noOrdering;
        neighbourPatch pipe_outlet;
    }

    pipe_outlet
    {
        nFaces 4976;
        startFace 16421195;
        type cyclicAMI;
        inGroups 1 (cyclicAMI );
        matchTolerance 0.0001;
        transform noOrdering;
        neighbourPatch rotor_inlet;
    }

    volute_inlet
    {
        nFaces 22460;
        startFace 16467763;
        type cyclicAMI;
        inGroups 1 ( cyclicAMI );
        matchTolerance 0.0001;
        transform noOrdering;
        neighbourPatch rotor_outlet;
    }

    rotor_outlet
    {
        nFaces 22578;
        startFace 16308981;
        type cyclicAMI;
        inGroups 1 ( cyclicAMI );
        matchTolerance 0.0001;
        transform noOrdering;
        neighbourPatch volute_inlet;
    }

decomposeParDict:

    numberOfSubdomains 32;
// preservePatches (pipe_outlet rotor_inlet rotor_outlet volute_inlet);
    method scotch;
    scotchCoeffs
    {
    }
TagsAMI

Activities

zordiack

2014-03-16 14:09

reporter   ~0002949

Update, also n=32 crashed with Courant number rising (timestep decreasing), it just took longer.

user4

2014-03-18 09:21

  ~0002956

The main thing which causes differences between parallel and non-parallel running is the linear solver. Make sure you use a tight enough tolerance and e.g. symGaussSeidel instead of PCG. Also keep an eye on your min and max AMI weight - if this gets too low it might give problems.

user889

2014-03-19 07:47

  ~0002958

I would recommend you to use snappyHexMesh. I think you use a different mesher because your nFaces for the AMIs are slightly different?

I ran Cases with AMI up to 1024 Cores with meshes not from snappy...my Cases always works but very very slow with many cores if I lower the cores count the running faster but not fast enough ... now I use snappy for my cases ...even if it is hard at the beginning but now my cases runs much faster... for example 6mio cells mesh on 256 cores per timestep 29seconds on the ami_rotor and ami_stator 100000 faces

As mattijs said ... have a look on your AMI weights espesially at the min ... if the goes to zero your Case will crash ... I also play around with the tolerance I did not recognize any effect also the solvers did not speed up my case with the other mesh from the other mesher ... hope it helps

user889

2014-03-19 09:55

  ~0002960

additional remark ... using symGaussSeidel gives a speed up against GaussSeidel for cyclicAMI case ..thanks mattijs for the hint

zordiack

2014-03-22 15:46

reporter  

fvSolution (1,881 bytes)    
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.2.x                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    location    "system";
    object      fvSolution;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

solvers
{
    p
    {
        solver          GAMG;
        tolerance       1e-07;
        relTol          0.01;
        
        smoother        GaussSeidel;

        cacheAgglomeration true;
        nCellsInCoarsestLevel 10;
        agglomerator    faceAreaPair;
        mergeLevels     1;
    }

    pFinal
    {
        $p;
        relTol          0;
    }

    "(U|k|epsilon|R|omega)"
    {
        solver          PBiCG;
        preconditioner  DILU;
        tolerance       1e-06;
        relTol          0.1;
    }

    "(U|k|epsilon|R|omega)Final"
    {
        $U;
        relTol          0;
    }
}

PIMPLE
{
    momentumPredictor yes;
    nOuterCorrectors 3;
    nCorrectors     2;
    nNonOrthogonalCorrectors 2;
    pRefCell        0;
    pRefValue       0;
}

relaxationFactors
{
    fields
    {
        p           0.3;
    }
    equations
    {
        "(U|k|epsilon|R)"   0.7;
        "(U|k|epsilon|R)Final" 1.0;
    }
}


// ************************************************************************* //
fvSolution (1,881 bytes)   

zordiack

2014-03-22 15:50

reporter   ~0002969

I attached my fvSolution file. I don't think there is a problem with the AMI weights, they seem fine to me:

solidBodyMotionFunctions::rotatingMotion::transformation(): Time = 9.974662104e-05 transformation: ((0 0 0) (0.9998772563 (0 -0.01566755817 0)))
AMI: Creating addressing and weights between 4992 source faces and 4976 target faces
AMI: Patch source sum(weights) min/max/average = 0.9924399218, 1.000000003, 0.9998715005
AMI: Patch target sum(weights) min/max/average = 0.9926369039, 1.000000096, 0.9998725489
AMI: Creating addressing and weights between 22578 source faces and 22460 target faces
AMI: Patch source sum(weights) min/max/average = 0.9999585282, 1.088634231, 1.000016179
AMI: Patch target sum(weights) min/max/average = 0.9995072751, 1.000012176, 1.000002772

I found the reason for the run crashing and it was an "user error" (initial timestep had to be larger that 1e-6 seconds, 1e-3 worked), but I'm still confused about why is the maximum Courant number (and thus adaptive timestep) different when number of domains changes?

Issue History

Date Modified Username Field Change
2014-03-16 12:45 zordiack New Issue
2014-03-16 14:09 zordiack Note Added: 0002949
2014-03-18 09:21 user4 Note Added: 0002956
2014-03-19 07:47 user889 Note Added: 0002958
2014-03-19 09:55 user889 Note Added: 0002960
2014-03-22 15:46 zordiack File Added: fvSolution
2014-03-22 15:50 zordiack Note Added: 0002969
2014-12-29 18:41 wyldckat Tag Attached: AMI
2015-03-05 19:06 henry Status new => closed
2015-03-05 19:06 henry Assigned To => henry
2015-03-05 19:06 henry Resolution open => no change required