View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0001640OpenFOAMBugpublic2015-03-30 14:392015-10-22 10:12
Reporteruser1114Assigned Tohenry  
PriorityhighSeverityblockReproducibilityalways
Status closedResolutionsuspended 
PlatformGNU/LinuxOSUbuntuOS Version12.04
Product Versiondev 
Summary0001640: Error when combining mesh re-distribution with refinemt
DescriptionWhen combining cell and fields redistribution and refinement, the execution stalls in a while(true) loop which seems like a face consistency-check plus action routine (2:1 conflicts). According to the debug information, the value of nChanged returned from faceConsistentRefinement is always greater than zero.

The loop is part of the file src/dynamicMesh/dynamicMesh/directTopoChange/directTopoChange/directActions/hexRef8.C.

The tools that I am using for each task are: dynamicRefineFvMesh for refinement, hexRef8 as mesh cutter, parMetis to develop a new distribution and the fvMeshDistribute to perform the actual distribution and field remapping.

The execution order from the start is Refine -> Re-distribute -> Work -> Stall in refinement.

The routine call trace is mesh.update() -> selectRefineCells -> meshCutter.consistentRefinement.

If we skip some re-distribution steps we may stumble into a what I think to be a related error, but this time in a unrefinement routine.
Steps To ReproduceTo reproduce this, I have developed a simple GIT patch (attached) that introduces the required changes to combine both features.
We use the interDyMFoam solver (to which the patch will add code) and the damBreakWithObstacle tutorial.

The changes required to the tutorial are:
File decomposeParDict: metis $->$ parMetis
Reproducible with only 2 procs, File decomposeParDict: numberOfSubdomains 2
File controlDict: adjustTimeStep no
File Allrun: runApplication decomposePar
File Allrun: runParallel interDyMFoam 2
TagsNo tags attached.

Activities

user1114

2015-03-30 14:39

 

refineProb.patch (4,697 bytes)    
From 6753f2f6011295d7143bc208db665038c2646954 Mon Sep 17 00:00:00 2001
From: RR <rbrt.ribeiro@gmail.com>
Date: Wed, 11 Mar 2015 03:23:48 +0000
Subject: [PATCH 1/2] Added re-distribute at runtime code

---
 .../solvers/multiphase/interDyMFoam/Make/options   |  5 ++-
 .../solvers/multiphase/interDyMFoam/interDyMFoam.C | 47 ++++++++++++++++++++++
 2 files changed, 50 insertions(+), 2 deletions(-)

diff --git a/applications/solvers/multiphase/interDyMFoam/Make/options b/applications/solvers/multiphase/interDyMFoam/Make/options
index d0bfa07..f44b4c3 100644
--- a/applications/solvers/multiphase/interDyMFoam/Make/options
+++ b/applications/solvers/multiphase/interDyMFoam/Make/options
@@ -7,8 +7,9 @@ EXE_INC = \
     -I$(LIB_SRC)/finiteVolume/lnInclude \
     -I$(LIB_SRC)/dynamicMesh/dynamicMesh/lnInclude \
     -I$(LIB_SRC)/meshTools/lnInclude \
-    -I$(LIB_SRC)/dynamicMesh/dynamicFvMesh/lnInclude
-
+    -I$(LIB_SRC)/dynamicMesh/dynamicFvMesh/lnInclude \
+    -I$(LIB_SRC)/decompositionMethods/decompositionMethods/lnInclude
+     
 EXE_LIBS = \
     -linterfaceProperties \
     -lincompressibleTransportModels \
diff --git a/applications/solvers/multiphase/interDyMFoam/interDyMFoam.C b/applications/solvers/multiphase/interDyMFoam/interDyMFoam.C
index e3bbbd3..64224fc 100644
--- a/applications/solvers/multiphase/interDyMFoam/interDyMFoam.C
+++ b/applications/solvers/multiphase/interDyMFoam/interDyMFoam.C
@@ -40,6 +40,11 @@ Description
 #include "twoPhaseMixture.H"
 #include "turbulenceModel.H"
 
+#include "fvMeshDistribute.H"
+#include "decompositionMethod.H"
+#include "mapDistributePolyMesh.H"
+
+
 // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
 
 int main(int argc, char *argv[])
@@ -60,6 +65,8 @@ int main(int argc, char *argv[])
 
     Info<< "\nStarting time loop\n" << endl;
 
+    int loop =0;
+
     while (runTime.run())
     {
 #       include "readControls.H"
@@ -90,6 +97,46 @@ int main(int argc, char *argv[])
         // Make the fluxes relative to the mesh motion
         fvc::makeRelative(phi, U);
 
+        {
+        	loop++;
+        	Info << "\n//************************************* Moving cells *************************************//\n\n";
+
+			Pout << "MyCells before: " << mesh.nCells() << endl;
+
+			IOdictionary decompositionDict(
+					IOobject("decomposeParDict", runTime.system(), mesh,
+							IOobject::MUST_READ, IOobject::NO_WRITE));
+
+			labelList finalDecomp;
+
+			// Create decompositionMethod and new decomposition
+			autoPtr<decompositionMethod> decomposer(
+					decompositionMethod::New(decompositionDict, mesh));
+
+			finalDecomp = decomposer().decompose(mesh.cellCentres());
+
+			Foam::labelList newDecompCounted = fvMeshDistribute::countCells(
+					finalDecomp);
+
+			// Global matching tolerance
+			scalar tolDim = 1E-6 * mesh.globalData().bb().mag();
+
+			// Mesh distribution engine
+			fvMeshDistribute distributor(mesh, tolDim);
+
+			// Do actual sending/receiving of mesh
+			Info << "Distributing... " << nl << endl;
+
+			//only re-distribute within a period of 4 iterations
+			//if (loop % 4 == 0)
+			autoPtr<mapDistributePolyMesh> map = distributor.distribute(
+					finalDecomp);
+
+			Pout << "MyCells after: " << mesh.nCells() << endl;
+
+			Info << "\n//************************************* End moving cells *************************************//\n\n";
+        }
+
         if (checkMeshCourantNo)
         {
 #           include "meshCourantNo.H"
-- 
1.9.1


From 75768e71ea55efa0011deb6df01e534ac80cc454 Mon Sep 17 00:00:00 2001
From: RR <rbrt.ribeiro@gmail.com>
Date: Thu, 12 Mar 2015 17:23:41 +0000
Subject: [PATCH 2/2] Change processorWeights to promote cell movement.

---
 src/decompositionMethods/parMetisDecomp/parMetisDecomp.C | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/src/decompositionMethods/parMetisDecomp/parMetisDecomp.C b/src/decompositionMethods/parMetisDecomp/parMetisDecomp.C
index 18b53fb..e2a8371 100644
--- a/src/decompositionMethods/parMetisDecomp/parMetisDecomp.C
+++ b/src/decompositionMethods/parMetisDecomp/parMetisDecomp.C
@@ -284,6 +284,22 @@ Foam::label Foam::parMetisDecomp::decompose
     // output: number of cut edges
     int edgeCut = 0;
 
+    /**********************************/
+    Field<real_t> processorWeights( Pstream::nProcs(),1);
+
+    int p;
+    if (Pstream::master())
+    p = int(rand() % Pstream::nProcs());
+
+    MPI_Bcast( &p, 1, MPI_INT, 0,comm );
+
+    processorWeights[p]=3;
+
+    processorWeights /= sum(processorWeights);
+    tpwgts=processorWeights;
+
+    Info << tpwgts << endl;
+    /**********************************/
 
     ParMETIS_V3_PartGeomKway
     (
-- 
1.9.1
refineProb.patch (4,697 bytes)   

user4

2015-03-30 15:30

  ~0004551

You are trying to do hard things. Dynamic load balancing is not really supported and probably requires a lot more work.

What is probably happening is that the refinement engine (hexRef8) has lost track of the refinement level of each cell. You would have to distribute the hexRef8 refinement structure (hexRef8::distribute). This in addition requires additional decomposition constraints since you want to move all cells originating from the same parent en bloc.

user1114

2015-03-31 16:34

  ~0004554

Thank you for your reply.
Not sure if fully I understand the last sentence. Why should we apply constraints since we are moving cell from the same parent? What kind of constraints do you reckon?

Thanks a lot.

henry

2015-10-22 10:12

manager   ~0005464

Awaiting answers to questions.

Issue History

Date Modified Username Field Change
2015-03-30 14:39 user1114 New Issue
2015-03-30 14:39 user1114 File Added: refineProb.patch
2015-03-30 15:30 user4 Note Added: 0004551
2015-03-31 16:34 user1114 Note Added: 0004554
2015-10-22 10:12 henry Note Added: 0005464
2015-10-22 10:12 henry Status new => closed
2015-10-22 10:12 henry Assigned To => henry
2015-10-22 10:12 henry Resolution open => suspended