Preprocessing
At the end of this class, you should be able to:
- Define the workflow and toolsets for preprocessing of CFD simulations
- Construct a geometry and a mesh for CFD
- Explain the parameters that influence the quality of the mesh
- Define the parameters to set for the numerical setup
The preprocessing step is not specific to the CFD workflow on HPC. In fact, many of these steps, such as geometry generation, mesh creation, and input file setup, are integral steps to any CFD simulation. Out of completeness, we present the main steps, as well as specific considerations for preprocessing when using HPC.
Preprocessing main steps, concepts, and toolsets
The steps for preprocessing CFD simulations are very broadly divided into:
- Construction of the geometry;
- Discretization of the fluid domain;
- Setup the simulation.
These steps are broken down in the following subsection, but first, we briefly discuss the particularities of the preprocessing workflow on HPC systems.
preprocessing workflow
For small non-HPC simulations, where memory or computational limitations are not an issue, the preprocessing, processing, and postprocessing steps can be done locally. With larger CFD simulations, this workflow needs to be revisited. Two main types of workflow emerge for the preprocessing step:
- Local preprocessing
- Remote preprocessing
The optimal workflow for a given problem depends on a number of factors including local workstation computational resources, toolset, and, even, the internet bandwidth. But the most significant constraint lies in the local workstation memory during the meshing stage. Complex mesh generation is often a memory intensive process that may constrain the user to rely on remote preprocessing on HPC systems. Let us explore these ideas in more detail.
Local preprocessing
For smaller CFD problems or simple meshing needs, local preprocessing is usually the preferred approach. Here, the user is on a local workstation and can generate the geometry and the mesh. Then using copying techniques seen in the first section can copy the resulting mesh to the remote HPC system for processing the simulation. This is usually the standard workflow for most users.
- Easy to setup and organize data
- Memory and computationally constrained
- Must copy the final mesh to the remote HPC system
Remote preprocessing
When the CFD problem is larger or the mesh is very complex, the limitations of the workstation constrain the user to resort to remote preprocessing. In this case, the preprocessing toolset is run on the remote HPC system, either interactively or headless (without gui). This allows the user to use the larger computational and memory resources of the remote system for preprocessing. As the mesh is generated on the same HPC system where the CFD simulation will be run, there is the added benefit of maintaining the data at the same location. Now, it is also possible to use a hybrid approach where, for example, the Computer-aided Design (CAD) is done locally and transferred to the remote system for the meshing phase. When considering remote preprocessing, it is important to take into account:
- availability of the preprocessing tools on remote system,
- licence constraints for commercial software,
- speed/lag time to run the software remotely.
- Access to larger memory and computational resources for preprocessing
- ’Data locality’: mesh is generated on the system where simulations are run
- During interactive mode, there may be significant lag due to internet latency
- Availability of the software/license on remote system
Toolsets
Many tools are available to complete the preprocessing step. Here, we provide a brief overview of some of the tools available. Please keep in mind that some CFD solvers, especially commercial tools, have geometry and meshing capabilities built in.
Open-source options:
- Gmsh: most versatile open-source meshing tool for CFD applications for structured and unstructured grids
- FreeCAD: tool to generate CAD definition of the CFD domain, must be combined with a mesher to generate a CFD mesh
- Salome: SMESH within this platform is a full fledge meshing tool that can accommodate complex geometries
- blockMesh: a utility within the openFoam framework to generate structured mesh
- snappyHexMesh: a utility within the openFoam framework to generate an unstructured mesh around an arbitrarily complex CAD geometry
- meshio: although it is formally not a meshing tool, this software can translate mesh files among many different mesh formats
Commercial tools
- Pointwise
- Cubit
- Autodesk
- Integrated meshers within commercial packages (e.g. Ansys Meshing, StarCCM+ Mesher )
For this course, we focus on Gmsh, which is a highly versatile open-source mesher that can create a structured or unstructured grid. It can also output natively or near-natively in a number of well-known meshing formats. A Gmsh tutorial can be found here. For users that do not have access to the Digital Research Alliance of Canada systems or that want to install the programs locally, please consult the overview page for links to the software.
Construct the geometry
For complex geometries, for example pumps, industrial applications, or HVAC fluid dynamics, the geometry of the simulation domain should be generated in a dedicated CAD software. Many meshing tools have limited capabilities to build complex geometries; for very simple geometries, the features of the meshing software may be sufficient. If the CAD can be natively developed within the meshing software, it should be favoured as it facilitates the integration of the geometry and mesh and enables an easier parametrization. Although some commercial tools have seamless integration between the CAD and meshing software, most meshing tools require an additional step to generate and transfer the CAD geometry. It should be noted that while some CAD software have meshing capabilities, they usually cater to solid mechanics community, which has very different types of constraints on their mesh generation.
The format of the CAD geometry depends on the compatibility with the meshing software. Typically, IGES or STEP formats are widely used in meshing software. For some meshing applications (such as snappyHexMesh) where only the outer hull of the geometry is needed, a fluid mesh can be generated from an STL (Stereolithography) file.
Example: Cylinder in crossflow (domain generation)
For this example, we construct a domain and mesh to simulate a cylinder in crossflow. For this purpose, we will use Gmsh. Gmsh is an widely used open-source meshing tool and is installed on the Digital Research Alliance clusters but you can also download it locally. We will not cover the usage of Gmsh in much detail and the interested reader can consult the tutorial online to supplement the knowledge provided here. Gmsh can be used either through its GUI (graphical user interface) or with an interpreted script. Note that when using the GUI, Gmsh automatically writes the script. Therefore, by saving and modifying the script, you can reproduce a mesh without a GUI. Furthermore, with a very large mesh, you can define the mesh locally (in the script) and run it remotely. For this example, we will directly work in the Gmsh script.
Let us assume that we have computed all the necessary domain size in a previous step and we have a general idea of the geometry we hope to create:
As we have a non-slip wall at the cylinder, we know that we need to pay close attention to the mesh at this location, so we create a zone around the cylinder which will have a higher mesh density. Now, we can parameterize these values into the Gmsh script below. In Gmsh, we must also, at this stage, make a decision on the mesh density (highlighted in the example). Note that the raw input file can be downloaded from the Git repository of the course.If we load the script, we should see the following three-dimensional geometry.
Generate mesh
Once the geometry is generated, the fluid domain can be discretized for CFD computation. At this stage, it is a good idea to be nimble and favour an iterative approach to generating the final production run mesh. It is often a good idea to start by generating a smaller mesh, visually inspecting the discretized grid, and, if needed, modifying the CAD geometry (e.g. domain size, removing smaller features of the CAD, smoothing unnecessary sharp corners, etc.) Estimates on the grid resolution requirements (computed in the last lecture) should be integrated in the definition of the final mesh.
As we discussed last lecture, grid sizing greatly influences computational costs in two ways:
- it determines the degrees of freedom in the problem;
- it constrains the time advancement step.
There are a couple of considerations when generating the mesh that are discussed below.
Start with a small mesh, output the results, and inspect the mesh before creating your final mesh. It may be a good idea to run a quick simulation on the coarse before committing to the generating the large mesh.
Deciding the type of mesh to use
There are two main types of mesh that are used in CFD: structured and unstructured mesh.
Structured mesh
Structured meshes are (typically) hexahedra-shaped mesh organized in a logical way. Given their regular hexahedral shape, they can minimize numerical errors in flux calculation between cells. The structured mesh can also have clear advantages in terms of computational efficiency, since there is implicit knowledge of the neighboring cells. For example, if a domain is discretized using
- Smaller storage and memory needs
- Reduced error in the flux calculations
- Lower computational cost
- Potentially faster computations
- Can result in more grid points: by construction, we may have unnecessary grid points due to the need to have a structured mesh
- Inability to mesh complex geometries
Most commercial and open-source CFD codes, such as SU2 and OpenFoam, permit the use of structured meshes but maintain an unstructured data structure for the computation of the interface fluxes.
Unstructured mesh
Most finite-volume CFD solvers are based on unstructured grids. In this case, the mesh information is not stored in a
- Easy to mesh complex geometries
- Can result in fewer grid points (grids can be locally clustered at specific locations)
- Easier to automate
- Computationally less efficient than structured meshes
- Requires more checks on the mesh quality
- Larger memory requirement
Domain discretization
The mesh generation process will depend heavily on the toolset used and the general meshing decisions (type of mesh, region of higher resolution etc.) A small mesh can be rapidly generated locally on your laptop; but large, complex meshes will require additional computational resources. Careful planning of the mesh and a nimble iterative approach should be preferred. Simultaneously, it is always a good idea to aim to make a parametric geometry and mesh such that future changes to the grid can be done without too many problems.
Continuing from the cylinder in crossflow example, we now focus on discretizing the fluid domain. For this example, we will use a structured mesh near the cylinder wall and an unstructured mesh elsewhere. This will enable us to minimize the error with a structured mesh where the gradients are the largest (near the wall) and save grid points in the far field with an unstructured mesh.
First, we define the number of grid points in the boundary layer (here there is no near wall clustering for simplicity), the number of points around the circumference of the cylinder, and number of points in the spanwise direction. Note that to make a structured mesh in Gmsh, we construct Transfinite Lines with which we create Transfinite Surface.
If this mesh were to be used in a CFD simulation, we would additionally need to define the boundaries of the computational domain (done in the final example in the class), for example:
For now, we can generate the mesh in Gmsh and obtain:
This mesh can then be exported from Gmsh in a format that can be read by the CFD solver.
Assessing the mesh quality
Assessing the mesh quality prior to a large scale simulation can avoid lengthy iterations on HPC systems (copying mesh over to HPC system, waiting in queue, assessing error, etc.). These mesh checks can typically be done within the meshing tool or in standalone mesh assessments scripts (for example, *checkMesh * in openFoam). The mesh assessment will typically report a number of different metrics, including the following:
-
Minimum and maximum mesh size: This information does not (necessarily) affect the quality assessment, but can provide useful insight for the simulation. As many time advancement methods are constrained by the CFL number, an unreasonably small mesh may unnecessarily constrain the time-advancement of the simulation. Especially for complex geometry, the assessment of the minimal mesh size with respect to the required resolution may highlight the need to revisit the mesh generation prior to running the simulation.
-
Aspect ratio: The aspect ratio corresponds to the ratio of the smallest to the largest edge length of the cells.
This formula is primarily for hexaheadral meshes; a slight modification of this formula (using an internal sphere) is used for tetrahedral or other mesh geometries. This computation is conducted in each cell of the domain and the maximum value is presented.
- Mesh orthogonality: Orthogonality is a measure of the angle between the vector of the face normal and the connecting line between two neighboring cells. In structured meshes, the orthogonality is null (assuming no non-cartesian mapping), for unstructured meshes, this value can be large, although below 90 degrees. High mesh orthogonality can lead to numerical errors in the flux calculation and numerical instability in the CFD computation.
- Volume ratio: This metric reveals the largest volumes ratio between neighboring cells. Strong differences in cell size can result in large numerical errors and, thus, instability of the simulation. The sensitivity of this parameter depends on a number of factors, such as the order of the numerical scheme. If this value is too large, a smoother transition should be considered.
where
- Skewness: Skewness is a measure of the departure from perfectly perpendicular vertex angles of the mesh (for hexahedral meshes). The definition for non-hexahedral meshes is slightly more involved (and not covered herein) but the basic idea remains the same.
General meshing metrics can be summarized as follow:
Mesh quality | Maximum value acceptable value |
---|---|
Aspect ratio | 20 |
Orthogonality | 85 deg |
Volume ratio | 80 |
Skewness | 0.85 |
These values are meant to provide general guidance for the mesh quality; we may still get a high-quality simulation despite not meeting these mesh quality criteria.
Now that the mesh is generated and output, we can check the quality of the final mesh. Here, we explore two different mesh checking tools, namely: (1) the native Gmsh mesh statistics, (2) the built-in SU2 mesh checker.
- Native Gmsh mesh statistics: Most meshing tools have mesh quality checks built in. Gmsh also has some limited statistics that roughly align with the above-mentioned. By calling Tool->Statistics, we have access to the following information:
- SICN: signed inverse condition number
- Gamma: inscribed radius / circumscribed radius
- SIGE: signed inverse error on the gradient of solution
These quantities can also be visualized, and corrections can be made to the mesh prior to simulating the final results. Using the checkMesh tool in openFoam, for example, can provide even greater statistical insight with the metrics shown above.
- Built-in SU2 mesh checker: An alternative option is to use the check mesh that is included in most CFD solvers. For this example, we use SU2 which runs a check mesh prior to starting the simulation. From the mesh generated above, we get the following output:
Configure numerical setup
The final stage of preprocessing is setting up the case.
- Linking the mesh to the CFD solver
- Setting up the initial conditions of the problem
- Setting up the boundary conditions
- Determining the governing equations (generally Euler or Navier-Stokes), state equations, and any other multiphysics
- Selecting the models (e.g. turbulence, acoustics, combustion)
- Select the decomposition of the domain for parallel computing
As these are case-specific, we will investigate these steps during the final example in this section.
We continue the example of the backward facing step by undertaking the preprocessing steps.
Construct domain
The size of the computational domain was determined in the previous class; based on these values, we can create the geometry that will be used. To make the mesh easily parametrizable, we define the important dimensions as variables in the Gmsh script:
Similarly, we can define the number of grid points based on the previous calculations (this particular case corresponds to a very coarse simulation and not the final simulation case).
Now we can use the variables to create all the points, the lines connecting the points, connecting all the lines to create surfaces, and, finally, the surfaces to create volumes.
To create a structured mesh in Gmsh, we need to define transfinite curves which we discretize with the corresponding number of points (as seen in the previous example). We can then Extrude the mesh in the spanwise direction.
Finally, we can define our boundary conditions on the mesh.
Click HERE to see the entire Gmsh script
Define the mesh
In the above, we generated the domain and, using the transfinite curves, we defined the mesh resolution and stretching. For illustrative purposes, we only used a very coarse mesh. In the git repository, we have included three different mesh with varying
Using the equations introduced in the last lesson, we can define the characteristics of all mesh as follows:
-
Knowing the Reynolds number (
), we can estimate the friction coefficient: -
With the friction coefficient, we can estimate the shear stress at the wall:
-
From which we can compute the friction velocity:
-
Based on the friction velocity and desired
, we can estimate the first cell height from the wall:
All these values are summarized in the table below and the corresponding mesh are located in the repository.
Grid description | dy (mm) | Grid count | |||
---|---|---|---|---|---|
Coarse | 5 | 0.0124 | 0.61 | 0.12 | 123,200 |
Medium | 2.3 | 0.0124 | 0.61 | 0.056 | 409,120 |
Fine | 1 | 0.0124 | 0.61 | 0.024 | 780,120 |
With these definitions, we can output the desired mesh into a format that is relevant for the selected CFD solver.
Checking mesh
Now we can check the mesh. In this summative example, we use the checkMesh
function in OpenFoam. Note that to run the checkMesh, we need to fully setup the case in openFoam. The checkMesh provides the following information:
A couple points to note:
- All checks withing checkMesh are successful
- The mesh is fully structured with only hexahedral mesh elements
Simulation setup
The standard SA model is used for near-wall turbulence modelling while Smagorinsky LES sub-grid scale (SGS) model is implemented at wall-normal distances larger than a value specified from default wall-distance coefficient. (Refer to Molina et al (2017) for SU2 formulations). In SU2, the 2nd order dual-time stepping scheme is used for time marching method with 5 inner iterations per time step. The total simulation duration is 400ms which corresponds to roughly 10 flow through times across the stream wise direction. Results are saved every 2ms.
Having finished this class, you should now be able to answer the following questions:
- What is the optimal workflow and toolset needed to preprocess a CFD simulation?
- What are the considerations in the geometry and mesh generation process?
- How do I determine, a priori, if my mesh is good?
- What parameters should I consider setting in the simulation?