RMA Test Cases

LocalConcurrency test cases

Combines different operations with different RMA sync modes to cover all possible conflict/sync variants for local concurrency problems / local buffer races.

Operations / Calls covered:

• Buffer accesses: buffer read, buffer write
• Local buffer write: MPI_Get, MPI_Rget
• Local buffer read: MPI_Put, MPI_Rput, MPI_Accumulate, MPI_Raccumulate, MPI_Get_accumulate, MPI_Rget_accumulate, MPI_Fetch_and_op, MPI_Compare_and_swap (Note: MPI_Get_accumulate and others may also write results in another local buffer, but we only consider the reading accesses here.)

Synchronization modes covered:

• Fences: MPI_Win_fence
• Lockall + Flush_all + Unlock_all
• Lockall + Flush_local_all + Unlock_all
• Lock + Flush + Unlock: MPI_Win_(lock|flush|unlock)
• Lock + Unlock
• Requests: MPI_R(put|get|accumulate|get_accumulate) + MPI_Wait
• PSCW
• Currently missing: Lock_all + Unlock_all

Currently missing compared to MBI (for LocalConcurrency):

• Conflict between MPI RMA operation and (non-)blocking P2P, e.g. MPI_Get + MPI_Isend/Send on the same buffer
• The conflicting access could be on a separate buffer or on the window itself (i.e., the window memory itself could be target of a local buffer access).

GlobalConcurrency test cases

Operations / Calls covered:

• Buffer accesses: buffer read, buffer write
• Local buffer read/write: MPI_Put / MPI_Get (we do not consider all other possibilities of local buffer accesses to avoid a test case explosion)
• Remote read: MPI_Get, MPI_Put
• Remote write: MPI_Put, MPI_Rput, MPI_Accumulate, MPI_Raccumulate, MPI_Get_accumulate, MPI_Rget_accumulate, MPI_Fetch_and_op, MPI_Compare_and_swap

Synchronization modes covered:

• Fences: MPI_Win_fence
• Lockall + Flush_all + Barrier + Unlock_all
• Lock + Unlock: Exclusive vs. Shared
• Requests: MPI_R(put|get|accumulate|get_accumulate) + MPI_Wait
• PSCW
• Optional (currently missing): Lock + Unlock, Lock_all + Unlock_all

Invalid Argument test cases

• Invalid target rank (-1, >= nranks, ...)
• Invalid buffer (NULL)
• Invalid Datatype (MPI_DATATYPE_NULL, freed data type)
• Invalid window (no window initialized, window freed to early)

Epoch Lifecycle

• RMA access before epoch is opened
• RMA access without closing epoch
• Double open of access epoch (double lock, non-collective fence)
• Mixing RMA access modes (fence + lock)

scripts/Infrastructure/Template.py

@@ -7,7 +7,6 @@ from scripts.Infrastructure.Instruction import Instruction
 from scripts.Infrastructure.MPICall import MPICall
 from scripts.Infrastructure.CorrectParameter import CorrectParameterFactory
 
-
 template = """// @{generatedby}@
 /* ///////////////////////// The MPI Bug Bench ////////////////////////
 
@@ -92,7 +91,8 @@ class TemplateManager:
         - replace_instruction(self, new_instruction=Instruction, identifier: str = None, idx: int | typing.List[int] = None): Replaces an instruction in the template with a new one.
     """
 
-    def __init__(self, min_ranks: int = 2, thread_level: str = None, has_finalize: bool = True, has_init: bool = True):
+    def __init__(self, min_ranks: int = 2, thread_level: str = None, has_finalize: bool = True, has_init: bool = True,
+                 allow_reorder: bool = True):
         """
         Initialize the TemplateManager
 
@@ -101,6 +101,7 @@ class TemplateManager:
             thread_level : the MPI Thread Level to use (None means use MPI_Init instead of Init_Thread)
             has_finalize (bool) : if call to MPI Finalize should be included
             has_init (bool) : if call to MPI Init should be included
+            allow_reorder (bool): allow reordering of instructions of different ranks for better redability
         """
         self._descr_full = ""
         self._descr_short = ""
@@ -110,6 +111,7 @@ class TemplateManager:
         self._has_finalize = has_finalize
         self._has_init = has_init
         self._stack_variables = {}
+        self._allow_reorder = allow_reorder
 
     def __str__(self):
         """
@@ -118,7 +120,31 @@ class TemplateManager:
         version = self.get_version()
         code_string = ""
         current_rank = 'all'
-        for inst in self._instructions:
+
+        instr_copy = []
+        if self._allow_reorder:
+            # "bucket-sort" the instructions to group those from different ranks together
+            buckets = [[] for _ in range(self._min_ranks)]
+            used_buckets = False
+            for instr in self._instructions:
+                rank = instr.get_rank_executing()
+                if rank != 'all' and rank != 'not0':
+                    used_buckets = True
+                    buckets[rank].append(instr)
+                else:
+                    # can not reorder beyond this point
+                    if used_buckets:
+                        for bucket in buckets:
+                            instr_copy = instr_copy + bucket
+                        # empty buckets
+                        buckets = [[] for _ in range(self._min_ranks)]
+                        used_buckets = False
+                    instr_copy.append(instr)  # current inst
+        else:
+            # no re-ordering
+            instr_copy = self._instructions.copy()
+
+        for inst in instr_copy:
             if inst.get_rank_executing() != current_rank:
                 if current_rank != 'all':
                     code_string = code_string + "}\n"