Skip to content
Snippets Groups Projects
Verified Commit 54454df6 authored by Simon Schwitanski's avatar Simon Schwitanski :slight_smile:
Browse files

Add level 1 logs

parent 6fcd90ed
No related branches found
No related tags found
No related merge requests found
Show whitespace changes
Inline Side-by-side
Showing
with 710 additions and 0 deletions
logs-20240722-083722-level1/itac/CallOrdering-ANY_SOURCE-001.c 0 → 100644
+ 60
− 0
View file @ 54454df6
/* ///////////////////////// The MPI Bug Bench ////////////////////////
Description: order of messages is indeterministic, may lead to a deadlock
Version of MPI: 1.0
This testcase can result in a Deadlock
Category: P2P
BEGIN_MBB_TESTS
$ mpirun -np 3 ${EXE}
| ERROR CallOrdering
| CallOrdering-ANY_SOURCE
END_MBB_TESTS
////////////////////// End of MBI headers /////////////////// */
#include <mpi.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char **argv) {
int nprocs = -1;
int rank = -1;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if (nprocs < 3)
printf(
"MBB ERROR: This test needs at least 3 processes to produce a bug!\n");
int *buf = (int *)calloc(10, sizeof(int));
if (rank != 0) {
buf[0] = rank;
MPI_Send(buf, 10, MPI_INT, 0, 0, MPI_COMM_WORLD);
}
if (rank == 0) {
for (int i = 1; i < nprocs; ++i) {
MPI_Recv(buf, 10, MPI_INT, MPI_ANY_SOURCE, 0, MPI_COMM_WORLD,
MPI_STATUS_IGNORE);
if (buf[0] != i) {
/*MBBERROR_BEGIN*/ MPI_Recv(buf, 10, MPI_INT, 1, 0, MPI_COMM_WORLD,
MPI_STATUS_IGNORE); /*MBBERROR_END*/
}
}
}
if (rank == 1) {
MPI_Send(buf, 10, MPI_INT, 0, 0, MPI_COMM_WORLD);
}
free(buf);
MPI_Finalize();
printf("Rank %d finished normally\n", rank);
return 0;
}
logs-20240722-083722-level1/itac/CallOrdering-ANY_SOURCE-001_0.elapsed 0 → 100644
+ 1
− 0
View file @ 54454df6
62.39588975906372
\ No newline at end of file
logs-20240722-083722-level1/itac/CallOrdering-ANY_SOURCE-001_0.md5sum 0 → 100644
+ 1
− 0
View file @ 54454df6
49d923fa959575b6eee4b22665e5464a
\ No newline at end of file
logs-20240722-083722-level1/itac/CallOrdering-ANY_SOURCE-001_0.txt 0 → 100644
+ 102
− 0
View file @ 54454df6
Running #114 /MBI/scripts/gencodes/P2P/CallOrdering-ANY_SOURCE-001.c
Wait up to 120 seconds
Compiling CallOrdering-ANY_SOURCE-001.c (batchinfo:1/1)
$ mpiicc /MBI/scripts/gencodes/P2P/CallOrdering-ANY_SOURCE-001.c -O0 -g -o CallOrdering-ANY_SOURCE-001
| icc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message.
|
Executing the command (cwd: None)
$ mpirun -check_mpi -genv VT_CHECK_TRACING on -np 3 ./CallOrdering-ANY_SOURCE-001
|
| [0] INFO: CHECK LOCAL:EXIT:SIGNAL ON
| [0] INFO: CHECK LOCAL:EXIT:BEFORE_MPI_FINALIZE ON
| [0] INFO: CHECK LOCAL:MPI:CALL_FAILED ON
| [0] INFO: CHECK LOCAL:MEMORY:OVERLAP ON
| [0] INFO: CHECK LOCAL:MEMORY:ILLEGAL_MODIFICATION ON
| [0] INFO: CHECK LOCAL:MEMORY:INACCESSIBLE ON
| [0] INFO: CHECK LOCAL:MEMORY:ILLEGAL_ACCESS OFF
| [0] INFO: CHECK LOCAL:MEMORY:INITIALIZATION OFF
| [0] INFO: CHECK LOCAL:REQUEST:ILLEGAL_CALL ON
| [0] INFO: CHECK LOCAL:REQUEST:NOT_FREED ON
| [0] INFO: CHECK LOCAL:REQUEST:PREMATURE_FREE ON
| [0] INFO: CHECK LOCAL:DATATYPE:NOT_FREED ON
| [0] INFO: CHECK LOCAL:BUFFER:INSUFFICIENT_BUFFER ON
| [0] INFO: CHECK GLOBAL:DEADLOCK:HARD ON
| [0] INFO: CHECK GLOBAL:DEADLOCK:POTENTIAL ON
| [0] INFO: CHECK GLOBAL:DEADLOCK:NO_PROGRESS ON
| [0] INFO: CHECK GLOBAL:MSG:DATATYPE:MISMATCH ON
| [0] INFO: CHECK GLOBAL:MSG:DATA_TRANSMISSION_CORRUPTED ON
| [0] INFO: CHECK GLOBAL:MSG:PENDING ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:DATATYPE:MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:DATA_TRANSMISSION_CORRUPTED ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:OPERATION_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:SIZE_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:REDUCTION_OPERATION_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:ROOT_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:INVALID_PARAMETER ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:COMM_FREE_MISMATCH ON
| [0] INFO: maximum number of errors before aborting: CHECK-MAX-ERRORS 1
| [0] INFO: maximum number of reports before aborting: CHECK-MAX-REPORTS 0 (= unlimited)
| [0] INFO: maximum number of times each error is reported: CHECK-SUPPRESSION-LIMIT 10
| [0] INFO: timeout for deadlock detection: DEADLOCK-TIMEOUT 60s
| [0] INFO: timeout for deadlock warning: DEADLOCK-WARNING 300s
| [0] INFO: maximum number of reported pending messages: CHECK-MAX-PENDING 20
|
| [0] ERROR: no progress observed in any process for over 1:00 minutes, aborting application
| [0] WARNING: starting emergency trace file writing
|
| [0] ERROR: GLOBAL:DEADLOCK:HARD: fatal error
| [0] ERROR: Application aborted because no progress was observed for over 1:00 minutes,
| [0] ERROR: check for real deadlock (cycle of processes waiting for data) or
| [0] ERROR: potential deadlock (processes sending data to each other and getting blocked
| [0] ERROR: because the MPI might wait for the corresponding receive).
| [0] ERROR: [0] no progress observed for over 1:00 minutes, process is currently in MPI call:
| [0] ERROR: MPI_Recv(*buf=0x470ab0, count=10, datatype=MPI_INT, source=1, tag=0, comm=MPI_COMM_WORLD, *status=0x1)
| [0] ERROR: (/MBI/scripts/gencodes/P2P/CallOrdering-ANY_SOURCE-001.c:47)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/MBI/logs-20240722-083722/itac/CallOrdering-ANY_SOURCE-001)
| [0] ERROR: [1] no progress observed for over 1:00 minutes, process is currently in MPI call:
| [0] ERROR: MPI_Finalize()
| [0] ERROR: (/MBI/scripts/gencodes/P2P/CallOrdering-ANY_SOURCE-001.c:57)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/MBI/logs-20240722-083722/itac/CallOrdering-ANY_SOURCE-001)
| [0] ERROR: [2] no progress observed for over 1:00 minutes, process is currently in MPI call:
| [0] ERROR: MPI_Finalize()
| [0] ERROR: (/MBI/scripts/gencodes/P2P/CallOrdering-ANY_SOURCE-001.c:57)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/MBI/logs-20240722-083722/itac/CallOrdering-ANY_SOURCE-001)
| [0] INFO: Writing tracefile CallOrdering-ANY_SOURCE-001.stf in /MBI/logs-20240722-083722/itac
|
| [0] INFO: GLOBAL:DEADLOCK:HARD: found 1 time (1 error + 0 warnings), 0 reports were suppressed
| [0] INFO: Found 1 problem (1 error + 0 warnings), 0 reports were suppressed.
|
|
| ===================================================================================
| = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
| = RANK 0 PID 132853 RUNNING AT n23t0001.hpc.itc.rwth-aachen.de
| = KILLED BY SIGNAL: 9 (Killed)
| ===================================================================================
|
| ===================================================================================
| = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
| = RANK 1 PID 132854 RUNNING AT n23t0001.hpc.itc.rwth-aachen.de
| = KILLED BY SIGNAL: 9 (Killed)
| ===================================================================================
|
| ===================================================================================
| = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
| = RANK 2 PID 132855 RUNNING AT n23t0001.hpc.itc.rwth-aachen.de
| = KILLED BY SIGNAL: 9 (Killed)
| ===================================================================================
|
Command return code: 255, elapsed time: 62.39588975906372
Finished #114 /MBI/scripts/gencodes/P2P/CallOrdering-ANY_SOURCE-001.c
logs-20240722-083722-level1/itac/CallOrdering-ANY_TAG-001.c 0 → 100644
+ 60
− 0
View file @ 54454df6
/* ///////////////////////// The MPI Bug Bench ////////////////////////
Description: order of messages is indeterministic, may lead to a deadlock
Version of MPI: 1.0
This testcase can result in a Deadlock
Category: P2P
BEGIN_MBB_TESTS
$ mpirun -np 2 ${EXE}
| ERROR CallOrdering
| CallOrdering-ANY_TAG
END_MBB_TESTS
////////////////////// End of MBI headers /////////////////// */
#include <mpi.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char **argv) {
int nprocs = -1;
int rank = -1;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if (nprocs < 2)
printf(
"MBB ERROR: This test needs at least 2 processes to produce a bug!\n");
int *buf = (int *)calloc(10, sizeof(int));
if (rank == 0) {
for (int i = 0; i < 10; ++i) {
MPI_Recv(buf, 10, MPI_INT, 1, MPI_ANY_TAG, MPI_COMM_WORLD,
MPI_STATUS_IGNORE);
if (buf[0] != i) {
/*MBBERROR_BEGIN*/ MPI_Recv(buf, 10, MPI_INT, 1, 0, MPI_COMM_WORLD,
MPI_STATUS_IGNORE); /*MBBERROR_END*/
}
}
}
if (rank == 1) {
for (int i = 0; i < 10; ++i) {
buf[0] = i;
MPI_Send(buf, 10, MPI_INT, 0, i, MPI_COMM_WORLD);
}
MPI_Send(buf, 10, MPI_INT, 0, 0, MPI_COMM_WORLD);
}
free(buf);
MPI_Finalize();
printf("Rank %d finished normally\n", rank);
return 0;
}
logs-20240722-083722-level1/itac/CallOrdering-ANY_TAG-001_0.elapsed 0 → 100644
+ 1
− 0
View file @ 54454df6
62.395663261413574
\ No newline at end of file
logs-20240722-083722-level1/itac/CallOrdering-ANY_TAG-001_0.md5sum 0 → 100644
+ 1
− 0
View file @ 54454df6
841cf3603a215f932d9657c838a0988e
\ No newline at end of file
logs-20240722-083722-level1/itac/CallOrdering-ANY_TAG-001_0.txt 0 → 100644
+ 90
− 0
View file @ 54454df6
Running #90 /MBI/scripts/gencodes/P2P/CallOrdering-ANY_TAG-001.c
Wait up to 120 seconds
Compiling CallOrdering-ANY_TAG-001.c (batchinfo:1/1)
$ mpiicc /MBI/scripts/gencodes/P2P/CallOrdering-ANY_TAG-001.c -O0 -g -o CallOrdering-ANY_TAG-001
| icc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message.
|
Executing the command (cwd: None)
$ mpirun -check_mpi -genv VT_CHECK_TRACING on -np 2 ./CallOrdering-ANY_TAG-001
|
| [0] INFO: CHECK LOCAL:EXIT:SIGNAL ON
| [0] INFO: CHECK LOCAL:EXIT:BEFORE_MPI_FINALIZE ON
| [0] INFO: CHECK LOCAL:MPI:CALL_FAILED ON
| [0] INFO: CHECK LOCAL:MEMORY:OVERLAP ON
| [0] INFO: CHECK LOCAL:MEMORY:ILLEGAL_MODIFICATION ON
| [0] INFO: CHECK LOCAL:MEMORY:INACCESSIBLE ON
| [0] INFO: CHECK LOCAL:MEMORY:ILLEGAL_ACCESS OFF
| [0] INFO: CHECK LOCAL:MEMORY:INITIALIZATION OFF
| [0] INFO: CHECK LOCAL:REQUEST:ILLEGAL_CALL ON
| [0] INFO: CHECK LOCAL:REQUEST:NOT_FREED ON
| [0] INFO: CHECK LOCAL:REQUEST:PREMATURE_FREE ON
| [0] INFO: CHECK LOCAL:DATATYPE:NOT_FREED ON
| [0] INFO: CHECK LOCAL:BUFFER:INSUFFICIENT_BUFFER ON
| [0] INFO: CHECK GLOBAL:DEADLOCK:HARD ON
| [0] INFO: CHECK GLOBAL:DEADLOCK:POTENTIAL ON
| [0] INFO: CHECK GLOBAL:DEADLOCK:NO_PROGRESS ON
| [0] INFO: CHECK GLOBAL:MSG:DATATYPE:MISMATCH ON
| [0] INFO: CHECK GLOBAL:MSG:DATA_TRANSMISSION_CORRUPTED ON
| [0] INFO: CHECK GLOBAL:MSG:PENDING ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:DATATYPE:MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:DATA_TRANSMISSION_CORRUPTED ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:OPERATION_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:SIZE_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:REDUCTION_OPERATION_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:ROOT_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:INVALID_PARAMETER ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:COMM_FREE_MISMATCH ON
| [0] INFO: maximum number of errors before aborting: CHECK-MAX-ERRORS 1
| [0] INFO: maximum number of reports before aborting: CHECK-MAX-REPORTS 0 (= unlimited)
| [0] INFO: maximum number of times each error is reported: CHECK-SUPPRESSION-LIMIT 10
| [0] INFO: timeout for deadlock detection: DEADLOCK-TIMEOUT 60s
| [0] INFO: timeout for deadlock warning: DEADLOCK-WARNING 300s
| [0] INFO: maximum number of reported pending messages: CHECK-MAX-PENDING 20
|
| [0] ERROR: no progress observed in any process for over 1:00 minutes, aborting application
| [0] WARNING: starting emergency trace file writing
|
| [0] ERROR: GLOBAL:DEADLOCK:HARD: fatal error
| [0] ERROR: Application aborted because no progress was observed for over 1:00 minutes,
| [0] ERROR: check for real deadlock (cycle of processes waiting for data) or
| [0] ERROR: potential deadlock (processes sending data to each other and getting blocked
| [0] ERROR: because the MPI might wait for the corresponding receive).
| [0] ERROR: [0] no progress observed for over 1:00 minutes, process is currently in MPI call:
| [0] ERROR: MPI_Finalize()
| [0] ERROR: (/MBI/scripts/gencodes/P2P/CallOrdering-ANY_TAG-001.c:57)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/MBI/logs-20240722-083722/itac/CallOrdering-ANY_TAG-001)
| [0] ERROR: [1] no progress observed for over 1:00 minutes, process is currently in MPI call:
| [0] ERROR: MPI_Send(*buf=0x46a500, count=10, datatype=MPI_INT, dest=0, tag=0, comm=MPI_COMM_WORLD)
| [0] ERROR: (/MBI/scripts/gencodes/P2P/CallOrdering-ANY_TAG-001.c:53)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/MBI/logs-20240722-083722/itac/CallOrdering-ANY_TAG-001)
| [0] INFO: Writing tracefile CallOrdering-ANY_TAG-001.stf in /MBI/logs-20240722-083722/itac
|
| [0] INFO: GLOBAL:DEADLOCK:HARD: found 1 time (1 error + 0 warnings), 0 reports were suppressed
| [0] INFO: Found 1 problem (1 error + 0 warnings), 0 reports were suppressed.
|
|
| ===================================================================================
| = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
| = RANK 0 PID 132345 RUNNING AT n23t0001.hpc.itc.rwth-aachen.de
| = KILLED BY SIGNAL: 9 (Killed)
| ===================================================================================
|
| ===================================================================================
| = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
| = RANK 1 PID 132346 RUNNING AT n23t0001.hpc.itc.rwth-aachen.de
| = KILLED BY SIGNAL: 9 (Killed)
| ===================================================================================
|
Command return code: 255, elapsed time: 62.395663261413574
Finished #90 /MBI/scripts/gencodes/P2P/CallOrdering-ANY_TAG-001.c
logs-20240722-083722-level1/itac/CallOrdering-mpi_send-mpi_recv-001.c 0 → 100644
+ 51
− 0
View file @ 54454df6
/* ///////////////////////// The MPI Bug Bench ////////////////////////
Description: Call Ordering: both ranks try to receive before sending
Version of MPI: 1.0
This testcase can result in a Deadlock
Category: P2P
BEGIN_MBB_TESTS
$ mpirun -np 2 ${EXE}
| OK
| CallOrdering-mpi_send-mpi_recv
END_MBB_TESTS
////////////////////// End of MBI headers /////////////////// */
#include <mpi.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char **argv) {
int nprocs = -1;
int rank = -1;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if (nprocs < 2)
printf(
"MBB ERROR: This test needs at least 2 processes to produce a bug!\n");
int *buf = (int *)calloc(10, sizeof(int));
if (rank == 0) {
MPI_Recv(buf, 10, MPI_INT, 1, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
MPI_Send(buf, 10, MPI_INT, 1, 0, MPI_COMM_WORLD);
}
if (rank == 1) {
MPI_Send(buf, 10, MPI_INT, 0, 0, MPI_COMM_WORLD);
MPI_Recv(buf, 10, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
}
free(buf);
MPI_Finalize();
printf("Rank %d finished normally\n", rank);
return 0;
}
logs-20240722-083722-level1/itac/CallOrdering-mpi_send-mpi_recv-001_0.elapsed 0 → 100644
+ 1
− 0
View file @ 54454df6
2.325329542160034
\ No newline at end of file
logs-20240722-083722-level1/itac/CallOrdering-mpi_send-mpi_recv-001_0.md5sum 0 → 100644
+ 1
− 0
View file @ 54454df6
c508d14d6634dcfe6d449b8c994e6f7d
\ No newline at end of file
logs-20240722-083722-level1/itac/CallOrdering-mpi_send-mpi_recv-001_0.txt 0 → 100644
+ 59
− 0
View file @ 54454df6
Running #99 /MBI/scripts/gencodes/P2P/CallOrdering-mpi_send-mpi_recv-001.c
Wait up to 120 seconds
Compiling CallOrdering-mpi_send-mpi_recv-001.c (batchinfo:1/1)
$ mpiicc /MBI/scripts/gencodes/P2P/CallOrdering-mpi_send-mpi_recv-001.c -O0 -g -o CallOrdering-mpi_send-mpi_recv-001
| icc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message.
|
Executing the command (cwd: None)
$ mpirun -check_mpi -genv VT_CHECK_TRACING on -np 2 ./CallOrdering-mpi_send-mpi_recv-001
|
| [0] INFO: CHECK LOCAL:EXIT:SIGNAL ON
| [0] INFO: CHECK LOCAL:EXIT:BEFORE_MPI_FINALIZE ON
| [0] INFO: CHECK LOCAL:MPI:CALL_FAILED ON
| [0] INFO: CHECK LOCAL:MEMORY:OVERLAP ON
| [0] INFO: CHECK LOCAL:MEMORY:ILLEGAL_MODIFICATION ON
| [0] INFO: CHECK LOCAL:MEMORY:INACCESSIBLE ON
| [0] INFO: CHECK LOCAL:MEMORY:ILLEGAL_ACCESS OFF
| [0] INFO: CHECK LOCAL:MEMORY:INITIALIZATION OFF
| [0] INFO: CHECK LOCAL:REQUEST:ILLEGAL_CALL ON
| [0] INFO: CHECK LOCAL:REQUEST:NOT_FREED ON
| [0] INFO: CHECK LOCAL:REQUEST:PREMATURE_FREE ON
| [0] INFO: CHECK LOCAL:DATATYPE:NOT_FREED ON
| [0] INFO: CHECK LOCAL:BUFFER:INSUFFICIENT_BUFFER ON
| [0] INFO: CHECK GLOBAL:DEADLOCK:HARD ON
| [0] INFO: CHECK GLOBAL:DEADLOCK:POTENTIAL ON
| [0] INFO: CHECK GLOBAL:DEADLOCK:NO_PROGRESS ON
| [0] INFO: CHECK GLOBAL:MSG:DATATYPE:MISMATCH ON
| [0] INFO: CHECK GLOBAL:MSG:DATA_TRANSMISSION_CORRUPTED ON
| [0] INFO: CHECK GLOBAL:MSG:PENDING ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:DATATYPE:MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:DATA_TRANSMISSION_CORRUPTED ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:OPERATION_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:SIZE_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:REDUCTION_OPERATION_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:ROOT_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:INVALID_PARAMETER ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:COMM_FREE_MISMATCH ON
| [0] INFO: maximum number of errors before aborting: CHECK-MAX-ERRORS 1
| [0] INFO: maximum number of reports before aborting: CHECK-MAX-REPORTS 0 (= unlimited)
| [0] INFO: maximum number of times each error is reported: CHECK-SUPPRESSION-LIMIT 10
| [0] INFO: timeout for deadlock detection: DEADLOCK-TIMEOUT 60s
| [0] INFO: timeout for deadlock warning: DEADLOCK-WARNING 300s
| [0] INFO: maximum number of reported pending messages: CHECK-MAX-PENDING 20
|
| [0] INFO: Writing tracefile CallOrdering-mpi_send-mpi_recv-001.stf in /MBI/logs-20240722-083722/itac
|
| [0] INFO: Error checking completed without finding any problems.
|
| Rank 0 finished normally
| Rank 1 finished normally
|
Command return code: 0, elapsed time: 2.325329542160034
Finished #99 /MBI/scripts/gencodes/P2P/CallOrdering-mpi_send-mpi_recv-001.c
logs-20240722-083722-level1/itac/CallOrdering-unmatched-mpi_allgather-001.c 0 → 100644
+ 50
− 0
View file @ 54454df6
/* ///////////////////////// The MPI Bug Bench ////////////////////////
Description: mpi_allgather is not called by all processes
Version of MPI: 1.0
This testcase can result in a Deadlock
Category: COLL
BEGIN_MBB_TESTS
$ mpirun -np 2 ${EXE}
| ERROR CallOrdering
| CallOrdering-unmatched-mpi_allgather
END_MBB_TESTS
////////////////////// End of MBI headers /////////////////// */
#include <mpi.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char **argv) {
int nprocs = -1;
int rank = -1;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if (nprocs < 2)
printf(
"MBB ERROR: This test needs at least 2 processes to produce a bug!\n");
int *buf = (int *)calloc(nprocs * (10), sizeof(int));
int *recv_buf = (int *)calloc(10 * nprocs, sizeof(int));
if (rank == 0) {
/*MBBERROR_BEGIN*/ MPI_Allgather(buf, 10, MPI_INT, recv_buf, 10, MPI_INT,
MPI_COMM_WORLD); /*MBBERROR_END*/
}
free(buf);
free(recv_buf);
MPI_Finalize();
printf("Rank %d finished normally\n", rank);
return 0;
}
logs-20240722-083722-level1/itac/CallOrdering-unmatched-mpi_allgather-001_0.elapsed 0 → 100644
+ 1
− 0
View file @ 54454df6
62.30204486846924
\ No newline at end of file
logs-20240722-083722-level1/itac/CallOrdering-unmatched-mpi_allgather-001_0.md5sum 0 → 100644
+ 1
− 0
View file @ 54454df6
3816040742e0bc85e85b91e8507cb791
\ No newline at end of file
logs-20240722-083722-level1/itac/CallOrdering-unmatched-mpi_allgather-001_0.txt 0 → 100644
+ 90
− 0
View file @ 54454df6
Running #71 /MBI/scripts/gencodes/COLL/CallOrdering-unmatched-mpi_allgather-001.c
Wait up to 120 seconds
Compiling CallOrdering-unmatched-mpi_allgather-001.c (batchinfo:1/1)
$ mpiicc /MBI/scripts/gencodes/COLL/CallOrdering-unmatched-mpi_allgather-001.c -O0 -g -o CallOrdering-unmatched-mpi_allgather-001
| icc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message.
|
Executing the command (cwd: None)
$ mpirun -check_mpi -genv VT_CHECK_TRACING on -np 2 ./CallOrdering-unmatched-mpi_allgather-001
|
| [0] INFO: CHECK LOCAL:EXIT:SIGNAL ON
| [0] INFO: CHECK LOCAL:EXIT:BEFORE_MPI_FINALIZE ON
| [0] INFO: CHECK LOCAL:MPI:CALL_FAILED ON
| [0] INFO: CHECK LOCAL:MEMORY:OVERLAP ON
| [0] INFO: CHECK LOCAL:MEMORY:ILLEGAL_MODIFICATION ON
| [0] INFO: CHECK LOCAL:MEMORY:INACCESSIBLE ON
| [0] INFO: CHECK LOCAL:MEMORY:ILLEGAL_ACCESS OFF
| [0] INFO: CHECK LOCAL:MEMORY:INITIALIZATION OFF
| [0] INFO: CHECK LOCAL:REQUEST:ILLEGAL_CALL ON
| [0] INFO: CHECK LOCAL:REQUEST:NOT_FREED ON
| [0] INFO: CHECK LOCAL:REQUEST:PREMATURE_FREE ON
| [0] INFO: CHECK LOCAL:DATATYPE:NOT_FREED ON
| [0] INFO: CHECK LOCAL:BUFFER:INSUFFICIENT_BUFFER ON
| [0] INFO: CHECK GLOBAL:DEADLOCK:HARD ON
| [0] INFO: CHECK GLOBAL:DEADLOCK:POTENTIAL ON
| [0] INFO: CHECK GLOBAL:DEADLOCK:NO_PROGRESS ON
| [0] INFO: CHECK GLOBAL:MSG:DATATYPE:MISMATCH ON
| [0] INFO: CHECK GLOBAL:MSG:DATA_TRANSMISSION_CORRUPTED ON
| [0] INFO: CHECK GLOBAL:MSG:PENDING ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:DATATYPE:MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:DATA_TRANSMISSION_CORRUPTED ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:OPERATION_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:SIZE_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:REDUCTION_OPERATION_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:ROOT_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:INVALID_PARAMETER ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:COMM_FREE_MISMATCH ON
| [0] INFO: maximum number of errors before aborting: CHECK-MAX-ERRORS 1
| [0] INFO: maximum number of reports before aborting: CHECK-MAX-REPORTS 0 (= unlimited)
| [0] INFO: maximum number of times each error is reported: CHECK-SUPPRESSION-LIMIT 10
| [0] INFO: timeout for deadlock detection: DEADLOCK-TIMEOUT 60s
| [0] INFO: timeout for deadlock warning: DEADLOCK-WARNING 300s
| [0] INFO: maximum number of reported pending messages: CHECK-MAX-PENDING 20
|
| [0] ERROR: no progress observed in any process for over 1:00 minutes, aborting application
| [0] WARNING: starting emergency trace file writing
|
| [0] ERROR: GLOBAL:DEADLOCK:HARD: fatal error
| [0] ERROR: Application aborted because no progress was observed for over 1:00 minutes,
| [0] ERROR: check for real deadlock (cycle of processes waiting for data) or
| [0] ERROR: potential deadlock (processes sending data to each other and getting blocked
| [0] ERROR: because the MPI might wait for the corresponding receive).
| [0] ERROR: [0] no progress observed for over 1:00 minutes, process is currently in MPI call:
| [0] ERROR: MPI_Allgather(*sendbuf=0x436110, sendcount=10, sendtype=MPI_INT, *recvbuf=0x47c0b0, recvcount=10, recvtype=MPI_INT, comm=MPI_COMM_WORLD)
| [0] ERROR: (/MBI/scripts/gencodes/COLL/CallOrdering-unmatched-mpi_allgather-001.c:41)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/MBI/logs-20240722-083722/itac/CallOrdering-unmatched-mpi_allgather-001)
| [0] ERROR: [1] no progress observed for over 1:00 minutes, process is currently in MPI call:
| [0] ERROR: MPI_Finalize()
| [0] ERROR: (/MBI/scripts/gencodes/COLL/CallOrdering-unmatched-mpi_allgather-001.c:47)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/MBI/logs-20240722-083722/itac/CallOrdering-unmatched-mpi_allgather-001)
| [0] INFO: Writing tracefile CallOrdering-unmatched-mpi_allgather-001.stf in /MBI/logs-20240722-083722/itac
|
| [0] INFO: GLOBAL:DEADLOCK:HARD: found 1 time (1 error + 0 warnings), 0 reports were suppressed
| [0] INFO: Found 1 problem (1 error + 0 warnings), 0 reports were suppressed.
|
|
| ===================================================================================
| = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
| = RANK 0 PID 132226 RUNNING AT n23t0001.hpc.itc.rwth-aachen.de
| = KILLED BY SIGNAL: 9 (Killed)
| ===================================================================================
|
| ===================================================================================
| = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
| = RANK 1 PID 132227 RUNNING AT n23t0001.hpc.itc.rwth-aachen.de
| = KILLED BY SIGNAL: 9 (Killed)
| ===================================================================================
|
Command return code: 255, elapsed time: 62.30204486846924
Finished #71 /MBI/scripts/gencodes/COLL/CallOrdering-unmatched-mpi_allgather-001.c
logs-20240722-083722-level1/itac/CallOrdering-unmatched-mpi_allgather-mpi_allreduce-001.c 0 → 100644
+ 51
− 0
View file @ 54454df6
/* ///////////////////////// The MPI Bug Bench ////////////////////////
Description: Collective mismatch: mpi_allgather is matched with mpi_allreduce
Version of MPI: 1.0
Category: COLL
BEGIN_MBB_TESTS
$ mpirun -np 2 ${EXE}
| ERROR CallOrdering
| CallOrdering-unmatched-mpi_allgather-mpi_allreduce
END_MBB_TESTS
////////////////////// End of MBI headers /////////////////// */
#include <mpi.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char **argv) {
int nprocs = -1;
int rank = -1;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if (nprocs < 2)
printf(
"MBB ERROR: This test needs at least 2 processes to produce a bug!\n");
int *buf = (int *)calloc(nprocs * (10), sizeof(int));
int *recv_buf = (int *)calloc(10 * nprocs, sizeof(int));
if (rank == 0) {
/*MBBERROR_BEGIN*/ MPI_Allgather(buf, 10, MPI_INT, recv_buf, 10, MPI_INT,
MPI_COMM_WORLD); /*MBBERROR_END*/
}
if (rank != 0) {
/*MBBERROR_BEGIN*/ MPI_Allreduce(buf, recv_buf, 10, MPI_INT, MPI_SUM,
MPI_COMM_WORLD); /*MBBERROR_END*/
}
free(buf);
free(recv_buf);
MPI_Finalize();
printf("Rank %d finished normally\n", rank);
return 0;
}
logs-20240722-083722-level1/itac/CallOrdering-unmatched-mpi_allgather-mpi_allreduce-001_0.elapsed 0 → 100644
+ 1
− 0
View file @ 54454df6
7.1542534828186035
\ No newline at end of file
logs-20240722-083722-level1/itac/CallOrdering-unmatched-mpi_allgather-mpi_allreduce-001_0.md5sum 0 → 100644
+ 1
− 0
View file @ 54454df6
f9a1ae8d4d7d932c760dba5e2007d772
\ No newline at end of file
logs-20240722-083722-level1/itac/CallOrdering-unmatched-mpi_allgather-mpi_allreduce-001_0.txt 0 → 100644
+ 87
− 0
View file @ 54454df6
Running #54 /MBI/scripts/gencodes/COLL/CallOrdering-unmatched-mpi_allgather-mpi_allreduce-001.c
Wait up to 120 seconds
Compiling CallOrdering-unmatched-mpi_allgather-mpi_allreduce-001.c (batchinfo:1/1)
$ mpiicc /MBI/scripts/gencodes/COLL/CallOrdering-unmatched-mpi_allgather-mpi_allreduce-001.c -O0 -g -o CallOrdering-unmatched-mpi_allgather-mpi_allreduce-001
| icc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message.
|
Executing the command (cwd: None)
$ mpirun -check_mpi -genv VT_CHECK_TRACING on -np 2 ./CallOrdering-unmatched-mpi_allgather-mpi_allreduce-001
|
| [0] INFO: CHECK LOCAL:EXIT:SIGNAL ON
| [0] INFO: CHECK LOCAL:EXIT:BEFORE_MPI_FINALIZE ON
| [0] INFO: CHECK LOCAL:MPI:CALL_FAILED ON
| [0] INFO: CHECK LOCAL:MEMORY:OVERLAP ON
| [0] INFO: CHECK LOCAL:MEMORY:ILLEGAL_MODIFICATION ON
| [0] INFO: CHECK LOCAL:MEMORY:INACCESSIBLE ON
| [0] INFO: CHECK LOCAL:MEMORY:ILLEGAL_ACCESS OFF
| [0] INFO: CHECK LOCAL:MEMORY:INITIALIZATION OFF
| [0] INFO: CHECK LOCAL:REQUEST:ILLEGAL_CALL ON
| [0] INFO: CHECK LOCAL:REQUEST:NOT_FREED ON
| [0] INFO: CHECK LOCAL:REQUEST:PREMATURE_FREE ON
| [0] INFO: CHECK LOCAL:DATATYPE:NOT_FREED ON
| [0] INFO: CHECK LOCAL:BUFFER:INSUFFICIENT_BUFFER ON
| [0] INFO: CHECK GLOBAL:DEADLOCK:HARD ON
| [0] INFO: CHECK GLOBAL:DEADLOCK:POTENTIAL ON
| [0] INFO: CHECK GLOBAL:DEADLOCK:NO_PROGRESS ON
| [0] INFO: CHECK GLOBAL:MSG:DATATYPE:MISMATCH ON
| [0] INFO: CHECK GLOBAL:MSG:DATA_TRANSMISSION_CORRUPTED ON
| [0] INFO: CHECK GLOBAL:MSG:PENDING ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:DATATYPE:MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:DATA_TRANSMISSION_CORRUPTED ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:OPERATION_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:SIZE_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:REDUCTION_OPERATION_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:ROOT_MISMATCH ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:INVALID_PARAMETER ON
| [0] INFO: CHECK GLOBAL:COLLECTIVE:COMM_FREE_MISMATCH ON
| [0] INFO: maximum number of errors before aborting: CHECK-MAX-ERRORS 1
| [0] INFO: maximum number of reports before aborting: CHECK-MAX-REPORTS 0 (= unlimited)
| [0] INFO: maximum number of times each error is reported: CHECK-SUPPRESSION-LIMIT 10
| [0] INFO: timeout for deadlock detection: DEADLOCK-TIMEOUT 60s
| [0] INFO: timeout for deadlock warning: DEADLOCK-WARNING 300s
| [0] INFO: maximum number of reported pending messages: CHECK-MAX-PENDING 20
|
|
| [0] ERROR: GLOBAL:COLLECTIVE:OPERATION_MISMATCH: error
| [0] ERROR: Different processes entered different collective operations on the same communicator.
| [0] ERROR: Collective call by local rank [0] (same as global rank):
| [0] ERROR: MPI_Allgather(*sendbuf=0x46ff30, sendcount=10, sendtype=MPI_INT, *recvbuf=0x4361d0, recvcount=10, recvtype=MPI_INT, comm=MPI_COMM_WORLD)
| [0] ERROR: (/MBI/scripts/gencodes/COLL/CallOrdering-unmatched-mpi_allgather-mpi_allreduce-001.c:38)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/MBI/logs-20240722-083722/itac/CallOrdering-unmatched-mpi_allgather-mpi_allreduce-001)
| [0] ERROR: Collective call by local rank [1] (same as global rank):
| [0] ERROR: MPI_Allreduce(*sendbuf=0x46a500, *recvbuf=0x46a560, count=10, datatype=MPI_INT, op=MPI_SUM, comm=MPI_COMM_WORLD)
| [0] ERROR: (/MBI/scripts/gencodes/COLL/CallOrdering-unmatched-mpi_allgather-mpi_allreduce-001.c:42)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/usr/lib/x86_64-linux-gnu/libc.so.6)
| [0] ERROR: (/MBI/logs-20240722-083722/itac/CallOrdering-unmatched-mpi_allgather-mpi_allreduce-001)
| [0] INFO: 1 error, limit CHECK-MAX-ERRORS reached => aborting
| [0] WARNING: starting emergency trace file writing
| [0] INFO: Writing tracefile CallOrdering-unmatched-mpi_allgather-mpi_allreduce-001.stf in /MBI/logs-20240722-083722/itac
|
| [0] INFO: GLOBAL:COLLECTIVE:OPERATION_MISMATCH: found 1 time (1 error + 0 warnings), 0 reports were suppressed
| [0] INFO: Found 1 problem (1 error + 0 warnings), 0 reports were suppressed.
|
|
| ===================================================================================
| = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
| = RANK 0 PID 130733 RUNNING AT n23t0001.hpc.itc.rwth-aachen.de
| = KILLED BY SIGNAL: 9 (Killed)
| ===================================================================================
|
| ===================================================================================
| = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
| = RANK 1 PID 130734 RUNNING AT n23t0001.hpc.itc.rwth-aachen.de
| = KILLED BY SIGNAL: 9 (Killed)
| ===================================================================================
|
Command return code: 255, elapsed time: 7.1542534828186035
Finished #54 /MBI/scripts/gencodes/COLL/CallOrdering-unmatched-mpi_allgather-mpi_allreduce-001.c
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment