Parallel Programming
in C/C++
Pthreads(Part-A), OpenMP(Part-B)
Lecture 5
06/10/2024
1
Identifying Parallelizable Code
do i= 1 to max,
a[i] = b[i] + c[i] * d[i] // Independent operations for each i
end do
• The iterations of this loop are independent, meaning that each iteration can
be computed simultaneously because none of the iterations depends on the
others.
• Why it’s parallelizable: No iteration depends on the result of another.
• Benefits: This allows us to distribute these calculations across multiple
threads or processors to increase efficiency.
2
Understanding Program, Process, and Thread Hierarchy
• A program can have multiple processes running simultaneously.
• Each process can create multiple threads to carry out different tasks.
• Threads share the same memory space within a process, making them
efficient for parallel tasks.
3
Parallel Programming in C/C++ (1 of 2)
• Pthreads: A POSIX (Portable Operating System Interface ) standard for
thread management, allowing low-level control over threads. Introduced
in 1995.
• OpenMP(Open Multi-Processing): A high-level API for parallel
programming, commonly used in scientific computing, introduced in
1997.
• Both are shared memory architectures
Feature Pthreads OpenMP
Control Low-level High-level
Use case General Scientific
Memory model Shared
4
Parallel Programming in C/C++ (2 of 2)
When to Choose each library:
• Use Pthreads when you need fine-grained control over threads, task
scheduling, and synchronization.
• Use OpenMP when you want a simpler interface for parallelizing
loops or large blocks of computation without worrying about the
details of thread management.
- Both libraries are designed to maximize performance in
different ways. Mixing them might result in poor performance
due to the interaction between the different models of thread
management. 5
PART-1: Pthreads
6
Writing Multithreaded Programs: Pthreads
• Multithreaded programs can be written using a variety of
language abstractions interfaces.
• One of the most widely used interfaces is
*Pthreads interface
7
Pthreads Overview
• Pthreads is a POSIX standard for describing a thread model
• You need to include pthread.h to use it in your code.
• Pthreads (POSIX Threads) provide fine-grained control over
thread creation, synchronization (e.g., mutexes, condition
variables), and management.
• It is a low-level API that requires the programmer to manually
handle all aspects of thread management, including resource
allocation, synchronization, and task division.
8
Creating Threads with Pthreads
pthread_create(&tid, &tattr, start_routine, arg);
• pthread_create(): Creates a new thread.
• &tid: Thread identifier (where the thread ID will be stored).
• &tattr: Thread attributes (NULL for default).
• start_routine: The function to be executed by the thread.
• arg: Argument to pass to the thread (NULL if no arguments).
NOTE: When thread attribute is NULL, thread with default properties is created
a
9
PTHREAD (creating thread)
pthread_t t 1 ;
pthread_create(&t1, NULL, &print_message, NULL);
o Pthread create a thread t1, during creation process, we
just pass the address of the newly created threads (&t1).
o Keep the attributes as NULL and left to the system
default setting for scheduling the thread.
o &print_message: pass address of the calling functions to
the thread to be executed by thread.
o NULL as the thread has no argument to be received,
that’s why we keep it null.
LINUX (UBUNTU)
Hello world with Pthreads
•An example Pthread program is shown in next slide.
•The main thread (executing function main) creates a child
thread and terminates.
•The child in turn runs the function and immediately
terminates.
•Since the main thread does not wait for the child thread to
terminate, it may terminate before the children does,
depending on how threads are scheduled on the available
processors. 11
Hello World with Pthreads What is happening?
•The main thread creates a
#include <stdlib.h>
child thread and terminates.
#include <stdio.h>
•The child thread runs the
#include <pthread.h>
assigned function and
void *myfunc1(void *p) {
terminates.
printf (“Hello World \n");
}
The main thread does not
Void main ( ) {
wait for the child thread to
pthread_t child1 ; //Declares a thread variable t1
pthread_create (&child1, NULL, myfunc1, NULL) ;
finish, which can lead to
printf ("No more Children!\n") ; premature termination of
exit(0); the main thread.
} // Expected Output: No more Children
Output:
No more Children
Complete Code: https://onlinegdb.com/xL4ZHn8qw 12
Pthreads - Basic Improved Example!!
#include <stdio.h> What is happening?
#include <pthread.h> • The main thread may
void *myfunc1(void *p) { terminate before the
printf (" Hello World \n"); child thread prints "Hello
} World."
main ( ) { • To prevent this, a dummy
pthread_t child1 ; loop is used to keep the
pthread_create (&child1, NULL, myfunc1, NULL) ; main thread active for a
printf ("No more Children!\n") ; short time.
for(int i=0; i<10000000; i++);
exit(0); Output:
} // Function1 called No more Children!
//Output:No more Children! Hello World
//Hello World 13
Two Threads Example What is happening?
#include <stdlib.h> • This example creates two
#include <stdio.h> threads, each running a
#include <pthread.h> different function
void *myfunc1(void *p) { (myfunc1 and myfunc2).
printf ("Function1 called \n");}
void *myfunc2(void *p) { • The main() function
printf ("Function2 called \n");} terminates after printing
"No more Children!"
int main ( ) {
without waiting for the
pthread_t child1 ;
threads to finish.
pthread_create (&child1, NULL, myfunc1, NULL) ;
pthread_t child2 ;
pthread_create (&child2, NULL, myfunc2, NULL) ;
Output:
printf ("No more Children!\n") ; No more Children!
exit(0);} 14
Pthreads with Sleep (Windows)
#include <stdio.h> Or Sleep- #include <windows.h>//
#include <pthread.h>
#include <unistd.h> // For POSIX usleep Sleep(ms) (S is capital)
void *myfunc1(void *p) {
printf("Function1 called\n");
What is happening?
} • The usleep() function is used to
delay thread creation to simulate
void *myfunc2(void *p) {
printf("Function2 called\n");
asynchronous behavior.
} • Why Use usleep? Introducing
delays allows control over when
int main() {
pthread_t child1, child2;
threads are created and executed
pthread_create(&child1, NULL, myfunc1, NULL);
usleep(3000); // Wait for 3 seconds Output:
pthread_create(&child2, NULL, myfunc2, NULL);
usleep(3000); // Wait for another 3 seconds Function1 called
printf("No more Children!\n"); Function2 called
exit(0); No more Children!
} 15
Cross-Platform Sleep
- Windows and Unix-like systems use different headers for implementing
delays.
• For Windows:
#include <windows.h>
Sleep(ms); // ms = milliseconds (e.g., Sleep(3000) waits for 3 seconds)
• For Mac/Unix:
#include <unistd.h>
usleep(seconds); // seconds = number of seconds (e.g., sleep(3) waits for 3 seconds)
- The Sleep() function takes milliseconds on Windows, while sleep() takes
seconds on Unix.
16
Thread Synchronization in POSIX using pthread_join
• In a multi-threaded environment, it's essential to wait for threads to complete
before proceeding.
• The pthread_join() function is used to wait for a thread to finish execution.
• As such the pthread_join() function blocks the main thread until a specific child
thread finishes execution.
• An enhanced version of a previous Two Threads Example, using pthread_join()
to synchronize the execution can be made by adding:
pthread_join(child1, NULL); // Wait for the first child thread to finish
pthread_join(child2, NULL); // Wait for the first child thread to finish
✓ In this case, the main thread waits for two child threads (child1 and child2) to finish execution using
pthread_join().
✓with NULL as the second parameter, which indicates no return value is expected.
17
Code Example for pthread_join
1. #include <stdio.h>
2. #include <pthread.h> What is happening?
3. void *myfunc1(void *p) {// Function for the first thread This code demonstrates how
4. printf("Function1 called \n");
5. }
pthread_join() is used to wait
for two child threads to
6. void *myfunc2(void *p) { // Function for the second thread complete before proceeding to
7. printf("Function2 called \n");} the final statement in the main
8. int main() {
thread.
9. pthread_t child1, child2; // Declare two thread variables
10. pthread_create(&child1, NULL, myfunc1, NULL); // Create the first thread to execute myfunc1
11. pthread_create(&child2, NULL, myfunc2, NULL); // Create the second thread to execute myfunc2
12. pthread_join(child1, NULL); // Wait for the thread child1 to finish
13. pthread_join(child2, NULL); // Wait for the thread child2 to finish
14. printf("No more Children!\n"); // Final message after all threads are done
15. return 0;} 18
Multiple Threads Example Using pthread_join()
#include <stdio.h>
#include <pthread.h>
#define NUM_THREADS 4 // Define the number of threads
What is happening?
// Function that will be executed by each thread • This code creates 4 threads,
void *hello(void *arg) {
printf("Hello Thread\n");
each executing the hello
} function, and waits for each
thread to complete using
int main() {
pthread_join().
pthread_t tid[NUM_THREADS]; // Array to hold thread IDs
// Create NUM_THREADS threads
for (int i = 0; i < NUM_THREADS; i++) {
pthread_create(&tid[i], NULL, hello, NULL);
}
// Wait for all threads to complete
for (int i = 0; i < NUM_THREADS; i++) {
pthread_join(tid[i], NULL);
}
return 0;
} 19
Thread Synchronization with Multiple Jobs
#include <stdio.h> What is happening?
#include <pthread.h>
#include <windows.h> // For Sleep • This example demonstrates creating multiple threads, each
incrementing a global counter and simulating some work.
pthread_t tid[4]; // Array for 4 thread IDs
int counter = 0; // Global counter for tracking jobs • The output demonstrates that while threads are started in
order, they may finish at different times due to the
// Function executed by each thread
void *myfunc1(void *p) { asynchronous nature of thread execution.
counter += 1; // Increment the job counter
printf("\n Job %d has started\n", counter);
int main() {
// Simulate work (loop represents a delay) int i = 0;
for (int j = 0; j < 10000; j++);
// Create 4 threads
printf("\n Job %d has finished\n", counter); while (i < 4) {
} pthread_create(&tid[i], NULL, myfunc1, NULL);
Output may vary based on system scheduling: i++;
}
Job 1 has started
Job 2 has started // Wait for all threads to finish
Job 2 has finished pthread_join(tid[0], NULL);
Job 4 has started pthread_join(tid[1], NULL);
pthread_join(tid[2], NULL);
Job 3 has started pthread_join(tid[3], NULL);
Job 4 has finished
Job 3 has finished return 0;
} 20
Complete Code: https://onlinegdb.com/HtppJx2WU
Introduction about Mutex
• A mutex (short for "mutual exclusion") is a synchronization primitive used in multithreaded
programming to prevent multiple threads from accessing shared resources or critical sections
simultaneously. This ensures data integrity by allowing only one thread to execute a piece of
code that accesses shared data at any given time.
Why Use a Mutex?
1. To avoid race conditions
- synchronization is needed When multiple threads attempt to modify shared data
concurrently. i.e. If two threads increment a global counter at the same time without
synchronization, the final value of the counter might be incorrect. This happens when
the outcome of the program depends on the timing or order of the threads' execution.
2.Preventing data corruption
- Using a mutex ensures that shared data is updated correctly, preventing corrupted or
unpredictable results.
3.Ensuring thread safety:
- Mutexes help make the program thread-safe by controlling access to shared resources,
ensuring only one thread can modify the resource at a time.
21
Mutex Functions in POSIX Threads (pthreads) (1 of 2)
How Does a Mutex Work? initiate → lock → unlock → destroy
1. Initiate (pthread_mutex_init)
- Before any thread can use the mutex, it must be initialized.
pthread_mutex_t lock;
pthread_mutex_init(&lock, NULL);
2. Lock (pthread_mutex_lock)
- When a thread needs access to a critical section, it locks the mutex. If another thread holds
the lock, the current thread will wait.
pthread_mutex_lock(&lock); // Locking the mutex
22
Mutex Functions in POSIX Threads (pthreads) (2 of 2)
How Does a Mutex Work? initiate → lock → unlock → destroy
3. Unlock (pthread_mutex_unlock)
- After completing the critical section, the thread unlocks the mutex, allowing other
threads to access the shared resource.
pthread_mutex_unlock(&lock); // Unlocking the mutex
4. Destroy (pthread_mutex_destroy)
- Once all threads have finished and the mutex is no longer needed, it is destroyed to
free up resources.
pthread_mutex_destroy(&lock); // Destroying the mutex
23
Using Mutex in a Critical Section (1 in 3)
• A critical section is a part of a program where shared resources are accessed. When multiple threads access
a critical section simultaneously, it can lead to data inconsistency. By using a mutex, we ensure that only one
thread can enter the critical section at a time.
#include <pthread.h> int main(void)
#include <stdio.h> { int i = 0;
#include <stdlib.h> int error;
#include <string.h> if (pthread_mutex_init(&lock, NULL) != 0) {
#include <unistd.h> printf("\n mutex init has failed\n");
pthread_t tid[4]; return 1;
int counter; }
pthread_mutex_t lock; while (i < 4) {
void* trythis(void* arg) pthread_create(&(tid[i]),NULL, &trythis, NULL);
{ i++;
pthread_mutex_lock(&lock); }
counter += 1; pthread_join(tid[0], NULL);
printf("\n Job %d has started\n", counter); pthread_join(tid[1], NULL);
for (int j = 0; j < 10000; j++) ; pthread_join(tid[2], NULL);
printf("\n Job %d has finished\n", counter); pthread_join(tid[3], NULL);
pthread_mutex_unlock(&lock); pthread_mutex_destroy(&lock);
return NULL; return 0;
} } 24
Complete Code: https://onlinegdb.com/F9PgoZpoO
Using Mutex in a Critical Section (2 in 3)
Output from previous run:
• Job 1 has started
• Job 1 has finished
• Job 2 has started
• Job 2 has finished
• Job 3 has started
• Job 3 has finished
• Job 4 has started
• Job 4 has finished
25
Using Mutex in a Critical Section (3 in 3)
Now Comment pthread_mutex_unlock(&lock); in trythis function then rerun
the code:
void* trythis(void* arg) What is happening?
{
pthread_mutex_lock(&lock); • Job 1 locks the mutex but never
counter += 1; releases it, causing a deadlock
printf("\n Job %d has started\n", counter); situation.
for (int j = 0; j < 10000; j++) ;
printf("\n Job %d has finished\n", counter); • As a result, no other thread can
// pthread_mutex_unlock(&lock); proceed because the mutex remains
return NULL; locked, and other jobs will not start.
}
Complete Code: https://onlinegdb.com/KBflX60yT
Mutex Deadlock: Failing to unlock a mutex Output
prevents other threads from accessing the • Job 1 has started
critical section, effectively freezing the • Job 1 has finished
program's execution.
26
Without Mutex in a Critical Section
• If the same code above is run without a mutex, the result would likely be incorrect, as threads
could interfere with each other while incrementing the shared counter variable.
#include <pthread.h> int main(void)
#include <stdio.h> { int i = 0;
#include <stdlib.h> int error;
#include <string.h> if (pthread_mutex_init(&lock, NULL) != 0) {
#include <unistd.h> printf("\n mutex init has failed\n");
pthread_t tid[4]; return 1;
int counter; }
pthread_mutex_t lock; while (i < 4) {
void* trythis(void* arg) pthread_create(&(tid[i]),NULL, &trythis, NULL);
{ i++;
//pthread_mutex_lock(&lock); }
counter += 1; pthread_join(tid[0], NULL);
printf("\n Job %d has started\n", counter); pthread_join(tid[1], NULL);
for (int j = 0; j < 10000; j++) ; pthread_join(tid[2], NULL);
printf("\n Job %d has finished\n", counter); pthread_join(tid[3], NULL);
//pthread_mutex_unlock(&lock); pthread_mutex_destroy(&lock);
return NULL; return 0;
} }
27
Complete Code: https://onlinegdb.com/AjdAYx6uf
Mutex Best Practices
• Always lock and unlock mutexes in the same order.
• Minimize the time spent in the critical section.
• Always make sure to unlock the mutex after the critical section,
otherwise, other threads will be blocked indefinitely
Uses of Mutexes:
• Keeping track of how many jobs a program has completed i.e. using
counters.
• Modifying an array or list that multiple threads need to update or
read from.
• Printing results in a consistent order from multiple threads to avoid
overriding and wrong outputs. 28
Passing Values in Threads
int main() {
#include <stdio.h> pthread_t child1, child2; // Define two thread
#include <pthread.h> variables
#include <stdlib.h> // Needed for exit(0) int i = 4, j = 10;
// Function to be executed by thread 1 // Create the first thread that runs myfunc1
pthread_create(&child1, NULL, myfunc1, &i);
void *myfunc1(void *p) {
int *pp = (int *)p; // Cast void pointer to integer pointer
// Create the second thread that runs myfunc2
int num = *pp; // Dereference to get the value pthread_create(&child2, NULL, myfunc2, &j);
printf("Function1 called - %d\n", num);
return NULL; // Return null since the function has a void* return type // Wait for both threads to finish
pthread_join(child1, NULL);
}
pthread_join(child2, NULL);
// Function to be executed by thread 2
// Print final message when threads are done
void *myfunc2(void *p) {
printf("No more Children!\n");
int *pp = (int *)p; // Cast void pointer to integer
pointer
// Exit the program
int num = *pp; // Dereference to get the value exit(0);
printf("Function2 called - %d\n", num); }
return NULL; // Return null since the function has a void*
return type
Output: Function1 called- 4
} Function2 called- 10
Complete Code: https://onlinegdb.com/7eXbvznjB
No more Children! 29
Passing Values in Threads same example, int num= *(int *) p;
#include <stdio.h> int main() {
#include <pthread.h> pthread_t child1, child2;
int i = 5, j = 10;
#include <stdlib.h> // For exit(0)
// Create thread for myfunc1
void *myfunc1(void *p) { pthread_create(&child1, NULL, myfunc1, &i);
int num = *(int *) p; // Cast the void pointer to
int pointer and dereference // Create thread for myfunc2
printf("Function1 called - %d\n", num); pthread_create(&child2, NULL, myfunc2, &j);
return NULL; // Wait for both threads to finish
} pthread_join(child1, NULL);
pthread_join(child2, NULL);
void *myfunc2(void *p) {
printf("No more Children!\n");
int num = *(int *) p; // Cast the void pointer to
int pointer and dereference
// Exit the program
printf("Function2 called - %d\n", num); exit(0);
return NULL; }
}
Output: Function1 called- 5
Function2 called- 10
Complete Code: https://onlinegdb.com/rD2kK5TUK No more Children! 30
