Jul 07, 2020 4:10 PM

Getting responses by multiple URLs in PHP using cURL multithreading

Getting responses by multiple URLs in PHP using cURL multithreading

In this article I will give an example of how to get responses by multiple URLs in PHP using cURL multithreading. You can execute multiple requests simultaneously in asynchronous mode with splitting into threads. In normal mode, executing multiple requests takes a lot of time, because they are executed sequentially one after another. But you can speed up this process by using asynchronous requests execution. You can also avoid 100% CPU utilization when using multithreading. Use this function to get responses by multiple URLs:

function getResponseByUrlsMultiThreads($urls, $threads = 10, $followLocation = false, $maxRedirects = 10)
{
    // Options
    $curlOptions = [
        CURLOPT_HEADER => false,
        CURLOPT_NOBODY => false,
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_TIMEOUT => 10,
        CURLOPT_CONNECTTIMEOUT => 10,
    ];

    if ($followLocation) {
        $curlOptions[CURLOPT_FOLLOWLOCATION] = true;
        $curlOptions[CURLOPT_MAXREDIRS] = $maxRedirects;
    }

    // Init multi-curl
    $mh = curl_multi_init();
    $chArray = [];

    $executeMethod = function ($mh, $chArray, &$result, &$running, &$currentThread) {
        usleep(100);
        while (curl_multi_exec($mh, $running) === CURLM_CALL_MULTI_PERFORM);
        curl_multi_select($mh);
        while ($done = curl_multi_info_read($mh)) {
            foreach ($chArray as $key => $ch) {
                if ($ch == $done['handle']) {
                    // Get response
                    $result[$key] = curl_multi_getcontent($ch);
                }
            }
            curl_multi_remove_handle($mh, $done['handle']);
            curl_close($done['handle']);
            $currentThread--;
        }
    };

    $result = [];
    $running = [];
    $currentThread = 0;

    $urls = !is_array($urls) ? [$urls] : $urls;
    foreach ($urls as $key => $url) {
        // Init of requests without executing
        $ch = curl_init($url);
        curl_setopt_array($ch, $curlOptions);

        $chArray[$key] = $ch;

        // Add the handle to multi-curl
        curl_multi_add_handle($mh, $ch);

        $currentThread++;
        if ($currentThread >= $threads) {
            while ($currentThread >= $threads) {
                $executeMethod($mh, $chArray, $result, $running, $currentThread);
            }
        }
    }

    do {
        $executeMethod($mh, $chArray, $result, $running, $currentThread);
    } while($running > 0);
    curl_multi_close($mh);

    return $result;
}

You will get an array with responses after calling this function:

print_r(getResponseByUrlsMultiThreads(['https://www.example.com', 'https://www.example.org']));

Other articles