README.md — Jetpack – WP Security, Backup, Speed, & Growth 16.1-beta.3

jetpack / vendor / wp-php-toolkit / http-client / README.md

jetpack / vendor / wp-php-toolkit / http-client Last commit date

ByteStream 1 week ago Middleware 1 week ago Transport 1 week ago examples 1 week ago LICENSE.md 1 week ago README.md 1 week ago class-client.php 1 week ago class-clientstate.php 1 week ago class-connection.php 1 week ago class-crawler.php 1 week ago class-httpclientexception.php 1 week ago class-httperror.php 1 week ago class-request.php 1 week ago class-response.php 1 week ago composer.json 1 week ago

README.md

610 lines

1	---
2	slug: httpclient
3	title: HttpClient
4	install: wp-php-toolkit/http-client
5
6	see_also:
7	- ../learn/04-talking-to-the-network.html \| Tutorial — Talking to the network \| Walks through a streaming downloader that resumes, fans out, and pipes bytes to disk without buffering.
8	- bytestream \| ByteStream \| Stream request and response bodies.
9	- filesystem \| Filesystem \| Persist large downloads without buffering them in memory.
10	- corsproxy \| CORSProxy \| Bridge browser-side tools to servers without CORS headers.
11	---
12
13	Async HTTP client without <code>curl</code> required. Uses sockets when curl is missing, supports concurrent requests and streaming responses.
14
15	## Why this exists
16
17	<p>A plugin installer starts with one request to download <code>plugin.zip</code>. A migration then adds progress reporting, a ten-request media window, resumable downloads, and a remote ZIP reader that feeds ZipFilesystem directly. Those workflows need the same request API from the first GET to the final streamed archive.</p>
18
19	<p>The HttpClient component gives the toolkit a small request/response model, middleware for redirects and caching, concurrent fetches, and response bodies exposed as byte streams. It runs through curl when PHP provides curl and through PHP sockets when it does not. Callers keep the same request and response model while the transport changes underneath.</p>
20
21	<p>Use it to fetch plugin metadata, submit import callbacks, mirror a media library, read a WXR export, or pipe a remote archive into Zip and Filesystem code.</p>
22
23	## GET a URL
24
25	<p class="callout"><strong>Network access in the demo runtime.</strong> Live request examples show the real API, but outbound HTTP in browser sandboxes may require a CORS proxy.</p>
26
27	<p>The smallest flow has three steps: create a request, wait until headers arrive, then consume the body stream. This is intentionally close to the Fetch API shape, but the body is a toolkit byte stream instead of a buffered string.</p>
28
29	<!-- snippet:
30	filename: get.php
31	runnable: true
32	-->
33	```php
34	<?php
35	require '/php-toolkit/vendor/autoload.php';
36
37	use WordPress\HttpClient\Client;
38	use WordPress\HttpClient\Request;
39
40	$client = new Client();
41	$stream = $client->fetch( new Request( 'https://example.com/' ) );
42
43	$response = $stream->await_response();
44	echo "status: " . $response->status_code . "\n";
45	echo "first 80 bytes: " . substr( $stream->consume_all(), 0, 80 ) . "\n";
46	```
47
48	## POST to a URL
49
50	<p>Uploads use the same shape. The only difference is that the request declares a method, request headers, and an upload body stream. Here the body is form-encoded text wrapped in <code>MemoryPipe</code>; a file upload could provide a file-backed read stream instead.</p>
51
52	<!-- snippet:
53	filename: post.php
54	runnable: true
55	-->
56	```php
57	<?php
58	require '/php-toolkit/vendor/autoload.php';
59
60	use WordPress\HttpClient\Client;
61	use WordPress\HttpClient\Request;
62	use WordPress\ByteStream\MemoryPipe;
63
64	$payload = http_build_query(
65	array(
66	'title' => 'Hello',
67	'tags' => 'http,php',
68	),
69	'',
70	'&'
71	);
72
73	$client = new Client();
74	$request = new Request( 'https://httpbin.org/post', array(
75	'method' => 'POST',
76	'headers' => array(
77	'content-type' => 'application/x-www-form-urlencoded',
78	'content-length' => (string) strlen( $payload ),
79	),
80	'body_stream' => new MemoryPipe( $payload ),
81	) );
82
83	$response = $client->fetch( $request )->json();
84	echo "Server saw form title: " . $response['form']['title'] . "\n";
85	```
86
87	## Build a JSON request object
88
89	<p>A <code>Request</code> is just data until a client enqueues it. That makes it easy to test request construction without network access. The constructor normalizes headers, calculates <code>content-length</code> when the body stream has a known length, and moves URL credentials into an Authorization header.</p>
90
91	<!-- snippet:
92	filename: request-object.php
93	runnable: true
94	-->
95	```php
96	<?php
97	require '/php-toolkit/vendor/autoload.php';
98
99	use WordPress\ByteStream\MemoryPipe;
100	use WordPress\HttpClient\Request;
101
102	$body = new MemoryPipe( json_encode( array(
103	'title' => 'Hello',
104	'tags' => array( 'docs', 'php' ),
105	) ) );
106	$body->close_writing();
107
108	$request = new Request( 'https://user:secret@api.example.test/posts', array(
109	'method' => 'POST',
110	'headers' => array( 'content-type' => 'application/json' ),
111	'body_stream' => $body,
112	) );
113
114	echo $request->method . ' ' . $request->url . "\n";
115	echo "content-type: " . $request->get_header( 'content-type' ) . "\n";
116	echo "content-length: " . $request->get_header( 'content-length' ) . "\n";
117	echo "authorization: " . substr( $request->get_header( 'authorization' ), 0, 10 ) . "...\n";
118	```
119
120	<!-- expected-output -->
121	```
122	POST https://api.example.test/posts
123	content-type: application/json
124	content-length: 39
125	authorization: Basic dXNl...
126	```
127
128	## Parse response headers
129
130	<p>Most applications receive <code>Response</code> objects from <code>await_response()</code>. Transports, middleware, and tests sometimes need the lower-level parser: <code>Response::from_http_headers()</code> turns raw HTTP header bytes into normalized status and case-insensitive headers.</p>
131
132	<!-- snippet:
133	filename: parse-response.php
134	runnable: true
135	-->
136	```php
137	<?php
138	require '/php-toolkit/vendor/autoload.php';
139
140	use WordPress\HttpClient\Request;
141	use WordPress\HttpClient\Response;
142
143	$request = new Request( 'https://api.example.test/posts/42' );
144	$raw = "HTTP/1.1 201 Created\r\n"
145	. "Content-Type: application/json\r\n"
146	. "Location: /posts/42\r\n"
147	. "Content-Length: 27\r\n\r\n";
148
149	$response = Response::from_http_headers( $raw, $request );
150
151	echo "status: " . $response->status_code . ' ' . $response->get_reason_phrase() . "\n";
152	echo "ok: " . ( $response->ok() ? 'yes' : 'no' ) . "\n";
153	echo "type: " . $response->get_header( 'CONTENT-TYPE' ) . "\n";
154	echo "size: " . $response->total_bytes . " bytes\n";
155	```
156
157	<!-- expected-output -->
158	```
159	status: 201 Created
160	ok: yes
161	type: application/json
162	size: 27 bytes
163	```
164
165	## Pick the right reading style
166
167	<p>There are three common ways to consume a response. Start simple, then move down the table only when the workflow demands it.</p>
168
169	<table><thead><tr><th>Style</th><th>Use when</th><th>Tradeoff</th></tr></thead><tbody><tr><td><code>consume_all()</code> or <code>json()</code></td><td>Small HTML, JSON, or API responses.</td><td>Buffers the full body.</td></tr><tr><td><code>Client::await_next_event()</code></td><td>Progress bars, streaming to disk, queues, failure handling.</td><td>You own the event loop.</td></tr><tr><td>Filesystem and parser composition</td><td>Remote ZIPs, WXR files, import pipelines.</td><td>Requires a stream-aware consumer.</td></tr></tbody></table>
170
171	## Choose a transport
172
173	<p>The transport is the I/O backend. It should not change your request, response, redirect, cache, or stream code; it only changes how bytes move across the network.</p>
174
175	<table><thead><tr><th>Transport</th><th>What it does</th><th>When to choose it</th></tr></thead><tbody><tr><td><code>auto</code></td><td>Uses curl when loaded, otherwise sockets.</td><td>Application default. Best when you want portability and the fastest available backend.</td></tr><tr><td><code>sockets</code></td><td>Uses PHP stream sockets, no curl extension.</td><td>Tests, Playground-style runtimes, hosts where curl is unavailable, or proving the dependency-free path works.</td></tr><tr><td><code>curl</code></td><td>Uses the curl extension.</td><td>Hosts where curl is available and you want to compare behavior or performance explicitly.</td></tr></tbody></table>
176
177	<p><code>concurrency</code>, <code>timeout_ms</code>, <code>cache_dir</code>, redirects, and response streaming sit above the transport, so the examples later on work with either backend.</p>
178
179	<!-- snippet:
180	filename: transports.php
181	runnable: false
182	-->
183	```php
184	<?php
185	require '/php-toolkit/vendor/autoload.php';
186
187	use WordPress\HttpClient\Client;
188
189	$default = new Client(); // Same as array( 'transport' => 'auto' ).
190
191	$portable = new Client( array(
192	'transport' => 'sockets',
193	) );
194
195	if ( extension_loaded( 'curl' ) ) {
196	$curl = new Client( array(
197	'transport' => 'curl',
198	) );
199	}
200	```
201
202	## Follow redirects and inspect the final request
203
204	<p>Redirects are middleware, not transport behavior. The client follows up to five redirects by default. The original <code>Request</code> keeps a chain to the final request, so importers can log where a source URL actually landed.</p>
205
206	<p>Current caveat: followed redirects are reissued as <code>GET</code> requests. That matches common browser behavior for <code>303</code> and many simple downloads, but do not rely on it for preserving non-GET methods across <code>307</code> or <code>308</code> responses.</p>
207
208	<!-- snippet:
209	filename: redirects.php
210	runnable: false
211	-->
212	```php
213	<?php
214	require '/php-toolkit/vendor/autoload.php';
215
216	use WordPress\HttpClient\Client;
217	use WordPress\HttpClient\Request;
218
219	$client = new Client();
220	$request = new Request( 'https://httpbin.org/redirect-to?url=https://example.com/' );
221	$stream = $client->fetch( $request );
222	$response = $stream->await_response();
223	$stream->consume_all();
224
225	$final = $request->latest_redirect();
226	echo "original: " . $request->url . "\n";
227	echo "final: " . $final->url . "\n";
228	echo "status: " . $response->status_code . "\n";
229	```
230
231	## Cache repeatable GET responses
232
233	<p>Pass <code>cache_dir</code> to add disk caching for cacheable GET and HEAD responses. Fresh cached responses replay the same header/body events as a network response, so crawlers and importers do not need a separate cache code path. Non-GET requests invalidate matching cache entries instead of being cached.</p>
234
235	<!-- snippet:
236	filename: cache.php
237	runnable: false
238	-->
239	```php
240	<?php
241	require '/php-toolkit/vendor/autoload.php';
242
243	use WordPress\HttpClient\Client;
244	use WordPress\HttpClient\Request;
245
246	$cache_dir = sys_get_temp_dir() . '/http-cache-' . uniqid();
247	mkdir( $cache_dir );
248
249	$client = new Client( array( 'cache_dir' => $cache_dir ) );
250	$url = 'https://httpbin.org/cache/60';
251
252	for ( $i = 1; $i <= 2; $i++ ) {
253	$stream = $client->fetch( new Request( $url ) );
254	$response = $stream->await_response();
255	$body = $stream->consume_all();
256	echo "request {$i}: HTTP " . $response->status_code . ', body=' . strlen( $body ) . " bytes\n";
257	}
258
259	echo "cache files: " . count( glob( $cache_dir . '/*' ) ) . "\n";
260	```
261
262	## Handle failures without losing the queue
263
264	<p>Failures arrive as events. That lets a crawler, importer, package installer, or media frontloader log one bad URL and keep processing the rest of the queue. Treat failure handling as part of the event loop, not as one global try/catch around the whole batch.</p>
265
266	<!-- snippet:
267	filename: failures.php
268	runnable: false
269	-->
270	```php
271	<?php
272	require '/php-toolkit/vendor/autoload.php';
273
274	use WordPress\HttpClient\Client;
275	use WordPress\HttpClient\Request;
276
277	$client = new Client( array( 'timeout_ms' => 5000 ) );
278	$client->enqueue( array(
279	new Request( 'https://example.com/', array( 'method' => 'HEAD' ) ),
280	new Request( 'https://example.invalid/missing' ),
281	) );
282
283	while ( $client->await_next_event() ) {
284	$request = $client->get_request();
285	$event = $client->get_event();
286
287	if ( Client::EVENT_GOT_HEADERS === $event ) {
288	echo "ok: " . $request->url . " HTTP " . $request->response->status_code . "\n";
289	} elseif ( Client::EVENT_FAILED === $event ) {
290	echo "failed: " . $request->url . "\n";
291	} elseif ( Client::EVENT_FINISHED === $event ) {
292	echo "finished: " . $request->url . "\n";
293	}
294	}
295	```
296
297	## Monitor download progress
298
299	<p>When you care about progress, use the event loop directly. Count bytes from each <code>EVENT_BODY_CHUNK_AVAILABLE</code> event and compare them with <code>Content-Length</code> when the server provides one.</p>
300
301	<!-- snippet:
302	filename: progress.php
303	runnable: true
304	-->
305	```php
306	<?php
307	require '/php-toolkit/vendor/autoload.php';
308
309	use WordPress\HttpClient\Client;
310	use WordPress\HttpClient\Request;
311
312	$url = 'https://raw.githubusercontent.com/WordPress/php-toolkit/trunk/components/Zip/Tests/fixtures/childrens-literature.zip';
313	$dest = sys_get_temp_dir() . '/progress-' . uniqid() . '.zip';
314
315	$client = new Client();
316	$request = new Request( $url );
317	$client->enqueue( array( $request ) );
318
319	$downloaded = 0;
320	$last_step = -1;
321	@unlink( $dest );
322
323	while ( $client->await_next_event() ) {
324	$event = $client->get_event();
325	$request = $client->get_request();
326
327	if ( Client::EVENT_GOT_HEADERS === $event ) {
328	echo "status: " . $request->response->status_code . "\n";
329	continue;
330	}
331
332	if ( Client::EVENT_BODY_CHUNK_AVAILABLE === $event ) {
333	$chunk = $client->get_response_body_chunk();
334	$downloaded += strlen( $chunk );
335	file_put_contents( $dest, $chunk, FILE_APPEND );
336
337	$total = $request->response->total_bytes;
338	if ( $total ) {
339	$step = min( 100, (int) floor( $downloaded / $total * 100 ) );
340	if ( $step >= $last_step + 25 \|\| 100 === $step ) {
341	echo "progress: {$step}% ({$downloaded}/{$total} bytes)\n";
342	$last_step = $step;
343	}
344	} else {
345	echo "downloaded: {$downloaded} bytes\n";
346	}
347	continue;
348	}
349
350	if ( Client::EVENT_FINISHED === $event ) {
351	echo "saved: {$dest}\n";
352	} elseif ( Client::EVENT_FAILED === $event ) {
353	echo "failed: " . $request->error->message . "\n";
354	}
355	}
356	```
357
358	## Keep a sliding window of 10 requests
359
360	<p>For large queues, do not enqueue everything at once. Keep at most ten active requests, enqueue another as each one finishes, and let the client multiplex only that window.</p>
361
362	<!-- snippet:
363	filename: sliding-window.php
364	runnable: true
365	-->
366	```php
367	<?php
368	require '/php-toolkit/vendor/autoload.php';
369
370	use WordPress\HttpClient\Client;
371	use WordPress\HttpClient\Request;
372
373	$urls = array();
374	for ( $i = 1; $i <= 25; $i++ ) {
375	$urls[] = 'https://example.com/?request=' . $i;
376	}
377
378	$client = new Client( array( 'concurrency' => 10 ) );
379	$pending = $urls;
380	$active = array();
381	$done = 0;
382
383	$enqueue_next = function () use ( &$pending, &$active, $client ) {
384	if ( ! $pending ) {
385	return;
386	}
387	$url = array_shift( $pending );
388	$request = new Request( $url, array( 'method' => 'HEAD' ) );
389	$active[ $request->id ] = $request;
390	$client->enqueue( array( $request ) );
391	};
392
393	for ( $i = 0; $i < 10; $i++ ) {
394	$enqueue_next();
395	}
396
397	while ( $active && $client->await_next_event() ) {
398	$request = $client->get_request();
399	$event = $client->get_event();
400
401	if ( Client::EVENT_GOT_HEADERS === $event ) {
402	echo "headers {$request->id}: " . $request->response->status_code . "\n";
403	continue;
404	}
405
406	if ( Client::EVENT_FINISHED === $event \|\| Client::EVENT_FAILED === $event ) {
407	unset( $active[ $request->id ] );
408	$done++;
409	echo "finished {$done}/25, active=" . count( $active ) . "\n";
410	$enqueue_next();
411	}
412	}
413	```
414
415	## Resume a partial download
416
417	<p>Resuming is an HTTP contract between you and the server. Save what you already have, send a <code>Range</code> request for the remaining bytes, and append only if the server returns <code>206 Partial Content</code>.</p>
418
419	<!-- snippet:
420	filename: resume-download.php
421	runnable: true
422	-->
423	```php
424	<?php
425	require '/php-toolkit/vendor/autoload.php';
426
427	use WordPress\HttpClient\Client;
428	use WordPress\HttpClient\Request;
429
430	$url = 'https://raw.githubusercontent.com/WordPress/php-toolkit/trunk/components/Zip/Tests/fixtures/childrens-literature.zip';
431	$dest = sys_get_temp_dir() . '/resume-' . uniqid() . '.zip';
432
433	$client = new Client();
434
435	// Simulate an interrupted first attempt by downloading only the first 32 KB.
436	$first = new Request( $url, array(
437	'headers' => array( 'range' => 'bytes=0-32767' ),
438	) );
439	$stream = $client->fetch( $first );
440	$response = $stream->await_response();
441	file_put_contents( $dest, $stream->consume_all() );
442
443	if ( 206 !== $response->status_code ) {
444	echo "Server did not honor Range; start over with a full download.\n";
445	exit;
446	}
447
448	$downloaded = filesize( $dest );
449	echo "partial file: {$downloaded} bytes\n";
450
451	$resume = new Request( $url, array(
452	'headers' => array( 'range' => 'bytes=' . $downloaded . '-' ),
453	) );
454	$stream = $client->fetch( $resume );
455	$response = $stream->await_response();
456
457	if ( 206 !== $response->status_code ) {
458	echo "Server did not resume; discard partial file and retry from byte 0.\n";
459	exit;
460	}
461
462	while ( ! $stream->reached_end_of_data() ) {
463	$n = $stream->pull( 8192 );
464	if ( 0 === $n ) {
465	break;
466	}
467	file_put_contents( $dest, $stream->consume( $n ), FILE_APPEND );
468	}
469
470	echo "complete file: " . filesize( $dest ) . " bytes\n";
471	echo "saved: {$dest}\n";
472	```
473
474	## Stream-unzip a remote archive
475
476	<p>Mount the remote archive with <code>ZipFilesystem</code>, then copy it into any writable filesystem. <code>SeekableRequestReadStream</code> caches received bytes to a temporary file so <code>ZipFilesystem</code> can read the central directory and seek to entries without first writing the ZIP yourself.</p>
477
478	<!-- snippet:
479	filename: stream-unzip.php
480	runnable: true
481	-->
482	```php
483	<?php
484	require '/php-toolkit/vendor/autoload.php';
485
486	use WordPress\HttpClient\Client;
487	use WordPress\HttpClient\ByteStream\SeekableRequestReadStream;
488	use WordPress\HttpClient\Request;
489	use WordPress\Filesystem\LocalFilesystem;
490	use WordPress\Zip\ZipFilesystem;
491	use function WordPress\Filesystem\copy_between_filesystems;
492	use function WordPress\Filesystem\ls_recursive;
493
494	$url = 'https://raw.githubusercontent.com/WordPress/php-toolkit/trunk/components/Zip/Tests/fixtures/childrens-literature.zip';
495	$root = sys_get_temp_dir() . '/remote-zip-' . uniqid();
496	mkdir( $root );
497
498	$client = new Client();
499	$reader = new SeekableRequestReadStream(
500	new Request( $url ),
501	array( 'client' => $client )
502	);
503
504	$response = $reader->await_response();
505	if ( ! $response->ok() ) {
506	echo "HTTP " . $response->status_code . "\n";
507	exit;
508	}
509
510	$zip = ZipFilesystem::create( $reader );
511	$local = LocalFilesystem::create( $root );
512
513	copy_between_filesystems( array(
514	'source_filesystem' => $zip,
515	'source_path' => '/',
516	'target_filesystem' => $local,
517	'target_path' => '/',
518	) );
519
520	$tree = ls_recursive( $local, '/' );
521	$files = 0;
522	array_walk_recursive( $tree, function ( $value, $key ) use ( &$files ) {
523	if ( 'type' === $key && 'file' === $value ) {
524	$files++;
525	}
526	} );
527
528	echo "extracted {$files} files\n";
529	echo "root: {$root}\n";
530	```
531
532	## Parallel fan-out: fetch many URLs at once
533
534	<p>Enqueue a batch of requests and react to events as they fire. The client multiplexes them — total wall time is roughly the slowest request, not the sum.</p>
535
536	<!-- snippet:
537	filename: fan-out.php
538	runnable: true
539	-->
540	```php
541	<?php
542	require '/php-toolkit/vendor/autoload.php';
543
544	use WordPress\HttpClient\Client;
545	use WordPress\HttpClient\Request;
546
547	$urls = array(
548	'https://wordpress.org/',
549	'https://make.wordpress.org/',
550	'https://developer.wordpress.org/',
551	);
552
553	$client = new Client();
554	$client->enqueue( array_map( function ( $url ) {
555	return new Request( $url, array( 'method' => 'HEAD' ) );
556	}, $urls ) );
557
558	$results = array();
559	while ( $client->await_next_event() ) {
560	$request = $client->get_request();
561	if ( Client::EVENT_GOT_HEADERS === $client->get_event() ) {
562	$results[ $request->url ] = $request->response->status_code;
563	} elseif ( Client::EVENT_FAILED === $client->get_event() ) {
564	$results[ $request->url ] = 'ERR ' . $request->error->message;
565	}
566	}
567
568	foreach ( $results as $url => $status ) {
569	printf( "%-40s %s\n", $url, $status );
570	}
571	```
572
573	## Stream a download to disk without OOM
574
575	<p>Process the body chunk-by-chunk via the event loop. Memory stays flat regardless of file size.</p>
576
577	<!-- snippet:
578	filename: stream-to-disk.php
579	runnable: true
580	-->
581	```php
582	<?php
583	require '/php-toolkit/vendor/autoload.php';
584
585	use WordPress\HttpClient\Client;
586	use WordPress\HttpClient\Request;
587
588	$dest = sys_get_temp_dir() . '/wp-readme.html';
589	$client = new Client();
590	$client->enqueue( array( new Request( 'https://wordpress.org/' ) ) );
591
592	$bytes = 0;
593	@unlink( $dest );
594
595	while ( $client->await_next_event() ) {
596	switch ( $client->get_event() ) {
597	case Client::EVENT_BODY_CHUNK_AVAILABLE:
598	$chunk = $client->get_response_body_chunk();
599	$bytes += strlen( $chunk );
600	file_put_contents( $dest, $chunk, FILE_APPEND );
601	break;
602	case Client::EVENT_FINISHED:
603	echo "Wrote {$bytes} bytes to {$dest}\n";
604	break;
605	}
606	}
607
608	echo "Peak memory: " . round( memory_get_peak_usage( true ) / 1024 / 1024, 2 ) . " MB\n";
609	```
610