README.md — Jetpack – WP Security, Backup, Speed, & Growth 16.1-beta.2

jetpack / vendor / wp-php-toolkit / data-liberation / README.md

jetpack / vendor / wp-php-toolkit / data-liberation Last commit date

BlockMarkup 6 days ago CSS 6 days ago DataFormatConsumer 6 days ago DataFormatProducer 6 days ago EntityReader 6 days ago EntityWriter 6 days ago Importer 6 days ago URL 6 days ago vendor-patched 6 days ago LICENSE.md 6 days ago README.md 6 days ago class-dataliberationexception.php 6 days ago class-dataliberationhtmlprocessor.php 6 days ago class-importentity.php 6 days ago composer.json 6 days ago rector.php 6 days ago

README.md

284 lines

1	---
2	slug: dataliberation
3	title: DataLiberation
4	install: wp-php-toolkit/data-liberation
5
6	see_also:
7	- ../learn/03-importing-content.html \| Tutorial — Markdown to WXR \| The chapter that walks through importing a folder of Markdown files into WordPress via the toolkit.
8	- markdown \| Markdown \| Use Markdown as a source or destination format.
9	- blockparser \| BlockParser \| Analyze serialized blocks inside post content.
10	- httpclient \| HttpClient \| Download media and remote source data while importing.
11	---
12
13	Streaming WordPress import/export. WXR, SQL, block markup — process entities one at a time instead of building whole-dataset object graphs.
14
15	## Why this exists
16
17	<p>WordPress content should be portable, but real migrations cross several formats. A site export might arrive as WXR, a Markdown folder, or entities from another CMS. URLs can hide in block attributes, HTML, CSS, feeds, GUIDs, and post meta. Importers must also resume after a failed media download or upload.</p>
18
19	<p>The DataLiberation component streams WordPress-shaped data through readers, transformers, and writers. It models posts, terms, comments, attachments, and metadata as <code>ImportEntity</code> objects, then lets a pipeline rewrite each entity without loading the full export into memory.</p>
20
21	<p>The API reflects specific migration bugs: relative URLs in known block attributes, URLs inside inline CSS, self-closing block comments that must keep their shape, and origin-only URLs whose trailing slash style should not change during a rewrite.</p>
22
23	<p>Reach for it when the job combines formats: build WXR from another CMS, rewrite a staging export for production, frontload remote assets, or compose Markdown, XML, HTML, CSS, and URL rewriting into one pipeline.</p>
24
25	## Write a WXR file in five lines
26
27	<p>Stream a single post into a WXR document via <code>WXRWriter</code>. The writer emits each entity to the output stream and only keeps the small amount of state needed for the current document.</p>
28
29	<!-- snippet:
30	filename: wxr-quickstart.php
31	runnable: true
32	-->
33	```php
34	<?php
35	require '/php-toolkit/vendor/autoload.php';
36
37	use WordPress\ByteStream\MemoryPipe;
38	use WordPress\DataLiberation\EntityWriter\WXRWriter;
39	use WordPress\DataLiberation\ImportEntity;
40
41	$pipe = new MemoryPipe();
42	$writer = new WXRWriter( $pipe );
43	$writer->append_entity( new ImportEntity( 'post', array(
44	'post_title' => 'Hello',
45	'content' => 'World.',
46	'post_id' => '1',
47	'status' => 'publish',
48	) ) );
49	$writer->finalize();
50	$writer->close_writing();
51	$pipe->close_writing();
52	$wxr = $pipe->consume_all();
53
54	echo "bytes: " . strlen( $wxr ) . "\n";
55	echo false !== strpos( $wxr, '<title>Hello</title>' ) ? "title exported\n" : "title missing\n";
56	echo false !== strpos( $wxr, '<wp:status>publish</wp:status>' ) ? "status exported\n" : "status missing\n";
57	```
58
59	<!-- expected-output -->
60	```
61	bytes: 475
62	title exported
63	status exported
64	```
65
66	## Build a WXR programmatically from any source
67
68	<p>The writer doesn't care where entities come from. Loop over rows from a CMS, a CSV, or a Notion API dump and emit posts plus their meta and comments.</p>
69
70	<!-- snippet:
71	filename: build-wxr.php
72	runnable: true
73	-->
74	```php
75	<?php
76	require '/php-toolkit/vendor/autoload.php';
77
78	use WordPress\ByteStream\MemoryPipe;
79	use WordPress\DataLiberation\EntityWriter\WXRWriter;
80	use WordPress\DataLiberation\ImportEntity;
81
82	$rows = array(
83	array( 'id' => 10, 'title' => 'About', 'body' => '<p>About us.</p>', 'tags' => array( 'company' ) ),
84	array( 'id' => 11, 'title' => 'Blog', 'body' => '<p>Hello world.</p>', 'tags' => array( 'news', 'launch' ) ),
85	);
86
87	$pipe = new MemoryPipe();
88	$writer = new WXRWriter( $pipe );
89
90	foreach ( $rows as $row ) {
91	$writer->append_entity( new ImportEntity( 'post', array(
92	'post_id' => (string) $row['id'],
93	'post_title' => $row['title'],
94	'content' => $row['body'],
95	'status' => 'publish',
96	'post_type' => 'post',
97	) ) );
98	foreach ( $row['tags'] as $i => $tag ) {
99	$writer->append_entity( new ImportEntity( 'term', array(
100	'term_id' => (string) ( $row['id'] * 100 + $i ),
101	'taxonomy' => 'post_tag',
102	'slug' => $tag,
103	'parent' => '0',
104	) ) );
105	}
106	}
107
108	$writer->finalize();
109	$writer->close_writing();
110	$pipe->close_writing();
111
112	$wxr = $pipe->consume_all();
113	echo "items: " . substr_count( $wxr, '<item>' ) . "\n";
114	echo "terms: " . substr_count( $wxr, '<wp:term>' ) . "\n";
115	echo false !== strpos( $wxr, '<title>Blog</title>' ) ? "Blog post exported\n" : "Blog post missing\n";
116	```
117
118	<!-- expected-output -->
119	```
120	items: 2
121	terms: 3
122	Blog post exported
123	```
124
125	## Read entities from a WXR file incrementally
126
127	<p><code>WXREntityReader</code> emits one entity at a time. Memory use is driven by the current entity and parser buffers rather than the total file size.</p>
128
129	<!-- snippet:
130	filename: wxr-read.php
131	runnable: true
132	-->
133	```php
134	<?php
135	require '/php-toolkit/vendor/autoload.php';
136
137	use WordPress\DataLiberation\EntityReader\WXREntityReader;
138
139	$wxr = <<<XML
140	<?xml version="1.0" encoding="UTF-8" ?>
141	<rss version="2.0" xmlns:wp="http://wordpress.org/export/1.2/" xmlns:content="http://purl.org/rss/1.0/modules/content/">
142	<channel>
143	<title>Demo</title>
144	<item><title>First</title><wp:post_id>1</wp:post_id><wp:post_type>post</wp:post_type><content:encoded>Body 1</content:encoded></item>
145	<item><title>Second</title><wp:post_id>2</wp:post_id><wp:post_type>post</wp:post_type><content:encoded>Body 2</content:encoded></item>
146	</channel>
147	</rss>
148	XML;
149
150	$reader = WXREntityReader::create();
151	$reader->append_bytes( $wxr );
152	$reader->input_finished();
153
154	while ( $reader->next_entity() ) {
155	$entity = $reader->get_entity();
156	echo $entity->get_type() . ': ' . json_encode( $entity->get_data() ) . "\n";
157	}
158	```
159
160	<!-- expected-output -->
161	```
162	site_option: {"option_name":"blogname","option_value":"Demo"}
163	post: {"post_title":"First","post_id":"1","post_type":"post","post_content":"Body 1"}
164	post: {"post_title":"Second","post_id":"2","post_type":"post","post_content":"Body 2"}
165	```
166
167	## Streaming transform: rewrite URLs while copying WXR
168
169	<p>Wire reader to writer to rewrite a WXR file on the fly. This pattern is how you migrate a staging export to production: swap <code>staging.example.com</code> for <code>example.com</code> while holding only the current entity and output buffers.</p>
170
171	<!-- snippet:
172	filename: rewrite-urls.php
173	runnable: true
174	-->
175	```php
176	<?php
177	require '/php-toolkit/vendor/autoload.php';
178
179	use WordPress\ByteStream\MemoryPipe;
180	use WordPress\DataLiberation\EntityReader\WXREntityReader;
181	use WordPress\DataLiberation\EntityWriter\WXRWriter;
182	use WordPress\DataLiberation\ImportEntity;
183
184	$source_xml = <<<XML
185	<?xml version="1.0" encoding="UTF-8" ?>
186	<rss version="2.0" xmlns:wp="http://wordpress.org/export/1.2/" xmlns:content="http://purl.org/rss/1.0/modules/content/">
187	<channel>
188	<item><title>Hello</title><wp:post_id>1</wp:post_id><wp:post_type>post</wp:post_type>
189	<content:encoded>Visit https://staging.example.com/about for more.</content:encoded></item>
190	</channel>
191	</rss>
192	XML;
193
194	$reader = WXREntityReader::create();
195	$reader->append_bytes( $source_xml );
196	$reader->input_finished();
197
198	$out_pipe = new MemoryPipe();
199	$writer = new WXRWriter( $out_pipe );
200
201	while ( $reader->next_entity() ) {
202	$entity = $reader->get_entity();
203	$data = $entity->get_data();
204	foreach ( array( 'post_content', 'content', 'description' ) as $field ) {
205	if ( isset( $data[ $field ] ) ) {
206	$data[ $field ] = str_replace( 'staging.example.com', 'example.com', $data[ $field ] );
207	}
208	}
209	if ( 'post' === $entity->get_type() ) {
210	$data['content'] = isset( $data['post_content'] ) ? $data['post_content'] : ( isset( $data['content'] ) ? $data['content'] : '' );
211	}
212	$writer->append_entity( new ImportEntity( $entity->get_type(), $data ) );
213	}
214
215	$writer->finalize();
216	$writer->close_writing();
217	$out_pipe->close_writing();
218
219	$wxr = $out_pipe->consume_all();
220	echo false !== strpos( $wxr, 'https://example.com/about' ) ? "new URL present\n" : "new URL missing\n";
221	echo false === strpos( $wxr, 'staging.example.com' ) ? "old URL removed\n" : "old URL still present\n";
222	```
223
224	<!-- expected-output -->
225	```
226	new URL present
227	old URL removed
228	```
229
230	## Render Markdown into a WXR import in one pipeline
231
232	<p>Compose <code>MarkdownConsumer</code> with <code>WXRWriter</code> to publish a folder of Markdown directly as a WordPress import file.</p>
233
234	<!-- snippet:
235	filename: md-to-wxr.php
236	runnable: true
237	-->
238	```php
239	<?php
240	require '/php-toolkit/vendor/autoload.php';
241
242	use WordPress\ByteStream\MemoryPipe;
243	use WordPress\DataLiberation\EntityWriter\WXRWriter;
244	use WordPress\DataLiberation\ImportEntity;
245	use WordPress\Markdown\MarkdownConsumer;
246
247	@mkdir( '/tmp/md-src', 0777, true );
248	file_put_contents( '/tmp/md-src/hello.md', "---\ntitle: Hello\n---\n\n# Hello\n\nFirst post." );
249	file_put_contents( '/tmp/md-src/second.md', "---\ntitle: Second\n---\n\nMore text here." );
250
251	$pipe = new MemoryPipe();
252	$writer = new WXRWriter( $pipe );
253
254	$id = 1;
255	foreach ( glob( '/tmp/md-src/*.md' ) as $path ) {
256	$consumer = new MarkdownConsumer( file_get_contents( $path ) );
257	$consumer->consume();
258	$writer->append_entity( new ImportEntity( 'post', array(
259	'post_id' => (string) $id++,
260	'post_title' => $consumer->get_meta_value( 'title' ) ?: basename( $path, '.md' ),
261	'content' => $consumer->get_block_markup(),
262	'status' => 'publish',
263	'post_type' => 'post',
264	'post_name' => basename( $path, '.md' ),
265	) ) );
266	}
267
268	$writer->finalize();
269	$writer->close_writing();
270	$pipe->close_writing();
271
272	$wxr = $pipe->consume_all();
273	echo "posts: " . substr_count( $wxr, '<item>' ) . "\n";
274	echo false !== strpos( $wxr, '<!-- wp:heading' ) ? "block markup exported\n" : "block markup missing\n";
275	echo false !== strpos( $wxr, '<title>Second</title>' ) ? "frontmatter title exported\n" : "frontmatter title missing\n";
276	```
277
278	<!-- expected-output -->
279	```
280	posts: 2
281	block markup exported
282	frontmatter title exported
283	```
284