External Entities

A family of Drupal modules that turns outside data (APIs, datasets, scraped pages) into Drupal entities that render, reference and search like local content, with editors able to override any field.

 

## What it is

External Entities is a layered family of Drupal modules for sites whose content lives partly elsewhere. Records from an external API or dataset become Drupal entities with fields, view modes and Manage display, so pages can show them, reference them and list them like any other content, while the data stays connected to its source.

A shared substrate handles what every integration needs: cached HTTP, one request per source however many parts of a page ask for it, and field-by-field mirroring. A new data source is therefore a small module that plugs into the framework, not a new integration. [Species External Entity](/products/species-ext) and [Editorial Intelligence](/products/editorial-intelligence) are built this way, and the [Research Library](/products/research-library) catalogue sits on top of it.



 

 





 

## At a glance

Substrate`entities_ext`, built on the engine of the contributed External Entities module; `request_bus`, the outbound call layer; `dynamic_mirror_base`, per-field mirroringGeneric types`entity_ext`: an external entity type backed by an HTTP API, defined in configuration instead of a custom storage pluginSource modulesOne per external source or dataset, among them `wikipedia_ext`, `data_ext`, `scripture_x`, `books_ext`, `scholar_ext`, `places_ext`, `artworks_ext`, `iiif_x`, `wikidata_x` and `authority_x`Drupal core10 or 11Requires`key`; `entities_ext` also requires `request_bus`, which itself needs only Drupal core and `key`StatusBuilt and in active development, no 1.0 release (modules at `0.1.0-dev`). The substrate runs in production on [Biodiversa](/projects/biodiversa), [Museo Avellonia](/projects/museo-avellonia) and the [Quercus Project](/projects/quercus-project); most source modules are in development on MADDev's own research platform.

 

 





 

## What it adds to a Drupal site

### External entity types

`entities_ext` defines no content type of its own. Each consuming module registers its external entity types on the shared substrate, for example `wikipedia_page`, `species_x` or `editorial_news`. The `xntt_views` submodule lists external entities in Views, `xnttsql` reads external SQL schemas, and `entities_ext_mng` watches each source for missing endpoints or changed fields and shields rendering while a source is down.

### Per-field mirroring

`dynamic_mirror_base` decides, field by field, where a value comes from: live from the source, a mirrored copy, or a local value kept for good. An editor can lock any field to a local value that survives every refresh from upstream. `dmb_ui` gives editors lock, edit, resync and unlock actions, and `dmb_harvest` refreshes mirrored values on a schedule.

### Single facts

`data_ext` resolves one external fact, such as a caption, a price or a coordinate, without mirroring a whole record. It ships with Library of Congress Classification and Universal Decimal Classification captions.

### Plugin types

- `@StorageClient` and `@ExternalSource`: one plugin per upstream API, tried in order of weight until one answers
- field mapper, property mapper, data processor and data aggregator plugins for shaping a source's data into fields
- `@DmbAdapter`: connects an entity type to per-field mirroring
- source plugins per module, such as `@WikipediaSource`, `@ScriptureSource`, `@ScholarSource`, `@ArtworkSource` and `@DatumSource`

### Services

`request_bus.bus` (also available as `entities_ext.request_bus`), `data_ext.resolver`, `wikidata_x.client`, `authority_x.resolver`, `scholar_ext.resolver`, `iiif_x.manifest_loader`.

### Administration

- `/admin/config/services/dmb/policy`: per-field mirroring policy
- `/admin/content/dmb-overrides`: every local override
- `/admin/config/services/dmb/harvest`: the refresh schedule
- `/admin/config/services/wikipedia-ext`: Wikipedia settings

### Developer workbench

`entities_ext_builder`, in development, builds external entity types without hand-written configuration: it probes a live endpoint, previews how fields will be extracted before anything is saved, and can use an AI provider to find an API's documentation. It holds no runtime logic, so uninstalling it leaves every external entity type it built working.



 

 





 

## Data sources and licences

- **Encyclopaedic:** Wikipedia and any MediaWiki site (`wikipedia_ext`); Wikidata (`wikidata_x`)
- **Books and research:** Open Library and Google Books, reconciled by ISBN (`books_ext`); OpenAlex, Crossref, arXiv, PubMed and Semantic Scholar (`scholar_ext`)
- **Names:** VIAF, ORCID, ULAN, Wikidata and ISNI identifiers reconciled to one person or organisation (`authority_x`)
- **Places:** Pleiades and the World Historical Gazetteer (`places_ext`)
- **Collections:** the Metropolitan Museum of Art, the Art Institute of Chicago and the Cleveland Museum of Art, none needing a key (`artworks_ext`); IIIF image manifests, versions 2 and 3 (`iiif_x`)
- **Texts:** Hebrew, Greek and Coptic scripture sources and translations (`scripture_x`)
- **Classification captions:** id.loc.gov and the UDC Summary (`data_ext`)

Every outbound call goes through `request_bus`, which caches and combines requests; sources that need a key keep it in the `key` module, not in code. Where content comes from an outside source, `copyrights_ext` writes its licence and credit into the site's copyright field as it is imported or mirrored, which is what [Copyrights Guard](/products/copyrights-guard) then enforces.



 

 





 

## Design decisions

- **Register a need, do not fire a request.** Callers tell the request bus what they need; it merges identical requests, combines the fields different parts of a page ask for, batches per source and remembers the answer, including "no data", for the rest of the request. A page with ten Wikipedia-backed sections makes one Wikipedia call.
- **A dead source never breaks a page.** A source that is down, slow or malformed returns nothing instead of throwing, and the page renders without it.
- **Only the fields a page asks for.** Loading an external entity resolves the fields actually demanded, not the whole external record.
- **Three settings per field, not one mode.** Where a value comes from (live, mirror or local), what happens when the source drops it (remove or keep), and where the value is stored are set independently; an editor's lock overrides all three.
- **Soft dependencies.** Modules depend hard only on the substrate and `key`; every other integration is checked at runtime, so the family keeps working when a module is absent.
- **Names that say what a module is.** `_ext` marks a source module, `_x` a module that extends the framework, and no suffix a utility with no dependency on it.



 

 





## Start a project.

Tell us which outside data your site depends on, and how current it has to be.



[Get in touch →](/contact)