Processing Backstage Entities

How to enrich catalog entities with Processors

Backstage logo © Spotify

Backstage is an open platform for building developer portals that can be easily extended to your needs by the power of entity processors. Entity processing in Backstage is done after the entity has been created by an entity provider. I recommend you first introduce yourself with the Backstage documentation about entities and maybe read through my previous articles about Backstage.

Basics

Each entity processor has to implement the CatalogProcessor interface. Processors can be developed directly into the application or plugin. Let’s start off by creating the processor to packages/backend/src/processors directory. We are going to call it myProcessor.ts:

// packages/backend/src/processors/myProcessor.ts
import {
  CatalogProcessor,
  CatalogProcessorCache,
  CatalogProcessorEmit,
} from '@backstage/plugin-catalog-backend';
import { Entity } from '@backstage/catalog-model';
import { LocationSpec } from '@backstage/plugin-catalog-common';

export class MyProcessor implements CatalogProcessor {
  getProcessorName(): string {
    return 'MyProcessor';
  }

  // Run first
  async preProcessEntity(
    entity: Entity,
    location: LocationSpec,
    emit: CatalogProcessorEmit,
    originLocation: LocationSpec,
    cache: CatalogProcessorCache,
  ): Promise<Entity> {
    return entity;
  }

  // Run after preProcess
  async validateEntityKind(entity: Entity): Promise<boolean> {
    // Return true if entity kind and it's fields are valid
    // and can be processed by this processor. Return false if the entity 
    // cannot be processed with this processor. Throw error if the entity
    // is invalid.
    return true;
  }

  // Run after validateEntityKind
  async postProcessEntity(
    entity: Entity,
    location: LocationSpec,
    emit: CatalogProcessorEmit,
    cache: CatalogProcessorCache,
  ): Promise<Entity> {
    return entity;
  }
}

Now, to get the processor to do something, you have to also add it to the catalog builder in packages/backend/src/plugins/catalog.ts :

// packages/backend/src/plugins/catalog.ts
// createPlugin()...
  const builder = CatalogBuilder.create(env);
  builder.addProcessor(new MyProcessor());
// ...

The order of processors is important as the pre-processing, validating, and post-processing are called in the same order as the processors are added. This means if you have two processors A and B, both added to the builder in that order, the execution order will be A-pre, B-pre, A-validate, B-validate, A-post, and B-post.

Adding functionality

Next, you should add at least one of the processing functions. For this example, we are going to do pre-processing for the entity. What the following code does is it reads the entity from its origin location, creates a group for the user's company, and adds a custom relation between these two. You can check out the catalog provider for this in my previous article. It’s good practice, if possible, to use existing well-known relations, instead of inventing your own as they are automatically supported by the Backstage.

// packages/backend/src/processors/myProcessor.ts
// Single user from https://jsonplaceholder.typicode.com/users
interface User {
  id: number;
  name: string;
  username: string;
  email: string;
  company?: {
    name: string;
    catchPhrase: string;
    bs: string;
  };
}

// ...
  // This function creates Group entity object
  private createCompanyGroup(user: User) {
    if (!user.company) {
      return null;
    }
    return {
      apiVersion: 'backstage.io/v1alpha1',
      kind: 'Group',
      metadata: {
        // Allowed characters are [a-z0-9A-Z-_.]
        name: user.company.name.toLowerCase().replaceAll(/\s/g, '-'),
        title: user.company.name,
        description: user.company.catchPhrase,
        tags: user.company.bs.split(" "),
      },
      spec: {
        type: 'company',
        children: [],
      },
    };
  }

  async preProcessEntity(
    entity: Entity,
    location: LocationSpec,
    emit: CatalogProcessorEmit,
    originLocation: LocationSpec,
    cache: CatalogProcessorCache,
  ): Promise<Entity> {
    // This processor only processes user entities
    if (entity.kind.toLowerCase() !== 'user') {
      return entity;
    }
    // Add more checks for what entities to process if necessary

    // Fetch the user from it's origin location
    // For example https://jsonplaceholder.typicode.com/users/8
    const res = await fetch(originLocation.target, {
      method: 'get',
    });
    const user: User = await res.json();

    // Create company entity JSON object from the response
    const company = this.createCompanyGroup(user);
    if (!company) {
      // If the company info is missing, emit error
      emit(processingResult.notFoundError(originLocation, 'Company not found'));
      return entity;
    }

    // Create the actual company entity and emit it
    const companyResult = processingResult.entity(
      originLocation,
      company,
    ) as CatalogProcessorEntityResult;
    emit(companyResult);

    // Create entity references for company and user
    // These are needed to be able to create relations between the entities
    const companyEntityRef = getCompoundEntityRef(companyResult.entity);
    const userEntityRef = getCompoundEntityRef(entity);

    // Emit custom relationship between company group and users
    emit(
      processingResult.relation({
        source: companyEntityRef,
        target: userEntityRef,
        type: 'employs',
      }),
    );

    emit(
      processingResult.relation({
        source: userEntityRef,
        target: companyEntityRef,
        type: 'employedTo',
      }),
    );

    return entity;
  }
// ...

Additionally, there are some other processing results you can emit:

// Example of general error in processing
emit(
  processingResult.generalError(originLocation, 'Invalid company given'),
);
// Example of input error in processing
emit(
  processingResult.inputError(originLocation, 'Invalid input from service'),
);
// Emit new Location entity to the catalog which is fetched by Backstage
emit(
  processingResult.location({
    ...originLocation,
    target: 'https://jsonplaceholder.typicode.com/companies',
  }),
);
// Emit refresh for entity
emit(processingResult.refresh(stringifyEntityRef(entity)));

NOTE: If you emit errors, the entity will be dropped out of the catalog. If the entity should still be visible, just return the original entity instead of emitting errors.

After all of the processors have completed processing, the entities, relations, and other information is passed to the Stitcher which combines the information and creates the final entities for the catalog and the search:

If you need to access existing entities in the catalog, you can create the catalog client in the catalog.ts file and pass it to the processor in the constructor. It’s also possible to pass other environment objects to the processor if needed.

// packages/backend/src/plugins/catalog.ts
// createPlugin()...  
const client = new CatalogClient({ discoveryApi: env.discovery });
const { config, logger } = env;
builder.addProcessor(new MyProcessor(client, config, logger));

Also, you should take cache into use when applicable. The CatalogProcessorCache is processor specific so it’s not shared with other processors. You can use for example HTTP ETag header values to cache responses from external sources.

And that’s about it for entity processors in Backstage. In the next article, we are going to go through Search collators which allow you to extend the Backstage search with new information.

I am Heikki Hellgren, Lead Developer, and technology enthusiast working at OP Financial Group. My interests are in software construction, tools, automatic testing, and continuous improvement. You can connect with me on Twitter and check out my website for more information.