Join Pipeline

Join Pipeline allows the user to combine the steps of two pipelines into a single pipeline.

This section covers the following topics:

Preparing the Example

We are going to use the example project from Split Pipeline, which was derived from the Sydney Buses example.

After creating the project and splitting the ingest pipeline we have this in our Network Diagram:

the network diagram after splitting

Restrictions

Currently, a pipeline is only a candidate to be joined to its upstream pipeline if all the following are true:

  • The pipeline has no route-to-sink
  • The source of the pipeline is a native stream
  • That native stream is not the source for any other pipeline
  • That native stream is the sink for only one upstream pipeline.

If any of these conditions does not hold, the Join menu option will not be available.

Joining Pipelines

We cannot simply re-join ingest and ingest_continued. The ingest_continued pipeline has a sink (nexus) which disables Join.

  1. First we need to disconnect the nexus sink from ingest_continued by removing the route-to-sink step.

    remove nexus sink
  2. Now we have an example of the kind of pipelines we can combine, we can go back to the Network Diagram and right click on ingest_continued, then select Join:

    ready to join

    You will notice that:

    • The ingest pipeline is opened The steps of ingest_continued have been added to the end of ingest in the Script section.
    • The sink categorized_buses is moved to the end of ingest, after the additional steps
    ingest after join
    • Back on the Network Diagram, the second pipeline ingest_continued has been deleted. The ingest pipeline still routes to categorized_buses.
    pipeline after join
  3. We can now disconnect the categorized_buses sink from ingest by removing the route to sink step:

    remove categorized_buses
  4. And we can re-route ingest to the nexus sink (be sure to select nexus as the sink):

    route to nexus
  5. And finally, from the Network Diagram you can delete the categorized_buses sink that was added when we split the pipelines originally. We are now back to where we started before the Split Pipeline:

    original network diagram