Split Pipeline

The Split Pipeline operation allows the user to split a single pipeline into two separate pipelines connected by a SQLstream native stream. This allows the developer to easily refactor his or her streaming application code.

The minimum requirement for using Split Pipeline is that the pipeline must have at least two steps (excluding any route-to-sink step).

To demonstrate how Split Pipeline works:

  1. Create a copy of the Sydney Buses gallery application.

    • If you are running in a VM or sqlstream/complete docker image you can also use the Welcome Page to start test data running.
    • You should see this network diagram:
    original network diagram
  2. Open the “ingest” pipeline by clicking the pencil icon near the top right corner. You should see a Script panel in the bottom left of the window, something like this pipeline guide, which has 10 script steps (the last one is the route to sink):

    original pipeline steps
  3. Pick any one of the script steps amd click the three vertical dots at the right. As long as you are not on the last script step or the route to sink you should be offered “Split to pipeline” as one of the menu options:

    menu for create a split
    • if we pick the last script step, we will see a menu without “Split to pipeline”:
    menu for last step
    • and if we pick the route to sink step, we will see a menu without either “Split to pipeline” or “View Data”:
    menu for route to sink
  4. Let us choose step 6 of the pipeline. Click on the three dots and then click the “Split to pipeline” option. A dialog box will appear where you must enter a name for the “Sink” (stream) and for the new “Pipeline”. Let us enter “categorized_buses” for the sink stream and “ingest_continued” for the pipeline.

    entering sink and pipeline names

    Note: Sink names and Pipeline names share the same namespace within the StreamLab project. Duplicate names are not allowed. StreamLab will tell you if the name is missing clashes with another object.

    warning for missing or duplicate names
  5. Click the “Split Pipeline” button. When StreamLab has finished you should see the first 6 steps remain in the pipeline; subsequent steps have been moved. Step 7 has been added as a route to sink to the new “categorized_buses” stream:

    the original pipeline after splitting
  6. All later steps (and any existing route to sink) in the “ingestion” pipeline will be moved to the new pipeline (named “ingest_continued” in the image). A new SQLstream stream (named “categorized_buses” in the image) will be created which will be the sink for the “ingest” pipeline and the source for the new “ingest_continued” pipeline.

    the network diagram after splitting
  7. When you open the “ingest_continued” pipeline you can see the steps that were numbered 7,8,9 plus the route to sink step 10 of the “ingest” pipeline are now steps numbered 1,2,3 plus route to sink step 4 of the “ingest_continued” pipeline.

    the new pipeline after splitting