Automating the CI/CD pipeline
In the final stage in the CI/CD pipeline you package and deploy your customizations using a script automation tool. A number of tools are available to do this; in this example, we use Jenkins.
Building a Jenkinsfile to automate your processes
The Jenkinsfile sets the execution sequence of your build
process. It includes the following stages and steps:
Note: You
can skip a stage if you do not have customizations for that stage.
- Stage 1: Package and deploy custom NiFi processors
- This stage consists of the following steps:
- Extract the code from your filerepository.
- Build the source code of the Custom Java Processors.
- Package the code of the Custom Java Processor in a .nar file. Using your preferred build automation utility like Jenkins+Maven plugin create a script that calls the pom.xml targets that will build and package the code into a .NAR file.
- Create a new custom NiFi container with a copy of the NAR file.
- Create a custom image of the .nar file that needs to be copied into
the NIFI docker container, using a
Dockerfile
,COPY commerce-custom-search-bundles.nar /opt/nifi/nifi-current/extensions/commerce-custom-search-bundles.nar
- Push the custom image into the Docker Registry.
- Deploy and Run the new custom NiFi image to see the available processor from the NIFI UI.
- Stage 2: Package and deploy custom NiFi process groups
- This stage consists of the following steps:
- Extract the code from your file repository.
- Using a Dockerfile, build a custom image of the NIFI Registry container with the JSON files in it.
- Deploy the custom NiFi Registry image.
- Start the custom NiFi Registry image.Note: Creating a new NiFi Registry is not a requirement. You can also implement a custom separate NiFi Registry that holds the custom process group.
- In the same Dockerfile used in step 2, include import_flow.sh utility to import the process groups. Refer to Import the connectors into runtime registry
- Stage 3: Package and deploy Ingest connectors
-
- Extract the code from your file repository.
- Using the following Curl command, upload the modified connector
definition JSON file into the Ingest container to persist the
descriptor into ZooKeeper.
post /connector/<*CONNECTOR_NAME*>/upgrade/
Example : Here is s Sample connection upgrade using Curl command:
curl -X POST "http://v91tk.hclcdev.com:30800/connectors/someid.sample/upgrade" -H "accept: */*" -H "Content-Type: application/json" -d "{\"id\":\"string\",\"name\":\"string\",\"description\":\"string\",\"environment\":[\"string\"],\"created\":\"string\",\"modified\":\"string\",\"schedule\":{\"enabled\":true,\"startDate\":\"string\",\"endDate\":\"string\",\"startTime\":\"string\",\"everyInterval\":\"string\"},\"pipes\":[{\"name\":\"string\",\"label\":\"string\",\"version\":0,\"properties\":[{\"name\":\"string\",\"value\":\"string\",\"scope\":{\"name\":\"string\",\"sourceName\":\"string\",\"destinationName\":\"string\",\"type\":\"CONTROLLER_SERVICE\"}}]}],\"schemas\":[{\"name\":\"string\",\"schema\":{}}]}"
- Customize properties and configuration of the default/existing components (update)
- Make property and configuration changes to the process groups, processors, connections, and controller services. To make these changes, place the update descriptor JSON file into the specified volume mount folder (/profile/apps/search-ingest.ear/search-ingest.war/WEB-INF/classes/deployments/customization). Based on this, the Search Ingest Service API automatically runs the Update API with this update descriptor JSON provided for customizing the properties and configuration of the default Ingest connectors.