Netflow is a network protocol developed by Cisco Systems to harvest information about IP traffic. It provides the ability to collect IP network traffic as it enters or exits an interface. The data provided by netflow can be analyzed to determine things such as the source and destination of traffic, class of service, the cause of congestion and many other factors. Netflow consists of data templates, options templates and flowsets containing the data respect to the incoming templates. In s-Server, the implementation is done for the collector, which processes the data received from the exporter. Netflow has evolved through the following versions :
In s-Server we support all versions of netflow.
The Netflow v9/IPFIX UDF (netflowCollector) is used to access the IP Flow information from the data networks; these network data are gathered by network elements like router and switches in the form of flow data and exported to collectors for further processing. The collected data provides fine-grained metering for highly flexible and detailed resource usage accounting.
A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device. These collected flows are exported to an external device, the NetFlow collector. Network flows are highly granular; for example, flow records include details such as IP addresses, packet and byte counts, timestamps, Type of Service (ToS), application ports, input and output interfaces, etc. Exported NetFlow data is used for a variety of purposes, including enterprise accounting and departmental chargebacks, ISP billing, data warehousing, network monitoring, capacity planning, application monitoring and profiling, user monitoring and profiling, security analysis, and data mining for marketing purposes.
As of now s-Server only supports UDP Protocol for Netflow v9 and IPFIX version. As per the lastest RFC, the supported protocols for IPFIX version (v10) are as follows :
s-Server aleady supports UTF-8 encoding for string type as part of the collector.
As of now, the template withdrawal functionality is not included in s-Server version of Netflow IPFIX Collector. Withdrawal of template is specific to the IPFIX version (v10) only. This mechanism is not applicable for UPD Protocol. Template withdrawal will be supported as part of TCP/SCTP Protocol implementation.
In case of UDP Protocol, the life cycle of the template is maintained as per the received time. As soon as a new template comes in to the collector, the old template is discarded and the new one is taken into consideration. The templates are considered in the sequence they are received.
The netflow message data contains a few fields as part of the message and a message header in each and every message. The details are as follows:
Field Name | v5 Supported | v9 Supported | IPFIX Supported | ** Field Details** |
---|---|---|---|---|
REPORTER | Yes | Yes | Yes | Source IP Deatils |
ROWTIME | Yes | Yes | Yes | Time of parsing the data |
NETFLOW_VERSION | Yes | Yes | Yes | Netflow version details |
FLOW_COUNT | Yes | Yes | No | Number of Flowset records, both template and data |
FLOW_LENGTH | No | No | Yes | Total number of bytes present in the flowset |
SYSTEM_UPTIME | Yes | Yes | No | Time in millisecond, the device was first booted |
EXPORT_TIME | No | No | Yes | Time the message left the exporter system expresed in seconds since the UNIX epoch of 1 January 1970 at 00:00 UTC |
UNIX_SECS | Yes | Yes | No | Seconds since 0000 Coordinated Universal Time (UTC) 1970 |
UNIX_NSECS | Yes | No | No | Residual nanoseconds since 0000 Coordinated Universal Time 1970 |
FLOW_SEQUENCE | Yes | No | No | Sequence counter of total flows seen |
SEQUENCE_NUMBER | No | Yes | Yes | Incremental sequence counter of all export packets sent by this export device; this value is cumulative, and it can be used to identify whether any export packets have been missed |
SOURCE_ID | No | Yes | No | The Source ID field is a 32-bit value that is used to guarantee uniqueness for all flows exported from a particular device. |
OBSERVATION_DOMAIN_ID | No | No | Yes | A 32-bit identifier of the Observation Domain that is locally unique to the Exporting Process. The Exporting Process uses the Observation Domain ID to uniquely identify to the Collecting Process the Observation Domain that metered the Flows. |
SCOPE_SYSTEM | No | Yes | No | Netflow v9 Specific Scope Type Detail |
SCOPE_INTERFACE | No | Yes | No | Netflow v9 Specific Scope Type Detail |
SCOPE_LINE_CARD | No | Yes | No | Netflow v9 Specific Scope Type Detail |
SCOPE_NETFLOW_CACHE | No | Yes | No | Netflow v9 Specific Scope Type Detail |
SCOPE_TEMPLATE | No | Yes | No | Netflow v9 Specific Scope Type Detail |
Note: These fields are added as part of the IANA_STD_DEF.csv, file present in the path plugin/IANA_STD_DEF.csv. For adding any missing fields, please reach out to the s-Server tech support team or replace the existing file to support the missing fields as in the file above on top of the existing entries. Make sure, no 2 field name should be associated with the same element Id. Field details are same for both Netflow v9 and Netflow Ipfix versions, and the details are as follows :
FIELD NAME | ELEMENT/FIELD ID | ABSTRACT DATA TYPE |
---|---|---|
OCTETDELTACOUNT | 1 | unsigned64 |
PACKETDELTACOUNT | 2 | unsigned64 |
DELTAFLOWCOUNT | 3 | unsigned64 |
PROTOCOLIDENTIFIER | 4 | unsigned8 |
IPCLASSOFSERVICE | 5 | unsigned8 |
TCPCONTROLBITS | 6 | unsigned16 |
SOURCETRANSPORTPORT | 7 | unsigned16 |
SOURCEIPV4ADDRESS | 8 | ipv4Address |
SOURCEIPV4PREFIXLENGTH | 9 | unsigned8 |
INGRESSINTERFACE | 10 | unsigned32 |
DESTINATIONTRANSPORTPORT | 11 | unsigned16 |
DESTINATIONIPV4ADDRESS | 12 | ipv4Address |
DESTINATIONIPV4PREFIXLENGTH | 13 | unsigned8 |
EGRESSINTERFACE | 14 | unsigned32 |
IPNEXTHOPIPV4ADDRESS | 15 | ipv4Address |
BGPSOURCEASNUMBER | 16 | unsigned32 |
BGPDESTINATIONASNUMBER | 17 | unsigned32 |
BGPNEXTHOPIPV4ADDRESS | 18 | ipv4Address |
POSTMCASTPACKETDELTACOUNT | 19 | unsigned64 |
POSTMCASTOCTETDELTACOUNT | 20 | unsigned64 |
FLOWENDSYSUPTIME | 21 | unsigned32 |
FLOWSTARTSYSUPTIME | 22 | unsigned32 |
POSTOCTETDELTACOUNT | 23 | unsigned64 |
POSTPACKETDELTACOUNT | 24 | unsigned64 |
MINIMUMIPTOTALLENGTH | 25 | unsigned64 |
MAXIMUMIPTOTALLENGTH | 26 | unsigned64 |
SOURCEIPV6ADDRESS | 27 | ipv6Address |
DESTINATIONIPV6ADDRESS | 28 | ipv6Address |
SOURCEIPV6PREFIXLENGTH | 29 | unsigned8 |
DESTINATIONIPV6PREFIXLENGTH | 30 | unsigned8 |
FLOWLABELIPV6 | 31 | unsigned32 |
ICMPTYPECODEIPV4 | 32 | unsigned16 |
IGMPTYPE | 33 | unsigned8 |
SAMPLINGINTERVAL | 34 | unsigned32 |
SAMPLINGALGORITHM | 35 | unsigned8 |
FLOWACTIVETIMEOUT | 36 | unsigned16 |
FLOWIDLETIMEOUT | 37 | unsigned16 |
ENGINETYPE | 38 | unsigned8 |
ENGINEID | 39 | unsigned8 |
EXPORTEDOCTETTOTALCOUNT | 40 | unsigned64 |
EXPORTEDMESSAGETOTALCOUNT | 41 | unsigned64 |
EXPORTEDFLOWRECORDTOTALCOUNT | 42 | unsigned64 |
IPV4ROUTERSC | 43 | ipv4Address |
SOURCEIPV4PREFIX | 44 | ipv4Address |
DESTINATIONIPV4PREFIX | 45 | ipv4Address |
MPLSTOPLABELTYPE | 46 | unsigned8 |
MPLSTOPLABELIPV4ADDRESS | 47 | ipv4Address |
SAMPLERID | 48 | unsigned8 |
SAMPLERMODE | 49 | unsigned8 |
SAMPLERRANDOMINTERVAL | 50 | unsigned32 |
CLASSID | 51 | unsigned8 |
MINIMUMTTL | 52 | unsigned8 |
MAXIMUMTTL | 53 | unsigned8 |
FRAGMENTIDENTIFICATION | 54 | unsigned32 |
POSTIPCLASSOFSERVICE | 55 | unsigned8 |
SOURCEMACADDRESS | 56 | macAddress |
POSTDESTINATIONMACADDRESS | 57 | macAddress |
VLANID | 58 | unsigned16 |
POSTVLANID | 59 | unsigned16 |
IPVERSION | 60 | unsigned8 |
FLOWDIRECTION | 61 | unsigned8 |
IPNEXTHOPIPV6ADDRESS | 62 | ipv6Address |
BGPNEXTHOPIPV6ADDRESS | 63 | ipv6Address |
IPV6EXTENSIONHEADERS | 64 | unsigned32 |
MPLSTOPLABELSTACKSECTION | 70 | octetArray |
MPLSLABELSTACKSECTION2 | 71 | octetArray |
MPLSLABELSTACKSECTION3 | 72 | octetArray |
MPLSLABELSTACKSECTION4 | 73 | octetArray |
MPLSLABELSTACKSECTION5 | 74 | octetArray |
MPLSLABELSTACKSECTION6 | 75 | octetArray |
MPLSLABELSTACKSECTION7 | 76 | octetArray |
MPLSLABELSTACKSECTION8 | 77 | octetArray |
MPLSLABELSTACKSECTION9 | 78 | octetArray |
MPLSLABELSTACKSECTION10 | 79 | octetArray |
DESTINATIONMACADDRESS | 80 | macAddress |
POSTSOURCEMACADDRESS | 81 | macAddress |
INTERFACENAME | 82 | string |
INTERFACEDESCRIPTION | 83 | string |
SAMPLERNAME | 84 | string |
OCTETTOTALCOUNT | 85 | unsigned64 |
PACKETTOTALCOUNT | 86 | unsigned64 |
FLAGSANDSAMPLERID | 87 | unsigned32 |
FRAGMENTOFFSET | 88 | unsigned16 |
FORWARDINGSTATUS | 89 | unsigned8 |
MPLSVPNROUTEDISTINGUISHER | 90 | octetArray |
MPLSTOPLABELPREFIXLENGTH | 91 | unsigned8 |
SRCTRAFFICINDEX | 92 | unsigned32 |
DSTTRAFFICINDEX | 93 | unsigned32 |
APPLICATIONDESCRIPTION | 94 | string |
APPLICATIONID | 95 | octetArray |
APPLICATIONNAME | 96 | string |
POSTIPDIFFSERVCODEPOINT | 98 | unsigned8 |
MULTICASTREPLICATIONFACTOR | 99 | unsigned32 |
CLASSNAME | 100 | string |
CLASSIFICATIONENGINEID | 101 | unsigned8 |
LAYER2PACKETSECTIONOFFSET | 102 | unsigned16 |
LAYER2PACKETSECTIONSIZE | 103 | unsigned16 |
LAYER2PACKETSECTIONDATA | 104 | octetArray |
BGPNEXTADJACENTASNUMBER | 128 | unsigned32 |
BGPPREVADJACENTASNUMBER | 129 | unsigned32 |
EXPORTERIPV4ADDRESS | 130 | ipv4Address |
EXPORTERIPV6ADDRESS | 131 | ipv6Address |
DROPPEDOCTETDELTACOUNT | 132 | unsigned64 |
DROPPEDPACKETDELTACOUNT | 133 | unsigned64 |
DROPPEDOCTETTOTALCOUNT | 134 | unsigned64 |
DROPPEDPACKETTOTALCOUNT | 135 | unsigned64 |
FLOWENDREASON | 136 | unsigned8 |
COMMONPROPERTIESID | 137 | unsigned64 |
OBSERVATIONPOINTID | 138 | unsigned64 |
ICMPTYPECODEIPV6 | 139 | unsigned16 |
MPLSTOPLABELIPV6ADDRESS | 140 | ipv6Address |
LINECARDID | 141 | unsigned32 |
PORTID | 142 | unsigned32 |
METERINGPROCESSID | 143 | unsigned32 |
EXPORTINGPROCESSID | 144 | unsigned32 |
TEMPLATEID | 145 | unsigned16 |
WLANCHANNELID | 146 | unsigned8 |
WLANSSID | 147 | string |
FLOWID | 148 | unsigned64 |
OBSERVATIONDOMAINID | 149 | unsigned32 |
FLOWSTARTSECONDS | 150 | dateTimeSeconds |
FLOWENDSECONDS | 151 | dateTimeSeconds |
FLOWSTARTMILLISECONDS | 152 | dateTimeMilliseconds |
FLOWENDMILLISECONDS | 153 | dateTimeMilliseconds |
FLOWSTARTMICROSECONDS | 154 | dateTimeMicroseconds |
FLOWENDMICROSECONDS | 155 | dateTimeMicroseconds |
FLOWSTARTNANOSECONDS | 156 | dateTimeNanoseconds |
FLOWENDNANOSECONDS | 157 | dateTimeNanoseconds |
FLOWSTARTDELTAMICROSECONDS | 158 | unsigned32 |
FLOWENDDELTAMICROSECONDS | 159 | unsigned32 |
SYSTEMINITTIMEMILLISECONDS | 160 | dateTimeMilliseconds |
FLOWDURATIONMILLISECONDS | 161 | unsigned32 |
FLOWDURATIONMICROSECONDS | 162 | unsigned32 |
OBSERVEDFLOWTOTALCOUNT | 163 | unsigned64 |
IGNOREDPACKETTOTALCOUNT | 164 | unsigned64 |
IGNOREDOCTETTOTALCOUNT | 165 | unsigned64 |
NOTSENTFLOWTOTALCOUNT | 166 | unsigned64 |
NOTSENTPACKETTOTALCOUNT | 167 | unsigned64 |
NOTSENTOCTETTOTALCOUNT | 168 | unsigned64 |
DESTINATIONIPV6PREFIX | 169 | ipv6Address |
SOURCEIPV6PREFIX | 170 | ipv6Address |
POSTOCTETTOTALCOUNT | 171 | unsigned64 |
POSTPACKETTOTALCOUNT | 172 | unsigned64 |
FLOWKEYINDICATOR | 173 | unsigned64 |
POSTMCASTPACKETTOTALCOUNT | 174 | unsigned64 |
POSTMCASTOCTETTOTALCOUNT | 175 | unsigned64 |
ICMPTYPEIPV4 | 176 | unsigned8 |
ICMPCODEIPV4 | 177 | unsigned8 |
ICMPTYPEIPV6 | 178 | unsigned8 |
ICMPCODEIPV6 | 179 | unsigned8 |
UDPSOURCEPORT | 180 | unsigned16 |
UDPDESTINATIONPORT | 181 | unsigned16 |
TCPSOURCEPORT | 182 | unsigned16 |
TCPDESTINATIONPORT | 183 | unsigned16 |
TCPSEQUENCENUMBER | 184 | unsigned32 |
TCPACKNOWLEDGEMENTNUMBER | 185 | unsigned32 |
TCPWINDOWSIZE | 186 | unsigned16 |
TCPURGENTPOINTER | 187 | unsigned16 |
TCPHEADERLENGTH | 188 | unsigned8 |
IPHEADERLENGTH | 189 | unsigned8 |
TOTALLENGTHIPV4 | 190 | unsigned16 |
PAYLOADLENGTHIPV6 | 191 | unsigned16 |
IPTTL | 192 | unsigned8 |
NEXTHEADERIPV6 | 193 | unsigned8 |
MPLSPAYLOADLENGTH | 194 | unsigned32 |
IPDIFFSERVCODEPOINT | 195 | unsigned8 |
IPPRECEDENCE | 196 | unsigned8 |
FRAGMENTFLAGS | 197 | unsigned8 |
OCTETDELTASUMOFSQUARES | 198 | unsigned64 |
OCTETTOTALSUMOFSQUARES | 199 | unsigned64 |
MPLSTOPLABELTTL | 200 | unsigned8 |
MPLSLABELSTACKLENGTH | 201 | unsigned32 |
MPLSLABELSTACKDEPTH | 202 | unsigned32 |
MPLSTOPLABELEXP | 203 | unsigned8 |
IPPAYLOADLENGTH | 204 | unsigned32 |
UDPMESSAGELENGTH | 205 | unsigned16 |
ISMULTICAST | 206 | unsigned8 |
IPV4IHL | 207 | unsigned8 |
IPV4OPTIONS | 208 | unsigned32 |
TCPOPTIONS | 209 | unsigned64 |
PADDINGOCTETS | 210 | octetArray |
COLLECTORIPV4ADDRESS | 211 | ipv4Address |
COLLECTORIPV6ADDRESS | 212 | ipv6Address |
EXPORTINTERFACE | 213 | unsigned32 |
EXPORTPROTOCOLVERSION | 214 | unsigned8 |
EXPORTTRANSPORTPROTOCOL | 215 | unsigned8 |
COLLECTORTRANSPORTPORT | 216 | unsigned16 |
EXPORTERTRANSPORTPORT | 217 | unsigned16 |
TCPSYNTOTALCOUNT | 218 | unsigned64 |
TCPFINTOTALCOUNT | 219 | unsigned64 |
TCPRSTTOTALCOUNT | 220 | unsigned64 |
TCPPSHTOTALCOUNT | 221 | unsigned64 |
TCPACKTOTALCOUNT | 222 | unsigned64 |
TCPURGTOTALCOUNT | 223 | unsigned64 |
IPTOTALLENGTH | 224 | unsigned64 |
POSTNATSOURCEIPV4ADDRESS | 225 | ipv4Address |
POSTNATDESTINATIONIPV4ADDRESS | 226 | ipv4Address |
POSTNAPTSOURCETRANSPORTPORT | 227 | unsigned16 |
POSTNAPTDESTINATIONTRANSPORTPORT | 228 | unsigned16 |
NATORIGINATINGADDRESSREALM | 229 | unsigned8 |
NATEVENT | 230 | unsigned8 |
INITIATOROCTETS | 231 | unsigned64 |
RESPONDEROCTETS | 232 | unsigned64 |
FIREWALLEVENT | 233 | unsigned8 |
INGRESSVRFID | 234 | unsigned32 |
EGRESSVRFID | 235 | unsigned32 |
VRFNAME | 236 | string |
POSTMPLSTOPLABELEXP | 237 | unsigned8 |
TCPWINDOWSCALE | 238 | unsigned16 |
BIFLOWDIRECTION | 239 | unsigned8 |
ETHERNETHEADERLENGTH | 240 | unsigned8 |
ETHERNETPAYLOADLENGTH | 241 | unsigned16 |
ETHERNETTOTALLENGTH | 242 | unsigned16 |
DOT1QVLANID | 243 | unsigned16 |
DOT1QPRIORITY | 244 | unsigned8 |
DOT1QCUSTOMERVLANID | 245 | unsigned16 |
DOT1QCUSTOMERPRIORITY | 246 | unsigned8 |
METROEVCID | 247 | string |
METROEVCTYPE | 248 | unsigned8 |
PSEUDOWIREID | 249 | unsigned32 |
PSEUDOWIRETYPE | 250 | unsigned16 |
PSEUDOWIRECONTROLWORD | 251 | unsigned32 |
INGRESSPHYSICALINTERFACE | 252 | unsigned32 |
EGRESSPHYSICALINTERFACE | 253 | unsigned32 |
POSTDOT1QVLANID | 254 | unsigned16 |
POSTDOT1QCUSTOMERVLANID | 255 | unsigned16 |
ETHERNETTYPE | 256 | unsigned16 |
POSTIPPRECEDENCE | 257 | unsigned8 |
COLLECTIONTIMEMILLISECONDS | 258 | dateTimeMilliseconds |
EXPORTSCTPSTREAMID | 259 | unsigned16 |
MAXEXPORTSECONDS | 260 | dateTimeSeconds |
MAXFLOWENDSECONDS | 261 | dateTimeSeconds |
MESSAGEMD5CHECKSUM | 262 | octetArray |
MESSAGESCOPE | 263 | unsigned8 |
MINEXPORTSECONDS | 264 | dateTimeSeconds |
MINFLOWSTARTSECONDS | 265 | dateTimeSeconds |
OPAQUEOCTETS | 266 | octetArray |
SESSIONSCOPE | 267 | unsigned8 |
MAXFLOWENDMICROSECONDS | 268 | dateTimeMicroseconds |
MAXFLOWENDMILLISECONDS | 269 | dateTimeMilliseconds |
MAXFLOWENDNANOSECONDS | 270 | dateTimeNanoseconds |
MINFLOWSTARTMICROSECONDS | 271 | dateTimeMicroseconds |
MINFLOWSTARTMILLISECONDS | 272 | dateTimeMilliseconds |
MINFLOWSTARTNANOSECONDS | 273 | dateTimeNanoseconds |
COLLECTORCERTIFICATE | 274 | octetArray |
EXPORTERCERTIFICATE | 275 | octetArray |
DATARECORDSRELIABILITY | 276 | boolean |
OBSERVATIONPOINTTYPE | 277 | unsigned8 |
NEWCONNECTIONDELTACOUNT | 278 | unsigned32 |
CONNECTIONSUMDURATIONSECONDS | 279 | unsigned64 |
CONNECTIONTRANSACTIONID | 280 | unsigned64 |
POSTNATSOURCEIPV6ADDRESS | 281 | ipv6Address |
POSTNATDESTINATIONIPV6ADDRESS | 282 | ipv6Address |
NATPOOLID | 283 | unsigned32 |
NATPOOLNAME | 284 | string |
ANONYMIZATIONFLAGS | 285 | unsigned16 |
ANONYMIZATIONTECHNIQUE | 286 | unsigned16 |
INFORMATIONELEMENTINDEX | 287 | unsigned16 |
P2PTECHNOLOGY | 288 | string |
TUNNELTECHNOLOGY | 289 | string |
ENCRYPTEDTECHNOLOGY | 290 | string |
BASICLIST | 291 | basicList |
SUBTEMPLATELIST | 292 | subTemplateList |
SUBTEMPLATEMULTILIST | 293 | subTemplateMultiList |
BGPVALIDITYSTATE | 294 | unsigned8 |
IPSECSPI | 295 | unsigned32 |
GREKEY | 296 | unsigned32 |
NATTYPE | 297 | unsigned8 |
INITIATORPACKETS | 298 | unsigned64 |
RESPONDERPACKETS | 299 | unsigned64 |
OBSERVATIONDOMAINNAME | 300 | string |
SELECTIONSEQUENCEID | 301 | unsigned64 |
SELECTORID | 302 | unsigned64 |
INFORMATIONELEMENTID | 303 | unsigned16 |
SELECTORALGORITHM | 304 | unsigned16 |
SAMPLINGPACKETINTERVAL | 305 | unsigned32 |
SAMPLINGPACKETSPACE | 306 | unsigned32 |
SAMPLINGTIMEINTERVAL | 307 | unsigned32 |
SAMPLINGTIMESPACE | 308 | unsigned32 |
SAMPLINGSIZE | 309 | unsigned32 |
SAMPLINGPOPULATION | 310 | unsigned32 |
SAMPLINGPROBABILITY | 311 | float64 |
DATALINKFRAMESIZE | 312 | unsigned16 |
IPHEADERPACKETSECTION | 313 | octetArray |
IPPAYLOADPACKETSECTION | 314 | octetArray |
DATALINKFRAMESECTION | 315 | octetArray |
MPLSLABELSTACKSECTION | 316 | octetArray |
MPLSPAYLOADPACKETSECTION | 317 | octetArray |
SELECTORIDTOTALPKTSOBSERVED | 318 | unsigned64 |
SELECTORIDTOTALPKTSSELECTED | 319 | unsigned64 |
ABSOLUTEERROR | 320 | float64 |
RELATIVEERROR | 321 | float64 |
OBSERVATIONTIMESECONDS | 322 | dateTimeSeconds |
OBSERVATIONTIMEMILLISECONDS | 323 | dateTimeMilliseconds |
OBSERVATIONTIMEMICROSECONDS | 324 | dateTimeMicroseconds |
OBSERVATIONTIMENANOSECONDS | 325 | dateTimeNanoseconds |
DIGESTHASHVALUE | 326 | unsigned64 |
HASHIPPAYLOADOFFSET | 327 | unsigned64 |
HASHIPPAYLOADSIZE | 328 | unsigned64 |
HASHOUTPUTRANGEMIN | 329 | unsigned64 |
HASHOUTPUTRANGEMAX | 330 | unsigned64 |
HASHSELECTEDRANGEMIN | 331 | unsigned64 |
HASHSELECTEDRANGEMAX | 332 | unsigned64 |
HASHDIGESTOUTPUT | 333 | boolean |
HASHINITIALISERVALUE | 334 | unsigned64 |
SELECTORNAME | 335 | string |
UPPERCILIMIT | 336 | float64 |
LOWERCILIMIT | 337 | float64 |
CONFIDENCELEVEL | 338 | float64 |
INFORMATIONELEMENTDATATYPE | 339 | unsigned8 |
INFORMATIONELEMENTDESCRIPTION | 340 | string |
INFORMATIONELEMENTNAME | 341 | string |
INFORMATIONELEMENTRANGEBEGIN | 342 | unsigned64 |
INFORMATIONELEMENTRANGEEND | 343 | unsigned64 |
INFORMATIONELEMENTSEMANTICS | 344 | unsigned8 |
INFORMATIONELEMENTUNITS | 345 | unsigned16 |
PRIVATEENTERPRISENUMBER | 346 | unsigned32 |
VIRTUALSTATIONINTERFACEID | 347 | octetArray |
VIRTUALSTATIONINTERFACENAME | 348 | string |
VIRTUALSTATIONUUID | 349 | octetArray |
VIRTUALSTATIONNAME | 350 | string |
LAYER2SEGMENTID | 351 | unsigned64 |
LAYER2OCTETDELTACOUNT | 352 | unsigned64 |
LAYER2OCTETTOTALCOUNT | 353 | unsigned64 |
INGRESSUNICASTPACKETTOTALCOUNT | 354 | unsigned64 |
INGRESSMULTICASTPACKETTOTALCOUNT | 355 | unsigned64 |
INGRESSBROADCASTPACKETTOTALCOUNT | 356 | unsigned64 |
EGRESSUNICASTPACKETTOTALCOUNT | 357 | unsigned64 |
EGRESSBROADCASTPACKETTOTALCOUNT | 358 | unsigned64 |
MONITORINGINTERVALSTARTMILLISECONDS | 359 | dateTimeMilliseconds |
MONITORINGINTERVALENDMILLISECONDS | 360 | dateTimeMilliseconds |
PORTRANGESTART | 361 | unsigned16 |
PORTRANGEEND | 362 | unsigned16 |
PORTRANGESTEPSIZE | 363 | unsigned16 |
PORTRANGENUMPORTS | 364 | unsigned16 |
STAMACADDRESS | 365 | macAddress |
STAIPV4ADDRESS | 366 | ipv4Address |
WTPMACADDRESS | 367 | macAddress |
INGRESSINTERFACETYPE | 368 | unsigned32 |
EGRESSINTERFACETYPE | 369 | unsigned32 |
RTPSEQUENCENUMBER | 370 | unsigned16 |
USERNAME | 371 | string |
APPLICATIONCATEGORYNAME | 372 | string |
APPLICATIONSUBCATEGORYNAME | 373 | string |
APPLICATIONGROUPNAME | 374 | string |
ORIGINALFLOWSPRESENT | 375 | unsigned64 |
ORIGINALFLOWSINITIATED | 376 | unsigned64 |
ORIGINALFLOWSCOMPLETED | 377 | unsigned64 |
DISTINCTCOUNTOFSOURCEIPADDRESS | 378 | unsigned64 |
DISTINCTCOUNTOFDESTINATIONIPADDRESS | 379 | unsigned64 |
DISTINCTCOUNTOFSOURCEIPV4ADDRESS | 380 | unsigned32 |
DISTINCTCOUNTOFDESTINATIONIPV4ADDRESS | 381 | unsigned32 |
DISTINCTCOUNTOFSOURCEIPV6ADDRESS | 382 | unsigned64 |
DISTINCTCOUNTOFDESTINATIONIPV6ADDRESS | 383 | unsigned64 |
VALUEDISTRIBUTIONMETHOD | 384 | unsigned8 |
RFC3550JITTERMILLISECONDS | 385 | unsigned32 |
RFC3550JITTERMICROSECONDS | 386 | unsigned32 |
RFC3550JITTERNANOSECONDS | 387 | unsigned32 |
DOT1QDEI | 388 | boolean |
DOT1QCUSTOMERDEI | 389 | boolean |
FLOWSELECTORALGORITHM | 390 | unsigned16 |
FLOWSELECTEDOCTETDELTACOUNT | 391 | unsigned64 |
FLOWSELECTEDPACKETDELTACOUNT | 392 | unsigned64 |
FLOWSELECTEDFLOWDELTACOUNT | 393 | unsigned64 |
SELECTORIDTOTALFLOWSOBSERVED | 394 | unsigned64 |
SELECTORIDTOTALFLOWSSELECTED | 395 | unsigned64 |
SAMPLINGFLOWINTERVAL | 396 | unsigned64 |
SAMPLINGFLOWSPACING | 397 | unsigned64 |
FLOWSAMPLINGTIMEINTERVAL | 398 | unsigned64 |
FLOWSAMPLINGTIMESPACING | 399 | unsigned64 |
HASHFLOWDOMAIN | 400 | unsigned16 |
TRANSPORTOCTETDELTACOUNT | 401 | unsigned64 |
TRANSPORTPACKETDELTACOUNT | 402 | unsigned64 |
ORIGINALEXPORTERIPV4ADDRESS | 403 | ipv4Address |
ORIGINALEXPORTERIPV6ADDRESS | 404 | ipv6Address |
ORIGINALOBSERVATIONDOMAINID | 405 | unsigned32 |
INTERMEDIATEPROCESSID | 406 | unsigned32 |
IGNOREDDATARECORDTOTALCOUNT | 407 | unsigned64 |
DATALINKFRAMETYPE | 408 | unsigned16 |
SECTIONOFFSET | 409 | unsigned16 |
SECTIONEXPORTEDOCTETS | 410 | unsigned16 |
DOT1QSERVICEINSTANCETAG | 411 | octetArray |
DOT1QSERVICEINSTANCEID | 412 | unsigned32 |
DOT1QSERVICEINSTANCEPRIORITY | 413 | unsigned8 |
DOT1QCUSTOMERSOURCEMACADDRESS | 414 | macAddress |
DOT1QCUSTOMERDESTINATIONMACADDRESS | 415 | macAddress |
POSTLAYER2OCTETDELTACOUNT | 417 | unsigned64 |
POSTMCASTLAYER2OCTETDELTACOUNT | 418 | unsigned64 |
POSTLAYER2OCTETTOTALCOUNT | 420 | unsigned64 |
POSTMCASTLAYER2OCTETTOTALCOUNT | 421 | unsigned64 |
MINIMUMLAYER2TOTALLENGTH | 422 | unsigned64 |
MAXIMUMLAYER2TOTALLENGTH | 423 | unsigned64 |
DROPPEDLAYER2OCTETDELTACOUNT | 424 | unsigned64 |
DROPPEDLAYER2OCTETTOTALCOUNT | 425 | unsigned64 |
IGNOREDLAYER2OCTETTOTALCOUNT | 426 | unsigned64 |
NOTSENTLAYER2OCTETTOTALCOUNT | 427 | unsigned64 |
LAYER2OCTETDELTASUMOFSQUARES | 428 | unsigned64 |
LAYER2OCTETTOTALSUMOFSQUARES | 429 | unsigned64 |
LAYER2FRAMEDELTACOUNT | 430 | unsigned64 |
LAYER2FRAMETOTALCOUNT | 431 | unsigned64 |
PSEUDOWIREDESTINATIONIPV4ADDRESS | 432 | ipv4Address |
IGNOREDLAYER2FRAMETOTALCOUNT | 433 | unsigned64 |
MIBOBJECTVALUEINTEGER | 434 | signed32 |
MIBOBJECTVALUEOCTETSTRING | 435 | octetArray |
MIBOBJECTVALUEOID | 436 | octetArray |
MIBOBJECTVALUEBITS | 437 | octetArray |
MIBOBJECTVALUEIPADDRESS | 438 | ipv4Address |
MIBOBJECTVALUECOUNTER | 439 | unsigned64 |
MIBOBJECTVALUEGAUGE | 440 | unsigned32 |
MIBOBJECTVALUETIMETICKS | 441 | unsigned32 |
MIBOBJECTVALUEUNSIGNED | 442 | unsigned32 |
MIBOBJECTVALUETABLE | 443 | subTemplateList |
MIBOBJECTVALUEROW | 444 | subTemplateList |
MIBOBJECTIDENTIFIER | 445 | octetArray |
MIBSUBIDENTIFIER | 446 | unsigned32 |
MIBINDEXINDICATOR | 447 | unsigned64 |
MIBCAPTURETIMESEMANTICS | 448 | unsigned8 |
MIBCONTEXTENGINEID | 449 | octetArray |
MIBCONTEXTNAME | 450 | string |
MIBOBJECTNAME | 451 | string |
MIBOBJECTDESCRIPTION | 452 | string |
MIBOBJECTSYNTAX | 453 | string |
MIBMODULENAME | 454 | string |
MOBILEIMSI | 455 | string |
MOBILEMSISDN | 456 | string |
HTTPSTATUSCODE | 457 | unsigned16 |
SOURCETRANSPORTPORTSLIMIT | 458 | unsigned16 |
HTTPREQUESTMETHOD | 459 | string |
HTTPREQUESTHOST | 460 | string |
HTTPREQUESTTARGET | 461 | string |
HTTPMESSAGEVERSION | 462 | string |
NATINSTANCEID | 463 | unsigned32 |
INTERNALADDRESSREALM | 464 | octetArray |
EXTERNALADDRESSREALM | 465 | octetArray |
NATQUOTAEXCEEDEDEVENT | 466 | unsigned32 |
NATTHRESHOLDEVENT | 467 | unsigned32 |
HTTPUSERAGENT | 468 | string |
HTTPCONTENTTYPE | 469 | string |
HTTPREASONPHRASE | 470 | string |
MAXSESSIONENTRIES | 471 | unsigned32 |
MAXBIBENTRIES | 472 | unsigned32 |
MAXENTRIESPERUSER | 473 | unsigned32 |
MAXSUBSCRIBERS | 474 | unsigned32 |
MAXFRAGMENTSPENDINGREASSEMBLY | 475 | unsigned32 |
ADDRESSPOOLHIGHTHRESHOLD | 476 | unsigned32 |
ADDRESSPOOLLOWTHRESHOLD | 477 | unsigned32 |
ADDRESSPORTMAPPINGHIGHTHRESHOLD | 478 | unsigned32 |
ADDRESSPORTMAPPINGLOWTHRESHOLD | 479 | unsigned32 |
ADDRESSPORTMAPPINGPERUSERHIGHTHRESHOLD | 480 | unsigned32 |
GLOBALADDRESSMAPPINGHIGHTHRESHOLD | 481 | unsigned32 |
VPNIDENTIFIER | 482 | octetArray |
BGPCOMMUNITY | 483 | unsigned32 |
BGPSOURCECOMMUNITYLIST | 484 | basicList |
BGPDESTINATIONCOMMUNITYLIST | 485 | basicList |
BGPEXTENDEDCOMMUNITY | 486 | octetArray |
BGPSOURCEEXTENDEDCOMMUNITYLIST | 487 | basicList |
BGPDESTINATIONEXTENDEDCOMMUNITYLIST | 488 | basicList |
BGPLARGECOMMUNITY | 489 | octetArray |
BGPSOURCELARGECOMMUNITYLIST | 490 | basicList |
BGPDESTINATIONLARGECOMMUNITYLIST | 491 | basicList |
To use the Netflow v9/IPFIX UDF, you first need to define it as a function. See the topics Writing a Java UDF in the Integration Guide and CREATE FUNCTION in the SQLstream Streaming SQL Reference Guide for more details on defining functions.
A simple declaration for determining the netflow v9/IPFIX parameters is as follows:
Create the UDF netflowCollector as below in the below example.
Configure the router/switch to dump the required CFLOW data to particular IP and Port.
Execute the UDF to listen to the IP and Port configured in step 2 above.
To execute the UDF in the sqllineClient, three parameters are required to be passed, the details are as followed :
CREATE OR REPLACE SCHEMA test;
DROP SCHEMA test CASCADE;
CREATE OR REPLACE SCHEMA test;
SET SCHEMA 'test';
SET PATH 'test';
create or replace function netflowCollector(bind_address varchar(128), port varchar(64), file_path varchar(250))
returns table(
ROWTIME TIMESTAMP NOT NULL,
REPORTER BINARY(16),
SCOPE_SYSTEM BINARY(4),
OCTETDELTACOUNT BIGINT,
PACKETDELTACOUNT BIGINT,
PROTOCOLIDENTIFIER TINYINT,
SOURCETRANSPORTPORT INT,
SOURCEIPV4ADDRES BINARY(4),
SOURCEIPV4PREFIXLENGTH TINYINT,
INGRESSINTERFACE INT,
DESTINATIONTRANSPORTPORT INT,
DESTINATIONIPV4PREFIXLENGTH TINYINT,
EGRESSINTERFACE INT,
IPNEXTHOPIPV4ADDRESS BINARY(4),
BGPSOURCEASNUMBER INT,
BGPDESTINATIONASNUMBER INT,
BGPNEXTHOPIPV4ADDRESS BINARY(4),
FLOWENDSYSUPTIME BIGINT,
FLOWSTARTSYSUPTIME BIGINT,
POSTOCTETDELTACOUNT BIGINT,
POSTPACKETDELTACOUNT BIGINT,
SOURCEIPV6ADDRESS BINARY(16),
DESTINATIONIPV6ADDRESS BINARY(16),
SOURCEIPV6PREFIXLENGTH TINYINT,
DESTINATIONIPV6PREFIXLENGTH TINYINT,
FLOWLABELIPV6 BIGINT,
ICMPTYPECODEIPV4 INT,
SAMPLINGALGORITHM TINYINT,
EXPORTEDOCTETTOTALCOUNT BIGINT,
EXPORTEDMESSAGETOTALCOUNT BIGINT,
EXPORTEDFLOWRECORDTOTALCOUNT BIGINT,
SAMPLERID TINYINT,
SAMPLERMODE TINYINT,
SAMPLERRANDOMINTERVAL BIGINT,
SAMPLERNAME VARCHAR(6),
FORWARDINGSTATUS TINYINT
)
LANGUAGE EXTERNAL
NO SQL
NO STATE
EXTERNAL NAME 'plugin/netflow';
--Create the View by calling the created UDF
create or replace view nf as select stream * from stream(test.netflowCollector('127.0.0.1', '2058',''));
--Select the data from the stream for the selected columns mentioned at the time of creating the UDF
select stream rowtime, * from nf ;
Netflow v9 Sample Output :
ROWTIME 2022-05-10 03:53:04.635
REPORTER
SCOPE_SYSTEM
OCTETDELTACOUNT 273
PACKETDELTACOUNT 3
PROTOCOLIDENTIFIER 17
SOURCETRANSPORTPORT 58271
SOURCEIPV4ADDRES
SOURCEIPV4PREFIXLENGTH
INGRESSINTERFACE 416
DESTINATIONTRANSPORTPORT 443
DESTINATIONIPV4PREFIXLENGTH
EGRESSINTERFACE 829
IPNEXTHOPIPV4ADDRESS
BGPSOURCEASNUMBER 0
BGPDESTINATIONASNUMBER 0
BGPNEXTHOPIPV4ADDRESS
FLOWENDSYSUPTIME 2359276866
FLOWSTARTSYSUPTIME 2359256525
POSTOCTETDELTACOUNT
POSTPACKETDELTACOUNT
SOURCEIPV6ADDRESS 26000001928B2BAF6C89B0EE81DB3793
DESTINATIONIPV6ADDRESS 2607F8B04017000A0000000000000018
SOURCEIPV6PREFIXLENGTH 44
DESTINATIONIPV6PREFIXLENGTH 32
FLOWLABELIPV6 0
ICMPTYPECODEIPV4
SAMPLINGALGORITHM
EXPORTEDOCTETTOTALCOUNT
EXPORTEDMESSAGETOTALCOUNT
EXPORTEDFLOWRECORDTOTALCOUNT
SAMPLERID 4
SAMPLERMODE
SAMPLERRANDOMINTERVAL
SAMPLERNAME
FORWARDINGSTATUS 64
CREATE OR REPLACE SCHEMA test;
DROP SCHEMA test CASCADE;
CREATE OR REPLACE SCHEMA test;
SET SCHEMA 'test';
create or replace function netflowCollector(bind_address varchar(128), port varchar(64), file_path varchar(250))
returns table(
ROWTIME TIMESTAMP NOT NULL,
NETFLOW_VERSION TINYINT,
EXPORT_TIME BIGINT,
SESSION_ID BIGINT,
TIMESTAMPS BIGINT,
NAT_BINDING_TIMER INT,
GMT_OFFSET CHAR(5),
PORT_RNG_ALLOC_TIMESTAMP BIGINT,
PORT_RNG_DEALLOC_TIMESTAMP BIGINT,
LAST_FLOW_TIMESTAMP BIGINT,
RAT_TYPE SMALLINT,
NETWORK_INST_NAME VARCHAR(65535),
ROUTE_CNXT_NAME VARCHAR(65535),
IMEI CHAR(16),
BND_TERMINATION_CAUSE SMALLINT,
UPF_NAME VARCHAR(65535),
SESSION_EVENT SMALLINT,
SESSION_PER_PUB_IP SMALLINT,
CHARGING_ID VARCHAR(65535),
SERVC_NW_ID VARCHAR(65535),
SGWC_IPV4_ADDR BINARY(4),
SGWC_IPV6_ADDR BINARY(16),
PGWC_IPV4_ADDR BINARY(4),
PGWC_IPV6_ADDR BINARY(16),
SUPI VARCHAR(65535),
GPSI VARCHAR(65535),
APN_NAME VARCHAR(65535),
NAT_POOL_NAME VARCHAR(65535),
SOURCEIPV4ADDRESS BINARY(4),
SOURCEIPV6ADDRESS BINARY(16),
MOBILEIMSI VARCHAR(65535),
MOBILEMSISDN VARCHAR(65535),
NATEVENT SMALLINT,
POSTNATSOURCEIPV4ADDRESS BINARY(4),
PORTRANGESTART INT,
PORTRANGEEND INT,
NATINSTANCEID BIGINT
)
LANGUAGE EXTERNAL
NO SQL
NO STATE
EXTERNAL NAME 'plugin/netflow';
--Create the view by calling the UDF
create or replace view nf as select stream * from stream(test.netflowCollector('127.0.0.1', '2058','unitsql/plugins/netflow'));
--Execute the select statement to get the stream of data
select stream rowtime,* from nf ;
Netflow IPFIX Sample output :
ROWTIME 2022-05-10 03:55:35.217
NETFLOW_VERSION 10
EXPORT_TIME 1596798127
SESSION_ID 8358680908399640577
TIMESTAMPS 1596798127602
NAT_BINDING_TIMER
GMT_OFFSET +0200
PORT_RNG_ALLOC_TIMESTAMP
PORT_RNG_DEALLOC_TIMESTAMP
LAST_FLOW_TIMESTAMP
RAT_TYPE 0
NETWORK_INST_NAME
ROUTE_CNXT_NAME
IMEI 999900000000030
BND_TERMINATION_CAUSE
UPF_NAME
SESSION_EVENT 1
SESSION_PER_PUB_IP
CHARGING_ID
SERVC_NW_ID
SGWC_IPV4_ADDR 00000000
SGWC_IPV6_ADDR 00000000000000000000000000000000
PGWC_IPV4_ADDR 00000000
PGWC_IPV6_ADDR 00000000000000000000000000000000
SUPI
GPSI
APN_NAME
NAT_POOL_NAME
SOURCEIPV4ADDRESS 31000005
SOURCEIPV6ADDRESS 00000000000000000000000000000000
MOBILEIMSI 226041000000003
MOBILEMSISDN 40700000003
NATEVENT
POSTNATSOURCEIPV4ADDRESS
PORTRANGESTART
PORTRANGEEND
NATINSTANCEID
Note: The file_path should be set in order to read the enterprise related CSV file. The file path needs to be given here.