github.com/zartbot/goflow@v0.0.0-20221001080520-456f7dfad288/README.md (about)

     1  # goFlow
     2  
     3  goFlow is a golang based Netflow/IPFIX/Cisco HighSpeed Logging collector.
     4  The flow record decoder is strictly followed by `RFC3954` and `RFC7011`
     5  
     6  Notice:
     7  
     8  This work is just used for Cisco FNF demo only and help customer verify the data collected fields.
     9  Please use commercial software like StealthWatch, DNA-C vManage and LiveAction for production network.
    10  
    11  ## Supported features & platforms 
    12  
    13  ### platforms
    14  - Cisco IOS XE series all supported Flexible Netflow records(AVC-ART etc..)
    15  - Cisco Viptela SDWAN
    16  - Cisco IOS XE series ETTA Security feature
    17  - Cisco IOS XE series Firewall and NAT HighSpeed Logging
    18  
    19  ### features 
    20  
    21  - Server list mapping
    22  - information Elemement extension 
    23  - malicious host mapping
    24  - spam and botnet mapping
    25  - DNS ptr mapping
    26  - Geolocation mapping
    27  
    28  ## Howto build and use it 
    29  
    30  check out the `example` folder and use `go build` command to build the file.
    31  
    32  if you need to enable geolocation mapping function , please download  ASN and City mmdb and put them in the `example/config/database/geolocation` folder
    33  
    34  #### Config performance monitor on IOS XE Router
    35  
    36  ```bash
    37  Router(config)#performance monitor context foo profile application-performance
    38  Router(config-perf-mon)# mode optimized
    39  Router(config-perf-mon)# exporter destination 192.168.99.101 source TenGigabitEthernet0/0/0.43 
    40  Router(config-perf-mon)# traffic-monitor all
    41  
    42  Router(config)# interface Gi0/1/0
    43  Router(config-if)# performance monitor context foo
    44  ```
    45  
    46  execute the goFlow example
    47  
    48  ```bash
    49  zartbot@zartbotLinux:~$ ./example
    50  ```
    51  
    52  #### example output
    53  
    54  ```bash
    55  [IPFIX] | ==========================================================DataFrame Start========================================================
    56  [IPFIX] | AgentIP: 192.168.99.199       | ExportTime: 2021-11-28 17:01:51 +0800 CST | DomainID/SetID: [  2560:309   ] 
    57  [IPFIX] | Record:[   0]--------------------------------------------------------------------------------------------------------------------
    58  [IPFIX] | Name: applicationAttribute                | Value: {Category:business-and-productivity-tools SubCategory:desktop-virtualization GroupName:other TrafficClass:multimedia-streaming BusinessRelevance:business-relevant p2pTech:no TunnelTech:no EncryptTech:yes SetName:desktop-virtualization-apps FamilyName:thin-client}
    59  [IPFIX] | Name: flowinfo                            | Value: {IPAddress_A:10.1.1.1 Hash_A:942535861 Hash_B:3031818525 Hash_A_Mod:5861 Hash_B_Mod:8525 Port_A:0 IPAddress_B:101.25.1.1 Port_B:12346 FlowID:10.1.1.1-101.25.1.1 FlowIDWithPort:10.1.1.1:0-101.25.1.1:12346 Direction:true}
    60  [IPFIX] | Name: hosta                               | Value: map[HostName: hostType:DNS/SSL risk:0]
    61  [IPFIX] | Name: flowname                            | Value: <-->
    62  [IPFIX] | Name: newConnectionDeltaCount             | Value: 0
    63  [IPFIX] | Name: services_waas_segment               | Value: 16
    64  [IPFIX] | Name: connectionSumDurationSeconds        | Value: 10120
    65  [IPFIX] | Name: conn_client_cnt_bytes_network_long  | Value: 805
    66  [IPFIX] | Name: applicationId                       | Value: 50335820
    67  [IPFIX] | Name: flowDirection                       | Value: 1
    68  [IPFIX] | Name: services_waas_passthrough_reason    | Value: 0
    69  [IPFIX] | Name: flowStartSysUpTime                  | Value: 3527529164
    70  [IPFIX] | Name: ipDiffServCodePoint                 | Value: 48
    71  [IPFIX] | Name: observationInterface                | Value: GigabitEthernet0/1/0
    72  [IPFIX] | Name: ingressVRFID                        | Value: 0
    73  [IPFIX] | Name: AgentID                             | Value: 192.168.99.199
    74  [IPFIX] | Name: Type                                | Value: conn
    75  [IPFIX] | Name: hostb                               | Value: map[HostName: hostType:DNS/SSL risk:0]
    76  [IPFIX] | Name: conn_client_ipv4_address            | Value: 10.1.1.1
    77  [IPFIX] | Name: biflowDirection                     | Value: 1
    78  [IPFIX] | Name: monitoringIntervalStartMilliSeconds | Value: 1638090000000
    79  [IPFIX] | Name: flowtype                            | Value: 2
    80  [IPFIX] | Name: flowEndSysUpTime                    | Value: 3527532852
    81  [IPFIX] | Name: conn_server_cnt_bytes_network_long  | Value: 0
    82  [IPFIX] | Name: applicationName                     | Value: {Name:pcoip Description:PC-over-IP - Virtual Desktop Infrastructure}
    83  [IPFIX] | Name: conn_server_ipv4_address            | Value: 101.25.1.1
    84  [IPFIX] | Name: conn_server_trans_port              | Value: 12346
    85  [IPFIX] | Name: observationPointId                  | Value: 2
    86  [IPFIX] | Name: responderPackets                    | Value: 0
    87  [IPFIX] | Name: initiatorPackets                    | Value: 5
    88  [IPFIX] | Name: CreateAt                            | Value: 2021-11-28 17:01:51 +0800 CST
    89  [IPFIX] | Name: flowrisk                            | Value: 0
    90  [IPFIX] | Name: protocolIdentifier                  | Value: 17
    91  ```
    92  
    93  
    94  ## Development
    95  
    96  1. Load Information Elements database.
    97  
    98  unlike other Netflow collector, goFlow use csv file defined all information elements,
    99  
   100  ```golang
   101  	iedb.ReadIANA("./config/database/informationElement/iana_ie.csv")
   102  	iedb.ReadCiscoIE("./config/database/informationElement/cisco_ie.csv")
   103  	iedb.ReadCiscoIE("./config/database/informationElement/key_ie.csv")
   104  
   105  	iedb.ShowIEDB()
   106  ```
   107  csv format as below:
   108  
   109  ```
   110  ElementID,EnterpriseNo,FieldType,DataType
   111  1,0,octetDeltaCount,unsigned64
   112  2,0,packetDeltaCount,unsigned64
   113  3,0,deltaFlowCount,unsigned64
   114  4,0,protocolIdentifier,unsigned8
   115  5,0,ipClassOfService,unsigned8
   116  ...
   117  
   118  ```
   119  
   120  2. Start Identity Services
   121  
   122  goFlow could leverage the existing database to search the server identitiy
   123  
   124  ```golang
   125  	logrus.Info("Starting Identification service....")
   126  	identity.LoadServerInfo("./config/database/server/server.csv")
   127  	identity.LoadReputationInfo("./config/database/security/reputation.csv")
   128  	identity.LoadBotnetInfo("./config/database/security/spam_bot_net.csv")
   129  	go identity.Run()
   130  ```
   131  
   132  3. Create Flow collectors 
   133  
   134  all flow record will be sent out to a streaming golang channel  `chan *datarecord.DataFrame`
   135  goFlow support multiple thread decoding, you may change the woker number in `NewIPFIXCollector` and `NewNFV9Collector`
   136  
   137  ```golang
   138  	//IPFIXDataFrameChan is dataframe output channel for future streaming process.
   139  	var IPFIXDataFrameChan = make(chan *datarecord.DataFrame, 10000)
   140  	var NFv9DataFrameChan = make(chan *datarecord.DataFrame, 10000)
   141  
   142  	//input Collector
   143  	var ipfixCollector []*ipfix.IPFIXCollector
   144  	for _, port := range []int{4739, 4740} {
   145  		c := ipfix.NewIPFIXCollector(ipaddr, port, IPFIXDataFrameChan, 1)
   146  		ipfixCollector = append(ipfixCollector, c)
   147  	}
   148  
   149  	var nfv9Collector []*netflowv9.NetflowV9Collector
   150  	for _, port := range []int{2055, 2056, 2057} {
   151  		c := netflowv9.NewNFV9Collector(ipaddr, port, NFv9DataFrameChan, 1)
   152  		nfv9Collector = append(nfv9Collector, c)
   153  	}
   154  
   155  	var wg sync.WaitGroup
   156  	wg.Add(2)
   157  
   158  	for _, c := range ipfixCollector {
   159  		go c.Run()
   160  	}
   161  
   162  	for _, c := range nfv9Collector {
   163  		go c.Run()
   164  	}
   165  
   166  ```    
   167  
   168  4. Consuming the datastream
   169  
   170  you may use the following method to start and comsuming the datarecord:
   171  
   172  ```golang
   173  	for {
   174  		select {
   175  		case e1 := <-IPFIXDataFrameChan:
   176  			RecordMap(e1)
   177  			e1.Print("IPFIX")
   178  
   179  		case e2 := <-NFv9DataFrameChan:
   180  			RecordMap(e2)
   181  			e2.Print("NFv9")
   182  		}
   183  	}
   184  ```
   185  
   186  You can  update and correlate record with optional template and  update record with addtional services here.
   187  
   188  ```golang
   189  func RecordMap(d *datarecord.DataFrame) {
   190  	d.TypeAssertion()
   191  	createAt := time.Unix(int64(d.ExportTime), 0)
   192  	for _, value := range d.Record {
   193  		value["CreateAt"] = createAt
   194  		value["AgentID"] = d.AgentID
   195  		value["Type"] = d.Type
   196  		//Optional Template Correlation
   197  		optiontemplatemap.UpdateInterfaceMap(value, d.AgentID)
   198  		optiontemplatemap.UpdateAppMap(value, d.AgentID)
   199  		optiontemplatemap.UpdateCiscoVarString(value, d.AgentID)
   200  		optiontemplatemap.UpdateC3PLMap(value, d.AgentID)
   201  		optiontemplatemap.UpdateDropCauseMap(value, d.AgentID)
   202  		optiontemplatemap.UpdateViptelaTLOCMap(value, d.AgentID)
   203  		optiontemplatemap.UpdateFWEventMap(value, d.AgentID)
   204  		optiontemplatemap.UpdateFWZonePairMap(value, d.AgentID)
   205  		optiontemplatemap.UpdateFWClassMapMap(value, d.AgentID)
   206  
   207  		//Additional Service
   208  		flowinfo.UpdateFlowInfo(value, true)
   209  		ciscoavc.ExtendField(value)
   210  		ciscoeta.ExtendField(value)
   211  	}
   212  }
   213  ```
   214  
   215  ## Export to external Database
   216  
   217  goFlow export as a golang channel which can be easily consumed by kafka and other applications. the output module has a simple elastic-search exporter.
   218  But we strongly recommend you to use `zartbot/golap` project for data pre-aggregation and in-line AI based performance scoring and reasoning.