github.com/zartbot/goflow@v0.0.0-20221001080520-456f7dfad288/README.md (about) 1 # goFlow 2 3 goFlow is a golang based Netflow/IPFIX/Cisco HighSpeed Logging collector. 4 The flow record decoder is strictly followed by `RFC3954` and `RFC7011` 5 6 Notice: 7 8 This work is just used for Cisco FNF demo only and help customer verify the data collected fields. 9 Please use commercial software like StealthWatch, DNA-C vManage and LiveAction for production network. 10 11 ## Supported features & platforms 12 13 ### platforms 14 - Cisco IOS XE series all supported Flexible Netflow records(AVC-ART etc..) 15 - Cisco Viptela SDWAN 16 - Cisco IOS XE series ETTA Security feature 17 - Cisco IOS XE series Firewall and NAT HighSpeed Logging 18 19 ### features 20 21 - Server list mapping 22 - information Elemement extension 23 - malicious host mapping 24 - spam and botnet mapping 25 - DNS ptr mapping 26 - Geolocation mapping 27 28 ## Howto build and use it 29 30 check out the `example` folder and use `go build` command to build the file. 31 32 if you need to enable geolocation mapping function , please download ASN and City mmdb and put them in the `example/config/database/geolocation` folder 33 34 #### Config performance monitor on IOS XE Router 35 36 ```bash 37 Router(config)#performance monitor context foo profile application-performance 38 Router(config-perf-mon)# mode optimized 39 Router(config-perf-mon)# exporter destination 192.168.99.101 source TenGigabitEthernet0/0/0.43 40 Router(config-perf-mon)# traffic-monitor all 41 42 Router(config)# interface Gi0/1/0 43 Router(config-if)# performance monitor context foo 44 ``` 45 46 execute the goFlow example 47 48 ```bash 49 zartbot@zartbotLinux:~$ ./example 50 ``` 51 52 #### example output 53 54 ```bash 55 [IPFIX] | ==========================================================DataFrame Start======================================================== 56 [IPFIX] | AgentIP: 192.168.99.199 | ExportTime: 2021-11-28 17:01:51 +0800 CST | DomainID/SetID: [ 2560:309 ] 57 [IPFIX] | Record:[ 0]-------------------------------------------------------------------------------------------------------------------- 58 [IPFIX] | Name: applicationAttribute | Value: {Category:business-and-productivity-tools SubCategory:desktop-virtualization GroupName:other TrafficClass:multimedia-streaming BusinessRelevance:business-relevant p2pTech:no TunnelTech:no EncryptTech:yes SetName:desktop-virtualization-apps FamilyName:thin-client} 59 [IPFIX] | Name: flowinfo | Value: {IPAddress_A:10.1.1.1 Hash_A:942535861 Hash_B:3031818525 Hash_A_Mod:5861 Hash_B_Mod:8525 Port_A:0 IPAddress_B:101.25.1.1 Port_B:12346 FlowID:10.1.1.1-101.25.1.1 FlowIDWithPort:10.1.1.1:0-101.25.1.1:12346 Direction:true} 60 [IPFIX] | Name: hosta | Value: map[HostName: hostType:DNS/SSL risk:0] 61 [IPFIX] | Name: flowname | Value: <--> 62 [IPFIX] | Name: newConnectionDeltaCount | Value: 0 63 [IPFIX] | Name: services_waas_segment | Value: 16 64 [IPFIX] | Name: connectionSumDurationSeconds | Value: 10120 65 [IPFIX] | Name: conn_client_cnt_bytes_network_long | Value: 805 66 [IPFIX] | Name: applicationId | Value: 50335820 67 [IPFIX] | Name: flowDirection | Value: 1 68 [IPFIX] | Name: services_waas_passthrough_reason | Value: 0 69 [IPFIX] | Name: flowStartSysUpTime | Value: 3527529164 70 [IPFIX] | Name: ipDiffServCodePoint | Value: 48 71 [IPFIX] | Name: observationInterface | Value: GigabitEthernet0/1/0 72 [IPFIX] | Name: ingressVRFID | Value: 0 73 [IPFIX] | Name: AgentID | Value: 192.168.99.199 74 [IPFIX] | Name: Type | Value: conn 75 [IPFIX] | Name: hostb | Value: map[HostName: hostType:DNS/SSL risk:0] 76 [IPFIX] | Name: conn_client_ipv4_address | Value: 10.1.1.1 77 [IPFIX] | Name: biflowDirection | Value: 1 78 [IPFIX] | Name: monitoringIntervalStartMilliSeconds | Value: 1638090000000 79 [IPFIX] | Name: flowtype | Value: 2 80 [IPFIX] | Name: flowEndSysUpTime | Value: 3527532852 81 [IPFIX] | Name: conn_server_cnt_bytes_network_long | Value: 0 82 [IPFIX] | Name: applicationName | Value: {Name:pcoip Description:PC-over-IP - Virtual Desktop Infrastructure} 83 [IPFIX] | Name: conn_server_ipv4_address | Value: 101.25.1.1 84 [IPFIX] | Name: conn_server_trans_port | Value: 12346 85 [IPFIX] | Name: observationPointId | Value: 2 86 [IPFIX] | Name: responderPackets | Value: 0 87 [IPFIX] | Name: initiatorPackets | Value: 5 88 [IPFIX] | Name: CreateAt | Value: 2021-11-28 17:01:51 +0800 CST 89 [IPFIX] | Name: flowrisk | Value: 0 90 [IPFIX] | Name: protocolIdentifier | Value: 17 91 ``` 92 93 94 ## Development 95 96 1. Load Information Elements database. 97 98 unlike other Netflow collector, goFlow use csv file defined all information elements, 99 100 ```golang 101 iedb.ReadIANA("./config/database/informationElement/iana_ie.csv") 102 iedb.ReadCiscoIE("./config/database/informationElement/cisco_ie.csv") 103 iedb.ReadCiscoIE("./config/database/informationElement/key_ie.csv") 104 105 iedb.ShowIEDB() 106 ``` 107 csv format as below: 108 109 ``` 110 ElementID,EnterpriseNo,FieldType,DataType 111 1,0,octetDeltaCount,unsigned64 112 2,0,packetDeltaCount,unsigned64 113 3,0,deltaFlowCount,unsigned64 114 4,0,protocolIdentifier,unsigned8 115 5,0,ipClassOfService,unsigned8 116 ... 117 118 ``` 119 120 2. Start Identity Services 121 122 goFlow could leverage the existing database to search the server identitiy 123 124 ```golang 125 logrus.Info("Starting Identification service....") 126 identity.LoadServerInfo("./config/database/server/server.csv") 127 identity.LoadReputationInfo("./config/database/security/reputation.csv") 128 identity.LoadBotnetInfo("./config/database/security/spam_bot_net.csv") 129 go identity.Run() 130 ``` 131 132 3. Create Flow collectors 133 134 all flow record will be sent out to a streaming golang channel `chan *datarecord.DataFrame` 135 goFlow support multiple thread decoding, you may change the woker number in `NewIPFIXCollector` and `NewNFV9Collector` 136 137 ```golang 138 //IPFIXDataFrameChan is dataframe output channel for future streaming process. 139 var IPFIXDataFrameChan = make(chan *datarecord.DataFrame, 10000) 140 var NFv9DataFrameChan = make(chan *datarecord.DataFrame, 10000) 141 142 //input Collector 143 var ipfixCollector []*ipfix.IPFIXCollector 144 for _, port := range []int{4739, 4740} { 145 c := ipfix.NewIPFIXCollector(ipaddr, port, IPFIXDataFrameChan, 1) 146 ipfixCollector = append(ipfixCollector, c) 147 } 148 149 var nfv9Collector []*netflowv9.NetflowV9Collector 150 for _, port := range []int{2055, 2056, 2057} { 151 c := netflowv9.NewNFV9Collector(ipaddr, port, NFv9DataFrameChan, 1) 152 nfv9Collector = append(nfv9Collector, c) 153 } 154 155 var wg sync.WaitGroup 156 wg.Add(2) 157 158 for _, c := range ipfixCollector { 159 go c.Run() 160 } 161 162 for _, c := range nfv9Collector { 163 go c.Run() 164 } 165 166 ``` 167 168 4. Consuming the datastream 169 170 you may use the following method to start and comsuming the datarecord: 171 172 ```golang 173 for { 174 select { 175 case e1 := <-IPFIXDataFrameChan: 176 RecordMap(e1) 177 e1.Print("IPFIX") 178 179 case e2 := <-NFv9DataFrameChan: 180 RecordMap(e2) 181 e2.Print("NFv9") 182 } 183 } 184 ``` 185 186 You can update and correlate record with optional template and update record with addtional services here. 187 188 ```golang 189 func RecordMap(d *datarecord.DataFrame) { 190 d.TypeAssertion() 191 createAt := time.Unix(int64(d.ExportTime), 0) 192 for _, value := range d.Record { 193 value["CreateAt"] = createAt 194 value["AgentID"] = d.AgentID 195 value["Type"] = d.Type 196 //Optional Template Correlation 197 optiontemplatemap.UpdateInterfaceMap(value, d.AgentID) 198 optiontemplatemap.UpdateAppMap(value, d.AgentID) 199 optiontemplatemap.UpdateCiscoVarString(value, d.AgentID) 200 optiontemplatemap.UpdateC3PLMap(value, d.AgentID) 201 optiontemplatemap.UpdateDropCauseMap(value, d.AgentID) 202 optiontemplatemap.UpdateViptelaTLOCMap(value, d.AgentID) 203 optiontemplatemap.UpdateFWEventMap(value, d.AgentID) 204 optiontemplatemap.UpdateFWZonePairMap(value, d.AgentID) 205 optiontemplatemap.UpdateFWClassMapMap(value, d.AgentID) 206 207 //Additional Service 208 flowinfo.UpdateFlowInfo(value, true) 209 ciscoavc.ExtendField(value) 210 ciscoeta.ExtendField(value) 211 } 212 } 213 ``` 214 215 ## Export to external Database 216 217 goFlow export as a golang channel which can be easily consumed by kafka and other applications. the output module has a simple elastic-search exporter. 218 But we strongly recommend you to use `zartbot/golap` project for data pre-aggregation and in-line AI based performance scoring and reasoning.